ARTLA: Augmented Remote Teaching, Learning and Assessment support for academics during COVID-19
The onset of
COVID-19 has necessitated us to replace our face-to-face interaction with students with preparing and instituting “Emergency Remote Teaching, Learning and Assessment” (ERT). This year, we want to follow a differentiated model of learning and teaching called
ARTLA (Augmented Remote Teaching, Learning and Assessment) which will combine contact tuition for smaller groups of students with online elements. The principles of good T&L&A in a F2F environment also apply to the online space - it is only the mode of delivery that is different. The point of departure for this process of adaptation is to rethink the outcomes, assessment and learning opportunities of your module for the online environment. It is important to design for the active involvement of students and to encourage students to take responsibility for their own learning. The DeLTA process, adapted for the ARTLA environment, is available
here
.
When planning a curriculum, we first have to be clear about what we want our students to learn, and then teach and assess accordingly in an aligned system of instruction (Biggs, 1996). Such a system is fully criterion referenced as the outcomes define what we should be teaching; how we should be teaching it; and how we could know how well students have learned it. In aligned teaching, there is maximum consistency throughout the system.
The curriculum is stated in the form of clear outcomes which state the level of understanding required rather than simply listing the topics to be covered. The teaching methods chosen are those that are likely to realise those outcomes; you get students to do the things that the outcomes nominate.
Finally, the assessment tasks address the outcomes, so that you can test to see if, and how well the students have learned what the outcomes state they should be learning. Assessment is about how well students achieve the intended outcomes, not about how well they report back to us what we have told them.
All components in such an aligned system thus address the same agenda and support each other. The students are "entrapped" in this web of consistency, optimising the likelihood that they will engage the appropriate learning activities. Biggs (1999) calls this network 'constructive alignment'.
One sub-section of the DeLTA-resource is called the DeLTA process for curriculum planning and renewal.
This sub-section focusses on the Outcomes, Assessment and Facilitation of Learning sections all coming together as Constructive Alignment. This sub-section is adapted from the Carpe Diem process developed by Prof Gilly Salmon, University of Western Australia.
Additional resources:
Teaching Teaching & Understanding Understanding, an award-winning video from the University of Aarhus, Denmark, written and directed by Claus Brabrand. To view the 19-minute short film or to order the DVD, click on the icon.
-cycle/constructive-alignment-outcomes/Web%20resources.jpg)
A South African version of the Teaching teaching and Understanding understanding video was created at CPUT. Contact Prof James Garraway at GarrawayJ@cput.ac.za if you are interested in a copy.
-cycle/constructive-alignment-outcomes/Article.jpg)
Aligning-teaching-for-constructing-learning.pdf
Outcomes
-cycle/constructive-alignment-outcomes/DeLTA%20outcomes%202.jpg)
The formulation of outcomes is fundamentally about the creation of knowledge. What knowledge, skills and attitudes do you expect your students to have by the end of a module or programme? When we plan a module or programme we identify certain "things" our students should learn and do. We need to tell them what these "things" are. This is why we formulate outcomes. Describing a module or program in terms of what students will do and what they will learn provides information up front that helps to guide and direct the lecturer's planning and the students' learning. Outcomes indicate what a lecturer see as important. And it helps to determine what to assess during or at the end of the course or program. Outcomes allow students to make judgements about whether they have learned what they needed to.
According to Phil Race (2012) the lecturer's knowledge is just information for the student. The question is then how does information become knowledge? Information only becomes knowledge when you "do" something with it. When you describe this "doing" that the students should do – it is done in terms of outcomes. For example, students have to apply/calculate/compare/analyse/interpret/integrate the information in order for them to create their own knowledge. Knowledge creation is thus an active process which the lecturer facilitates through the identified learning opportunities.
Knowledge can be created in three different domains: cognitive or knowledge, psychomotor or skills, affective or attitudinal. Knowledge within each of these three domains can be created on different levels – from basic to advanced. Each level implies the previous lower levels, but it is important to formulate the outcomes on the highest level.
When we teach we should have a clear idea of what we want our students to learn. More specifically, on a topic by topic basis, we should be able to stipulate how well each topic needs to be understood. Accordingly, we have to state our objectives in terms that require students to demonstrate their understanding, not just simply tell us about it in invigilated exams. The first step in designing the curriculum objectives, then, is to make clear what levels of understanding we want from our students in what topics, and what performances of understanding would give us this knowledge.
Aligning-teaching-for-constructing-learning.pdf
Below are some resources that might be useful when writing / evaluating outcomes:
Taxonomies
Outcomes define the result that students must achieve as a result of learning opportunities in three domains namely: the cognitive (knowledge) the psychomotor (skills) and the affective (attitudes/values). Taxonomies organize lecturers' expected learning outcomes in these domains into a hierarchy from less to more complex and they are thus helpful when you think about the expected learning outcomes for your module or program. The most well-known and used taxonomy for the cognitive domain was created by Benjamin Bloom.
1. Taxonomy for learning in the cognitive (knowledge/know) domain (Bloom 1956 & 1999)
| | Level | Definition | Sample student learning activities – use these to formulate your outcomes |
| 1 | Remembering/ know about(lowest cognitive level) | to recall or remember facts without necessarily understanding them: observing and recalling information; knowing dates, events, places; knowing concepts & major ideas | name, arrange, define, duplicate, label list, memorize, name, order, recognize, relate, recall, reproduce, list, tell, describe, identify, show, label, collect, examine, tabulate, quote |
| 2 | Understanding/ Comprehension | to understand and interpret learned information: understanding information; grasping meaning; comparing; contrasting; ordering; grouping; inferring causes; predicting consequences; translating knowledge into new contexts | summarize, interpret, contrast, predict, associate, report, distinguish, estimate, differentiate, extend, classify, describe, discuss, explain, express, discuss, translate, review, restate, locate, recognize |
| 3 | Applying | to put ideas and concepts to work in solving problems: using concepts, theories in new situations; solving problems using required skills or knowledge | apply, demonstrate, calculate, complete, illustrate, show, solve, examine, modify, relate, change, classify, experiment, discover, choose, dramatize, employ, interpret, operate, practice, schedule, sketch, use |
| 4 | Analysing | to break information into its components to see interrelationships and ideas: seeing patterns; organizing parts; recognizing hidden meanings; identifying components | explain, compare, select, analyze, appraise, calculate, categorize, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test, separate, order, connect, classify, arrange, divide, infer |
| 5 | Evaluating | to judge the value of information based on established criteria: comparing and discriminating between ideas; assessing value of theories, presentations; making choices based on reasoned argument; verifying value of evidence; recognizing subjectivity | assess, decide, rank, measure, recommend, convince, select, judge, explain, discriminate, support, conclude, compare, appraise, argue, attach, defend, predict, rate, evaluate, conclude, summarize |
| 6 | Creating | to use creativity to compose and design something original: using old ideas to create new ones; generalizing from given facts; relating knowledge from several areas; predicting; drawing conclusions | combine, integrate, modify, rearrange, substitute, plan, create, design, invent, compose, formulate, prepare, generalize, arrange, assemble, collect, construct, develop, manage, organize, propose, set up, rewrite |
This taxonomy for the cognitive domain was originally put together in 1956 and was adapted recent years.
When one describes these cognitive levels as actions (using verbs), one gets a good idea of the intellectual skills that are characterised thereby:
- Knowledge: what do the students know; what content do they know?
- Comprehension: to what degree do students understand that which they know? How deep is their comprehension?
- Application: Can the students apply the knowledge e.g., by using it in a context other than the one in which the knowledge was originally acquired?
- Analysis: Can the students analyse information and systematise knowledge e.g., by categorising different aspects and/or comparing them with one another and arranging them according to specific criteria?
- Synthesis: Can students recognise the interdependence and links within the acquired knowledge content and can they integrate knowledge at an overarching level so that it leads to new insight?
- Evaluation: Are the students able to make their own deductions and themselves judge, for example, the merits of an intellectual argument or the quality of a creative adaptation of previous knowledge?
Bloom's Taxonomy
The Solo taxonomy
The SOLO Taxonomy helps to map levels of understanding that can be built into the intended learning outcomes and to create the assessment criteria or rubrics. Constructive alignment can be used for individual courses, for degree programmes, and at the institutional level, for aligning all teaching to graduate attributes.
As indicated earlier, outcomes should be formulated in the domains of knowledge, skills and values/attitudes. The section above dealt with the formulation of outcomes in the cognitive domain. Similar to the cognitive domain, the affective and psychomotor domains also have taxonomies.
2. Taxonomy for learning in the Affective (attitudes/values/being) domain (Rothwell & Kazanas 1992)
| | Level | Definition | Sample student learning activities – use these to formulate your outcomes |
| 1 | Receiving | Paying attention | Listen to, perceive, be alert to, show tolerance of |
| 2 | Responding | Minimal participation | Reply, answer, follow along, approve, obey, appreciate |
| 3 | Valuing | Internalising preferences, accepting values/beliefs | Attain, assume, support, attempt, participate, continue, accept |
| 4 | Organisation | Development or acquisition of a new value system | Organise, select, judge, decide, identify with, challenge |
| 5 | Characterisation | Adopting and practicing a total new philosophy of life | Belief, practice, continue, carry out, defend, dispute, join, praise, question, share, support |
3. Taxonomy for learning in the Psychomotor (skills/doing) domain ("see one do one teach one")
| | Level | Definition | Sample student learning activities – use these to formulate your outcomes |
| 1 | Perception
(consciously incompetent) | Observing behaviours involved in a task, "see one" | Watch, observe, take note, listen |
| 2 | Guided response
(consciously competent) | Performing a task with assistance, "do one" | Demonstrate, practice, use, play, implement feedback, reflect on own performance |
| 3 | Complex, overt response(unconsciously competent) | Acting without assistance, performing automatically, habitually, most advanced learned movement, "teach one" | Perform, do, use, bend, operate, shorten, stretch, give feedback to others, reflect on own performance |
This document provides a summary of the taxonomies of learning domains as discussed above: Summary of all domains.pdf
Formulating outcomes
To formulate an outcome, it is recommended that you write a set of clear statements about what the learners/students
- will know and understand;
- what they will be able to do, and at what level;
- and according to what values and with what attitudes they will think and act
after completion of the learning experience.
| This means that you do the following: |
1. Determine the course of the learning opportunity
Example: After completion of the module/programme … |
| Then spell out the following (not necessarily in this order): |
2. a subject
Example: After completion of this module you (or the candidates / students …) will …
(It is often recommended that a form of address in the first person is used). |
3. one or more verbs, preferably action verbs
Example: After completion of this module, you will be able to … identify, indicate, distinguish, compare, expose, evaluate, etc. |
4. an object
Example: After completion of this module, you will be able to identify a specific set of characteristics, elaborate a specific use of a formula, compare various things with each other, can solve a specific problem etc. |
5. a specific framework (context) or specific circumstances
Example: After completion of this module, you will be able to perform a task within a specific context or under specific circumstances (e.g., within a specific time or utilising specific resources and/or sources, with a specific aim in mind, within a specified group context etc.) |
6. a specific set of values and norms and/or attitudes which direct thought and action
Example: After completion of this module, you will be able to perform a specific task under specific circumstances with the critical awareness and social sensitivity that can be required of … |
Below two podcasts that might assist you when writing outcomes:
The Graduate Attributes as identified in the SU Teaching and Learning Strategy should also be included as outcomes:
Keep in mind...
At any level of education, some memorization of information is essential, but the mere transmission of knowledge is never an appropriate goal for a course or program. Even in the most basic and introductory of courses, expected learning outcomes should emphasize, at least, comprehension and application of knowledge. On a post-graduate level the outcomes should be on the higher cognitive levels (level 3 and up)
Ensure your learning outcomes focus not on what you as lecturer will do but on what students will be able to do at the end of the module. A phrase such as "students will be exposed to..." is not about student outcomes.
Avoid vague terms such as know, appreciate, understand, be familiar with, or learn; such terms could suggest that one has to think more carefully about what you want students to get out of your course.
Tell your students in your learning outcomes what they will be expected to do to demonstrate their understanding i.e. that they have achieved the outcome. If the outcome involves understanding, perhaps students will outline, explain, describe, model, or apply what they have learned in a new context. If the outcome involves critical or creative thinking, perhaps they will synthesize, evaluate, or extend what they have learned.
Each discipline and topic will of course have its own appropriate verbs that reflect different levels of understanding, the topic content being the objects the verbs take.
Formulating SMART outcomes (see video clip at bottom of page) could be useful:
S = specific (not fuzzy)
M = measureable (must be able to assess if reached)
A = attainable (what is possible given infrastructure and logistics)
R = relevant (linked to aim of programme)
T= time (feasible within module/semester, rotation)
An online tool that could assist with the writing of outcomes according to Bloom's taxonomy is called The Differentiator and it is available by clicking on the web icon:
