from monologue to dialogue: enhancing feedback in learning professor david nicol deputy-director...

From Monologue to Dialogue: enhancing feedback in learning

Professor David Nicol Deputy-Director

Centre for Academic Practice and Learning Enhancement (CAPLE

Director, REAP project (www.reap.ac.uk)University of Strathclyde

Queen’s University Belfast, Centre for Educational Development, 20th May 2009

NSS: Assessment and feedback (2008)

No

Survey Statement England

Scotland

Northern Ireland

5. The criteria used in marking have been clear in advance

69 69 69

6. Assessment arrangements and marking have been fair

74 74 73

7. Feedback on my work has been prompt

56 51 53

8. I received detailed comments on my work

61 52 52

9. Feedback on my work has helped clarify things I did not understand

56 51 50

22.

Overall, I am satisfied with the quality of the course

82 86 83

Feedback as ‘transmission’

Elite to mass higher educationQuality of the input message (what is written by teachers) versus the quality of interactionFeedback as monologue trying to carry the burden of dialogueStudents as passiveNSS might lead to narrow focus for interventions

Feedback as ‘dialogue’

Contingent: in response to learners’ needs. Multi-faceted: deriving from many sources (peers) and available in different formats.Catalytic: triggering inner dialogues around disciplinary concepts and ideas.Motivating: encouraging feedback seeking rather than feedback avoidance behaviours.Actionable: expressed as changes in thinking and in action

Plan

BackgroundRe-engineering Assessment Practices (REAP) projectConcepts and ideasSuggestions to enhance practiceCase study of practice from REAPGuidelines for implementation

Background

Research: HE Academy 2004: Literature review: 7 principles of good assessment and feedback practice in relation to development of learner self-regulation (Nicol and Macfarlane-Dick, 2004: 2006)Local Implementations: Scottish Funding Council 2005-7: The Reengineering Assessment Practices (REAP) project (£1m) www.reap.ac.ukPolicy and strategy: University of Strathclyde Assessment Policy and Practice Guidelines Synthesis: QAA Scotland 2009: First Year Experience: Assessment and Feedback publication (66 pages)(http://www.enhancementthemes.ac.uk/documents/firstyear/FirstYear_TransformingAssess.pdf

http://www.enhancementthemes.ac.uk/documents/firstyear/FirstYear_TransformingAssess.pdf


Re-engineering Assessment Practices project

Scottish Funding Council (£1m)Strathclyde, Glasgow and Glasgow CaledonianLarge 1st year classes (160-900 students)A range of disciplines (19 modules ~6000 students)Many technologies: online tests, simulations, discussion boards, e-portfolios, e-voting, peer/feedback software, VLE, online-offlineLearning quality and teaching efficienciesAssessment for learner self-regulation

First Year: The academic experience

What is important in the first year?

Coping with transitionUnderstanding what is requiredEngagement with academic programmesReceiving support and feedback Experiences of successFeeling in control of own learningBelief that you can succeedA sense of belonging within the academic and social culture

Based on research by Yorke (UK) and Tinto (US)

Background (1)

Gibbs, G. & Simpson, C (2004) Conditions under which assessment supports students learning, Learning and Teaching in Higher Education, 1, 3-31.

See: Formative Assessment in Science Teaching (FAST) project at: http://www.open.ac.uk/science/fdtl/

Gibbs and Simpson (2004)

Assessment tasks [Conditions 1-4]1. Capture enough study time (in and out of class)2. Are spread out evenly across timeline of study3. Lead to productive activity (deep vs surface)4. Communicate clear and high expectations

i.e concern here is with ‘steers’ about how much work to do

Background (2)

Literature ReviewNicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 34 (1), 199-218Nicol, D & Milligan, C. (2006), Rethinking technology-supported assessment practices in relation to the seven principles of good feedback practice. In C. Bryan & K. Clegg, Innovative assessment in higher education, Routledge.

BackgroundStudent Enhanced Learning through Effective Feedback [SENLEF] project funded by HE AcademyREAP project: www.reap.ac.uk

http://www.reap.ac.uk/

Rethinking assessment and feedback

1. Consider self and peers as much as the teacher as sources of assessment and feedback

Tap into different qualities than teacher can provide Saves time Provides considerable learning benefits (lifelong

learning)

2. Focus on every step of the cycle: Understanding the task criteria (Sadler, 1983)Applying what was learned in action

3. Not just written feedback: Also oral, computer, vicarious, formal and informal

Seven principles of good feedback

Good feedback should:1. Clarify what good performance is (goals, criteria,

standards).2. Facilitate the development of reflection and self-

assessment in learning 3. Deliver high quality feedback to students: that

enables them to self-correct4. Encourage peer and student-teacher and peer

dialogue around learning5. Encourage positive motivational beliefs & self

esteem through assessment6. Provide opportunities to act on feedback7. Provide information to teachers that can be used to

help shape their teaching (making learning visible)8. Source: Nicol and Macfarlane-Dick (2006)

Two super principles

SUPER-PRINCIPLE 1: time and effort on task (structured engagement) i.e. steers on how much work to do and when – Gibbs and Simpson 4 conditions

SUPER-PRINCIPLE 2: developing learner self-regulation (empowerment/self-regulation) i.e steers to encourage ownership of learning – the seven principles discussed above.

Case examples from REAP – applying these conditions/ principles

Discussion points








help shape their teaching (making learning visible)

Source: Nicol and Macfarlane-Dick (2006)



standards).

Teacher models expert performanceStudents derive criteria from exemplarsModel answersStudents create assessment criteria for a task


Good feedback should:2. Facilitate the development of reflection and self-

assessment in learning

Time on taskStudents identify what was strong and weak when they hand in an assignmentExplain concepts underpinning problem-solving questionsProvide an abstract with an essayRespond to teacher-feedback commentsSelect feedback comments relevant to a task from a list


Good feedback should:3. Delver high quality feedback comments to

students: that enables them to self-correct

The nature of feedback comments

Understandable: expressed in a language that students will understandSelective: only two or three worked out commentsSpecific: pointing to examples in the student’s submission where the feedback appliesTimely: provided in time to inform the next piece of work or happening when most usefulContextualised: framed with reference to the learning outcomes or assessment criteriaNon-judgemental: descriptive rather than evaluative, focused on learning goals not just performance goals Balanced: focused on positive as well as areas in need of improvementForward looking: suggesting how students might improve subsequent assignmentsTransferable: focused on processes, skills and self-regulatory abilities


Good feedback should:3. Delver high quality feedback comments to

students: that enables them to self-correct

‘Feed forward’ rather than feed backFeedback on processes and skillsFeedback on students’ self-assessmentsDon’t give feedback – point to problem areas


Good feedback should:4. Encourage teacher-student and peer dialogue around

learning

Students request feedback Students respond to feedback receivedAudio feedbackDiscussions of feedback in tutorials or one to one

Peer commenting and critiquingCritical reviewsElectronic voting methodsCollaborative assignments


Good feedback should:5. Encourage positive motivational beliefs and self-

esteem

Focus on learning goals rather than performance comparisonsWithhold marks until feedback responded toAvoid overuse of praiseProvide descriptive rather than evaluative feedback (reader-response feedback)Align formative and summative (e.g. MCQs)


Good feedback should:6. Provide opportunities to use feedback

Provide feedback as action pointsDrafts and redraftsReward use of feedback in a new task


Good feedback should:7. Being responsive to feedback needs

Requested feedback Online testing enabling ‘just-in-time’ teachingElectronic voting methodsOne-minute papersMonitoring discussion boards

Discussion points

Psychology

560 first year studentsMixture of psychology majors (130) and those taking psychology only for one year (430)6 topic areas, 48 lectures, 4 tutorials, 12 practicalsAssessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in experiment (5%), essay exam (66%)

Problems identified

No practice in writing skills but required in the examMore detail provided in lectures than mentioned in exams (not enough independent reading)No feedback except on MCQs (percent correct)Didn’t want to increase staff workloadWanted to improve overall exam marksAnd standard of entrant to second year

Discussion point

What would you do to improve the student experience in first year psychology?

You can use any technology (or combination of technologies) but you must consider costs and staff time constraint.

Psychology Redesign

Discussion board in WebCTStudents in 85 discussion groups of 7-8, same groups throughout yearAlso open discussion board for classFriday lectures dropped Students discover for themselves through collaboration what would have been presented in the Friday lectureSeries of online tasks

Structure of group tasks

6 cycles of 3 weeks (one cycle x major course topic)

• First week: ‘light’ written task (e.g. define terms) = 7 short answers (all answer)

• Second week = guided reading • Week three: ‘heavy’ written task: students

answer guided questions and then collaborate in writing a 700-800 word essay.

Within each week:• The Monday lecture – introducing material• Immediately after lecture, task posted online –

for delivery the following Monday• Model answers (selected from students) posted

for previous week’s task

The teaching role

Participation in the discussions was compulsory but not marked (this year there is 2% mark for participation)The course leader provided general feedback to the whole class – often motivational He encouraged students to give each other feedbackAnd he selected the model answersThe group discussions were not moderatedAround 8 teaching assistants monitored the discussions and reported non-participation to the teacher

Guidelines

Comment constructively on each other’s work. If you think that someone has missed a detail or is confused then HELP - point out how an answer might be better. This isn’t showing off – it’s being supportive – don’t leave someone stuck with half an answer or one that’s wrong or confused just because you don’t want to seem to ‘know better’ – it might be you that needs the help next time. Build answers in your online group discussion space, i.e. ‘show your working’ online where possible so that you can keep track of your progress as a group (and also so that Teaching Assistants can check on who is doing what and who isn’t).

Source: Jim Baxter, Psychology, Strathclyde University

Online Project 1 – Classical Conditioning Phenomena.

Each Group Member should read the Passer chapter from the beginning to at least as far the section which begins ‘Applications of Classical Conditioning’. Satisfy yourself that you can answer EACH of the questions below. Then agree as a group who will answer what.

Project 1 is to answer these questions as fully as you can:

1) What type of response is susceptible to Classical Conditioning? 2) Why does Extinction occur? 3) What is Spontaneous Recovery? 4) What does the phenomenon of Spontaneous Recovery tell us about the nature of Extinction in Classical Conditioning? 5) What is Generalisation? 6) What is Discrimination? 7) What is Higher Order Conditioning?

Project 9: An example of ‘heavy’ task

The Task – 800 word essay:

Assess the strengths and weaknesses of Freud’s and Eysenck’s theories of personality. Are the theories incompatible?readings suggestedquestions provided – all should tryand advice on how to divide task given

Topic: Online Project 9 - Online Group Projects 45 Date: 11 February 2007 Subject: hey?? Author: John Hey, is it OK with everyone if I post up the part of my essay, describing Freud’s theories, some of his strengths and weaknesses’

Topic: Online Project 9 - Online Group Projects 45Date: 11 February 2007

Subject: Re:hey?? Author: Donna yeah cool, il describe Eysneck (is that even how its spelt’’) is any1 else gettin confused’ freuds sayin that adult personality is influenced by parenting, either excessive or lax at each of the stages that can result in regression or fixation. whereas Eysneck is saying that differences in personality are a result of differences in the level of arousal’’so there not compatible coz Freuds- emphassing the importance of childhood experiences and Ey is saying that it is biological’’but then genetic differences account for like half of the difference or sumfin’’’

n each is emphasising the importance of bio/env and not saying that that is the only influence on adult personality’’’ AND- if the ego and the superego develop to control the impulses of the id, which is innate and unconscious then how can behaviour and personality be a result of childhood experiences if the ids pesent at birth’ ahh but then i spose that ego and superego have contact with reality.... c y am conffudled!! please help dont even no if this is anything 2 do with question, who knows.xx


Subject: Re:hey?? Author: JohnI think you are on the right lines. In my essay, I said that Eysenck’s theories of personality can be viewed in contrast to Freud’s theories of personality. I wrote that both their theories can be correct although they emphasised different things. Freud emphasised the significance of childhood experiences (the three main psychosexual stages) and the unconscious in influencing adult personality traits whereas Eysenck emphasised more the heritability of traits and psychological factors such as Psychoticism, Extraversion and Neuroticism as main influences on the personality of an individual. I hope that make more sense. That’s kinda what I wrote in my essay, anyway!


Subject: Re:hey?? Author: Donna yeh that makes sense! thanks. i havent wrote the full essay yet but il put up a summary of eysencks theory just now. well in like 10mins. ta xx

Benefits

Students worked exceptionally hardWritten responses of exceedingly high standard Discussions about learning and leaner responsibilityHigh levels of motivation: atmosphere in class improvedOnline interactions showed powerful ‘scaffolding’Feedback possible with 560 students: peer and self-feedback (model answers)Easy for tutors to monitor participationImproved mean exam performance (up from 51-57%, p<0.01)

Has it worked?

I read more about Psychology and read it earlier in each semester than I would have done without

the online projects

0

10

20

30

40

50

60

1 2 3 4 5

agreed-disagreed

Nu

mb

ers

I learned more in Psychology because of online projects than I did in my other subjects

0

10

20

30

40

50

60

1 2 3 4 5

agreed-disagreed

nu

mb

ers

Series1

I found that reading other people's contributions helped me understand Psychology

010203040

50607080

1 2 3 4 5

agreed-disagreed

nu

mb

ers

Series1

The feedback based on other students' work helped me understand how to improve my own

answers

0

10

20

30

40

50

60

1 2 3 4 5

agreed-disagreed

nu

mb

ers

Series1

Online postings/interaction

24,362 messages posted by groups Average number of postings per student 44.31067 postings to general open discussion forumStudents set up online study groups for other subjects

Structured tasks online triggered important social-cognitive processes

Relation to the Gibbs & Simpson’s four assessment conditions

1. Tasks require significant study out of class (condition 1)

2. Tasks are distributed across topics and weeks (condition 2)

3. They move students progressively to deeper levels of understanding (condition 3)

4. There are explicit goals and progressive increase in challenge (condition 4)

Relation to 7 feedback principles

1. Standard format and model answers provide progressive clarification of expectations (principle 1)

2. Students encouraged to self-assess against model answer (principle 2)

3. Course leader provides motivational and meta-level feedback and selects model answers (principle 3)

4. Online peer discussion aimed at reaching consensus is core feature of design about response (principle 4)

5. Focus on learning not just marks, sense of control/challenge enhanced motivation (principle5)

6. Repeated cycle of topics and tasks provide opportunities to act on feedback (principle 6)

7. VLE captures all interactions allowing course leader to monitor progress and adapt teaching (principle 7)

What can we learn from this case study?

Use of simple technologies (discussion board)Considerable thought gone into the learning design [which is transferable]The drivers were learning improvements rather than technology (context of use)Key finding across all REAP studies was need to balance structure and learner controlAn important finding was the way that the social and the academic processes were shown to be mutually supportive

Discussion points

EMPOWERMENT/SELF-REGULATION

SOCIALEXPERIENCE

ENGAGEMENT

Figure 1: Framework for Analysis of Assessment and Feedback practice

ACADEMICEXPERIENCE

12 Principles of Good Assessment and Feedback

Practice

Good formative assessment and feedback practices should:

1. Help clarify what good performance is (goals, criteria, standards)2. Encourage ‘time an effort’ on challenging learning tasks3. Deliver high quality feedback information that helps learners self-correct4. Provide opportunities to act on feedback 5. Ensure that summative assessment supports formative

learning processes6. Encourage interaction and dialogue around learning (peer, teacher-

student)7. Facilitate the development of self-assessment and reflection in learning8. Give choice in the topic, method, criteria, weighting or timing of

assessments.9. Involve students in decision making about assessment policy and

practice10. Support the development of learning groups and communities11. Encourage positive motivational beliefs and self-esteem12. Provide information to teachers that can be used to help shape the

teaching

Dynamics of Implementation


SOCIALEXPERIENCE

ENGAGEMENT

Principle 1: Clarify what good performance is

ACADEMICEXPERIENCE

• Students create criteria• Students add own criteria • Students identify criteria

from samples of work• Exemplars of different

performance levels provided

• Students rephrase criteria in own words

• Provide document with criteria


SOCIALEXPERIENCE

ENGAGEMENT

Principle 2: Encourage time and effort on challenging tasks

ACADEMICEXPERIENCE

• Students given large project and expected to structure their own learning

• Ask students to determine the milestones and reward them for keeping to the deadlines

• Set out a series of learning tasks with milestones


SOCIALEXPERIENCE

ENGAGEMENT

ACADEMICEXPERIENCE

6.Encourage interaction and dialogue around learning (peer and teacher-student)

2. Encourage time & effort on challenging learning tasks

+


SOCIALEXPERIENCE

ENGAGEMENT

ACADEMICEXPERIENCE

6.Encourage group discussion of how that feedback might be used in tutorials

3. Deliver individual written feedback on students’ work

+


SOCIALEXPERIENCE

ENGAGEMENT

ACADEMICEXPERIENCE

Students self-assess own performance using online MCQs

Students self-assess using MCQs and provide confidence ratings

Principle 7

MCQ and Confidence-Based Marking

Mark if correct

Penalty if wrong

Degree of certainty C=1 Low C=2 Medium C=3 High

1 2 3

0 - 2 - 6

Scoring regime for confidence-based marking

Ref: Tony Gardner-Medwin (2006), Confidence-based marking: towards deeper learning and better exams.


SOCIALEXPERIENCE

ENGAGEMENT

ACADEMICEXPERIENCE

Students self-assess own performance using online MCQs

Students self-assess using MCQs and provide confidence ratings

Students create MCQs and feedback for wrong and right answers 6. Encourage

interaction and dialogue around learning (peer and teacher-student)

+

Principle 6Principle 7

Guidelines for Implementation

1. A single principle or many?2. Active involvement of students3. Tight-loose – maintain fidelity to the principles

(tight) but encourage disciplines develop their own techniques of implementation (loose)

4. Clarify students’ responsibilities 5. Evaluate changes (process indicators)6. And where ICT can add value7. Alternate solo and group work8. Share your learning designs

My Publications

Nicol, D (2009), Transforming assessment and feedback: Enhancing integration and empowerment in the first year, Published by Quality Assurance Agency, Scotland(http://www.enhancementthemes.ac.uk/documents/firstyear/FirstYear_TransformingAssess.pdfNicol, D (2009), Assessment for learner self-regulation: Enhancing achievement in the first year using learning technologies, Assessment and Evaluation in Higher Education, 34(3), 335-352Nicol, D (2007), Laying the foundation for lifelong learning: cases studies of technology supported assessment processes in large first year classes, British Journal of Educational Technology, 38(4), 668-678Nicol, D (2007) E-assessment by design: using multiple-choice tests to good effect, Journal of Further and Higher Education.31(1), 53-64.Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge.Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 199-218.See also www.reap.ac.uk for copies.



http://www.reap.ac.uk/

From Monologue to Dialogue: enhancing feedback in learning

Interactive Keynote Session

Queen’s University Belfast, Centre for Educational Development, 20th May 2009

Feedback as ‘dialogue’

Contingent: in response to learners’ needs Multi-faceted: deriving from many sources (peers) and available in different formatsCatalytic: triggering inner dialogues around disciplinary concepts and ideasMotivating: encouraging feedback seeking rather than feedback avoidance behaviours.Actionable: expressed as changes in thinking and in action

Productive use of study time

Assessment tasks should:1. Capture enough study time (in and out of class)2. Are spread out evenly across timeline of study3. Lead to productive activity (deep vs surface)4. Communicate clear and high expectations

Concern here is with ‘steers’ about how much work to do

Source: Gibbs and Simpson (2004)

Seven feedback principles

Good feedback practice should: 1. Clarify what good performance is (goals, criteria,






help shape their teaching (making learning visible)

Source: Nicol and Macfarlane-Dick (2006)

Another Perspective (1)1-4. Students don’t engage in time on taskDon’t learn to manage their timeLess opportunity to identify misunderstandingsHave less reason to get feedback from peersP1. Unclear goals/criteriaUnder-performance by studentsFeedback doesn’t connectStudents don’t’ engage in effective self-assessmentP2. Absence of formal opportunties for self-assessmentStudents don’t learn to make evaluative judgment about

own workHave less ownership and understanding of assessment

criteriaP3. Poor quality teacher feedbackStudents become dependent on teacherFeedback does not feed-forward

Another perspective (2)P4. Lack of peer dialogueTeacher has to provide all the feedbackStudents don’t scaffold each other’s learningP5. Judgemental feedbackDamaged motivationCan lead to fixed mindsetStudents give up or leave courseP6. No clear opportunities action on feedbackStudents don’t see the benefitMake less improvement Staff time might be wastedP7. Teacher not aware of student difficultiesLess able to target feedback to student needsOver or under production of feedback

Discussion in Groups

Consider your own classes/modules:Identify any bottlenecks/issues in your own teachingHow might you apply the assessment/feedback principles to address these bottlenecks and enhance student learning?Identify any questions raised in your discussion groups (e.g. What difficulties do you envisage in application of the principles? What other recipes would you suggest?)

Plenary report backBe prepared to share a good idea with others in the plenary and/orA ‘question worth asking’ that might help illuminate any important issues.

Implementation strategy

1. The big idea: ‘learner autonomy’2. The problem: assessment 3. The principles: specified but not over-specified

solutions4. Local implementations with a common

framework5. Multi-professional support6. Evaluations of implementations 7. Dissemination internal and external8. Scaling up from course teams to departments9. Embedding

Written feedback in mass higher education

Students don’t read and/or act on commentsEvidence of poor quality comments (e.g. Duncan, 2007)No guidelines on what might be effective (see Hattie and Timperley, 2007)Multiple purposes and audiences for commentsNegative effects of feedback (Kluger & DeNisi, 1998)Matching feedback to individual needs (Pelligrino, Chudowsky and Glaser, 2001)Students viewed as passive (Boud, 2007)No evidence that improving feedback comments on their own improves learning! (despite Black and Wiliam, 1998)

Another Perspective (1)1-4. Students don’t engage in time on taskDon’t learn to manage their timeLess opportunity to identify misunderstandingsHave less reason to get feedback from peersP1. Unclear goals/criteriaUnder-performance by studentsFeedback doesn’t connectStudents don’t’ engage in effective self-assessmentP2. Absence of formal opportunties for self-assessmentStudents don’t learn to make evaluative judgment about

own workMore likely to be dependent on teacher for feedbackHave less ownership and understanding of assessment

criteriaP3. Wrong kind of teacher feedbackStudents become dependent on teacherStudents don’t believe staff interested in them

Another perspective (2)P4. Lack of peer dialogueTeacher has to provide all the feedbackStudents don’t scaffold each other’s learningP5. Negative feedbackDamaged motivationCan lead to fixed mindsetStudents give up or leave courseP6. No clear opportunities action on feedbackStudents don’t see the benefitStaff time wastedMake less improvementP7. Teacher not aware of student difficultiesLess able to target feedback to student needsOver or under production of feedback

from monologue to dialogue: enhancing feedback in learning professor david nicol deputy-director...

Documents

feedback practice

feedback publication

encouraging feedback

assessment arrangements

assessment suppo

action slide

implementation slide

feedback avoidance behaviours