Transforming teaching and learning through feedback:

a question of design and digitalization

David Boud

Centre for Research in Assessment and Digital Learning, Deakin University

Emeritus Professor, University of Technology Sydney

Overview

• What is the problem with feedback?

• Changing conceptions of feedback

• A project: Closing the Feedback Loop

– Case studies of effective feedback

• Summary: designs for feedback

• Digital enablement

The problem with feedback

• Learners complain that they do not get enough feedback

• Both parties describe it as confronting

• Both parties agree that it is very important

• Educators resent that although they put considerable time into generating information, learners take little notice of it

• Educators typically think their feedback is more useful than their learners think

• Feedback is typically ‘telling’ and analytic in flavour, often lacking strategies for improvement, and often lacking opportunities for further task attempts

Ende 1995, Hattie 2009, Boud and Molloy 2013, Johnson & Molloy 2017

The project:“Feedback for Learning: Closing the Assessment Loop”

Asks:

“What works, when, and why?” and “What is enabling excellent feedback?”

Takes a social constructivist view of feedback and an ecological perspective on higher education

• Conducted a large-scale, mixed-methods study, including identifying cases of effective feedback

• Informed by literature and expertise from team, evaluator and reference group

• To develop a framework of “conditions for success”

• To deliver workshops and re-usable materials for the sector

The project: “Feedback

for Learning:

Closing the Assessment

Loop”

Our definition

“Feedback is a process

in which learners make sense of information

about their performance

and use it

to enhance the quality of their work or learning strategies.”

www.feedbackforlearning.org

This is not feedback

“I left feedback on their final essays, which they never collected”

This is not feedback

This is feedback

Conceptualising feedback

• In higher education, how is our view of feedback changing?

• What are the implications of these changing conceptions?

Conventional ‘feedback’ (Mark 0)

• Adjunct to ‘marking’

• Undertaken by teachers on students

• Hope that it might be taken up

• But, no direct response is required or expected

• More than hopefully useful information-looking for effect

• Given/done to receivers

• Sequenced to require improvement

• Given in time to allow for improved work

Evolution of feedback designs: Feedback Mark 1

Feedback Mark 1:Iterative task design

Activity 3

Activity 2

Activity 1

Degree of

task

challenge

Time

Overlap of

learning

outcomes

Overlap of

learning

outcomes

Molloy and Boud (2013)

Other challenges

Evolution of feedback designs: Feedback Mark 2

• Feedback Mark 1 (importance of effect) plus:

– Dialogic

– Participatory and agentic (students are not just objects)

– Others as well as experts

– Development of evaluative judgement

Orientation to standards of work & purpose of feedback

Activity 1

Learner judges work

Learner asks for specific feedback information

Others judge work

Compare judgements

Plan for improved work

Activity 2

Feedback Mark 2An example

Cases of effective feedback• Student survey and focus groups

identified courses where feedback was working well

• In-depth interviews with multiple teaching staff and students to understand what is working well and why

• Cases are useful exemplars of effective feedback – but also the lessons learnt in enabling feedback

feedbackforlearning.org

Case studies

1. Developmental and diverse feedback: helping first-year learners to transition into higher education

2. Personalised feedback at scale: Moderating audio feedback in first-year psychology

3. In-class feedback: a flipped teaching model in first-year physics

4. Authentic feedback through social media in second year digital media

5. Layers and loops: scaffolding feedback opportunities in first-year biology

6. Multiple prompt strategies across contexts: feedback in classroom, lab and professional practice

7. Investing in educators: enhancing feedback practices through the development of strong tutoring teams

Emphasis on:multiple feedback strategies and modes, applicability across disciplines, scalability, training of tutors in feedback processes, links to literature, resource implications

feedbackforlearning.org

Design of feedback

• Establish feedback as a process and not an input

• Are learning/assessment tasks positioned within the unit/module to enable feedback to occur?

• Are tasks (and what precedes them) designed to stimulate worthwhile learning?

• Are subsequent tasks positioned to enable students to utilize feedback information they have received?

• Are comments made to students designed explicitly to lead to improvements in their work?

What can digitalization enable?

• Prompting good design: – advance planning, deliberate sequencing of tasks, explicitness of activities

• Providing tools for managing tasks and feedback:– digital submission and response

• Enabling tracking: – is there improvement on key outcomes? what comments did the student receive

previously?

• Enabling student self-testing:– students can test themselves and receive non-human specific feedback

information when they need it

• Permiting use of more powerful modalities:– Audio, video, screencast feedback modes

The feedback-enabled curriculum

• Has early strategies to shift learner identity to becoming more self-regulated and builds their feedback literacy

• Positions feedback as part of learning, not as an adjunct of assessment

• Equips students to be skilled and comfortable with negotiating learning outcomes, feedback processes and information needs

• Fosters ongoing ‘dialogue’ between students and teachers about feedback processes, the nature of standards and the practicing of judgement.

• Introduces activities to enable students to calibrate judgement (of their own work and that of others)

Conclusions

• Change the language of feedback and focus on what makes a difference

• Design feedback loops into learning activities with opportunities for students to act on comments

• Position students as initiators of feedback at every opportunity

Ten feedback strategies to make a difference

1. Have students identify and state what kind of comments they would like to get

2. Design follow-on tasks so that students can apply information received

3. Have students respond to information with plans for what they are going to do

4. Have students judge their own work against criteria before they submit it

5. Prompt peer feedback sessions that focus on producing improved work

6. Distinguish between mark justification and feedback information when making comments

7. Move feedback comments from late in the sequence to earlier when students have time to act on them

8. Focus on comments for improvement rather than corrections

9. Refer frequently to models and exemplars of good work

10.Train students to be feedback literate (ie. know what feedback is and how they can make it work for themselves)

Find other strategies in the case studies of excellent practice at feedbackforlearning.org

David Boud, Rola Ajjawi, Phillip Dawson and Joanna Tai (Eds)

Developing Evaluative Judgement in Higher EducationLondon: Routledge, 2018

Other contributors:Elizabeth Barrett, Sue Bennett, Jaclyn Broadbent, Alison Bullock, David Carless, Kennedy Chan, Gloria Dall'Alba, Barney Dalgarno, Cath Ellis, Peter Goodyear, John Hattie, Michael Henderson, Christina Johnson, Gordon Joughin, Jenny Keating, Gregor Kennedy, Romy Lawson, Margaret Lo, Lori Lockyer, Jason Lodge, Lina Markauskaite, Karen Mattick, Elizabeth Molloy, Lynn Monrouxe, Robert Nelson, Ernesto Panadero, Charlotte Rees, Sam Sevenhuysen, Darralll Thompson, Jessica To.

References

Boud, D. and Molloy, E. (2013). Rethinking models of feedback for learning: the challenge of design, Assessment and Evaluation in Higher Education, 38, 6, 698-712.

Boud, D. and Molloy, E. (Eds) (2013). Feedback in higher and professional education. London: Routledge

Dawson, P. (2017). Assessment rubrics: towards clearer and more replicable design, research and practice. Assessment & Evaluation in Higher Education, 42(3), 347-360.

Hattie, J. and Gan, M. (2011). Instruction based on feedback. In Meyer, R.E. and Alexander, P.A. (Eds) Handbook of Research on Learning and Instruction, New York: Routledge.

Hattie, J. and Timperley, H. (2008). The power of feedback’, Review of Educational Research, 77: 81-112.

Jolly, B. and Boud, D. (2013). Written feedback: what is it good for and how can we do it well? In Boud, D. and Molloy, E. (Eds) Feedback in higher and professional education. London: Routledge, 104-124.

Shute, V.J. (2008). Focus on formative feedback. Review of Educational Research, 78:153- 189.Winstone, N. E., Nash, R. A., Rowntree, J., & Menezes, R. (2016). What do students want most

from written feedback information? Distinguishing necessities from luxuries using a budgeting methodology. Assessment & Evaluation in Higher Education, 41(8), 1237-1253.

What does feedback literacy look like?

Deakin University CRICOS Provider Code: 00113B

Carless, D., & Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315-1325

Interesting questions in feedback literacy

Deakin University CRICOS Provider Code: 00113B

• What are the benefits of feedback literacy?

• How can we notice (understand, interrogate, measure, etc) feedback literacy of students?

• How can we develop students’ feedback literacy for the long term?

• How can we improve enactment of feedback literacy?

Some of CRADLE’s research on feedback literacy

Deakin University CRICOS Provider Code: 00113B

Published

Carless, D., & Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315-1325

In progress

Elaboration of feedback literacy competencies

• Boud and Molloy

Feedback literacy scale

• Development of instrument to measure feedback literacy

• Collaboration with Yan Zi(Education University Hong Kong)

In development

Developing feedback capabilities for study, life and work through behaviour change

• ARC Discovery application for submission in 2019

• Collaboration with CRADLE Fellow Jaclyn Broadbent (Psychology) and Liz Molloy (UniMelb)

The Conversation, CC BY-ND

Survey• Open to all coursework students and

staff in two universities

• 4514 students completed the survey, including:• 3002 undergraduates

• 109 Honours students

• 1138 Masters students

• 265 students completing a postgraduate diploma or certificate

• 406 academic staff completed the survey,

Is this enough to improve feedback?

While it is the overall feedback process that makes a difference, the information we communicate to learners is still very important.

There is evidence that some kinds of comments lead to negative outcomes.

What constitutes effective comments on students’ work?

Hattie’s model for feedback comments

• Comments can be directed at four different levels of operation of the student. Feedback will be ineffective if directed at an inappropriate level.

• The responses of students and their efficacy are dependent on the focus and type of comments they get.

• If the focus is inappropriate to their needs, feedback may be ineffective, because the student is unable to transform information into action where it is needed most.

Hattie and Timperley 2008; Hattie and Gan, 2011

Levels of operation at which comments are pitched:

• Task focused

• Process focused

• Self-regulation focused

• Person focused

Levels of operation at which comments are pitched:

• Task focused

– Most common

• Process focused

– More effective

• Self-regulation focused

– Most needed

• Person focused

– Mostly ineffective

Elements of self regulation focus

• capacity to create ‘internal’ feedback.

• ability to self-assess.

• willingness to invest effort into seeking and dealing with feedback information.

• degree of confidence or certainty in the correctness of the response.

• attributions about success or failure.

• level of proficiency at seeking help.

Guidance for those offering comments

• Be wary of old nostrums and supposed ‘good practice’

• Involve the learner – if they are positioned as passive recipients they will act as such

• Think about what you really want to influence– It may not be good use of your time to offer simple corrections

• Always do it when student are in a position to act on it– Not at the end of a unit!

• Comment as if it were a part of an ongoing dialogue– One-off, disconnected input is very unlikely to influence

Key points for excellent practice

Design: • are tasks positioned within the unit to enable feedback to occur and for students to improve

their work?• Are tasks (and what precedes them) designed to stimulate worthwhile learning?Inputs to students:• Are comments to students designed to lead to specific improvements in their work?Responses of students:• Are they expected from the start of the unit/task to be active players?• Are they necessarily expected to respond to and act on inputs from others to produce

improved work?Feedback to staff• Are you monitoring students’ work with a view to adjusting the unit/course to create bigger

positive effects on their learning?