Rola Ajjawi, Sean McAleer &

Tom Mammen

Improving dialogic feedback

in online distance learning

Acknowledgement: David Walker

Learning outcomes

Participants will be able to: a) discuss best practice guidelines of feedback and assessment in medical education; b) reflect on own assessment and feedback practices; c) critique findings from interACT research project; and c) generate strategies for uptake of principles in own practice, including overcoming challenges and identifying rewards.

Drivers

Overview

• Welcome and introductions

• Presentation of literature synthesis of best practice principles of assessment and feedback

• Appreciative inquiry activity of current practices in relation to assessment and feedback

• Case Study presentation of UoD JISC research

• Grading of assessment with cover page activity

• Summary of discussion and future action

ACTIVITY: Jot down your definition of feedback and then discuss with peer (5minutes)

What is feedback?

“Feedback should help the student

understand more about the learning

goal, and more ways to bridge the gap

between their current status and the

desired status.”1

Feedback encapsulates:

1. Where am I going?

2. How am I going?

3. Where to next? (or how do I

get there?)2

Influence of feedback

• Affective processes of increased effort

and motivation and/or cognitive

processes of restructuring knowledge

• Learners (particular those studying at a

distance) require reassurance that they

are heading in the right direction

• Constructive feedback often results in

improved performance2-3

We have a problem…

http://www.motor-trade-insider.com/index.php/2010/11/houston-we-have-a-problem/

Commonly reported concerns of students

• Feedback is too late to influence learning

• Feedback is cryptic e.g. “more”

• Feedback provides no explanation for action e.g. “good”

• Good students miss out

• Feedback is “one off” – no chance to try again based on the feedback

Tutors think they provide more detailed feedback than students perceive

Tutors view their feedback to be more useful compared to what students think4

Critique of monologic feedback4-10

• Lack of learner engagement with feedback • Lack of understanding of feedback • Transmitted feedback creates dependency on

teacher • No evidence of feed forward • Not utilising self- and peer- feedback • Lack of a shared context for assessment for teacher

and learner • High teacher effort— low efficiency • Reduced staff satisfaction as developmental potential

of feedback not seen

Influence of feedback

• Feedback can help to develop learner

self-regulation4,5

• Professional socialisation into

profession11

What of the person ‘receiving’ the feedback?

• Interplay between fear, confidence and reasoning in receptivity to feedback 12

• Tendency for ‘deflection’ when there is a discrepancy between learners’ internal perceptions (self-evaluation) and the external teacher’s perceptions (feedback) 10

• Learner re-interprets the external feedback to make it conform with their own hope, intention or interpretation of their performance13

• Perceived credibility of feedback giver essential14-16

Purpose of feedback?

Feedback should develop the students’

capacity to make evaluative judgements

about their own and others work1,8

Feedback should serve the function of

progressively enabling students to

better monitor, evaluate and regulate

their own learning, independently of the

teacher7

What is feedback?

“Feedback should be conceptualised as

a dialogical and contingent two-way

process that involves coordinated

teacher–student and peer-to-peer

interaction as well as active learner

engagement.”5

ACTIVITY: In your pack is a list of principles reported in the literature. In your group critique these principles and reach a consensus on 4-5 guiding principles (10minutes)

Teacher-focussed strategies

• Faster turnaround of feedback

• Improve quality of the feedback

• Explicate timings

• Generate assessment rubrics

• Look at sequencing of assignments

• Faculty development and benchmarking

Learner-focussed strategies • Educate and empower students

to take an active role in feedback • Engage students in generating

assessment criteria and discussing standards

• Create opportunities for students to process and use feedback (e.g. self-explaining)

• Engage students in ways to develop evaluative judgements (self- and peer-review)

• Generating a dialogue about feedback

http://www.strath.ac.uk/media/other/learningteaching/student_leaflet.pdf

Effective Instructor Feedback : perceptions of online graduates17

Student involvement – students need to be consulted re the type of feedback they regard as helpful

Being positively constructive – should not demean the effort of learning

Gentle guidance – firstly by making expectations of the course explicit; secondly guidance of students as they worked through the course

Timeliness – students should know when they would receive feedback

Future orientation – feedback should also help student deal with what was in front of them - preparedness

ACTIVITY: In your group discuss: a) strategies that you currently implement in your practice that align with your principles; and b) strategies that you could implement. What are the challenges? Additional strategies are included in your pack (15 minutes)

interACT: Interactive Assessment and Collaboration via Technology

Dr Susie Schofield – Project Manager and Lecturer CME

Dr David Walker – Senior Learning Technologist LLC

Karen Barton – Project officer CME Grant Murray – Learning Technologist CME

Embarked on a literature review…

And synthesised some principles:

1. Supports personalised learning

2. Ensures feedback leads to learning

3. Focuses on student self-development

4. Stimulates dialogue

5. Inducts students in the educational culture and assessment and feedback practices

6. Is acceptable to all stakeholders

Baseline activities

• Document analysis: external examiner reports (2006-2011); end of course evaluations (2006-2011); additional evaluation surveys conducted in 2010 and 2011 as part of our curriculum review; and the HEA postgraduate student experience survey

• Focus groups with staff including part-time tutors

• Feedback audit

Baseline activities revealed

1. Inconsistency in the quality and quantity of feedback provided

2. Assessment design (e.g. over-assessment, over-reliance on essays, lack of formative assessment)

3. Timeliness of the feedback

4. Lack of assessment and feedback dialogue

Reason for Feedback

Focus of Feedback

• Modular approach to sequencing of assessment

• Explicating feedback times in a student-tutor charter

• Development of assessment rubrics for all summative

• Increased use of formative assessments

Re-engineered our assessment

Equivalent 2

chapters Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters

PG Certificate in Medical Education

Equivalent 2

chapters Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters

Equivalent 2

chapters

Revised PG Certificate in Medical Education

Chapters 2 4 6 8 10

Chapters 2 4 6 8 10

Chapters 2 4 6 8 10

Chapters 2 4 6 8 10

Formative assessment

Medium stakes assessment

High stakes assessment

©Mark Russell 2010 University of Hertfordshire

ESCAPE project

Faculty development

• Workshops about feedback and importantly agreement on how to give feedback (process)

Include self-review

• Self-review is a scenario where students make evaluative judgements about the own work and produce a written feedback commentary19

Content (understanding of theory / principles and application to own context)

- Understanding of learning theory

- Evidence of critical reflection on learning theory and key learning and teaching principles

- Evidence of application of learning theory to own practice

Self-evaluation:

Tutor feedback:

Style, format and language (e.g. structure, coherence, flow, formatting, use of language)

Self-evaluation:

Tutor feedback:

Sources and references (e.g. range of references cited, relevance, consistency, accuracy and completeness of referencing)

Self-evaluation:

Tutor feedback:

Which aspect(s) of your assignment would you specifically like feedback on?

Student comment:

Tutor feedback:

How did previous feedback inform this assignment? Student comment:

Name of Tutor: Date:

Engaging students in processing of feedback through the wiki

When you receive your assignment with feedback please upload a copy to your wiki and include a reflection on the following four questions:

1. How well does the tutor feedback match with your self-evaluation?

2. What did you learn from the feedback process?

3. What actions, if any, will you take in response to the feedback process?

4. What if anything is unclear about the tutor feedback?

Aspects I would like feedback on: • Ideas of not preparing too rigidly in order to be flexible

within sessions – practical advice would be welcomed! • As the first essay I have written in nearly 20 years, I would

like to know whether the standard overall was acceptable • Please advise me how I can enter a specific page number in

a reference when using Endnote (I wanted to add “p. 58” to the first reference used in the text since this is an exact quote, but failed to find out how I can do it, despite using the help option of the software).

• Feedback on whether my peers have had similar thoughts for their own teaching, or other ideas that have been commonly developed would be beneficial in case I have not thought or considered them.

• any part of it

How did previous feedback inform this assignment

• It made me realise that instead of focusing on a single or a few key teaching principles, I focused on many of them without going into much detail. Also I had used bullet points in the text.

• Feedback that my writing style was agreeable was reassuring. I appreciated knowing my use of literature was valid and supportive in the previous assignment, so have tried to continue applying the literature to my work.

• I tried to be careful to define and reference jargon

• It was really helpful in writing present assignment

I really like the cover sheet and request for self assessment and what you would like feedback about ... consistent with principles of feedback!! (M&L end of module feedback)

ACTIVITY: In your group you have two assignments. Half your group should read assignment A and other half assignment B. Provide feedback on the cover page then discuss the feedback (15 minutes)

Added challenges of:

• Relationship and trust • Asynchronous dialogue and

shared understanding • Reliance on written feedback • Technology

For sharing feedback

The challenges of student engagement in online courses17

• Faculty unable to pick up nonverbal cues from students

• Faculty not able express themselves easily – in terms of encouragement, concern etc.

• The anonymity of the online environment make it easier for students to withdraw/disappear

• Students may enrol because they may feel it will require less time/easier – therefore prone to disengagement

Future developments

• Develop a toolkit of e-activities designed to improve self, peer and tutor dialogic feedback through the use of available technologies such as blogs, Friendfeed, Skype/webinars, or synchronous chat

• Introduce a patchwork assignment at the end of the Certificate and Diploma

• Work to refine technology

Project website http://blog.dundee.ac.uk/interact

Importance of feedback

• If students are to understand their level of

performance and how to improve they need

feedback

• Feedback does not need to be the responsibility

of teachers7

• Feedback should not be viewed as a

transmission process but a dialogue5,10

• Feedback should feedforward and assessment

strategy should allow this

Feedback is … a dialogue

http://www.flickr.com/photos/khalidalbaih/5653817859

Readings in your pack

• Boud, D., & Molloy, E. (2012). Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in Higher Education, 1-15.

• Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218.

Contact Details Dr Rola Ajjawi

Centre for Medical Education

University of Dundee

Email: r.ajjawi@dundee.ac.uk

Twitter: @r_ajjawi

http://blog.dundee.ac.uk/interact/

Dr Sean McAleer

Centre for Medical Education

University of Dundee

Email: j.p.g.mcaleer@dundee.ac.uk

1. Sadler, D. R. (2010) Beyond feedback: developing student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35:5, 535-550.

2. Hattie, J. & TIimperley, H. (2007). The Power of Feedback. Review of Educational Research, 77, 81-112.

3. Veloski, J., Boex, J. R., Grasberger, M. J., Evans, A., & Wolfson, D. B. (2006). Systematic review of the literature on assessment, feedback and physicians’ clinical performance*: BEME Guide No. 7. Medical Teacher, 28(2), 117-128.

4. Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31(2), 219-233.

5. Nicol, D. (2010). From monologue to dialogue: improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education, 35(5), 501 - 517.

6. Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144.

7. Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218.

8. Boud, D., & Associates. (2010). Assessment 2010: Seven propositions for assessment reform in higher education. Sydney: Australian Learning and Teaching Council.

9. Higgins, R., Hartley, P., & Skelton, A. (2002). The Conscientious Consumer: Reconsidering the role of assessment feedback in student learning. Studies in Higher Education, 27(1), 53-64.

10. Boud, D., & Molloy, E. (2012). Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in Higher Education, 1-15.

11. Molloy, E. K. (2009). Time to Pause: Feedback in Clinical Education. In C. Delaney & E. K. Molloy (Eds.), Clinical Education in the Health Professions. Sydney: Elsevier.

12. Eva, K., Armson, H., Holmboe, E., Lockyer, J., Loney, E., Mann, K., et al. (2011). Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Advances in Health Sciences Education, 1-12.

13. Carless, D., Salter, D., Yang, M., & Lam, J. (2010). Developing sustainable feedback practices. Studies in Higher Education, 36(4), 395-407.

14. Watling, C., & Lingard, L. (2010). Toward meaningful evaluation of medical trainees: the influence of participants’ perceptions of the process. Advances in Health Sciences Education, 1-12.

15. Watling, C., Driessen, E., van der Vleuten, C. P. M., & Lingard, L. (2012). Learning from clinical work: the roles of learning cues and credibility judgements. Medical Education, 46(2), 192-200.

16. Watling, C., Driessen, E., van der Vleuten, C. P. M., Vanstone, M., & Lingard, L. (2012). Understanding responses to feedback: the potential and limitations of regulatory focus theory. Medical Education, 46(6), 593-603.

17. Getzlaf B, Perry B, Toffner G, Lamarche K and Edwards M., (2009) Effective Instructor Feedback: Perceptions of Online Graduate Students. The Journal of Educators Online, 6(2)July.

18. Cull S, Reed D and Kirk K. (2010) Student Motivation and Engagement in Online Courses http://serc.carleton.edu/NAGTWorkshops/online/motivation.html

19. Nicol, D. (2012). Assessment and feedback - in the hands of the student [Online]. JISC. Available: http://jiscdesignstudio.pbworks.com/w/page/50118521/Assessment%20and%20feedback%20-%20in%20the%20hands%20of%20the%20student [Accessed 01/02/12].