automatic feedback for motivation and self-regulation

Automatic feedback for motivation and self-regulation

Professor Denise Whitelock

The Open University, Walton Hall,

Milton Keynes MK7 6AA, UK

[email protected]

DMW UOC Open Seminar May 2013

I hate marking but want the tasks and feedback to assist student learning


The e-Assessment and automatic feedback Challenge

• Constructivist Learning – Push

• Institutional reliability and accountability – Pull

.



www.storiesabout.comwww.storiesabout.com/

[email protected]

MCQs: Variation on a theme


Example of LAPT Certainty-Based Marking, UK cabinet ministers demo exercise showing feedback, University College, Tony Gardner-Medwin

Drug Chart Errors and Omissions, Medicines Administration Assessment, Chesterfield Royal Hospital

MCQs: High Stakes Assessment


Example of practice Thinking Skills Assessment" (TSA) question, Admissions Interview, Cambridge Assessment, Steve Lay

Example of practice Thinking Skills Assessment" (TSA) feedback, Admissions Interview, Cambridge Assessment, Steve Lay

Scaffolding and High Stakes assessment

• Math for Science

• Tutor less course

• Competency led

• No point to cheat

• Web home exam

• Invigilation technologies


Self diagnosis

• Basic IT skills, first year med students (Sieber, 2009)

• Competency based testing

• Repeating tests for revision

• Enables remedial intervention



Characteristics Descriptor

Authentic Involving real-world knowledge and skills

Personalised Tailored to the knowledge, skills and interests of each student

Negotiated Agreed between the learner and the teacher

Engaging Involving the personal interests of the students

Recognise existing skills Willing to accredit the student’s existing work

Deep Assessing deep knowledge – not memorization

Problem oriented Original tasks requiring genuine problem solving skills

Collaboratively produced Produced in partnership with fellow students

Peer and self assessed Involving self reflection and peer review

Tool supported Encouraging the use of ICT

Elliott’s characteristics of Assessment 2.0 activities

Authentic Assessment: Building e-portfolios on a chef’s course


food preparation for e-portfolio, Modern Apprenticeship in Hospitality and Catering,

West Suffolk College, Mike Mulvihill

Evidence of food preparation skill for e-portfolio, Modern Apprenticeship in Hospitality and Catering, West Suffolk College, Mike Mulvihill

Peer Assessment and the WebPA Tool

• Loughborough (Loddington et al, 2009)

• Self assess and peer assess with given criteria

• Group mark awarded by tutor

• Students rated:• More timely feedback

• Reflection

• Fair rewards for hard work

• Staff rated:• Time savings

• Administrative gains

• Automatic calculation

• Students have faith in the administrative system


Mobile Technologies and Assessment

• MCQs ,PDAs Valdiva & Nussbaum(2009)

• Polls,instant surveys

• Simpson & Oliver (2007)

• Draper (2009) EVS



Gains from Formative Assessment

• Mean effect size on standardised tests between 0.4 to 0.7 (Black & Williams, 1998)

• Particularly effective for students who have not done well at school http://kn.open.ac.uk/document.cfm?docid=10817

• Can keep students to timescale and motivate them

• How can we support our students to become more reflective learners and engage in formative assessment tasks?


Collaborative formative assessment with

Global Warming

DMW, Institute of Educational Technology, September 1997DMW, Institute of Educational Technology, September 1997


Global Warming


Next: ‘Yoked’ apps via BuddySpace

Student A

Student B(‘yoked’, butwithout full

screen sharingrequired!)


Global Warming: Simlink Presentation

LISC: Aily Fowler

• Kent University ab-initio Spanish module

• Large student numbers

• Skills-based course

• Provision of sufficient formative assessment meant unmanageable marking loads

• Impossible to provide immediate feedback

• leading to fossilisation of errors


The LISC solution: developed by Ali Fowler• A CALL system designed to enable students

to:• Independently

practise sentence translation

• Receive immediate (and robust) feedback on all errors

• Attend immediately to the feedback (before fossilisation can occur)


How is the final mark arrived at in the LISC System?

• The two submissions are unequally weighted

• Best to give more weight to the first attempt

• since this ensures that students give careful consideration to the construction of their first answer

• but can improve their mark by refining the answer

• The marks ratio can vary (depending on assessment/feedback type)

• the more information given in the feedback, the lower the weight the second mark should carry


Heuristics for the final mark

• If the ratio is skewed too far in favour of the first attempt…

• students are less inclined to try hard to correct non-perfect answers

• If the ratio is skewed too far in favour of the second attempt…

• students exhibit less care over the construction of their initial answer


What about emotional support in the feedback?

• Difficult at times to receive written feedback

• Not just a cognitive response

• How can Bales help?



Coding into Categories

• Bales analysis

• Psychology 1950s

• Analyses talk

• Includes socio-emotive categories

• Flander’s (1970)categories inappropriate as also includes classroom control


Bales Categories

• Four main groupings

• A. Positive reactions; agreeing and boosting the other person

• B. Directing/teaching

• C. Questions: requesting information, clarification etc

• D. Negative reactions: disagreement


Coding the tutor comments

Categories Specific Examples

Positive Reactions

A1

A2

A3

1. Shows solidarity

2. Shows tension release

3. Shows agreement

Jokes, gives help, rewards others

Laughs, shows satisfaction

Understands, concurs, complies, passively accepts

Attempted Answers

B1

B2

B3

4. Gives suggestion

5. Gives opinion

6. Gives information

Directs, proposes, controls

Evaluates, analyses, expresses feelings or wishes

Orients, repeats, clarifies, confirms

Questions

C1

C2

C3

7. Asks for information

8. Asks for opinion

9. Asks for suggestion

Requests orientation, repetition, confirmation, clarification

Requests evaluation, analysis, expression of feeling or wishes

Requests directions, proposals

Negative Reactions

D1

D2

D3

10. Shows disagreement

11. Shows tension

12. Shows antagonism

Passively rejects, resorts to formality, withholds help

Asks for help, withdraws

Deflates others, defends or asserts self

Bales’ Interaction Process


Identifying trends: H801

0 5 10 15 20 25

A Pass 1

A Pass 2

A Pass 3

A Pass 4

B Pass 1

B Pass 2

B Pass 3

B Pass 4

C Pass 1

C Pass 2

C Pass 3

C Pass 4

D Pass 1

D Pass 2

D Pass 3

D Pass 4

Ba

les'

In

tera

cti

on

al

Ca

teg

ori

es

at

ea

ch

Pa

ssL

leve

l

Number ofIincidences

Graph to show conflated Bale’s categories against mean number of incidences in H801 scripts


Identifying trends: H801

5.96

17.13

5.73

1.61

A

B

C

D

Pie Chart to show the mean number of incidences per pass per conflated Bales' Interactional Category for all four levels of

pass in H801 scripts

Key:

A = Positive reactions

B = Responses

C = Questions

D = Negative reactions


Identifying trends

A

B

C

D

A

B

C

D

A

B

C

D

Pie Charts to show the mean number of incidences per conflated Bales Interactional Category for ‘Pass 1’ and ‘Pass 4’ in the following courses:

Key:

A = Positive reactions C = Questions B = Responses D = Negative reactions

Pass 4

Pass 1

B820 S103

A

B

C

D

A

B

C

D

A

B

C

D

H801


What is Open Mentor?

• “An open source mentoring tool for tutors”

• “Open source” = free and easy to use, and to embed in an institutions infrastructure and working practices

• “mentoring” = designed to help people learn how to give feedback effectively, through reflection and social networks

• “tutors” = primarily intended for teaching staff, but with clear applications for those involved in quality


How Open Mentor handles comments

• “Good work”

• “Yes, well done”

• “Yes, but is this useful?”

• “Can you explain what you mean”

• “This does not follow”

• A = positive reactions

• A = positive reactions

• B = attempted answers, and not a positive reaction

• C = questions

• D = negative reactions


Explaining OpenMentor’s Rules

• Four categories

• A – Positive Reactions

• B – Attempted Answers

• C – Questions

• D – Negative Reactions


‘A’ - Positive Reactions

Category Examples of Rules Examples of commentsA - Positive Reactions1. Shows solidarity A1 ...excellent... Excellent Conclusions.

A1 ...(good|comprehensive)... Good, you are drawing on hard facts here.A1 ...nicely... Very nicely stated. Your analysis is thorough and your

conclusions consistent regarding the attractiveness of the budget airline sector. This is a good example of critical thinking.

A1 ...well presented... Very well presented diagram with interesting information.A1 ...effective use... Effective use of the case material here.A1 …well (structured|stated)… Report very well structured.A1 ...(well|clear)(ly)*

(structured|structure|summary| summarised|presented|presentation)...

The corporate vs. business unit strategy is well presented and nicely tied to strategies.

A1 ...reasonable.... A reasonable structure as listed in your table of contents.A1 ...useful point(s)... Generally useful points in this section.

2. Shows tension release A2 ... a helpful...

A2 …(thanks|thank you)…3. Shows agreement A3 ...yes... Yes, the intellectual reactions are both real.

A3 ...indeed... Indeed – if it has one basic strategy it is surely differentiation, though it still has to control costs.

4. Praise then direction A4 good...but... Good model, good quote, but be careful about what industry you analyse ??


‘B’ Attempted Answers

Category Rules Examples of commentsB - Attempted Answers

B4 …perhaps… Perhaps even better here to explain the link in your mind between "analysis of stratgeies" and "strategic issues".

B4 …requires…B4 …take care… Take care with your STEP analysis not to make it too industry

focussed.B4 …useful to… Innovation is closely linked to structure and culture- it would be

useful to see some book 6-8 concepts here too.

B4 …you (might|ought)… You ought to have explicitly stated these.

B4 Don’t|never … Don’t introduce new frameworks just for the sake of it in the conclusion. The conclusion should be pulling together what went before.

B4 Please (see|refer to|look at)... Please make sure to read and understand the question correctly

5. Gives opinion B5 I (am|think)... I think I can see where you are going, though a numbered report format might have demonstrated the approach better

B5 This is.... this is an introduction rather than a “summary”B5 ...sounds...like... This sounds as if it could be very popular!!B5 ...not sure... I am not sure about the balance between the environmental

analysis and the review of the resources, capabilities (power, culture, structures and systems) as raised in the question.

B5 I (thought|agree|suggest)… I thought it was because they did not need any external input and saw a significant market sector they could address themselves.

B5 I (do|don’t) think… I don’t think this exercise has helped to develop your analysis. I also think that the development of the perspectives is superficial

6. Gives information B6 …(demonstrates|shows) this….B6 Also… Also, cross link to Leadership issues, Pettigrew on Strategic

Thinking tooB6 ...Q1... etc Q1 = 59/100


Is the rule set generic?

Comments Classified from Test Data

0102030405060708090

100

B820 TestSet 1

B820 TestSet 2

B820 TestSet 3

A850 M878 S809

Test Data Set

% o

f co

mm

ents

cla

ssif

ied

OpenMentor Transfer: JISC funded

• JISC funded project

• Transfer OpenMentor technology to King’s and Southampton

• What changes are needed for cross institutional use?

• Identify strengths and limitations of OM for training tutors


Transferring OM to other HEIs


• Transferred to Southampton and Kings London

• Participating Tutors given face to face training

• King's College:

• 3 tutors.

• 25. learning experts at TEL forum gave feedback after a demonstration

• Southampton

• 3 tutors.

• Interviews and questionnaire

• Open University

• 3 distance education tutors

• Questionnaire and epistolary interviews

• 113 students in a Masters course focussing on Innovation in eLearning and 5 tutors.

Lessons learned after completion of first cycle of trials

• Open Mentor’s theoretical framework was robust enough to facilitate and encourage dialogue and reflective activities

• Tutors positive about the system’s functions to support provision of feedback

• Suggestions for change

• a module for user authentication and management

• the development of OM reports to help tutors to progress towards the ideal ‘state’ of feedback provided.

• used for training purposes as an academic development tool.

• Our contact details, blog and references -http://omtetra.ecs.soton.ac.uk/wordpress/


What can we learn from modelling tutors marking to construct a formative

e-assessment tool?

• Open Comment project builds on the work of

OpenMentor

• Free text entry for History and Philosophy students

• Immediate feedback (in context) to students

• Influenced by ELIZA (Weizenbaum, 1963)


Stuart Watt

Changed to reduce the "modelled on" description

Open Comment addresses the problem of free text entry

• Automated formative assessment tool

• Free text entry for students

• Automated feedback and guidance

• Open questions, divergent assessment

• No marks awarded

• For use by Arts Faculty


Causal models of explanation

• First step:

• Identification of question types where students exhibit causal reasoning

• Looked for questions with:

• Salient variables

• Description of these variables

• Identification of trends

• Identification of relationship between the variables i.e. causality


Praise for effort and not just ability

• Praise for ability per se can hinder learning (Mueller & Dweck, 1998)

• Praise = being clever

• Negative feedback now without ability

• Disempowering and demoralising


Mueller & Dweck (1998)

• Raven’s Matrices (IQ)

• First test pupils praise either for effort or ability

• Second test most difficult

• Third test medium difficulty. Score up 1 points for pupils praised for effort. Down 1 point ability


Stages of analysis by computer of students’ free text entry for Open Comment: advice with respect to content (socio-emotional support stylised example)• STAGE 1a: DETECT ERRORS E.g. Incorrect dates,

facts. (Incorrect inferences and causality is dealt with below)

• Instead of concentrating on X, think about Y in order to answer this question Recognise effort (Dweck) and encourage to have another go

• You have done well to start answering this question but perhaps you misunderstood it. Instead of thinking about X which did not…….. Consider Y



Computer analysis continued

• STAGE 2a: REVEAL FIRST OMISSION

• Consider the role of Z in your answer Praise what is correct and point out what is missing Good but now consider the role X plays in your answer

• STAGE 2b: REVEAL SECOND OMISSION

• Consider the role of P in your answer Praise what is correct and point out what is missing Yes but also consider P. Would it have produced the same result if P is neglected?


Final stages of analysis• STAGE 3:REQUEST

CLARIFICATION OF KEY POINT 1

• STAGE 4:REQUEST FURTHER ANALYSIS OF KEY POINT 1(Stages 3 and 4 repeated with all the key points)

• STAGE 5:REQUEST THE INFERENCE FROM THE ANALYSIS OF KEY POINT 1 IF IT IS MISSING

• STAGE 6:REQUEST THE INFERENCE FROM THE ANALYSIS OF KEY POINT 1 IF IT IS NOT COMPLETE

• STAGE 7:CHECK THE CAUSALITY

• STAGE 8:REQUEST ALL THE CAUSAL FACTORS ARE WEIGHTED

Where are we now?

• Opening up with Open Source

• Moving towards vision and not losing sight of it through tool adaptation

• More work to do for Arts

• Open Comment - pedagogical model open to test

• Feedback

• Changing pedagogy

• Another handle on misconceptions



Open Comment drivers for reflection

• Students are able to find facts similar to X

• Know how X might be disputed

• Are able to make predictions about X

• Know how to use X in an argument

• Know how far X can be pushed

• Supported with tools and strategies for effort

SAFeSEAProfessor Denise WhitelockProfessor John Richardson

Professor Stephen Pulman

An automated tool supporting

online writing and

assessmentof essays providing

accurate targeted feedback

SAFeSEA: Supportive Automated Feedback for Short Essay Answers


About SAFeSEA

• Effect of summarisation

• What are the beneficial factors?

• Correlate measures of learner activity and essay improvement

• Effect of hints

• http://www.open.ac.uk/iet/main/research-scholarship/research-projects/supportive-automated-feedback-short-essay-answers


OpenEssayist: SAFeSEA Web application for summarisation-based formative feedback

DMW UOC May 2013

localhost:8065

phaeros.open.ac.uk:80

openEssayist

PHP, Epiphany[Symfony2]

User

openEssayist RESTful API

PHP, Epiphany

User

User

pyEARESTful API

Python, Flask

localhost:8064

AfterTheDeadlineSpell/Grammar

checker

Java

User

localhost:9998

Apache TikaText Extractor

Java

Orchestrator

(Open)Learner Model

pyEssayAnalyser

Python, NLTK

localhost:8065

phaeros.open.ac.uk:80

openEssayist

PHP, Epiphany[Symfony2]

User

openEssayist RESTful API

PHP, Epiphany

User

User

pyEARESTful API

Python, Flask

localhost:8064

AfterTheDeadlineSpell/Grammar

checker

Java

User

localhost:9998

Apache TikaText Extractor

Java

Orchestrator

(Open)Learner Model

pyEssayAnalyser

Python, NLTK

Key words and phrases visualized in the essay context. Sentences in light-grey (green) background are key sentences as extracted by the EssayAnalyser (the number at the start of the sentence indicates its key-ness ranking);

bigrams are indicated in bold (red) and boxed.

DMW UOC May 2013

The structural elements of the essay can be used jointly with the key word extraction to highlight relevant information within specific parts of the essay, here the introduction (and the assignment question)

DMW UOC May 2013

Key words and phrases as separate lists

DMW UOC May 2013

Dispersion of key words across the essay http://www.open.ac.uk/iet/main/research-scholarship/research-projects/supportive-automated-feedback-short-essay-answers

DMW UOC May 2013

Can we find ways of using graph visualization techniques on the key words and key sentences, to make them helpful and meaningful to students?

DMW UOC May 2013

SAFeSEA

• Support for essay writing

• Shape landscape of eLearning and Learning Analytics

• Improves the student experience

• Support advances in NLP


Feedback

• Students must decode feedback and then act on it Boud (2000)

• Students must have the opportunity to act on feedback Sadler (1989)

• Gauging efficacy through action


Badge System: Mozilla



Characteristics Descriptor

Authentic Involving real-world knowledge and skills

Personalised Tailored to the knowledge, skills and interests of each student

Negotiated Agreed between the learner and the teacher

Engaging Involving the personal interests of the students

Recognise existing skills Willing to accredit the student’s existing work

Deep Assessing deep knowledge – not memorization

Problem oriented Original tasks requiring genuine problem solving skills

Collaboratively produced Produced in partnership with fellow students

Peer and self assessed Involving self reflection and peer review

Tool supported Encouraging the use of ICT

Elliott’s characteristics of Assessment 2.0 activities

A d v i c e f o r A c t i o n

The 4Ts Pyramid


Tool Development

Training of staff

TransformationTasks

TransferLearning

National Union of Students’ Principles of Effective Assessment Times Higher Education, 29th January 2009

• Should be for learning, not simply of learning

• Should be reliable, valid, fair and consistent

• Should consist of effective and constructive feedback

• Should be innovative and have the capacity to inspire and motivate.

• Should be conducted throughout the course, rather than being positioned as a final event

• Should develop key skills such as peer and reflective assessment


Final thoughts

• There is a growing consensus in the field of assessment that times are changing and that assessment needs to become more embedded/central in the teaching learning cycle (Hatzipanagos & Rochon 2011).

• Our project provides another phase in this type of research where the balance of socio emotive content contained in feedback cannot be ignored (Draper, 2009).

• Feedback that encourages the student to actively change their ideas and ways of organising their answers and discourse within a given subject domain is what is required and advocated by Whitelock (2011) as “advice for action”.


“Advice for Action”, Whitelock (2011)

• Helping students find out what they do not know and how to remedy the situation can avoid the trauma of assessment

• Are we on the way with new e-tools?


References• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J.

(2013) ‘OpenEssayist: Extractive Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium

• Whitelock, D., Gilbert, L., Hatzipanagos, S., Watt, S., Zhang, P., Gillary, P. & Saucedo, A. (2012) Supporting tutors with their feedback using OpenMentor in three different UK Universities. 10th International Conference on Computer Based Learning in Science, CBLIS 2012, Barcelona, Spain. 26-29 June 2012.

• Whitelock, D., Gilbert, L. & Gale, V. (2011) ‘Technology-Enhanced Assessment and Feedback: How is evidence-based literature informing practice?’ International Computer Assisted Assessment Conference, DeVere Grand Harbour Hotel, Southampton, 5/6 July 2011. http://caaconference.co.uk/wp-content/uploads/WhitelockB-CAA2011.pdf

• Whitelock, D. (2010) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global. pp. 319–342.


References (2)

• Whitelock, D. & Watt, S. (2008) ‘Putting Pedagogy in the driving seat with Open Comment: an open source formative assessment feedback and guidance tool for History Students.’ CAA Conference 2008, Loughborough University, 8/9 July 2008, edited by Farzana Khandia pp. 347-356 ISBN 0-9539572-7-6 http://kn.open.ac.uk/public/document.cfm?docid=11638

• Whitelock, D. & Watt, S. (2007) e-Assessment: How an we support tutors with their marking of electronically submitted assignments? Ad-Lib Journal for Continuing Liberal Adult Education, Issue 32, March 2007 pp 7-9, ISSN 1361-6323.


References (3)

• Whitelock, D. (2006) Electronic Assessment: Marking, Monitoring and Mediating Learning. In McAndrew, P. and Jones, A. (eds) Interactions, Objects and Outcomes in learning. Special Issue of International Journal of Learning Technology. Vol. 2, Nos 2/3 pp 264-276.

• Whitelock, D. & Watt, S. (2006) OpenMentor: opening tutors eyes to the written support given to students in their assignments. JISC Conference 2006, Information & Communication Technology in Education and Research. International Conference Centre, Birmingham, 14 March 2006.


automatic feedback for motivation and self-regulation

Education

evsdmw uoc open seminar

carrydmw uoc open seminar

themedmw uoc open seminar

assessment mcqs

peer assessment

cambridge assessment

authentic assessment

high stakes assessment