Feedback as dialogue and learning technologies: can e-assessment be formative?

Stylianos HatzipanagosKing's College London

April 2010

The seminar will:

Put forward a conceptual model of formative assessment and examine how this can be made to work purposefully within the specific constraints of ODL;

Explore the relationship between formative assessment and learning technologies and the opportunities for computer mediated communication and dialogue in these environments.

Closing the loop: two projects

explored policy, teaching practices and tutor and

student views in three Open and Distance Learning environments.

investigated relationship between formative assessment and learning technologies

Projects team

♣ Paul Black (King’s)♣ Stylianos Hatzipanagos (King’s)♣ Ana Lucena (IOE)♣ Bob McCormick (OU)♣ Steven Warburton (King’s)

Types of assessment

summative (SA) numerical results about student performance generally carried out at the end of a course/programme formative (FA) a continuous/periodic appraisal of students’

performance may inform and improve competence and contribute to

further learning (Sadler, 1989; Boston, 2002; Black et al., 2003; York, 2004)

Formative or assessment for learning(Albon, 2003; Wiliam et al., 2004; Nicol & MacFarlane-Dick, 2004;

Black, 2005)

Duality “SA/FA” may not represent opposite poles of assessment (Hargreaves, 2005; Dylan, 2006)

FA is SA with feedback, which can be used by the learner (Taras 2005)

positive implications for student learning allows students to play a more active role in management of own

learning (Nicol 1997)

Purposes of assessment types (Hatzipanagos & Warburton 2009, adapted from Bull & McKenna 2004)

Type of assessment

Feedback to student

Feedback to tutor

Self testing Determining level/progression

diagnostic 2 2 1

formative 1 2 1 2

summative 2 1

Primary purpose: 1, Secondary purpose: 2

Dimensions of FA Power (autonomy and ownership)DialoguePeer/self assessmentFeedback (attributes)VisibilityReflectionAction

Black et al. (2003), Gibbs & Simpson (2004), Giroux (1992), McConnell (2006).

Feedback Comments – what messages are being conveyed?

! ‘be more critical’

‘you’re argument needs to be more academic’

X ‘More use of analysis would have helped’

‘requires further elaboration’

‘you need to think

carefully about your structure’

‘You write in a journalistic style’

?

Medland 2009

FA and technologies Learning technologies promote innovative assessment

practices and lead to deeper thinking about how tutors conceptualise assessment in higher education (McCormick 2004).

assessment practices have been supported by technology for many years. However…

….main focus on developing tools such as objective tests rather than addressing fundamental issues, such as how they can be used to support effective assessment approaches (Nicol and Milligan, 2006).

Characteristics of FA technologies: potential

Can be used as living record of student learningBlur boundaries between formal and informal learning

spacesChallenge dominant model of hierarchical, tutor

centred education Embrace an ideology of openness, dialogue and

ownership.

Open & Distance Learning

ODL environments : necessity for FA practices. proactive in FA practices out of need to provide

systematic feedback to students.

e-literacy in pedagogical terms

Required set of competencies for teaching practitioners is more complex and consists of:

Updated knowledge of e-learning resources and their role in enriching the learner experience.

Expertise of range of tools and of the pedagogical affordances that each provides

Ability to evaluate technologies and engage in reflection on how they impact on learning

Staff Perspective‘...part of the feedback given to pupils in class is like so

many bottles thrown out to sea. No one can be sure that the message they contain will one day find a

receiver’ (Perrenoud, 1998: 87)

Student Perspective

Dearing Report (1997)

National Student Survey (2004-present)

Methodology Open-ended interviews with 17 tutors and 20 students to

explore perceptions of assessment. Informed content of online questionnaire that was

administered to students within the three ODL environments. In the online survey, students were asked to express their level

of agreement to a number of statements about a five-point Likert scale.

Qualitative data using also open-ended questions in the questionnaire.

Institutional differences

Huge diversity in practices in all three of the environments.

From the three environments, two were broadly similar, and they were characterised by consistent elements of good practice.

One of the three environments had in place an infrastructure to provide more systematic provision of feedback. In addition, there was in place a framework, which emphasised periodic assessment rather than end of year assessments.

Institutional cultures and the nature of an institution determine assessment practices:

King’s ODL and the External programmes broadly similar.

External programmes : there is an indication of huge diversity in practices.

Good practice in policy : policy documents that set the frameworks within which the tutors will operate

and increased use of e-assessment. Infrastructure to provide more systematic provision of feedback. Framework, which emphasises periodic assessment rather than end of year

assessments. Particularly at the OU tutors are continuously monitored about their assessment

practices.

Tutor engagement

Disjunction between beliefs, ambitions and pragmatic approaches to the use of FA. Practitioners’ attitudes to FA context-dependent and sometimes discipline-oriented.

Classification of approaches in two substantial groups and one smaller:

The first two claimed that they were proponents of FA; they either used FA in their practices (group 1) or claimed that they did not, however they would consider it if pragmatic constraints allowed it (group 2)Third smaller group did not consider FA necessary for their context.

Significant number of tutors engaged in FA but not extensively in what we consider FA.

Notion of FA varied e.g. often was equated to ‘continuous assessment’. Formats of assessment have changed because of the

possibilities new technologies can afford.

Student engagementTarget audience of institutional groups were diverse in terms

of perceptions.Students’ attitudes to assessment were not discipline

dependent. It was the broad context (the ODL environment) that determined attitudes.

Facilitation of feedback mechanisms using computer mediated communication was recognised as a significant component of the assessment process.

Majority of learners recognised the challenges in providing a suitably formative environment in these settings.

Difficulties in defining their personalised learning environment and the affordances of the tools they used.

Q31. I am interested in the marks and not in the feedback.

3.8% 13.4% 41.8% 41.0%Always Often Sometimes Never

0

50

100

150

200

250

300

350

400

Series1

Q24. I would learn more if I received feedback

28.9% 27.1% 44.0%more frequently with longer comments shortly after submission of

my work

0

50

100

150

200

250

300

350

400

Series1

Q28. The feedback guides me on what I need to do to improve my work.

37.9% 43.5% 16.9% 1.6%Always Often Sometimes Never

0

50

100

150

200

250

300

350

400

Series1

Q13. I have used a computer to receive feedback.

83.6% 13.2% 3.8%Yes No Not sure

0

100

200

300

400

500

600

700

800

Series1

e-AssessmentThe projects considered whether current formative

assessment practices can cope with emerging technologies.

Learning technologies (a fundamental channel of student support)

Received favourably, though overall there was no sense of ownership or always a clear perception of purpose regarding the affordances of each technology.

e-assessment: range of technologies

Non-formative

Objective tests (non adaptive and they ‘disagree’ with certain disciplines)

Model answers received or revealed after students submit an answer, as non-personalised feedback

Electronic submission of coursework

Formative

Communication tools in VLEsOnline tutorial environments Games that allow monitoring and intervention Audio to canvas opinions/understanding of concepts/issues (audio more

meaningful conceptually than video)Tools such as certainty based markingVideoconferencingSocial software: Blogs, Wikise-portfolios

Key processes of formative assessment

Traditional technologies

Social software

email discussion

board quiz blog wiki

Power (ownership and autonomy)

H L L H H

(distributed)

Dialogue

L to M H L H (informal

tone) M

Timeliness

L to M H variable H H

Visibility

L variable L H H

Action

L to M L to M M H H

Community (peer support)

L variable L H (via

aggregation) H

Reflection

variable M M H M

Relationship between social software & FA

…social dimensions of social software allow for formative assessment practices to be re-invented or at the very least facilitated by participative and student focussed interventions.

(Hatzipanagos & Warburton 2009)

Four Stages to embed FA practices in the assessment cycleStage Zero Assessment mostly through exams and end of assessment term projects.

No provision of feedback.

Limited or no peer/self assessment opportunities.

Limited or no use of learning technologies to support assessment practices.

Stage One Generalised feedback on student work but of limited customisation to the needs of the individual learner.

Examiner reports with model answers for monitoring/evaluating assessment practice.

Use of learning technologies to support assessment through objective tests.

Stage Two All the above in Stage One plus:

Periodic/continuous assessment for learners to rehearse arguments that they will use in end of assessment period assessments.

Feedback is monitored, to ensure that students will act upon negotiated targets and the feedback loop is closed.

Learner responses to feedback become an essential part of the assessment cycle.

Use of learning technologies such as computer mediated communication to facilitate the assessment cycle.

Stage Three All the above in Stage Two plus:

Peer/self assessment.

Student involvement in setting marking criteria.

Use of learning technologies to peer review, and to construct knowledge collaboratively.

Hatzipanagos 2010

Conclusions

Towards an understanding of the significance of formative assessment in ODL.

A conceptual model of formative assessment and how this can be made to work purposefully within the specific constraints of ODL environments.

FA practices can be problematic in courses with emphasis on end of year assessments as the ‘closing the loop’ component of the assessment process very rarely takes place.

FA can enrich e-learning approaches by making the feedback central to all e-assessment activities.

e-assessment tools promote a dialogue in relation to feedback, peer and self assessment activities which by their nature place the student at the centre of the educational process.

Some questions to explore

1. Are assessment practices in HE stifling innovation? 2. How formative SA practices must be to have an

impact on student learning? 3. Can self or peer assessment practices be summative

or only formative?4. How important is context on assessment practices

and student performance?5. How do we accredit learning that takes place in

informal spaces afforded by learning technologies

Further work ♣ Comparison of current technology supported FA

mechanisms across the ODL landscape.♣ identify types of processes and learning technology

tools that might best support effective feedback♣ encourage effective feedback approaches that both

empower and enhance the learning experience for the distance learner.

If you are interested to find out more …

email s.hatzipanagos@kcl.ac.uk