Assessment in Higher Education: where next?

GEES conferenceSally Brown

PVC (Academic)Leeds Metropolitan University

Approaches to innovative HE assessment

How can we ensure students make the

most of assessment as part of their

overall learning experience? How can staff incorporate innovative

assessment approaches that are

manageable and fit-for-purpose? How can HEIs ensure that assessment

strategies support students and staff

while assuring quality?

Some uncomfortable questions

Are we certain that the way we assess students is working?

Is Phil Race right when he says assessment is broken?

Where is the quality in quality assurance? Do students understand what we are trying to

do with assessment? Can assement actually help students learn?

Current trends in HE design, delivery and assessment A move towards blended learning as

opposed to e-learning or traditional face-to-face approaches (assessment implications?);

Recognition of the centrality of assessment for (rather than just of) learning (stop marking, start assessing!);

The importance of teams rather than individuals in assignment design;

Recognition of divergent international approaches to assessment

My predictions for the next decade

The move away from universities being the guardians of content, where everything is about delivery, towards universities having two major functions:

Recognising and accrediting achievement, where ever such learning has taken place (not necessarily in our university but from anywhere);

Supporting student learning and engagement.

Some contextual issues Impact of the recession (staffing, £ etc) Central importance of student retention

(balanced with maintaining quality) Impacts of the National Student Survey and

various other means of expressing student (dis) satisfaction;

NUS interest: the National Feedback amnesty

External scrutiny of quality of assessment, particularly the quality and speed of turnaround of feedback.

“Assessment methods and requirements probably have a greater influence on how and what students learn than any other single factor. This influence may well be of greater importance than the impact of teaching materials” (Boud 1988)

“Students can avoid bad teaching: they can’t avoid bad assessment.” (Boud 1994)

What are students getting like? More value(s)-conscious; More litigious? More diverse; Blurred distinction between part-time and full-

time students; Demonstrating the impact of different

approaches to study in schools (new 14-19 curriculum);

Having increased expectations of diverse kinds of support.

Implications of widening participation

Ever more diverse student population; Retention of diverse students is paramount; Research (Yorke etc) tells us assessment is

central to retention; Feedback and feed forward are at the heart

of retention; Detailed and timely feedback is hugely

demanding of staff.

Some sample innovatory approaches to assessment

Assessment of learning in practice settings e.g. use of PDAs on site in clinical settings (ALPS);

Use of blogs to assess reflective practice; Groups projects to replace final year

dissertations; Exploratory work on computer-based

assessment of short answer questions; Assessment of multiple small tasks to

demonstrate achievement of practical competence (OSCEs, PASS)

Audio Feedback (‘Sounds good’) www.soundsgood.org.uk

My fit-for-purpose model of assessment: the key questions

Why are we assessing? What is it we are actually assessing? How are we assessing? Who is best placed to assess? When should we assess?

To integrate assessment we need to realign it with the curriculum by:

Exploring ways in which assessment can be made integral to learning;

Constructively aligning (Biggs 2003) assignments with planned learning outcomes and the curriculum taught;

Providing realistic tasks: students are likely to put more energy into assignments they see as authentic and worth bothering with;

Us more and better formative assessment.

The guru Sadler“The indispensable conditions for improvement are

that the student comes to hold a concept of quality roughly similar to that held by the teacher, is able to monitor continuously the quality of what is being produced during the act of production itself, and has a repertoire of alternative moves or strategies from which to draw at any given point. In other words, students have to be able to judge the quality of what they are producing and be able to regulate what they are doing during the doing of it”. (Sadler 1989).(my italics)

Formative and summative feedback: two ends of a continuum

Formative assessment is primarily concerned with feedback aimed at prompting improvement, is often continuous and usually involves words.

Summative assessment is concerned with making evaluative judgments, is often end point and involves numbers.

What’s the difference between formative & summative assessment?“Summative contrasts with formative assessment in

that [the former] is concerned with summing up or summarizing the achievement status of a student, and is geared towards reporting at the end of a course of study especially for purposes of certification. It is essentially passive and does not normally have immediate impact on learning, although it often influences decisions which may have profound educational and personal consequences for the student..” (Sadler 1989).

Using formative and summative assessment appropriately

Students often don’t value formative assessment; All assessment needs to be fair, consistent, reliable,

valid, manageable and transparent; Many assessment systems fail to clarify for students

the purposes of different kinds of assessment activity;

Low-stakes formative assessment helps students, especially those from disadvantaged backgrounds, understand the rules of the game.

Some problems with formative assessment

Students may not take it as seriously as summative assessment, if it doesn’t ‘count’;

It can be hugely time consuming; Students are likely to need different kinds of

formative assessment at different stages in their ‘learning journeys’;

It can be difficult to gauge how best to do it with groups of students who may be at different stages of development.

How can we make students take feedback more seriously? We can: Spend time and energy helping students to

understand the importance of feedback and the value of spending some time after receiving work back to learn from the experience. Most students don’t do this at the moment, concentrating principally on the mark.

Withhold the mark until after the student has received and responded to feedback;

Provide assessed opportunities for reflection on previously marked work.

We can make feedback timely Aim to get formative feedback on work back to

students very quickly, while they still care and while there is till time for them to do something with it.

The longer students have to wait to get work back, especially if they have moved into another semester by the time they receive their returned scripts, the less likely it is that they will do something constructive with lecturer’s hard-written comments.

Don’t bother with detailed feedback for non-continuing students

We can concentrate formative feedback where it can do most good

Don’t give detailed written feedback to students on work that is handed back at the end of the semester if that area of study is no longer being followed by the student: just give a mark or grade;

Give more incremental feedback throughout the semester (and if university systems don’t allow this, change the systems!);

Ensure that students at the top end of the ability range don’t feel short changed by minimal feedback.

Use formative assessment to promote independence

Investigate how learning can be advanced in small steps using a ‘scaffolding’ approach;

Provide lots of support in the early stages when students don’t understand the ‘rules of the game’ and may lack confidence;

This can then be progressively removed as students become more confident in their own abilities.

Consider providing opportunities for multiple assessment

Consider allowing resubmissions of work as part of a planned programme;

Students often feel they could do more will work once they have seen the formative feedback and would like the chance to have another go;

Particularly at the early stages of a programme, consider offering them the chance to use formative feedback productively;

Feedback often involves a change of orientation, not just the remediation of errors.

Use formative assessment to help students with reading Help them also to understand that there are different

kinds of approaches needed for reading depending on whether they are reading for pleasure, for information, for understanding or reading around a topic;

Guide them to become active readers with a pen and Post-its in hand, rather than passive readers, fitting the task in alongside television and other noisy distractions;

Give them clear guidance in the early stages about how much they need to read and what kinds of materials they need to focus on.

From Bowl (2003) Non-traditional students in HE

The hardship was not understanding. When they give you an assignment and say it was on this handout. But my difficulty is not understanding what to do at first… I think that there’s a lack of my reading ability, which I can’t blame anyone for. I can only blame myself because I don’t like reading. And if you don’t read, you’re not going to learn certain things. So I suppose that’s to do with me…..it’s reading as well as putting what you read into your essay. You can read it and understand it. I can read and understand it, but then you have to incorporate it into your own words. But in the words they want you to say it in, not just: She said this, and this is the way it should be. The words, the proper language.

(Bowl 2003 p90).

Use formative assessment to help students with writing Devote energy to helping students understand

what is required of them in terms of writing; Work with them to understand the various

academic discourses that are employed within the subject/institution;

Help them to understand when writing needs to be personal and based on individual experience, such as in a reflective log, and when it needs to be formal and using academic conventions like passive voice and third person, as in written reports and essays.

Involve students in their own and each other’s assessment

Consider ways of getting students to give each other meaningful formative feedback;

Shared reflection is the best means available to help them really get inside the criteria;

Asking students to review each others draft material prior to submission particularly helps those who lack confidence about what kinds of things are expected of them.

Providing opportunities for multiple assessment? Consider allowing resubmissions of work as part

of a planned programme; Students often feel they could do more will work

once they have seen the formative feedback and would like the chance to have another go;

Particularly at the early stages of a programme, consider offering them the chance to use formative feedback productively;

Feedback often involves a change of orientation, not just the remediation of errors.

To integrate assessment we need to realign it with the curriculum by:

Exploring ways in which assessment can be made integral to learning.

Constructively aligning (Biggs 2003) assignments with planned learning outcomes and the curriculum taught;

Providing realistic tasks: students are likely to put more energy into and play fairer with assignments they see as authentic and worth bothering with.

Making assessment work well Intra-tutor and Inter-tutor reliability need to be

assured; Practices and processes need to be transparently

fair to all students; Cheat and plagiarisers need to be

deterred/punished (see bonus features); Assessment needs to be manageable for both staff

and students; Assignments should assess what has been

taught/learned not what it is easy to assess.

Students benefit if we can make feedback timely Aim to get feedback on work back to students

very quickly, while they still care and while there is till time for them to do something with it.

The longer students have to wait to get work back, especially if they have moved into another semester by the time they receive their returned scripts, the less likely it is that they will do something constructive with lecturer’s hard-written comments.

Can we provide opportunities for multiple assessment? Consider allowing resubmissions of work as part

of a planned programme; Students often feel they could do better once

they have seen the formative feedback and would like the chance to have another go;

Particularly at the early stages of a programme, we can consider offering them the chance to use formative feedback productively;

Feedback often involves a change of orientation, not just the remediation of errors.

Using formative assessment to promote independence and learning

Investigate how learning can be advanced in small steps using a ‘scaffolding’ approach;

Provide lots of support in the early stages when students don’t understand the ‘rules of the game’ and may lack confidence;

This can then be progressively removed as students become more confident in their own abilities.

Play fair with students by avoiding using ‘final language’ (Boud)

Avoid destructive criticism of the person rather than the work being assessed.

Try not to use language that is judgmental to the point of leaving students nowhere to go.

Words like “appalling”, “disastrous” and “incompetent” give students no room to manoeuvre.

However, words like ”incomparable” and “unimprovable” don’t help outstanding students to develop ipsatively either.

However

“The sound recording on this assignment is exemplary. I always feel as a tutor that I should give you ideas on how to improve your work further but I doubt you will ever in your life do a drum recording that is better balanced and pitched than this”

Play fair by giving feedback to students with diverse abilities

Students at the top end of the ability range sometimes feel short changed by minimal feedback;

Students with many weaknesses easily become dispirited if there is too much negative feedback;

Consider giving an assessment sandwich. Start with something positive, go into the detailed critique and find something nice to say at the end (to motivate them to keep reading!);

Explore ways to incentivise reading of feedback; Consider which medium to use for students with

disabilities (e.g. don’t use bad handwriting for those with visual impairments or dyslexia!).

Conclusions Concentrating on giving students detailed and

developmental formative feedback is the single most useful thing we can do for our students, particularly those who have had a struggle to achieve entry to higher education.

Summative assessment may have to be rethought to make it fit for purpose;

To do these things may require considerable imagination and re-engineering, not just of our assessment processes but also of curriculum design as a whole if we are to move from considering delivering content the most important thing we do.

Useful references: 1

Biggs J (2003) Teaching for Quality Learning at University (Maidenhead: SRHE & Open University Press)

Bowl, M (2003) Non-traditional entrants to higher education ‘they talk about people like me’ Stoke on Trent, UK, Trentham Books

Brown, S. Rust, C & Gibbs, G (1994) Strategies for Diversifying Assessment Oxford Centre for Staff Development.

Boud, D. (1995) Enhancing learning through self-assessment London: Routledge.

Brown, G. with Bull, J. and Pendlebury, M. (1997) Assessing Student Learning in Higher Education London: Routledge.

Brown, S. and Glasner, A. (ed.) (1999) Assessment Matters in Higher Education, Choosing and Using Diverse Approaches, Maidenhead: Open University Press.

Brown, S. and Knight, P. (1994) Assessing Learners in Higher Education, London: Kogan Page.

Useful references 2

Brown, S., Race, P. and Bull, J. (eds.) (1999) Computer Assisted Assessment in Higher Education London: Routledge.

Carroll J and Ryan J (2005) Teaching International students: improving learning for all Routledge SEDA series

Falchikov, N (2004) Improving Assessment through Student Involvement: Practical Solutions for Aiding Learning in Higher and Further Education, London: Routledge.

Gibbs, G (1999) Using assessment strategically to change the way students learn, In Brown S. & Glasner, A. (eds.), Assessment Matters in Higher Education: Choosing and Using Diverse Approaches Maidenhead: SRHE/Open University Press.

Kneale, P. E. (1997) The rise of the "strategic student": how can we adapt to cope? in Armstrong, S., Thompson, G. and Brown, S. (eds) Facing up to Radical Changes in Universities and Colleges, 119-139 London: Kogan Page.

Useful references 3 Knight, P. and Yorke, M. (2003) Assessment, learning and

employability Maidenhead, UK: SRHE/Open University Press. McDowell E & Brown S 1998 Assessing students: cheating and plagiarism, Red

Guide 10/11 University of Northumbria, Newcastle Mentkowski, M. and associates (2000) p.82 Learning that lasts:

integrating learning development and performance in college and beyond San Francisco: Jossey-Bass.

Peelo, M and Wareham, T (eds) (2002) Failing Students in higher education Buckingham, UK, SRHE/Open University Press.

Sadler, D R (1989) Formative assessment and the design of instructional systems Instructional Science 18, 119-144.

Sadler, D R (1998) Formative assessment: revisiting the territory Assessment in Education: Principles, Policy and Practice 5, 77-84

Pickford, R. and Brown, S. (2006) Assessing skills and practice London: Routledge.

Useful references 4

Race, P. (2001) A Briefing on Self, Peer & Group Assessment in LTSN Generic Centre Assessment Series No 9 LTSN York. Race P. (2006) The lecturer’s toolkit (3rd edition) London: Routledge.

Race P (2006) The Lecturers toolkit 3rd edition London Routledge Race P and Pickford r (2007) Making Teaching work: Teaching

smarter in post-compulsory education, London, Sage Rust, C., Price, M. and O’Donovan, B. (2003). Improving students’

learning by developing their understanding of assessment criteria and processes. Assessment and Evaluation in Higher Education. 28 (2), 147-164.

Ryan J (2000)A Guide to Teaching International Students Oxford Centre for Staff and Learning Development

Stefani L and Carroll J (2001)A Briefing on Plagiarism http://www.ltsn.ac.uk/application.asp?app=resources.asp&process=full_record&section=generic&id=10

Yorke, M. (1999) Leaving Early: Undergraduate Non-completion in Higher Education, London: Routledge.

Bonus features

Plagiarism

Cheating and plagiarism: why have concerns about them increased?

Mass access to HE. More reported bad practice. Changes in assessment practice. Wider use of communication and information

technologies, especially the web. Higher stakes: the importance of getting good

grades.

How do new technologies change things?

“Plagiarism used to be hard work…hours in the library, researching what to copy…..the plagiarist used to learn a lot while trying to get out of doing the work”

Jim Evans, University of Warwick, quoted by Jude Carroll, Oxford Brookes University.

What is plagiarism?

Passing off someone else’s work as your own.

Wholesale ‘lifting’ of entire assignments/ texts.

Patching and paraphrasing. Purchasing or commissioning work.

What about unintentional plagiarism?

Readers may not be conscious how much they have themselves absorbed;

Schools may encourage the learning and re-use of model answers;

In some cultures, your teacher or text books are honoured sources and there is nothing in appropriate about repeating their words.

Tell-tale signs Outdated or obscure references; Dramatic changes in writing ability; Inappropriate cultural references; Layout peculiarities; American spellings/vocabulary (other than

from US students and those used by conventional spell checkers).

Modified from Bill Johnson (2003) The concept of plagiarism Manchester Metropolitan University ‘Learning and Teaching in Action’ Volume 2 No 1

Four strategies to control plagiarism and cheating

Use strict controls. Make the rules clear and have known

penalties (and apply them). Design assessment instruments that make

cheating difficult. Develop a climate that will reduce the

likelihood of cheating.

Design assessment instruments that make cheating difficult

Use open–book rather than closed book exams.

Use assignments reliant on personal experience.

Ask students to produce learning/reflective journals and critical incident accounts.

Use vivas and orals. Design differentiated assignments. Provide assignments with choice and

individual activity.

More anti-plagiarism assignments

Use computer based assessment. Involve 3rd party verification. Ask them to provide photocopied annotated

source material. Use group assessment. Involve an element of peer assessment. Give students tasks in learning teams. Monitor the production of assessed work: use

staged assignments.

Bonus features

Using technologies to support assessment

Use technologies to support assessment for learning Employ computer-assisted formative

assessment with responses to student work automatically generated by email;

Students seem to really like having the chance to find out how they are doing, and attempt tests several times in an environment where no one else is watching how they do;

We can monitor what is going on across a cohort, so we can concentrate our energies either on students who are repeatedly doing badly or those who are not engaging at all in the activity.

Giving feedback electronically: you can use Emailed comments from you to students on their

individual work. As students sit the exam, put up generic

feedback on the VLE Email overall generic comments on assignments

to the whole cohort of students or through a computer conference.

Use computer-delivered feedback. (There is an interesting OU research project currently being undertaken to give formative feedback to students on electronically submitted work).

Computer-based assessment is valuable to learning when:

It provides students with multiple opportunities to test and re-test their understanding;

It offers formative feedback in response to wrong (and right) answers;

It blurs the distribution between teaching and assessment.

CAA requires capital investment because:

Any fool can produce CAA (and a lot of fools do);

Good CAA requires good pedagogic understanding, up-to-date content and sound technological systems;

The investment of time to produce, pilot and test good CAA resources must be extensive.

Social networks can support assessment when:

There are clear purposes for the assessment of learning opportunities;

They foster a genuine shared learning environment, aiming to build a learning community;

Students feel equally able to contribute to network communities.

They are less successful when:

Academics attempt to hijack students’ own social networks for their own purposes;

There is no clear focus for learning or assessment activities in shared spaces;

Tasks and activities are minimal/ tokenistic.

Key questions on future TEL developments for assessment How can we assess student learning effectively

and efficiently in ways that go beyond multi-choice questions, so we are testing deeper learning behaviours than simply recall and memorisation?

How can we evaluate more effectively what learning students have already and what more they need to do in order to achieve the outcomes we can recognise as being worthy of a degree or other award?

How can we help students learn to differentiate between poor quality and good quality content?

More questions How can we help students learn cooperatively in

ways that emulate but don't replicate their social learning spaces (they hate it when they find faculty staff“ intruding onto face book" and yet they don't use the learning spaces we set up for them much)?

Are there technology supported templates we can develop which would be better than the cumbersome and people-intensive APEL (Accreditation of Prior Experiential learning) processes we currently use.

And some more

How can we best make use of the technologies almost all students have nowadays for assessment (mobile phones, MP3 players, iPods, PDAs to support learning?

Some of our distance learning students in developing countries, don't have much in the way of PCs or laptops but lots of them have mobile phones.