Formative thresholded assessment: Reflections on the evaluation of a faculty-wide change in assessment practice

Sally Jordan, Department of Physical Sciences, The Open University @SallyJordan9

Acknowledgements

●Those who had the idea of formative thresholded assessment, in particular Joy Manners, Stuart Freake, John Bolton, Andrew Norton.

●Those who contributed to the evaluation: John Bolton, Lynda Cook, John Golding, Janet Haresnape, Richard Jordan, Kerry Murphy, Karen New, Ruth Williams.

●Members of the formative thresholded evaluation group and/or the Science Faculty Assessment Group: Terri Battrick, Audrey Brown, Simon Clark, Anisha Dave, Basiro Davey, Ellie Dommett, Anne-Marie Gallen, Ellen Heeley, Katherine Leys, Jane Loughlin, Jean McCloughry, Donal O’Donnell, Simone Pitman, Claire Rothwell, Peter Taylor, Vicky Taylor, Joy Wilson, Carlton Wood.

●Ben Palmer for help in extracting data; Doug Clow for advice on data wrangling; Anne Adams and Ann Jones for advice on qualitative techniques.

●Saroj Datta, Basiro Davey and Ellie Dommett for their related work on SDK125.●Associate lecturers who reported on their S141 students’ reasons for non-submission:

Amy Clarke, Catherine Halliwell, Jane Kendall-Nicholas, Maria Townsend, Pam Spence, Sarah Chyriwsky.

●Members of the module teams involved in the project, in particular Claire Rothwell and Crispian McArthur for S104 and Diane Butler and Jessica Bartlett for S141.

●Tracey Moore, Elaine McPherson, Jennie Bellamy, Gill Knight and other qualification and curriculum managers for supplying data about TMA submission rates and insight into reasons for anomalies.

●Financial assistance from eSTEeM, and support from Diane Ford, Steve Swithenby and Nick Braithwaite.

2

The Open University

●Founded in 1969

●Supported distance learning

●200 000 students, mostly studying part-time●Most undergraduate qualifications are completely open entry,

so students have a wide range of previous qualifications

●Normal age range from 18 to ??

●20 000 of our students have declared a disability of some sort

●13 000 of our students live outside the UK●Students study on modules (mostly 30- and 60-credit) which

are combined together into qualifications in a number of ways.

3

Science Faculty assessment practice

●Tutor-marked assignments (TMAs) and sometimes interactive computer-marked assignments (iCMAs) combine together into overall continuous assessment score (OCAS).

●Examination and/or end-of-module assessment (EMA) gives overall examination score (OES).

●Module result reflects both OCAS and OES.

●Students receive detailed feedback comments on TMAs and iCMAs, so they have a significant formative function.

●Assignments serve formative (“for learning”)/summative (“of learning”) double duty.

Historically…

4

Our change in emphasis

●Model A Students have to reach threshold (usually 40%) for OCAS (overall), but their module result is then determined on the strength of their OES alone.

●Model B OCAS is formative, but students have to demonstrate engagement by scoring more than 30% in x out of y TMAs and iCMAs; final score is determined by OES alone.

“Formative thresholded assessment”

5

Drivers for change

●“when assessments serve both formative and summative purposes… formative work will always be threatened due to the dominance of summative requirements” (Brearley & Cullen 2012).

●To free students from anxiety over the minutiae of grading of TMAs and iCMAs, placing greater focus on feedback and dialogue between students and associate lecturers.

●To establish more honest assessment strategies. As Freake (2008, p.4) points out: “In practice, the summative nature of the continuous assessment is somewhat illusory for the vast majority of Physics and Astronomy students since their course grade is determined by their exam mark, which is generally significantly lower than their continuous assessment mark.”

6

Drivers for change

●“To allow questions to be re-used so “more effort would be put into optimising the questions, the tutor notes and feedback, and the hints and answers provided by iCMAs, because they would be reused each year. Improvements could be made if necessary in the light of experience” (Freake, 2008, p. 4).

●To remove one of the barriers to more frequent presentation of the courses and to flexible study rates (since new assignments would not need to be produced for each presentation) and to encourage the use of ‘”little and often” assessment, for pacing (Gibbs & Simpson, 2004-5).

7

Evaluation

●Has the move to formative thresholded assessment led to a change in module completion and pass rates?

●Has the move to formative thresholded assessment led to a change in TMA and iCMA submission rates?

●Has the move to formative thresholded assessment led to a change in TMA performance?

●How do the levels of student engagement on iCMAs compare for summative, formative thresholded, and purely formative use?

●Has the move to formative thresholded assessment led to a change in the extent or type of plagiarism cases that are detected?

●What are the relative merits of Model A and Model B formative thresholded assessment?

●What do students and associate lecturers know about and think of our assessment strategies?

Our research questions:

8

Findings

TMA submission rates and module completion for two presentations of a level 3 life science module.

9

Findings

TMA submission rates and module completion for three presentations of a level 2 environmental science module.

10

Findings

TMA submission rates and module completion for three presentations of a level 2 Earth science module.

11

Impact of TMA submission on overall success

Not surprisingly, on average, students who submit all assignments do better on the examination or end-of-module assessment.

Some students would probably have been better prepared for the exam if they had done all the TMAs.

However, there is evidence that some students are making a sensible decision to omit a TMA (usually the last one), or to worry less about this assignment, and so to concentrate on revision.

N.B. Correlation does not imply causality!

12

Student engagement with iCMAs

“Learning analytics”

13

Engagement with iCMAs

Number of questions attempted in purely formative use (L) and formative thresholded use (R).

14

Engagement with iCMAs

Proportion of responses correct at first, second and third try, and not at all, for an identical iCMA in formative thresholded (top) and formative use.

15

Engagement with iCMAs

The proportion of responses correct at 1st, 2nd and 3rd try, showing the extent of repeating and blank responses for the same S104 question in summative, formative thresholded (Model A) and purely formative use.

16

iCMAs as scaffolding

Overall use of formative thresholded iCMAs with hard cut-off dates.

17

Impact on plagiarism

●A review of plagiarism cases in summer 2014 found that, whilst the re-use of assignments had been accompanied by an increase in plagiarism cases, it was not possible to attribute causality, and the increase in cases had not been as large as feared.

●The introduction of “assessment commentaries” has reduced the public availability of model answers and, arguably, given more useful feedback to weak students.

●In formative thresholded assessment, students who copy assignment answers are only really cheating themselves.

18

Student opinion

I really like the fact that I can treat the TMAs with the same dedication as I have in all the courses, but haven’t got the stress of aiming for 100% with each one. Looking back on my previous courses I think I stressed over my TMAs far too much, perhaps to the detriment of revising throughout the year in preparation for the exam. I always wanted each TMA to be perfect so that I could gain the maximum marks for the overall course assessment. The reality for me has always been that my exam grade is the one which has determined by final grade. This year I am still working hard on my TMAs but I am not stressing about every tiny detail like I used to . I think I can actually say I am enjoying my TMAs. Using this method of working has also freed up some of my quite limited time to allow me to go back and look over the previous books.

A comment from one student:

19

Student perception

●The most shocking finding of the whole evaluation has been the realisation that very large numbers of students and staff do not understand our previous, well-established assessment strategies. We need to make our practices clear.

●Students and associate lecturers also called for uniformity of practice across qualifications (though the findings of a university-wide consultation on qualification-focused assessment were less clear on this point).

●How clear and consistent is your assessment practice?

20

For the future

●Formative thresholded assessment has been very successful, but there is a reasonable anxiety that our novice students may end up being summatively assessed for the first time in an examination or another high-stakes assessment.

●We are also uneasy about over-reliance on examinations for the assessment of skills such as problem solving and the writing up of experiments; and it is difficult (impossible?) to assess collaborative activity in an examination.

●We have therefore introduced two-stage OES on all new Science Faculty level 2 modules, with 25% of the available marks coming from a tutor-marked assignment (with feedback provided in the usual way) and 75% from an examination.

21

References

Brearley, F.Q. & Cullen, W.R. (2012). Providing Students with Formative Audio Feedback. Bioscience Education, 20, 22-36.

Freake, S. (2008). New assessment models for Physics and Astronomy…”the times they are a-changing.” OpenCETL Bulletin, 3, 4-6.

Gibbs, G. & Simpson, C. (2004-5). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education, 1, 3-31.

The full report “Thresholded assessment: Does it work?” is available on the eSTEeM website: http://www.open.ac.uk/about/teaching-and-learning/esteem/projects/themes/innovative-assessment/thresholded-assessment-does-it-work 22

Sally Jordan

Department of Physical SciencesThe Open University

sally.jordan@open.ac.uk

@SallyJordan9

www.open.ac.uk/blogs/SallyJordan

23