Overview of HeLFElectronic Management of

Assessment SurveyResults 2013

Dr Barbara Newland, Brighton

Lindsey Martin, Edge Hill

Alice Bird, Liverpool John Moores

Electronic Management of Assessment (EMA) includes a wide range of activities so the following definitions were used:

Definitions

eSubmission electronic submission of an assignment

eMarking electronic marking (including offline marking eg in Word)

eFeedback electronic feedback (ie text, audio, video but not hard copy)

eReturn electronic return of marks

To identify current practice with regard to Electronic Management of Assessment (EMA) in UK HE

To gain a snapshot of the strategic overview identifying key issues relating to strategic change, policies and practices

To reflect on longitudinal developments from findings of 2011 and 2012 surveys.

Aim

A network of senior staff in institutions engaged in promoting, supporting and developing technology enhanced learning

Over 138 nominated Heads from UK Higher Education institutions

A regular programme of well attended events

Represents the interests of its members to various national bodies and agencies including the Higher Education Academy and JISCwww.helf.ac.uk

Heads of eLearning Forum (HeLF)

The survey was available to HeLF members who were asked to respond with regard to their knowledge of their own institution.

The survey was available in March/April 2013 and took about 10 minutes to complete

The questions were a mixture of closed multiple-choice and multiple selection as well as open response type

Participants were assured that all data collected in the survey would be held anonymously and securely

No personal data was asked for or retained unless the participant indicated a willingness to participate in the follow-up activity

The results are being analysed using quantitative and qualitative methods

52 responses from HeLF members – 38% response rate

Methodology

More positive attitudes towards EMA and its normalising within their institutions

Challenges in relation to buy-in, take up and roll-out processes, functionality, service disruption and standardisation and whether the latter is desirable and achievable.

Main findings

Institution-wide policy

eSubmission eMarking eFeedback0

5

10

15

20

25

30

35

40

45

50

YesNo

Closer to having an institution-wider policy than a year ago?

eSubmission eMarking eFeedback0

5

10

15

20

25

30

YesNoDon't know

Is the move to eFeedback leading to more standardisation in terms of the feedback provided?

University-wide

Department-wide

Neither

Which software does your institution recommend for eFeedback in text format?

Turnitin (stand-alone)Turnitin (integrated into VLE)VLEHome grownOther

Current practice

05

10152025303540

University-wideSome department-wideIndividual academics

Is there evidence of increased use of mobile devices?

eSubmission eMarking eFeedback0

2

4

6

8

10

12

14

16

Academic staffStudents

Have you experienced critical issues with the eSubmission, eMarking or eFeedback software over the past year?

YesNoDon't know

Technical issues◦ service interruptions and outages◦ limited functionality and bugs with integration tools

Institutional circumstances◦ lack of university standardisation◦ lack of institutional policy◦ lack of consistency across Schools which meant that

the existing technology could not meet all the requirements of Schools

Skills - staff skills gap

Critical issues

eSubmission is a high stakes activity

Concerns about the service being hosted by a third party and also with quality standards

Other challenges:◦ unsatisfactory problem solving◦ unannounced upgrades and maintenance downtimes

during assignment submission periods◦ service not being able to cope with large numbers of

students◦ lost assignments although these were eventually

recovered

Technical concerns

Where Turnitin unavailability coincided with submission deadlines, one institutional representative reported that they had to issue advice to academics to extend deadlines.

For another institution individual staff made their own contingencies, for example, paper submission and paper feedback.

“Caused a lot of stress!”

Solutions?

Does your institution have a fully integrated EMA approach?

YesNoUnder consideration

Does your EMA system cope with the following marking processes?

Anon

ymou

s m

arking

Double

mar

king

Exte

rnal e

xam

iner

s0

5

10

15

20

25

30

YesNoDon't know

How would you rate attitudes to eSubmission?

Positive Negative Neutral Don't know0

5

10

15

20

25

30

35

40

45

50

Academic staffAdministrative staffStudents

How would you rate attitudes to eFeedback?

Positive Negative Neutral Don't know0

5

10

15

20

25

30

35

40

45

Academic staffAdministrative staffStudents

How would you rate academic staff attitudes to eMarking?

PositiveNegativeNeutralDon't know

‘There has been a change in attitude towards eFeedback, with a number of members of staff recognising that they already do this in some form.’

It is regarded as an urgent sector-wide agenda and a common student expectation, and for many institutions it is increasingly becoming embedded in departmental practice

Changing attitudes

Students

Staff

Senior management

National agendas

‘Student use. Getting positive stories from staff and student users.’

‘Positive student feedback through module evaluations and staff, student consultative committees.

Drivers

Consultation with all stakeholders

Leadership by senior management - buy-in and support

Effective communication strategy - visible campaigns,

online awareness-raising and support resources

Departmental champions

Digital literacy - the provision of training

Support from central/departmental learning technologists

Technical – robust, easy to use systems integrated with

other central systems.

Critical success factors

While mandatory policy has been a key driver in some institutions, a number of respondents suggest that inclusive policy and flexible processes work well

‘It depends very much on the culture of your organisation. Here a partnership model between central services, and School / faculty admin and academics tends to work. Top down imposition of systems is less successful. In other institutional cultures this may be different.’

Institutional differences

‘e-Submission is quite straight forward. However, we could have managed the process of e-marking and feedback better. Academic staff need a lot of time to come round to the idea if they are changing years of established practice.’

‘Never underestimate the effort involved with winning hearts and minds of colleagues.’

Challenges and concerns

Assessment is mission critical – it determines whether students achieve, progress and gain awards (and at which level)

All stakeholders in the assessment process are wary of making errors and need to be convinced of the merits in changing long-established practices

Change in practice supported by technology is only worthwhile if it effectively enhances stakeholders’ experiences and/or improves the overall process workflow

Change management - EMA

‘There is a general acceptance of e-submission policy, with fewer complaints, and anecdotal evidence points to greater academic participation in emarking and efeedback’.

Usage has moved from individual early adopters to more widespread, formalised use. There is an increased appetite for standardisation across faculties. However, a very small proportion of respondents report no, little or slow moving change in attitudes.

Summary

Dr Barbara Newland b.a.newland@brighton.ac.uk

Lindsey MartinLindsey.Martin@edgehill.ac.uk

Alice BirdA.Bird@ljmu.ac.uk

Acknowledgement: Dr Rachel Masika, University of Brighton

Contact details