evaluating e-learning efficacy university of york, uk wayne britcliffe and simon davis edinburgh...
TRANSCRIPT
EvaluatingE-Learning Efficacy
University of York, UK
Wayne Britcliffe and Simon Davis
Edinburgh Napier Learning and Teaching conference14th June 2012
What’s going on?
Drivers
Initially approached by ourDepartment of Health Sciences:– Health Sciences have extensive and embedded
online support for teaching activity– Large student numbers and a number
of CPD courses as well as UG and PG courses– Devolved VLE administration/acknowledged VLE Coordinator
Approach backed by Health Sciences senior management– Their primary goal was to establish the level of online
engagement from staff and students and alsotry and get an idea of its impact
Language and Linguistics, Law, Economics, Environment, Management– Improve dialogue and inform departmental strategy, baseline
development etc
Evaluation challenges
Students range from Undergraduates on campus through to CPD distance learners
Large number of modules to ‘audit’ (e.g. HS = 217) How to broadly classify and indicate activity
without being judgemental aboutthe effectiveness of that activity?
Establishing actual impact on learning or quantifying where value has been added can be very difficult
E-Learning evaluation not built into course design and the focus of evaluation endeavours quite wide
Difficulty in getting good survey (survey fatigue) and focus group coverage across the whole offering
Process – Step 1
Consultation– Departmental VLE coordinator liaised
with departmental management Establish reporting requirements Get senior backing for survey/report activity
– E-Learning Team liaised with the departmental VLE co’ordinator to develop evaluation plan Scoping of evaluation aims Identify useful review and
performance indicators Establish data collection methods
Evaluation plan
Focus • Establish level of on-line activity per course focussed around usage of each course’s discrete VLE module
• Try and establish the impact of the on-linesupport/activity on learning
Key Questions • How has each VLE module beenused by teaching staff?
• How has each VLE module been used andreceived by the students?
Stakeholders • For feedback : The department’s teaching staff, the department’s students
• For evaluation : ELDT, dept’s support staff and management
Time Scale/Dependencies
• Need established late 2011• HS draft report required by mid-March 2012
Instruments and methods
• Statistical querying of the VLE database • Survey of staff users• Survey of student users• Focus groups with students• Systematic audit of current VLE sites
Process – Step 2
Data Capture– Statistical querying of the VLE database
VLE administrator written queries based on stat data that is deemed useful
– Survey of staff users Develop survey questions
– Survey of student users Develop survey questions
– Focus groups with students/staff Develop focus group questions
– Systematic audit of all currentacademic year on-line sites Decide on ‘positioning’ criteria and also
on what to qualitatively look at
Data Capture – Closer look at module audit – 3E
Systematic audit of all current academic year on-line sites
Enhance – Health Sciences Example
Extend – Health Sciences Example
Empower – Health Sciences Example 1
Empower – Health Sciences Example 2
Process – Step 3 – Analysis and recommendations
EARL (reading lists)
Process – Step 3 – Analysis and recommendations
Some survey feedback from students:
“Better training for teaching staff, so they are able to offer the different opportunities for as example group files/file exchange/place to save articles and other documents to make them accessible to the whole group. This would decrease the amount of emails sent between groups and make communication easier.” “The Wiki section of the VLE
could use clear directions on conventions and getting started.”
“I don't consider myself computer illiterate, however The VLE in my opinion is extremely difficult to negotiate.”
Process – Step 3 – Analysis and recommendations
Some survey feedback from staff:
“Departmental strategy for implementation of VLE in teaching should be developed (and adhered to) - based on needs/requirements of both staff and students - this could (should) be tied into staff development so that awareness of what the VLE can do and how it can support their pedagogical framework and desired course outcomes is increased.”
“VLE sites are generally set up by someone else, and I'm not sure if they consult about structure, nomenclature or branding.”
“I find it difficult to invest in developing and maintaining the sites due to workload and technical skills.”
Process – Step 3 – Analysis and recommendations
In Brief: Early draft recommendations– Staff support and training/student support
Work with staff (targeting programmes rather than individuals) to help improve understanding and integration of E-learning delivery. Improve student induction to on-line tools/content and support conduits
– Template development Develop templates that suit specific programmes
(not one size fits all) and make sure staff understand how to use them
Moving forward
Refine evaluation processes– Bespoke survey questions/focus group questions– Create a transferable framework
Staff support and training– PGCAP: 3 E forms basis of “Intro to E-Learning
session” – Bespoke dept training: “Going further with VLE” – Position local exemplars within the framework– Build evaluation into module design so it’s easier
to establish actual impact
References
Smyth et al., 2011, Benchmark for the use of technology in modules, Edinburgh Napier University
Higher Education Academy (2008), Challenges and Realisations from the Higher Education Academy/JISC Benchmarking and Pathfinder Programme: An End of Programme Review by the Higher Education Academy, Evaluation and Dissemination Support Team, September 2008, available at http://elearning.heacademy.ac.uk/weblogs/pathfinder/wp-content/uploads/2008/09/Bench_and_PathFinalReview20080926.pdf
ACODE (2007), ACODE benchmarks for e-learning in universities and guidelines for use, June 2007, available at http://www.acode.edu.au/resources/acodebmguideline0607.pdf