‘Eyes Wide Shut’: The Challenge of Evaluating E-

Learning

Alan Masson Vilinda Ross

University of Ulster

SERA 2006

Session Overview

CETL (NI) Overview E-Learning Issues Evaluation of E-Learning VLE Data Data Health Risks Data Usage – Realistic Scenarios Making Sense of the Data Wrap

CETL Overview

Aim: “promote, facilitate and reward the adoption of a “learner centred” reflective practice approach to the development of teaching and learning, in particular wrt the use of e-learning technologies”

Posts: 1x Centre Director, 1x Academic Staff Developer, 1x Research Associate, 2 Tech Posts, 4 Content Developers, Learner Advocate

Cultural challenge: effecting changes in “teaching” practices - key to learning experience

CETL (NI): Utilising Institutional E-Learning Services to Enhance the Learning Experience

Promoting reflective practice

• Full “cycle” support– Reflect– Inspire– Develop– Evaluate

• Learner focus emphasised

e-learning “evaluation” issuesExpectations

(still) emergent and high profile – demand for evaluation

Quantitative “data” readily available

Tensions between: reporting and evaluationevaluation and reflection

Evaluation of e-learning

Scope & Activity? institutional structures, policies etc. e-teaching student learning

CETL perspectiveAssist evaluation of practiceExamine macro impact on teaching AND

learning

VLE - a source of useful data? WebCT (CE, Vista)

Logging (user centric)

Tool Usage (Communication, Assessment) (resource centric)

Reports - individual, module, cohort, subject, demographics……

Data Extraction (Log files, Review periods, Extensive data, Filter out

variables, Statistical analysis)

User location (IP address), time… session profile

Extensive VLE Reporting Options

Evaluating VLE reports?

Data Health Risks

Data requires context - Library book “impact” issue

Confidence in Data / Relevance Eyes Wide Shut vs. Open

‘Rule In’ (not trust anything except) - Eyes Open

‘Rule Out’ (trust everything except)

- Eyes Shut

Data Usage – Realistic Scenarios

Targeting subjective work (thresholds and trends) Reinforcing studies (indicative)

CETL priority - cultural change (different agenda) Raise awareness Promote reflection Challenge practices Provide new perspectives Requires careful presentation

Making Sense of the available data resourceData at Face Value vs. Probing DataDifficult to quantify (needs contexts)Making Inferences (meaningful trends,

identifying potential sample groups)Experience, Expectation (reinforcing

theory)

Benefits of approach so far

• Can better focus on learner experience• Complementary approach - allows focus

on practices and interactions• Detailed review of “data” - identifying

number of added value uses

• Language / context important to manage expectations

Complementary tools

• Survey methodologies– Teacher focused– Learner focused

• Learner reflective journals• Focus groups• Exploring conversational approaches to evaluation

– Teacher / learner models of practice

Resource intensive activities - getting best impact?

Wrap

Evaluation of E-Learning

Eyes Wide Open

Development of a transparent method of

recording practice

• Hybrid Learning Model drawing on Verpoorten’s 8LEM model and Bennett’s 30 learning verbs

Example of flash cards used to facilitate modeling of practice

Early example: hybrid learning model of T&L practice