Designing useful evaluations

Dr Rachel A Harris

JISC Assessment & Feedback Programme Online workshop

7th November 2011

Inspire Research Ltd

Aims of the session

1. Look at the evaluation cycle.2. Consider project ‘logic’.3. Review anticipated outcomes & impact.4. Appraise which project activities to evaluate.5. Identify evaluation questions.6. Review how the evaluation will be undertaken.7. Find out what other projects are doing!8. Consider the baseline 9. Other evaluation issues?

Inspire Research Ltd

Curriculum Delivery Programme & the role of Evaluation

The JISC & Becta funded programme invited projects to “transform how they deliver and support learning across a curriculum area through the effective use of technology”.– Emphasis on gathering and using evidence– Identifying what works– Adding value by generating evidence

– How did projects determine if delivery was ‘transformed’ and the technology was ‘effective’?

Project outputs: http://jiscdesignstudio.pbworks.com/

Inspire Research Ltd

Initial evaluation stages

• Envision potential impact /achievements:– potential indicators of impact/achievement,– credible evidence of impact, from different

stakeholders’ perspectives.

• Baseline:– first set of evaluation data– review the current context – identify existing practice– determine stakeholders’ attitudes– clarify challenges (and identify barriers)

Inspire Research Ltd

A multitude of approaches

• Action research (Atelier-D, Duckling, KUBE)• Independent internal/external evaluator

(Cascade)• Appreciative inquiry (ESCAPE)• Balanced scorecard (COWL)• Formative evaluation (MAC)• Micro & macro data sources (Springboard TV)• Rich qualitative methods (ISCC)

Inspire Research LtdMcNiff and Whitehead’s (2006) ‘Action-reflection cycle’

Observe

Reflect

ActEvaluate

Modify

Move in new directions

ActionResearch

Baseline

Pilot

Revisit

Review

Inspire Research Ltd

Innovation ServiceEfficiency

VLE Support for Courses

Online Delivery of Generic Content

Online Assignment Handling

Course Design

Online Payment & Enrolment

Admin handling time

Adoption rate

IT set-up and support time

User acceptance

Technology confidence

Technology engagement

Technology best practice

Ease of use

Service response time

Customer satisfaction

Aims

Focus Areas

Evaluation Areas

Usage rate and areas

CIPP Evaluation (Cascade)

Inspire Research Ltd

Assessment & Feedback ProgrammeThe programme focuses on “large-scale changes in assessment and feedback practice, supported by technology

“with the aim of enhancing the learning and teaching process and delivering efficiencies and quality improvements”.– Pedagogic enhancements– Workload efficiencies – Building the business case

Inspire Research Ltd

Designing your evaluation (I)

1. What do you want to happen as a result of your project’s assessment and feedback innovation?(Anticipated Outputs and Outcomes, & Anticipated Impact)

2. Who and/or what might this innovation impact on? (Stakeholders & think about levels)

3. How will you know the intended outcomes have been achieved? Or What indicators could you use to demonstrate what has been achieved? (Measures)

Inspire Research Ltd

What do you want to know?&What indicators could you use?

Inspire Research Ltd

Report back

Inspire Research Ltd

Big picture evaluation questions

• Process evaluation – administrative and pedagogical design, and implementation.

• Outcome evaluation – value of outcomes.

• Learning – barriers and enablers, surprises, causal explanations.

• Overarching questions about value/worth.

• Forward or outward focused evaluation questions – reusability, sustainability.

Inspire Research Ltd

Questions ‘cheat sheet’

• How well was the project designed & implemented?• How valuable were the outcomes to participants

(students, lecturing staff, administrators)? To the institution, the community, the economy?

• What were the barriers and enablers?• What else was learned?• Was the project worth implementing?• To what extent are the project’s content, design or

implementation likely to be valuable elsewhere?

Inspire Research Ltd

Share your evaluation questions

Some examples:• What value do students and tutors gain from using

spoken e-feedback?• How has providing spoken rather than written feedback

impacted on tutor workload? • What value did teachers and students gain from

collaborating long-term on assessment and feedback?• What impact has providing ‘balance of assessment’ data

had on programme teams?

Inspire Research Ltd

Designing your evaluation (II)

• How will you find out whether the intended outcomes have been achieved? (What, How and When evidenced)

• How will you use this information?

• Who else might this information be of use to?

• What other questions should be considered?

Inspire Research Ltd

Existing sources What existing sources of evidence are

you planning to use?1. Previous course evaluations 2. Departmental timesheets3. Data from previous studies4. Centrally held institutional data5. NSS data6. No plans to use existing data7. Others?

Inspire Research Ltd

How will you find out?

Inspire Research Ltd

The Baseline

• Where are you starting from?

• Qualitative versus quantitative evidence

• Hitting a moving target• Previous projects’ baselines:

http://jiscdesignstudio.pbworks.com/w/page/46422956/Example%20baseline%20reports

Inspire Research Ltd

What happens next?

• Reflect on your thinking from today.• Review the feedback on your draft

plans.• Consider how you might want to

refine your evaluation plans.• Use your evaluation questions to

inform baseline activities.• Programme mapping!

Inspire Research Ltd

Any questions?

Programme Evaluation Resources https://programmesupport.pbworks.com/w/page/42511148/Evaluation%20Resources

Dr Rachel A Harristwitter: raharris email: rachel@inspire-research.co.ukweb: www.inspire-research.co.uk