evaluating educational interventions paul stevenson
TRANSCRIPT
Evaluating Educational Interventions
Paul Stevenson
Background
• Local evaluation of EBP training showed:Skills & Knowledge improved ….
..but limited change in behaviour.
•Miller, G E The assessment of clinical skills/performance
•Academic medicine, 1990, 65, S63-7
• How can I make the training more effective?
• I had used the Kirkpartick Model for Evaluation
Kirkpatrick Model – Learner Focused
Level 1 – Learner reaction
Level 2 – Modification of learner attitudes & perceptions,
and learner acquisition of knowledge & skills
Level 3 - Change in learner behaviour
Level 4 - Benefits to the organisation/patient resulting from learner performance
Kirkpatrick’s four levels of evaluation (1967,1994), adapted by Barr et al (2000)
Level 1 – Learner reaction
Level 2 – Modification of learner attitudes & perceptions,
and learner acquisition of knowledge & skills
Level 3 - Change in learner behaviour
Level 4 - Benefits to the organisation/patient resulting from learner performance
Kirkpatrick Model
Kirkpatrick ModelTendency for declarative outcome measures
Shows if the intervention has been effective……but provides little information about why
Learner FocusedFails to consider contextual/environmental variablesView that training is solely responsible for any outcome
One DirectionalTendency to focus on the impact of the learner on their environment
• What alternative methods could be used?
Transfer of LearningBaldwin & Ford (1988) transfer of training into practice. Three areas of influence on transfer :
1. Aspects of the course
2. Characteristics of the learner
3. Features of the work environment
Common Barriers
• The most commonly mentioned barriers to EBP are:
Learner
•Personal lack of interest
•Lack of skills
•Low self-efficacy
•Previous experience
Training
•Appropriate content
•Delivery method
Environment
•Organisational culture
•Insufficient time
•Lack of resources
•Lack of support from colleagues and management
Salbach et al, 2007; Pollock & Legg, 2000; Jette et al, 2003; McColl & Smith, 1998; Freeman & Sweeny,2001; Retsas, 2000; Cranney & Warren, 2001; Metcalf et al, 2001
Diffusion of Innovation
Diffusion of innovation model, Rogers (1995)
LET evaluation
Lewin’s (1951) equation B=f (P,E) Behavior is a function of the person and the environment
Adapting this to evaluate the impact of an educational intervention we get: BC= f (L,E,T)
Behavioral change is a function of:L – Learner E – Environment T- Training
Rather than focus solely on the impact of the learner on the environment, we also measure the impact of the environment on the learner.
Systems Approach
• System: Integrated whole composed of several parts
• Look at the educational intervention holistically
• Look for relationships with other parts of the organisation (L&D, HR, Clinical Audit)
Example Measures
Learner:Pre-post course evaluation formTest exercise/paper to show learning has occurredLikert/VAS measure of self-efficacy, desire for change and other psychological aspects of the learner.
Post-course follow up – has learning been retained, examples of how training has impacted on work, reasons/barriers if new learning has not been used.
Monitor usage statistics such as Athens Usage
Example MeasuresEnvironment:
Floor Walking – observe the working environment: is there access to appropriate resources, space/time for staff to implement EBP.
Dialog with Line Management – Creates a two way exchange of information
Integrate Library Services with the wider organisation – If you are working in the same environment as your learners you will experience the culture/environment first-hand.
Example Measures
Training:Pre-post course evaluation formTest exercise/paper to show learning has occurred
Post-course follow up – has learning been retained, examples of how training has impacted on work.
Peer-observation
Advantages
Truly understand the wider impact of the training
Identify and have a better understanding of the barriers which exist and can start to resolve them.
Increases profile and importance of library services within organisation / departments
