back to our future - eppicinc.files.wordpress.com · © 2012 atlantic training inc. back to our...

© 2012 Atlantic Training Inc.

Back to our Future: Evidence-Based Practice for ISPI

Dick Clark Atlantic Training Inc. and U.S.C.

[email protected] -- www.cogtech.usc.edu

THE Performance Improvement Conference

Toronto, Canada April 18, 2012

Illustration by Jonathan Rosen for Mother Jones

mailto:[email protected]

http://www.cogtech.usc.edu

© Atlantic Training Inc., 2012

What is evidence-based practice?

“ A commitment to a constant reexamination of practices

through research and outcome analyses to enhance our

knowledge base and our decision-making”

Sackett et al, 1997 “Evidence-based Medicine”


What is evidence-based performance improvement ?

“The use of empirically verified interventions and solutions to

improve human performance (results) at the individual, team,

enterprise and community levels”

Hugh Oakley-Browne

Oakely-Browne Associates Limited

http://www.oakley-browne.co.nz/


Benefits of Evidence-Based PI

More opportunities realized and more problems resolved

Reduce wasted effort

Significantly improved productivity

Quicker results with more front end effort

Reduced costs

Reduction in negative second and third order effects

As CPT’s we fully commit to “Do No Harm”

Solutions can often be commoditized

New members will be attracted to ISPI

Clients are more satisfied and confident of ROI


Risks of Evidence-based PI

Some members may resist trusting others about evidence

We may have difficulty agreeing about what the evidence “says”

Some elected ISPI officers and staff may resist implementing

Some clients may refuse to implement

Revising standards, training, policy and procedures is expensive


What is evidence?

Level Source and type of Evidence in Most EBP fields

1st Best Meta-analysis or replicated randomized controlled trials (RCT)

including a control condition and baseline measures.

2nd Best At least one RCT with placebo or active control condition,

evidence from multiple time series with intervention,

comprehensive reviews of multiple, peer reviewed studies.

3rd Best Single study with adequate controls, uncontrolled observation

study with multiple sites and/or people.

4. NOT Anecdotal reports, case studies, participant observations, experts

reporting their experience, opinions, common sense, or fads.


ISPI: From Evidence-based to “Friendly”?

1960’s: National Society for Programmed Instruction - Air Force Lab influenced by B.F. Skinner’s research - Found a 33% decrease in time and 9% increase in learning - Scientist-practitioner model based on evidence

1980’s: National Society for Performance and Instruction – Diverse mix of researchers and practitioners - Engineering model based on evidence and clinical skill

1990’s - Now:

International Society for Performance Improvement

- Influenced by the need to be more “friendly”

- Clinical model based on experience rather than evidence


What we’ve gained and lost

Gained

•Wider focus on performance

•More experienced practitioners

•Greater variety of interventions

•Advocacy for popular solutions

•International scope and membership

•Less critical “tent” accepting all or

most approaches

Lost

• In-depth study of performance

• Experienced researchers

• Commitment to evidence for impact

• Advocacy for evidence

• Total membership declining

• Challenges to failed or questionable

approaches


The Bottom Line:

• �ISPI is partly evidence-based but has become a market place where nearly all ideas are welcome including far too many that do not work and some that make performance worse.

• ISPI must commit unconditionally to the kind of evidence-based practice that has improved fields such as medicine, psychology, engineering, management and insurance.

• As members we should demand that our professional association support only evidence-based practice and help us implement it.

• We must learn how to operate in teams of HPT experts, researchers and clients to always search out the best evidence and require reliable and valid measurement of the HPT process and results.


2 Nobel Prizes for performance research

Daniel Kahneman 2002 Herb Simon 1978

“Thinking Fast and Slow” “Reason in Human Affairs”


4 facts we’ve learned about ourselves

1. We are overconfident and so assume that what we expect will happen.

• We ignore evidence if we did not expect the results – even if we

collect the evidence.

2. We assume we are rational and that our thinking will solve problems

• We are unaware that non-conscious expectations control nearly all of

our decisions

3. We feel unique and so believe what works for others will not work for us

• We reject evidence if we did not collect it in our own organizations

4. We believe successful organizations know what made them successful

• They often do not so benchmarking often targets the wrong factors


We’ve ignored evidence about:

• Downsizing: Almost always has negative results

• Learning Styles: None of them aid learning or performance

• Feedback: 2/3 makes performance worse or has no impact at all

• SME trainers: Can’t describe 70% of what trainees need to know

• Reactions as measures of learning: 50% wrong so useless

• Financial Incentives: 20% average increase in performance

• Discovery Learning: Does not work for anyone but experts

• Games: Are more expensive and less effective than other options

• Generational Differences: Evidence that they don’t matter

• Social Media: Do not influence learning or motivation

• Memory tests: Do not tell us anything about application knowledge


4 Facts we’ve learned about ourselves

1. We are overconfident and so assume that what we expect will happen





our decisions






Non-conscious decisions and goal choices

• Most of our decisions are made automatically, before we consciously make a choice, but we believe we made a deliberate, conscious choice..

• Non-conscious environmental cues influence our values, self confidence, mood, persistence, mental effort, actions and decisions.

• When we are try to process too much information too quickly, we overload our minds, consciousness “crashes” and we “daydream”.

• Non-conscious decisions, thinking and behavior are pleasurable because they stimulate the same brain structures as drug addiction.

• Motivated intentions to avoid mistakes seem often to cause mistakes when working memory is overloaded by stress or anxiety.

• When we make a prediction and what we predicted does not happen we most often later remember predicting what happened.


4. Facts we’ve learned about ourselves






our decisions



4. We assume successful organizations know what made them successful



Google’s 8 Habits of Good Managers

In order of importance: 1.Be a good coach.

2.Empower your team and don’t micromanage.

3.Express interest in team success and individual well being

4.Be productive and focus on results.

5.Be a good communicator and a great listener.

6.Help employees develop their careers.

7.Have a clear vision and strategy for the team.

8.Have key technical skills so you can help your team.

Google’s earlier

approach

was reversed


4. Facts we’ve learned about ourselves






our decisions




• They often do not, so benchmarking often targets the wrong factors


Evidence about what does work

• Not learning styles but the APA Big Five and Prior Experience

• Not error focused feedback but strategy focused feedback

• Not SME trainers but Cognitive Task Analysis and non SME trainers

• Not Reactions for learning but hands on application during and after

• Not Recognition but well designed Financial Incentives for work

• Not Discovery learning but Fully Guided Training for all but experts

• Not Games but Simulations that assist during applied practice

• Not Generational Differences but APA Big Five and Prior Experience

• Not Memory Tests but Hands on Application during and after


New Instructional Research Foundations

Principle Description

Task-

centeredness

Authentic tasks that represent the

domain/learning outcomes

Activation

Connect to learner’s prior

experience/knowledge/larger

knowledge structure

Demonstration Demonstrate and give examples of

correct performance

Application Part-task and whole-task practice

with corrective feedback

Integration

Reflection, discussion, public

performance, exploration of real

life uses

Merrill, M. D. “First Principles of Instruction,” In C. M.

Reigeluth & A. Carr (Eds.), Instructional Design Theories and

Models III (Vol. III), 2009

First Principles of Instruction Pittsburgh Science of Learning Center

Knowledge-Learning-Instruction Framework

Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center)


Clark and Mayer’s principles for E-learning design

Principle Description

Multimedia Use words and graphics rather than words alone

Contiguity Place printed words near corresponding graphics; Synchronize spoken words with corresponding graphics

Modality Present words as audio narration rather than on-screen text

Redundancy Explain visuals with words in audio OR text, not both

Coherence Avoid interesting but unnecessary material; avoid extraneous audio, graphics, words

Personalization Use conversational rather than formal style; Use effective on-screen coaches; Make the author visible

Segmenting Break content into bite-size segments

Pre-training Teach key concepts prior to procedures or processes

Examples Transition from worked examples to problems via fading; Promote self-explanation of worked-out steps;

Supplement worked examples with explanations

Practice Mirror the job; Provide explanatory feedback; Adapt the amount and placement of practice to job performance

requirements; Transition from examples to practice gradually

Collaboration Insufficient evidence for guidelines on social learning

Learner Control/

Navigation

Give experienced learners control; Make important instructional events the default; Consider adaptive control; Give

pacing control

Build Thinking Skills Use job-specific cases; Make thinking processes explicit; Define job-specific problem-solving processes

Source: E-learning and the Science of Instruction, Ruth Clark and Richard Mayer, 3nd ed., 2011


Prior experience of control in similar situations

Value

Self Efficacy

Attributions

Emotions

Starting

Persisting

Mental Effort

Motivation

Outcomes

Specific

Influences

Root

Cause

“What makes me or us

Effective or Successful?”


Evidence about Organizational Management


Evidence-Based Organizations


Social Media ARE Evidence-Based


Evidence-based Professional Associations?

Evidence-Based HPT

Organizations

Non Evidence-Based HPT

Organizations


What is Evidence-Based Practice at ISPI?

As members we should insist on: 1. An informed, respectful and constant dialogue about the

findings from well-designed, clinical studies of performance at the individual, team and organizational level.

2. Conferences, education, information and products that feature only those interventions that have the strongest empirical and application support for attaining desired goals.

1. Providing us, as members, with access to HPT strategies that

allow us to apply the best evidence, our own expertise and the values of our clients to make decisions that matter to them.


Are you willing to support evidence-based practice at ISPI?

PLEASE RAISE YOUR HAND IF YOU ARE WILLING TO SUPPORT EVIDENCE-BASED PERFORMANCE IMPROVEMENT FOR ISPI

THANK YOU!

We can do this if we

work together!

back to our future - eppicinc.files.wordpress.com · © 2012 atlantic training inc. back to our...

Documents