Joshua D. Southwick, MRC, CRCDavid Vandergoot, PhD

Why Continuing Education? Intended Outcomes of Continuing Education Actual Results of Continuing Education Evaluating Training

◦ Why Evaluate?◦ How to Evaluate?

Approaches to Evaluation Guiding Principles

Recommendations Examples Practice & Share (if time)

2

3

We Need Qualified Rehabilitation Counselors◦ The Rehabilitation Act of 1973, as amended,

requires “qualified vocational rehabilitation counselors” to provide services

◦ Ethically relevant: CRCs “practice only within the boundaries of their

competence” (CRCC, p. 11) CRCs “recognize the need for continuing

education . . . to maintain competence in the skills they use” (p. 11)

4

5

Continuing Professional Development (CPD) involves “the continuous acquisition of new knowledge, skills, and attitudes to enable competent practice.”1

◦ Well-trained employees may feel less frustration, more job satisfaction and more job commitment2

◦ After pre-service, graduates should expect to learn new skills or hone existing skills3

Gaining specialty-specific expertise Understanding ever-changing challenges arising within the

field Becoming familiar with promising and evidence-based

practices emerging from new empirical research

6

1. Peck, McCall, McLaren, & Rotem, 2000, p. 4322. Allen & van der Velden, 20013. Leahy et al., 2009

Certification or Licensure Maintenance Team Building Networking Increase Organizational Effectiveness and

Efficiency◦ Increased capacity to serve individuals

Better services for persons with disabilities◦ Greater consumer satisfaction

7

Training has often been less effective than expected:◦ Managerial Training – people are learning but not

applying1

◦ Medical Professionals2, 3

Training has been effective:◦ medium to large effect sizes for training outcome

criteria related to learning (e.g., knowledge; d=0.63), behavior (e.g., job-related behavior changes; d=0.62), and results (e.g., productivity; d=0.62)4

1. Powell & Yalcin (2010)2. Davis, O’Brien, Freemantle, Wolf, Mazmanian, & Taylor-Vaisey (1999)3. Green & Seifert (2005)4. Arthur, Bennett, Edens, & Bell (2003)

8

New Zealand study, Flett, Biggs, & Alpass (1994)◦ Finding: professional training decreased

occupational stress, thereby increasing the rehabilitation practitioner’s ability to work effectively

Christensen, Boisse, Sanchez, & Friedmann (2004)◦ Finding: a one-day training workshop impacted VR

counselors’ knowledge and reported practice in substance abuse screening (*never to rarely)

9

Despite the intended outcomes of continuing education, the return on this training investment remains, to a great extent, unmeasured and unknown.

Most training programs are evaluated only for the participants reactions (i.e., satisfaction; Alliger & Janak, 1989; Van Buren & Erskine, 2002)

10

11

It is important to evaluate continuing education in order to validate and improve such training efforts

When budgets are tight, it may be necessary to justify training expenses

12

Comic showing 2 men at a chalkboard. The chalkboard has a complicated formula on it. In step 2 of the formula are the words "then a miracle occurs." One man says: "I think you should be more explicit here in step two."

13

Continuing Education Training

Better Outcomes for Persons with Disabilities

Kirkpatrick’s Four Levels Logic Model

14

The Four Levels◦ Level 1: Reaction – To what degree participants

react favorably to the learning event.◦ Level 2: Learning – To what degree participants

acquire the intended knowledge, skills, and attitudes based on their participation in the learning event.

◦ Level 3: Behavior – To what degree participants apply what they learned during training when they are back on the job.

◦ Level 4: Results - To what degree targeted outcomes occur, as a result of the learning event(s) and subsequent reinforcement.

15

The Four LevelsLevel 1:Reaction

To what degree participants react favorably to the learning event.

Level 2:Learning

To what degree participants acquire the intended knowledge, skills, and attitudes based on their participation in the learning event.

Level 3:Behavior

To what degree participants apply what they learned during training when they are back on the job.

Level 4:Results

To what degree targeted outcomes occur, as a result of the learning event(s) and subsequent reinforcement.

*From Kirkpatrick & Kirkpatrick (2010)

Reaction – 78% Learning – 32% Behavior – 19% Results – 7% (Reported across multiple

disciplines)

16

*Reported across multiple disciplines

Morin, L., & Renaud, S. (2004). Participation in corporate university training: Its effect on individual job performance. Canadian Journal of Administrative Sciences, 21(4), 295-306.

A logic model shows the rationale or program theory for how program planners believe that the resources and activities invested in a program will produce the expected outcomes.

Used to:◦ Visually display the components of a program◦ Identify measures that will be useful in evaluating

the program outcomes

17

InputsResources (human, financial, organizational, community)ActivitiesImplementation; how resources are used (projects, events,

actions)OutputsParticipationDirect products (deliverables)OutcomesImpact (expected changes or benefits)Short-term = learningMedium-term = action or behaviorLong-term = conditionsAdapted from University of Wisconsin-Extension-Cooperative

Extension, 2003; W. K. Kellogg Foundation, 2004

18

*Adapted from University of Wisconsin-Extension-Cooperative Extension, 2003; W. K. Kellogg Foundation, 2004

InputsActivitiesOutputsOutcomes

19

Activities

Inputs

Outcomes

Outputs

Inputsprogram plannersinstructor preparationtraining materialsmoneyfacilitiestechnologyActivitiesUsing resources to implement a:continuing education programconferenceWorkshopwebinarOutputsverification of attendance at the trainingCEU credits earnedsatisfaction scoresOutcomesShort-term: increases in participants’ knowledge, skills, and confidenceMedium-term: participants implement new knowledge and skillsLong-term: increases in agency-level performance, efficiency, and consumer satisfaction

20

Which criteria should be measured in order to most accurately assess the outcomes of continuing education?

What types of measures can act as indicators that professionals are developing?

21

Knowledge Translation (KT) Organization Development (OD)

22

“A move beyond the simple dissemination of knowledge into actual use of knowledge”1

Barriers to KT / research utilization2:◦ Environmental & organizational factors (culture,

leadership)◦ Individual factors (age, years of service)◦ Difficulty accessing research (database access,

time)◦ Difficulty determining the relevance of research

1. Straus, S. E., Tetroe, J., & Graham, I. (2009). Defining knowledge translation. Canadian Medical Association Journal, 181(3-4), 165-168.

2. Johnson, K., Brown, P., Harniss, M., & Schomer, K. (2010). Knowledge translation in rehabilitation counseling. Rehabilitation Education, 24(3-4), 239-250.

23

24

Organizational Development: A method for designing, implementing, and reinforcing intentional organizational changes1

Key characteristic of OD:◦ The action taken is deliberately and consciously

designed to bring about change over a specified time period, and there must be some way to demonstrate and/or measure the degree to which the change occurred2

1. Cummings & Worley, 2009

2. Worley & Feyerherm, 2003

25

Notes:1. Kirkpatrick’s levels2. Knowledge Translation principles3. Organization Development principles 26

KT2

Requirements for pre-approval of CRC continuing education credits:◦ >60 minutes◦ Focus is to increase knowledge of or skills in

rehabilitation counseling◦ Clearly defined learning objectives or expected

outcomes◦ Participants complete an evaluation of the program’s

value (not an evaluation of learning)◦ Accessible, barrier free location◦ For CE through written means, multiple choice

questions are required.CRCC (2011)

27

VR agency does an in-service training1. They identify learners needs/wants; Involve learners

in training planning process (Adult learning theory; OD)

2. Set objectives for learning (Use these in the evaluation questions & indicators)

3. Hold training4. Evaluate:

Reaction (satisfaction survey) Learning (pre-post quiz) Behavior (2-3 month follow up survey on Objectives)

(Knowledge Translation) Results (3-12 month follow up on agency indicators

specifically related to Objectives) (Organization Development)

28

29

Timing◦ 2-12 months post training? Or the amount of time

estimated for participants to implement new skills.

Measurements not too distal from training objectives (if too distal, you won’t see the impact)

May also want to assess organizational culture (did it support or hinder implementation)

30

Include planned assessments in the course outline

Build assessments so that they are fully integrated into the course

Ensure participants that their satisfaction scores will remain anonymous (i.e., they will not be tracked by IP address)

Provide immediate feedback when feasible

31

32

Online training provided to 42 counselors Trained to implement a case management

model in 10 sites Trainees were administered a knowledge

check as a post-training assessment Ongoing training provided using Case

Reviews Performance evaluated using benchmarks

of key model indicators aggregated by site

Conduct training and evaluate the extent of content learned using a knowledge check

Assess interim performance by conducting case reviews using a protocol reflective of model processes and providing one-on-one instruction as needed

Evaluate relationship of performance on case review protocol with model performance indicators (interim assessment reported here)

Eventually relate performance on model indicators with employment outcomes (will not be available for several years)

Provide online trainingAssess KnowledgeConduct quarterly case reviewsProvide TA as neededMonitor site performance monthlyAnalyze individual and site data(back to) Conduct quarterly case reviews

35

Provide online

training

Assess Knowledge

Conduct quarterly

case reviews

Provide TA as needed

Monitor site performance

monthly

Analyze individual and

site data

Did those who completed all the online courses do better than those who did not complete the courses on their first Case Review?

Score on First Case Review 8 did not complete all the courses 34 did complete them Means:

◦ Completers averaged 76% ◦ Non Completers averaged 86%◦ This is not a significant difference

Implication – more assistance needed

Did those who completed all the online courses do better than those who did not complete the courses as averaged over all their Case Reviews?

The average score of all case reviews Means:

◦ Completers averaged 80%◦ Non Completers averaged 82%◦ This is not a significant difference

Providing training in and of itself may not be sufficient to achieve desired performance

What is the relationship between the average course grade and Case Review scores?

First Case Review score: Correlation = .09 Averaged Case Review scores: Correlation

= .12 These are both significant at the .01 level Implication – Although these results are in the

desired direction, they are weak and reinforce the need for ongoing technical assistance

What is the degree of improvement in Case Review Ratings over time?

Mean Case Review Ratings: Quarter 1 – 77.9Quarter 2 – 70.7Quarter 3 – 81.9Quarter 4 – 85.3Quarter 5 – 89.0

Implication – providing ongoing technical assistance indicates improved conformance to model expectations

Quarter Mean Case Review Rating

1 77.9

2 70.7

3 81.9

4 85.3

5 89.0

This analysis was based on aggregated data by site

How do Case Review scores, as measured by aggregating most recent two reviews, relate to the most recent performance indicators?

Correlation with: ◦ Key indicators most reflective of course content

= .59◦ Total benchmark score = .65◦ Ranking of site performance = .71

Training with follow up technical assistance leads to desired performance. Simply providing training may not lead to success.

Looking at data using the individual as unit of analysis and also using aggregated site data as unit of analysis leads to enhanced understanding of the impact of training and technical assistance

44

Professional Conference 2 follow-up surveys were sent

◦ 1st survey immediately following the conference◦ 3 month follow-up survey after the conference

Sought to identify factors that facilitated or hindered Knowledge Translation

Sought to identify the types of collaboration that resulted from networking at the conference

45

N = 98 Reported KT = 76 Facilitators of KT

◦ Personal Interest = 72% (55)◦ Opportunity in current situation = 61% (46)◦ Belief that applying the knowledge/skills will make a

positive difference = 45% (34)◦ High Self-efficacy = 39% (30)◦ Supportive policies and/or superiors = 34% (26)◦ Peer Interest = 33% (25)

New collaborations as a result of the conference:◦ About 500 new collaborative projects (not unique)

reported among 76 responders

46

Reported no KT = 22 Barriers to KT

◦ Lack of personal interest = 0◦ Lack of peer interest = 1◦ *Lack of opportunity = 8◦ Lack of supportive policies and/or superiors = 0◦ Low Self-efficacy = 0◦ Belief that applying the knowledge/skills will not

make a difference = 1◦ *Lack of Time = 6

47

48

49

Validation of the effectiveness of continuing education

Timely feedback to continuing education providers

Evaluate at levels beyond just Reaction: Learning, Behavior, Results

The very act of evaluating (and planning to evaluate) can positively impact:◦ Organization of the continuing education activity◦ Engagement of the participants during the continuing

education activity◦ Transfer of knowledge and skills to workplace

behaviors◦ Results for the entire organization

50

Research:◦ A potential framework for gathering data on the

effectiveness of continuing education programs (learning, behavior, results, impact).

Practice:◦ A tool in planning, evaluating, validating, and

improving ongoing counselor training efforts.◦ Better ensure that continuing education increases

the competence of practitioners and thereby improves services and outcomes for service recipients.

51

52