Polling Question

How many people are participating in this webinar at your location today?

Just me! 72 83 94 105 More than 106

Polling Question

What member section do you belong to?

Health Care Education AssociationFederal/Government Health Care Educator

Hospital/Health SystemMedical Education and Communication Company

Medical SchoolMedical Specialty Society

State Medical SocietyPharmaceutical

Other

Education Outcomes Assessment for

Regularly Scheduled Series (RSS)

Tuesday, May 18, 20102:00 – 3:00 PM ET

Presenter

Jason Olivieri, MPHMedical Education Outcome Analyst

Nemours Foundation

I have no commercial relationships to disclose

CME OutcomesExpanded CME Framework Description Source of Data

Satisfaction (Level 2) The degree to which the expectations of the participants about the setting and delivery of the CME activity were met

Questionnaires completed by attendees after a CME activity

Declarative knowledge (Level 3A) Degree to which participants state what the CME activity intended them to know

Objective : Pre- and posttests of knowledgeSubjective : Self-report of knowledge gain

Procedural knowledge (Level 3B) Degree to which participants state how to do what the CME activity intended them to know how to do

Objective : Pre- and posttests of knowledgeSubjective : Self-report of knowledge gain

Competence (Level 4) Degree to which participants showin an educational setting how to do what the CME activity intended them to be able to do

Objective : Observation in educational settingSubjective : Self-report of competence; intention to change

Performance (Level 5) Degree to which participants do what the CME activity intended them to be able to do in their practices

Objective : Observation of performance in patient care setting; patient charts; administrative databasesSubjective : Self-report of performance

Moore DE, et al. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof 2009;29:1-15.

Interpreting a level 4+ CME outcome

• Physician participants in this RSS are

200% more likely to treat their diabetic patients in accordance with the latest

clinical guidelines

• Why? What was it about the activity that led to its success? How did the participants perceive it?

RSS

• Occur as frequently as once-a-week

• Types:

– Single discipline (e.g., Anesthesia Case

Conference)

– Multidisciplinary (e.g., Grand Rounds)

• You don’t have to evaluate every session

Webinar pre survey: How many RSS do you oversee? (n = 52)

0

20

40

60

80

100

120

140

160

180

# of RSS

Median = 20, range = 0 – 167!

RSS evaluation goals

• Assess the activity for satisfying criteria associated with “good” CME

• Assess participants for changes in competency, performance or patient health (ACCME, Criteria 11)

• Identify practice gaps (ACCME, Criteria 2)

• Increase response rate

Webinar pre survey: How do you

currently evaluate your RSS? (n = 52)

• ARS• Chart review

• Commitment to change evaluation• Designated evaluators• Focus groups

• Paper evaluations (post, quarter, annual)• Pre-post surveys• QI data

• Web-based survey

Where to start?

• 0% of 10 CME evaluations between 2000-2006 using self-developed instruments provided reliability or validity information

• J Contin Educ Health Prof 2007;27:16-27

Established CME assessment tools

1. Rothman AI, Sibbald G. Evaluating Medical Grand Rounds. J Contin Educ Health Prof 2002;22:77-83.

2. Wood TJ, et al. The development of a participant questionnaire to assess continuing medical education presentations. Med Educ2005;39:568-72.

3. Shewchuk RM, et al. A Standardized Approach to Assessing Physician Expectations and Perceptions of Continuing Medical Education. J Contin Educ Health Prof 2007;27:173-82.

CME assessment tool #1: Rothman & Sibbald*

• Rate presenters on ten items reflecting “best educational practice”

• 5-point scale

*J Contin Educ Health Prof 2002;22:77-83.

1. Provided objectives

2. Demonstrated thorough knowledge

3. Clear and organized

4. Stimulated enthusiasm

5. Appropriate level and detail

6. Effective visuals

7. Effective presentation style

8. 25% session interactive

9. Good rapport with audience

10. Invited questions and participation

CME assessment tool #2: Wood et al*

• Rate quality of CME presentation in three domains (nine items total)

• 7-point scale

*Med Educ 2005;39:568-72.

Domain 1: Presenter

1. Enthusiasm

2. Interaction with audience

3. Apparent knowledge of topic

Domain 2: Presentation

4. Information was presented in an organized manner

5. Used case-based methods; related information presented to practical problems

6. Quality of audiovisual

Domain 3: Content

7. Volume and complexity of the information was appropriate

8. Related content to current evidence in literature

9. Content was relevant to practice

CME assessment tool #3: Shewchuk et al*

• Assess participant perceptions of CME quality on ten items

• 7-point scale

*J Contin Educ Health Prof 2007;27:173-82.

1. Addressed my most pressing concerns

2. Addressed competencies identified by my specialty

3. Provided fair and balanced content

4. Provided clear evidence to support content

5. Included opportunities to learn interactively

6. Provided me with supporting materials or tools

7. Included opportunities to solve cases relevant to my work

8. Translated evidence to practice

9. Addressed barriers to optimal performance

10. Allowed me to assess what I have learned

CME OutcomesExpanded CME Framework Description Source of Data

Satisfaction (Level 2) The degree to which the expectations of the participants about the setting and delivery of the CME activity were met

Questionnaires completed by attendees after a CME activity

Declarative knowledge (Level 3A) Degree to which participants state what the CME activity intended them to know

Objective : Pre- and posttests of knowledgeSubjective : Self-report of knowledge gain

Procedural knowledge (Level 3B) Degree to which participants state how to do what the CME activity intended them to know how to do

Objective : Pre- and posttests of knowledgeSubjective : Self-report of knowledge gain

Competence (Level 4) Degree to which participants showin an educational setting how to do what the CME activity intended them to be able to do

Objective : Observation in educational settingSubjective : Self-report of competence; intention to change

Performance (Level 5) Degree to which participants do what the CME activity intended them to be able to do in their practices

Objective : Observation of performance in patient care setting; patient charts; administrative databasesSubjective : Self-report of performance

Moore DE, et al. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof 2009;29:1-15.

Competency outcome (criteria 11)

Do you intend to make any changes in your practice Do you intend to make any changes in your practice Do you intend to make any changes in your practice Do you intend to make any changes in your practice as a result of participating in this CME activity?as a result of participating in this CME activity?as a result of participating in this CME activity?as a result of participating in this CME activity?

□ If yes, please provide at least one example _____________

□ If no, please specify why not ________________________

Practice gaps (criteria 2)

1.1.1.1. What questions in practice are you having that you are not What questions in practice are you having that you are not What questions in practice are you having that you are not What questions in practice are you having that you are not getting answers to?getting answers to?getting answers to?getting answers to?

2.2.2.2. What patient problems or patient challenges do you feel What patient problems or patient challenges do you feel What patient problems or patient challenges do you feel What patient problems or patient challenges do you feel that youthat youthat youthat you’’’’re not able to address appropriately or to your re not able to address appropriately or to your re not able to address appropriately or to your re not able to address appropriately or to your satisfaction?satisfaction?satisfaction?satisfaction?

3.3.3.3. What patient problems are your patients saying back to What patient problems are your patients saying back to What patient problems are your patients saying back to What patient problems are your patients saying back to you needs more attention or more followyou needs more attention or more followyou needs more attention or more followyou needs more attention or more follow----up?up?up?up?

http://education.accme.org/video/accme-video-faq/traditional-needs-assessment

RSS evaluation goals

�Provide feedback to faculty

�Assess changes in competency

� Identify practice gaps

? Increase response rate

Final assessment tools

• Hypothesis: Less evaluation = more response rate

• Quarterly, web-based evaluation for “single discipline”RSS– Evaluates perceptions of CME quality (Shewchuk et al tool)

– Includes intent to change and needs assessment questions– SurveyMonkey

• Post-activity paper-based evaluation for “multidisciplinary” RSS– Evaluates quality of CME presentation (Wood et al tool)– Includes intent to change and needs assessment questions

Webinar pre-survey question: Have you ever used a web-based survey to evaluate CME outcomes (n = 52)

Yes, 62%

No, 38%

Table 1. Participation in case-based RSS between Apr-Jun 2009 (Q2)

RSS title Unique participants

Total sessions

Attendance average

Valid emails

Previous SurveyMonkey

opt-outs

Evaluation response rate (after 1 invite)

Cardiac Teaching Conference

41 12 20 39 1 56% (22/39)

Tumor Board –AIDHC

35 13 16 34 0 65% (22/34)

Oncology Teaching Conference

18 10 7 17 1 76% (1317)

Uroradiology Case Conference

12 9 6 10 1 80% (8/10)

Evidence-based Lecture Series

9 5 4 6 0 50% (3/6)*

Surgery M&M Case Conference

10 9 7 9 0 44% (4/9)

Patient Safety Case Conference

15 3 8 14 1 57% (8/14)**

Tumor Board- JAX 45 8 19 39 1 59% (23/39)

Surgery/Radiology Series - JAX

16 13 10 14 1 71% (10/14)

avg = 61%

*response rate after 3 survey invitations

**response rate after 2 survey invitations

Webinar pre survey: Average paper survey response rate? (n = 41)

0

10

20

30

40

50

60

70

80

90

100

# of RSS

per

cen

t

Median = 55%, range = 2 – 100!

Table 2. Response rate for quarterly, web-based RSS evaluation (2009)

RSS title Q3 Q4

Cardiac Teaching Conference 57% (27/47)* 43% (15/35)

Tumor Board – AIDHC 65% (28/43)* 52% (15/29)

Oncology Teaching Conference 73% (19/26)* 61% (11/18)*

Uroradiology Case Conference 50% (8/16) 73% (8/11)*

Evidence-based Lecture Series 80% (8/10)* 50% (2/4)*

Surgery M&M Case Conference 60% (6/10)* 50% (4/8)

Patient Safety Case Conference 40% (6/15) 43% (3/7)

Tumor Board- JAX 67% (35/52)* 59% (22/37)

Surgery/Radiology Series - JAX 82% (14/17)* 64% (9/14)*

avg = 64% avg = 55%

*response rate after 2 survey invitations

Survey example (web)

Web-based RSS email invite

Subject:Subject:Subject:Subject: CME/CE evaluation for Nemours Tumor Board

Body:Body:Body:Body:Hi [CustomValue] [LastName]:

Our records indicate that you attended one or more sessions of Nemours "Tumor Board" between July andSeptember 2009. Tumor Board typically occurs every Monday at 12:30 PM. A total of 10 sessions were heldbetween July and September 2009.

In an effort to reduce the time you spend completing course evaluations (usually one at each session), thesurvey link below will direct you to a single evaluation covering this entire period.

Here is a link to the survey: https://www.surveymonkey.com/s.aspx

In order to meet the requirements of the Accreditation Council for Continuing Medical Education to provide CME credit for this activity, we must measure the educational outcomes. Please respond to this survey (it takes < 5 minutes) so that we can continue providing credit for this series.

If you have any questions about this evaluation, please contact me via phone (904) 697-3877 or email(jolivier@nemours.org).

Thank you for your participation!

Jason Olivieri Nemours Office of CME

Please note: If you do not wish to receive further emails from us, please click the link below, and you will beautomatically removed from our mailing list. https://www.surveymonkey.com/optout.aspx

Q&A

We welcome your questions and comments.

Please use the chat box on the left side of your screen to submit a question.

Thank you!

Please take a moment to complete the brief evaluation.

We appreciate your feedback!

Thank You!

Join us for the Alliance’s next Takeout Tuesday webinar!

Taking Education to the Healthcare Team: In-situ Simulation in Acute MI Care as a Model for Team-

focused CME

Tuesday, June 15, 20102:00 – 3:00 PM ET

www.acme-assn.org