METHODS FOR DEVELOPING ASSESSMENT INSTRUMENTS TO GENERATE USEFUL DATA IN THE PRESENCE OF VAGUE COURSE OBJECTIVES

Patrick B. BarlowTiffany L. SmithEric Heidel, PhDWilliam Metheny, PhD

On the AgendaSETTING THE SCENE• Who? What? Where? When?

WHY?• Assessment in Graduate Medical

Education

FIVE PRACTICAL TIPS• How we addressed the problem

USE OF ASSESSMENT RESULTS• How were these methods useful?

DISCUSSION• Questions?

SETTING THE SCENEWho? What? Where? When?

Office of Medical Education, Research, and Development (OMERAD)

• Job Description• Consultation and education

• What was happening with GME at our institution?• New Office Structure

• PhD students in ESM brought in• Office given the reigns of the clinical research skills curricula

WHY DOES THIS NEED TO BE ADDRESSED?

Assessment in Graduate Medical Education

What is EBM?

Evidence-Based Medicine in GME1,2

Critical Appraisal

Biostatistics

Clinical Epidemiology

What We Know About Resident Knowledge of Clinical Research Skills?

• Error rate in reporting and interpreting statistics in medicine is estimated between 30-90%

3

• Consistent…• Lack of knowledge• Lack of confidence

4,5

An example…

FIVE PRACTICAL TIPSHow we addressed the problem

TIP ONEKnow Your Situation

• Learning environment factors• Statistics and research methods as a topic

• No formal “courses”, nothing is “required”

• No previous learning objectives, syllabus, or assessment structure

• Work environment factors• Hospital obligations

• Attending physician buy-in & priorities

TIP ONEKnow Your Situation

•Population-specific factors• Variable background experience• Low average competence and confidence• Realities of being a physician

•Availability of resources• Limited time• Limited money

TIP TWOClarify Your Purpose

• Ask two questions:• How will the assessment audience benefit from the results?• How will the students benefit from the assessment results?

• In our case• Audience (OMERAD, GSM faculty/administration)• Students (Residents, fellows, physicians, & staff)

TIP THREEUse What You Have

• Gather the Necessary Background Data• Existing content• Faculty interviews• Direct observation• Literature• Clinical/Work experience

• Three benefits• What instructors think the students are learning

• What is being taught• Where the gaps are in the curriculum

TIP FOURFit the Instrument to Your Purpose, Not the Other Way Around

•Again, consider situational factors•Resources for types of assessment instruments•What worked for us

• Background knowledge probe6

TIP FIVEGet Consistent and Critical Feedback

Assessment must be viewed as a never-ending, iterative process• An instrument is

developed or modified

• The instrument is tested

• Testing generates feedback

• Feedback leads to modifications…

“Feedback Loop” of Assessme

nt Practice

Develop/Modify

TestFeedback

TIP FIVEGet Consistent and Critical Feedback

Assessment must be viewed as a never-ending, iterative process• An instrument is

developed or modified

• The instrument is tested

• Testing generates feedback

• Feedback leads to modifications…

• These modifications are tested

“Feedback Loop” of Assessme

nt Practice

Develop/Modify

TestFeedback

USE OF RESULTS

An Integrated Assessment Model for a Dynamic Learning Environment

Improvements to the Course:Learning Objective

Development

“List of Topics”

BEFORE

ConcreteLearning Objectiv

es

AFTER

• Multiple sources of data• Assessment• Experiential• Evaluation

• Data identified• Salient topics• Missing

content• Student needs

• Need for a responsive curriculum

“after this module

participants should be able to…”

1. Start with course learning objectives

2. Identify test “topics” from learning objectives

3. Expand each topic to as many “concepts” as possible

4. Collapse list of concepts to remove redundancy

5. Create/modify items

Improvements to the Assessment:Test blueprint process used to improve

the assessment instrumentModule 2:Comparing Research Designs

Apply each design to their own

area of research

Identify major

epidemiologic research designs

Next Steps

• Continue instrument & curriculum revisions• Standardized assessment for residents, fellows, physicians on clinical research skills and statistics.

DISCUSSIONQuestions? Comments?

References1. Green, M. L. (2000). Evidence-based medicine training in graduate medical

education: past, present and future. Journal of evaluation in clinical practice, 6(2), 121–38. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/10970006

2. Stewart, M.G. (2001). ACGME Core Compentencies Accreditation Council for Graduate Medical Education. Retrieved from http://www.acgme.org/acWebsite/RRC_280/280_coreComp.asp

3. Novack, L., Jotkowitz, A., Knyazer, B., & Novack, V. (2006). Evidence-based medicine: assessment of knowledge of basic epidemiological and research methods among medical doctors. Postgraduate Medical Journal, 82(974), 817–822. Retrieved from http://pmj.bmj.com/content/82/974/817.abstract

4. West, C. P., & Ficalora, R. D. (2007). Clinician Attitudes Toward Biostatistics. Mayo Clinic Proceedings, 82(8), 939–943. Retrieved from http://www.mayoclinicproceedings.com/content/82/8/939.abstract

5. Windish, D. M., Huot, S. J., & Green, M. L. (2007). Medicine Residents’ Understanding of the Biostatistics and Results in the Medical Literature. JAMA: The Journal of the American Medical Association, 298(9), 1010–1022. Retrieved from http://jama.ama-assn.org/content/298/9/1010.abstract

6. Angelo, T. & Cross, K.P. (1993). Classroom Assessment Techniques. San Francisco: Jossey-Bass.

7. Fink, L.D. (2003). Creating Significant Learning Experiences: An integrated approach to designing college courses. San Francisco, CA: John Wiley & Sons, Inc.