You’ve Been Shown: Now It’s Your Turn to Ask!

CLEAR 2004 Kansas City, Missouri

Responding to Your Questions About Testing

Credentialing and Exam Issues: Sessions at CLEAR 2004

• Ten Year Review of CBT

• When Certification Replaces Licensure: Show me Accreditation

• ADA: Let Us Show You What Works

• Show Me What Globalization Means for Credentialing Programs

• Multiple Tracks to Licensure: Show me the Way

• Test Piracy and Security in a Wired World

CLEAR 2004 Kansas City, Missouri

• Show Me the Validity

• Show Me How to Get Past MCQs: New Opportunities in Measurement

• Show me an Evidential Approach to Assessment Design

• The Case of the Repeating Licensure Examination Candidate: Psychometric, Legal, and Policy Issues

CLEAR 2004 Kansas City, Missouri

Credentialing and Exam Issues: Sessions at CLEAR 2004

Panelists:Carol Morrison Assistant Vice President National Board of Medical Examiners

Elizabeth Witt Senior Statistician American Board of Emergency Medicine

Rose McCallin Director of Office of Examination Services

State of Colorado

CLEAR 2004 Kansas City, Missouri

Moderator: Chuck Friedman Assistant Vice President, ACT

Speakers

You . . . the audience!!!

CLEAR 2004 Kansas City, Missouri

ValidityReliability

Item Analysis

Item Review

Item Writing

Passing ScoreScoring

Job Analysis

Test Blueprint

Standard Administration

Valid

ity

Reliability

Item

Anal

ysis

Item

Writin

g

Pass

ing

Scor

e

Scorin

g

Test Blueprint

Standard

Administ

ratio

n

Job Analysis Item

Rev

iew

Should state licensing boards require continuing education?

CLEAR 2004 Kansas City, Missouri

How do I find out if my test has been compromised? If so, what should I do?

CLEAR 2004 Kansas City, Missouri

CLEAR 2004 Kansas City, Missouri

What should I do if I do or not do if I think a candidate or candidates have cheated?

Test Compromise?

• Missing test book– When, who?– Statistical analyses

• Memorization of items– Memorize and reconstruct items– Study guides– Internet postings– Statistical analyses

Is a self assessment exam good enough for continuing competency?

CLEAR 2004 Kansas City, Missouri

Four Components ofContinuous Certification

• Professional StandingProfessional Standing

• Lifelong Learning and Self AssessmentLifelong Learning and Self Assessment

• Assessment of Cognitive ExpertiseAssessment of Cognitive Expertise

• Assessment of Practice PerformanceAssessment of Practice Performance

Considerations in Determining the Value of a Self-Assessment

• How is the exam developed and scored?

• Can you be certain who took the exam?

• How high are the stakes for maintaining the credential?

• Does the self-assessment process foster learning?

• Is it easy to participate as part of one’s normal professional activities?

Is performance testing useful in state licensure?

CLEAR 2004 Kansas City, Missouri

Performance Assessment

• Does performance assessment measure something different?

• Reliability, validity• How do you score it?• Cost, complexity

• Answer: It depends!

CLEAR 2004 Kansas City, Missouri

What is your opinion about open book exams?

What should I do if a candidate receives an incorrect score report?

CLEAR 2004 Kansas City, Missouri

How do you determine if a request for accommodations is appropriate? If a test is administered under special conditions, should the score report be “flagged”?

CLEAR 2004 Kansas City, Missouri

Is Request Appropriate?

• Use expert consultant• Level the playing field, don’t tilt it too far the

other way

To Flag or Not to Flag?

• Many testing companies have stopped flagging• Some claim that flagging may lead to unintended

consequences• Standards for Educational and Psychological

Testing (APA/AERA/NCME)– Flag if do not have evidence of score

comparability

Do we have to equate our tests?

Is there one best way?

ONE BEST WAY?

CLEAR 2004 Kansas City, Missouri

Reasons to Consider Reasons to Consider

NOT EquatingNOT Equating• Small sample size

• Multidimensional exam

• Changing content outline

What If We Can’t Equate?

• Create a detailed blueprint

• Write the best possible items

• Build forms that are closely parallel in both content and statistics

• Conduct frequent standard setting studies

• Include common items across forms; study differences qualitatively

CLEAR 2004 Kansas City, Missouri

An examinee complains that he failed the test by one point/question. What do you tell him?

Other Sources of Information• CLEAR Exam Review• Other CLEAR Publications

– ADA Handbook– Development, Administration, Scoring and

Reporting of Credentialing Exams• CLEAR web site: Frequently Asked

Questions• Joint Standards: AERA, NCME, APA• Professional psychometricians

CLEAR 2004 Kansas City, Missouri