Assessment in EducationAssessment in Education

Patricia O’SullivanOffice of Educational Development

UAMS

Last Month Dr. Allen described a situation about an individual asking her how to get to Freeway Medical Center. How would you determine how successful Dr. Allen was?

ObjectivesObjectives Provide terminology and principles

fundamental to assessment Introduce various assessment

approaches Review tips for a common assessment:

Multiple Choice Questions Review elements for a less common

assessment: Portfolio

What is Assessment?What is Assessment?

Appraising performance through measurement to make an evaluation

For what purpose? Formative Summative

What Assessments are Used at What Assessments are Used at UAMS?UAMS?

What Makes a Good What Makes a Good Summative Assessment?Summative Assessment?

Fair and practical Based on appropriate criteria that

are shared with the learners Valid Reliable

ValidityValidity Key to the interpretability and

relevance of scores Do the scores have a plausible

meaning? What are the implications of

interpreting the scores? What is the utility of the scores? What are the actual social

consequences?

Validity Evidence for Proposed Validity Evidence for Proposed InterpretationInterpretation

Content/Construct Evidence Content matches the objectives &

teaching Score is shown to be meaningful

Criterion/Predictor Evidence Score relates to future outcomes

ReliabilityReliability Assessment data must be

reproducible Written tests

• Internal consistencyCronbach’s alpha or KR-20 (values

range 0-1)• Test-retest

Correlation ranging from 0-1

Reliability (cont.)Reliability (cont.)

Rater-based assessments Interrater consistency or agreement

(generally correlation coefficient or intraclass correlation)

OSCE & performance assessments Generalizability Theory

Improve ValidityImprove Validity

Match assessment with objectives Increase the sample of objectives

and content on the assessment Review blueprint (or produce one) Use test methods appropriate to

objectives Ensure test security

Improve ReliabilityImprove Reliability

Clear questions Appropriate time allotted Simple, clear and unambiguous

instructions High quality scoring Increase number of observations or

questions

Norm or Criterion ReferencedNorm or Criterion Referenced

Norm-referenced (relative standard) Use the results of the assessment to set the

standards (e.g. proportion to receive each grade); performance judged by comparison with others

Criterion-referenced (absolute standard) Learners achieve some minimal standard of

competency

Standard Setting ProceduresStandard Setting Procedures

Judgments based on Holistic impressions of the exam or

item pool Content of individual test items Examinee’s test performance

Anghoff Standard Setting Anghoff Standard Setting ProcedureProcedure

Judges are instructed to think of a group of “minimally acceptable” persons

For each item estimate the proportion of this group that will answer item correctly

Sum up proportions to get the minimum passing score for a single judge

Average across judges

Assessment MethodsAssessment Methods ACGME Toolbox Additional methods

Other written assessments• Essay• Short answer and computation• Patient management problems• Modified essay questions

Create a game Self-assessment

Matching Assessment Method Matching Assessment Method and Objectivesand Objectives

Examine ACGME matrix

Alternative Testing ModesAlternative Testing Modes

Take-home tests Open-book tests Group exams Paired testing

Reducing Test AnxietyReducing Test Anxiety

Make the first exam relatively easier

Give more than one exam Give advice on how to study Encourage study groups Give a diagnostic test early

Reducing Test AnxietyReducing Test Anxiety

Before a test explain format Provide sample items Provide a pool of final items with

syllabus

Technology and AssessmentTechnology and Assessment

Computer based testing Use of images and video clips

Computer Adaptive Testing (CAT)

Take a TestTake a Test

Multiple Choice Questions: Multiple Choice Questions: Writing TipsWriting Tips

http://www.nbme.org/about/itemwriting.asp Simple suggestions:

Design for one best answer Place hand over stem. Can you

guess the answer? Use a template Attend to Graveyard of Test Items

Template ExamplesTemplate Examples A (patient description) is unable to

(functional disability). Which of the following is most likely to have been injured?

Following (procedure), a (patient description) develops (symptoms and signs). Laboratory findings show (findings). Which of the following is the most likely cause?

GraveyardGraveyard B-type: matching heading with

words or phrases (can use heading more than once)

D-type: complex matching—two step process

K-type: stem with four options and choices are A=1,2,3 only B=1,3 only C=2,4 only D=4 only, E=all are correct

PortfoliosPortfolios

Case Records

Logs

Conferences

Evaluations

PortfoliosPortfolios

Definition A purposeful collection of student

work that exhibits to the student (and/or other) the student’s efforts, progress, or achievement in (a) given area(s).

This collection must include student participation in Selection of portfolio content The criteria for selection The criteria for judging merit Evidence of student

reflection

PortfoliosPortfolios

Learning Showing progress Showing effort Showing

remediation

Assessment Showing

competence

Case Records

Presentations

Evaluations

Logs

Criteria__________________

Criteria__________________

Criteria

Reflection

Portfolio

• Why selected

• How reflects skills

and abilities

Using Portfolios the Learner Using Portfolios the Learner Engages in AssessmentEngages in Assessment

Making choices Critically self-examining Explaining what they are doing

Well-structured PortfolioWell-structured Portfolio

Menu Selection criteria Scoring rubric Benchmarks Trained raters

Determine Reliability and ValidityDetermine Reliability and Validity Reliability: Generalizability theory

Found needed 6 entries with 2 raters Validity: number of studies

Resident performance across years Resident performance in areas Resident perception Resident performance when program made

an intervention Correlation of portfolio performance and

other areas

10-Step Approach10-Step Approach

1. Define the purpose

2. Determine competencies to be assessed

3. Select portfolio materials

4. Develop an evaluation system

5. Select and train examiners

10-Step Approach10-Step Approach

6. Plan the examination process

7. Orient learners

8. Develop guidelines for decisions

9. Establish reliability and validity evidence

10. Design evaluation procedures