Oral Assessment in an

Advanced Lab Course

2009 Topical Conference on Advanced Laboratories

University of Michigan, Ann Arbor23-25 July 2009

25 July 2009 David Bailey - Oral Assessment 2

What’s the point?

25 July 2009 David Bailey - Oral Assessment 3

Not about Oral Presentations

i.e. not about student talksAt Toronto students give oral presentations in– 2nd Year Lab– Research & Reading

CoursesNot in Advanced Lab

25 July 2009 David Bailey - Oral Assessment 4

Oral Assessment

• Assess– Lab accomplishments & knowledge– Oral communication skills

• Assist instructor understanding of– notebook– lab work

25 July 2009 David Bailey - Oral Assessment 5

Communications vs Physics

Need to decide if marks will be allocated for communication skills

– ExplicitlyA specific part of marking scheme

– IndirectlyLousy communicators loose marks because examiners can’t figure out what the student knows or accomplished.

25 July 2009 David Bailey - Oral Assessment 6

Toronto Grading Scheme• Simulated Data Analysis 2 %• 3 Experiments 60 %

– 3/10 from interview

• Formal Report 18 %• Final Oral 20 %

Total Oral Component 38%About 1/3 of this mark

is communications skill, i.e. 10%

25 July 2009 David Bailey - Oral Assessment 7

Benefits & Drawbacks

• Positives– efficient and flexible

• Negatives– stressful– perceived to be

more subjectivethan markingnotebooks.

25 July 2009 David Bailey - Oral Assessment 8

Oral Examinations

“a stressful method of student assessment with poor objectivity and reliability and doubtful validity”

– P. T. Jayawickramarajah, “Oral examinations in medical education”, Medical Education 19 (1985) 290-293.

25 July 2009 David Bailey - Oral Assessment 9

Complaints

This past year, for the first time we got complaints from a few students that some professors were unfair and too tough in their interviews.

They wanted the whole mark based on the notebook.

25 July 2009 David Bailey - Oral Assessment 10

Efficient Questioning

Based on Fig. 5 from J. R. Platt, “On Maximizing the Information Obtained from Science Examinations, Written and Oral”, Am. J. Phys. 29 (1961) 111-122.

WeakStudent

Strong Student

Knowledge

Ignorance Efficient D, E

Inefficient A, B, C

25 July 2009 David Bailey - Oral Assessment 11

Assessment Format

• Directed Discussioni.e. We ask questions

• First question is usually variant of “What is the point?”

• At least 1 question each– about hardware– that requires notebook– about physics

• open (note)book

25 July 2009 David Bailey - Oral Assessment 12

Experiment Mark

~ 1/3 each on – student’s work in lab– student notebook– interview

but components are strongly interdependent.

25 July 2009 David Bailey - Oral Assessment 13

Experiment Interview

• 20 minutes long– formerly 1 hour (or more)

• 1 Examiner– Professor supervising experiment

who has already “read” notebook

• Discussion– clarifies understanding of notebook

and what was done in lab– provides immediate feedback

25 July 2009 David Bailey - Oral Assessment 14

Final Oral Exam

• 25 minutes long– 5 minutes between exams

• 3 examiners– 2 Professors and 1 TA

try to have 1 expert and 1 novice

• 2 experiments are discussed– student chooses 1st

• open (note)book

25 July 2009 David Bailey - Oral Assessment 15

Oral Exam MarksAfter student leaves,

– each examiner writes down a mark in secret without any discussion

– take average

May then be a minute of discussion if the marks vary widely– does not change the student’s mark– for examiner education only

25 July 2009 David Bailey - Oral Assessment 16

Bias and AccuracyBias

– Unavoidable– Be self-aware– Use multiple assessors

Accuracy– Notebooks can be revisited, but

not clear marks are more accurate

“After 4 decades, I can now just about tell an A notebook from a B notebook” David H., U. Toronto

25 July 2009 David Bailey - Oral Assessment 17

0%

20%

40%

60%

<50 50 60 70 80 90

Students

CMQMAPL

F 50 60 70 80 90 Grades

Students

40%

20%

0%

Quantum Mech.

Advanced Lab

Classical Mech.

Grade Distributions

Averages

71±0.9

68±0.6

80±0.6

25 July 2009 David Bailey - Oral Assessment 18

Marking Correlations (Correlation Coefficients)

40

60

80

100

40 60 80 100

100

80

60

4040 50 80 100

Oral vs Experiments(0.61±0.03)

X Quantum vs Classical(0.68±0.03)

Oral vs Classical(0.61±0.04)

Oral vs Quantum(0.50±0.04)

25 July 2009 David Bailey - Oral Assessment 19

Marking Correlations (Correlation Coefficients)

40

60

80

100

40 60 80 100

100

80

60

4040 50 80 100

Oral vs Experiments(0.61±0.03)

X Quantum vs Classical(0.68±0.03)

Expt vs Classical(0.64±0.04)

Expt vs Quantum(0.43±0.05)

25 July 2009 David Bailey - Oral Assessment 20

Examiner Variations

Oral examiner averages vary little 77-81

Slightly more variation in spread8-11

But marks are well correlated>0.76

25 July 2009 David Bailey - Oral Assessment 21

Marking Spreads

Expect smaller mark differences for Orals than for Experiments, where both both in marker and student work varies.

Average RMS spread– by different examiners for same

student on same Oral Exam4.1±0.1

– for different experiments by same student

7.2±0.3

25 July 2009 David Bailey - Oral Assessment 22

SummaryOral Interviews and Examinations

– powerful tools for assessing student competence and achievement.

– relatively efficient and flexible

but– stressful– perceived to be more subjective and

inconsistent

Toronto’s experience has been good.