Is a Mock Best Option MCQ a Valid Measurement of Student Performance in Final Clinical Medicine

A. Patterson, M. Shuhaibar, C. O’Morain, M. Hennessy, S.R. McCann University of Dublin, Trinity College, Ireland

AMEE CONFERENCE 2009

BACKGROUND: Validity is one of the core principles on which assessment is based. Validity is the extent to which a test actually measures what it claims to measure1. This study builds on previous work2 examines both predictive and face validity of an examination. Predictive validity examines if the test can predict future performance in a particular area. Face validity is the least scientific measure of validity but it pertains to the students’ perception of the test and if they believe it is a measure of their overall knowledge. If a test has predictive validity it allows students see how they are progressing in the course and highlights if they need to amend their study strategies to perform better.

OBJECTIVES: to compare student performance in a mock best option multiple choice question examination with their performance in a Final Clinical Medicine MCQ examination and their clinical examinations for 2008 and 2009. In addition, the analysis investigated if the mock examination completed by students three months prior to the real examination could predict how students perform in the real examination. Also, to ascertain students’ perceptions of the examination.

DISCUSSION: Students performed better in the real examination versus the mock examination, there was a period of three months between the tests when an intensive course in Clinical Medicine was provided for students. The predictive validity of the mock examination was tested and proved.Students valued the results of the mock examination as a valuable indicator of how they were proceeding in the course and commented that it reduced the anxiety of sitting the real examination.

CONCLUSIONS: The mock examination is a valuable examination that provides students with an opportunity to see how they are progressing in the course. It is under consideration to convert the mock into a real examination. However, as students perform better in the real examination and the tests are examining equivalent knowledge, this study suggest it should be continued as a purely formative examination.

REFERENCES: 1.Brown, G., Bull, J. and Pendlebury, M., 1997. Assessing student learning in higher education. , Routledge, London2.A. Patterson, M. Shuhaibar, Prof. C. O’Morain, The Predictive Validity of a Mock MCQ Examination of Student Performance in a Final Clinical Medicine MCQ examination, INMED Conference, Ireland, 2009

RESULTS: The results for 2008 and 2009 were compared for students who completed both tests using a Pearson correlation coefficient (N= 103, 2008 and N = 116, 2009). There was a strong, positive correlation between both tests, 2008 and 2009 r=0.63, 2009 r = 0.387 respectively. Both correlations were significant. The mock examination results were also compared with students clinical performance and their overall mark. There were significant correlations with all these measurements.A paired t-test was performed to see if there was any significant difference in student performance between the two tests. Students completed an evaluation of the examination.There was no significant difference between student performance on both tests with t<0.001. The means for the mock and the real examinations were 60.6 ± 10.5 and 68.7 ± 12.5, 52.3 ± 8.8 and 67.9 ± 9.5 for 2008 and 2009 respectively.

“It was an insight. Thank

you.” “It has given me

the motivation to study”

 

2008N= 103

Mock MCQ

Real MCQ

Minor cases

Major cases

Final Mark

Mock MCQ

0.626** 0.433** 0.423** 0.655**

Real MCQ

0.626** 0.563** 0.566** 0.790**

Minor cases

0.433** 0.563** 0.576** 0.819**

Major cases

0.423** 0.566** 0.576** 0.803**

Final Mark

0.655** 0.790** 0.819** 0.803**

2009N= 103

Mock MCQ

Real MCQ

Minor cases

Major cases

Final Mark

Mock MCQ

0.387** 0.415** 0.224** 0.526**

Real MCQ

0.387** 0.354** 0.218* 0.587**

Minor cases

0.415** 0.354** 0.298** 0.789**

Major cases

0.224** 0.218* 0.298** 0.639**

Final Mark

0.526** 0.587** 0.789** 0.639**

**Correlation is significant at the 0.01 level* Correlation is significant at the 0.05 level