taking the guesswork out of writing better multiple choice questions presented to amatyc february 2,...
TRANSCRIPT
Taking the Guesswork out of Writing Better Multiple Choice Questions
Presented to AMATYC February 2, 2015 by Patricia L. Gregg, PhD (Patti) Georgia Perimeter College
Multiple choice tests have their place …
… but it isn’t necessarily “everywhere”
Question “stem”
Should give a clear indication of what the question is about
Don’t be too broad or generic(e.g. Which of the following is true?)
Phrase in form of a question Don’t use incomplete statements, fill in the blank, etc.
Be consistent with grammar and syntax
Don’t give clues about right answer based on the wording of the stem
Use simple and positive constructions(e.g. Name the most common cause of …)
Avoid convoluted and negative constructions(e.g. Name the least uncommon cause of …)
Use scenarios when possible but Don’t include a lot of distracting or irrelevant information (red herrings)
Answer options
Categories should be mutually exclusive
Avoid overlapping categories
Have one clear right (or best) answer Avoid “all of the above” “none of the above” “both a and c” etc.
Be consistent with grammar and syntax
Don’t give clues about right answer based on the wording of the options
Should be same length and construction
Don’t make the right answer longer or shorter than other options, or phrased differently
Should be equally plausible No obvious wrong answers
Aren‘t plausible distractors “trick questions”?
Plausible distractors help to
• minimize lucky guesses
• ensure that you are truly measuring students’ learning, not their ability to recognize blatantly irrelevant information
• diagnose key concepts that students may be misunderstanding or confusing with one another
Poll #1
ITEM ANALYSIS can help you understand
• which of your incorrect answer options are plausible “distractors”
• whether your test question effectively “discriminates” between high-performing and low-performing students
Can we test higher order thinking with multiple choice questions?
Yes, but we need to be vigilant!
Too often we claim to be assessing application, analysis, interpretation, etc. but we are really assessing recognition, computation or calculation.
• Not all word problems test higher order skills!
• Examples of question items that could measure higher order skills would require students to:
Interpret data or graphs
Recognize spurious conclusions
Distinguish between relevant and irrelevant information
_____triangles have three sides of equal length.
a. Isosceles
b. Equilateral
c. Right
d. Scalene
Leading with the blank is an awkward construction, especially for non-native English speakers and others with reading challenges.
Which type of triangle has all three sides of equal length?
a. Isosceles
b. Equilateral
c. Right
d. ScaleneAdapted from Cheung & Bucat, How Can We Construct Good Multiple Choice Items?
This question tests the same knowledge as the previous example, but the construction is more familiar.
The mean of a distribution of test scores is the:
a. Most frequently occurring score
b. 50th percentile
c. Arithmetic average
d. Measure of score range
A university developed an aptitude test to use for admission into its Honors Program. The test was administered to a group of seven applicants who obtained the following scores: 70, 72, 72, 80, 89, 94, 98. The mean score on the aptitude test is:
a. 72 c. 82
b. 80 d. 90from Zimmaro, Writing Good Multiple-Choice Exams
In order to correctly answer this question, the student must still know the definition of “mean” but also must be able to calculate the mean. However, the question does not assess higher order skills such as application or interpretation.
We want students to see the relevance of course content, but too many details in a word problem end up testing reading speed and comprehension, instead of math skills!
Suppose you are a mathematics professor who wants to determine whether or not your teaching of a unit on probability has had a significant effect on your students. You decide to analyze their scores from a test they took before the instruction and their scores from another exam taken after the instruction. Which of the following t-tests is appropriate to use in this situation?
a. Dependent samples.
b. Heterogenous samples.
c. Homogenous samples.
d. Independent samples.
When analyzing your students’ pretest and posttest scores to determine if your teaching has had a significant effect, an appropriate statistic to use is the t-test for:
a. Dependent samples.
b. Heterogenous samples.
c. Homogenous samples.
d. Independent samples.
Burton et. al., How to Prepare Better Multiple Choice Items: Guidelines for University Faculty
cited in Duquesne University, Good, Better, Best: Multiple Choice Exam Construction
Test items with carefully constructed answer options can help you diagnose what concepts your students may be confusing and/or misunderstanding.
The number of bacteria N(t) present in a culture at time t hours is given byN (t) = 2200(2)t. Find the number of bacteria present when t = 4 hours.
(a) 35200(b) 17600(c) 70400(d) 19360000
Test item courtesy of a GPC faculty member
The correct answer is (a).
A student would select (b) if they multiplied 2 x 4 instead of raising 2 to the 4th power.
A student would get (c) if they multiplied 2200 by 32, thinking that (2)4 is 32 rather than 16.
The student who selects (d) demonstrates no understanding of the order of operation. In this case, the student multiplies 2200 by 2 and then raises it to 4. Since that is not one of the choices, 19360000 (4400 raised to 2) is the closest.
http://www.act.org/compass/sample/pdf/numerical.pdf
How could this item be stronger?
Do we really need 5 answer options?
The COMPASS publishers probably do this to minimize the probability of guessing the correct answer, BUT it is really hard to come up with 5 plausible alternatives, so if one is blatantly wrong, we may as well just have 4 options.
Do all the answer options contribute to diagnosis?
It seems unlikely that a student would select 0% or 45% unless they had almost no grasp of simple arithmetic, even if they didn’t understand how to calculate percentages.
Does the number of math courses taken contribute to the strength of the question?
It may be an unnecessary complication for weaker readers.
The skill being testing by this item is calculation of percentages.
We want the student to demonstrate their understanding that the second group is 60% of 25% not 60% of the total class; therefore the last remaining group is 40% of 25%.
One possible re-write:This year, 75% of the graduating class of Harriet Tubman High School had been accepted to college. Of the remaining class members, 60% had applied to college but had not been accepted. What percent of the graduating class did not apply to college?
The calculation is exactly the same, but the red herrings are removed.
http://www.act.org/compass/sample/pdf/numerical.pdf
Poll #2
How could this item be stronger?Ensure that all answer options contribute to diagnosis.
Eliminate extraneous information e.g. exercise 3 times per week for 20 minutes.
IF this test item were intended to measure higher order skills, we would keep the exercise information and possibly add some additional “what if” scenarios with different variables.
PAPERSBurton, Sudweeks, Merrill & Wood (1991, 33 pp.) https://testing.byu.edu/handbooks/betteritems.pdf
Cheung & Bucat (2002, 10 pp.) http://www3.fed.cuhk.edu.hk/chemistry/files/constructMC.pdf
Zimmaro (2010, 41 pp.) https://ctl.utexas.edu/sites/default/files/documents/Writing-Good-Multiple-Choice-Exams-04-28-10.pdf
INSTITUTIONAL WEBSITES Brigham Young University:https://testing.byu.edu/handbooks/14%20Rules%20for%20Writing%20Multiple-Choice%20Questions.pdf
Duquesne University: http://www.duq.edu/about/centers-and-institutes/center-for-teaching-excellence/teaching-and-learning/multiple-choice-exam-construction
Indiana University: http://www.iub.edu/~best/pdf_docs/better_tests.pdf
University of Texas: http://ctl.utexas.edu/teaching/assessment/planning/multiple-choice
Vanderbilt University: http://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/
COMMERCIAL PUBLISHERSCenseo Corporation, Guidelines for Writing Effective Tests: A Practical “Short Course” for Test Authors (2007, 9 pp). Document not available online; appears company may no longer be in business; I have a PDF copy to share.
Magna Publications:http://www.facultyfocus.com/articles/educational-assessment/seven-mistakes-avoid-writing-multiple-choice-questions/
Patricia (Patti) GreggAssociate Director, Assessment and Reporting
Georgia Perimeter College
678-891-2571
@DrPattiG
www.linkedin.com/pub/patricia-gregg