writing effective questions and tests

Writing Effective Writing Effective Questions and TestsQuestions and Tests

Peter FenrichPeter FenrichInstructional Development ConsultantInstructional Development Consultant

Learning and Teaching Centre (bcit.ca/ltc)Learning and Teaching Centre (bcit.ca/ltc)British Columbia Institute of TechnologyBritish Columbia Institute of Technology

Tel: 604-432-8817Tel: [email protected][email protected]

Bridge-in Have you ever :Have you ever :

- been frustrated or angry after an - been frustrated or angry after an exam?exam?

- wondered what an exam will cover?- wondered what an exam will cover?

- been confused about what a question- been confused about what a question

was asking?was asking?

- thought a test question was too picky?- thought a test question was too picky?

Outcomes

You should be able to:You should be able to:

- list ideas that help to test effectively- list ideas that help to test effectively

- state characteristics of well-written - state characteristics of well-written questionsquestions

- describe how to reduce learner - describe how to reduce learner frustration during testingfrustration during testing

- write effective alternative-answer, - write effective alternative-answer, multiple-choice, short-answer, matching, multiple-choice, short-answer, matching, and long-answer questionsand long-answer questions

Test understandingTest understanding Reinforce learningReinforce learning Apply learning to problemsApply learning to problems MotivateMotivate CertifyCertify Provide feedbackProvide feedback Keep students actively involved!Keep students actively involved!

Testing Effectively

Why ask questions?

Only base questions on the learning Only base questions on the learning outcomes.outcomes.

Only test one learning outcome per Only test one learning outcome per question, in general.question, in general.

Avoid testing content that is directly Avoid testing content that is directly based on the material. Why?based on the material. Why?

Ensure earlier questions do Ensure earlier questions do notnot provide provide answers for later questions.answers for later questions.

Gradually increase the level of difficulty of Gradually increase the level of difficulty of the questions. Why?the questions. Why?

Testing Effectively

You can test effectively if you:

Use short clear statements.Use short clear statements. Use familiar and appropriate language.Use familiar and appropriate language. Only use abbreviations that every learner Only use abbreviations that every learner

knows.knows. Avoid acronyms.Avoid acronyms. Have one fact or concept determine the Have one fact or concept determine the

answer, for each lower-level question.answer, for each lower-level question.

Characteristics of well-written questions

For well-written questions:

Do NOT provide clues to answers, such Do NOT provide clues to answers, such as by grammar.as by grammar.

Do NOT let the sequence of choices Do NOT let the sequence of choices influence (e.g., students guessing with influence (e.g., students guessing with either “b” or “c”, the most common either “b” or “c”, the most common answers). How do you do this?answers). How do you do this?

Use simple positive statements.Use simple positive statements. Avoid negatives and especially double Avoid negatives and especially double

negatives.negatives.


Well-written questions continued:

Based on the previous pages, list two or Based on the previous pages, list two or more ideas that you should focus on for more ideas that you should focus on for writing questions well.writing questions well.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________


Things to Note

Think back to when you were a student. Think back to when you were a student. Were you ever frustrated with something Were you ever frustrated with something about a test? What caused you to be about a test? What caused you to be frustrated?frustrated?

Reducing Student Frustration

Causes of Frustration

Test important ideas. How?Test important ideas. How? Create fair tests. How?Create fair tests. How? Include media. Why?Include media. Why? Arrange questions from easy to hard, to Arrange questions from easy to hard, to

increase confidence.increase confidence. Ask similar question types together. Ask similar question types together.

Why?Why?


Reduce Student Frustration

Keep questions realistic.Keep questions realistic. Assuming it will Assuming it will notnot make the question make the question

trivial, should you provide formulas?trivial, should you provide formulas?

Why or why not?Why or why not? In general, should you allow enough time In general, should you allow enough time

for slower students to complete the test?for slower students to complete the test?

Why or why not?Why or why not? It is often fair to give students part-marks.It is often fair to give students part-marks.


Reduce Student Frustration

Based on the previous pages, list two or Based on the previous pages, list two or more ideas that you should do to reduce more ideas that you should do to reduce student frustration.student frustration.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________


Things to Note

Alternative-answer questions are most Alternative-answer questions are most commonly written as True-False commonly written as True-False questions. questions.

Other forms of alternative-answer Other forms of alternative-answer questions include Yes/No, High/Low, questions include Yes/No, High/Low, Agree/Disagree, and Fact/Opinion Agree/Disagree, and Fact/Opinion answers.answers.

Alternative-answer Questions


Identify relationshipsIdentify relationships Identify attitudes, values, and beliefsIdentify attitudes, values, and beliefs Apply known principles to new situationsApply known principles to new situations


Use these questions to:

Test the most facts in the shortest timeTest the most facts in the shortest time Are easy to write and scoreAre easy to write and score Only measure one learning outcomeOnly measure one learning outcome Are non-ambiguousAre non-ambiguous GGives students a break since there is a ives students a break since there is a

50% chance of answering correctly50% chance of answering correctly


Advantages (if written properly)

Difficulty in measuring complex learningDifficulty in measuring complex learning Not reliably measuring the skill when Not reliably measuring the skill when

students answer from recognition not students answer from recognition not memorymemory

Being subject to guessingBeing subject to guessing


Disadvantages

Use statements that are absolutely and Use statements that are absolutely and completely true or false (or yes or no …).completely true or false (or yes or no …).

Check that statements are specific and Check that statements are specific and direct, not implied.direct, not implied.

Name the authority when opinions or Name the authority when opinions or values are tested.values are tested.

Use about the same number of true and Use about the same number of true and false (or yes and no …) questions.false (or yes and no …) questions.


Question Writing Guidelines

True or false?True or false?

The sport shownThe sport shown

is called hoop ball.is called hoop ball.


Is this a well-written question?


This person isThis person is

incompetent.incompetent.




Peter Fenrich Peter Fenrich

believes that the dogbelieves that the dog

does not want to be does not want to be

in the picture.in the picture.




These two people These two people

are genetically are genetically

related.related.



Write down key things you shouldWrite down key things you should

remember about alternative-answerremember about alternative-answer

questions.questions.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________


Things to Note

Multiple choice questions usually have a Multiple choice questions usually have a statement completed or question statement completed or question answered by one choice out of a list of answered by one choice out of a list of choices.choices.

Consider asking for more than one Consider asking for more than one correct answer. For example, you could correct answer. For example, you could have three out of five answers correct. have three out of five answers correct. Why might that be better?Why might that be better?

Multiple-choice Questions

Multiple Choice Questions

These questions can be easier to write These questions can be easier to write because it can be difficult to create four because it can be difficult to create four good wrong answers.good wrong answers.

These questions can reinforce learning. These questions can reinforce learning. For example, if you only ask for one For example, if you only ask for one reason for reducing pollution, it could reason for reducing pollution, it could imply that there is only one reason.imply that there is only one reason.


If More Than One Right Choice

Measure numerous learning outcomes in Measure numerous learning outcomes in a short time a short time

Have your students differentiate between Have your students differentiate between a list of similar choices a list of similar choices

Reduce the guess factor as compared to Reduce the guess factor as compared to alternative-answer questionsalternative-answer questions


Use MC questions to:

Can (arguably) test all levels of learningCan (arguably) test all levels of learning Can test subtle differencesCan test subtle differences Test many facts in a short timeTest many facts in a short time Easy to scoreEasy to score Are non-ambiguousAre non-ambiguous



Difficult to measure some skillsDifficult to measure some skills

What are some examples?What are some examples? Difficult to make all answers believableDifficult to make all answers believable Subject to guessing and the process of Subject to guessing and the process of

eliminationelimination Not reliably measuring the skill when Not reliably measuring the skill when

students answer from recognitionstudents answer from recognition Some students have learned how to Some students have learned how to

master multiple-choice testsmaster multiple-choice tests


Disadvantages

Put all repeated words in the stem. Why?Put all repeated words in the stem. Why? Have a purpose for each answer choice.Have a purpose for each answer choice. Have short alternatives of similar lengths.Have short alternatives of similar lengths. Rarely use “all of the above”. Why?Rarely use “all of the above”. Why? Ensure “none of the above” addresses a Ensure “none of the above” addresses a

learning outcome.learning outcome.



Have 4 or 5 choices. When is 3 okay?Have 4 or 5 choices. When is 3 okay? Consider using humour for a choice.Consider using humour for a choice. Be careful with opposite answer choices Be careful with opposite answer choices

where one of the two must be right.where one of the two must be right. Have answer choices that have Have answer choices that have

something in common, if possible. Why?something in common, if possible. Why? Put answer choices in a logically-Put answer choices in a logically-

sequenced, vertical list that complete a sequenced, vertical list that complete a sentence. Why?sentence. Why?


More Question Writing Guidelines

Shooting the ballShooting the ball

through the hoop is through the hoop is

called acalled a

A. goalA. goal

B. home runB. home run

C. basketC. basket

D. touch downD. touch down

E. strikeE. strike


For a paper-based test, how can this question be improved?

An advantage of multiple-choiceAn advantage of multiple-choice

questions is that multiple-choice questions is that multiple-choice

questionsquestions

a) are easy to understanda) are easy to understand

b) are easy to writeb) are easy to write

c) are simple to answerc) are simple to answer

d) are simple to scored) are simple to score


In pairs, discuss how to fix this question

Circle every correct answer.Circle every correct answer.

Well-written multiple-choice questionsWell-written multiple-choice questions

a) always test skills reliablya) always test skills reliably

b) are the best way to testb) are the best way to test

c) can assess small differencesc) can assess small differences

d) can test many learning outcomesd) can test many learning outcomes


More than one right


remember about multiple-choiceremember about multiple-choice

questions.questions.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________


Things to Note

Matching questions require learners to Matching questions require learners to match items in one column to those of match items in one column to those of another column.another column.

Matching Questions

Matching Questions

Measure numerous learning outcomes in Measure numerous learning outcomes in a short timea short time

Have your students differentiate between Have your students differentiate between a list of similar choicesa list of similar choices

Reduce the guess factorReduce the guess factor

Matching Questions

Use matching questions to:

Being able to test complex learningBeing able to test complex learning The ability to test subtle differencesThe ability to test subtle differences Testing many facts in one questionTesting many facts in one question Being easy to scoreBeing easy to score Being non-ambiguousBeing non-ambiguous

Matching Questions


Difficulty in writing good questionsDifficulty in writing good questions Being subject to guessing and the Being subject to guessing and the

process of eliminationprocess of elimination Not reliably measuring the skill when Not reliably measuring the skill when

students answer from recognition not students answer from recognition not memorymemory

Matching Questions

Disadvantages

Have clear directions, on the same page, Have clear directions, on the same page, that include:that include:

- what is contained in each column- what is contained in each column

- an explanation of the relationship- an explanation of the relationship

- whether choices can be used more - whether choices can be used more thanthan

once or not at allonce or not at all

Matching Questions


Have the givens column (left) shorter than Have the givens column (left) shorter than the choices column (right) or allow the choices column (right) or allow choices to be used more than once. choices to be used more than once.

Put the choices column entries in a Put the choices column entries in a logical order to reduce searching time.logical order to reduce searching time.

Have similar appearing choices.Have similar appearing choices. Have a maximum of nine matches.Have a maximum of nine matches.

If you have more, what should you do?If you have more, what should you do?

Matching Questions


German wordGerman word English wordEnglish word

AbendAbend ______ a. Daya. Day

MittagMittag ______ b. Eveningb. Evening

MorgenMorgen ______ c. Morningc. Morning

NachtNacht ______ d. Nightd. Night

TagTag ______ e. Noone. Noon

WocheWoche ______

Match words on the right with the leftMatch words on the right with the left

Matching Questions


remember about matching questions.remember about matching questions.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

Matching Questions

Things to Note

Short-answer questions can have a Short-answer questions can have a variety of forms, such as completing a variety of forms, such as completing a sentence, answering a question, and sentence, answering a question, and providing a list.providing a list.

Short-answer Questions


Test recall rather than recognition of Test recall rather than recognition of information information

Check computational and problem-Check computational and problem-solving skillssolving skills

Evaluate knowledge of procedural skillsEvaluate knowledge of procedural skills Minimize the guess factorMinimize the guess factor


Use short-answer questions to:

Testing many facts in a short timeTesting many facts in a short time Being easy to score Being easy to score

(i.e., assign a number to the score)(i.e., assign a number to the score) Having a low guess factorHaving a low guess factor Reliably measuring the skill since Reliably measuring the skill since

students answer from memory not students answer from memory not recognitionrecognition



Hard to measure complex learningHard to measure complex learning Challenges in scoring objectivelyChallenges in scoring objectively Relatively high likelihood of having Relatively high likelihood of having

ambiguityambiguity Takes more time to provide feedbackTakes more time to provide feedback Takes students more time to answerTakes students more time to answer


Disadvantages

Provide rules for spelling, grammar, extra Provide rules for spelling, grammar, extra words, units, significant figureswords, units, significant figures

Only have specific keywords omittedOnly have specific keywords omitted Generally, answers should be one word Generally, answers should be one word

or phraseor phrase Be careful about the answer slot’s sizeBe careful about the answer slot’s size Have one answer slot per question unless Have one answer slot per question unless

the student should provide a listthe student should provide a list

- Consider providing an exact number of - Consider providing an exact number of answer slots for a listanswer slots for a list



If the answer is “keywords”, which of the If the answer is “keywords”, which of the following is the better format for following is the better format for learners?learners?

a) _________ should complete the a) _________ should complete the

beginning of a statement.beginning of a statement.

b) It is best for _________ to completeb) It is best for _________ to complete

the middle of a statement.the middle of a statement.

c) The statement should be completed c) The statement should be completed

by _________. by _________.




remember about short-answer questions.remember about short-answer questions.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________


Things to Note

Long-answer questions can entail a Long-answer questions can entail a variety of challenges, such as requiring variety of challenges, such as requiring learners to apply, analyze, synthesize, or learners to apply, analyze, synthesize, or evaluate related products, processes, evaluate related products, processes, information, ideas, and/or procedures.information, ideas, and/or procedures.

This can result in a/an report, essay, This can result in a/an report, essay, story, paragraph, letter, memo, email, story, paragraph, letter, memo, email, scientific paper, article, description, scientific paper, article, description, commentary, presentation, video or audio commentary, presentation, video or audio recording…recording…

Long-answer Questions


To make the challenge realistic, long-To make the challenge realistic, long-answer questions can be based on case answer questions can be based on case studies or problem-based learning studies or problem-based learning situations. situations.

You can decide whether your long-You can decide whether your long-answer questions will refer to resources answer questions will refer to resources or have limits on length, time, or scope.or have limits on length, time, or scope.


Long-answer Questions Cont’d

Use long-answer questions to assess a Use long-answer questions to assess a student’s ability to:student’s ability to:

- Write- Write

- Evaluate information- Evaluate information

- Synthesize ideas- Synthesize ideas

- Analyze information- Analyze information

- Organize thoughts and ideas- Organize thoughts and ideas

- Apply information- Apply information

- Explain, describe, and/or recall content- Explain, describe, and/or recall content


Long-answer Questions Cont’d

The ability to test many high-level skills.The ability to test many high-level skills. There is a low guess factor.There is a low guess factor.


Advantages (if written well)

It can be difficult to score objectively (or It can be difficult to score objectively (or evenly). A well-written rubric can help evenly). A well-written rubric can help with this.with this.

It can require a lot of time to mark.It can require a lot of time to mark. It can take a fair amount of time for It can take a fair amount of time for

students to do what was requested.students to do what was requested.


Disdvantages

Have very clear directions that include:Have very clear directions that include:

- the criteria for scoring (which you should - the criteria for scoring (which you should thoroughly analyze before your students thoroughly analyze before your students answer the question)answer the question)

- guidelines for the length of the response - guidelines for the length of the response (e.g., number of words) (e.g., number of words)

- rules for marking, spelling, and - rules for marking, spelling, and grammar, if neededgrammar, if needed

Provide enough space for students to Provide enough space for students to respondrespond



Scoring long-answer questions can be a Scoring long-answer questions can be a challenge. One method is to simply challenge. One method is to simply compare your students' answer to an compare your students' answer to an ideal answer. For example, students ideal answer. For example, students would receive marks based on how well would receive marks based on how well they cover each outcome.they cover each outcome.

As mentioned, rubrics can help.As mentioned, rubrics can help.


Scoring long-answer questions


remember about long-answer questions.remember about long-answer questions.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________


Things to Note

In general, do NOT In general, do NOT compare students to compare students to determine marks. determine marks. Normalizing scores to fit a Normalizing scores to fit a bell (or “normal”) curve is bell (or “normal”) curve is against BCIT policy.against BCIT policy.

The criterion is whether The criterion is whether the learner achieved the the learner achieved the learning outcome. You learning outcome. You can have numerous high can have numerous high or low marks.or low marks.

Scoring

Scoring

If marks are relatively high then:If marks are relatively high then:

- The learning outcomes may be too - The learning outcomes may be too

simple.simple.

- Your tests may be too easy.- Your tests may be too easy.

- Your teaching may be excellent.- Your teaching may be excellent.

- Everything may be as it should be.- Everything may be as it should be.

Scoring

High Scores

If marks are relatively low then:If marks are relatively low then:

- Learning outcomes may be set too high.- Learning outcomes may be set too high.

- Your tests may be too hard.- Your tests may be too hard.

- Your students may need to work harder,- Your students may need to work harder,

need to be motivated, or have a problemneed to be motivated, or have a problem

such as a lack of time.such as a lack of time.

- Your teaching may need improvement.- Your teaching may need improvement.

- Everything is going as well as it can.- Everything is going as well as it can.

Scoring

Low Scores

What questions about What questions about writing effective writing effective

questions and testsquestions and tests do you still have? do you still have?

writing effective questions and tests

Education

wwhhyy oorr wwhhyy nnoott

hhaavvee yyoouu eevveerr

cclluueess ttoo aannsswweerrss

student frustrationthings

aanndd lloonngg

lleevveell qquueessttiioonn

iinn ggeenneerraall

aavvooiidd aaccrroonnyymmss