Crafting questions that measure what

you want to measure

Liz Norman

Massey University

General issues with question

wording

CommunicationThe examination question is the question setter’s expression

of the question setter’s task.

The student’s answer represents the student’s expression of

the student’s interpretation of the question.

The marker evaluates the marker’s interpretation of the

student’s expression of the student’s answer.

The marker uses the marker’s interpretation of the setter’s

expression of the setter’s task to evaluate the student’s

answer.

Modified from Pollitt & Ahmed (1999) New Model of the Question Answering Process. IAEA. Bled, Slovenia

Expectations and stereotypes

Examples:

• “treatment”

• all differential diagnoses vs those only applicable

in a particular case

• expectation that Qs will ask about what

something is rather than what it is not

• expectation of hard questions

Elephant RidingJan Kemp

Climbing up

the back of an elephant

you spring into

the toehold of its tail

held in place by the mahout

grab the ropes

strapped round its belly

& haul yourself up.

She rises

from buckled knees under you

moves like a ship

you’re high

under the hanging ashoka leaves

as you flow forward

her fly-bitten ears grey sails flap.

she flings the odd young-leaved branch

into her mouth

with her triumphant trunk.

Crisp et al. (2008). Tales of the expected: The influence of students' expectations on question validity

and implications for writing exam questions. Educational Research, 50(1), 95-115.

Year 11 Science exam Q

Discuss the diagnosis and management of a cat

with both chronic renal failure and hyperthyroidism.

(24 minutes)

Contextualising Qs

• Context is good because it brings relevance and

authenticity

• Allows assessment of concrete or specific examples

not abstract concepts or generalisations

• Allows assessment of applied learning (doing not

just knowing)

• All these carry with them a potential for bias.

Contextualising Qs

• Other disadvantages:

– More words used

– More intended/unintended demand

– Familiarity – schemas/sterotypes

– Focus may direct candidates to the wrong aspects

– Images can be particularly distracting

You are presented with a poorly controlled diabetic dog. The dog is an 8 year old female spayed Labrador weighing 36 kg with a body condition score of 7/9. The dog has been on porcine lente insulin (Caninsulin) twice daily for 3 months. Various doses have been tried during this time and the dog is currently receiving 70 units twice daily. The dog is still polyuric and polydipsic with 3+ to 4+ glucosuria.

Discuss the possible causes of poor control which you should consider when evaluating this dog.

List the possible causes of consistently high blood

glucose values in a dog being treated with

subcutaneous insulin injections for the treatment of

diabetes mellitus. Outline your approach to

investigating these possible causes.

Question 1

A crate of 12 cans of cola costs $4.20. How much

do 7 crates of cola cost?

Question 2

A ski pass costs $4.20. How much would it cost for

7 days?

Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus.

Assessment in Education: Principles, Policy & Practice, 14(2), 201-232.

Ah

me

d &

Po

llitt

(2

00

7)

Imp

rovin

g th

e q

ua

lity o

f co

nte

xtu

aliz

ed q

ue

stion

s: A

n e

xpe

rim

en

tal

investig

atio

n o

f fo

cus. A

sse

ssm

en

t in

Ed

uca

tion

: P

rin

cip

les,

Po

licy &

Pra

ctice, 1

4(2

), 2

01

-232

.

Guidelines for using context

• If you are going to use context use a natural real

one

• Contextualise purposefully for what it brings to

the task

• Only use images when the Q could not be asked

without them

• Don’t decorate!

Don’t write questions; write tasks

What is your diagnosis?

State the most likely diagnosis

State the most likely diagnosis and explain your reasoning

Discuss the differential diagnoses you would consider in this case

or …..

Instructional verb examples

Compare: to find similarities between things, or to look for characteristics and features that resemble each other.

Contrast: to find differences or to distinguish between things.

Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.

Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.

Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.

Illustrate: to use a picture, diagram or example to clarify a point.

Specify boundaries of the answer

Speciese.g. “in both dogs and cats…”

Quantities and amountse.g. “Provide 5 reasons why…”

With reference toe.g. “ With reference to the published research from ..”

Timeeg: “in the first 24 hours”

Part of the questionEg: “for one of your differentials….”

What are the

clinical signs of

hypothyroidism

in dogs and how

do they arise?

List the three most

common owner-

observed clinical signs

of hypothyroidism in

dogs and explain how

thyroid hormone

deficiency leads to each

of these signs.

Guidance

Question 1

Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)

Question 2

a) Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats (10 marks)

b) Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)

c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).

Guidance

You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario.(20 marks)

Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide

Wording questions – examples

of problems

Name two (2) diagnostic tests you would run next to

investigate the cause of this dog’s current illness.

Outline your approach to confirming the initial

clinical diagnosis and a management and

prevention plan for this problem. This discussion

should include an outline on further observations

taken about ….

…list in dot point form: the gross pathological

features, the characteristic histopathological

changes, and the clinical pathology changes. In

your discussion, list one antemortem

test/procedure that can be used to aid in the

diagnosis …

A veterinarian asks you for assistance in designing

a protocol for the delivery of a vaccine for cats in

their practice. What factors would you take into

consideration in designing this protocol?

Are there any clinical features which can help you

determine this patient’s prognosis?

State what you believe is your most likely

diagnosis.

Discuss commonly found tumours and tumour-like

disorders associated with the oral cavity and

dental tissues of the horse.

How would you localise the site of the lesion?

Answer provided in the marking scheme:

Spinal lesion between T3 and L3

Write notes on

a) considerations in the selection of stockpersons

and animal attendants

Describe and give reasons for your further clinical

examination (if any), and recommendations to the

owner/trainer for diagnostics and treatment.

Explain what you think is really important and why.

Explain if you think there is any controversy in

treatment plans.

Sentence-level clarity

• Simple sentence structures

• Grammatically correct

• Remove superfluous words

Question timing

How long it will take the candidate

• Unstructured tasks – open ended

• Need to control the demands of the Q carefully

• Question reading time

• Writing speed

Klatt & Klatt (2011) Acad Med. 86:1079–1083

Poppy is a 9 year old 12 kg female neutered fox terrier cross who has been on treatment for diabetes mellitus with Caninsulin (porcine lente insulin) for 4 months. She is currently receiving 5 units subcutaneously at 7:30 am and 5 units at 7:30 pm. At the time of each injection she is fed a mixture of “Optimum” dog roll and Pedigree Pal Meaty Bites. Her last visit was 2 weeks ago when her dose of insulin was decreased by 2 units.

Since the last visit Poppy has been happy and active and is eating all her food at mealtimes. Her body weight has not changed since the last visit and her body condition score is 5/9. Her owner reports her to be drinking about 600mL of water daily. Her owner is very happy with Poppy’s progress and comments that she is her normal self again. There has been no signs of hypoglycaemia.

You detect no abnormalities on physical examination and admit Poppy to perform a serial blood glucose curve. The owner has administered the morning insulin dose and fed her as usual. The following results are obtained.

Based on your assessment of the clinical information and the glucose curve shown what dose of insulin would you recommend Poppy be given now? (5 marks)

A client rings you up to arrange his quarterly visit as he is frustrated by ongoingproblems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also had a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also think she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear.

On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present.

You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)

Summers & Catarro (2003) Australian Occupational Therapy Journal 50(3): 148-157

Effect of time stress

• Time stress

– Increases the use of schemas (sterotyping)

– Decreases working memory processing capacity

– Decreases the ability to maintain relevant information

and suppress irrelevant information

http://www.slideshare.net/liznorman