Written ExamAssessing knows and knows

how

Departemen Obstetri dan Ginekologi FKUI-RSCM

Three levels of test

1. Knowledge (recall or recognition of specific information)

2. Combined comprehension and application

3. Problem solving (transferring existing knowledge and skills to new situations)

Guidelines for Basic Item Content

• Clinical vignettes• Focus on key concepts and principles

that are essential information• Test material relevant to learning in

clinic• Avoid items that only require recall of

isolated facts• Avoid esoteric or interesting topics

that are not essential

Tools

Case vignettes should include:

age, gender site of care presenting complaint Duration patient history physical findings +/- diagnostic studies +/- initial treatment

Stems should:

not be completely based on real patients

include reference material when it would be realistic in practice

not use the patient’s or doctor’s own words

not include patients who lie

Multiple Choice Questions (MCQs)

Introduction

MCQs can assess taxonomically higher-order cognitive processing such as interpretation, synthesis and application of knowledge rather than the test of recall of isolated facts.

Should not be used as a sole assessment method in summative examinations, but alongside other test forms

Advantage

Can assess a broad range of learner knowledge in a short period of time (time effective)

Appropriately constructed MCQs result in objective testing that can measure knowledge,comprehension application and analysis

reliable, discriminatory, reproducible and cost-effective

Disadvantage

they test recognition (choosing an answer) rather than recall (constructing an answer)

allow for guessing difficult and time-

consuming to construct

Writing MCQs

Begins with identifying the most important information or skill that is to be learned

Focus on important and relevant content, avoiding testing the knowledge of medical trivia

Controversial items should be avoided

Components of MCQs

The item : entire unit and consists of a stem and several options.

The stem : the question, statement or lead-in to the question.

options or choices : the possible answers keyed response : The correct option foils or distractors : The incorrect options

Parts of MCQs “item” A 62-year-old woman comes to the

physician because of bleeding from the vagina. She states that her last menstrual period came 11 years ago and that she has had no bleeding since that time. She has hypertension and type 2 diabetes mellitus. Examination shows a mildly obese woman in no apparent distress. Pelvic examination is unremarkable. An endometrial biopsy is performed that shows grade I endometrial adenocarcinoma

Which of the following is the most appropriate next step in management?  a. Chemotherapy b. Cone biopsyc. Dilation and curettaged. Hysteroscopye. Hysterectomy *

vignette

Lead in

distractors Option

Stem

answer

Technical Item Flaws

Issues Related toTestwiseness

• Grammatical cues• Logical cues• Absolute terms - such as always or never • Long correct answer• Word repeats• Convergence strategy

• Options are long, complicated, or double• Numeric data are not stated consistently• Terms in the options are vague (eg, “rarely,”

“usually”)• Language in the options is not parallel• “None of the above” is used as an option• Stems are tricky or unnecessarily complicated• The answer to an item is “hinged” to the

answer of a related item

Issues Related to Irrelevant Difficulty

Make sure the item can be answered without looking at the options, (the options are 100% true / false).

Avoid technical Item Flaws Write distractors that are plausible and

the same relative length as the answer.

General Guidelines for Item Construction

General….

• Avoid using absolutes such as always, never, and all in the options; also avoid using vague terms such as usually and frequently.

• Avoid negatively phrased items. If you must use a negative stem, use only short (preferably single word) options.

And most important of all: Focus on important concepts; don’t waste time testing trivial facts

MCQ writing step

Do Don’t

Before writing

assess learning outcomes or important concepts

Identify the cognitive level at which the MCQ should be pitched

Decide on the topic and content area

assess trivial, insignificant facts

Writing the stem

a common clinical case Include as much information as

required to arrive at the correct answer, i.e. a long stem (with short options)

synthesise for the candidate, technical item flaws, such as

A word in the stem repeated in the option(s)

Tricky/complicated stems Clues to the answer in the stem

Writing the lead-in

clearly indicate how to answer the MCQ

preferably be a question Refer back to the topic & content

area, when constructing the lead-in

Use questions and avoid phrases e.g. Regarding epilepsy:

Avoid technical item flaws, such as: Absolute terms, e.g. always,

never Frequency terms, e.g. rarely Which of the following

statements is correct?’ This type of lead-in may lead to heterogeneous options

Negative questions

MCQ writing step

Do Don’t

Checking the stem and lead-in

Lead-in and stem must give enough information to answer the MCQ, without/before reading the options

Both should be clear, precise and simple

create a ‘test within a test’.

Writing the options

only one clear answer Distractors clearly incorrect, but plausible short and uncomplicated All options homogeneous List in a logical order The positions of the correct option should vary

with other MCQs All options are of similar length Use coherent, consistent terminology

technical item flaws, such as:

Related to testwiseness

Related to irrelevant difficulty

After writing 1. Does the MCQ assess an important concept?2. Does the MCQ test factual recall of knowledge, application or evaluation?3. Can the MCQ be answered by only reading the stem & lead-in?4. Are all the options homogeneous?5. Is the MCQ (stem, lead-in and options) devoid of technical item flaws?

EXTENDED MATCHING QUESTIONS

EXTENDED MATCHING QUESTIONS

Extended Matching items are multiple-choice items organized into sets that use one list of options for all items in the set.

A well-constructed Extended-Matching set includes four components (R-type):1. a theme2. an option list3. a lead-in statement4. at least two item stems

1. Identify the theme for the set.

2. Write the lead-in for the set.

3. Prepare the list of options.

4. Write the items.

5. Review the items. Check to make sure that

there is only a single “best” answer for each

question.

Overview of the Steps for Writing Extended Matching Items

Example of an EMQs Theme: Bleeding per vagina Options

A. Cervicitis G. Endometrial cancerB. Atrophic vaginitis H. Endometrial polypC. Cervical cancer I. Incomplete miscarriageD. Cervical polyp J. Threatened miscarriageE. Complete miscarriage K. Toxic-shock syndromeF. Ectopic pregnancy

Instructions/Lead-inFor each of the patients described below choose the SINGLE most appropriate diagnosis from the list of options above. Each optionmay be used once, more than once or not at all.

.

Item:

1. A 62-year-old woman comes to the physician because of bleeding from the vagina. She states that her last menstrual period came 11 years ago and that she has had no bleeding since that time. She has hypertension and type 2 diabetes mellitus. Examination shows a mildly obese woman in no apparent distress. Pelvic examination is unremarkable. Ultrasound revealed thickened endometrium 15 mm, with increase vascularization.

2. A 25 years old woman comes to emergency because of bleeding from vagina. She states that her last menstrual period came 2 months ago and she already did pack test and the result was positive. On vaginal examination the doctor found that the ostium was open and there was conception tissue in it.

Short Answer Questions

Introduction

Short-Answer questions require a student to respond to a question by writing a word or short phrase.

Questions can vary in expected student response- from one word to short sentence, and include forms : complete the sentence supply the missing line short descriptive or qualitative answers

Advantage

Assess higher-order or critical thinking skills

Provide authentic experience.

easier to mark. Evaluate student

thinking and reasoning

Disadvantage

Assess a limited sample of the range of content.

Are difficult and time consuming to grade.

Procedures for setting and marking

Make the questions precise Direct questions are better than incomplete

statements. If a numerical answer is required, indicate the

units and degree of precision required. Prepare a structured marking sheet Allocate marks or part-marks for acceptable

answer(s). Be prepared to accept other equally acceptable

answers, some of which you may not have predicted.

Example of short answer questions

A 62-year-old woman comes to the physician because of bleeding from the vagina. She states that her last menstrual period came 11 years ago and that she has had no bleeding since that time. She has hypertension and type 2 diabetes mellitus. Examination shows a mildly obese woman in no apparent distress. Pelvic examination is unremarkable. Ultrasound revealed thickened endometrium 15 mm, with increase vascularization. What is the most likely diagnosis of this patient

Thank you…..