sp110623
DESCRIPTION
Nursing MCQ sessionTRANSCRIPT
1
Some perspectives onMCQ assessment
For School of Nursing and
Midwifery July 2011
Source: http://www.dreamstime.com/item.php?imageid=7299038
2
Assessment generally
“Examinations are formidable even to the best prepared, for the greatest fool may ask more than the wisest man can answer.”
Charles Caleb Colton (1780 - 1832)
3
General thoughts
MCQs can be a useful element of an overall assessment design
Can’t be abstracted from the contextNo marks / diagnostic / formative / interactive
Few marks / semi-formative / assignment
High stakes / summative / examination
Design and evaluation approach should be appropriate for the context
4
A proposed MCQ process
From: Considine, J; Botti, M and Thomas, S. Design, format, validity and reliability of multiple choice questionsfor use in nursing research and education. Collegian, 2005 Jan; 12 (1): 19-24.
5
An alternative/pragmatic processDesign
Content/Face validity
Use
Construct validity
Reliability
Keep / Replace
6
Design – unit assessment plan
Assignment No. Assignment details Marks
1 Introduction to DSO 3 %
2 Information literacy & the Library 5 %
3 Referencing 10 %
4 Professional ethics report 15 %
5 Multi-choice test 1 6 %
6 Multi-choice test 2 6 %
7 Major report and presentation 25 %
Exam Must pass! 30 %
7
Design – MCQ test plan
ContentAreas
Processes
1. Knowledge 2. Comprehension 3. Application %
A 3.75% 5.25% 6% 15
B 6.25% 8.75% 10% 25
C 3.75% 5.25% 6% 15
D 7.5% 10.5% 12% 30
E 3.75% 5.25% 6% 15
% 25 35 40 100
8
Design – NCLEX-RN MCQ test plan
Source: 2010 NCLEX-RN® Detailed Test Plan
9
Face validity
A sub-set of content validity
Use an ‘expert panel’ to : assess item clarity assess item readability assess consistency of style check for errors
10
Construct validity
Key check – do ‘experts’ agree on the correct answer?
Item difficulty – the proportion of examinees that get the correct answer
Distracter evaluation
Item discrimination – the ability of an item to distinguish between knowledgeable students and others
11
BlackBoard Vista item analysis report
12
Review of items
Where items have been developed based on a systematic test plan and good content validity, they should only be discarded on the basis of clearly poor construct validity and reliability
13
MCQ Feedback
Butler and Roediger (2008) found that:
“In comparison with the no-feedback condition, both immediate and delayed feedback increased the proportion of correct responses and reduced the proportion of intrusions (i.e., lure responses from the initial multiple-choice test) on a delayed cued recall test.
Educators should provide feedback when using multiple-choice tests.”
14
MCQ Feedback – strategies
Provide consolidated feedback after the test has been completed
Explanation of why the key is correct
Explanation of why a distracter is incorrect
Explanation of the general reasoning associated with an item
15
MCQ Feedback – strategies
Directing students to associated resources – text chapters/sections/pages, etc.
Providing a run-through/de-brief in class after the test
Randomisation of MCQ presentation to counter rote learning of ‘saved’ tests
16
A usefulresource?
Source: https://www.ncsbn.org/1287.htm
17
Development of the NCLEX-RN Test Plan
“A total of 12,000 newly licensed RNs are asked about the frequency and importance of performing 155 nursing care activities. Nursing care activities are then analyzed in relation to the frequency of performance, impact on maintaining client safety and client care settings where the activities are performed. This analysis guides the development of a framework for entry-level nursing practice that incorporates specific client needs, as well as processes fundamental to the practice of nursing.”
18
NCLEX-RN test plan structure
Safe and Effective Care EnvironmentManagement of CareSafety and Infection Control
Health Promotion and Maintenance
Psychosocial Integrity
Physiological IntegrityBasic Care and ComfortPharmacological and Parenteral TherapiesReduction of Risk PotentialPhysiological Adaptation
19
Design – NCLEX-RN MCQ test plan
20
Detailedtest plan
21
Detailedtest plan
22
Sample items – management of care
For each of the eight content areas a sample item is provided
23
Steps in item writing
1. Select an area of the test plan for the focus of the item
2. Select a subcategory from the chosen area of the test plan
3. Select an important concept within that subcategory
4. Use the concept selected and write the stem
5. Write a key to represent important information the entry-level nurse should know
6. Identify common errors, misconceptions or irrelevant information
7. Use the previous information and write the distracters
8. Complete the item using the stem, key and distracters
24
Steps in item writing - example
Step 1. Select an area of the test plan for the focus of the itemSafety and Infection Control
Step 2. Select a subcategory from the chosen area of the test planStandard Precautions/Transmission-Based Precautions/Surgical Asepsis
Step 3. Select an important concept within that subcategoryEvaluate infection control precautions implemented by staff members
25
Steps in item writing - example
Step 4. Use the concept selected and write the stemThe nurse and nursing assistant are caring for a client with vancomycin-resistant enterococci (VRE). Which of the following activities by the nursing assistant would require immediate follow-up?
Step 5. Write a key to represent important information the entry-level nurse should knowContact Isolation: Assisting the client to ambulate in the hallway
26
Steps in item writing - example
Step 6. Identify common errors, misconceptions or irrelevant informationLack of understanding of isolation precautionsUncertainty related to specific diagnosis
Step 7. Use the previous information and write the distractersLeaving a blood pressure cuff in the client’s room to be used by the client onlyPutting on a protective gown to assist the client to sit in a chairTaking the gloves off before leaving the client’s room
27
Steps in item writing - example
Step 8. Complete the item using the stem, key and distracters
The nurse and nursing assistant are caring for a client with vancomycin-resistant enterococci (VRE). Which of the following actions performed by the nursing assistant would require immediate follow-up from the nurse?
a. Leaving a blood pressure cuff in the client’s room to be used by that client only
b. Putting on a protective gown to assist the client to sit in a chair
c. Taking the gloves off before leaving the client’s room
d. Assisting the client to ambulate in the hallway (key)
28
Case scenario exercise
Reduction of Risk Potential
The nurse is caring for client who had a procedure three hours ago. Write an item that includes assessment data the nurse would observe in this client and which data should the nurse respond to first.
29
Reduction of Risk Potential – model
The nurse is caring for a client who had a cardiac catherization 3 hours ago. Which of the following findings would be essential for the nurse to follow-up?
a. blood pressure increase from 103/68 to 110/70 over the past one hour
b. blood urea nitrogen (BUN), 22 mg/dL (key)
c. pulse, 101
d. decrease in respiratory rate from 18 to 16 over the past one hour
30
Case scenario exercise
Psychosocial Integrity
A nurse on an inpatient psychiatric unit observes a client pacing the hallway, mumbling and occasionally yelling aloud “Stop it!” Write an item describing the action the nurse should take in this situation.
31
Psychosocial Integrity – model MCQ
The nurse is caring for a client who is mumbling, pacing in the hallway, and occasionally yelling “Stop it!” Which of the following actions should the nurse take?
a. Remove other clients from the area.
b. Escort the client back to the client’s room.
c. Request that the client be quiet and not disrupt others.
d. Use distraction to re-focus the client to reality. (key)
32
Thank you for your time