question development
DESCRIPTION
How do I create good multiple choice items?. Question Development. Agenda. Black Box Why we are here. Black Box. Back to the Classroom. Black Box. What happens inside…. Black Box. Does Formative Assessment raise standards? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/1.jpg)
Question DevelopmentHow do I create good multiple choice items?
![Page 2: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/2.jpg)
Agenda
Black BoxWhy we are here
![Page 3: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/3.jpg)
Black Box
![Page 4: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/4.jpg)
Back to the Classroom
![Page 5: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/5.jpg)
Black Box
What happens inside…
![Page 6: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/6.jpg)
Black Box
Does Formative Assessment raise standards?
(I prefer “achievement” to standards.)
1. enhances communication between student and teacher.
2. actively involves students
3. yields results that are then used to adapt teaching
4. affects the motivation and self-esteem of students
![Page 7: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/7.jpg)
Black Box
Is there room for improvement in the way we assess?
Issue #1 cultivate understanding, not rote learning; work
collaboratively with other teachers; emphasize quality, not
quantity.
Issue #2 emphasize learning, not grading; encourage each student’s
personal improvement; convey your belief that students can
learn
Issue #3 analyze student work to discern their learning needs; make
feedback meaningful
![Page 8: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/8.jpg)
Black BoxHow can we improve formative
assessment?
Feedback to students should be about the particular qualities of their work.
Students should be trained in self-assessment so that they can understand the main purposes of their learning and what they need to do to achieve.
Effective teaching includes opportunities for students to express their understanding.
![Page 9: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/9.jpg)
Black BoxSo…if I am in the black box, how do
I respond?1. I learn from professional development.
2. I start in low gear and patiently find my own way of incorporating new learning.
3. While remaining aware of external tests, I take leading responsibility for assessing ongoing work to monitor and improve progress.
4. Continually reflect on my work.
![Page 10: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/10.jpg)
Research
Black & Wiliam Research Questions:1. Does better formative assessment
equal higher test scores?2. Does formative assessment need
improving?3. If so, what improvement is needed?
![Page 11: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/11.jpg)
Research FindingsStudy Standard Deviation GainsBloom (1984) 1.0 to 2.0Black and William (1998) .5 to 1.0Meisels, et al. (2003) .7 to 1.5Rodriguez (2004) .5 to 1..8Harrie and Timperley (2007) .5 to 1.0
![Page 12: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/12.jpg)
Why are we here?
What’s coming with Common Core Curriculum?
EOCT vs. GHSGTContinuous Improvement
![Page 13: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/13.jpg)
Our Element
The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.
EQ? How can well designed assessments evaluate student performance on the GPS and provide information about student strengths and weaknesses?
![Page 14: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/14.jpg)
Determine and define vocabulary Identify key terms within the
standard(s)/element(s). Define each term. In your group:
▪ Underline key terms in the element.▪ Define key terms
The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.
![Page 15: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/15.jpg)
Study the standard(s)/element(s)Generate key questions related to
the standard(s) and/or element(s).Answer each question.
The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.
![Page 16: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/16.jpg)
Identify strong and weak workCollect and rate strong and weak
samples of work related to the standard(s) and/or element(s). List examples of strong and weak work Use information on handout to help
The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.
![Page 17: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/17.jpg)
Scaffold understanding of the standard(s) and/or element(s)Paraphrase the standard(s) and/or
element(s).
The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.
![Page 18: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/18.jpg)
Develop “I Can” statementsDescribe the standard(s) and/or
element(s) as statements of intended learning (e.g., I can use information from what I read to draw conclusions (make inferences), I can use mathematical vocabulary to describe how I solved a problem, etc.).
The teacher uses a variety of summative assessment strategies to evaluate student status relative to mastery of the Georgia Performance Standard.
![Page 19: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/19.jpg)
Depth of Knowledge (DOK) Jigsaw Activity
1. Count off at your table 1, 2, 3, 4, 1, 2, …
2. All 1’s meet; All 2’s meet; All 3’s meet; All 4’s meet
3. In “Expert Groups” read and discuss everything about your DOK Level
4. Return to your “Home Group”5. Experts report out to the “Home Group”
![Page 20: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/20.jpg)
Common Verbs
Describe Explain IdentifyIllustrate Trace CompareContrast Predict ApplySort Justify AnalyzeEvaluate Summarize DefineList Differentiate DistinguishDefend Conclude Categorize
![Page 21: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/21.jpg)
![Page 22: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/22.jpg)
Rules are made for breaking?Basic rules for creating multiple
choice itemsReduce systematic errors Increase reliability of test items
Basically – design the questions to determine what the student really knows or does not know… Not whether or not the student has “test
taking skills”
![Page 23: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/23.jpg)
“Misguided” Test Taking TipsGuess if you are not sureLook for answer choices that
grammatically complete the question If one answer is much longer than
the others – it is most likely the correct answer
If “All of the Above” is provided and you are reasonably sure of two answer choices , “All of the Above” is most likely correct.
![Page 24: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/24.jpg)
Multiple Choice Items
Who was president during World War II?A. Franklin RooseveltB. George WashingtonC. William ClintonD. Ronald Reagan
Item StemKey
Distracters
![Page 25: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/25.jpg)
Rule #1
Make the problem clear to the student
Poor Example:1. Antigens
A. Attack memory cellsB. Attack killer T Cells
C. Attack pathogens D. Attack helper T cells
![Page 26: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/26.jpg)
Rule #1
Better Example:
1. Which of these do antigens attack?
A. PathogensB. Helper T cellsC. Killer T cellsD. Memory cells
![Page 27: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/27.jpg)
Rule #2 State the item stem in the positive
whenever possible.Poor Example:1. Which of these is not a reason why President
Roosevelt decided not to involve the United States in World War II until the bombing of Pearl Harbor?
A. The US was upset with Great Britain over the sinking of merchant ships.B. The US was involved in another war at the time.C. The US was already allied with Germany at the time of Pearl Harbor.D. The US had not been attacked directly until Pearl Harbor.
![Page 28: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/28.jpg)
Rule #2Better Example1. Which of these best describes why the
United States became involved militarily in World War II?
A. The US was directly attacked.B. Great Britain asked for assistance
against German submarines.
C. French resistance needed military arms after Germany invaded.
D. The US was obligated to become involved by its membership in NATO
![Page 29: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/29.jpg)
Rule #3
Make sure the item stem does not give away the correct answer.
Poor Example:1. What led to the formation of the States'
Rights Party? A. The level of federal taxation B. The demand of states for the right to
make their own laws C. The industrialization of the South D. The corruption of federal legislators
on the issue of state taxation
![Page 30: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/30.jpg)
Rule #3
Better Example:
1. What led to the formation of the States' Rights Party?
A. The level of federal taxation B. The demand by states to make
their own laws C. The industrialization of the South D. The corruption of federal
legislators on the issue of state taxation
![Page 31: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/31.jpg)
Rule #4
Emphasize qualifiers such as most likely and best in the stem.
Poor Example:1. Which measurement unit below
would be best to use if you wanted to measure the mass of a bumblebee?
A. gram B. kilogramC. millimeterD. centimeter
![Page 32: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/32.jpg)
Rule #4
Better Example
1. Which measurement unit below would be best to use if you wanted to measure the mass of a bumblebee?
A. gram B. kilogramC. millimeterD. centimeter
![Page 33: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/33.jpg)
Rule #5
Make sure the answer choices are plausible.
Poor Example:1. Who was president during World
War II?A. Franklin RooseveltB. George WashingtonC. George W. BushD. Jesse James
![Page 34: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/34.jpg)
Rule #5
Better Example1. Who was president during World
War II?
A. Dwight EisenhowerB. Lyndon JohnsonC. Franklin RooseveltD. Woodrow Wilson
![Page 35: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/35.jpg)
Rule #6 Develop answer choices that are parallel in
grammar and in length.Poor Example:1. In English class, Cynthia must write a descriptive
essay about the person she admires the most. She wants to write about her father. Which of these would be the best way she could begin to write her essay?
A. Look through old picture of her and her fatherB. She could call her best friend to talk about himC. By making a list of activities she has done with her father.D. Asking her father to come to her class
![Page 36: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/36.jpg)
Rule #6
Better Example:
1. In English class, Cynthia must write a descriptive essay about the person she admires the most. She wants to write about her father. Which of these would be the best way she could begin to write her essay?
A. By looking at her baby picturesB. By asking her father to come to her
classC. By calling her best friend to talk about
her fatherD. By listing all of the things he has done
for her and others
![Page 37: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/37.jpg)
Rule #7
Avoid using “All of the above” and “None of the above.”
Poor Example:1. Which of these should be used to
measure in millimeters?
A. A yard stickB. A metric rulerC. A digital scaleD. None of the above
![Page 38: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/38.jpg)
Rule #7
Better Example:1. Which of these should be used to
measure in millimeters?
A. A yard stickB. A metric rulerC. A digital scaleD. A graduated cylinder
![Page 39: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/39.jpg)
Rule #8 Place the answer choices in logical order.
Numbers in numerical order One-word answer choices in alphabetical order Order sentences from shortest to longest
Poor Example:1. A teacher is planning a field trip and will need
school buses to transport students A school bus holds 36 students. If 1,128 students will be transported, how many buses are needed?A. 32B. 31.33C. 36D. 31
![Page 40: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/40.jpg)
Rule #8
Better Example1. A teacher is planning a field trip and
will need school buses to transport students A school bus holds 36 students. If 1,128 students will be transported, how many buses are needed?
A. 31B. 31.33C. 32D. 32.33
![Page 41: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/41.jpg)
Rule #9
Avoid clues in the answer choices that give away the correct answer.
Poor Example1. Which of the following revolutions
resulted in increased business by speeding travel?A. industrialB. technologyC. textileD. transportation
![Page 42: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/42.jpg)
Rule #9
Better Example
1. Which of the following best describes a result of the transportation revolution in the 1800s?
A. Suburban areas grewB. Demands for slave labor decreasedC. Manufacturing and production increasedD. People relied less on mass transportation
![Page 43: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/43.jpg)
Rule #10 Make sure that the correct response is
the only correct response.Poor example1. Chicago Bears: football::
_________:baseball
A. Anaheim AngelsB. Los Angeles GalaxyC. Atlanta BravesD. Tampa Bay Buccaneers
![Page 44: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/44.jpg)
Rule #10
Better example
1. large: big :: triumph : ___________
A. lossB. ruleC. smallD. success
Seinfeld teaches History - Cooperative Learning
![Page 45: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/45.jpg)
The Work
Look at what you will be teaching from Jan. 6th to Feb. 1st
Focus on questions that address the standards
Assign questions: DOK Tag Standard/Element Tag
Create a bank of 30-35 questions (80% at DOK 2 or 3)
![Page 46: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/46.jpg)
Question FormatDOK1 GPS Type Question Stem Here
A. Type Answer Choices Here
B. Type Answer Choices Here
C. Type Answer Choices Here
D. Type Answer Choices Here
1. Indicate DOK and Standard/Element
2. Leave Column Blank 3. Leave
Row Blank
![Page 47: Question Development](https://reader035.vdocument.in/reader035/viewer/2022062520/56816439550346895dd60166/html5/thumbnails/47.jpg)
Resources
www.edinformatics.com Released questions from standardized
state tests across the nationhttp://vhsprofessionallearning.wikisp
aces.com/ PowerPoint 5 Step Protocol Template for Question Banks