observation, measurement, and data analysis in per: methodological issues and challenges
DESCRIPTION
Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges. David E. Meltzer Department of Physics and Astronomy Iowa State University Ames, Iowa Supported in part by National Science Foundation grants DUE #9981140 and REC #0206683. Collaborators - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/1.jpg)
Observation, Measurement, and Data Analysis in PER:
Methodological Issues and Challenges
David E. Meltzer
Department of Physics and Astronomy
Iowa State University
Ames, Iowa
Supported in part by National Science Foundation grants DUE #9981140 and REC #0206683
![Page 2: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/2.jpg)
Collaborators
Tom Greenbowe (Department of Chemistry, ISU)
Mani K. Manivannan(Southwest Missouri State University)
Graduate Students
Jack Dostal (ISU/Montana State)
Ngoc-Loan NguyenTina FanettiLarry EngelhardtWarren Christensen
![Page 3: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/3.jpg)
Outline
• Observing Instruments
![Page 4: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/4.jpg)
Outline
• Observing Instruments
• Issues related to assessment of an individual student– context dependence of students’ ideas
– multidimensionality of student mental states (“models”)
– time dependence (rate of change) of students’ thinking
![Page 5: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/5.jpg)
Outline
• Observing Instruments
• Issues related to assessment of an individual student– context dependence of students’ ideas
– multidimensionality of student mental states (“models”)
– time dependence (rate of change) of students’ thinking
• Measures of learning gain (g, d, etc.)
![Page 6: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/6.jpg)
Outline
• Observing Instruments
• Issues related to assessment of an individual student– context dependence of students’ ideas– multidimensionality of student mental states (“models”)– time dependence (rate of change) of students’ thinking
• Measures of learning gain (g, d, etc.)
• Issues related to assessment of many students– “hidden variables” in students’ pre-instruction state– sample-selection bias
![Page 7: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/7.jpg)
Outline• Observing Instruments
• Issues related to assessment of an individual student– context dependence of students’ ideas– multidimensionality of student mental states (“models”)– time dependence (rate of change) of students’ thinking
• Measures of learning gain (g, d, etc.)
• Issues related to assessment of many students– “hidden variables” in students’ pre-instruction state– sample-selection bias
• Dynamic Assessment (Time-dependent assessment)
![Page 8: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/8.jpg)
Tools of Physics Education Research
• Conceptual surveys or “diagnostics”: short-answer or multiple-choice questions emphasizing qualitative understanding, e.g., FCI, MBT, FMCE, CSEM, etc.
![Page 9: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/9.jpg)
Tools of Physics Education Research
• Conceptual surveys or “diagnostics”: short-answer or multiple-choice questions emphasizing qualitative understanding, e.g., FCI, MBT, FMCE, CSEM, etc.
• Students’ written explanations of their reasoning
![Page 10: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/10.jpg)
Tools of Physics Education Research
• Conceptual surveys or “diagnostics”: short-answer or multiple-choice questions emphasizing qualitative understanding, e.g., FCI, MBT, FMCE, CSEM, etc.
• Students’ written explanations of their reasoning
• Interviews with studentse.g. “individual demonstration interviews” (U. Wash.)
![Page 11: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/11.jpg)
Tools of Physics Education Research
• Conceptual surveys or “diagnostics”: short-answer or multiple-choice questions emphasizing qualitative understanding, e.g., FCI, MBT, FMCE, CSEM, etc.
• Students’ written explanations of their reasoning
• Interviews with studentse.g. “individual demonstration interviews” (U. Wash.)
• Observations of student group interactions
![Page 12: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/12.jpg)
Observations of Student Group Interactions
• Very time consuming– real-time observation and/or recording
![Page 13: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/13.jpg)
Observations of Student Group Interactions
• Very time consuming– real-time observation and/or recording
• Identify more fruitful and less fruitful student group behaviors e.g. R. Thornton, PERC 2001
![Page 14: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/14.jpg)
Observations of Student Group Interactions
• Very time consuming– real-time observation and/or recording
• Identify more fruitful and less fruitful student group behaviors e.g. R. Thornton, PERC 2001
• Characterize student-technology interactions e.g. V. Otero, PERC 2001; E. George, M. J. Broadstock, and J. Vasquez-Abad, PERC 2001
![Page 15: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/15.jpg)
Observations of Student Group Interactions
• Very time consuming– real-time observation and/or recording
• Identify more fruitful and less fruitful student group behaviors e.g. R. Thornton, PERC 2001
• Characterize student-technology interactions e.g. V. Otero, PERC 2001; E. George, M. J. Broadstock, and J. Vasquez-Abad, PERC 2001
• Identify productive instructor interventions e.g. D. MacIsaac and K. Falconer, 2002
![Page 16: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/16.jpg)
Caution: Careful probing needed!
• It is very easy to overestimate students’ level of understanding.
![Page 17: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/17.jpg)
Caution: Careful probing needed!
• It is very easy to overestimate students’ level of understanding.
• Students frequently give correct responses based on incorrect reasoning.
![Page 18: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/18.jpg)
Caution: Careful probing needed!
• It is very easy to overestimate students’ level of understanding.
• Students frequently give correct responses based on incorrect reasoning.
• Students’ written explanations of their reasoning, and interviews with students, are indispensable diagnostic tools.
![Page 19: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/19.jpg)
Ignoring Students’ Explanations Affects both Validity and Reliability
.
![Page 20: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/20.jpg)
T. O’Brien Pride, S. Vokos, and L. C. McDermott, Am. J. Phys. 66, 147 (1998)
Posttest Variant #1N = 435
[comparison of KE and p, two objects different mass, acted upon by same force]
(t=const.) (x, tconst.)
kinetic energy comparisonmomentum comparison
Posttest Variant #2N = 320
Ignoring Students’ Explanations Affects both Validity and Reliability
![Page 21: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/21.jpg)
T. O’Brien Pride, S. Vokos, and L. C. McDermott, Am. J. Phys. 66, 147 (1998)
Posttest Variant #1N = 435
[comparison of KE and p, two objects different mass, acted upon by same force]
Correct answer, correct
explanation
(t=const.)
Correct answer, correct
explanation
(x, tconst.)
kinetic energy comparison 35% 30%momentum comparison
Posttest Variant #2N = 320
Ignoring Students’ Explanations Affects both Validity and Reliability
![Page 22: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/22.jpg)
T. O’Brien Pride, S. Vokos, and L. C. McDermott, Am. J. Phys. 66, 147 (1998)
Posttest Variant #1N = 435
[comparison of KE and p, two objects different mass, acted upon by same force]
Correct answer, correct
explanation
(t=const.)
Correct answer, correct
explanation
(x, tconst.)
kinetic energy comparison 35% 30%momentum comparison 50% 45%
Posttest Variant #2N = 320
Consistent results when explanations taken into account
Ignoring Students’ Explanations Affects both Validity and Reliability
![Page 23: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/23.jpg)
T. O’Brien Pride, S. Vokos, and L. C. McDermott, Am. J. Phys. 66, 147 (1998)
Posttest Variant #1N = 435
[comparison of KE and p, two objects different mass, acted upon by same force]
Correct answer, correct
explanation
(t=const.)
Correct answer,
explanation ignored
(t=const.)
Correct answer, correct
explanation
(x, tconst.)
Correct answer,
explanation ignored
(x, tconst.)
kinetic energy comparison 35% 65% 30% 45%momentum comparison 50% 45%
Posttest Variant #2N = 320
Consistent results when explanations taken into account
Ignoring Students’ Explanations Affects both Validity and Reliability
![Page 24: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/24.jpg)
T. O’Brien Pride, S. Vokos, and L. C. McDermott, Am. J. Phys. 66, 147 (1998)
Posttest Variant #1N = 435
[comparison of KE and p, two objects different mass, acted upon by same force]
Correct answer, correct
explanation
(t=const.)
Correct answer,
explanation ignored
(t=const.)
Correct answer, correct
explanation
(x, tconst.)
Correct answer,
explanation ignored
(x, tconst.)
kinetic energy comparison 35% 65% 30% 45%momentum comparison 50% 80% 45% 55%
Posttest Variant #2N = 320
Consistent results when explanations taken into account
Ignoring Students’ Explanations Affects both Validity and Reliability
![Page 25: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/25.jpg)
Context Dependence
• physical context– minor variations in “surface features,” e.g., soccer ball
instead of golf ball
![Page 26: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/26.jpg)
Context Dependence
• physical context– minor variations in “surface features,” e.g., soccer ball
instead of golf ball
• form of question– e.g., free-response or multiple-choice
![Page 27: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/27.jpg)
Context Dependence
• physical context– minor variations in “surface features,” e.g., soccer ball
instead of golf ball
• form of question– e.g., free-response or multiple-choice
• mode of representation– verbal (words), graphs, diagrams, equations
![Page 28: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/28.jpg)
Context Dependence
• physical context– minor variations in “surface features,” e.g., soccer ball
instead of golf ball
• form of question– e.g., free-response or multiple-choice
• mode of representation– verbal (words), graphs, diagrams, equations
• physical system – vary physical elements and/or form of interaction
e.g., car pushes truck vs. ice-skater collision
![Page 29: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/29.jpg)
Context Dependence of Student Responses
• Changing physical context may significantly alter students’ responses– E.g., FCI #13, forces on steel ball thrown straight up. When
changed to “vertical pistol shot,” many who originally included upward force in direction of motion changed to correct response (gravity only). H. Schecker and J. Gerdes, Zeitschrift für Didaktik der Naturwissenschaften 5, 75 (1999).
![Page 30: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/30.jpg)
Context Dependence of Student Responses
• Changing physical context may significantly alter students’ responses– E.g., FCI #13, forces on steel ball thrown straight up. When
changed to “vertical pistol shot,” many who originally included upward force in direction of motion changed to correct response (gravity only). H. Schecker and J. Gerdes, Zeitschrift für Didaktik der Naturwissenschaften 5, 75 (1999).
• Changing form of question may significantly alter students’ responses– E.g., free-response final-exam problems similar to several FCI
posttest questions. In some cases, significant differences in percent correct responses among students who took both tests. R. Steinberg and M. Sabella, Physics Teacher 35, 150 (1997).
![Page 31: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/31.jpg)
Different Results with Different Representations
Example: Elementary Physics Course at Southeastern Louisiana University. (DEM and K. Manivannan, 1998)
![Page 32: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/32.jpg)
Different Results with Different Representations
Example: Elementary Physics Course at Southeastern Louisiana University. (DEM and K. Manivannan, 1998)
Newton’s second-law questions from FMCE. (nearly identical questions posed in graphical, and “natural language” form.)
![Page 33: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/33.jpg)
E.
F.
G.
The force is toward the left and is decreasing in strength (magnitude).
The force is toward the left and is of constant strength (magnitude).
The force is toward the left and is increasing in strength (magnitude).
Direction of Force
A.
B.
C.
The force is toward the right and is increasing in strength (magnitude).The force is toward the right and is of constant strength (magnitude).
The force is toward the right and is decreasing in strength (magnitude).
Direction of Force
D. No applied force is needed
1. 1. Which force would keep the sled moving toward the right and speeding up at a steady rate (constant acceleration)?
2. 2. Which force would keep the sled moving toward the right at a steady (constant) velocity?
3. 3. The sled is moving toward the right. Which force would slow it down at a steady rate (constant acceleration)?
4. 4. Which force would keep the sled moving toward the left and speeding up at a steady rate (constant acceleration)?
R. Thornton and D. Sokoloff, Am. J. Phys. 66, 38 (1998)
![Page 34: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/34.jpg)
Time
F o r c e
+
-
0Time
F o r c e
+
-
0Time
F o r c e
+
-
0Time
F o r c e
+
-
0Time
F o r c e
+
-
0 Time
A B C D E F G F
o r c e
Time
F o r c e
+
-
0
Questions 14-21 refer to a toy car which can move to the right or left along a horizontal line (the positive part of the distance axis).
+0Assume that friction is so small that it can be ignored.
You may use a choice more than once or not at all. If you think that none is correct, answer choice .J
A force is applied to the car. Choose the one force graph ( through ) for each statement below which could allow the described motion of the car to continue.
A H
The car moves toward the right (away from the origin) with a steady (constant) velocity.
__14.
The car is at rest.__15.
The car moves toward the right and is speeding up at a steady rate (constant acceleration).
__16.
The car moves toward the left (toward the origin) with a steady (constant) velocity.
__17.
The car moves toward the right and is slowing down at a steady rate (constant acceleration).
__18.
The car moves toward the left and is speeding up at a steady rate (constant acceleration).
__19.
The car moves toward the right, speeds up and then slows down.
__20.
The car was pushed toward the right and then released. Which graph describes the force after the car is released.
__21.
None of these graphs is correct.J
F o r c e
+
-
0 Time
H
+
-
0
![Page 35: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/35.jpg)
Different Results with Different Representations
Example: Elementary Physics Course at Southeastern Louisiana University. (DEM and K. Manivannan, 1998)
Newton’s second-law questions from FMCE. (nearly identical questions posed in graphical, and “natural language” form.)
Posttest (N = 18)
“force graph” questions: 56%
“natural language” questions*: 28%
*Force Sled Questions #1-4
![Page 36: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/36.jpg)
Warning Just because you saw it once does not
necessarily mean you’ll see it the next time
![Page 37: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/37.jpg)
Warning Just because you saw it once does not
necessarily mean you’ll see it the next time
• Class averages on full sets of test items tend to be very stable, one measurement to the next (e.g., different year)
• Measurements on individual test items fluctuate significantly
![Page 38: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/38.jpg)
Warning Just because you saw it once does not
necessarily mean you’ll see it the next time
• Class averages on full sets of test items tend to be very stable, one measurement to the next (e.g., different year)
• Measurements on individual test items fluctuate significantlyExample: Algebra-based physics, male students at ISU, FCI #29Original: forces acting on office chair at rest on floor [no graphic]Variant (“Gender FCI”; L. McCullough): forces acting on diary at rest on nightstand [drawing of system is shown]
FCI #29original version
% correctvariant
% correctsignificance
Spring 2001 30% (n = 69) 60% (n = 65) p = 0.0005
![Page 39: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/39.jpg)
Warning Just because you saw it once does not
necessarily mean you’ll see it the next time
• Class averages on full sets of test items tend to be very stable, one measurement to the next (e.g., different year)
• Measurements on individual test items fluctuate significantlyExample: Algebra-based physics, male students at ISU, FCI #29Original: forces acting on office chair at rest on floor [no graphic]Variant (“Gender FCI”; L. McCullough): forces acting on diary at rest on nightstand [drawing of system is shown]
FCI #29original version
% correctvariant
% correctsignificance
Spring 2001 30% (n = 69) 60% (n = 65) p = 0.0005
Fall 2001 40% (n = 55) 37% (n = 46) n.s.
![Page 40: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/40.jpg)
Warning Just because you saw it once does not
necessarily mean you’ll see it the next time
• Class averages on full sets of test items tend to be very stable, one measurement to the next (e.g., different year)
• Measurements on individual test items fluctuate significantlyExample: Algebra-based physics, male students at ISU, FCI #29Original: forces acting on office chair at rest on floor [no graphic]Variant (“Gender FCI”; L. McCullough): forces acting on diary at rest on nightstand [drawing of system is shown]
FCI #29original version
% correctvariant
% correctsignificance
Spring 2001 30% (n = 69) 60% (n = 65) p = 0.0005
Fall 2001 40% (n = 55) 37% (n = 46) n.s.
Replication is important, especially for surprising results
![Page 41: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/41.jpg)
Superposition of Mental States
• Students tend to be inconsistent in applying same concept in different situations, implying existence of “mixed-model” mental state. E.g., use “impetus” model in one case, Newtonian model on
another. I. Halloun and D. Hestenes, Am. J. Phys. 53, 1058 (1985).
![Page 42: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/42.jpg)
Superposition of Mental States
• Students tend to be inconsistent in applying same concept in different situations, implying existence of “mixed-model” mental state. E.g., use “impetus”
model in one case, Newtonian model on another. I. Halloun and D. Hestenes, Am. J. Phys. 53, 1058 (1985).
• Time-dependent changes in degree of consistency of students’ mental states may correlate with distinct learning patterns with different physical concepts. E.g., students learn to recognize presence of
normal force, but still believe in “force in direction of motion.” L. Bao and E. F. Redish, PERS of AJP 69, S45 (2001).
![Page 43: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/43.jpg)
Issues Related to Multiple-Choice ExamsCf. N. S. Rebello and D. A. Zollman, PERS of AJP (in press)
.
![Page 44: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/44.jpg)
Issues Related to Multiple-Choice ExamsCf. N. S. Rebello and D. A. Zollman, PERS of AJP (in press)
• Even well-validated multiple-choice questions may miss significant categories of responses.
![Page 45: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/45.jpg)
Issues Related to Multiple-Choice ExamsCf. N. S. Rebello and D. A. Zollman, PERS of AJP (in press)
• Even well-validated multiple-choice questions may miss significant categories of responses.
• Selection of distracters made available to students can significantly affect proportion of correct responses.
![Page 46: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/46.jpg)
Issues Related to Multiple-Choice ExamsCf. N. S. Rebello and D. A. Zollman, PERS of AJP (in press)
• Even well-validated multiple-choice questions may miss significant categories of responses.
• Selection of distracters made available to students can significantly affect proportion of correct responses.
• As a result of instruction, new misconceptions may arise that are not matched to original set of distracters.
![Page 47: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/47.jpg)
Deciphering Students’ Mental Models from their Exam Responses
.
![Page 48: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/48.jpg)
Deciphering Students’ Mental Models from their Exam Responses
• Distinct patterns of incorrect responses may
correlate to transitional mental states. R. Thornton, ICUPE Proceedings (1997)
![Page 49: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/49.jpg)
Deciphering Students’ Mental Models from their Exam Responses
• Distinct patterns of incorrect responses may
correlate to transitional mental states. R. Thornton, ICUPE Proceedings (1997)
• Varying the selection of answer options can alter
the models ascribed to students’ thinking. R. Dufresne, W. Leonard, and W. Gerace, Physics Teacher 40, 174 (2002).
![Page 50: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/50.jpg)
Deciphering Students’ Mental Models from their Exam Responses
• Distinct patterns of incorrect responses may correlate to transitional mental states. R. Thornton, ICUPE Proceedings (1997)
• Varying the selection of answer options can alter the models ascribed to students’ thinking. R. Dufresne, W. Leonard, and W. Gerace, Physics Teacher 40, 174 (2002).
• Students’ justifications for incorrect responses may change as a result of instruction. J. Adams and T. Slater (1997)
![Page 51: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/51.jpg)
Deciphering Students’ Mental Models from their Exam Responses
• Distinct patterns of incorrect responses may correlate to transitional mental states. R. Thornton, ICUPE Proceedings (1997)
• Varying the selection of answer options can alter the models ascribed to students’ thinking. R. Dufresne, W. Leonard, and W. Gerace, Physics Teacher 40, 174 (2002).
• Students’ justifications for incorrect responses may change as a result of instruction. J. Adams and T. Slater (1997)
• Precision design of questions and answer options necessary for accurate targeting of students’ mental models. Bao and Redish, PERS of AJP 69, S45 (2001); Bao, Hogg, and Zollman, AJP, 70, 772 (2002).
![Page 52: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/52.jpg)
D. Maloney, T. O’Kuma, C. Hieggelke, and A. Van Heuvelen, PERS of AJP 69, S12 (2001).
![Page 53: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/53.jpg)
![Page 54: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/54.jpg)
Hypothetical Student Models on Relation Between Electric Field and Equipotential Lines
• Model 1 [correct]: field stronger where lines closer together. Responses: #1: D #2: B or D
• Model 2: field stronger where lines farther apart
Responses: #1: C #2: A or C
• Model 3: field stronger where potential is higherResponses: #1: E #2: A or C
• Model 4: Mixed models, all other responses
![Page 55: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/55.jpg)
Evolution of Student Models Algebra-based physics at ISU (1998-2001)
n = 299 Pre-test Post-test
Model #1 20% 63%
Model #2 14% 2%
Model #3 9% 8%
Model #4 57% 27%
[disappears]
[remains]
![Page 56: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/56.jpg)
Caution: Models much less firm than they may appear
Spring 2002: 116 Students in same course gave answers pre-instruction with explanations to the two questions.
n explanation consistent with model
Model #1 15 5 (33%)
Model #2 19 2 (11%)
Model #3 21 7 (33%)
Model #4 61 —
Patterns of student thinking that seemed to be present on pretest may actually have been largely random.
![Page 57: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/57.jpg)
Interview Evidence of Students’ Mental State-Function
.
![Page 58: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/58.jpg)
Interview Evidence of Students’ Mental State-Function
• Initially, students may offer largely formulaic responses e.g., equations, verbatim repetition of phrases, etc.
![Page 59: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/59.jpg)
Interview Evidence of Students’ Mental State-Function
• Initially, students may offer largely formulaic responses e.g., equations, verbatim repetition of phrases, etc.
• Later responses may contradict earlier ones; sometimes resolvable by student, sometimes not. Sometimes they have no well-defined concept.
![Page 60: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/60.jpg)
Interview Evidence of Students’ Mental State-Function
• Initially, students may offer largely formulaic responses e.g., equations, verbatim repetition of phrases, etc.
• Later responses may contradict earlier ones; sometimes resolvable by student, sometimes not. Sometimes they have no well-defined concept.
• Even with minimum-intensity probing, students may in time succeed in solving problem that was initially intractable.
![Page 61: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/61.jpg)
Interview Evidence of Students’ Mental State-Function
• Initially, students may offer largely formulaic responses e.g., equations, verbatim repetition of phrases, etc.
• Later responses may contradict earlier ones; sometimes resolvable by student, sometimes not. Sometimes they have no well-defined concept.
• Even with minimum-intensity probing, students may in time succeed in solving problem that was initially intractable.
If student “learns” during interview, have we measured “knowledge” or “learning ability”?
![Page 62: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/62.jpg)
Time Dependence of Student Learning
.
![Page 63: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/63.jpg)
Time Dependence of Student Learning
• Multi-dimensionality of student mental states (i.e., diversity of individual model states) suggests possible correlations with diverse learning trajectories and learning rates.
![Page 64: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/64.jpg)
Time Dependence of Student Learning
• Multi-dimensionality of student mental states (i.e., diversity of individual model states) suggests possible correlations with diverse learning trajectories and learning rates.
• Can initial learning rate be correlated with final learning gains? Ambiguous results so far. (DEM, 1997)
![Page 65: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/65.jpg)
Time Dependence of Student Learning
• Multi-dimensionality of student mental states (i.e., diversity of individual model states) suggests possible correlations with diverse learning trajectories and learning rates.
• Can initial learning rate be correlated with final learning gains? Ambiguous results so far. (DEM, 1997)
• To date there has been little focus on assessing physics students’ learning rates.
![Page 66: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/66.jpg)
Measures of Learning Gain
.
![Page 67: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/67.jpg)
Measures of Learning Gain
• Single exam measures only instantaneous knowledge state, but instructors are interested in improving learning, i.e., transitions between states.
![Page 68: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/68.jpg)
Measures of Learning Gain
• Single exam measures only instantaneous knowledge state, but instructors are interested in improving learning, i.e., transitions between states.
• Need a measure of learning gain that has maximum dependence on instruction, and minimum dependence on students’ pre-instruction state.
![Page 69: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/69.jpg)
Measures of Learning Gain
• Single exam measures only instantaneous knowledge state, but instructors are interested in improving learning, i.e., transitions between states.
• Need a measure of learning gain that has maximum dependence on instruction, and minimum dependence on students’ pre-instruction state.
search for measure that is correlated with instructional activities, but has minimum correlation with pretest scores.
![Page 70: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/70.jpg)
Normalized Learning Gain “g”R. R. Hake, Am. J. Phys. 66, 64 (1998)
In a study of 62 mechanics courses enrolling over 6500 students, Hake found that mean normalized gain <g> on the FCI is:
• virtually independent of class mean pretest score (r = +0.02);
gainpossiblemaximum
gain
scorepretestscoremaximum
scorepretestscoreposttestg
![Page 71: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/71.jpg)
Normalized Learning Gain “g”R. R. Hake, Am. J. Phys. 66, 64 (1998)
In a study of 62 mechanics courses enrolling over 6500 students, Hake found that mean normalized gain <g> on the FCI is:
• virtually independent of class mean pretest score (r = +0.02);
• = 0.230.04(s.d.) for traditional instruction, nearly independent of instructor;
• =0.480.14(s.d.) for courses employing “interactive engagement” active-learning instruction.
These findings have been largely confirmed in hundreds of physics courses worldwide
gainpossiblemaximum
gain
scorepretestscoremaximum
scorepretestscoreposttestg
![Page 72: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/72.jpg)
Effect Size d: Measure of Non-Overlap
deviationstandardsquaremean
scorepretestmeanscoreposttestmeand
![Page 73: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/73.jpg)
Effect Size d: Measure of Non-Overlap
Small Effect Size
Score
Nu
mb
er
of
stu
de
nts
Pretest
Posttest
Large Effect Size
Score
Nu
mb
er
of
stu
de
nts
Pretest
Posttest
pretest
posttest
deviationstandardsquaremean
scorepretestmeanscoreposttestmeand
![Page 74: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/74.jpg)
Effect Size d: Measure of Non-Overlap
Small Effect Size
Score
Nu
mb
er
of
stu
de
nts
Pretest
Posttest
Large Effect Size
Score
Nu
mb
er
of
stu
de
nts
Pretest
Posttest
pretest
posttest
deviationstandardsquaremean
scorepretestmeanscoreposttestmeand
Large effect size does notnecessarily imply “significant” gain!
![Page 75: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/75.jpg)
But is g really independent of pre-instruction state?
Possible “hidden variables” in students’ pre-instruction mental state:
![Page 76: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/76.jpg)
But is g really independent of pre-instruction state?
Possible “hidden variables” in students’ pre-instruction mental state:
• mathematical skill R. Hake et al., 1994; M. Thoresen and C. Gross, 2000; D. Meltzer, PERS of AJP (in press)
![Page 77: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/77.jpg)
But is g really independent of pre-instruction state?
Possible “hidden variables” in students’ pre-instruction mental state:
• mathematical skill R. Hake et al., 1994; M. Thoresen and C. Gross, 2000; D. Meltzer, PERS of AJP (in press)
• spatial visualization ability R. Hake 2002
![Page 78: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/78.jpg)
But is g really independent of pre-instruction state?
Possible “hidden variables” in students’ pre-instruction mental state:
• mathematical skill R. Hake et al., 1994; M. Thoresen and C.
Gross, 2000; D. Meltzer, PERS of AJP (in press)
• spatial visualization ability R. Hake 2002
• gender L. McCullough 2000; R. Hake 2002
![Page 79: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/79.jpg)
But is g really independent of pre-instruction state?
Possible “hidden variables” in students’ pre-instruction mental state:
• mathematical skill R. Hake et al., 1994; M. Thoresen and C. Gross, 2000; D. Meltzer, PERS of AJP (in press)
• spatial visualization ability R. Hake 2002
• gender L. McCullough 2000; R. Hake 2002
• reasoning ability J. M. Clement, 2002
![Page 80: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/80.jpg)
But is g really independent of pre-instruction state?
Possible “hidden variables” in students’ pre-instruction mental state:
• mathematical skill R. Hake et al., 1994; M. Thoresen and C. Gross, 2000; D. Meltzer, PERS of AJP (in press)
• spatial visualization ability R. Hake 2002
• gender L. McCullough 2000; R. Hake 2002
• reasoning ability J. M. Clement, 2002
and even pretest score!? C. Henderson, K. Heller, and P. Heller, 1999
![Page 81: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/81.jpg)
Normalized Gain vs. CSE Pretest Score (ISU 1998)
0.0
0.2
0.4
0.6
0.8
1.0
0 15 30 45 60 75
CSE Pretest Score (% correct)
No
rmal
ized
Gai
n "
g"
r = 0.0
![Page 82: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/82.jpg)
Distribution of Gains: ISU 1998(high and low CSE pretest scores)
02468
10
0.00
-0.0
9
0.1
0-0.
19
0.20
-0.2
9
0.30
-0.3
9
0.40
-0.4
9
0.50
-0.5
9
0.60
-0.6
9
0.70
-0.7
9
0.80
-0.8
9
0.90
-1.0
0
g
# st
ud
ents Bottom half CSE
pretest scoresTop half CSEpretest scores
![Page 83: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/83.jpg)
Distribution of Gains: ISU 1998(high and low CSE pretest scores)
02468
10
0.00
-0.0
9
0.1
0-0.
19
0.20
-0.2
9
0.30
-0.3
9
0.40
-0.4
9
0.50
-0.5
9
0.60
-0.6
9
0.70
-0.7
9
0.80
-0.8
9
0.90
-1.0
0
g
# st
ud
ents
Bottom half CSEpretest scores
Top half CSEpretest scores
![Page 84: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/84.jpg)
Normalized Gain vs. Math Pretest(ISU 1998)
0.0
0.2
0.4
0.6
0.8
1.0
0 10 20 30 40
Math Pretest Score (Max = 38)
No
rma
lize
d G
ain
"g
" r = +0.46p = 0.0002
![Page 85: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/85.jpg)
Distribution of Gains: ISU 1998(high and low math pretest scores)
02468
10
0.00
-0.0
9
0.1
0-0.
19
0.20
-0.2
9
0.30
-0.3
9
0.40
-0.4
9
0.50
-0.5
9
0.60
-0.6
9
0.70
-0.7
9
0.80
-0.8
9
0.90
-1.0
0g
# st
uden
ts Bottom half mathpretest scores
![Page 86: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/86.jpg)
Distribution of Gains: ISU 1998(high and low math pretest scores)
02468
10
0.00
-0.0
9
0.1
0-0.
19
0.20
-0.2
9
0.30
-0.3
9
0.40
-0.4
9
0.50
-0.5
9
0.60
-0.6
9
0.70
-0.7
9
0.80
-0.8
9
0.90
-1.0
0g
# st
uden
ts
Bottom half mathpretest scoresTop half mathpretest scores
![Page 87: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/87.jpg)
Sample-Selection Bias
.
![Page 88: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/88.jpg)
Sample-Selection Bias
• self-selection factor in interview samples– interviewees tend to be above-average students
![Page 89: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/89.jpg)
Sample-Selection Bias
• self-selection factor in interview samples– interviewees tend to be above-average students
• biasing due to student availability– students attending recitations may have above-
average grades
![Page 90: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/90.jpg)
Sample-Selection Bias
• self-selection factor in interview samples– interviewees tend to be above-average students
• biasing due to student availability– students attending recitations may have above-
average grades
• spring semester/fall semester differences– possible tendency for off-sequence courses to
attract better-prepared students
![Page 91: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/91.jpg)
0
5
10
15
20
25
30
Total Grade Points
Perc
enta
ge o
f Sam
ple
Full Class, N = 424, median grade = 261
Interview Sample, N = 32, median grade = 305
Grade Distributions: Interview Sample vs. Full Class
(DEM, 2002)
![Page 92: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/92.jpg)
0
5
10
15
20
25
30
Total Grade Points
Per
cent
age
of S
ampl
eFull Class, N = 424, median grade = 261
Interview Sample, N = 32, median grade = 305
Interview Sample:34% above 91st percentile; 50% above 81st percentile
Grade Distributions: Interview Sample vs. Full Class
(DEM, 2002)
![Page 93: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/93.jpg)
Grade Comparison: Students attending recitation vs. All students
J. Dostal and DEM, 2000
N Scores on Exam #1 [before using worksheets]
Students using special worksheets
(ALL attended recitation session)
129 69% (std.dev. = 20%)
![Page 94: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/94.jpg)
Grade Comparison: Students attending recitation vs. All students
J. Dostal and DEM, 2000
N Scores on Exam #1 [before using worksheets]
Students using special worksheets
(ALL attended recitation session)
129 69% (std.dev. = 20%)
Students not using special worksheets
(MOST attended recitation session)
325 65% (std. dev. = 18%)
Difference of 4% is statistically significant (p < 0.05)
(Same difference found on final exam on “non-worksheet” questions)
![Page 95: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/95.jpg)
Score Comparison on Vector Concept Quiz fall-semester courses vs. spring-semester courses
N. L. Nguyen and DEM, PERS of AJP (in press)
Algebra-based physics: A-I (mechanics); A-II (E&M)Calculus-based physics: C-I (mechanics); C-II (E&M, thermo, optics)
(Quiz given during first week of class)
![Page 96: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/96.jpg)
![Page 97: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/97.jpg)
Score Comparison on Vector Concept Quiz fall-semester courses vs. spring-semester courses
N. L. Nguyen and DEM, PERS of AJP (in press)
Algebra-based physics: A-I (mechanics); A-II (E&M)Calculus-based physics: C-I (mechanics); C-II (E&M, thermo, optics)
(Quiz given during first week of class)
Fall Spring
A-I 44% (n = 287) 51%* (n = 233) p < 0.001
C-I 65% (n = 192) 74%* (n = 416) p = 0.0003
* “off-sequence” course
![Page 98: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/98.jpg)
Score Comparison on Vector Concept Quiz fall-semester courses vs. spring-semester courses
N. L. Nguyen and DEM, PERS of AJP (in press)
Algebra-based physics: A-I (mechanics); A-II (E&M)Calculus-based physics: C-I (mechanics); C-II (E&M, thermo, optics)
(Quiz given during first week of class)
Fall Spring
A-I 44% (n = 287) 51%* (n = 233) p < 0.001
C-I 65% (n = 192) 74%* (n = 416) p = 0.0003
A-II 63%* (n = 83) 61% (n = 118)not significant
C-II 83%* (n = 313) 78% (n = 389)p < 0.01
* “off-sequence” course
![Page 99: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/99.jpg)
Fundamental Quandary: Assessment of “Knowledge” or “Learning”?
(To analyze motion of particle, initial position and momentum required. And to analyze student understanding? . . .)
![Page 100: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/100.jpg)
Fundamental Quandary: Assessment of “Knowledge” or “Learning”?
(To analyze motion of particle, initial position and momentum required. And to analyze student understanding? . . .)
• To assess the impact of the teaching environment, we examine students “before” and “after.” How do we measure magnitude of learning effect?
![Page 101: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/101.jpg)
Fundamental Quandary: Assessment of “Knowledge” or “Learning”?
(To analyze motion of particle, initial position and momentum required. And to analyze student understanding? . . .)
• To assess the impact of the teaching environment, we examine students “before” and “after.” How do we measure magnitude of learning effect?
• Two students at same instantaneous knowledge “point” may be following very different trajectories. How can they be distinguished?
![Page 102: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/102.jpg)
Fundamental Quandary: Assessment of “Knowledge” or “Learning”?
(To analyze motion of particle, initial position and momentum required. And to analyze student understanding? . . .)
• To assess the impact of the teaching environment, we examine students “before” and “after.” How do we measure magnitude of learning effect?
• Two students at same instantaneous knowledge “point” may be following very different trajectories. How can they be distinguished?
(Imagine ensemble of points representing individual students’ mental state-functions. The trajectory of the ensemble is influenced by the teaching “force field,” but also depends on initial momentum distribution.)
![Page 103: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/103.jpg)
“Dynamic” Assessment?Cf. C. S. Lidz, Dynamic Assessment (Guilford, New York, 1987)
.
![Page 104: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/104.jpg)
“Dynamic” Assessment?Cf. C. S. Lidz, Dynamic Assessment (Guilford, New York, 1987)
• Even within the time period of a single interview, a student’s mental state may vary significantly.– random fluctuation, or secular change?
![Page 105: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/105.jpg)
“Dynamic” Assessment?Cf. C. S. Lidz, Dynamic Assessment (Guilford, New York, 1987)
• Even within the time period of a single interview, a student’s mental state may vary significantly.– random fluctuation, or secular change?
• Full description of mental state function requires dynamical information, i.e., rates of change, reaction to instructional “perturbation,” etc. (and remember student state-function is multi-dimensional!)
![Page 106: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/106.jpg)
“Dynamic” Assessment?Cf. C. S. Lidz, Dynamic Assessment (Guilford, New York, 1987)
• Even within the time period of a single interview, a student’s mental state may vary significantly.– random fluctuation, or secular change?
• Full description of mental state function requires dynamical information, i.e., rates of change, reaction to instructional “perturbation,” etc. (and remember student state-function is multi-dimensional!)
• Full analysis of teaching/learning environment will require broad array of interaction parameters.
![Page 107: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/107.jpg)
“Dynamic” Assessment?Cf. C. S. Lidz, Dynamic Assessment (Guilford, New York, 1987)
• Even within the time period of a single interview, a student’s mental state may vary significantly.– random fluctuation, or secular change?
• Full description of mental state function requires dynamical information, i.e., rates of change, reaction to instructional “perturbation,” etc. (and remember student state-function is multi-dimensional!)
• Full analysis of teaching/learning environment will require broad array of interaction parameters.
• Simplification a practical necessity (just as in all other physics research!), but can’t lose sight of underlying reality.
![Page 108: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/108.jpg)
Conclusion
.
![Page 109: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/109.jpg)
Conclusion
• “Detector” design for data collection in PER has just begun to scratch the surface.
![Page 110: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/110.jpg)
Conclusion
• “Detector” design for data collection in PER has just begun to scratch the surface.
• We need to improve identification and control of variables.
![Page 111: Observation, Measurement, and Data Analysis in PER: Methodological Issues and Challenges](https://reader033.vdocument.in/reader033/viewer/2022051416/568134df550346895d9c12e5/html5/thumbnails/111.jpg)
Conclusion
• “Detector” design for data collection in PER has just begun to scratch the surface.
• We need to improve identification and control of variables.
• Dynamic, time-dependent assessment is likely to increase in importance.