collecting and analyzing data: measuring student success rebecca orr, ph.d. professor of biology
TRANSCRIPT
![Page 1: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/1.jpg)
COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS
Rebecca Orr, Ph.D.
Professor of Biology
![Page 2: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/2.jpg)
Agenda• Efforts made to improve student success:
• Online homework using MasteringBiology• Peer tutoring• Online quizzing •
• Strategies for data collection:• Preparing to begin• Modifying as collection progresses• Involving your students
• Case study: Impact of pre-exam quizzing on student outcomes
• Lessons learned
![Page 3: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/3.jpg)
Student Success Challenges
• Have you ever heard: • “I studied so hard, and it just didn’t work!”
• My question: • How can student success be improved in the introductory biology (majors) course?
![Page 4: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/4.jpg)
My Evolution of Mastering Use
• Began using Mastering to create assignments and made them available for optional student use. • Student feedback indicated a trend:
• Successful students were using Mastering • Course evaluation surveys by students said:
“Make Mastering mandatory!”• Less successful students did not tend to utilize
Mastering
• Moved to making Mastering mandatory by assigning a portion of lecture credit (10%) to Mastering.
![Page 5: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/5.jpg)
Is There a Significant, Positive Correlation Between Completing MasteringBiology and Exam Scores?
• Students required to complete MasteringBiology homework assignments (10% lecture grade).
• Overall exam average was compared to percentage MasteringBiology completed by end of semester.
• Spearman's Rank Order correlation was run:(rs(138) = 0.571, P = .000)
• A statistically significant, positive correlation between percentage of MasteringBiology homework completed and exam average was found.
![Page 6: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/6.jpg)
Results
• Students completing more work in MasteringBiology were more successful on exams.
![Page 7: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/7.jpg)
Problem: How to Increase Student Success in Introductory (Majors) Biology?
• My Strategy, Spring 2011 semester:
• Evaluate the efficacy of providing peer tutors vs. requiring MasteringBiology homework (using exam scores to measure)• Offered optional, online pre-exam quizzes to all using
MasteringBiology
• Results:
• Peer tutoring appears to have no benefit over offering required MasteringBiology homework.
• Significant, positive correlation between participating in optional quizzing and increased exam score observed.
![Page 8: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/8.jpg)
Cognitive Science of Learning
• Introduction of Desirable Difficulties:
“Conditions of instruction that appear to create difficulties for the learner, slowing the rate of
apparent learning, often optimize long-term retention and transfer.”
![Page 9: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/9.jpg)
Testing Serves as Learning Events
• Testing as a learning vehicle vs. an assessment vehicle.
• Evidence shows: Practice of recalling information is more powerful than re-presentation of information.
• Reality is: Students re-present information to themselves (i.e., reading a chapter over and over again, underlining things, etc.), focusing on storage rather than retrieval.
![Page 10: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/10.jpg)
Can An Online Homework Platform Be Used For Quizzing to Increase Student Success?
• Introduced required pre-exam quizzing delivered via MasteringBiology
• Students required to take pre-exam quizzes during the Fall 2011 semester (n=199) and Spring 2012 semester (n=174)
• My Question: Does taking pre-exam quizzes using
Mastering result in significantly higher exam scores?
![Page 11: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/11.jpg)
Preparation
• Before beginning, set up spreadsheet to collect data necessary to answer questions.
• My predesigned columns included:• Name (deleted prior to sending to statistician)• Exam # and grade• Number of pre-exam quizzes taken• Grade on each pre-exam quiz• Set up color coding to designate “drops” and “not in my course”
• Data gathered by exporting grade book from Mastering.
![Page 12: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/12.jpg)
Student Participation Compliance in Required Pre-exam QuizzingFall 2011 & Spring 2012
% Students Completing Quizzes prior to Exam
% Quizzes Completed
Exam 1
n= 362
Exam 2
n= 339
Exam 3
n= 321
Exam 4
n= 305
100% 78.5 59.9 62.9 59.3
67% 16.3 22.4 * *
50% * * 26.5 25.9
33% 3.0 11.5 * *
0% 2.2 7.4 10.3 14.8
![Page 13: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/13.jpg)
Comparison of Exam Averages100% Quiz Takers to 0% Quiz Takers
Fall 2011
Spring 2012
![Page 14: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/14.jpg)
Comparison of Exam Averages by Percentage of Quiz Taking: Spring 2012
![Page 15: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/15.jpg)
Analysis by Groups
• Scores on Exams 1 and 2 were averaged, and students were grouped by these averages.
• Groups Established:• Pass: 70%+• Fail: <70%• Low (“weak student”): <60%• Middle (“average student”): 60%-80%• High (“strong student”): >80%
• Student scores were compared by group for 100% quiz takers vs. 0% quiz takers.
![Page 16: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/16.jpg)
Performance of Passing vs. Failing Students: 100% Quiz Takers vs. 0% Quiz Takers
Exam 3
Exam 4
![Page 17: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/17.jpg)
Performance by Low, Middle and High Students: 100% Quiz Takers vs. 0% Quiz Takers
Exam 3
Exam 4
![Page 18: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/18.jpg)
Data Analysis Often Generates New Questions!
• Can students’ compliance with pre-exam quiz requirement be increased?
• If percent pre-exam quiz compliance increases, will significant gains in student success in the course be observed?
![Page 19: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/19.jpg)
Impact of Data Collection on Course Approach • Tell the students about the data!
• Ask them to predict! Who earns higher exam scores:• Quiz takers or non quiz takers?• Quiz takers that fail their quizzes or non quiz takers?
• Show them the data, and ASK:• “What group do you want to be in?!”
The blue group? Or the red group?!
![Page 20: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/20.jpg)
Pre-exam Quiz Participation Rate Increased Significantly in Fall 2012 vs. Spring 2012
Sprin
g: 100% Parti
cipati
on
Fall:
100% Parti
cipati
on
Sprin
g: 0% P
articip
ation
Fall:
0% Parti
cipati
on0
1020304050607080
Exam 1Exam 2Exam 3Exam 4
![Page 21: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/21.jpg)
Percentage of ABC vs. DFW Grades Earned in Spring 2012 vs. Fall 2012
ABC DFW0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
SpringFall
![Page 22: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/22.jpg)
Study Results• Student participation in pre-exam quizzing results in
significantly higher exam grades.
• Benefit realized to student is based on participation (rather than performance) in quizzing.
• Participation in pre-exam quizzing results in significantly higher exam scores for students of diverse abilities.
• Stronger students may see slightly greater benefits from pre-exam quizzing (as evidenced by exam averages).
• Significant increase in student compliance with required quizzing was accompanied by significantly higher percentage of ABC rates vs. DFW rates.
![Page 23: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/23.jpg)
Student Perception of Mastering: Homework and Pre-Exam Quizzing
Mas
terin
g Hom
ewor
k Help
ed M
e Kee
p ...
Mas
terin
g Hom
ewor
k In
crea
sed
Und...
Quiz
zes
Incr
ease
d Und
erst
andin
g
Quiz
zing
Result
s Le
d to
+ S
tud.
..
Plan to
con
tinue
quiz
zing
in fu
ture
0102030405060708090
100
(% S
tud
ent
Ag
reem
ent)
![Page 24: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/24.jpg)
Study Conclusions
• Pre-exam quizzing using online platform seems to provide a viable testing environment for student learning.
• Using online platform allows instructors more time to teach!
• All students showed positive gains in exam scores as a result of participating in pre-exam quizzing.
![Page 25: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/25.jpg)
Lessons learned
• Begin process of gaining IRB approval right away.
• Consult with your statistician about what you want to investigate before you start.
• Speak with colleagues about your investigation to get a feel for questions they may raise.
• If introducing a new teaching method or technology, consider giving a student survey to gather their thoughts and comments.
![Page 26: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/26.jpg)
Lessons learned (cont.)• Design your data collection tools before beginning the
study.• Gather and organize more than you think you need!• Stay current in updating master spreadsheet throughout
the semester:• Falling behind can become overwhelming!
• New questions often arise as you review the data.
• Modifications in what you record may be necessary. Much easier to do this “as you go” than to go back and wade through the entire semester’s worth of data!
• Tell students about your study- involve them in the process when possible.
![Page 27: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/27.jpg)
Benefits of Studying Student Success Data
• Allows you to identify what you are doing that is truly effective.
• Sharing results with students motivates them and helps them to feel that your requirements will yield positive gains for them.
• Provides intellectual challenge to you that provides additional purpose and interest!
![Page 28: COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology](https://reader036.vdocument.in/reader036/viewer/2022062421/56649c915503460f9494babe/html5/thumbnails/28.jpg)
Acknowledgements• Shellene Foster
Professor of Mathematics, Collin College
• Pushpa Ramakrishna, Ed.D. Professor of Biology, Chandler-Gilbert Community College
References• 1. Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way:
Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough, & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56-64). New York: Worth Publishers.
• 2. Halamish, V., & Bjork, R. A. (2011). When does testing enhance retention? A distribution-based interpretation of retrieval as a memory modifier. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 801-812.