![Page 1: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/1.jpg)
Technology In and Out of the Classroom: The Effect of Online
Learning on Student Performance
Jodi N. BeggsEconomists Do It With Models
7th Annual Economics Teaching ConferenceOctober 27, 2011
www.economistsdoitwithmodels.com
![Page 2: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/2.jpg)
Motivation• Personal bias • Increasing fiscal constraints at both the university and K-12 levels• Increased technology availability and cost effectiveness• Adoption rate of online learning is very high
• 2.6 million students took at least one online course in fall 2005• 80 percent of doctoral/research institutions in the US offer online
courses• Virtually every institution with over 15,000 students offers online
courses• Online courses also increasing prevalent at the K-12 level
![Page 3: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/3.jpg)
Motivation• Personal bias • Increasing fiscal constraints at both the university and K-12 levels• Increased technology availability and cost effectiveness• Adoption rate of online learning is very high
• 2.6 million students took at least one online course in fall 2005• 80 percent of doctoral/research institutions in the US offer online
courses• Virtually every institution with over 15,000 students offers online
courses• Online courses also increasing prevalent at the K-12 level
Online instruction represents a potential opportunity for both cost savings and educational enhancement, but the effect on student
understanding needs to be understood before new modes of instruction are widely adopted
![Page 4: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/4.jpg)
Brown and Liedholm 2002• Data from Principles of Microeconomics at Michigan
State University• 3 modes of instruction: live, hybrid, virtual• Same textbook, multiple-choice examinations,
course web sites and email• Live: 3 class hours per week• Hybrid: 2 class hours per week plus online materials• Required “Problems in Microeconomics”
• Virtual: Videos of live lectures plus hybrid materials
![Page 5: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/5.jpg)
The Data (Means)Live Hybrid Virtual
Total Score (%) 65.49 64.51 61.19
Subscore 1 72.23 73.06 71.84
Subscore 2 62.48 58.77 55.57
Subscore 3 58.02 57.13 50.34
Female 0.52 0.46 0.42
Black 0.17 0.05 0.03
Math Courses 1.53 1.17 1.62
ACT 22.84 23.52 24.46
GPA 2.86 2.84 2.80
Credits 51.48 39.95 56.42
N 363 258 89
![Page 6: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/6.jpg)
Score Regressions By Course TypeLive Hybrid Virtual
Female -5.70 -1.08 -2.03Black -4.70 -5.94 -4.93Math Courses
1.65 0.69 -3.74
ACT 0.63 0.74 1.15GPA 15.93 12.72 15.06Credits 0.030 0.044 -0.052Adjusted R2 0.50 0.37 0.36
![Page 7: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/7.jpg)
Score Predictions By Instruction Type
Hypothetical TypeActual Type Live Hybrid VirtualLive 65.49 64.20 59.72Hybrid 65.76 64.51 63.80Virtual 66.98 66.02 61.19
![Page 8: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/8.jpg)
Discussion• Differences in performance across modes of instruction increase
as questions get higher level• Differences could be at least partially attributable to differences
in student effort• Statistics on time spent on course work indicate that students
in live sections put in more effort• Gender differences largely go away in hybrid and virtual
environments• Could reflect differences in effort across genders• Race differences also go away to some degree
• Selection bias is present since students are selecting into different forms of the course
![Page 9: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/9.jpg)
Discussion• Differences in performance across modes of instruction increase
as questions get higher level• Differences could be at least partially attributable to differences
in student effort• Statistics on time spent on course work indicate that students
in live sections put in more effort• Gender differences largely go away in hybrid and virtual
environments• Could reflect differences in effort across genders• Race differences also go away to some degree
• Selection bias is present since students are selecting into different forms of the course Selection bias can only
be eliminated via a randomized
experiment…
![Page 10: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/10.jpg)
Figlio, Rush and Yin 2010• Large introductory course at a major research university• Randomly assigned to live or internet lectures• Students were given an incentive to participate in
the experiment• Volunteers not markedly different from non-
volunteers• All other aspects of the course held constant• Multiple-choice exams• Live lecture condition most similar to the hybrid
condition introduced earlier
![Page 11: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/11.jpg)
Scores By Method of Instruction
Exam 1 Exam 2 Final Exam
Average Score
Live 84.536 76.692 75.939 79.940Online 83.301 76.904 74.302 78.502Difference 1.235 -0.212 1.637 1.440
![Page 12: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/12.jpg)
Scores By Method of Instruction
Exam 1 Exam 2 Final Exam
Average Score
Live 84.536 76.692 75.939 79.940Online 83.301 76.904 74.302 78.502Difference 1.235 -0.212 1.637 1.440
None of the differences are statistically significant!
![Page 13: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/13.jpg)
Differences Matter for Some Demographics
Group Live-Online Differential
White students 1.117
Black students 2.828
Hispanic students 11.276
Asian students 4.319
Male students 3.480
Female students 1.780
Low-achievers 4.054
High-achievers 1.169
![Page 14: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/14.jpg)
Differences Matter for Some Demographics
Group Live-Online Differential
White students 1.117
Black students 2.828
Hispanic students 11.276
Asian students 4.319
Male students 3.480
Female students 1.780
Low-achievers 4.054
High-achievers 1.169Could again reflect differences in effort
![Page 15: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/15.jpg)
Caveats• Live/online contamination• Attendance data suggests that some live students
were watching lectures online• However, live participants had better attendance
than live non-participants• Potential peer effects• Representativeness of study volunteers• Familiarity of students with online platforms may affect
the live/online differential• Results may not generalize to non-economics material
![Page 16: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/16.jpg)
Caveats• Live/online contamination• Attendance data suggests that some live students
were watching lectures online• However, live participants had better attendance
than live non-participants• Potential peer effects• Representativeness of study volunteers• Familiarity of students with online platforms may affect
the live/online differential• Results may not generalize to non-economics materialDo other studies
support these findings?
![Page 17: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/17.jpg)
Dept. of Education Meta Analysis• Literature search from 1996 through July 2008• Studies screened for several attributes• Contrasted an online to a face-to-face condition• Measured student learning outcomes• Used a rigorous research design• Provided adequate information to calculate an effect
size• 51 independent effects identified• 7 at the K-12 level, the rest with older students• Various subjects examined, the most popular being
medicine or health care
![Page 18: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/18.jpg)
Findings• On average, students in online learning conditions performed
better than those receiving face-to-face instruction• 11 individual effects significantly favored an online or hybrid
condition• 2 individual effects significantly favored face-to-face
condition• Differences were largest when pure face-to-face instruction
contrasted with a hybrid of online and face to face• Elements such as video or online quizzes do not appear to
influence the amount that students learn in online courses
![Page 19: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/19.jpg)
Discussion• Because online conditions usually included
additional learning time and instructional elements, it’s not clear that the online medium is driving results• This is supported by the observation that
studies in which online learners spent more time on the course than their face-to-face counterparts saw greater benefits from online instruction
![Page 20: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/20.jpg)
Key Success Factors for Technology Integration
• Leverage top instructors• Winner-take-all market
• Use technology to supplement and economize, not purely to replace• Enable efficient division of labor
• Provide incentives/commitment devices/etc. to dissuade procrastination and lack of engagement
![Page 21: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/21.jpg)
For Further Reading…David N. Figlio, Mark Rush, and Lu Yin, “Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning.” National Bureau of Economic Research Working Paper 16089.
Byron W. Brown and Carl E. Liedholm, “Can Web Courses Replace the Classroom in Principles of Microeconomics?” American Economic Review, May 2002 (Papers and Proceedings), 92(2), pp. 444-448.
U.S. Department of Education, Office of Planning, Evaluation, an Policy Development, Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies, Washington, D.C., 2009.
![Page 22: Technology In and Out of the Classroom: The Effect of Online Learning on Student Performance Jodi N. Beggs Economists Do It With Models 7th Annual Economics](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2e5503460f94a057a6/html5/thumbnails/22.jpg)
For more information, go tohttp://www.economistsdoitwithmodels.com/Carlson