Technology In and Out of the Classroom: The Effect of Online

Learning on Student Performance

Jodi N. BeggsEconomists Do It With Models

7th Annual Economics Teaching ConferenceOctober 27, 2011

www.economistsdoitwithmodels.com

Motivation• Personal bias • Increasing fiscal constraints at both the university and K-12 levels• Increased technology availability and cost effectiveness• Adoption rate of online learning is very high

• 2.6 million students took at least one online course in fall 2005• 80 percent of doctoral/research institutions in the US offer online

courses• Virtually every institution with over 15,000 students offers online

courses• Online courses also increasing prevalent at the K-12 level

Motivation• Personal bias • Increasing fiscal constraints at both the university and K-12 levels• Increased technology availability and cost effectiveness• Adoption rate of online learning is very high

• 2.6 million students took at least one online course in fall 2005• 80 percent of doctoral/research institutions in the US offer online

courses• Virtually every institution with over 15,000 students offers online

courses• Online courses also increasing prevalent at the K-12 level

Online instruction represents a potential opportunity for both cost savings and educational enhancement, but the effect on student

understanding needs to be understood before new modes of instruction are widely adopted

Brown and Liedholm 2002• Data from Principles of Microeconomics at Michigan

State University• 3 modes of instruction: live, hybrid, virtual• Same textbook, multiple-choice examinations,

course web sites and email• Live: 3 class hours per week• Hybrid: 2 class hours per week plus online materials• Required “Problems in Microeconomics”

• Virtual: Videos of live lectures plus hybrid materials

The Data (Means)Live Hybrid Virtual

Total Score (%) 65.49 64.51 61.19

Subscore 1 72.23 73.06 71.84

Subscore 2 62.48 58.77 55.57

Subscore 3 58.02 57.13 50.34

Female 0.52 0.46 0.42

Black 0.17 0.05 0.03

Math Courses 1.53 1.17 1.62

ACT 22.84 23.52 24.46

GPA 2.86 2.84 2.80

Credits 51.48 39.95 56.42

N 363 258 89

Score Regressions By Course TypeLive Hybrid Virtual

Female -5.70 -1.08 -2.03Black -4.70 -5.94 -4.93Math Courses

1.65 0.69 -3.74

ACT 0.63 0.74 1.15GPA 15.93 12.72 15.06Credits 0.030 0.044 -0.052Adjusted R2 0.50 0.37 0.36

Score Predictions By Instruction Type

Hypothetical TypeActual Type Live Hybrid VirtualLive 65.49 64.20 59.72Hybrid 65.76 64.51 63.80Virtual 66.98 66.02 61.19

Discussion• Differences in performance across modes of instruction increase

as questions get higher level• Differences could be at least partially attributable to differences

in student effort• Statistics on time spent on course work indicate that students

in live sections put in more effort• Gender differences largely go away in hybrid and virtual

environments• Could reflect differences in effort across genders• Race differences also go away to some degree

• Selection bias is present since students are selecting into different forms of the course

Discussion• Differences in performance across modes of instruction increase

as questions get higher level• Differences could be at least partially attributable to differences

in student effort• Statistics on time spent on course work indicate that students

in live sections put in more effort• Gender differences largely go away in hybrid and virtual

environments• Could reflect differences in effort across genders• Race differences also go away to some degree

• Selection bias is present since students are selecting into different forms of the course Selection bias can only

be eliminated via a randomized

experiment…

Figlio, Rush and Yin 2010• Large introductory course at a major research university• Randomly assigned to live or internet lectures• Students were given an incentive to participate in

the experiment• Volunteers not markedly different from non-

volunteers• All other aspects of the course held constant• Multiple-choice exams• Live lecture condition most similar to the hybrid

condition introduced earlier

Scores By Method of Instruction

Exam 1 Exam 2 Final Exam

Average Score

Live 84.536 76.692 75.939 79.940Online 83.301 76.904 74.302 78.502Difference 1.235 -0.212 1.637 1.440

Scores By Method of Instruction

Exam 1 Exam 2 Final Exam

Average Score

Live 84.536 76.692 75.939 79.940Online 83.301 76.904 74.302 78.502Difference 1.235 -0.212 1.637 1.440

None of the differences are statistically significant!

Differences Matter for Some Demographics

Group Live-Online Differential

White students 1.117

Black students 2.828

Hispanic students 11.276

Asian students 4.319

Male students 3.480

Female students 1.780

Low-achievers 4.054

High-achievers 1.169

Differences Matter for Some Demographics

Group Live-Online Differential

White students 1.117

Black students 2.828

Hispanic students 11.276

Asian students 4.319

Male students 3.480

Female students 1.780

Low-achievers 4.054

High-achievers 1.169Could again reflect differences in effort

Caveats• Live/online contamination• Attendance data suggests that some live students

were watching lectures online• However, live participants had better attendance

than live non-participants• Potential peer effects• Representativeness of study volunteers• Familiarity of students with online platforms may affect

the live/online differential• Results may not generalize to non-economics material

Caveats• Live/online contamination• Attendance data suggests that some live students

were watching lectures online• However, live participants had better attendance

than live non-participants• Potential peer effects• Representativeness of study volunteers• Familiarity of students with online platforms may affect

the live/online differential• Results may not generalize to non-economics materialDo other studies

support these findings?

Dept. of Education Meta Analysis• Literature search from 1996 through July 2008• Studies screened for several attributes• Contrasted an online to a face-to-face condition• Measured student learning outcomes• Used a rigorous research design• Provided adequate information to calculate an effect

size• 51 independent effects identified• 7 at the K-12 level, the rest with older students• Various subjects examined, the most popular being

medicine or health care

Findings• On average, students in online learning conditions performed

better than those receiving face-to-face instruction• 11 individual effects significantly favored an online or hybrid

condition• 2 individual effects significantly favored face-to-face

condition• Differences were largest when pure face-to-face instruction

contrasted with a hybrid of online and face to face• Elements such as video or online quizzes do not appear to

influence the amount that students learn in online courses

Discussion• Because online conditions usually included

additional learning time and instructional elements, it’s not clear that the online medium is driving results• This is supported by the observation that

studies in which online learners spent more time on the course than their face-to-face counterparts saw greater benefits from online instruction

Key Success Factors for Technology Integration

• Leverage top instructors• Winner-take-all market

• Use technology to supplement and economize, not purely to replace• Enable efficient division of labor

• Provide incentives/commitment devices/etc. to dissuade procrastination and lack of engagement

For Further Reading…David N. Figlio, Mark Rush, and Lu Yin, “Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning.” National Bureau of Economic Research Working Paper 16089.

Byron W. Brown and Carl E. Liedholm, “Can Web Courses Replace the Classroom in Principles of Microeconomics?” American Economic Review, May 2002 (Papers and Proceedings), 92(2), pp. 444-448.

U.S. Department of Education, Office of Planning, Evaluation, an Policy Development, Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies, Washington, D.C., 2009.

For more information, go tohttp://www.economistsdoitwithmodels.com/Carlson