sr ithaka interactive learning online at public universities

8/12/2019 Sr Ithaka Interactive Learning Online at Public Universities

1/53

May 22, 2012

Interactive Learning Online

at Public Universities:Evidence from

Randomized TrialsWilliam G. Bowen

Matthew M. Chingos

Kelly A. Lack

Thomas I. Nygren


2/53

Interactive Learning Online at Public Universities: Evidence from Randomized Trials May 22, 2012 1

Ihaka S+R is a sraegic consuling and research service provided by IHA K,a no-or-profi organizaion dedicaed o helping he academic communiy usedigial echnologies o preserve he scholarly record and o advance research andeaching in susainable ways. Ihaka S+R ocuses on he ransormaion o schol-arship and eaching in an online environmen, wih he goal o ideniying hecriical issues acing our communiy and acing as a caalys or change. JSOR,a research and learning plaorm, and Porico, a digial preservaion service, arealso par o IHAK.

Copyrigh 2012 IHA K. Tis work is licensed under he Creaive CommonsAtribuion No Derivaive Works 3.0 Unied Saes License. o view a copy ohe license, please see htp://creaivecommons.org/licenses/by-nd/3.0/us
http://creativecommons.org/licenses/by-nd/3.0/ushttp://creativecommons.org/licenses/by-nd/3.0/us


3/53


Preface

Higher educaion is acing serious challenges in he Unied Saes. Tere isincreasing concern abou rising coss, he qualiy o educaion, and ha henaion is losing is compeiive edge. Online learningspecifically highlyineracive, closed-loop, online learning sysems ha we call ILO or IneraciveLearning Onlineholds he promise o broadening access o higher educaiono more individuals, while also lowering coss or sudens. Bu is he qualiyhere?

In our firs repor in his area, Barriers o Adopion o Online Learning Sysemsin U. S. Higher Educaion, we highlighed a broad, widely held concern abouhe qualiy o learning oucomes achieved hrough online learning. Bu do weacually know how ineracive online learning sysems really compare o hein-classroom experience? Tis second repor was designed o help find answers.

We used a sricly quaniaive mehodology o compare he wo learning approachesin a rigorous way. In six differen public insiuions, we arranged or he sameinroducory saisics course o be augh. In each insance, a conrol group wasenrolled in a radiional classroom-based course; hen, a reamen group ook ahybrid course using a prooype machine-guided mode o insrucion developed aCarnegie Mellon Universiy in concer wih one ace-o-ace meeing each week.Sudens were assigned o hese wo groups by means o a careully designedrandomizaion mehodology. Te research we conduced was designed o answerhese quesions:

Can sophisicaed, ineracive online courses be used o mainain or improvebasic learning oucomes (masery o course conen, compleion raes, andime-o-degree) in inroducory courses in basic subjecs such as saisics?

Are hese courses as effecive, or possibly more effecive, or minoriy andlow-socioeconomic-saus sudens and or oher groups subjec o sereoype

hrea? Or, are hese groups less well suied o an online approach?

Are such courses equally effecive wih no-so-well-prepared sudens andwell-prepared sudens?

Te resuls o his sudy are remarkable; hey show comparable learning oucomes orhis basic course, wih a promise o cos savings and produciviy gains over ime.

More research is needed. Even hough he analysis was rigorous, i was a singlecourse. We need o learn more abou he adapabiliy o exising plaorms oroffering oher courses in differen environmens. Ihaka S+R is commited oconinuing his research and shar ing our findings broadly.

We look orward o coninuing o engage wih al l hose who care abou highereducaion o help deliver on he poenial ha new echnologies provide.

DEANNA MARCUM

[email protected]

Managing Director, Ithaka S+R
mailto:Deanna.Marcum%40ithaka.org?subject=mailto:Deanna.Marcum%40ithaka.org?subject=


4/53


Interactive Learning Online at Public Universities:

Evidence from Randomized Trials

2 Preface

4 Introduction

9 Educational Outcomes in Public Universities

23 Costs and Potential Savings

26 Summary Observations

29 Acknowledgements

30 Appendices


5/53


Interactive Learning Online at

Public Universities: Evidence fromRandomized TrialsWilliam G. Bowen, Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren1

May 22, 2012

Introduction

Te opic o online learning in higher educaion is o obvious imporance. Teserious economic and social problems acing he U.S.high unemploymen,slow growh, and severe inequaliiesare relaed, many believe, o ailures o he

1 The authors are all associated with Ithaka S+R (the Strategy and Research arm of ITHAKA), which sponsored

this study. Bowen is a senior advisor t o Ithaka S+R, Chingos is a senior research consultant at Ithaka S+R

and a fellow at the Brookings Institution's Brown Center on Educat ion Policy, Lack is a research analyst,

and Nygren is a project director and senior business analyst for Ithaka S+R. The authors wish to thank the

foundat ions that supported this work: the Carneg ie Corporation of New York, the William and Flora Hewlett

Foundation, the Spencer Foundation, and a fourth foundation that has asked to remain anonymous. We also

thank our colleagues at ITHAKAand L arry Bacow, Johanna Brownell, Jackie Ewenstein, and Kevin Guthrie inparticularfor their generous help all along the way. But most o f all, we wish to thank our faithful friends on

the participating campuses for their hard work and patience with us; their names are appended to this repor t.

A number of these individuals (as well as others) have commented on a draft of the repor t, but the authors

are, of course, fully responsible for the views expressed here and for any errors t hat remain.

Ithaka S+R has sponsored three studies of online learning, of which this is the longest lasting. The two other

studies are now available on the Ithaka S+R website. See Barriers to Adoption of Online Learning Systems

in U.S. Higher Education by Lawrence S. Bacow, William G. Bowen, Kevin M. Guthrie, Kelly A. Lack, and

Matthew P. Long, and Current Status of Research on Online Learning in Postsecondary Educat ion by William

G. Bowen and Kelly A. Lack (bo th available online at http://www.sr.ithaka.org/).


6/53


U.S. educaion sysem, including higher educaion.2Levels o educaionalatainmen in his counry have been sagnan or almos hree decades, whilemany oher counries have been making grea progress in educaing largernumbers o heir ciizens. Tere is growing concern ha he U.S. is losing iscompeiive edge in an increasingly knowledge-driven world. Also, subsanialachievemen gaps relaed o race and socioeconomic saus persis and have agrea deal o do w ih worry ing inequiies. Moreover, here are good reasons o

believe ha hese wo problems are closely relaed.3

The Cost Squeeze in Higher Education

A he same ime, higher educaion, especially in he public secor, is increasinglyshor o resources. Saes coninue o cu back appropriaions in he ace o fiscalconsrains and pressures o spend more on oher hings, such as healh careand reiremen expenses.4Caliornia is a dramaic case in poin. Lack o undinghas caused Caliornia colleges and universiies o reduce he size o heir ener-ing classes a he very ime when increasing numbers o sudens are seekingo enroll.5Higher uiion revenues migh be an escape valve, bu here is greaconcern abou uiion levels and increasing resenmen among sudens andheir amilies ha is having poliical reverberaions. Presiden Obama, in his

2 The authors agree that there is an important connection between educational outcomes and the economic

performance of a c ountry. But we would warn against exaggerating the power of the connection. In the case

of the U.S., for example, the recent recession and the slow rate of g rowth seen in the last few years surely

owe more to the 2008 financial excesses than they do to deficiencies in the countrys higher education

system. As Jacob Weisberg pointed out in Newsweek in 2010 with respect to the recent recession, there are

no strong candidates for a single factor t hat would have caused the crisis in the absence of any others

(Weisbergs piece can be f ound online at http://www.thedailybeast.com/newsweek/2010/01/08/what-

caused-the-great-recession.html).

3 SeeEquity and Excellence in American Higher Education by William G. Bowen, Martin A. Kurzweil, and

Eugene M. Tobin (2005) for an extended discussion of the historical record and of the likely connections,

going forward, between achievement gaps and overall levels of educational at tainment. See also David

Leonhardts October 8, 2011 column in the New York Times, The Depression: If Only Things Were That Good,

in which he argues that the U.S. is worse off today than it was in the 1930s because innovation is lagging

which he attributes in no small part to deficiencies in education (http://www.nytimes.com/2011/10/09/sun-

day-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=all.) Of course, lagging

rates of educational at tainment have their origins in low high school graduation rates. See Henry M. Levin

and Cecilia E. Rouse, The True Cost of High School D ropout, New York Times, January 25, 2012. (http://

www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.html). But these problems

are then compounded by low completion rates among those who both graduate from high school and enter

college; see Crossing the Finish Line: Completing College at Americas Public Universities(2009) by William

G. Bowen, Matthew M. Chingos, and Michael S. McPherson.

4 A report released in spring 2012 by the State Higher Education Executive Officers, entitled State Higher

Education Finance FY 2011 (http://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdf), docu-

ments the dire economic circumstances of many public institutions.

5 In November 2008, California State University became the first public university to limit enrollment when,

despite a 20% increase in applications from prospective first-year students, it decided to reduce its s tudent

body by 10,000 students, following a $200 million decrease in tax revenue that academic year coupled with

an additional $66 million cut (see Under Financial Stress, More Colleges Cap Enrollments (November

26, 2008) in TIME, http://www.time.com/time/nation/article/0,8599,1861861,00.html). The University of

California and California Community College systems have since followed suit in the face of limited funding

available from the state (see the August 5, 2009 article Budget cuts devastate California higher educa-

tion in The Washington Examiner, http://washingtonexaminer.com/science-and-technology/2009/08/

budget-cuts-devastate-california-higher-education).

Levels of educational attainment in

this country have been stagnant for

almost three decades, while many

other countries have been making

great progress in educating larger

numbers of their citizens.
http://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdfhttp://www.time.com/time/nation/article/0,8599,1861861,00.htmlhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://washingtonexaminer.com/science-and-technology/2009/08/budget-cuts-devastate-california-higher-educationhttp://www.time.com/time/nation/article/0,8599,1861861,00.htmlhttp://www.sheeo.org/finance/shef/SHEF_FY2011-EARLY_RELEASE.pdfhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2012/01/26/opinion/the-true-cost-of-high-school-dropouts.htmlhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.nytimes.com/2011/10/09/sunday-review/the-depression-if-only-things-were-that-good.html?_r=1&pagewanted=allhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.htmlhttp://www.thedailybeast.com/newsweek/2010/01/08/what-caused-the-great-recession.html


7/53


2012 Sae o he Union address and in subsequen speeches, has decried risinguiions, called upon colleges and universiies o conrol coss, and proposed o

wihhold access o some Federal programs or colleges and universi ies ha didno address affordabiliy issues or mee compleion ess.6

oday, a variey o higher educaion insiuions mus conron he challengeo how o manage coss in he ace o igher unding. Whi le he proporion oeducaion spending drawn rom uiion revenues rose across all insiuions,

increases in uiion ofen oupaced increases in educaion and relaed spending(i.e. spending on insrucion, suden services, and some suppor and maine-nance coss relaed o hese uncions), calling ino quesion he susainabiliyo he curren unding model.7Moreover, he firs survey o provoss and chieacademic officers byInside Higher Edound ha on he quesion o insiuionaleffeciveness in conrolling coss, over 15 percen o all provoss gave heirinsiuions marks o 1 or 2 on effeciveness [on a scale rom 1 o 7, wih 7 being

very effecive].8I is equal ly noeworhy ha veryew chie academic officers(and especially hose a boh public and privae docoral universiies) gave heirinsiuions high marks on his meric. Recogniion o he problem is widespread;soluions have been hard o come by.

A undamenal source o he problem is he cos disease, based on he handicrafnaure o educaion wih is atendan lack o opporuniies or gains in pro-duciviy, which one o he auhors o his repor (Bowen) promulgaed in he1960s, in collaboraion wih William J. Baumol. Bu he ime may (finally!) bea hand when advances in inormaion echnology wil l permi, under he righcircumsances, increases in produciviy ha can be ranslaed ino reducions in

6 See Remarks by the President in State of the Union Address, January 24, 2012 (transcript available at (http://

www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-address). Three days later,

Obama spoke about college affordability at the University of Michigan (transcript available at http://www.

whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michigan).

This speech does not, however, contain more details concerning how affordability is to be measured or

what penalties are to be imposed on those who fail to pass the requisite tests. As Molly Broad, president of

the American Council on Education, said af ter the speech: The devil is in the [unspecified] details (Mixed

Reviews of Obama Plan to Keep Down College Costs, January 28, 2012, New York Times, http://www.nytimes.

com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.html).7 According to the College Boards 2011 Trends in College Pricing Report (http://trends.collegeboard.org/

downloads/College_Pricing_2011.pdf), tuition at public two-year universities increased, on average, by

8.7% relative to the 2010-2011 academic year, and tuition at public four-year institutions for the 2011-2012

academic year increased, on average, by 8.3% for instate students and by 5.7% for out of state students.

In keeping with the trend over the previous four years, students at tending private institutions experienced

smaller percentage increases (4.5% for private not-for-profit four-year institutions and 3.2% fo r private for-

profit institutions).

8 See Scott Jaschik, Mixed Grades: A Survey of Provosts, Inside Higher Education, January 25, 2012, http://

www.insidehighered.com/news/survey/mixed-grades-survey-provosts.

Higher education, especially in the

public sector, is increasingly short

of resources. States continue to

cut back appropriations in the

face of scal constraints and

pressures to spend more on

other things, such as healthcare and retirement expenses.
http://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://www.insidehighered.com/news/survey/mixed-grades-survey-provostshttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://trends.collegeboard.org/downloads/College_Pricing_2011.pdfhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.nytimes.com/2012/01/28/education/obamas-plan-to-control-college-costs-gets-mixed-reviews.htmlhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/27/remarks-president-college-affordability-ann-arbor-michiganhttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-addresshttp://www.whitehouse.gov/the-press-office/2012/01/24/remarks-president-state-union-address


8/53


he cos o insrucion.9Greaerand smareruse o echnology in eaching iswidely seen as a promising way o conroll ing coss while also reducing achieve-men gaps and improving access. Te exploding growh in online learning isofen cied as ev idence ha, a las, echnology may offer pahways o progress.10Online learning is seen by a growing number o people as a way o breakingree o cenury-old rigidiies in educaional sysems ha we have inheried. Temuch-discussed book on disrupive echnologies and universiies by ClayonChrisensen and Henry Eyring is perhaps he bes example o he atenion beinggiven o online echnologies as a way o changing prooundly he way we educaesudens.11

Tere are, however, also concerns ha a leas some kinds o online learning arelow qualiy and ha onl ine learning in general de-personalizes educaion.In thisregard, it is crit ically important to recognize issues of nomenclature: online learningis hardly one thing.I comes in a dizzying variey o flavors, ranging rom simply

videoaping lecures and posing hem or any-ime access, o uploading maeri-als such as s yllabi, homework assignmens, and ess o he Inerne, all he wayo highly sophisicaed ineracive learning sysems ha use cogniive uors andake advanage o muliple eedback loops. Te varieies o online learning can be

used o each many kinds o subjecs o differen populaions in diverse insiuional

9 Bowens co-author in the promulgation of the cost disease, William J. Baumol, has continued to discuss

its relevance not only for education but also for sect ors such as the performing arts and heath care. For

the initial statement of this proposition, see William J. Baumol and William G. Bowen,Performing Ar ts: The

Economic Dilemma, Twentieth Century Fund (1968). In essence, the argument is that in fields such as t he

performing arts and education, there is less opportunity t han in other fields to improve productivity (by, for

example, substituting capital for labor), that unit labor costs will therefore rise inexorably as these sectors

have to compete for labor with ot her sectors in which productivity gains are easier to come by, and that the

relative costs of labor-intensive activities such as chamber music and teaching will therefore continue to

rise. As Bowen argued in his Romanes lecture, for a number of years advances in information technology

have in fact increased productivity, but these increases have been enjoyed primarily in the form o f more

output (especially in research) and have generally led to higher, not lower, total costs. (For the text of the

Romanes lecture, see William G. Bowen, At a Slight Angle to the Universe: The University in a Digitized,

Commercialized Age, Princeton University Press, 2001; the text is also available on the Andrew W. Mellon

Foundation website: http://www.mellon.org/internet/news_publications/publications/romanes.pdf.)

10 A November 2011 report by the Sloan Consortium and the Babson Survey Research Group shows that between

fall 2002 and fall 2010, en rollments in online courses increased much more quickly t han tot al enro llments

in higher education. During this time period, the number of online course enrollments grew from 1.6 million

to 6.1 million, amounting to a compound annual rate of 18.3% (compared with a rate o f 2% for course

enrollments in general)although between fall 2009 and fall 2010 online enrollments grew more slowly,

at 10.1%. More than three of every 10 students in higher education now t ake at least one course online. Inaddition to the growth in what we call online or hybrid courseshowever nebulous that terminology may

bewe also fee l the pervasiveness of the Internet in higher education by the increasing use of it in the

form o f course manag ement systems or vi rtual reading materials/electronic tex tbooks incorporated in to the

curriculum. Even courses that are called traditional almost always involve some use of digit al resources.

11 See Clayton M. Christensen, and Henry J. Eyring, The Innovative University: Changing the DNA of Higher

Education from the Inside Out, San Francisco: Jossey-Bass, 2011. An October 2, 2011 New York Times op-ed

piece by Bill Keller, aptly titled The University of Wherever, is another illustration of the high visibility and

high stakes of the debate over online education (http://www.nytimes.com/2011/10/03/opinion/the-universi-

ty-of-wherever.html?pagewanted=all).

There are also concerns that at

least some kinds of online learning

are low quality and that online

learning in general de-personalizes

education. In this regard, it is

critically important to recognize

issues of nomenclature: onlinelearning is hardly one thing. It

comes in a dizzying variety of

avors.
http://www.mellon.org/internet/news_publications/publications/romanes.pdfhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.nytimes.com/2011/10/03/opinion/the-university-of-wherever.html?pagewanted=allhttp://www.mellon.org/internet/news_publications/publications/romanes.pdf


9/53


setings. A key poin, i an obvious one, is ha here is no one approach ha isrigh or every suden or every seting. In imporan respecs, he online learn-ing markeplace reflecs he diversiy o A merican higher educaion isel.12

As resisan as some may si ll be even o hink abou seeking produciv iy gainsin order o reduce eaching coss, here is simply no denying he need o lookmore closely han ever beore a he relaion beween cerain oupus (approxi-maed, or example, by degrees conerred) and inpus (he mix o labor and

capial ha defines educaional producion uncions).13

I is essenial ha helimied resources available o higher educaion be used as effecively as possible.For hese reasons, he research repored here is concerned wih boh educaionaloucomes and coss, seen as wo blades o he scissors.

Organization of This Report

Te nex secion o his repor describes a wo-year effor we have made o esrigorously he learning oucomes achieved by a prooype ineracive learningonline course delivered in a hybrid mode (wih some ace-o-ace insrucion) onpublic universiy campuses in he Norheas and Mid-Alanic. Beore preseningour findings, we devoe space o explaining our randomizaion mehodology

boh because he findings can only be undersood agains he backdrop o he

mehodology and because he research design may be o independen ineres osome readers.14Tis secionwhich conains he resuls o he main par o ourresearchis ollowed by a brieer discussion o he poenial cos savings hacan conceivably be achieved by he adopion o hybrid-orma online learningsysems. We explain why we avor using a cos simulaion approach o esimaepoenial savings, bu we relegae o Appendix B he highly provisional resuls

we obained by employing one se o assumpions in a cos simulaion model. Weend he main body o he repor wih a shor conclusion ha considers barrierso he adopion o online learning sysems ha are ruly ineracive, seps hamigh be aken o overcome such barriers, and he need o ake a sysem-wideperspecive in addressing hese exremely imporan issues.

12 As Henry Bienen (president emeritus of Northwestern and chairman of the board of Rasmussen College, a

for-profit unive rsit y, as well as chairman of ITHAKA) points out, f or many institutions seeking to address the

needs of adult learners and others who are not candidates for places in traditional colleges and universi-

ties, there is no choice: online education, in some form, is the only way that many people can acquire more

skills and earn a college degree, the returns on which have skyrocketed in the past three dec ades. But online

education is also increasingly common in colleges and universities that educate traditional students. It is

seen as a revenue-generating force in many institutions, both four-year and two-year and both public and

private. See Barriers to Adoption of Online Learning Systems in U.S. Higher Education by Bacow et al.13 Some argueand we heartily agreethat the output of higher education has broader dimensions and

includes both research results and also the contribution that the entire system of higher education makes

to the effective functioning of a democratic society. But it will not do to allow emphasis on these larger (and

hard-to-measure) contributions to obscure the need to look carefully, and with a somewhat skeptical eye, at how

effectively institutions utilize resources to achieve straightforward aims such as improving graduation rates.

14 Readers interested in methodology may be especially interested in Appendix C to this report, which contains

a detailed discussion of lessons learned from our experience in carrying out this complicated research

project. We wish only that we had had access to this recitat ion of what to do and what not to do be fore we

started on this adventure! We learned many of these lessons the hard way.


10/53


Educational Outcomes in Public Universities

Te firs and mos ambiious par o our research was direced a assessing heeducaional oucomes associaed wih wha we erm ineracive learning onlineor ILO. By ILO we reer o highly sophisicaed, ineracive online coursesin which machine-guided insrucion can subsiue or some (hough no usu-ally al l) radiional, ace-o-ace insrucion. Course sysems o his ype akeadvanage o daa colleced rom large numbers o sudens in order o offer eachsuden cusomized insrucion, as well as allow insrucors o rack sudensprogress in deail so ha hey can provide heir sudens wih more argeed andeffecive guidance. As several leaders o higher educaion made clear o us inpreliminary conversaions, absen real ev idence abou learning oucomes here isno possibiliy o persuading mos radiional colleges and universiies, and espe-cially hose regarded as hough leaders, o push hard or he inroducion o ILOechnologies ha begin o subsiue machine-guided insrucion or radiionalorms o eaching in appropriae setings.

We se ou o provide a leas enaive answers o hese quesions:

Can sophisicaed, ineracive online courses be used o mainain or improvebasic learning oucomes (masery o course conen, compleion raes, andime-o-degree)?

Are hese courses as effecive, or possibly more effecive, or minoriy andlow-socioeconomic-saus sudens and or oher groups subjec o sereoypehrea?

Are hey equal ly effecive wih no-so-well-prepared sudens and well-pre-pared sudens?

Are hey equal ly effecive in a var iey o campus setingscommuniy col-leges versus our-year colleges, commuer colleges versus colleges wih more

sudens in residence?

Research Design

In hinking abou research design, we began by looking closely a exisingresearch. Tere have been lierally housands o sudies o online learning, buunorunaely he grea majoriy are deficien in one way or a noherofen orreasons beyond he conrol o he principal invesigaors. 15Very ew look direclya he eaching o large inroducory courses in basic fields a major public univer-siies, where he grea majoriy o undergraduae sudens pursue eiher associaeor baccalaureae degrees, presumably because very ew ILO courses have been

15 A detailed summary of existing research has been compiled by our staff (especially Lack); but it is too lengthy

to include here. See Current Status of Research on Online Learning in Postsecondary Education by Bowen

and Lack.

The most ambitious part of our

research was directed at assessing

the educational outcomesassociated with what we term

interactive learning online or

ILO. By ILO we refer to highly

sophisticated, interactive online

courses in which machine-guided

instruction can substitute for some

(though not usually all) traditional,

face-to-face instruction.


11/53


offered in hese setings.16Very ew o he sudies use randomized assignmenechniques o creae reamen and conrol groups ha can be used o reduceoherwise ubiquious selecion effecs ha make i hard o inerpre findings.

o overcome hese limiaions, we decided o work wih seven insances o aprooype ILO saisics course a six public universiy campuses (including woseparae courses in wo deparmens on one campus) ha agreed o cooperae ina careu lly designed research projec uilizing random assignmen echniques.

wo o hese campuses are par o he Sae Universiy o New York (SUNY);wo are par o he Universiy o Maryland; and wo are par o he Ciy Univer-siy o New York (CUNY). Te individual campuses involved in his sudy were,rom SUNY, he Universiy a A lbany and SUNY Insiue o echnology; romhe Universiy o Maryland, he Universiy o Maryland, Balimore Couny andowson Universiy; and, rom CUNY, Baruch College and Ciy College. Teseven courses, wih heir all 2011 enrollmens, are shown in able 1.

We also atemped o include hree communiy colleges in New York andMaryland. We were ulimaely unable o include daa rom hese campuses inour sudy or several reasons. A one o he hree communiy colleges, muliplechanges in leadership compromised he implemenaion o he randomizedresearch proocol. A he second communiy college, a large number o sudyparicipans never ook he course, and among hose who did, almos a quar-er swiched ino a orma differen rom he one o which hey were randomlyassigned. Addiionally, daa on final exam and sandardized es scores wereunavailable or a subsanial proporion o his campus sudy paricipans. A hehird communiy college, much o he daa were provided oo lae o incorporaeino our primary analysis. We srongly cauion readers agains assuming ha hefindings repored here or our-year colleges necessarily apply o communiy col-leges. Vigorous effors nowihsanding, we were unable o obain hard evidenceon his key quesion.

16 Our focus on students attending public institutions is not meant to denigrate the importance of either the

private non-profit sector o r the for-profit sector. Nor is it meant to denigrate professional programs aimed

at working adults. But it is the public colleges and universities, which educate more than three-quarters of

undergraduates at degree-granting institutions (according to the College Boards 2011 report, cited above),

that face the most consequential challenges in raising attainment rates and closing achievement gaps while

simultaneously reducing costs and restraining tuition increases.


12/53


TABLE 1. PARTICIPATING COURSES/INSTITUTIONS, FALL 2011

Course Enrollment Study Participants

Institution A 850 97

Institution B 877 229

Institution C 235 92

Institution D 86 16

Institution E, Department 1 337 31

Institution E, Department 2 473 50

Institution F 188 90

Total 3,046 605

Notes: Study participants are students who consented to be in our study and were randomly assigned to a traditional or

hybrid format of the introductory statistics class.

We do no claim ha hese six campuses are a sais ical ly valid sample o evenpublic higher educaion, never mind all o higher educaion. Bu his se o si xdoes include: (a) major urban universiies wih large commuing populaionso sudens, as well as universiies wih more residenial sudens; and (b) largenumbers o minoriy sudens and sudens rom low-socioeconomic-saus ami-

lies (as shown in ables 2 and 3). Tus, he populaion o insiuions and su-dens in he sudy is boh large enough and diverse enough o allow us o exploremos o he quesions lised above in he conex o our-year public insiuions.

More specifically, his research was designed o es as rigorously as possiblehe learning effeciveness o a paricular ineracive saisics course developeda Carnegie Mellon Universiy (CMU)viewed as a prooype o oher ILO

The population of institutions

and students in the study is both

large enough and diverse enoughto allow us to explore most of

the questions listed above in

the context of four-year public

institutions.


13/53


coures.17While he CMU course can be delivered in a ully onl ine environmen,in his sudy i was used in a hybrid mode in which mos o he insrucion wasdelivered hrough he ineracive online maerials, bu he online insrucion

was supplemened by a one-hour-per-week ace-o-ace session in which sudenscould ask quesions or be given a rgeed assisance.

Te exac research proocol varied by campus in accordance wih local policies,pracices, and preerences, and we describe hese proocols in deail in Appendix

able A1, and on Ihaka S+Rs websie where here is a narraive descripion;Appendix able A1 also presens summar y daa on enrollmens and secion sizesin each orma (ofen he hybrid-orma secions were somewha smaller han heradiional-orma secions). Te general procedure ollowed was: 1) a or beorehe beginning o he semeser, sudens regisered or he inroducory sais-ics course were asked o paricipae in our sudy, and modes incenives wereoffered;182) sudens who consened o paricipae filled ou a baseline survey;3) sudy paricipans were randomly assigned o ake he class in a radiional orhybrid orma; 4) sudy paricipans were asked o ake he CAOS es o sais i-cal lieracy19a he beginning o he semeser; and 5) a he end o he semeser,

17 We prefer the ILO acronym to others, including the OLI acronym used by CMU to stand for Open LearningInitiative. The term ILOfor interactive learning onlineis not specific to CMUs suite of courses, and

ILO emphasizes the interactive features of this kind of online learning. This is in contrast with more com-

mon types of online learning which largely mimic classroom teaching without taking advantage of the unique

online environment to provide added value, that is, anything beyond t hat which can be achieved in a physi-

cal classroom.

The CMU statistics course (which can be accessed at http://oli.web.cmu.edu/openlearning/) includes textual

explanations of concepts and an inventory of worked examples and practice problems, some of which require

the students to manipulate data for themselves using a statistical software package. Both the statistics

course and other courses in the OLI suite were originally intended to be comprehensive enough to allow stu-

dents to learn the material independently without the guidance of an instructor; since it was developed, how-

ever, the statistics course has been used at a variet y of higher education institutions, sometimes in a hybrid

mode. (Taylor Walsh describes the history of the development o f this course, which was financed largely by

the Hewlett Foundation over a number of years, in her 2010 book Unlocking the Gates: How and Why Leading

Universities Are Opening Up Access to Their Courses,Princeton University Press, 2010.) Among the main

strengths of the CMU statistics course is its ability to embed interactive assessments into each instructional

activity, and its three key feedback loops: system to student, as the student answers questions; system to

teacher, to inform student-instructor interactions; and system to course developer, to identify aspec ts of the

course that can be improved. In addition to of fering assessments to measure how well students understand a

particular concept, the CMU course also asks students to complete self-assessments, to give the instruct or

and/or learning scientists a sense of how well students think they understand t he concept. However, while

instructors can delete and re-order modules, CMU does not offer much oppor tunity for customization, nor is

the course adaptive in terms of redirect ing students to extra prac tice sessions or additional reading if their

incorrect answers indicate that they do not understand a concept and need more help. Thus, although the

CMU statistics course is cer tainly impressive, we refer to it as a proto type because we believe it is an early

representative of what will likely be a wave of even more sophisticated systems in the not-too-distant future.18 See Appendix A for a description of the research protocol and incentives used on each campus.

19 The CAOS test, or Comprehensive Assessment of Outcomes in Statistics, is a 40-item multiple-choice

assessment designed to measure students sta tistical literacy and reasoning skills. One characteristic of

the CAOS test is that (for a variet y of reasons) scores do not increase by a large amount over the course

of the semester. Among students in our study who took t he CAOS test at both the beginning and end of the

semester, the average score increase was 5 percentage points. For more information about the CAOS test,

see https://app.gen.umn.edu/artist/caos.html, or delMas, Robert, Joan Gar field, Ann Ooms, and Beth Chance,

Assessing Students Conceptual Understanding After a First Course in Statistics, 6.2 (2007): 28-58,

accessed July 28, 2010, http://www.stat.auckland.ac.nz/~iase/serj/SERJ6(2)_delMas.pdf.
http://oli.web.cmu.edu/openlearning/https://app.gen.umn.edu/artist/caos.htmlhttp://www.stat.auckland.ac.nz/~iase/serj/SERJ6%282%29_delMas.pdfhttp://www.stat.auckland.ac.nz/~iase/serj/SERJ6%282%29_delMas.pdfhttps://app.gen.umn.edu/artist/caos.htmlhttp://oli.web.cmu.edu/openlearning/


14/53


sudy paricipans were asked o ake he CAOS es o saisical lieracy again,as well as complee anoher quesionnaire. Appendix able A2 provides henumbers o sudens on each campus who were randomized ino each ormaand who compleed each daa collecion insrumen.

Adminisraive daa on paricipaing and non-paricipaing sudens were gah-ered rom he paricipaing ins iuions daabases. Te baseline survey admin-isered o sudens included quesions on sudens background characerisics,

such as socioeconomic saus, as well as heir prior ex posure o saisics andhe reason or heir ineres in possibly aking he saisics course in a hybridorma. Te end-o-semeser survey asked quesions abou heir experiences inhe saisics course. Sudens in sudy-affiliaed secions o he saisics courseook a final exam ha included a se o iems ha were idenical across all heparicipaing secions a ha campus (or, in he case o he campus ha had wodeparmens paricipaing in he sudy, all paricipaing secions in ha depar-men). Te scores o sudy paricipans on his common porion o he exam

were provided o he research eam, along wih background adminisraive daaand final course grades o all sudens (boh paricipans and, or comparisonpurposes, nonparicipans) enrolled in he saisics course in he all 2011

semeser. All o hese daa are described in deail on he Ihaka S+R websie,which also includes copies o he survey insrumens.

Our inenion was o provide a rigorous side-by-side comparison o specificlearning oucomes or sudens in his hybrid version o he saisics course andcomparable sudens in a radiionally-augh version o he same course. Werecognize, however, ha while we were reasonably successul in randomizingsudens beween reamen and conrol groups (see documenaion in he nexsecion o his repor), we could no randomize insrucors in eiher group andhus could no conrol or differences in eacher qualiy.20Tis is one reason,among ohers, ha we do no regard he research design o his projec as

20 Instructor surveys reveal that, on average, the instructors in traditional format sections were much more

experienced than their counterparts teaching hybrid-format sections (median years of teaching experience

was 20 and 5, respectively). Moreover, almost all of the instructors in the hybrid-format sections were using

the CMU online course for either the first or second time, whereas many of the instructors in the traditional-

format sec tions had taug ht in this mode for years. The experience-advantage, therefore, is clear ly in favor

of the teachers of the tradit ional-format sections. The questionnaires also revealed that a number of the

instructors in hybrid-format sections began with negative perceptions of online learning. In part for these

reasons, a leader of one of the se ts of institutions in this study believes that results for the hybrid-format

sections would be improved vis--vis results in the traditional-format sections if the exper iment were

repeated and instructors in the hybrid-format sections were better motivated and better trained. But this is,

of course, a conjecture.

Our intention was to provide a

rigorous side-by-side comparison

of specic learning outcomesfor students in this hybrid

version of the statistics course

and comparable students in a

traditionally-taught version of the

same course. However, while we

were reasonably successful in

randomizing students between

treatment and control groups, we

could not randomize instructorsin either group and thus could not

control for differences in teacher

quality.


15/53


anyhing close o perec.21Sill, his is he firs effor o which we are awareo carry ou he kind o randomized sudy o oucomes in large inroducorycourses on public universiy campuses ha we hink has been needed.

One wise decision we made was o conduc spring-erm pilos on as many cam-puses as possible in advance o he al l-erm 2011 research phase o he sudy

when we reaed oucomes as suiable or measuremen. Te spring-erm pilosidenified a number o pracical aspecs in which he sudy could be improved,

and a memo on lessons learned rom he spring-erm pilos is included in hisrepor as Appendix C.22

I remains only o add ha, as Appendix C illusraes, his is verydifficulresearch o do, in large par because so many deailshow bes o presenhe course, o recrui suden and aculy paricipans, o randomize sudens

beween reamen and conrol groups, o collec good daa including back-ground inormaion abou he suden paricipans, and o saisy InsiuionalReview Board requiremens in a imely wayneed o be worked ou wih heday-o-day involvemen o campus saff no direcly responsible o us. We havegrea respec or oher invesigaors who have coped wih hese problems, ofenin setings ouside higher educaion.

Findings

Te grea advanage oindeed, he main moivaion orconducing a ran-domized experimen is ha sudens in he reamen and conrol groups areexpeced o have he same average characerisics, boh observed and unob-served. Te resuls in able 2 indicae ha he randomizaion worked properly inha radiional and hybrid-orma sudens in ac have similar characerisics.Tere are a handul o small differences ha are saisically significan bu, ingeneral, he differences beween sudens augh in he radiional orma andsudens augh in he hybrid orma are no meaningul. 23

21 Randomization procedures were limited by the fact that Institutional Review Board (IRB) requirements

precluded randomization of students enrolled in the course without their consent. Instead, we had f irst to

use incentives to encourage students to participate in the study, with the understanding that they would then

be randomized between t reatment and control groups. We were able, however, to compare the character-

istics of participants and non-part icipants, and the two groups turned out to be very similar; see Table 3.

The study is, of course, limited in that it involves only a single course, but having a common hybrid course

across the six campuses (i.e. the CMU statistics course) controls fo r one source of variance in outcomes.

We deliberately chose the CMU statist ics course because we think that the greatest near-term opport unity

to take advantage of interactive online technologies is in introductory-level courses that serve large student

populations in fields in which there is more or less one right answer to most questions. Somewhat different

pedagogies would be needed, we suspect, in courses that are more value-laden and dependent on discussion

of various perspectives.22 We are indebted to James Kemple, now Executive Director of the Research Alliance for New York City Public

Schools, and formerly the Director of t he K-12 Education Policy division at MDRC, for much useful advice. D r.

Kemple has long experience with randomized trials. Lessons learned from the pilot s included how to present

the project, the ef fective use of modest incentives for participants, and techniques that could improve ran-

domization. We hope that others will benefit fr om our experience (see Appendix C) in mounting this research

project.

23 A regression of format assignment on all of the variables listed in Table 2 (and institution dummies) fails to

reject the null hypothesis of zero coef ficients for all variables (except the institution dummies) with p=0.12.

A Hotelling test fails to reject the null of no difference in means with p=0.27.


16/53


In addiion o esing he success o our effors o randomize sudens, able 2also serves o describe he populaion o sudens who par icipaed in our sudy.Tey are a very diverse group. Hal o he sudens come rom amilies w ihincomes less han $50,000 and hal are firs-generaion college sudens. Fewerhan hal are whie, and he group is abou evenly divided beween sudens wihcollege GPAs above and below 3.0. Mos sudens are o radiional college-goingage (younger han 24), are enrolled ull-ime, and are in heir sophomore or

junior year.

Tese sudens are a diverse group, bu do hey resemble he enire populaiono sudens enrolled in he inroducory sais ics courses included in our sudy?Sudy paricipans were randomly assigned o a secion orma, bu he sudyparicipans hemselves are a sel-seleced populaionbecause o InsiuionalReview Board requiremens only sudens who agreed o be in he sudy wererandomly assigned, and scheduling complicaions also l imied he populaiono paricipans. Overall, 605 o he 3,046 sudens enrolled in hese saisicscourses paricipaed in he sudy. An even larger sa mple size would have beendesirable, bu he logisical challenges o scheduling a leas wo secions (onehybrid secion and one radiional secion) a he same ime, so as o enable

sudens in he sudy o atend he saisics course regardless o heir (random-ized) orma assignmen, resriced our prospecive paricipan pool o helimied number o paired ime slos available. Also, as already noed, Insi-uional Review Boards required suden consen in order or researchers orandomly assign hem o he radiional or hybrid orma. No surprisingly, somesudens who were able o make he paired ime slos eleced no o paricipaein he sudy. All o hese complicaions nowihsanding, our final sample o 605sudens is by no means smalli is in ac quie large in he conex o his ypeo research.24

24 Of the 46 studies examined in the Means et al. (2009) meta-analysis, only 5 had sample sizes of over 400,

and of the 51 independent effect sizes the authors abs tracted, 32 came from studies with fewer than 100

study participants.

The students who participated

in our study are a very diverse

group. Half of the students come

from families with incomes less

than $50,000, and half are rst-

generation college students. Fewer

than half are white.


17/53


TABLE 2. RANDOMIZATION OF STUDY PARTICIPANTS

Traditional Hybrid Adj. Diff. Sig?

Male 46% 39% -7% +

Asian 24% 23% -1%

Black 14% 14% 0%

Hispanic 20% 14% -5% +

White 41% 46% 4%

Other/Missing 1% 3% 2%

Average Age 21.9 22.0 0.0

Age


18/53


TABLE 3. STUDENT CHARACTERISTICS BY STUDY PARTICIPATION

Participant Non-Part. Adj. Diff. Sig?

Male 42% 44% -1%

Asian 23% 17% 1%

Black 14% 13% 0%

Hispanic 17% 10% 3% *

White 44% 47% 6% *

Other/Missing 2% 13% -10% **

Average Age 21.9 21.6 -0.3

Age


19/53


Te resuls in able 3 indicae ha he 605 sudy paricipans, while no ul lyrepresenaive o all saisics sudens in any ormal sense, have broadly simi-lar characerisics. Tere are sais ically significan differences beween sudyparicipans and non-paricipans on several characerisics, bu mos o hedifferences are small in magniude. For example, paricipans are more likelyo be enrolled ull-ime, bu only by a margin o 90 versus 86 percen. Courseoucomes are also broadly similar, wih paricipans earning similar grades and

being only sl ighly less likely o complee and pass he course as compared onon-paricipans.

Our analysis o he daa is sraighorward; we compare he oucomes o su-dens randomly assigned o he radiional orma o he oucomes o sudensrandomly assigned o he hybrid orma. In a small number o cases4 perceno he 605 sudens in he sudyparicipans atended a differen orma sec-ion han he one o which hey were randomly assigned. In order o preservehe randomizaion procedure, we associaed sudens wih he secion y pe o

which hey were randomly assigned. Tis is someimes called an inen o reaanalysis. Under cerain assumpions, he effec o acually ak ing he course inhe hybrid orma (as opposed o jus being randomly assigned o do so) can be

calculaed by increasing our esimaes by 4 percen.25

Tis is someimes calledhe reamen on he reaed esimae, which in our sudy is very similar o heinen o rea esimae because mos sudens ook he course in he orma o

which hey were randomly assigned.

How did learning oucomes compare across he reamen and conrol groups?We firs examine he impac o assignmen o he hybrid orma, rela ive o heradiional orma, in erms o he rae a which sudens compleed and passedhe course, heir perormance on a sandardized es o saisics (he CAOSes), and heir score on a se o final exam quesions ha were he same in hewo ormas.26Our main resuls are summarized in Figure 1 (page 19), and heregression resuls are repored in Appendix able A3. 27We find no saisicallysignifican differences in learning oucomes beween sudens in he radiional and hybrid-orma secions. Hybrid-orma sudens did perorm slighly beterhan radiional-orma sudens on hree oucomes, achieving pass raes ha

25 The key assumption is that being randomly assigned to hybrid or traditional did not have an effect on student

outcomes independent of its e ffect on the format in which students were enrolled. This assumption would

be violated if, for example, students hoped for a certain outcome of the random assignment and were disap-pointed when they did not get their preferred assignment, which in turn caused them to do worse in

the course.

26 All of our results control for course-specific dummy variables, since students were randomized within cours-

es; these variables also control for unobserved student characteristics that are constant within institutions.

However, we obtain similar results when we do not control for institution dummies, as would be expec ted

given that the probability of being assigned to the hybrid section was constant across courses (50%).

27 Note that the pass rate in Figure 1 and Appendix Table A3 cannot be used to calculate the percentage of stu-

dents who failed the course because the non-passing group includes students who never enrolled or withdrew

from the course without receiving a grade.


20/53


wereabou hree percenage poins higher, CAOS scores abou one percenagepoin higher, and final exam scores wo percenage poins higherbu none ohese differences passes radiional ess o saisical significance.28

Figure 1. Effect of Hybrid Format on Student Learning Outcomes

0 10 20 30 40 50 60 70 80%

Traditional

Hybrid

PassRate

CAOSPost-Test

FinalExam

54.7%

56.7%

76.4%

79.7%

46.9%47.5%

Notes: None of the traditional-hybrid differences above were statistically signicant at the 10% level. See Appendix Table

A3 for more information about the results depicted here.

I is imporan o repor ha hese differences (or raher, he lack o saisicallysignifican differences) are ai rly precisely esimaedsee boh he acual coe-ficiens and he small sandard errors o he effec es imaes repored in Appen-dix able A3.29In oher words, we can be quie confiden ha he averageeffecs were in ac close o zero. As we explain shorly, we also find ha he same

basic resuls hold or subgroups, and ha disribuions o key oucomes are verysimilar or boh he reamen and conrol group sudens. One commenaor,Michael S. McPherson, presiden o he Spencer Foundaion, observed ha wha

we have here are quie precisely esimaed zeros.

Ta is, i here had in ac been pronounced differences in oucomes beweenradiional-orma and hybrid-orma groups, i is h ighly likely ha we would

have ound hem.30

Our findings are srik ingly differen in his consequenialrespec rom a hypoheical finding o no significan difference which resuledrom a coefficien o some magniude accompanied by a very large sandard erroror by big differences in he disr ibuions o oucomes. A hypoheical finding ohis k ind would mean, in effec, ha we don know much: ha he rue resulscould be almos anywhere.

28 The effect on CAOS test scores in standard deviation units (using the distribution of the control group) was

0.05. We also examined performance using item-level CAOS post-test data. Specifically, we grouped the 40

items into the 20 items on which delMas et al.s (2007) national sample of s tudents exhibited significant

growth (over the course of a semes ter) and the remaining 20 items. We found similar hybrid-format effec tsfor the two g roups o f items.

29 We cluster standard errors by section in order to capture section-specific shocks to student outcomes (such

as the quality of the instructor). Students who were randomly assigned but never enrolled in the course are

grouped as a section within each course for the purpose of computing clustered st andard errors.

30 Some degree of caution is warranted in interpreting the results for the CAOS post-test because the average

students CAOS score only increased by five percentage points over the course of the semester (among

students who took both the pre-test and the post-test). This may have resulted in part f rom some students

not taking the CAOS test seriously because, in most cases, i t was not part of their grade. However, it is

reassuring that the results for the CAOS test are consistent with results fo r pass rates and final exam scores.

In other words, we can be quite

condent that the average effects

were in fact close to zero. We alsond that the same basic results

hold for subgroups, and that

distributions of key outcomes are

very similar for both the treatment

and control group students.


21/53


Tese findings conrol or suden characeris ics, including race/ehniciy,gender, age, ull-ime versus par-ime enrollmen saus, class year in college,parenal educaion, language spoken a home, and amily income. Tese conrolsare no sricly necessary since sudens were randomly assigned o secion or-ma, bu we include hem in order o increase he precision o our resuls and oconrol or any remaining imbalance in observable characerisics. However, weobain nearly idenical resuls when we do no include hese conrol variables

jus as we would expec given he apparen success o our random assignmenprocedure.

Our resuls are also robus o a variey o alernaive mehodologies used oanalyze he experimenal daa. Tese resuls are repored in Appendix able A4,and one is worh highlighing. A limiaion o our main resuls or CAOS pos-es and final exa m scores is ha we only observe hese oucomes or sudens

who compleed he course and ook hese exams. Tis is unlikely o be a signifi-can limiaion given ha we do no find any significan effecs o secion ormaon course compleion and pass raes. Bu as an addiional check, we assignedsudens or whom we did no observe a CAOS pos-es score heir score on heCAOS pre-esin oher words, we assumed ha heir score did no change over

he course o he semeser. Sudens who did no ake eiher he pre-es or hepos-es were assigned he average pre-es score a heir insiuion. Te resul-ing se o real and impued pos-es scores yielded very similar resuls o hoseobained using only he real daa.

Te lack o differences in mean oucomes beween ormas could mask differ-ences in he disribuion o oucomes.31Figure 2 (page 21) shows ha his is nohe case or CAOS pos-es scores. Te disribuions o scores or radi ionaland hybrid orma sudens are largely similar, alhough scores are sl ighly morespread-ou or hybrid-orma sudens. We obain a similar finding or final examscores (no shown).32(Tis kind o comparison o disribuions is no possible orpass raes, which only ake on a value o 0 or 1 or an individual suden.)

Resuls broken down by individual insiuion (Appendix able A5) do noreveal any noeworhy paterns. Tese resuls are much noisier because heyare based on smaller numbers o sudens, bu hey do no indicae ha hehybrid orma was paricularly effecive or ineffecive a any individual insiu-ionwih he possible excepion o Insiuion F, where coefficiens are posiiveacross all our oucomes, alhough only saisically significan in he case o oneoucome.

31 We are indebted to Stephen Stigler, a professor at the University of Chicago and a member of the ITHAKA

board, for emphasizing to us the importance of considering this question.

32 In general, results that use final exam scores should be interpreted cautiously given limitations in these

exams and their implementation. Some institutions included only a handful of questions that were common

across the sections of the course (and we only use data from the common questions). At one institution,

common questions were administered to some students a fter the end of the semester because the actual

final exam only included common quest ions in t wo out o f six sections . At ano ther ins titut ion, fina l exam data

were not available for the students of two instruct ors (covering three out of six traditional-format sections).

Excluding either or both of these institutions produces similar results, but it should be noted that the e ffect

on final exams in standard deviation units is substantial in size (0.19; see Appendix Table A4) and imprecisely

estimated.

Results broken down by

individual institution do not

reveal any noteworthy patterns.


22/53


Figure 2. Distributions of CAOS Post-Test Scores

Traditional

Hybrid

Traditional

4

Density

CAOS Post-Test Score

Kernel =epanechinov, bandwidth=0.03335

2

0

0 .2 .4 .6 .8

Hybrid

We also calculaed resuls separaely or subgroups o sudens defined in ermso var ious characeris ics, including race/ehniciy, gender, parenal educa-ion, primary language spoken, CAOS pre-es score, hours worked or pay, andcollege GPA. We did no find any consisen evidence ha he hybrid-orma

effec varied by any o hese characerisics (Appendix able A6).33

Tere wereno groups o sudens ha benefied rom or were harmed by he hybrid ormaconsisenly across muliple learning oucomes.

Our main resuls provide compelling evidence ha, on average, sudens learnedjus as much in he hybrid orma as hey would have had hey insead aken hecourse in he radiional ormawih learning measured in radiional ways,in erms o course compleion, course grades, and perormance on a naional eso saisical lieracy. Tis seemingly bland resul is in ac very imporan, in l igho perhaps he mos common reason given by aculy and deans or resising heuse o ILO-ype insr ucion: We worry ha basic suden learning oucomes(pass raes and masery o conen) will be hur, and we won expose our su-

dens o his risk . Te research repored here suggess srongly ha such worriesare no well ounded.

We also examined how much sudens liked he hybrid orma o he course rela-ive o radiional orma sudens (Figure 3 [page 22] and Appendix able A7).

We ound ha sudens gave he hybrid orma a modesly lower overal l rainghan he one given by sudens aking he course in radiional orma (he raing

was abou 11 percen lower). By similar margins, hybrid sudens repored eel ing

33 The one exception is our finding that completion and pass rates were significantly higher in the hybrid course

for st udents with family incomes o f at least $50,000 per year, but not for s tudent s with family inc omes o f

less than $50,000 per year. However, we hesitate to att ach much significance to this result given that we do

not find such a pattern for our other measure of socioeconomic status (parental education) or for measures

of academic preparation. This finding could be the result of random noise in the coef ficients (especially given

the large number of coefficients reported in Appendix Table A6).


23/53


ha hey learned less and ha hey ound he course more difficul. 34Tese hreedifferences, hough modes in size, were saisically significan a he 10 percenlevel. Bu here were no saisically significan differences in sudens repors ohow much he course raised heir ineres in he subjec mater.

Figure 3. Effect of Hybrid Format on Student Evaluations of Course

0.0

0.5

1.0

1.5

2.0

2.5

DifficultyAmountLearned

RaisedInterest

OverallRating

Traditional

Hybrid

Raiting(0

4)

2.3

2.0+

2.0+

2.5+

1.7 1.7

2.2 2.3

Notes: + indicates traditional-hybrid difference is statistically signicant at the 10% level. See Appendix Table A7 for moreinformation about the results depicted here.

A leader o one o he universi ies ha ac ively paricipaed in our sudyopined ha a deec o he CMU prooype course is ha i has no addiciveor Disney-like appeal; i was, as his person pu i, designed by cogniivescieniss (no offense inended!). In conras, some sudens in he radiionalorma may have been reaed o an occasional colorul sory, personal recol-lecions o he insrucor, or oher reamens someimes used by aculy oimprove sudens opinions o heir course. 35

We also asked sudens how many hours per week hey spen outside of class

working on he sais ics class. Hybrid-orma sudens repored spending 0.3hour more each week, on average, han radiional-orma sudens. Tis differ-

34 Students responses to the open-ended questions on the end-of-semester surveys indicate that many

students in the hybrid format would have liked more face-to-face time with t he instructor than one hour

each week; others felt that the instructor could have bet ter used the face-to-face time to make the weekly

sessions more structured and/or helpful in explaining the material and going over concepts students did not

understand. A number of students in the hybrid course also indicated they would have benefited from more

practice problems or examples, and many were frustrated by the difficulty o f checkpoint assessments in the

course and by problems they encountered using the statistical software packages to complete assignments.

35 The question of what is really going on herewith no differences in learning outcomes as measured con-

ventionally combined with a (to be sure, small) difference in qualitative assessmentsrelates to a larger

literature on measured learning outcomes versus more subjective measures of student satisfact ion with

online or hybrid courses relative to their satisfaction with face-to-face courses. Studies (some more rigorous

than others) pert aining to the latter t opic abound; a few examples include: Hannay, Maureen, and Tracy

Newvine. Perceptions of Distance Learning: A Comparison o f Online and Traditional Learning. 2.1 (2006):

1-11. Accessed April 24, 2012. http://jolt.merlot.org/documents/MS05011.pdf; Horspool, Agi, and Carsten

Lange. Applying the Scholarship of Teaching and Learning: Student Perceptions, Behaviours and Success

Online and Face-to-Face. 37.1 (2011): 73-88. Ac cessed April 24, 2012. http://www.tandfonline.com/doi/abs

/10.1080/02602938.2010.496532; and Meyer, Katrina A. Student Perceptions of Face-to-Face and Online

Discussions: The Advantage Goes To . .4: 53-69. Accessed April 24, 2012. http://sloanconsortium.org/

jaln/v11n4/student-percept ions-face-face-and-online-discussions-advantage-goes.

Our results indicate that hybrid-

format students took about

one-quarter less time to achieve

essentially the same learning

outcomes as traditional-format

students.
http://jolt.merlot.org/documents/MS05011.pdfhttp://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://sloanconsortium.org/jaln/v11n4/student-perceptions-face-face-and-online-discussions-advantage-goeshttp://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://www.tandfonline.com/doi/abs/10.1080/02602938.2010.496532http://jolt.merlot.org/documents/MS05011.pdf


24/53


ence, which is no saisically significan, implies ha, in a course where a radi-ional secion mees or 3 hours each week and a hybrid secion mees or 1 hour,he average hybrid-orma suden would spend 1.7 less hours each week in totaltime devoed o he course, a difference o abou 25 percen. Tis resul is con-sisen wih oher evidence ha ILO-ype ormas do succeed in achieving hesame learning oucomes as radiional-orma insrucion in less imewhichpoenially has imporan implicaions or scheduling and he rae o coursecompleion.36

In sum, our resuls indicae ha hybrid-orma sudens ook abou one-quarerless ime o achieve essenially he same learning oucomes as radiional-ormasudens. Te hree main limiaions o his analysis are: (1) we were no able orandomly assign insrucors o secion ormaswhich would have been di-ficul, i no impossible, o do, especially in a small scale sudy, or o conrol ordifferences in how radiional-orma secions were augh;37(2) he limiaionso he CMU prooype o an ILO courseno cusomizaion and no addiciveeaures; and (3) our inabiliy o repor resuls or communiy colleges. Despiehese limiaions, hese resuls reflec a serious, rigorous, assessmen o he rela-ive efficacy o echnology-enhanced learning (ILO-syle hybrid insrucion)

compared o he radiional mode o insrucion. Tey are, we believe, he besevidence available o dae on an imporan se o quesions. Tere is, w ihoudoub, much more research ha can and should be carried ou bu, a he mini-mum, his sudy suppors a no-harm-done conclusion regarding a leas onecurren prooype o an ILO sysem.

36 The authors of this paper, interested to see whether the hybrid course might enable students to learn the

material in the statistics course in a shorte r period of time, conducted a separate, quasi-experimental study

in the summer of 2011, involving one of the campuses used in our larger fall 2011 study. This summer probe

occurred over two shortened (five or six-week) summer sessions; while we collected a large amount of

data and used numerous controls in our analysis, we did not randomly assign students to the hybrid or theface-t o-face format so that we could obtain lar ger sample sizes (from what was already a much smaller pool

of students taking the course during the summer). The results of the summer study revealed no sta tistically

significant differences in the percentage of students who passed t he course, in final exam grades, or in the

end-of-semester CAOS test results, bet ween the students who took the course in a hybrid format and stu-

dents who took the course in a face-to-face format. A separate study by Marsha Lovett, associate director of

the Eberly Cente r for Teaching Excellence, and her colleagues at CMU produced similar findings. In that study,

which was conducted at CMU, the performance of students who were randomly assigned to the face-to-face

format of a stat istics course, which met four times a week for 50 minut es each time, was compared w ith

the performance of s tudents who were randomly assigned to a hybrid format, which met twice a week for 50

minutes each time. The researchers found little dif ference in the amount of time students repor ted spending

on the course outside of class each week (about 2.8 hours for the hybrid-format student s, compared with

about 2.7 hours per week for face-to-face students). In addition, students also were able to learn in eight

weeks the same amount of material that students would ordinarily take 15 weeks in a face-to-face formatto learn. In this respect, the use of the CMU course could be said to increase learning efficiency. (For more

information about this study, see Lovet t, Meyer, and Thilles 2008 art icle in the , entitled The Open Learning

Initiative: Measuring the effectiveness of t he OLI statistics course in accelerat ing student learning, available

online at http://jime.open.ac.uk/2008/14.)

37 One commentator, Michael McPherson, noted that right now, quite apart from any use of online technologies,

very different instructional methods are used to teach introductory statistics courses even within the same

universitynever mind across universities. But little ef fort appears to be made to compare learning out-

comes and cost effectiveness across different approachesa point Derek Bok, former president of Harvard

who continues to write on this subject, keeps emphasizing.
http://jime.open.ac.uk/2008/14http://jime.open.ac.uk/2008/14


25/53


Costs and Potential Savings

I is an open quesion wheher subsequen progress in improving hybrid ILOcourses o his k indin paricular, effors o achieve greaer cusomizaionopporuniies or aculy,38and o make i more un (more addicive) or bohsudens and aculy o use he sysemwill lead o more posiive conclusionsconcerning learning effeciveness. We jus don know. Digial learning is sil l inearly days, and i is enirely possible ha uure versions o hybrid courses wi llno jus mainain he same learning oucomes bu increase suden learning rela-ive o he saus quo. Bu we do know enough now o jusiy careul exploraiono poenial cos savings.

Produciviymeasured as oupus divided by inpushas been increasedby he use o echnology in oher secors o he economy, ofen resuling inincreased oupu. Our muli-campus sudy o learning oucomes in undergradu-ae educaion has shown ha a leading prooype hybrid learning sysem did nolead o a saisically significan increase in oupus (suden learning), bu couldpoenially increase produciviy noneheless by using ewer inpusherebyachieving cos savings. Unil now, echnology-induced produciviy gains in

higher educaion have been aken mainly in he orm o increased oupumoreand aser research, and so on; he ime may be a hand when cos savings should

be sough and emphasized.39

Te cos blade o he scissors is a leas as chal lenging o sudy as he learningblade. Tis may be one reason why so ew sudies have paid much atenion ocoss. (Carol wiggs work wih he Naional Cener or Academic ransorma-ion projec is a noable excepion.40) A firs blush, i would seem o be sraigh-orward o compare he side-by-side coss o he hybrid-online version o hesaisics course and he radi ional version. No so. Our early effors o do jusha were unsuccessul. Te big problem, we learned, is ha conemporaneouscomparisons can be near-useless in projecing seady-sae coss because hecoss o doing almos anyhing or he firs ime are very differen rom he cosso doing he same hing numerous imes. Ta admoniion is especially rue inhe case o online learning. Te cos implicaions o some educaional iner-

venions can be measured immediaely and wih relaively litle difficuly. Forexample, he higher cos associaed wih a decrease in he size o a class is simplyhe cos o he addiional insrucors and space required o accommodae a largernumber o classes wih ewer sudens in each one. Tis cos w ill be more or lesshe same in he firs year he inervenion is implemened as in he enh year.

In he case o hybrid learning, however, here are subsanial sar-up cossha have o be considered in he shor run bu are l ikely o decrease over ime,

hereby making shor-erm coss significanly greaer han long-erm coss. Forexample, he developmen o sophisicaed hybrid courses wil l be a cosly efforha would only be a sensible invesmen i he sar-up coss were eiher paid or

by ohers (oundaions and governmens) or shared by many insiuions and

38 The pervasive desire for more ability to customize is a key finding of Barriers by Bacow et al. (see pages

21-22 in particular).

39 This is another conclusion of the Barriers study by Bacow et al.; see pages 22 and 24-25 in particular.

40 See the National Center for Academic Transformation website at www.thencat.org.

We conceptualize the research

question here not as how much

will institutions save right now by

shifting to hybrid learning? but

rather as under what assumptions

will cost savings be realized, over

time, by shifting to a hybrid format,

and how large are those savings

likely to be?
http://www.thencat.org/http://www.thencat.org/


26/53


amorized over ime. Tere are also ransiion coss enailed in moving rom heradiional, mosly ace-o-ace model (ha may, however, employ some elemenso simple online models, such as video-aped lecures or homework assign-mens online) o a hybrid model ha akes advanage o more sophisicaed ILOsysems employing machine-guided insrucion, cogniive uors, embeddedeedback loops, and some orms o auomaed grading. Insrucors need o berained o ake ull advanage o such sysems. Tere may also be conracual lim-is on secion size ha were designed wih he radiional model in mind bu hado no make sense or a hybrid model. I is possible ha hese consrains can bechanged in a nex round o conrac negoiaions, bu ha oo wi ll ake ime.

o overcome (or avoid!) hese problems, we hink here is much o be said orcarrying ou simulaed cos probes, and we made a very rough atemp o do

jus his on hree o he campuses included in he lea rning oucomes par ohe sudy. We concepualize he research quesion here no as how much willinsiuions save righ now by shifing o hybrid learning? bu raher as under

wha assumpions wi ll cos savings be realized, over ime, by shifing o a hybridorma, and how large are hose savings likely o be? Our basic approach waso sar by looking in as much deail as possible a he acual coss o eaching a

basic course in radiional orma (usually, bu no always, he saisics course)in a base year. Ten, we worked wih leaders on hese campuses o simulae heprospecive, seady-sae coss o a hybrid-online version o he same course,looking hree o five years ino he uure. Tese ex ploraory simulaions were

based on explici assumpions, especially abou saffing, ha were incorporaedino spreadsheeswhich in urn allowed us o see how sensiive our resuls

were o variaions in key assumpions. We ocused heavily on personnel coss,because o boh heir imporance and our abiliy o examine hem wih someprecision. Oher coss, including space coss, were also considered. We hopedha he simulaions would, a he minimum, give us a leas a rough sense o hepoenial impac on coss o inroducing hybrid learning and, more specifically,

show us how much room here would be or insiuions o share cos savingswih acul y and sudens on a coninuing basis.

We ocus on insrucor compensaion because hese coss comprise a subsan-ial porion o he recurring cos o eaching and are he mos sraighorwardo measure. Space coss are also an imporan caegory o coss ha are likely o

be reduced by shifing o a hybrid learning model (he mos imporan caegoryin some siuaions), bu such coss are more difficul o measure accuraely ahe level o an individual course. A hybrid model also affords boh aculy andsudens significanly greaer scheduling flexibiliy, a poenially very imporan

benefi ha will no be capured by our simulaions. On he oher hand, here arealso oher y pes o coss ha we do no consider here, such as increases in inor-maion echnology (I) suppor coss associaed wih moving a signi fican shareo learning aciviies online. Such added coss can be ar rom riv ial.

We did exploraory simulaions or wo ypes o radi ional eaching models: (1)a model in which sudens are augh in secions o roughly 40 sudens per sec-ion; and (2) a model in which all sudens atend a common lecure and are henassigned o small discussion secions led by eaching assisans. We compare hecurren coss o each o hese radi ional eaching models o simulaed coss oa hybrid model in which more insrucion is delivered online, sudens atend

Our simulations illustrate that

hybrid learning offers opportunities

for signicant savings incompensation costs, but that the

degree of cost reduction depends

on the exact model of hybrid

learning used.


27/53


weekly ace-o-ace sessions wih par-ime insrucors, and he course is over-seen by a enure-rack proessor (wih adminisraive responsibiliies delegaedo a par-ime insrucor).

We have decided, however, no o presen he acual calculaions and resuls ohese simulaions in he body o his repor. Tey are oo speculaive and subjeco considerable variaion depending on how a paricular campus waned oorganize is eaching. Te danger o specious precision is grea, and i would be

wrong o atach much imporance o paricular numbers. Suffice i o say ha hecrude models we employed sugges savings in compensaion coss ranging rom36 percen o 57 percen in he all-secion model, and 19 percen in he lecure-secion model. Appendix B presens hese resuls and many more calculaionsand figures showing how sensiive poenial savings are as we vary assumpionsabou secion sizes and compensaion.

Tese simulaions illusrae ha hybrid learning offers opporuniies or sig-nifican savings in compensaion coss, bu ha he degree o cos reduciondepends (o course) on he exac model o hybrid learning usedespecially herae a which insrucors are compensaed and secion size. A large share o cossavings is derived rom shifing away rom ime spen by expensive proessors

oward boh machine-guided insrucion ha saves on saffing coss overall andoward ime spen by less expensive saff in Q&A setings. O course, enuredproessors canno be laid off in order o realize hese savings and, in any case,orce reducions are no required o save significan amouns o money. Insiu-ions ha ace pressures o expand enrollmen are in an especially good posiiono realize savings by shifing he mix o eaching ime. When more sudens areo be augh, hybrid models make his possible wihou increasing he demandsmade on enured aculy. Recruimen coss may hereby be reduced along wihcompensaion coss per suden, and debaes over mainaining commimenso exising aculy are avoided. Over ime, cerainly, saff size can be aleredhrough atriion. A lso, he ime o proessors can be reallocaed oward smaller,more advanced classeswhich many preer o each (such reallocaions mayno save he insiuion money, hough hey may improve he overall educaionalexperience o many sudens).

In hese simulaions, we have assumed ha he number o sudens in he coursewill remain consan. However, as already suggesed, many insiuions aceincreasing demand or places in heir classes. Te hybrid learning model is veryatracive in such circumsances or wo primary reasons: (a) less space is neededin general; and (b) hybrid courses provide boh sudens and eachers wihgreaer scheduling flexibiliy. Increased enrollmen can a lso lead o increasedcompensaion cos savings (per suden) because he fixed coss o he proessor

in charge o he course, and an adminisraive coordinaor, would be spread overa larger number o sudens. For he same reason, he larges savings w ill be real-ized in courses wih he larges enrollmen, all else equal.

Our simulaion approach underesimaes subsanially he savings rom mov-ing oward a hybrid model in many setings because we do no accoun or spacecoss. We are relucan o pu a dollar figure on space coss because capial cossare difficul o apporion accuraely down o he course level. However, i is moresraighorward o calculae he percenage change in he need or classroom


28/53


space ha would resul rom shi fing oward a hybrid model. Te hybrid coursemees or one hour each week, whereas he radiional course ypically meesor hree or our hours each week. Consequenly, he hybrid course requires 67percen o 75 percen less classroom use han he radiional course, assumingha he course is augh in secions, ha secion size is held consan, and hahe hybrid course does no have addiional space requiremens o is own, such asaddiional compuer labs.

In he shor run, insiuions canno sell or demolish heir buildings. However,in he long run, using hybrid models or some large inroducory courses wouldallow insiuions o expand enrollmen wihou a commensurae increase inspace cossa major cos savings (cos avoidance) relaive o wha insiuions

would have had o spend had hey sayed wih a radiional model o insrucion.An imporan poin here is ha he hybrid model need no jus save

sr ithaka interactive learning online at public universities

Documents