![Page 1: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/1.jpg)
How to Research MOOCs:
A Primer (with Results)
D. Christopher Brooks, Ph.D.
J.D. Walker, Ph.D.
20th Annual Online Learning Consortium
International Conference
29 October 2014
![Page 2: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/2.jpg)
![Page 3: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/3.jpg)
![Page 4: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/4.jpg)
(Ebben & Murphy, 2014)(Ebben & Murphy, 2014)
![Page 5: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/5.jpg)
![Page 6: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/6.jpg)
![Page 7: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/7.jpg)
Microwaves : Kitchens ::
MOOCs : Higher Education
![Page 8: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/8.jpg)
How to Research MOOCs
Your Background with MOOCs
How many of you…
• have participated in the design, teaching, or
support of a MOOC?
• work for a university that has offered courses
as MOOCs?
• work for a company that
offers a MOOC platform (EdX,
Coursera, Udacity, etc.)?
![Page 9: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/9.jpg)
![Page 10: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/10.jpg)
How to Research MOOCs
MOOCs at Minnesota: Background
• Offered about 12 MOOCs since summer
2013:
– Statistical Molecular Thermodynamics
– Social Epidemiology
– Health Informatics
– Global Food Systems/Sustainability
– Canine Theriogenology
– Resilience in Children
– Fundamentals of Fluid Power
– Creative Problem-Solving
• Total nominal enrollment: 333,495
![Page 11: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/11.jpg)
How to Research MOOCs
Why Researching MOOCs Is Hard
• Students not very
motivated
• Very diverse and
different student
population
• Lack of baseline
information
• Non-random attrition
![Page 12: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/12.jpg)
How to Research MOOCs
Advantages of
MOOC Research
• Strength in numbers
• Large amount of
non-self-report data
Kyle Bowen, http://classhack.com/post/76426264075/reallybigdata
![Page 13: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/13.jpg)
MOOC Research Progression:
Early Stages = Descriptive/Exploratory
S. Gross. “Blind Men and the Elephant Systems Thinking”
![Page 14: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/14.jpg)
MOOC Research
Early Stages: Descriptive/Exploratory
• What are MOOCs?
• Who takes MOOCs?
• Why do people take MOOCs?
• Who teaches MOOCs?
• What happens in MOOCs?
Kyle Bowen, http://classhack.com/post/41632460268/moocmart
![Page 15: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/15.jpg)
Early Stages: Descriptive/Exploratory
Duke University Bioelectricity White Paper
(Belanger & Thornton, 2013)
![Page 16: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/16.jpg)
Early Stages: Descriptive/Exploratory
Minnesota Faculty and Student Analysis
• Research questions:
– Time & effort
– Student characteristics
• Data sources:
– Time & effort diaries
– Faculty interviews
– Pre and post surveys
– Coursera dataD. Christopher Brooks, “OZ: Road Work”
![Page 17: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/17.jpg)
Planning Executing Total
Statistical Molecular
Thermodynamics 207.50 205.50 413.00
Sustainability &
Food Systems272.50 157.50 430.00
Interprofessional
Health Care
Informatics263.00 56.00 319.00
Social
Epidemiology114.50 74.50 189.00
Canine
Theriogenology for
Dog Enthusiasts19.92 52.35 72.27
Unidentified & TAs 234.95 242.00 476.95
TOTAL 1,112.37 787.85 1,900.22
AVERAGE 222.47 157.57 380.04
Early Stages: Descriptive/Exploratory
University of MinnesotaInstructor & TA Self-Reported Effort
![Page 18: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/18.jpg)
Early Stages: Descriptive/Exploratory
University of Minnesota:Faculty Experience: Overview
Personal Satisfaction
Despite considerable effort
Student & Professional Connections
Broad Reach
Despite low completion
rates
Professional Satisfaction
Keeping Things Fresh
Despite the public debate
Think about
Teaching
![Page 19: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/19.jpg)
Early Stages: Descriptive/Exploratory
University of Minnesota:Faculty Experience: Beyond the MOOC
BIG IDEAS
Student-Centered
Discussion & Activities
Brief Lecture
![Page 20: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/20.jpg)
Early Stages: Descriptive/Exploratory
University of Minnesota:Post-MOOC Follow-Up: Why Offer MOOCs?
• 90% materials reuse!
• “I could go on and on about
the benefits [teaching the
MOOC has] brought to my
students, and to my teaching
and research.”
Kyle Bowen, http://classhack.com/post/76426180711/five-
updated
![Page 21: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/21.jpg)
University-related reasons
(3.929)
Access-related reasons
(1.475)
Professional reasons
(1.312)
Enjoyment-related reasons
(1.029)
Early Stages: Descriptive/Exploratory
University of MinnesotaReasons for Enrolling: Factor Analysis
1. This subject is relevant to my academic field of study
2. This class teaches skills that will help my job/career
3. Because this course is offered by a prestigious
university
4. I think taking this course will be fun and enjoyable
5. I am not geographically close to educational
institutions
6. Traditional courses are too expensive
7. I was interested in taking a course with this professor
8. This course is offered by the University of Minnesota
9. General interest in the topic
10. To help me decide whether to take further
college/university classes
11. To make professional connections
12. To obtain a badge or certification that will be useful to
me professionally
• Total variance explained: 64.55%
• (Eigenvalues in parentheses)
![Page 22: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/22.jpg)
Early Stages: Descriptive/Exploratory
University of Minnesota:
Reasons for Enrolling
![Page 23: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/23.jpg)
Early Stages: Descriptive/Exploratory
ECAR Faculty and Student Surveys:MOOCs? What’s a MOOC?
![Page 24: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/24.jpg)
Early Stages: Descriptive/Exploratory
ECAR Faculty and Student Surveys:Support/Opposition of MOOCs in Higher Ed
![Page 25: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/25.jpg)
Early Stages:
Descriptive/Exploratory
ECAR Faculty
and Student
Surveys:
Student
Experiences with
MOOCs
![Page 26: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/26.jpg)
Early Stages: Descriptive/Exploratory
ECAR Faculty and Student Surveys:Undergraduate Degrees > MOOC Certificates
![Page 27: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/27.jpg)
![Page 28: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/28.jpg)
MOOC Research Progression:
Intermediate Stages = Correlational Analysis
https://xkcd.com/925/
![Page 29: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/29.jpg)
Intermediate Stages: Correlational Analysis
Minnesota Student Population Analysis: Reasons for Enrolling by English Proficiency
![Page 30: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/30.jpg)
Intermediate Stages: Correlational Analysis
Minnesota Student Population Analysis:Reasons for Enrolling by Retrospective Intent
![Page 31: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/31.jpg)
Intermediate Stages: Correlational Analysis
Minnesota Student Population Analysis: Reasons for Enrolling by Demographics
![Page 32: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/32.jpg)
• Strivers: outside US, male, younger, non-native
English, work/study in same field, professional
reasons
• Grazers: inside US, older, native English,
complete less, enjoyment reasons
Intermediate Stages: Correlational Analysis
Minnesota Student Population Analysis:
Two Populations of Students
![Page 33: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/33.jpg)
Intermediate Stages: Correlational Analysis
Stanford University
Analyzing Learning Subpopulations
(Kizelcec, Piech, & Schneider, 2013)
![Page 34: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/34.jpg)
Intermediate Stages: Correlational Analysis
Phil Hill (http://mfeldstein.com)
MOOC Student Archetypes
![Page 35: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/35.jpg)
Intermediate Stages: Correlational Analysis
Success & Completion
• Defining success
and completion
matters
• How one defines
success and
completion matters
• Who defines
success matters
• The denominator
matters
Audrey Watters, “Say ‘MOOC’ One More Time,”
https://www.flickr.com/photos/surreal_badger/8573233
746/.
![Page 36: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/36.jpg)
Intermediate Stages: Correlational Analysis
University of Minnesota:Predicting Completion: Student/Faculty Defined
Intent Self-Reported 50% Completed Total Points
Rea
so
ns f
or
En
roll
ing
University + + + 0
Professional + 0 0 0
Access + 0 0 0
Enjoyment + 0 0 0
De
mo
gra
ph
ics English Proficiency
+ - 0 +
Location: USA 0 + - 0
Age – 0 - 0
Sex 0 0 0 -
Ob
sta
cle
sto
Co
mp
leti
on
Tech Unfamiliar 0 0 -
Connection Problems + 0 -
Computer Problems 0 0 0
Time Zone Issues 0 0 0
Family Issues 0 - -
Work Issues 0 - -
Intent 0 + +
Self-Reported +
![Page 37: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/37.jpg)
Intermediate Stages: Correlational Analysis
University of Minnesota:Predicting Completion: Student/Faculty Defined
Intent Self-Reported 50% Completed Total Points
Rea
so
ns f
or
En
roll
ing
University + + + 0
Professional + 0 0 0
Access + 0 0 0
Enjoyment + 0 0 0
De
mo
gra
ph
ics English Proficiency
+ - 0 +
Location: USA 0 + - 0
Age – 0 - 0
Sex 0 0 0 -
Ob
sta
cle
sto
Co
mp
leti
on
Tech Unfamiliar 0 0 -
Connection Problems + 0 -
Computer Problems 0 0 0
Time Zone Issues 0 0 0
Family Issues 0 - -
Work Issues 0 - -
Intent 0 + +
Self-Reported +
![Page 38: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/38.jpg)
Intermediate Stages: Correlational Analysis
University of Minnesota:Predicting Completion: Student/Faculty Defined
Intent Self-Reported 50% Completed Total Points
Re
as
on
s f
or
En
roll
ing
University + + + 0
Professional + 0 0 0
Access + 0 0 0
Enjoyment + 0 0 0
De
mo
gra
ph
ics English Proficiency + - 0 +
Location: USA 0 + - 0
Age – 0 - 0
Sex 0 0 0 -
Ob
sta
cle
sto
Co
mp
leti
on
Tech Unfamiliar 0 0 -
Connection Problems + 0 -
Computer Problems 0 0 0
Time Zone Issues 0 0 0
Family Issues 0 - -
Work Issues 0 - -
Intent 0 + +
Self-Reported +
![Page 39: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/39.jpg)
Intermediate Stages: Correlational Analysis
Clickstream Data and Learner Intentions
Chen, B. et al. “How do MOOC learners’ intentions relate to their behaviors and overall
outcome?”
![Page 40: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/40.jpg)
Intermediate Stages: Correlational Analysis
Clickstream data and demographics
Guo & Reineke, (2014), “Demographic Differences in How Students Navigate through
MOOCs”
![Page 41: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/41.jpg)
![Page 42: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/42.jpg)
MOOC Research Progression:
Mature Stages = Controlled
Comparative Designs
Bu
tte
red
ca
t figu
res e
xtr
acte
d fro
m G
reg
Will
iam
s' W
ikiW
orld
![Page 43: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/43.jpg)
• Recommender systems MOOC, fall 2013,
Professor Joseph Konstan
Mature Stage: Controlled Comparative Designs
Computer Science: MOOC vs Hybrid Class
http://militantrecommender.blogspot.com/
![Page 44: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/44.jpg)
Pre- and post-course surveys and knowledge test:
Q. What is the core idea behind dimensionality
reduction recommenders?
a. To reduce the computation from polynomial to linear.
b. To strip off any product attributes so products appear simpler.
c. To reduce the computation time from O(n3) to O(n2).
d. To transform a ratings matrix into a pair of smaller taste-space
matrices.
e. I have no idea.
Mature Stage: Controlled Comparative Designs
Computer Science: MOOC vs Hybrid Class
![Page 45: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/45.jpg)
Mature Stage: Controlled Comparative Designs
Computer Science: MOOC vs Hybrid Class
![Page 46: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/46.jpg)
Mature Stage: Controlled Comparative Designs
Computer Science: MOOC vs Hybrid Class
Normalized gains: (Post-test – Pre-test / 100 – Pre-test)
![Page 47: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/47.jpg)
Mature Stage: Controlled Comparative Designs
Computer Science: MOOC vs Hybrid Class
![Page 48: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/48.jpg)
Mature Stage: Controlled Comparative Designs
Computer Science: MOOC vs Hybrid Class
![Page 49: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/49.jpg)
Mature Stage: Controlled Comparative Designs
Computer Science: MOOC vs Hybrid Class
![Page 50: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/50.jpg)
Mature Stage: Controlled Comparative Designs
Learning in a Physics MOOC: Pre- & Posttest
Colvin et al. (2014). “Learning in an Introductory Physics MOOC: All Cohorts Learn Equally, Including an On-
Campus Class.”
![Page 51: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/51.jpg)
Mature Stage: Controlled Comparative Designs
A/B Test of Instructor Involvement
Tomkin et al. (2014). “Do professors matter? Using an a/b test to evaluate the impact of instructor involvement on MOOC student
outcomes.”
![Page 52: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/52.jpg)
![Page 53: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/53.jpg)
Questions?
![Page 54: How to Research MOOCs: A Primer (with Results)](https://reader034.vdocument.in/reader034/viewer/2022052623/559c58e61a28ab656e8b456f/html5/thumbnails/54.jpg)
How to Research MOOCs:
A Primer (with Results)
Thanks!
D. Christopher Brooks, Ph.D. ([email protected])
ECAR: http://www.educause.edu/ecar
Twitter: @dcbphd
J.D. Walker, Ph.D. ([email protected])
University of Minnesota: http://z.umn.edu/research