an open research approach - 2015 open education week tu delft
TRANSCRIPT
9-3-2015
Challenge the future
DelftUniversity ofTechnology
An ‘open research’ approach
Thieme Hennis – [email protected]
Pieter de Vries - [email protected]
TU Delft Online Learning
Level (Example) topics of interest and stakeholders
Strategic University role into the future, open en online education, business models, ROIDeans, DelftX, MT, marketing
Organizational Support structure, documentation and guidelines, collaboration protocols, data use, teacher satisfactionO&S
Course Course design, use of media, pedagogy, methods and tools, student support, experiments, interaction & collaborationCourse teams, Media Center
Technology/edX Design of tools for learning, teaching, and data analysis/visualizationedX consortium, ‘TUD tech’
Students Learner demographics, background, performance, engagement, satisfaction, etc.Students
Many opportunities
• … to research MOOCs
• … and use MOOCs for research
But..
• Limited capacity
• No/little history in educational research at university
• Potential to connect with researchers worldwide
Open research…
Facilitating research activities
1. …with relevant parties globally and locally
2. …on a broad range of relevant topics
3. …by giving them access to our data and research
instruments
4. …and managing the collaboration workflow
5. …and publishing the results together
1a. Who are these ‘relevant
parties’?
• MOOC Teachers / course teams
• University students / PhD’s
• Researchers from partner institutes
• Researchers globally with expertise on a specific,
relevant topic
1b. Our current ‘MOOC research
network’
• Stanford University - Psychology• UC St. Barbara - Psychology• University of South Australia - Pedagogy, coll. learning• Edinburgh University - Communities of Inquiry• Harvey Mudd College - Gender, diversity• TU München - Gender, diversity• Memphis university - Linguistics• UVA - Methodologies
• TU Delft - Privacy/ethics- Course-relevant topics (researching course teams)
2a. The ‘broad range’ of relevant
topics
• Who is the DelftX MOOC learner? - UNISA
• Does self-affirmation of personal values increase engagement of at-
risk students (motivation research) – Stanford, UC St. Barbara
• (How) does communication in MOOC forums influence social
centrality and performance? – Memphis Univ.
• How well does the Community of Practice framework fit learning in a
MOOC context? What effects of social and teaching presence on social
networks and collaboration? – Edinburgh univ.
• Why do female students have lower performance in our MOOCs? How to
design socially inclusive courses? – Harvey Mudd College & TU
München
• Etc..
2b. Choice of topics
Research questions come/emerge from
• Evaluation outcomes (DelftX MOOCs)
• Literature, current developments in educational research
• Teachers, developers, and other support staff
• External parties interested in research collaboration
3. Giving access to data and
instrumentsData sources Research & evaluation
edX subscription data Number of participants, dropouts, location, age, gender, schooling
edX student data, learning analytics Progress, tests results, quizzes, exams, etc.Forum participationVideo clicks, navigation
External data, other media (i.e. Facebook groups, discussions)
Social networks, content and discourse analysis
Surveys (pre, mid, post) Information about demographics, intention, expectations, satisfaction, media use, etc.Interventions embedded in surveys
Interviews / self-assessment: teachers, NMC, DelftX
Experiences with workflow andorganizationQuestions and expectations for evaluation
4b. The Open Research Toolbox
Collaborating• Google Drive folder with
• Documentation for partners• Data files
Documentation• Memorandum of Understanding• Ethical clearance documentation• Paper outline• All available data sources, instruments, and courses
Communication• Monthly Skype conversations (depending on stage in process)• Email
5. Selection of publications
• 6 Working papers
• 5 course reports
• 1 comparative analysis paper
• Understanding Social Learning Behaviors (SEFI’14 - Skrypnyk, Hennis, De Vries, 2014)• Learners’ Social Centrality and Performance (EDM’15 – Dowell, Joksimovic, Skrypnyk,
Hennis, De Vries, 2015)• An exploratory study in the concerns for information privacy (MSc thesis - Hassing,
2015)• Scalability and Flexibility through Open Research (LAK’15) Hennis, De Vries, 2015)
• Who is the Learner in the DelftX Engineering MOOCs? (SEFI’15 - Hennis, Skrypnyk, De
Vries, 2015)
• Diversity in Engineering MOOCs, a first Appraisal (SEFI’15 – Ihsen, Yves, Hennis, De
Vries 2015)
• Reconsidering Retention in MOOCs: the Relevance of Formal Assessment and Pedagogy
(EMOOCS’15 - Skrypnyk, De Vries, Hennis, 2015)
Potential of ‘open research’
• Being highly flexible in what you research
• Being able to scale the research efforts
• Research network and learn from their expertise
• Interdisciplinary research
Next steps
• Optimize the work flow & instruments
• Research across courses (other edX partners/MOOC providers)
• More data cleaning capacity + open data workflow
• Extend the research HUB function (also: organizations)
• Better integration and involvement course teams to achieve
design based research → feedback and forward loops between
research and teaching (online)
Additional ideas:
• Tender for MOOC research
• Promote TU Delft research areas
• Students as researchers (crowdsource research models)
Gender gap
• Low percentage female students at start
• Even lower percentage female students finishing
• Stereotype threat?→ collaboration Univ. Munich
& Harvey Mudd College
Ihsen, S, Yves, J, Hennis, T. A. & Vries, P. de. (2015). Diversity in Engineering MOOCs, a first Appraisal. In SEFI 43th Annual Conference. Orleans, FR.
Performance attitude
•Among young, male
students
• ‘Spike’ around grades 6
and 10
• Reason to join MOOC:
“To get a certificate..”
Young vs Old
Male vs female
Hennis, T. A., Skrypnyk, O., & Vries, P. de. (2015). Who is the Learner in the DelftX Engineering MOOCs. In SEFI 43th Annual Conference. Orleans, FR.
Dropout rate
• Non-starters are biggest dropout group (up to ~ 80%!)
• Then: after first Homework Assignment (HWA.01)
• GRADE HWA.01 IS IMPORTANT PREDICTOR
• Avg. grade HWA.01 of group that does not continue: to HWA.02: 3.58
• Avg. grade of group that does continue: 8.89
• Also: pass rate is > 80% of group who performs better than 20%
• Including the ‘explorers’
• “No Time” most important factor
Pedagogy and Assessment design
→ Collaboration with University of South Australia
Skrypnyk, O., Vries, P. de. & Hennis, T. A.(2015). Reconsidering Retention in MOOCs: the Relevance of Formal Assessment and Pedagogy. In EMOOCs’15, Mons, BE.
(Professional) background students
• Many professionals: 50-70%
• “No experience” predictor of early
dropout / not starting• Not during course!
• Relevance for course team: • Recommended level course ~
student level?
• Who are my students?
• etc
Hennis, T. A., Skrypnyk, O., & Vries, P. de. (2015). Who is the Learner in the DelftX Engineering MOOCs. In SEFI 43th Annual Conference. Orleans, FR.
Forum (1): performance en structuur
• Forum activity ~ grade
• Facebook groups vs forum• More ‘social’ behavior and peer-
support• Not anonymous• Better performance
• edX forum social network analysis• Few ‘superposters’ responsible for
most posts• Few ‘communities’
Skrypnyk, O., Hennis, T. A., & Vries, P. de. (2014). Understanding social learning behaviours of
xMOOC completers Conference Topic : Technology in Learning. In SEFI 42th Annual Conference.
Birmingham, UK.
Forum (2): Language ~ social capital
+ Community of Inquiry
• Community of Inquiry• Measuring social and teaching
presence• Effects on engagement, performance
and forum activity
• Linguistic analysis• Aumated textual analysis: how
conversations influence development of social capital
→ with Memphis Univ & Univ of Southern Australia
Dowell, N, Skrypnyk, O., Joksimovic ́, S, Graesser, A, Dawson, S, Gasevic, D, Hennis, T. A.,, Vries, P.
de., Kovanovic, V. (2015). Modeling Learners’ Social Centrality and Performance through Language
and Discourse. In EDM’15. Madrid, Spain.
Forum (3): cultural group &
preference to collaborate
• EUR / Anglo > alone• Asia / African /Latin >
together
→ edX forum: does not show this cultural preference!
(vooral EUR/Anglo)