1

Using Experiments and Cognitive Science Research to Improve the Design of

Online Resources for Learning

Joseph Jay Williams

josephjaywilliams@stanford.edu www.josephjaywilliams.com/research-overview

Online Education & Learning Online

• New research area?• Convergence of

computational & behavioral scienceNIPS “Data-Driven Education”New ACM conference “Learning at Scale”CHI

Novel Research Opportunity: Real-World + Laboratory

Integrate Research & Practice

• Randomized assignment• Experimental Control• Rich data

• Generalizable theories• “in vivo” experiments• Diverse populations

• Practical improvements• Disseminate research• Generate Funding

• Real-world environment

• Authentic activities• Practical Challenges

5

Overview

• Explanation & Learning• Teaching Learning Strategies• Motivational Messages• Experimental Paradigm• Experiment-focused Design

6

Why does explaining “why?” help learning?

• General boost to Learning Engagement vs.

• The Subsumptive Constraints Account: Interpret target of why-explanation in terms of a broader generalization (Williams & Lombrozo, 2010)

• 2 x 3 = 6• Why?

Explanation and Learning: Lab Online

• Discovery & transfer (Williams & Lombrozo, 2010,

Cognitive Science)

• Use of prior knowledge (Williams & Lombrozo, 2013,

Cog. Psych.)

• Erroneously overgeneralize at expense of exceptions (Williams et al, 2013, JEP:

General)

• Promotes belief revision – given sufficient anomalies (Williams, Walker, Maldonado &

Lombrozo, 2012; 2013, Cog Sci Conference; in prep)

• Prompts in online (math) exercises (Williams, Paunesku, Haley, Sohl-Dickstein, 2013, AIED Moocshop; ongoing)

GLORP DRENT

8

Learning Task & Experimental Paradigm

• Online (Math) Exercise

1. Number of Problems Completed

2. Percent Correct

9

Logic of my Previous Research

Explain why this is correct.

Elaborate on what you are thinking now.

• Post-Study test questions• Transfer/Generalization

questions• Questions about key principle• Memory for details

10

Future: Generate, Receive, Compare

Geza KovacsSimultaneously Learning AND Crowdsourcing Improvement of Learning Resources

Williams, Thille, Siemens, Trumbore, Stigler. How online resources can facilitate interdisciplinary collaboration. Invited talk to be presented at SIG on Computer and Internet Applications in Education, AERA 2014.

E.g. Explain why this is correct.

Generate

E.g. Another student said: This is correct because…

Receive

E.g. What are the similarities and differences?

Compare

Teaching Learning Strategies

• Spend time teaching specific content, or general strategies?

• Online: Collect data that would be extremely difficult to get in the real world

• Online: Repeatedly reinforce habits & educational behaviors

• Teach “What? Why? How?” self-questioning/explanation strategies (Palinscar & Brown,

1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams & Lombrozo, 2010, Cognitive Science)

• Understanding vs. Problem-solving • vs. Interpreting vs. Practice-as-Usual

11

Experimentally manipulate additional prompts

Clickable link.

[Click here to learn about the “What? Why? How?” strategy]

+ Prompts embedded into hints

Self-questioning strategy: What? Why? How?

Embedded Prompts between Hint/Solution Steps

Prompts

Prompt type

Prompted-understanding

Prompted-problem-solving

Prompted-interpretation

What? What does this step

mean to you?

What are you doing or thinking right

now?

What is this step saying? Restate it in

your own words.

Why? Why is it helpful to

take this step?

Why is what you are currently doing

helpful? Why is it useful for achieving

your goal?

How? How do you know this step is right?

How well is your current approach to

this problem working?

16

Explanation studyPractice-as-

usual Control Self-Regulating Thinking

Reflecting on Meaning

Problems Attempted

Problems Correct

Practice Tasks Completed

Mastery Tasks Completed

Problem Accuracy

17

Explanation studyPractice-as-usual No Textbox Textbox

Problems Attempted

Problems Correct

Practice Tasks Completed

Mastery Tasks Completed

Problem Accuracy

Learning Behavior Support

• Clickable link to Drop-Down text with suite of strategies:Are you stuck?Click here for some tips.

• Provide previously examined prompts. • Use mouseover and drop-down text to

reveal information “as requested”, rich traversal of options, guided by student

Natural Link to Learning Strategy Training

• If you want to learn more about strategies to keep motivated and learn well, go to tiny.cc/learningassistant or XX or YY

20

3. Add motivational messages

Practice-as-usual

Remember, the more you practice the smarter you become!

Growth Mindset Message

21

3. Embedded in vivo Experiment

• Growth Mindset Message

• "Remember, the more you practice the smarter you become.”,

• "Mistakes help you learn. Think hard to learn from them.”

• Practice-as-usual

• Benefit of Growth Mindset Message?

Jascha Sohl-Dickstein

22

Results: More motivated?

• Growth Mindset Message > Practice-as-Usual

• extra problems attempted

• more problems correct

• Percent Correct: Problems correct/Problems attempted

• increase in Percent Correct

23

Practice-as-usualGrowth Mindset Message

Positive Message

3. Add motivational messages

Some of these problems are hard. Do your best!Remember, the more you practice the smarter you become!

24

Does any positive message work?

• Growth Mindset Message

• "Remember, the more you practice the smarter you become.”,

• "Mistakes help you learn. Think hard to learn from them.”

• Positive Message• "Some of these problems are

hard. Just do your best."• "This might be a tough problem,

but we know you can do it.”

• Practice-as-usual

25

Effects of Positive Messages?

• Positive Messages ~= Practice-as-Usual

• Growth Mindset > Positive• extra problems attempted • more problems correct • increase in Percent Correct

26

Computational Modeling

• Williams, Mitchell, Heffernan. MOOC Research Initiative grant from Gates Foundation & Athabasca. Investigating the benefits of embedding motivational messages in online exercises.

• 2 million users on 12 kinds of fractions exercises, ~100 problems each

• Moderators & Mediators• Item Response Theory• Non-parametric Bayesian clustering of Users (CrossCat, JMLR)• Model latent knowledge states

27

Synthesize Scientific Findings

• Williams, J.J. (2013)Improving Learning in MOOCs by Applying Cognitive Science. Paper presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN.

• www.josephjaywilliams.com/education

28

Experimental Paradigm: R.E.P.E.A.T.• Williams, J. J. (2013). Finding connections

between basic experimental research and realistic online education contexts. In J. J. Williams (chair), Online Learning and Psychological Science: Opportunities to integrate research and practice. Symposium conducted at the annual convention of the Association for Psychological Science. 

• Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society. (pdf)

29

Experiments

• Williams, J.J. & Williams, B.A. (under review). Online A/B Tests & Experiments: A Practical But Scientifically Informed Introduction. Course proposal submitted to ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada. (pdf)

• Williams, J.J., Heffernan, N., & Koedinger, K. Experiments at Scale: Instrumenting MOOCs for experimentation and course-improving data analysis. Tutorial proposal submitted to the First Annual ACM Conference on Learning at Scale.

30

Experiment-Focused Design

• Williams, J.J. & Williams, B. A. (2013). Using Interventions to Improve Online Learning. Paper to be presented at the NIPS 2013 Workshop on Data Driven Education.

31

Review

• Explanation & Learning• Teaching Learning Strategies• Motivational Messages• Experimental Paradigm• Experiment-focused Design• Williams, J.J., Klemmer, S., Kizilcec, R., &

Russel, D. (under review). Learning Innovations at Scale. Workshop proposal submitted to ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada. 

32

Acknowledgements

• Jascha Sohl-Dickstein• Jace Kohlmeier & Khan Academy• Sam Maldonado• Lytics Lab (lytics.stanford.edu)• VPOL (Vice Provost of Online

Learning, online.stanford.edu)