tranformational model of translational research that leverages educational technology for fast...

1

Transformational Model of Translational Research that Leverages Educational Technology for Fast Data-Discovery Feedback Loops

John Stamper Pittsburgh Science of Learning Center Human-Computer Interaction Carnegie Mellon University Connecting How we Learn to Educational Practice and Policy: Research Evidence and Implications International Conference 23-24 January 2012

2

Vision for PSLC

• Why? Chasm between science & practice –  Low success rate (<10%) of randomized field trials

• LearnLab = a socio-technical bridge between lab psychology & schools – E-science of learning & education – Social processes for research-practice

engagement

• Purpose: Leverage cognitive theory and computational modeling to identify the conditions that cause robust student learning

“rigorous, sustained scientific research in education” (NRC, 2002)

3

PSLC is multidisciplinary

170+ multidisciplinary researchers from California to Germany Ken Koedinger - Carnegie Mellon Co-Director Charles Perfetti - University of Pittsburgh Co-Director Executive Committee: Vincent Aleven (HCI), Maxine Eskenazi (LTI; Diversity Director), Julie Fiez (Psych), Geoff Gordon (ML), David Klahr (Psych; Education Director), Marsha Lovett (Psych), Tim Nokes (Psych), Lauren Resnick (Psych), Carolyn Rose (LTI), John Stamper (HCI)

4

The Setting & Inspiration •  Rich tradition of research on

Learning and Instruction at CMU & University of Pittsburgh –  Basic Cognitive Science –  Research in schools –  Intelligent tutors

•  PSLC inspiration: Educational technology as research platform to launch new learning science

Built in generalization to practice, dissemination.

5

Design

Deploy

Data

Discover

Translational Research Feedback Loop

Real World Impact of Cognitive Science Algebra Cognitive Tutor • Based on computational models of

student thinking & learning • Course used nation wide

– Over 2600 schools, 500K students use for ~80 minutes per week

• Spin-off:

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

7

Scaling Up

Learning & Educational Science

Educational Practice

8

Translational Research 1: Bringing Cognitive Science to School

Design Cognitive Tutor courses: Tech, Text, Training

Deploy Address social context

Research base

Cognitive Psychology Artificial Intelligence

Practice base

Math Educators Standards

9

Which kind of problem is most difficult for Algebra students?

Story Problem

As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work?

Word Problem

Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with?

Equation

x * 6 + 66 = 81.90

Data contradicts common beliefs of researchers and teachers

High School Algebra Students

70%61%

42%

0%

20%

40%

60%

80%

100%

Story Word Equation

Problem Representation

Per

cen

t C

orre

ct

Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.

0!10!20!30!40!50!60!70!80!90!

100!

Elementary!Teachers!

Middle!School!

Teachers!High School!Teachers!

% Correctly ranking equations as hardest!

Nathan & Koedinger (2000). An investigation of teachers’ beliefs of students’ algebra development. Cognition and Instruction.

Expert Blind Spot!

11

3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology Use ACT-R theory to individualize instruction •  Cognitive Model: A system that can solve problems in

the various ways students can

If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d

Then rewrite as abx + c = d

If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a

•  Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

12

3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology Use ACT-R theory to individualize instruction •  Cognitive Model: A system that can solve problems in

the various ways students can

If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d

Then rewrite as abx + c = d

•  Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Hint message: “Distribute a across the parentheses.” Bug message: “You need to

multiply c by a also.”

•  Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

Known? = 85% chance Known? = 45%

13

Translational Research 1: Bringing Cognitive Science to School



Research base


Practice base


14

Cognitive Tutor Algebra: Problems that engage intuition & interest Health Care

Extinction

Local Facts

Smoking Risks

Importance of Math Education

15

Cognitive Tutor Algebra: Rich Interactions

16

Cognitive Tutor Algebra: Rich Interactions

Cognitive Tutor Algebra course yields significantly better learning

Course includes text, tutor, teacher professional development 8 of 10 full-year controlled studies demonstrate significantly better student learning

0

10

20

30

40

50

60

Iowa SAT subset ProblemSolving

Represent-ations

Traditional Algebra Course

Cognitive Tutor Algebra

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

18

Scaling success? Yes Done? No! Why not? •  Final performance particularly in urban schools

is still far from desirable •  Weaknesses in field study results

–  Not all studies are random assignment –  Two null results

•  Many design decisions not guided by science

•  We can use the deployed technology to collect data, make discoveries, and continually improve the instructional design

19

Scaling Down

Learning & Educational Science

Educational Practice

20

Translational Research 2: Fielded Systems Provide Data for New Discoveries



Data Qual, quant; process, product

Discover Cognition, learning, instruction, context

Research base


Practice base


21

How are cognitive models developed? Cognitive Task Analysis

Traditional methods •  Structured interviews &

think alouds of experts & novices => Create symbolic model

Newer methods •  Data-Driven •  Educational Data Mining => Create statistical model => symbolic model

Meta-analysis: CTA produces 1.7 effect size (Lee, 2004)

22

Good Cognitive Model => Good Learning Curve •  An empirical basis for determining when

a cognitive model is good •  Accurate predictions of student task

performance & learning transfer – Repeated practice on tasks involving the

same skill should reduce the error rate on those tasks

=> A declining learning curve should emerge

23

A good cognitive model produces a learning curve

Without decomposition, using just a single “Geometry” skill,

Is this the correct or “best” cognitive model?

no smooth learning curve.

a smooth learning curve.

But with decomposition, 12 skills for area,

(Rise in error rate because poorer students get assigned more problems)

Inspect curves for individual knowledge components (KCs)

Some do not => Opportunity to improve model!

Many curves show a reasonable decline

25

Can a data-driven process be automated & brought to scale?

Yes! •  Combine Cognitive Science,

Psychometrics, Machine Learning … •  Collect a rich body of data •  Develop new model discovery

algorithms, visualizations, & on-line collaboration support

26

Automating the Cognitive Model Discovery Process

Learning Factors Analysis •  Input

–  Factors that may differentiate tasks –  Student performance across tasks & over time

•  Output: Best cognitive model

Cen, H., Koedinger, K., Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement.

27

Discovery of new cognitive models: Strategy & Results •  “Mixed initiative” human & machine discovery

–  Visualizations to aid human discovery –  AI search for statistically better models

•  Better models discovered in Geometry, Statistics, English, Physics

Stamper, J., Koedinger, K.R. (2011) Human-machine Student Model Discovery and Improvement Using DataShop.

28

LFA –Model Search Process

OriginalModel

BIC = 4328

4301 4312

4320

43204322

Split by Embed Split by Backward Split by Initial

43134322

4248

50+

4322 43244325

15 expansions later

Automates the process of hypothesizing alternative cognitive models & testing them against data

•  Fully automated machine learning guided search

•  Input: Existing proposed models •  Output: Best cognitive model based on

splitting and merging existing models

29

Summary •  Most ed field trials yield null results

– Need better data & cumulative theory •  Optimal instructional design requires

discoveries – The student is not like me

•  Scale up success: Cognitive Tutor Algebra

•  LearnLab: E-science infrastructure to support science of learning

30

Opportunities Ahead

•  Better models => better instruction •  Combine cog sci & machine learning

– Machine learning competitions

– PSLC’s DataShop has 300+ datasets

– SimStudent learns new models

31

Thank you! Acknowledgements •  Cognitive Tutors

John R. Anderson (Psych), Albert Corbett (HCI), Steve Ritter (Carnegie Learning), …

•  Cognitive Task Analysis Mitchell Nathan (UW Ed Psych), Mimi McLaughlin (HCI), Neil Heffernan (WPI CS), Marsha Lovett (Psych) …

•  Cognitive Model Discovery Brian Junker (Stats), Hao Cen (Machine Learning), Geoff Gordon (ML) …

•  Pittsburgh Science of Learning Center –  Kurt VanLehn (ASU CS) -- original PSLC co-director –  Ken Koedinger (HCI/Pysch), Charles Perfetti (Upitt Psych),

David Klahr (Psych), Lauren Resnick (Upitt Psych), Vincent Aleven (HCI), Maxine Eskenazi (LTI), Carolyn Rose (LTI/HCI)

–  All 200+ past & current members!