©2012 carnegie learning, inc. in-vivo experimentation steve ritter founder and chief scientist...
TRANSCRIPT
©2012 Carnegie Learning, Inc.
In-vivo Experimentation
Steve RitterFounder and Chief Scientist
Carnegie Learning
©2012 Carnegie Learning, Inc.
An attempt to find meaning in three acts
• Design: Geometry Contiguity (Vincent Aleven, Kirsten Butcher)
• Modeling: Adjusting learning curve parameters (Cen, Koedinger, Junker)
• Personalization: Word problem content (Candace Walkington)
©2012 Carnegie Learning, Inc.
DESIGN
©2012 Carnegie Learning, Inc.
Geometry angles
©2012 Carnegie Learning, Inc.
Contiguity
Early Version Commercial Version(Carnegie Learning)
Research Version
(Carnegie Mellon)
Butcher, K., & Aleven, V. (2008). Diagram interaction during intelligent tutoring in geometry: Support for knowledge retention and deep transfer. In C. Schunn (Ed.) Proceedings of the Annual Meeting of the Cognitive Science Society, CogSci 2008. New York, NY: Lawrence Earlbaum.
Hausmann, R.G.M. & Vuong, A. (2012) Testing the Split Attention Effect on Learning in a Natural Educational Setting Using an Intelligent Tutoring System for Geometry. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society. (pp. 438-443). Austin, TX: Cognitive Science Society.
©2012 Carnegie Learning, Inc.
Early Tutor
©2012 Carnegie Learning, Inc.
Revised (commercial) tutor
©2012 Carnegie Learning, Inc.
Geometry Contiguity
• Design and field experimentation– Butcher and Aleven (2008)• Diagram interaction led to better transfer
and retention
• Analysis of impact– Hausmann and Vuong (2012)• Unit-level effects mixed• Advantage for harder skills
©2012 Carnegie Learning, Inc.
Geometry Angles
©2012 Carnegie Learning, Inc.
Lessons
• Change is constant• Transition from research to
production always requires adaptation
©2012 Carnegie Learning, Inc.
MODELING
©2012 Carnegie Learning, Inc.
Skillometer
©2012 Carnegie Learning, Inc.
Expression Writing
©2012 Carnegie Learning, Inc.
What gets learned?
©2012 Carnegie Learning, Inc.
Bayesian Knowledge Tracing
Cognitive tutor traces these skills differently
©2012 Carnegie Learning, Inc.16
Learning Curve Parameter Fitting• Field study looking at learning area of geometric figures
– One group used adjusted learning parameters based on previous year’s data
• Optimized group took 12% less time to reach same performance
• Significant learning gain in both groups• No difference in learning gain between groups (p = 0.772 )
0
20
40
60
80
100
120
Square
Parallelogram
Triangle
Trapezoid
Polygon
Circle
Optimized
Control
©2012 Carnegie Learning, Inc.
Lessons
• Learning efficiency is a great outcome
• Small, systemic changes can have big impact
• Optimizing skills requires appropriate skill model– Koedinger, McLaughlin and Stamper
(2012) - LFA
©2012 Carnegie Learning, Inc.
PERSONALIZATION
©2012 Carnegie Learning, Inc.
Word problem customization
©2012 Carnegie Learning, Inc.
Personalization field study
• Students who got problems related to their interests made fewer errors
• Also affected subsequent unit• Interaction with readability
©2012 Carnegie Learning, Inc.
Lessons
• Content matters– Challenge for knowledge component
modeling
• Are we personalizing preferences, reading level or both?
©2012 Carnegie Learning, Inc.
Summary
• It’s not about whether A is better than B– It’s about why A is better than B