![Page 1: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/1.jpg)
©2012 Carnegie Learning, Inc.
In-vivo Experimentation
Steve RitterFounder and Chief Scientist
Carnegie Learning
![Page 2: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/2.jpg)
©2012 Carnegie Learning, Inc.
An attempt to find meaning in three acts
• Design: Geometry Contiguity (Vincent Aleven, Kirsten Butcher)
• Modeling: Adjusting learning curve parameters (Cen, Koedinger, Junker)
• Personalization: Word problem content (Candace Walkington)
![Page 3: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/3.jpg)
©2012 Carnegie Learning, Inc.
DESIGN
![Page 4: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/4.jpg)
©2012 Carnegie Learning, Inc.
Geometry angles
![Page 5: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/5.jpg)
©2012 Carnegie Learning, Inc.
Contiguity
Early Version Commercial Version(Carnegie Learning)
Research Version
(Carnegie Mellon)
Butcher, K., & Aleven, V. (2008). Diagram interaction during intelligent tutoring in geometry: Support for knowledge retention and deep transfer. In C. Schunn (Ed.) Proceedings of the Annual Meeting of the Cognitive Science Society, CogSci 2008. New York, NY: Lawrence Earlbaum.
Hausmann, R.G.M. & Vuong, A. (2012) Testing the Split Attention Effect on Learning in a Natural Educational Setting Using an Intelligent Tutoring System for Geometry. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society. (pp. 438-443). Austin, TX: Cognitive Science Society.
![Page 6: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/6.jpg)
©2012 Carnegie Learning, Inc.
Early Tutor
![Page 7: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/7.jpg)
©2012 Carnegie Learning, Inc.
Revised (commercial) tutor
![Page 8: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/8.jpg)
©2012 Carnegie Learning, Inc.
Geometry Contiguity
• Design and field experimentation– Butcher and Aleven (2008)• Diagram interaction led to better transfer
and retention
• Analysis of impact– Hausmann and Vuong (2012)• Unit-level effects mixed• Advantage for harder skills
![Page 9: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/9.jpg)
©2012 Carnegie Learning, Inc.
Geometry Angles
![Page 10: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/10.jpg)
©2012 Carnegie Learning, Inc.
Lessons
• Change is constant• Transition from research to
production always requires adaptation
![Page 11: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/11.jpg)
©2012 Carnegie Learning, Inc.
MODELING
![Page 12: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/12.jpg)
©2012 Carnegie Learning, Inc.
Skillometer
![Page 13: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/13.jpg)
©2012 Carnegie Learning, Inc.
Expression Writing
![Page 14: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/14.jpg)
©2012 Carnegie Learning, Inc.
What gets learned?
![Page 15: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/15.jpg)
©2012 Carnegie Learning, Inc.
Bayesian Knowledge Tracing
Cognitive tutor traces these skills differently
![Page 16: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/16.jpg)
©2012 Carnegie Learning, Inc.16
Learning Curve Parameter Fitting• Field study looking at learning area of geometric figures
– One group used adjusted learning parameters based on previous year’s data
• Optimized group took 12% less time to reach same performance
• Significant learning gain in both groups• No difference in learning gain between groups (p = 0.772 )
0
20
40
60
80
100
120
Square
Parallelogram
Triangle
Trapezoid
Polygon
Circle
Optimized
Control
![Page 17: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/17.jpg)
©2012 Carnegie Learning, Inc.
Lessons
• Learning efficiency is a great outcome
• Small, systemic changes can have big impact
• Optimizing skills requires appropriate skill model– Koedinger, McLaughlin and Stamper
(2012) - LFA
![Page 18: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/18.jpg)
©2012 Carnegie Learning, Inc.
PERSONALIZATION
![Page 19: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/19.jpg)
©2012 Carnegie Learning, Inc.
Word problem customization
![Page 20: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/20.jpg)
©2012 Carnegie Learning, Inc.
Personalization field study
• Students who got problems related to their interests made fewer errors
• Also affected subsequent unit• Interaction with readability
![Page 21: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/21.jpg)
©2012 Carnegie Learning, Inc.
Lessons
• Content matters– Challenge for knowledge component
modeling
• Are we personalizing preferences, reading level or both?
![Page 22: ©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning](https://reader031.vdocument.in/reader031/viewer/2022032104/56649ce05503460f949a9978/html5/thumbnails/22.jpg)
©2012 Carnegie Learning, Inc.
Summary
• It’s not about whether A is better than B– It’s about why A is better than B