valerie shute, florida state university s y s t e m s ari workshop on adaptive training...
TRANSCRIPT
Valerie Shute, Florida State UniversityS Y S T E M S
ARI Workshop on Adaptive Training Technologies, Charleston, SC (March 3-5, 2009)
A D A P T I V E
Shute, V. J. & Zapata-Rivera, D. (2008). Adaptive technologies. In J. M. Spector, D. Merrill, J. van Merriënboer, & M. Driscoll (Eds.), Handbook of Research on Educational Communications and Technology (3rd Edition) (pp. 277-294). New York, NY: Lawrence Erlbaum Associates, Taylor & Francis Group.
AdaptiveContent
Diagnosis
Assessment
Evidence
Simple Logic
Definitions (foundation)
Rationale (motivation)
Four-process adaptive cycle (frame)
Current technologies (hard & soft)
Concrete example (via my bag)
Future visions (briefly)
Definitions
1. Adaptivity
Refers to a natural or artificial system’s ability to alter its behavior (etc.) according to the environment.
Adaptive technologies (hard/soft) allow an instructional system to alter its behavior according to learner needs (etc.).
Typically linked with a learner model (see next slide).
2. Learner Model
Representation of a learner, maintained by an adaptive system.
Models can be used to give personalized assistance to individuals based on cognitive and noncognitive aspects of their profile.
Learner models have been used in many areas, especially advanced educational and training systems.
3. Hard Technologies
Devices used in adaptive systems to capture learner info or present content.
Used to detect performance data or affective states (e.g., boredom, excitement, confusion, etc.) or present stuff in a more accessible manner.
Best when coupled with soft technologies (next slide).
Eye tracking device
Talking tactile tablet
4. Soft Technologies
Usually algorithms, programs, or envir’s that broaden the types of interaction between learners and computer.
For example, an adaptive algorithm can be used in a program to: (a) select a task that provides the most info about a learner, or (b) suggest additional resources tailored to the learner’s needs.
Task Response Types: E = Equation | G = Graphic | N = Numeric | T = TextualOrder within topic (skill) specified by topic, based on educational appropriateness
Classroom test or instructional unit
Score reportClassroom test or instructional unit
Mathematics Intervention Module
Engaging Instructional Unit Locator Test Set initial Student Model values and sequence leaf topics for presentation
Leaf Topic (Skill)1
Brief Instructional Object (Overview of skill area)
Student presented Hard E task
Task scored
Is the student's response correct?
Student is provided targeted,
progressive feedback
Tries = 3? Student presentedEasy E task
Task scored
Is the student's response correct?
Student is provided targeted,
progressive feedback
Tries = 3?
Detailed Instructional Object E
E tasks
G tasks N tasks T tasks
Leaf Topic 2 Leaf Topic 3 Leaf Topic n
IntegratedTask Set
Teacher is provided with summary
feedback
Proficient in standard?
Individualized instructionNot proficient in some standard
No
No
No
No
Yes
Yes
Yes
No
3 MC items per leaf topic Leaf topics sequenced from most mastered to least mastered
1 - 8 tasks integratingskills to reflect the standard as a whole.
[Relationship to student model TBD]
= Optional (Teacher-selectable?)
Add'l Practice
Teacher-selectable options:
1: All students2: Student choice3. Do not deliver
Multiple-choice items, feedback
limited to answers/rationales
Student Model updated
Student Model updated
Yes
Yes
SMART MIM v 1.3March 24, 2005
For IDMS 8/2005 Release
Loop as appropriate
Standard-level
Instructional Recap
5. Adaptive Systems
AC systems monitor and adjust room temperature, and cruise-control systems monitor and adjust your car speed.
Similarly, adaptive educational/training systems monitor important learner characteristics and make (or suggest) appropriate adjustments to support and enhance relevant competencies.
Actual room temperature
Desiredtemperature
Heating/cooling
Temperaturedifference
Thermostat heating/AC system
6. Goal of Adaptive Systems
… to create a sound and flexible environment that supports learning for persons with a wide range of abilities, disabilities, interests, backgrounds, traits, states, etc.
The challenge rests mainly on accurately identifying and estimating these learner variables then leveraging the info to improve learning/skill.
Rationale
People differ across countless dimensions.
Different dimensions are more/less suited for different types of instruction/training.
Adaptive systems can enhance learning/skill via extra practice opportunities, alternative multimedia options (especially useful to those with disabilities), tailored instruction/training, etc.
Why Adapt?Why
Adaptive systems are helpful/relevant in the world of business and education …
And they are (and will be) of growing importance in terms of supporting U.S. Army’s evolving training needs.
Adaptive Cycle
4-Process Adaptive Cycle
Adaptive technology is intended to support learning (effectiveness, efficiency, and/or engagement).
This requires accurate diagnoses.
Learner info used as basis for content selection.
Our 4-process cycle combines & extends: (a) a simpler 2-process adaptive model (Dx/Rx), and (b) a process model to support assessment (Mislevy, Steinberg, & Almond, 2003).
Shute & Zapata-Rivera, 2008
4-Process Adaptive Cycle
PresentCapture
SelectAnalyze
Learner Model
Learner
Alternative CyclesScenario Description
Complete cycle (1, 2, 3, 4, 5, 6)
All processes are exercised and cycle will continue until goals of the instructional/training activity have been met.
Modifying the model (1, 2, 3, 4, 5, 6, 9)
Learner allowed to interact with the learner model. The nature of the interaction and effects on the learner model can vary (e.g., overwriting the value of a particular variable).
Monitoring path (1, 2, 3)
Learner continuously monitored; info is analyzed and used to update learner profiles. This path spins off to a 3rd party (e.g., surveillance system, profiling for risk-analysis).
Short (or temporary) memory cycle (1, 7, 5, 6)
Adaptation based on info gathered from the latest interaction(s) between learner and the system. No permanent LM is maintained.
Diagnosis Over TimeL
ear
ne
r In
form
ati
on
Each agent maintains a personal view of the learner.
LM info and content can be distributed in different places.
Agents can communicate with each other directly or through an LM server to share information that can be used to help the learner achieve learning goals.
Communication: Agents/Learners
Overview of Technologies
Capture
Analyze Select
PresentQuantitativeTechniques
QualitativeTechniques
CognitiveVariables
NoncognitiveVariables
• Bayesian nets
• Machine learning
• Stereotypes
• Plan recognition• . . . .
• Performance data
• Eye-gaze tracker
• Speech capture
• Gesture/posture
• Haptic devices
• . . . .
• Personalized content
• Multiple representations
• Accommodations
• Meaning equivalencies
• . . . .
What variables should be taken into account when implementing an adaptive system?
What are the best technologies and methods that you use or recommend?
Cristina Conati Jim Greer Tanja Mitrovic Julita Vassileva Beverly Woolf
Experts’ Views
What to adapt?
How to adapt?
Learner variables Instructional variablesCognitive abilities (e.g., math skills, reading skills, cognitive development stage, problem solving, analogical reasoning)
Feedback type (e.g., hints, explanations) and timing (e.g., immediate, delayed)
Metacognitive skills (e.g., self-explanation, self-assessment, reflection, planning)
Content sequencing (e.g., concepts and learning objects as well as tasks, items, or problems to solve)
Affective states (e.g., motivation, attention, engagement)
Scaffolding (e.g., support and fading as warranted, rewards)
Additional variables (e.g., personality, learner styles, social skills such as collaboration, perceptual skills)
View of material (e.g., overview, preview, and review as well as visualization of goal or solution structure)
What to Adapt?
Approach RationaleProbability and decision theory
Rule-based approaches often used in adaptive systems, but using probabilistic LMs provides formal theories of decision making for adaptation. Decision theory takes into account uncertainty in both model assessment and adaptation outcome, & combines it with formal representation of system objectives to identify best actions.
Concept mapping
To adapt content (e.g., sequences of concepts, learning objects, hints) to the learner, employ a concept map with prerequisite relationships, an overlay model of the students’ knowledge, and a reactive planning algorithm.
Unsupervised machine learning
Most existing LMs built by relying on expert knowledge (either for direct model definition or labeling data) to be used by supervised machine learning techniques. But expert knowledge can be very costly, and for some innovative applications such knowledge may be nonexistent. Alternatively – use unsupervised machine learning to build LMs from unlabeled data using clustering techniques for defining classes of user behaviors during environment interactions.
How to Adapt?
ChallengesThe main barriers to moving ahead in the area of adaptive technologies include the following:
Obtaining useful and accurate learning info on which to base adaptive decisions.
Maximizing benefits to learners while minimizing costs associated with adaptive technologies.
Addressing issues relating to learner control (of environment and LM) and privacy.
Figuring out the bandwidth problem (re: scope of learner data).
Valid LM
Increase ROI
Control/Privacy
Grain size
Example
Diagnosis This is the part of the cycle on which I now focus.
Sine qua non
Flow & Grow
Shute, V. J., Ventura, M., Bauer, M. I., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow. In U. Ritterfeld, M. J. Cody, & P. Vorderer (Eds.), The Social Science of Serious Games: Theories and Applications. Philadelphia, PA: Routledge/LEA.
Games Flow Engagement Learning
but… Games lack assessment infrastructure
Assessments determine what’s been learned
Typical assessments disrupt flow
Thus we need stealth assessments in games
Evidence-centered design can accomplish this
Games to Learning
New advances in measurement let us to administer formative assessment (FA) during learning to Extract ongoing, multifaceted info from a learner Make accurate inferences of competencies React in immediate and helpful ways.
When FA is so seamlessly woven into the fabric of the learning environment that it’s invisible, this is stealth assessment.
Stealth Assessment
E C D
Introducing
Competency Model
What do you want to say about the person?
Evidence Model
What observations would provide best evidence for what you want to say?
Task/Action Model Model
What kinds of tasks let you make the necessary observations?
Assessment Design
Competency Model: Organization of competencies & claims to be made about students, and current mastery estimates.
Evidence Model: Criteria or rubrics for evidence of claim (i.e., specific student performance data; observables).
Task Model: A range of templates and parameters for task development to elicit evidence needed for the evidence model.
Assessment Models & Metrics
Monitor & Diagnose Success
StatModel
EvidenceRules
Competency Evidence Task
CaptureAnalyze
Design & Diagnosis
Elder Scrolls IV
OblivionElder Scrolls IV
Oblivion
First person 3D RPG set in a medieval world
Can be one of many characters (e.g., knight, mage, elf), each who has (or can obtain) various weapons, spells, and tools
Primary goal—gain rank & complete quests (like America’s Army)
Quests may include locating a person to obtain info, figuring out a clue for future quests, etc.
Multiple mini quests along the way, and a major quest that results in winning the game (100s of hr of game play)
Players have the freedom to complete quests in any order
Elder Scrolls IV: Oblivion
In Oblivion (like AA), problem solving plays a key role in quests since the player has to figure out what to do and how to do it.
Problem solving often viewed as the most important cognitive activity in everyday & professional contexts, but it’s seldom explicitly assessed or rewarded in formal instructional/training settings.
Assessment and support of problem solving skills are very important to improve long-term learning potential.
Quests: Problem Solving
There are many character skills to improve in Oblivion which are frequency based (i.e., number of actions relative to a skill).
Learning to play the game and developing skills require many hours of game play, and many hours of game play implies persistence—in the face of success and failure.
Persistence has been shown to significantly predict achievement—in academic, business, and military worlds.
Quests: Persistence
In many games (and combat games in particular), attention plays a key role in success.
In Oblivion, you need to attend to factors such as: health, fatigue, enemy maneuvering, escape plan, etc.
The central role of attention in learning has been demonstrated for decades.
Quests: Attention
Success in Oblivion
Cognitive Noncognitive
ReadingComp
ListeningComp
SpeakingSkill
WorkingMemory
DomainKnowledge
ProblemSolving
ReflectionExploratory
Behavior
PersistenceCreativity
Creative ProblemSolving
Attention
Efficiency Novelty
Oblivion Competency Model
Efficiency Novelty
Scene 2Scene 1
Evidence Model
Competency Model
Action Model
Creative Problem Solving
Scene 1
Action Indicators Scene 2
Problem Solving Creativity
Unobservables
Observables
The Glue
Example ECD Models
Relevant* Action Novelty Efficiency
Swim across the river n = 0.12 e = 0.22
Levitate over the river n = 0.33 e = 0.70
Freeze water with a spell and slide across n = 0.76 e = 0.80
Find a bridge over the river n = 0.66 e = 0.24
Dig a tunnel under the river n = 0.78 e = 0.20
* Relevant refers to any action included in a successful solution.
Problem: Cross river filled with dangerous fish to get to the cave on the other side.
Action Model with Indicators
Novelty: 1 – frequency
Efficiency: Inverse fn (resources, time)
Action: Find a bridge over the river
Indicators: Novelty = 1 - 0.34 = 0.66 Efficiency = 1 / [(3 × 0.4) + (5 × 0.6)] = 0.24
• Resources Used = Weapon (1, fight monster with sword) + Health (1, damage from monster) + Object (1, magic potion) = 3 resources (weight = 0.4)
• Time expended = 5 minutes (weight = 0.6)
Indicators Per Action
CreativeProblemSolving
LowHigh
0.600.40
Creativity
LowHigh
0.110.89
ProblemSolving
LowHigh
0.640.36
ObservedNovelty
0 to 0.250.25 to 0.50.5 to 0.750.75 to 1
0 0 0
1
0.78 ± 0.07
Novelty
LowHigh
0.020.98
Efficiency
LowHigh
0.860.14
ObservedEfficiency
0 to 0.250.25 to 0.50.5 to 0.750.75 to 1
1 0 0 0
0.20 ± 0.07
Dig a tunnel under the river: e = 0.20; n = 0.78
Bayes Model—Case 1
CreativeProblemSolving
LowHigh
0.180.82
Creativity
LowHigh
0.030.97
ProblemSolving
LowHigh
0.120.88
ObservedNovelty
0 to 0.250.25 to 0.50.5 to 0.750.75 to 1
0 0 0
1
0.80 ± 0.07
Novelty
LowHigh
0.010.99
Efficiency
LowHigh
0.020.98
ObservedEfficiency
0 to 0.250.25 to 0.50.5 to 0.750.75 to 1
0 0 0
1
0.76 ± 0.07
Freeze water and slide across: e = 0.76; n = 0.80
Bayes Model—Case 2
Bayes nets can be used in various ways to improve learning and performance.
They continuously observe & integrate evidence of performance for accurate, real-time estimates of competencies.
Info on competencies may be used by (a) trainers (to adjust instruction), (b) the system (to select new gaming experiences), and/or (c) trainees (to reflect on how they’re doing).
Supporting “Grow”
Re: learning, current estimates of competencies can be integrated into the game and displayed as progress indicators.
This elevates valued competencies to the same level as health & weapons!
Supporting “Grow” (cont.)
To address military training challenges and harness the potential of immersive games, I presented an ECD-inspired idea which involved the following:
• Specify valuable competencies to be acquired from the game
• Define evidence models that link game behaviors to competencies
• Update the learner model at certain intervals
Next step—adapt content in the game to fit the current needs of player/learner.
Example Summary
Future Visions
Broad themes included:
Lifelong learner models under control of each learner and with aggregation of info possible across models. Issues: privacy and user control of personal data, its use and reuse (Kay).
Ecological approach to adaptivity, where environment contains repositories of artificial agents (representing learning objects) and personal agents (representing learners). Each agent maintains a model of other agents and users to help achieve its goals. Continuously accumulating info, with natural selection re: objects (McCalla).
Getting benefits to exceed costs of adaptive technologies. Adaptivity is worthwhile within a restricted range of settings, so it’s important to identify settings and conducting good adaptive experiments (Jameson).
Experts’ Future Visions
Gord McCallaJudy Kay Anthony Jameson
Evaluation Studies Needed. To advance adaptive systems, we need controlled evaluations of technologies and systems (e.g., Shute, Hansen, & Almond—You can’t fatten a hog by weighing it, or can you?). Such studies will let us gauge the added value of expensive technologies in relation to important outcomes.
What / How to Adapt? Traits targeted for adaptation should clearly improve the pedagogical effectiveness of the system. This depends on if (a) a trait is relevant to achieve system goals, (b) there’s enough variability on the trait to justify personalization, and (c) there’s sufficient knowledge on how to adapt to learner diffs on the trait.
Overall Summary
Human beings, viewed as behaving systems, are quite simple. The apparent complexity of
our behavior is largely a reflection of the complexity of the environment in which we
find ourselves.
~Herbert A. Simon
Capture
Get Back
Gathering personal (cognitive and noncognitive) info about the learner as she interacts with the environment.
Analyze
Get Back
The creation and maintenance of a learner model. Typically the info is represented in terms of inferences on current states. In the 4-process figure, it’s the smaller human icon (i.e., the LM).
Select
Info (i.e., content in the broadest sense) is selected according to the LM and goals of the system (e.g., next learning object, test item, type of feedback). This process is used to determine how & when to intervene.
Get Back
Present
Based on results from the select process, specific content is presented to the learner. This involves using different media, devices, and technologies to efficiently and effectively convey info.
Get Back
Special delivery!