A Mixed Reality System for Enabling Collocated After Action
ReviewJohn Quarles
Samsun Lampotang
Ira Fischler
Paul Fishwick
Benjamin Lok
Mixed Reality for Training
• Training + Mixed Reality(MR) = Improved Training– i.e Military [x], industrial[x], medical [x]
– MR is used during the training session
• After Action Review (AAR)– Playback and review of one’s training session
• i.e. AAR in the HPS
– Currently: video reviews
– helpful for bringing trainees to competency
• AAR + MR = ?
After Action Review
• Students need:– Repetition, directed instruction, Feedback
• Educators need– Assessment Tools
• Video Based Review offers:– Assessment Tools and Repetition:
• video playback (FF,REW,etc)
– instruction and feedback:• Expert must be present during student review
Video Based AAR
• Widely used• Limitations
– Fixed, exocentric viewpoint– Minimal interaction (Play, Pause, etc)– See only what the camera sees (i.e
occlusions are problematic)– Lost information
MR Based AAR
• Video review = lost information– Goal: maximize information flow with MR
• Advantages of MR systems in general:– Egocentric viewpoint– interactivity – i.e. TUIs– Collocated virtual information (i.e. gaze)
• Can we enhance AAR with MR?
Innovations
• The Augmented Anesthesia Machine Visualization and Interactive Debriefing System (AAMVID)– MR based AAR system for Anesthesia education– Educator version: AAMVID-E– Student version: AAMVID-S
• User evaluation (19 students, 3 educators)
• General goal: Evaluate MR for use in AAR
Previous Work
• AAR [x],[x],[x]
• Recording Virtual Experiences– [x],[x],[x]
• AAR of Virtual Human Experiences– IPSViz[x]
Anesthesia Education and MR• The Augmented Anesthesia Machine [x]
– merges and abstract simulation of an anesthesia machine with a real machine (magic lens + TUI)
– Improves learning
AAR in Anesthesia Education
• Currently Video Based– Students perform a test (i.e. a fault test)– Educators and students review and critique
performance
AAR in Anesthesia Education
• MR system: AAMVID– Based on the AAM– Students (AAMVID-S):
• View their own or an expert’s interaction in situ
• Follow a directed tutorial guided by the expert’s interaction visualization
– Educators (AAMVID-E):• Can visualize and filter aggregate
student data
• AAMVID addresses video review limitations
Student AAMVID-S System
playback controls
Gaze visualizationinteraction
visualization
Student AAMVID-S System
• 3 Visualization Modes– User Review
• View what the student did
– Expert Review• View what the expert did
– Expert Tutorial• Collocates:
– expert’s gaze and interaction event boxes– Real time abstract simulation of the student’s machine
• Interaction with the real machine while mimicking expert• Hands-on experience in correcting the fault
AAMVID-S Evaluation
• Usability and performance evaluation of AAMVID-S
• 19 psychology students:
Day 1 (90 min)
Training with AAM
Day 2 (90 min)
1.Testing2. AAR of Tests with AAMVID
Day 2: Testing
• Three Fault Tests– Experimenter caused a problem in the machine
• i.e. disabled a the flow of nitrous oxide
– Participant had to decide• Whether or not a fault was present
• What the fault was (if present)
• How to correct the fault (if present)
Metrics
• Repeated Measures design:• Performance Measures:
– Participants were asked questions (each scored 0-4)• Fault absent or present• What was the fault?• How does the fault affect the patient?• How to correct the fault?
• Confidence Measures:– Self reported confidence ratings for above questions (0-
4)
Results
• Insert graphs that show significant improvements in performance and confidence.
Discussion
• Main results:– AAMVID Improved machine fault
understanding• Performance question answers improved.
– AAMVID increased confidence in fault correction ability
• Confidence scores increased
– Users prefer Expert Tutorial Mode
Educator AAMVID-E System
Aggregate Gaze Heat Mapping
Interactive Filtering(i.e. on spatial cog data)
Educator AAMVID-E System
Interactive Filtering(i.e. on spatial cog data) Interaction Graph
(markov model of class interaction)
Informal Evaluation
• 3 Experts in anesthesia education– Dr. Samsun Lampotang– David Lizdas– Dr. Nicholas Gravenstein
Observations
• Magic lens vs desktop?
• What worked and what didn’t work
Conclusions
• Presented AAMVID– Improved student understanding and
confidence– Offered needed assessment abilities to
educators that were not available before
• MR based AAR– can offer visualizations and interactivity that
video based AAR cannot