![Page 1: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/1.jpg)
A Mixed Reality System for Enabling Collocated After Action
ReviewJohn Quarles
Samsun Lampotang
Ira Fischler
Paul Fishwick
Benjamin Lok
![Page 2: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/2.jpg)
Mixed Reality for Training
• Training + Mixed Reality(MR) = Improved Training– i.e Military [x], industrial[x], medical [x]
– MR is used during the training session
• After Action Review (AAR)– Playback and review of one’s training session
• i.e. AAR in the HPS
– Currently: video reviews
– helpful for bringing trainees to competency
• AAR + MR = ?
![Page 3: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/3.jpg)
After Action Review
• Students need:– Repetition, directed instruction, Feedback
• Educators need– Assessment Tools
• Video Based Review offers:– Assessment Tools and Repetition:
• video playback (FF,REW,etc)
– instruction and feedback:• Expert must be present during student review
![Page 4: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/4.jpg)
Video Based AAR
• Widely used• Limitations
– Fixed, exocentric viewpoint– Minimal interaction (Play, Pause, etc)– See only what the camera sees (i.e
occlusions are problematic)– Lost information
![Page 5: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/5.jpg)
MR Based AAR
• Video review = lost information– Goal: maximize information flow with MR
• Advantages of MR systems in general:– Egocentric viewpoint– interactivity – i.e. TUIs– Collocated virtual information (i.e. gaze)
• Can we enhance AAR with MR?
![Page 6: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/6.jpg)
Innovations
• The Augmented Anesthesia Machine Visualization and Interactive Debriefing System (AAMVID)– MR based AAR system for Anesthesia education– Educator version: AAMVID-E– Student version: AAMVID-S
• User evaluation (19 students, 3 educators)
• General goal: Evaluate MR for use in AAR
![Page 7: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/7.jpg)
Previous Work
• AAR [x],[x],[x]
• Recording Virtual Experiences– [x],[x],[x]
• AAR of Virtual Human Experiences– IPSViz[x]
![Page 8: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/8.jpg)
Anesthesia Education and MR• The Augmented Anesthesia Machine [x]
– merges and abstract simulation of an anesthesia machine with a real machine (magic lens + TUI)
– Improves learning
![Page 9: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/9.jpg)
AAR in Anesthesia Education
• Currently Video Based– Students perform a test (i.e. a fault test)– Educators and students review and critique
performance
![Page 10: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/10.jpg)
AAR in Anesthesia Education
• MR system: AAMVID– Based on the AAM– Students (AAMVID-S):
• View their own or an expert’s interaction in situ
• Follow a directed tutorial guided by the expert’s interaction visualization
– Educators (AAMVID-E):• Can visualize and filter aggregate
student data
• AAMVID addresses video review limitations
![Page 11: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/11.jpg)
Student AAMVID-S System
playback controls
Gaze visualizationinteraction
visualization
![Page 12: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/12.jpg)
Student AAMVID-S System
• 3 Visualization Modes– User Review
• View what the student did
– Expert Review• View what the expert did
– Expert Tutorial• Collocates:
– expert’s gaze and interaction event boxes– Real time abstract simulation of the student’s machine
• Interaction with the real machine while mimicking expert• Hands-on experience in correcting the fault
![Page 13: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/13.jpg)
AAMVID-S Evaluation
• Usability and performance evaluation of AAMVID-S
• 19 psychology students:
Day 1 (90 min)
Training with AAM
Day 2 (90 min)
1.Testing2. AAR of Tests with AAMVID
![Page 14: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/14.jpg)
Day 2: Testing
• Three Fault Tests– Experimenter caused a problem in the machine
• i.e. disabled a the flow of nitrous oxide
– Participant had to decide• Whether or not a fault was present
• What the fault was (if present)
• How to correct the fault (if present)
![Page 15: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/15.jpg)
Metrics
• Repeated Measures design:• Performance Measures:
– Participants were asked questions (each scored 0-4)• Fault absent or present• What was the fault?• How does the fault affect the patient?• How to correct the fault?
• Confidence Measures:– Self reported confidence ratings for above questions (0-
4)
![Page 16: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/16.jpg)
Results
• Insert graphs that show significant improvements in performance and confidence.
![Page 17: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/17.jpg)
Discussion
• Main results:– AAMVID Improved machine fault
understanding• Performance question answers improved.
– AAMVID increased confidence in fault correction ability
• Confidence scores increased
– Users prefer Expert Tutorial Mode
![Page 18: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/18.jpg)
Educator AAMVID-E System
Aggregate Gaze Heat Mapping
Interactive Filtering(i.e. on spatial cog data)
![Page 19: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/19.jpg)
Educator AAMVID-E System
Interactive Filtering(i.e. on spatial cog data) Interaction Graph
(markov model of class interaction)
![Page 20: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/20.jpg)
Informal Evaluation
• 3 Experts in anesthesia education– Dr. Samsun Lampotang– David Lizdas– Dr. Nicholas Gravenstein
![Page 21: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/21.jpg)
Observations
• Magic lens vs desktop?
• What worked and what didn’t work
![Page 22: A Mixed Reality System for Enabling Collocated After Action Review John Quarles Samsun Lampotang Ira Fischler Paul Fishwick Benjamin Lok](https://reader036.vdocument.in/reader036/viewer/2022062803/56649f415503460f94c60c6c/html5/thumbnails/22.jpg)
Conclusions
• Presented AAMVID– Improved student understanding and
confidence– Offered needed assessment abilities to
educators that were not available before
• MR based AAR– can offer visualizations and interactivity that
video based AAR cannot