1. Research Project NumberHydraulic Excavator HMI DESIGNUSING Augmented Reality Georgia Institute of Technology | Milwaukee School of Engineering | North Carolina A&T State University | Purdue University University of Illinois, Urbana-Champaign | University of Minnesota | Vanderbilt UniversityPresenter: Michael EkundayoAdvisor: Dr. Eui ParkGraduate Supervisor: Joseph AkyeampongNorth Carolina A&T State UniversityCCEFP REU 2012June 4 August 10, 2012

2. Outline 1.Background 2.Research Objective 3.Technical Solutions 4.Implementation 5.Lessons Learned 6.Idea 7.Future Work 8.Q&AEvent2Event Date(s) 3. Background Fluid Power SystemsHydraulics PneumaticsCar liftAirCompressor Jet engine Wind wiring turbinesEvent 3Event Date(s) 4. Background HydraulicExcavators Work Domains include: Construction (commercial and residential) Mining Water & Sewer Forestry & AgricultureEvent4Event Date(s) 5. Background Excavator HMI Issues Ease of Use (Cognitive) Less Intuitive Information deficient Safety (Physical) Layout ofDisplays/Controls Repetitive use ofcontrolsEvent 5Event Date(s) 6. Background Virtual vs.Augmented Reality Virtual RealityAugmented Reality A computer generated, A combination of the ainteractive, three- real scene and a virtualdimensional environment scene generated by ain which a person iscomputer that augmentsimmersed. the scene with additionalinformation.Event 6Event Date(s) 7. Background - Augmented Reality TechnologiesAn AR system consists of: a camera for acquiring video images of the real world. a computer for generating virtual objects. display unit (e.g. monitor/head- mounted/projector) for merging the virtual and real scenes. a tracking unit (e.g. mechanical/optical/inertia), trackers) for obtaining real-time position/orientation of the user. fiducial markers for registering (or overlaying) virtual objects on the real world interaction devices (e.g. joystick/mouse/haptic etc.) to manipulate/navigate the augmented environment.Event 7Event Date(s) 8. Examples of AR Applications SkinVaders is the firstgame to use AugmentedReality to physically bringa person body into a game. Google glasses developedfor A.R. application Event8 Event Date(s) 9. Examples of ARApplicationsIKEA uses augmentreality to fit a customerwith a product by imagingthe room with theproducts like furniture,refrigerators, dishwasherand other productsEvent 9Event Date(s) 10. Research ObjectiveTo develop an excavator HMI utilizing augmented reality (AR)technologyEvent 10Event Date(s) 11. Role of AR in Hydraulic Excavator HMI Design The role of AR in hydraulic excavator HMI design is to identify additional information that can be used to augment the operators cognition to enhance his/her task performance. AR design strategies will be used to recommend improved HMI designs for hydraulic excavators.Event11Event Date(s) 12. Technical Solutions (Ease ofUse) Heads Up Display Originated in jet fighters to provideflight information to pilots withoutrequiring them to look away from theirusual viewpoints. sought to ease informationoverload and to make theinstrumentation less complicated Common in aircrafts; beingintroduced in carsHydraulic excavators will providegreat application platform to easethe operators informationoverloadEvent 12Event Date(s) 13. Approach/TechnicalSolutions Haptic technology is a technologyHaptic Technologywhich takes advantage of the sense of touch by applying forces, vibrations, or motions to the user. Haptic devices (or haptic interfaces) are mechanical devices that mediate communication between the user and the computer For hydraulic excavators it will provide an additional modality (sensory cue) to facilitate the operators cognition, thereby leading to enhanced usability.Event 13Event Date(s) 14. ImplementationEvent 14Event Date(s) 15. What I LearnedI have learned about Fluid Power Systems Hydraulic Excavators Augmented Reality Heads Up Displays Haptic Devices Ergonomics Time-management Readiness: to be prepared for everythingand anythingEvent15Event Date(s) 16. Idea Google Earth has been explored for AR/MR relatedapplications to give remote viewing of geo-spatialinformation (Frohlich et al., 2006) and urban planning(Phan & Choo, 2010). Google Earth can be used with AR to guide the designand implementation of HMIs for hydraulic excavators.Question to be Addressed: How to connect Google earth with A.R. in real time?Event 16Event Date(s) 17. Future Work Assist with conducting usability testingwith AR-simulated HMIs Assist with performing ergonomicsanalysis of HUD-inspired HMI in Jack.Event17Event Date(s) 18. Q&A?????????Event18Event Date(s)