hand tracked controls -...
TRANSCRIPT
![Page 1: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/1.jpg)
Hand Tracked Controls: Design and Implementation for Serious VR
Matt NewportCTO/Co-founder - Osso VR
![Page 2: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/2.jpg)
A bit about me
● Matt Newport – [email protected]● CTO/Co-founder Osso VR
● http://ossovr.com/● VR surgical training
● Started in games industry 2000● Software engineer with a focus on rendering● EA, Relic (THQ), various smaller studios
● Left games to work on Serious VR 2014
![Page 3: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/3.jpg)
Osso VR
![Page 4: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/4.jpg)
Overview
● Basic design decisions for hand tracked controls● Show hands or controllers?● Pre-set or flexible grips (‘tomato presence’)● How to handle hands passing through solid objects
● Hand tracking tips and tricks● Dropped objects● Passing from hand to hand
● The importance of throwing in VR● Implementing hand tracked interactions
![Page 5: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/5.jpg)
Hands or Controllers?
![Page 6: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/6.jpg)
Picking Stuff Up
● Two main choices for hand or controller:
● Remains visible
● Disappears
● This is best illustrated with a GIF!
![Page 7: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/7.jpg)
![Page 8: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/8.jpg)
● Requires one or more pre-set grip positions for each object
● Hand position in VR doesn’t match user’s actual hand position
● User can’t freely adjust grip
● Looks more natural on video
● Doesn’t require any custom setup per object
● Users don’t seem to mind / notice
● Users can freely adjust their grip
● Can look strange on video
Disappearing Hands?
Hand disappears Show object held in hand
![Page 9: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/9.jpg)
Show Objects in Hand: The Gallery (Video Review by FunshineX)
![Page 10: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/10.jpg)
Hand Disappears
![Page 11: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/11.jpg)
Hands passing through objects
● Hands in VR should always track 1:1 with real hands
● How to handle passing through solid objects?
● Our solution: allow hands to pass but held objects break out of grip
![Page 12: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/12.jpg)
Hands passing through objects
![Page 13: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/13.jpg)
Downside of showing handsThere can be multiple different natural grips for an object depending on the user’s intent (pour, carry, use as a weapon)
![Page 14: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/14.jpg)
Flexible grip choice
![Page 15: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/15.jpg)
Passing from hand to hand
![Page 16: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/16.jpg)
Throwing
![Page 17: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/17.jpg)
Throwing
![Page 18: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/18.jpg)
Dropping stuff on the floor
![Page 19: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/19.jpg)
Dropping stuff on the floor
![Page 20: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/20.jpg)
More complex interactions
● Attaching and detaching parts
● Screwing
● Drilling
● Hammering
● Positioning the C-arm (X-ray machine)
● And more…
![Page 21: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/21.jpg)
Interactions: physics based or not?
● I’m not a physics engine expert!
● We’re using PhysX in Unity● Your results may vary with other engines
● When it works, physics based interactions can be a time saver and produce good results
● Unfortunately it often doesn’t work well● Particularly for two handed interactions
● Experiment and decide on a case by case basis
![Page 22: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/22.jpg)
Case study: screwing
![Page 23: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/23.jpg)
Physics (joint based) didn’t work
![Page 24: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/24.jpg)
Our custom solution
● Use parenting in Unity
● Custom script does a bunch of quaternion math to try and produce a plausible rotation based on hand movements while enforcing constraints we care about
● Trigger haptics proportional to the local space rotation to give a feeling of threads
![Page 25: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/25.jpg)
Case study: drilling
![Page 26: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/26.jpg)
Physics (collision based) didn’t work
![Page 27: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/27.jpg)
Physics (joint based) worked better
![Page 28: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/28.jpg)
Our physics based solution
● Drill is locked to a configurable joint constraining motion to one linear axis and rotation around that axis
● Script adjusts the range of motion of the joint according to the time the drill is active
● When the drilling is complete, make the joint breakable to allow drill to be removed
![Page 29: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/29.jpg)
Case study: hammering
![Page 30: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/30.jpg)
Physics (collision based) had promise
![Page 31: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/31.jpg)
Our custom solution
● We opted not to use physics in this case
● May revisit this
● Physics based solution might give us displacement proportional to impact force ‘for free’
● Custom script detects hammer blows and sets position of nail
![Page 32: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/32.jpg)
Haptics and audio
● Haptics helps sell interactions
● When haptics and audio are used well together the whole is greater than the sum of the parts
● Haptics that are driven by real world position and rotation changes work well
![Page 33: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/33.jpg)
NewtonVR
● If you’re interested in experimenting with physics driven interactions check out NewtonVR - open source Unity plugin
● Developed in part by two of our employees when they were at Tomorrow Today Labs
● We currently have custom interaction code but may switch to NewtonVR in future
![Page 34: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/34.jpg)
Conclusions
● Hand interactions are a powerful tool for training applications
● Make sure to get the basics right and think about the right choices for your application
● Custom interactions can be time intensive to develop but the payoff is usually worth it
● Physics based interactions are great if you can get them to work reliably
![Page 35: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/35.jpg)
Resources
● Owlchemy Labs Oculus Connect 2 talk: Being There: Designing Standing VR Experiences with Tracked Controllers - https://youtu.be/hjc7AJwZ4DI
● Collection of useful links: http://www.uxofvr.com/● My blog: http://blog.mattnewport.com/● Email me: [email protected]● Osso VR: http://ossovr.com/● NewtonVR – Physics based interaction for Unity:
https://github.com/TomorrowTodayLabs/NewtonVR
![Page 36: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/36.jpg)
Video Credits
● “Three Bs on the B” - The Gallery: Call of the Starseed - HTC Vive VR Gameplay –FunshineX● https://youtu.be/8MJm1EWWHi0
● Baby Beth throwing toys out of the cot –jennymcdiarmid● https://youtu.be/8tTRDmdi18g
![Page 37: Hand Tracked Controls - twvideo01.ubm-us.nettwvideo01.ubm-us.net/o1/vault/vrdc2016/Newport_MattHand-Tracke… · Overview Basic design decisions for hand tracked controls Show hands](https://reader033.vdocument.in/reader033/viewer/2022042721/5f9a24b53c383853891f8680/html5/thumbnails/37.jpg)
Questions?