sae ar/vr - the challenges of creating a vr application with unity
DESCRIPTION
Unity3D is a great authoring tool to create 3D applications. Regarding immersive virtual reality, its capabilities are limited. We will discuss the challenges and solutions to create great VR applications in Unity!TRANSCRIPT
![Page 2: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/2.jpg)
!
• Founder & President!
• 12 years in VR!
!
• - French railways!
!
•
Sébastien Kuntz
![Page 3: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/3.jpg)
• Simplify VR!
• Creation!
• Deployment!
• So YOU can spend more time creating applications
![Page 4: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/4.jpg)
Plan• Unity!• Challenges!
• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Rendering speed!
• MiddleVR for Unity!• Conclusion
![Page 5: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/5.jpg)
Plan
Unity!• Challenges!
• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!
• MiddleVR for Unity!• Conclusion
![Page 6: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/6.jpg)
• Platform for creating interactive 3d applications!
• Widely used!
• 2 million registered developers!
• Certified by the US Army and Air Force!
• Clients : Bigpoint, Cartoon Network, Coca-Cola, Disney, Electronic Arts, LEGO, Microsoft, NASA, Nickelodeon, Ubisoft, US Army, Warner Bros.
![Page 7: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/7.jpg)
• High-quality rendering!
• Deferred renderer!
• Occlusion culling!
• Realtime shadows!
• Ambient occlusion!
• Light mapping
![Page 8: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/8.jpg)
![Page 9: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/9.jpg)
Unity 3D Features• Programming!
• C#, Javascript!
• Profiler!
• Physics!
• NVidia PhysX!
• Cloth!
• Soft/rigid bodies!
• Ragdolls!
• Cars
3D formats !
FBX, Collada, 3DS, OBJ…!
Audio engine!
Networking!
Terrain
![Page 10: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/10.jpg)
Editor
• Simple to use!
• Create your own tools inside Unity
![Page 11: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/11.jpg)
Asset store
• Buy / sell assets!
• 3D models!
• Textures!
• Shaders!
• Scripts!
• Editor plugins
![Page 12: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/12.jpg)
Desktop
![Page 13: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/13.jpg)
Plan• Unity!
Challenges!• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!
• MiddleVR for Unity!• Conclusion
![Page 14: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/14.jpg)
Plan• Unity!• Challenges!
Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!
• MiddleVR for Unity!• Conclusion
![Page 15: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/15.jpg)
Virtual reality
Presence in a virtual world
(c) eMagin(c) Barco
![Page 16: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/16.jpg)
Experience of presenceis in your brain !
![Page 17: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/17.jpg)
Cognitive presencePerceptive presence
![Page 18: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/18.jpg)
• Build hardware
To create and maintain presence, you need to ...
• Build software
• Build user experience
![Page 19: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/19.jpg)
Plan• Unity!• Challenges!
• Presence!
Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!
• MiddleVR for Unity!• Conclusion
![Page 20: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/20.jpg)
Device management
Physical device!!
Driver Application
![Page 21: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/21.jpg)
Device management
Tracker 1! Driver Tracker 1
Application
Tracker 2!!
Driver Tracker 2
![Page 22: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/22.jpg)
![Page 23: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/23.jpg)
Plan• Unity!• Challenges!
• Presence!• Device management!
Display management!• Cluster management!• Interactions!• Deployment!• Speed!
• MiddleVR for Unity!• Conclusion
![Page 24: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/24.jpg)
Correct perspective
(c) Johnny Lee
![Page 25: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/25.jpg)
Correct perspective
![Page 26: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/26.jpg)
Stereoscopy
![Page 27: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/27.jpg)
Warping & Blending
![Page 28: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/28.jpg)
Graphics cards limitations
• Number of outputs!
• Processing power
![Page 29: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/29.jpg)
multiple graphics cards
• Pro!
• More outputs!
• More processing power!
• Con!
• 3D engines are not optimized for multi-pipe
![Page 30: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/30.jpg)
Cluster
• Pro!
• More outputs!
• More processing power!
• Con!
• 3D engines don’t support clustering
![Page 31: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/31.jpg)
• No seams across displays require 3 layers of sync!
• 3D objects synchro (Scenelock)!
• New images display (Swaplock)!
• Left/Right eyes (Genlock)
Cluster
![Page 32: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/32.jpg)
Plan• Unity!• Challenges!
• Presence!• Device management!• Display management!• Cluster management!
Interactions!• Deployment!• Speed!
• MiddleVR for Unity!• Conclusion
![Page 33: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/33.jpg)
Interactions
• Many different 3D user interactions!
• Navigation!
• Selection!
• Manipulation!
• Draw!
• System control!
• etc.
![Page 34: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/34.jpg)
Interactions are dependent on hardware
![Page 35: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/35.jpg)
Menus
![Page 36: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/36.jpg)
Haptics (force feedback)
![Page 37: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/37.jpg)
Plan• Unity!• Challenges!
• Presence!• Device management!• Display management!• Cluster management!• Interactions!
Deployment!• Speed!
• MiddleVR for Unity!• Conclusion
![Page 38: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/38.jpg)
Deployment
• Modify application to!
• Change drivers!
• Change displays!
• Change interactions
![Page 39: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/39.jpg)
Plan• Unity!• Challenges!
• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!
Rendering speed!• MiddleVR for Unity!• Conclusion
![Page 40: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/40.jpg)
Rendering speed
![Page 41: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/41.jpg)
Plan• Unity!
Challenges!• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Rendering Speed!
• MiddleVR for Unity!• Conclusion
![Page 42: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/42.jpg)
Application
![Page 43: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/43.jpg)
Plan• Unity!• Challenges!
• Presence!• Device management!• Display management!• Cluster management!• Interactions!• Deployment!• Speed!
MiddleVR for Unity!• Conclusion
![Page 44: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/44.jpg)
• Build hardware
To create and maintain presence, you, the author, need to ...
• Build software
• Build user experience
![Page 45: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/45.jpg)
“VR is mainly a software problem”!Jaron Lanier
![Page 46: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/46.jpg)
What is MiddleVR ?
• VR plugin !
• Functionalities ○Manage input devices
•3D trackers, keyboard, mouse, joystick !
○Manage display •Viewports, cameras, stereoscopy, cluster
![Page 47: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/47.jpg)
Desktop
![Page 48: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/48.jpg)
![Page 49: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/49.jpg)
http://www.youtube.com/watch?v=EEKH-1EilXs
MiddleVR tutorial
![Page 50: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/50.jpg)
![Page 51: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/51.jpg)
Device management
![Page 52: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/52.jpg)
Device management
![Page 53: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/53.jpg)
Supported Devices• Via VRPN :
• A.R.T • Vicon • Optitrack • Wiimote !
• Polhemus Fastrak Tracker, 3Space Tracker, Patriot Trackers,
• Ascension Flock of Bird • Intersense IS600, IS900 • 3dTech HiBall-3000 Wide Area Tracker, • Advanced Realtime Tracking Gmbh DTrack
Client, • World Viz Precision Position Tracker PPT
1.2, • Natural Point Optitrack Rigid Body Toolkit, • Phasespace optical tracking OWL, • ...
• Native drivers • Oculus Rift • Leap Motion • Kinect (Microsoft SDK) • PNI SpacePoint
Fusion • GameTrak • Razer Hydra • Motion Analysis (beta)
![Page 54: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/54.jpg)
First integration of Haption’s haptic devices
http://www.youtube.com/watch?v=lnoE_VclnOI
![Page 55: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/55.jpg)
Stereoscopy
• Active stereoscopy (OpenGL Quad-Buffer)!
• Which Unity can’t natively handle!
• Passive stereoscopy
![Page 56: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/56.jpg)
Cluster & Multi-pipe
• Scenelock!
• Swaplock!
• Genlock!
• Multi-pipe (soon)
![Page 57: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/57.jpg)
Supported displays
• Virtually any HMD • VR-Wall, Powerwall • Workbench, Holobench • HoloStage • CAVEs • 3D TVs • zSpace • Igloo Vision • …
![Page 58: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/58.jpg)
Deployment
• Abstractions!
• Devices!
• 3D Nodes (User)!
• Interactions (soon)
![Page 59: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/59.jpg)
Device management
Driver Tracker 1
Application
Driver Tracker 2
![Page 60: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/60.jpg)
Device abstraction
Virtual Device
Application
Driver Tracker 1
Driver Tracker 2
![Page 61: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/61.jpg)
Device management
Driver
Virtual Device Application
Simulation
Replay
![Page 62: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/62.jpg)
Virtual Device 1
Hand (3D Node) Application
Virtual Device 2
Inverse Kinematics
User abstraction
![Page 63: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/63.jpg)
Interactions• MiddleVR includes high-level interactions!
• Navigation!
• Selection!
• Manipulation!
• … More to come!
• Future: Interactions abstractions!
• Dynamically change interactions based on hardware
![Page 64: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/64.jpg)
Interactions abstraction?Grab-air
Navigation
Application
Point
Steering
Selection
Manipulation
Raycasting
Go-go
Hand
Homer
Hand
WIM
![Page 65: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/65.jpg)
Menus
• Ability to create menus and GUIs in HTML5 (soon)
![Page 66: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/66.jpg)
Speed• Unity!
• Many options for optimizations!
• Occlusion culling!
• Static batching!
• Lightmap baking!
• etc.!
• 32-bit editor only!
• But player is 32-bit & 64-bit !!
• Subdivide parts in editor!
• Compositing!
• Cloud
![Page 67: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/67.jpg)
Compositing Stereoscopy
![Page 68: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/68.jpg)
Compositing Image decomposition
![Page 69: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/69.jpg)
VR in the Cloud
• Temporary increase in power!
• Combine with compositing
![Page 70: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/70.jpg)
Perceptive illusions
• Playing with limitations of human perception!
• Sensitivity of senses!
• Selectivity of attention!
• Useful to lower cost/complexity of VR systems
![Page 71: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/71.jpg)
Perceptive illusions
(c) Franck Steinicke - Wurzburg University
Redirected walking
![Page 72: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/72.jpg)
Change blindness :
(c) Evan Suma - USC/ICT
Perceptive illusionsChange blindness
![Page 73: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/73.jpg)
![Page 74: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/74.jpg)
![Page 75: SAE AR/VR - The challenges of creating a VR application with Unity](https://reader033.vdocument.in/reader033/viewer/2022061200/54774c22b4af9fe8558b470d/html5/thumbnails/75.jpg)