![Page 1: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/1.jpg)
1
FOSS Virtual & Augmented Reality
Monado & OpenXR
Jakob Bornecrantz
01/02/2020
![Page 3: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/3.jpg)
3
● What is AR & VR● OpenXR● Monado● Status● Demos● Wrapping up
![Page 4: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/4.jpg)
4
● What is AR & VROpenXRMonadoStatusDemosWrapping up
![Page 5: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/5.jpg)
5
What is Augmented Reality?
● Augment━ Verb━ /ɔːɡˈmɛnt/━ make greater by adding to it; increase
![Page 6: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/6.jpg)
6Open First
![Page 7: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/7.jpg)
7Open First
![Page 8: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/8.jpg)
8
What is Virtual Reality?
● I reject your reality and substitute my own━ Dungeonmaster, 1984
■ (popularized by Adam Savage)
![Page 9: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/9.jpg)
9Open First
![Page 10: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/10.jpg)
10Open First
![Page 11: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/11.jpg)
11
AR + VR = ?
● Also known as XR● X = A | V● Not eXtended Reality
![Page 12: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/12.jpg)
12
What is AR & VR● OpenXR
MonadoStatusDemosWrapping up
![Page 13: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/13.jpg)
13
Program
Platform
Hardware
How do we XR?
![Page 14: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/14.jpg)
14
Program
Platform
Hardware
How do we XR?
API
![Page 15: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/15.jpg)
15
![Page 16: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/16.jpg)
16
● It’s just like OpenGL & Vulkan━ Made by Khronos
● Out now● Action Based● FOSS Implementation! :D
What’s cool about OpenXR?
![Page 17: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/17.jpg)
17
● Type of controller not in focus━ Only after the fact
● Context━ From program━ To runtime━ Runtime decides what button does what
Action Based
![Page 18: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/18.jpg)
18
What is AR & VROpenXR
● MonadoStatusDemosWrapping up
![Page 19: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/19.jpg)
19
Program
Platform
Hardware
How do we XR?
![Page 20: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/20.jpg)
20
Program
Monado
Hardware
Where does Monado fit in?
![Page 21: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/21.jpg)
21
OpenXR State Tracker
DriversCompositor
Program
xrt_compositorxrt_device
OpenXR
Platform
xrt_prober
Monado
Hardware
![Page 22: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/22.jpg)
22
OpenXR State Tracker
Drivers
GL Wrapper
Program
xrt_gl_compositorxrt_device
OpenXR
Platform
xrt_prober
Complex Truth
OpenXR Loader
Compositor
xrt_fd_compositor
HardwareAuxiliary
![Page 23: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/23.jpg)
23
OpenXR State Tracker
Hardware
CompositorDrivers
xrt_compositorxrt_device
Vulkan, X11, Wayland and KMS
Platform
xrt_prober
Linux Hardware
Linux APIsMisc APIs
ProgramOpenXR
Desktop Stack
![Page 24: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/24.jpg)
24
OpenXR State Tracker
IPC Layer
Hardware
CompositorDrivers
xrt_fd_compositorxrt_device
xrt_fd_compositorxrt_device
Vulkan and KMS
Platform
xrt_prober
xrt_prober
Linux Hardware
Linux APIsMisc APIs
Process boundary
With IPC - Anticipated
![Page 25: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/25.jpg)
25
● Between different components━ Auxiliary code has their own interfaces
● Purely C● Not stable![1]
XRT Interfaces
![Page 26: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/26.jpg)
26
● HMDs● Controllers● Aspect pattern
━ Optional structs━ HMD info
xrt_device● Functions
━ Input Sync━ Output Set━ View/Position Retrieval
● Info (immutable)━ Input/Outputs━ Tracking Origin━ Screen position (HMD)━ Distortion (HMD)
![Page 27: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/27.jpg)
27
● Base class━ xrt_[gl|vk|fd]_compositor━ GL & VK are wrappers
■ Client side only
━ FD is “real” compositor■ Written in Vulkan■ Only “talks” in FDs
● xrt_swapchain━ xrt_[gl|vk|fd]_swapchain━ acquire, wait & release
xrt_compositor● Functions
━ More explicit then OpenXR━ Create Swapchain━ Prepare, Begin & End━ Wait, Begin, Discard & End━ Poll Events (Refactor?)
● Info (immutable)━ Supported Formats
![Page 28: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/28.jpg)
28
● Policy holder━ Reads configuration━ Decides what devices open
● Device prober━ Used by OpenXR ST
● Platform code━ Abstract native interfaces━ Used by device drivers
● Should be split in two━ xrt_prober━ xrt_platform
xrt_prober● Functions
━ Look for devices━ Select (opens devices)━ List & Open Video devices━ Misc USB
● Info (immutable)━ None
![Page 29: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/29.jpg)
29
What is AR & VROpenXRMonado
● StatusDemosWrapping up
![Page 30: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/30.jpg)
30
● Mostly complete OpenXR support● In-process Compositor
━ Mesh shader
● Video processing framework● Tracking PS Move● Nifty debug UI
Status
![Page 31: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/31.jpg)
31
● North Star*● Daydream Ctrl*● OSVR HDK● Vive (Pro) & Index
━ Positional tracking through libsurvive
● PlayStation VR● PlayStation Move● Razer Hydra● OpenHMD
━ Port to native drivers where it makes sense.
Hardware support
![Page 32: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/32.jpg)
32
● 35.5K lines of codes━ Drivers: 6.9K
■ 9 drivers■ Vive: 1.5K■ HDK: 0.5K
━ Compositor: 6.6K━ Auxiliary: 7.6K━ State Trackers: 10.1K━ Targets: 1.3K
Code
![Page 33: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/33.jpg)
33
● Complete OpenXR support● Improve PS Move Tracking● PSVR Tracking● Out of process compositor
Next Up
![Page 34: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/34.jpg)
34
● Safe space● System UI● Setup UI● Lighthouse Tracking● AR and SLAM
Even further out
![Page 35: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/35.jpg)
35
What is AR & VROpenXRMonadoStatus
● DemosWrapping up
![Page 36: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/36.jpg)
36
Tracking
![Page 37: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/37.jpg)
37
Debug UI
![Page 39: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/39.jpg)
39
What is AR & VROpenXRMonadoStatusDemos
● Wrapping up
![Page 40: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/40.jpg)
40
● Gnome-Shell━ Output hot-plug━ non-desktop
● Intel Missing Extension━ GL_EXT_memory_object_fd
Rant
![Page 41: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/41.jpg)
41
● Lots of work needed● GNOME & KDE UIs?
━ Monado interface library?
● Switching between apps?
Some thoughts
![Page 43: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/43.jpg)
43
● Monado━ https://monado.dev/━ https://monado.pages.freedesktop.org/monado━ https://gitlab.freedesktop.org/monado/monado
● OpenXR━ https://www.khronos.org/openxr
● Work on GFX & XR━ Internships @ Collabora━ Work @ Collabora
Links
![Page 44: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/44.jpg)
44Open First
Talk to me about
● XR, VR & AR● FOSS XR● Volt Programming Language● FPGAs & FOSS● Amiga (FPGA), mc68k (LLVM)● Voxel/SVO rendering● FOSS & Society● Joining Collabora!
![Page 45: Augmented Reality - FOSDEM€¦ · 1 FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz jakob@collabora.com 01/02/2020](https://reader034.vdocument.in/reader034/viewer/2022042404/5f1bff302393b177ed3a933d/html5/thumbnails/45.jpg)
45
FOSS Virtual & Augmented RealityQuestions?