![Page 1: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/1.jpg)
JonathanReeves
October9,2017
SeeWhatWeSee- SharingMixedRealityExperienceswithWebRTC
1©2017Arvizio,Inc.
![Page 2: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/2.jpg)
Introduction
MixedRealityisagamechangingtechnologyforenterprisesincludingAEC,education,energy,engineering&healthcare
Keyrequirementsforenterpriseadoption:§ Mixedrealityplatformsmustscaletohandlecomplex3Dmodels§ Experiencesshouldbesharedbothlocallyandremotely§ IntegrationwithexistingITsystemsessentialforseamlessworkflow
Arvizio’s MRStudio platformgoal:§ Dynamicservingoflarge3DmodelswithrichLOD(levelofdetail)§ Sharingofexperiencesandcollaborationacrosslocations§ FlexibledataintegrationwithexistingITsystemsandIoTdata
2©2017Arvizio,Inc.
![Page 3: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/3.jpg)
TheMRStudioPlatform
MRStudioImmerse
MRStudioRemoteASPENscalable3DmodelLODprocessingReal-timesession&sharingservicesOnPrem,edgeorclouddeployment
Large3DmodelvisualizationwithLODSharing&collaborationacrosslocationsSupportformultipleAR/MRdevicetypes
Viewmixedrealityscenesremotely2Wayvideo,audio&annotation
MRStudioDirector
IoTData&ITSystems
4G/5GorWiFi
3©2017Arvizio,Inc
IPNetwork
WebRTC
![Page 4: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/4.jpg)
MRStudioDirector
Advanced3DVisualization§ Optimizeslarge3DmodelswithLOD(LevelofDetail)§ Mesh,CAD,Pointcloud,BIM,GIS&DICOM§ Workflowwizardsformanyspatialdataformats
Real-timeSharingServices§ SharedviewingformultipleHoloLensusers§ HoloLensusersorMRSDirectormayleadsessions§ WebRTCgatewayforsecureremoteparticipation
DataIntegration§ ConnectsMRStudiowithexternalITsystems§ ModularApacheNiFi dataflowbasedarchitecture§ ModulestoconnectwithIoTsensors,bigdataetc.
4©2017Arvizio,Inc.
![Page 5: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/5.jpg)
LODDataPyramid
5©2017Arvizio,Inc..
Full 3D model
2nd Level of Detail
3rd Level of Detail
4th Level of Detail
![Page 6: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/6.jpg)
ImmerseAppFramework
3DModelViewingwithLODLifeSizeModelWalkThroughSharingforMultipleHoloLens
RemoteVideoChatMixedRealityRTCFeed
DataintegrationwithITsystemsBigDataSystems,IoTStreams
MRStudioDirector
Annotation
MRStudioImmerse
6©2017Arvizio,Inc..
![Page 7: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/7.jpg)
Local&RemoteCollaboration
7©2017Arvizio,Inc.
HoloSync allowsMRSDirectortointeractivelypublish3Dscenesandmodels
MRStudioRemoteActiveviewingandannotationoftheHoloLensfeedCreatetextualorgraphicalhologramsinthescene
WebRTC
LODDataServedonDemand
WebRTCAudio,Video&SpatialDataExchange
![Page 8: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/8.jpg)
WebRTCRecap
VP8/9&H.264VideoCompression- AppleoffersonlyH.264inIOS11
STUN&TURNAllowNATRouterTraversal
ICE– InteractiveConnectivityEstablishmentmanagestheconnectivityprocess
8©2017Arvizio,Inc.
![Page 9: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/9.jpg)
Remoteviewing&annotationoverWebRTC
9©2017Arvizio,Inc.
§ Remoteusersmayplaceholographicannotationsintothefieldofview§ Supportedonmanybrowsersacrossmultipleplatforms§ Bandwidthefficient- operatesoverbandwidthconstrainedwideareanetworks§ HoloLensfeedislimitedto720p,alsoheadmovementcanbeachallenge
MRDeviceView
RemoteAnnotationToolbar
![Page 10: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/10.jpg)
CapturingtheSharedExperienceHoloLensSpectatorView
§ LocalviewingofmixedrealityscenesinHDbutnoremotestreaming§ Calibrationcheckerboardrequired,complexsoftwareinstall§ RequiresdedicatedHoloLens&digitalSLR,milledaluminumbracket§ Powerfulconcept,couldthisbeimprovedupon?
10©2017Arvizio,Inc.
![Page 11: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/11.jpg)
MRStudioViewPointStreamingofMixedRealityinHD
§ Localviewing/recording&remoteWebRTCstreamingofmixedrealityscenes§ Headsetrequiredduringinitialsetuponly§ Usedepthcameraorwebcams(staticplacement)inplaceofDSLR§ Potentialtoworkacrossmultipletypesofdevice
11©2017Arvizio,Inc.
![Page 12: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/12.jpg)
CameraCaptureUnity App
ComposerWebRTCGateway
Director Sharing Server
Depth SensingStereo Camera
MR Studio Director
WebRTCClient
LocalHD View(s)
RemoteView
PC
Audio
MRStudioViewPoint Architecture
12©2017Arvizio,Inc.
Shared Viewing
![Page 13: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/13.jpg)
MRStudioUseCasesAEC(Architecture,Engineering&Construction)
§ Visualizationduringthedesign,engineeringandconstructionprocess§ Bringingprojectstolifeduringthesalesprocess§ Accelerateprojectapprovalprocess
IndustrialDesign&Engineering§ 3Dvisualizationofcomplexmachinery§ Assistedmaintenancetaskswithremotevideoexpertassistance
Training&Education§ Training&educationacrossindustries§ Classroom,lectures,onsitetrainingandremoteparticipants
Healthcare§ MRhospitalroomconcept,MRassistancefortelemedicine§ Presurgeryvisualization,prepareforcomplexsurgeries
13©2017Arvizio,Inc.
![Page 14: See What We See -Sharing Mixed Reality Experiences with WebRTC · WebRTC Recap VP8/9 & H.264 Video Compression -Apple offers only H.264 in IOS 11 STUN & TURN Allow NAT Router Traversal](https://reader033.vdocument.in/reader033/viewer/2022042219/5ec4ea012d6ea64d964e080c/html5/thumbnails/14.jpg)
Demonstration