distributed collaborative virtual & augmented reality · 2019. 9. 19. · virtual (vr),...

© Competence Centre for Virtual Reality and Cooperative Engineering w. V. – Virtual Dimension Center (VDC)

Dr.-Ing. Dipl.-Kfm. Christoph Runde

Franziska Jöckel

Virtual Dimension Center (VDC)

Fellbach

Auberlenstr. 13

70736 Fellbach

www.vdc-fellbach.de

Distributed Collaborative Virtual &

Augmented Reality VDC-Whitepaper


Basics Areas of ApplicationIntroduction Market Overview Outlook

1. Introduction

2. Basics

2.1 Definition

2.2 Benefits

2.3 Fundamental Concepts

2.4 Network Architectures

2.5 Challenges and Solutions

3. Areas of Application

4. Market Overview

5. Summary & Outlook

Structure

19.09.2019 2



"Distributed work" is becoming more and more important: a study by Forrester

Research revealed that more than 80 percent of employees already work

regularly with people who are at a different location (eg home office, other

office building, another company location abroad, etc.).1

Interdisciplinary communication is becoming increasingly important: change in

the working environment (digitization, new ways of working, etc.); increasing

complexity of tasks, idea and decision-making processes; domain-specific

know-how as undocumented experiential knowledge

“Traditional alternatives" for distributed, interdisciplinary collaboration:

o On-site meetings: Disadvantage of high travel costs and times

o Remote meetings in the form of telephone and video conferencing:

especially when discussing 3D content lack of visualization options, lack of

group awareness

1 https://www.wiwo.de/erfolg/management-der-zukunft/arbeitsform-der-zukunft-so-fuehren-sie-virtuelle-teams/19385778.html

Starting Situation (1/2)

19.09.2019 3

Introduction

https://www.wiwo.de/erfolg/management-der-zukunft/arbeitsform-der-zukunft-so-fuehren-sie-virtuelle-teams/19385778.html



Virtual (VR), augmented (AR) and mixed (MR) reality technologies enable

interactive, remote collaboration, and they are becoming increasingly

important in the enterprise context

Head-mounted displays (HMDs) are becoming cheaper and more powerful

The VR HMD market has been split equivalently until just a few years ago,

and now there is a spillover of consumer solutions with increased

performance in industrial applications; even game engines are now used

industrially A scenario in which multi-user environments / virtual rooms are

also used for meetings in the corporate context is conceivable

Technological trends such as the integration of eye tracking, face capture and

gesture recognition into current VR systems are an indicator that

interpersonal aspects, including emotions, should be mapped in the future

Starting Situation (2/2)

19.09.2019 4

Introduction



Collaborative virtual / augmented environments are

virtual spaces where multiple users work together in

real time

In comparison “multi-user-VR": does not define, if an

interaction with each other or against each other

(especially in gaming context) takes place

Environments can be completely virtual (VR) or a

combination of virtual and real world (AR)

The goal is to bring people together in a collaborative

virtual environment, preferably from multiple locations,

in order to exchange experiences and ideas

Participants can be represented by avatars

Participants can communicate with each other

(linguistically, gesture) and interact with other

participants or virtual objects

Definition „Collaborative virtual / augmented environments“

19.09.2019 5

Image: Facebook

Social VR-platformFacebook Spaces

Image: Spatial.is

AR-collaboration, realistic avatarrepresentation bySpatial

Basics

Image: MiddleVR

VR productpresentation;connection ofdifferent devices



Enables immersive meetings and workshops over

long distances

Enables exchange of information in real time, in

particular useful regarding visualization information of

3D models

Support of corporate groups and internationalization

activities

Reduction of travel costs and times

Acceleration of development cycles

Reduced error rates through increased and early

involvement of experts

Accelerated reactivity to incidents

Greatest benefit especially if the focus is on the

interdisciplinary planning of a complex object

Benefits

19.09.2019 6

Image: Eversheim 1995, S. 17

Basics



In most cases, a collaborative virtual environment

(CVE) system consists of several remote processes

that share the same set of data

The processes are usually distributed among many

(remote) computers connected through a network

A distributed system usually has the following

characteristics:

o Virtual environment data is replicated

o Passing on updates is time-critical

o Updates are asynchronous (eg non-blocking)

The replication of the data is important for performance reasons

Each computer that participates in the CVE simulation typically

renders the scene 30 to 100 times per second

The part of the scene being rendered must be read from

memory and sent to the rendering device

Distributed virtual environments: fundamental concepts (1/2)

19.09.2019 7

Image: Theoktisto, Fairén

Components of CVEs; yellow highlightedelements are referring directly tocollaboration aspects

Basics



Latency: the delay at which data sent to receiver

arrives, depending on the data line (telephone, LAN,

satellite), distance

Bandwidth: amount of data that can be transmitted per

unit of time

Reliability:

o Loss of information

o Corruption of data

o Visibility of information by strangers

Network protocol: Rules by which two applications

communicate

o Data formats, semantics (meaning of the message), error

handling

o Many protocols in practical use: ftp, http, realtime audio,

etc.

o For communication between applications, many protocols

are used simultaneously

Distributed virtual environments: fundamental concepts (2/2)

19.09.2019 8

Virtual environmentwith 3 avatarsImage: Fraunhofer IPA

Basics



Hardware compatibility

Data security, data hosting

Data preparation of the models to be visualized

Supported data formats

Mapping of physical laws

Communication: language, gestures, facial expressions

Interaction possibilities

Avatar representation and configuration

Usability / user interface

User management

Maximum number of users

Documentation of work results

Costs

Further important criteria regarding collaborative VR/ AR solutions

19.09.2019 9

Basics



Core idea of distributed virtual environments:

o All participants are present in the same environment

o State must be consistent for all participants

Trade-off consistency-performance:

o Information is generated on one host and mirrored on another

o Outdated information through latency in the transmission

o Without synchronization, mirrored information can only be trusted to a

limited extent

o Consistent states can only be achieved by synchronization, but these only

without having high update rates Contradiction between speed and

consistency

Thus, distributed virtual environments are either fast-changing dynamic

worlds or consistent worlds that provide identical information to all hosts

Distributed virtual environments: State management (1/2)

19.09.2019 10

Basics



Basically there are three approaches to state management:

1. Central information management

o State of the virtual environment is managed centrally

o Synchronized writing access

o Centre is the bottleneck, but can also distribute targeted

2. High-frequency information distribution

o Goal: Fast updates at the expense of network-wide consistency

o States of all hosts are sent completely and frequently to all other hosts

o Problem: Prevent multiple hosts from simultaneously manipulating the same object (eg via lock

manager)

3. State prediction

o Idea: Each host constructs approximations of the actual state between updates

o 2 phases: Prediction (dead reckoning) and convergence (in the actual state) after update

o Use of laws of physics, collision detection

19.09.2019 11

Distributed virtual environments: State management (2/2)

Basics



19.09.2019 12

Distributed virtual environments: Network architectures (1/2)

The most common network architectures:

Centralized Server: Centralized Primaries Modelo Based on the primary server replication model

o Primary scene is on a computer (server)

o All other computers are called clients and they only hold

the scene backup replicas

o When a client wants to update data, he must send an

update request to the server

o If the update is accepted after the scene consistency

check, the primary replica is updated and the message is

sent to all backup replicas to update their values

Peer-to Peer: Active Replicationo All computers replicate all collaborative scene data,

keeping the scenes fully synchronized

o Performance is limited by the slowest computer in the

system

Centralizedprimariesconsistencymodel

ActivereplicationconsistencymodelImage: Peciva, 2007

Image: Peciva, 2007

Basics



19.09.2019 13

Distributed Server (cf. concept Distributed

Scene Graph):

– Distributed Primaries Modelo Primary scene is distributed on the computers

o Different parts of the scene belong to different

computers and each computer is allowed to write

only in its part of the scene

– Data Ownershipo Primaries are allowed to migrate between computers

more flexible than Distributed Primaries Model

o If the computer owns a data item, it can read and

overwrite its values; the update will be sent to all

other computers

o Another computer may request access to data item

and if approved, also read and overwrite it

Distributed primariesconsistency model

Data ownershipconsistency model

Image: Peciva, 2007

Image: Peciva, 2007

Distributed virtual environments: Network architectures (2/2)

Basics



Technology:

Costs and selection of technology

Compatibility software and hardware

Bandwidth, latency

Privacy / data security

Creation of 3D data

User requirements:

Build up know-how

Physical restriction, cyber sickness

Social acceptance

Further:

Structure of a VR meeting (preparation,

(temporal) procedure, etc.)

Uncertain benefit

Challenges regarding the implementation of collaborative V/AR applications

19.09.2019 14

Image: VDC Fellbach

Results of theDesign ThinkingWorkshop „Collaborative VR/ AR“

Basics



Within the company

Live and promote a culture of change

Hiring VR specialists

Inteligent data policy

Ensure access to VR equipment

External

Set up of test environments / experimental rooms

Definition of standards

Publication of success stories

19.09.2019 15

Image: VDC Fellbach

Test environmentat VDC Fellbach

Image: VDC Fellbach

Image: VDC Fellbach

Approaches to implementing collaborative V/AR applications

Basics





Usage scenarios are diverse and just as

different

Variety of possible properties require

careful selection and testing

Main application fields:

1. Entertainment: Multi-player games (not

included in the following)

2. Presentation

3. Distance training

4. Assistance / Remote Support /

Teleoperations

Areas of application

19.09.2019 16

Image: Innoactive

VR training in logistics @ Audi with Innoactive

Image: ESI

Image: Re´flekt

IC.IDO enablescollaborative andimmersive Design Reviews

Remote Service Support by meansof AR technology

Areas of Application



19.09.2019 17

Area of application: Presentation

Virtual, interactive presentation of a product

Complex and large products presentable

Securing a common understanding, reducing

misunderstandings

Enables easy demonstrating of variants

In product development, design reviews

(distributed development on the same subject,

intuitive and interactive exploration, testing

prototype savings, product quality improvement,

lead time reduction)

Marketing/ sales, market research (accelerate

sales processes, product reviews)

Layout planning, virtual inspections

Image: rumii

Image: WeAre

VR meeting / conceptpresentation(rumii)

VR productpresentation withannotations(WeAre)

Image: vr-on

Layout planning: virtual inspectionof a factory (vr-on)




19.09.2019 18

Area of application: Training

Learning levels: explain, accompany,

examine

Introduction of documentation, simulators

Simulation scenarios

Visualization of the hidden

Training of dangerous situations

Training of processes in the team

Shortening downtime

Training already during planning / without

occupancy of the object / safe with

scenario technique

e.g. product trainings, process-oriented

trainings (eg assembly, surgery, etc.)

Image: University of Southern California

Assistant Steve explains andissuesinstructions

Image: vr-on

VR producttraining (vr-on)

Image: realworld-one

Training session(realworld-one)




19.09.2019 19

Plant training simulator ofthe Virtual Reality und Multimedia Park Turin:Definition of scenarios,processing of scenarios,linked 2D and 3D-perspective

Image: Virtual Reality and Multimedia Park, Turin

Area of application: Training




19.09.2019 20

Area of application: Assistance/ Remote Support

Use of a 3D environment as a knowledge

management platform to support remote

service personnel

Linking 3D environment to real system:

online 3D quickly conveys perspective of the

colleague on site

Use online 3D where cameras can not be

used (harsh environment, signal latencies)

On-site service staff can be supported with

3D information by AR systems

Image: Fh-IPA

Remote configuration of a machine tool

Image: Fh-IPA

Service technician in head office cansupport the on-site technician byoverlayingadditional information via a AR display

Image: tepcon

AR solution forsupport of servicingpurposes




VR / AR use cases have been segmented according to their implementation

complexity and achievable benefits for the automotive industry, manufacturing

and supply sectors

Remote expert / support solutions can be found in all three segments in

the “Must Do" segment - bringing with them a high level of benefit

combined with a low level of implementation complexity

Remote collaboration for design agreements and similar is classified as a

“Must Do“ as well (with particularly high benefit in the automotive sector)

AR is generally perceived as more beneficial, though more complex than VR

applications

Repair and maintenance are at the heart of implementation efforts: using real-

time visualization and information to complete a maintenance or repair task

Areas of application: Evaluation (1/2)

19.09.2019 21




19.09.2019 22

Image: Capgemini

Classification of „Remote Expert“ and „Remote Collaboration“ solutions as„Must Do“ (high benefitwith relatively lowcomplexity) – Example forthe automotive industry

Areas of application: Evaluation (2/2)




Presentation/ Design Review / Training

(VR / AR):

ESI

Innoactive

Lightshape

MiddleVR

Realworld One

Solid White

TechViz

vr-on

WeAre

WorldViz

VR-Meeting platforms:

Altspace VR

Facebook Spaces

VRChat

Market overview – Companies in this technological field

19.09.2019 23

Remote expert/ Assistance

applications (AR):

Atheer

CDM Tech

Ilogic

Re´flekt

Tepcon

TruPhysics

TZM

Wikitude

XM Reality

Market Overview

A detailed overview (over 60 solutions) of VR / AR

collaboration solutions can be found on the VDC

intranet.

A collection of specific application examples and

success stories will be published in the course of 2019.



Communication:

o (Spatial) audio

o Recognition of viewing direction,

gesture

o Virtual laser pointer

o Virtual whiteboards

o Speech-to-text

o 3D drawing function

Import function of 3D objects, pdfs,

etc.

Variation of the meeting environment

Configuration of avatars

Viewpoint navigation

Screenshots

Example functions of collaborative V/ AR solutions

19.09.2019 24

Interaction with 3D models:

o Exploded view

o Cross section

o Coloring

o Measuring

o Moving/ Rotating

o Annotations

Structure/ organisation:

o Access management, permissions

o Role management (moderator/

presenter, participants, etc.): sharing

perspective, teleport groups, muting,

etc.

o Documentation of the sessions

Market Overview



Use cases of distributed, collaborative VR / AR:

o Remote collaboration for presentation, meetings,

trainings

o Remote expert solutions

Top benefits:

o According to Capgemini: increasing efficiency and

productivity, increasing safety, saving time, reducing

complexity

o According to Deloitte: reduction of travel costs, increase

of flexibility and efficiency of work processes

Remote expert solutions are particularly promising

The magnitude of revenues generated in the future

depends heavily on developments in the hardware

sector (HoloLens & Co.)

Summary

19.09.2019 25

Summary

Image: Deloitte

VR/ AR study ofCapgemini andDeloitte

Image: Capgemini



19.09.2019 26

Outlook

Acquisition and transmission of gesture

as it is an important communication channel:

eg pointing direction, shrugging

Acquisition and transmission of facial

expressions important for face to face

communication: interpretation

Realistic avatars by scanning the

participants 3D projections in virtual or

remote conference rooms

Haptic feedback in specific interactions

increased immersion

Image: TU Chemnitz

Image: heise.de

Gesture recognitionin front of Powerwall (forscene navigation)

Kinect recordsmovements of mouthand eyebrows andthereby animatesavatars : wrinkling, nodding, smiling, lipmovements

Image: spatial.is

Realistic avatars, developed based on a 2D picture

Outlook



Sources:

WeAre Blogeintrag (2019):„How to conduct remote meetings more effectively“, verfügbar unter: https://weare-rooms.com/blog/

vr-on Whitepaper (2019): „Virtual Reality Kollaboration“

P. Ivanov (2017): „So leiten Sie virtuelle Teams“, verfügbar unter: https://www.wiwo.de/erfolg/management-der-zukunft/arbeitsform-der-

zukunft-so-fuehren-sie-virtuelle-teams/19385778.html

Capgemini Studie (2018): „Augmented and Virtual Reality in Operations: A guide for investment”, verfügbar unter:

https://www.capgemini.com/wp-content/uploads/2018/09/AR-VR-in-Operations1.pdf

Deloitte, Fraunhofer FIT, Bitkom (2016): „Head Mounted Displays in deutschen Unternehmen. Ein Virtual, Augmented und Mixed Reality

Check“ , verfügbar unter: https://www2.deloitte.com/content/dam/Deloitte/de/Documents/technology-media-telecommunications/Deloitte-

Studie-Head-Mounted-Displays-in-deutschen-Unternehmen.pdf

J. Peciva Dissertation (2007): „Active Transactions in Collaborative Virtual Environments“

VDC Fellbach Whitepaper (2013): „Collaborative Virtual Engineering“, verfügbar unter: https://www.vdc-

fellbach.de/fileadmin/assets/Publikationen/Whitepaper/2013_VDC-Whitepaper_Collaborative_Virtual_Engineering.pdf

V. Theoktisto, M. Fairén: „Enhancing Collaboration in Virtual Reality Applications”

Further literature:

S. Ren, F. D. Mckenzie (2015):„Collaborative Virtual Environment For Engineering Laboratory“

D. Livingstone, G. Creechan, J. Scullion (2013): „Learning about Collaborative Virtual Environments by Creating Collaborative Virtual

Environments“

J. Mütterlein, T. Hess (2018): „Specifics of Collaboration in Virtual Reality: How Immersion Drives the Intention to Collaborate“

C. Toinard (2000): „Patterns for Collaborative Virtual Reality”

E. Langbehn (2017): “Influence of Avatar Appearance on Presence in Social VR”

R. Rabätje, S. Menzel, M. Wochnig (2019): „Herausforderungen bei kollaborativen Anwendungen mit VR-HMDs“

D. Lake, H. Liu, M. Bowman (2010): “Distributed scene graph to enable thousands of interacting users in a virtual environment”

19.09.2019 27

Sources & Literature

https://weare-rooms.com/blog/

https://www.wiwo.de/erfolg/management-der-zukunft/arbeitsform-der-zukunft-so-fuehren-sie-virtuelle-teams/19385778.html

https://www.capgemini.com/wp-content/uploads/2018/09/AR-VR-in-Operations1.pdf

https://www2.deloitte.com/content/dam/Deloitte/de/Documents/technology-media-telecommunications/Deloitte-Studie-Head-Mounted-Displays-in-deutschen-Unternehmen.pdf

https://www.vdc-fellbach.de/fileadmin/assets/Publikationen/Whitepaper/2013_VDC-Whitepaper_Collaborative_Virtual_Engineering.pdf


VDC-members working in this technological field:

19.09.2019 28


Thank you very much for your interest!

You are interested in this topic and you are looking for contact persons/

implementation partners? Please contact us.

Dr.-Ing. Dipl.-Kfm. Christoph Runde

Virtual Dimension Center (VDC)

Fellbach

Auberlenstr. 13

70736 Fellbach

www.vdc-fellbach.de