The Virtual Twin: Controlling Smart Factories using aspatially-correct Augmented Reality Representation

Mareike Kritzler, Markus Funk, FlorianMichahelles

Siemens Corporate TechnologyBerkeley, CA, USA

firstname.lastname@siemens.com

Wolfgang RohdeSiemens PLM Software Inc.

Charlotte, NC, USAwolfgang.rohde@siemens.com

ABSTRACTWith the advent of digitalization in industrial settings, referredto as Industry 4.0, more and more machine data is being col-lected and analyzed during a machine’s life cycle. The avail-ability to leverage machine data allows the concept of theDigital Twin. The more data that is collected the more statis-tically relevant and reliable the digital representation. Thisdata within the Digital Twin is usually hard to access and toconsume by humans. Typically, there is just a web-based userinterface or basic queries available to mine the data. There-fore, we propose the concept of a Virtual Twin, which is aninteractive virtual representation (virtual 3D model) of a phys-ical object. The benefit of a Virtual Twin is that machinesand equipment are arranged in the same way and at the sameposition in relation to each other as their real-world counter-parts. The virtual 3D representation functions as an intuitivegateway to the corresponding Digital Twin. In this paper, wepresent the concept and implementation of the Virtual Twinand discuss potential use cases as well as areas of application.

ACM Classification KeywordsH.5.2. Information Interfaces and Presentation (e.g. HCI):User Interfaces

Author KeywordsDigital Twin; Virtual Twin; Augmented Reality; HoloLens

INTRODUCTION AND BACKGROUNDDigitalization of manufacturing and industrialization, referredto as Industry 4.0, aims at decision making and control ofmanufacturing shop floors based on data. One key concept inthis development is the so-called Digital Twin [1]. The DigitalTwin is an up-to-date and accurate digital representation ofthe physical object’s data values, states, and properties, e.g.shape, position, and/or throughput. This data is obtained bysensors that are attached to physical objects monitoring theobjects’ real-time status, working condition and communica-tion with other physical objects and systems. The Digital Twin

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

IoT ’17 October 22–25, 2017, Linz, Austria

© 2017 Copyright held by the owner/author(s).

ACM ISBN 978-1-4503-5318-2/17/10.

DOI: https://doi.org/10.1145/3131542.3140274

embodies the entire life cycle from the planning stage to liveoperational data up to modernization of an object or process.It therefore provides the foundation for connected productsand service. Thus, the Digital Twin can be seen as a digitalinformational construct of a physical system. This constructassembles all the relevant data as well as higher level domainspecific knowledge, e.g. maintenance intervals, in one centralplace.

While having a digital representation of a physical objectthat is positioned on the shop floor is a powerful tool, theaccess and interaction with this tool is usually limited, none-intuitive for humans and prone to errors. Traditionally, theseDigital Twins can be accessed using either a terminal, which isinstalled directly on a machine, or a computer, which accessesthe Digital Twin using a web interface (see Fig. 1a). If theaccess to the Digital Twin is not adjacent to the physical object,operators may have an issue identifying the correct DigitalTwin for an object on the shop floor. Augmented Reality(AR) technology can help to make the connection betweenthe digital and the physical possible. Previous work [2, 3] hasalready shown that AR can overlay a real-world object withdata and information of the Digital Twin. What if the physicalobject is in a remote location or not accessible due to harshconditions: heat, nuclear radiation or explosives, for example?

In order to bridge the gap, in this paper we introduce the ideaof a Virtual Twin which is a virtual representation of real-worldobjects located on a virtual shop floor. By retaining the layoutof the shop floor in a virtual representation, the data of theDigital Twin can be combined with the spatial representationof the real-world shop floor without being physically presentat a physical factory. Using this Virtual Twin, the data of theDigital Twin can be easily accessed by non-experts who arefamiliar with the physical layout of the shop floor.

A VIRTUAL TWIN FOR FASTER DATA ACCESSThe goal of the Virtual Twin representation is to support plantmanagers, supervisor, and service technicians by intuitivelyaccessing the data of a Digital Twin. We achieve that byusing the exact same spatial arrangement of objects for theVirtual Twin representation as they have on the physical shopfloor. Allowing users to easily access data using their priorspatial knowledge of the physical site. The Virtual Twin can beutilized to show virtual factory components and their currentstatus and values to a user and be an interface to the real-worldcontrol components (e.g. switch on/off). The Virtual Twin

(a) (b) (c)Figure 1. (a) A traditional web-based user interface that can be used to view data of a Digital Twin and control parameters of a smart factory. (b) Anexample visualization of a Virtual Twin using a spatial model of a smart factory to align the data and controls with the environment. (c) An interactivemodel of a smart factory that is connected to the cloud and can be remote controlled with our Virtual Twin application.

can be viewed from any location independent of its real-worldcounterpart or can be positioned on-site and aligned on top ofthe real-world factory components as an overlay. Allowinga human to have easy access to the digital information: TheVirtual Twins serve as a human-centric user interface for all ofthe digital information available for a specific machine. Thiscould be their physical properties, performance indicator, orrun-time information, such as error status during the executionof a work plan. In a scenario, where the physical machine isyet to be built, a Virtual Twin can show a human user what areal-world machine is supposed to look like.

Our proof-of-concept implementation uses a MicrosoftHoloLens for displaying a 3D model of a physical smart fac-tory for accessing its Digital Twin and controlling the shopfloor remotely (see Fig. 1b). By optically focusing on a stationof the smart factory, new information is requested from thesmart factory. In order not to overload the Virtual Twin, onlythe information that is current is being requested. The request,first goes to the cloud interface, which directs the request backto the smart factory. By designing the system with a cloudbuffer, a traditional Digital Twin interface for the smart fac-tory data is still possible. The Virtual Twin system then sendsthe data back to the HoloLens from the Cloud and displaysthe data directly on top of the machine model (see Fig. 1b).As objects in a smart factory are subject to changes of theirposition, e.g. mobile robotic units or changes in factory floorsetups, our system is designed to adapt to changes in positionsby listening to the position attribute of the Digital Twin.

To test our Virtual Twin system, we created a scenario using asmart factory model which is physically located in Charlotte,NC and a service technician wearing a HoloLens in a facilitylocated in Preston, NJ. The physical smart factory model (seeFig. 1c) consists of Raspberry Pis and a model train that areused as stations and bots of the smart factory. In our proof-of-concept implementation, the throughput times of the differentstations were manipulated by remotely turning the stationson and off using a digital switch that was next to the virtualmachine representation. Giving a service technician the liveproduction data needed and the ability to remotely control asmart factory.

Our initial tests show that interacting with smart factories andother smart environments on-site and from a remote locationcan be simplified by using a spatially-correct 3D representation

- a Virtual Twin - compared to interacting with a web-baseduser interface of the Digital Twin. In addition to adding ahands-free character to the data access, the main benefit forusers in using a Virtual Twin is that users can experiencethe factory floor remotely, which allows for faster access toinformation due to spatial representation of the factory in anAR headset. Thereby, a service technician can for exampleaccess “the machine that is located in the back of the factoryon the left side of the isle” instead of having to physicallyreconcile the correct machine by asset tag or identifier.

CONCLUSION AND FUTURE WORKIn this paper, we introduced the concept of a Virtual Twin,which provides a spatially-correct 3D representation of smartfactories for providing easier and more intuitive data access toa Digital Twin. Through a proof-of-concept implementationusing a Microsoft HoloLens and a model of a smart factory,we gain first hand insight in using a Virtual Twin for control-ling smart factories remotely. We argue that using a VirtualTwin will therefore help factory managers, supervisors andservice technicians to be more efficient and require less timeon-site. Using this system can prevent downtime and increasethe productivity of smart factories. In future work, we willfocus on further evaluating this concept with different types ofstakeholders that are involved in smart factories. We want todiscuss the different aspects of the visualization for stakehold-ers with different interests. E.g. a factory manager might wantto interact differently with a Virtual Twin than a customer, thatmight want to receive real-time updates on their productionorder.

REFERENCES1. Michael Grieves and John Vickers. 2017. Digital twin:

Mitigating unpredictable, undesirable emergent behaviorin complex systems. In Transdisciplinary Perspectives onComplex Systems. Springer, 85–113.

2. Steven J Henderson and Steven Feiner. 2009. Evaluatingthe benefits of augmented reality for task localization inmaintenance of an armored personnel carrier turret. InMixed and Augmented Reality, 2009. ISMAR 2009. 8thIEEE International Symposium on. IEEE, 135–144.

3. Dirk Reiners, Didier Stricker, Gudrun Klinker, and StefanMüller. 1998. Augmented reality for construction tasks:Doorlock assembly. Proc. IEEE and ACM IWAR 98, 1(1998), 31–46.