coversheet€¦ · 12/03/2018 · augmented reality at the systemic level provides the facility...

LA-UR-18-22070Approved for public release; distribution is unlimited.

Title: Augmented Reality Tools for Nuclear Facilities

Author(s): Mascarenas, David Dennis Lee; Morales Garcia, John Evan; Bleck, BrianMark; Katko, Benjamin Joseph; Green, Andre Walter; Dasari, Sudeep Rao;Gertsen, Henry; Ballor, JoAnn; McClain, Oscar; Mellor, Miranda; Yang,Yongchao; Blackhart, Craig; Cattaneo, Alessandro; Trujillo, Julio B.;Farrar, Charles Reed; Wysong, Andrew Russell; Harden, Troy Anthony

Intended for: Waste Management 2018, 2018-03-18 (Pheonix, Arizona, United States)

Issued: 2018-03-12 (Draft)

Disclaimer:Los Alamos National Laboratory, an affirmative action/equal opportunity employer, is operated by the Los Alamos National Security, LLC forthe National Nuclear Security Administration of the U.S. Department of Energy under contract DE-AC52-06NA25396. By approving thisarticle, the publisher recognizes that the U.S. Government retains nonexclusive, royalty-free license to publish or reproduce the publishedform of this contribution, or to allow others to do so, for U.S. Government purposes. Los Alamos National Laboratory requests that thepublisher identify this article as work performed under the auspices of the U.S. Department of Energy. Los Alamos National Laboratorystrongly supports academic freedom and a researcher's right to publish; as an institution, however, the Laboratory does not endorse theviewpoint of a publication or guarantee its technical correctness.

HoloLensMaster

Smart Cart 02

Safe 01

Scale 03

Augmented Reality Tools for Nuclear Facilities

David Mascareñas, John Morales, Brian Bleck, Benjamin Katko, Andre Green, Sudeep Dasari, Henry Gertsen, JoAnn Ballor, Oscar, McClain, Miranda Mellor, Yongchao Yang, Craig Blackhart,

Alessandro Cattaneo, Julio Trujillo, Charles Farrar, Andrew Wysong, Troy Harden,

Gloveboxes at PF‐4 Facility

Augmenting human senses can help reduce administrative criticality safety violations while working in gloveboxes

2016 ‐ Commercially Available Augmented Reality headsets have Arrived!

We Use the Microsoft HoloLens

• Windows 10 machine• Advanced Sensors and

Optics: Depth camera, RGB cameras, microphones, Inertial Measurement unit (IMU),

• Spatial Sound• Wireless comms: 802.11,

Bluetooth• Spectator Views (3rd

Person)

UNCLASSIFIED

Spatial mapping is advantageous because it knows where you are and where you

have been

HoloLens Internal Spatial Map

Spatial knowledge is key to information that is

• Relevant to user position

• Relational to the objects that surround

the user

UNCLASSIFIED

Our motivation is to augment a nuclear facility personnel with knowledge using a next generation, human-machine interface tool of the future for the

effective, efficient, and safe operation within a nuclear facility

SystemicFacility Level

Experiential Technician Level

Knowledge Waiting to be augmented

UNCLASSIFIED

Past

Concept for a Smart Nuclear Facility

In-Situ Generalized Augmented Reality

Nuclear Facility

Future

l3 add video once hololens is done updatingl211257, 7/13/2017

UNCLASSIFIED

Augmented reality at the systemic level provides the facility manager with a

complete view of real-time actions being performed within the facility

Goal:Augmenting the facility operator(s) knowledge with real-time data streams

Generalized Virtual Nuclear Facility

2D Floor Plan of Generalized Nuclear Facility

UNCLASSIFIED

The development of the Generalized Nuclear Facility has static and dynamic

elementsGeneralized

Nuclear facility

Static Element

s

Room Hallway Glove Box

Glove Box

Nuclear Material Handling and

Movement Documents

Current State of glove box

Interaction with Dynamic Elements

Controlling Sub-Elements

Drop Box Safe Cage Floor

Space

Dynamic

Elements

Personnel

Canisters Trolley Carts Drums

UNCLASSIFIED

The dynamic elements, such as material canisters,within the facility are mobile. Data can be associated

with these objects that can be used to facilitate planning, executing, and documenting material moves

l4 do a transition from photo1 to photo 2l211257, 7/13/2017

Augmented Reality in the Glovebox Setting

RFID + Scale

(1) Scan Operator Tag (2) Scan Item Tag (3) Weigh Item

Mass (g)

RFID Smart Cart and Safe

Planning the Material Move

Room 11.

Room 11. Operator authenticates themselves

into the system so that their actions are associated with themselves.

2. Take can, fill with material from primary can. Can should have a QR code that can be used to bring up safety information.

3. Use the networked scale in the room to weigh the can. The weight of the can is automatically associated with a timestamp and the properties of the can. The scale should have an NFC reader associated with it.

4. Place Can on smart cart. Smart Cart Should register user and can ID. Smart cart should communicate with user through the augmented reality headset . The smart cart should feature an NFC reader. It should ask for the user to scan in and then for the can to scan so the cart knows it owns the can and the user owns the cart.

3. Leave Room 1.

Room 21. Room 2 will have a QR code on the door that can be

associated with safety information for the room. 2. The smart cart with can is brought into room 2. The can is

removed from the smart cart and is then associated with a safe using an NFC reader. The safe is now considered the owner of the can. The safe also features a QR code that can be used to bring up holograms communicating information regarding the contents of the safe.

StartStart

FinishFinish

Smart Cart

Nuclear Material Canister

NFC Tag Reader With Microprocessor

and Wireless Networking

Operator

Augmented Reality Headset

NFC OperatorID Tag

Software Framework – Primary Components

Modules Settings Daemons

Relevant Settings• Modules are individual

programs to perform specific tasks that generally require user attention.

• Examples include collecting documentation and planning material moves.

• ModuleAPI.cs – All other elements of the system interface through the API

Relevant Settings

Daemon• Daemons are

background processes• Examples of Daemons

include automatically displaying information associated with the safety of nuclear materials canisters , and running criticality simulations.

• DaemonAPI.cs – All other elements of the system interface through the API.

• Setting can be used to turn Daemons on and off.

• Setting are used to configure the operation of Modules and Daemons

• Allows users to modify the settings within the app. Ideally users should be able to access the settings for the current app

Operators

Operator• Operator Info• Should eventually all

connect to a database. • Associates all current

actions performed with the augmented reality headset with the operator selected.

• Authenticates current operator.

Primary Components

Software Framework – support components

Databases

Databases• Material Safety

Datasheets• Criticality Safety

Information• Material chain‐of‐

custody• Equipment inventory• Operator

Qualifications • Machine Learning

Networking

Networking• External Sensors• External Database

information• External Agent

Communications• Includes appropriate

security (encryption)

Support Components

p

Modeling, Simulation and Optimization

Large‐scale computational resources to support facility operations.• High fidelity nuclear

criticality models.• Computationally/data

intensive optimization.• Computation requiring

global information for the facility.

Video Demo

Smart Nuclear Infrastructure Demonstration

When starting the system the operator is presented with a MainMenu button to access the features of the smart nuclearinfrastructure augmented reality interface.

Typically the operator would first indicate to the system their identity so that the system can attribute facility interactions with the appropriate operator.


The operator can then tell the software to connect with the smart nuclear infrastructure wireless network.


An example of a mock nuclear material canister that has been outfitted with a QR code that allows the augmented reality headset to bring up information associated with the canister using a holographic interface.


If the operator air‐taps the info button a holographic displayis made visible in the augmented reality headset showinginformation associated with the canister.


The holographic display can even be used to display video ofthe process of loading the canister which is important fordocumentation requirements in certain nuclear facilities.


Mock nuclear material is taken from the canister and placedon a networked balance. The measurement from the balanceassociated with operator and balance information is displayedin the augmented reality headset.


After weighing is complete the mock nuclear material is placeinside a new canister. The smaller canister on the right isprimary containment, and the smaller canister is then placedinside the secondary containment canister on the left.


The new canister is then placed on a smart cart to move it toanother room in the facility. The canister is associated withthe cart and the operator for the move as shown in theupdate information displayed in the augmented realityheadset.


A photo showing the new canister with the mock nuclearmaterial being moved with the smart cart to a new location inthe facility.


The doors of the mock facility have been instrumented withQR codes. The augmented reality headset can recognize theR codes. A holographic interface can provide information tothe operator regarding the contents of the room if they air‐tap the info button.


Once inside the room the operator can associate the mocknuclear canister and operator with a mock safe using the NFCtags.


Once the canister and operator and safe have beenassociated together, the operator can place the canisterinside the safe.


The safe is also instrumented with a QR code. This QR codecan be recognized with augmented reality headset so aholographic display can be summoned to provide the userinformation about the safe on demand.


We have also begun integrating additional sensors into the AR Headset

Next step is to fuse data with AR sensors and statistical classifiers for Enhanced Facility AwarenessThermal SHM methods?

Thermal Imagery examples

Conclusions and Path Forward:

>>• The emergence of Augmented Reality headsets opens up

many opportunities for vast improvements in structural inspection, smart infrastructure, facility management, construction, integration with IoT, visualization and training.

• If a form of Trusted‐3rd person Artificial Intelligence could be developed it could also be used for automatic work verification. This could be important for problems such as mitigating corruption in the construction industry. Particularly in the developing world.

• Integration with BIM and other Facility Management systems to build truly smart infrastructure.

• Smart PPE• Mitigates need for inspection robots

Mock Nuclear Facility

UNCLASSIFIED

Questions?2014 20142017 2017