augmented reality maintenance modelling - mer innovate · augmented reality: definition augmented...

29
Augmented Reality maintenance modelling

Upload: others

Post on 23-May-2020

19 views

Category:

Documents


0 download

TRANSCRIPT

Page 2: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Context and problematic

• Wind turbine are off-shore • ~1 hour for accessing a wind farm.

• Accessibility depends on weather conditions.

=> Little time to perform maintenance.

• Expert can’t always be physically on site.

• Expert may not be allowed to be part of the crew.

2

Need to

improve communication

Page 3: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Summary

Definition of augmented reality

Principles

Display technologies

Existing frameworks

Augmented reality maintenance modelling

Collaboration configuration

Works done

Next steps

3

Page 4: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented Reality: definition

Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive in real time 3. Registered in 3-D

Augmentation: virtual object added to the real world.

Real

environment

Virtual

environment

Augmented

reality

Augmented

virtuality

Reality-virtuality continuum

Mixed reality

4

Page 5: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented Reality: blower maintenance demonstrator

Demo done with

• Unity

• Vuforia SDK

5

Page 6: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented Reality : how it works?

Principle for augmented reality vision-based tracking (Azuma 1997)

• A camera gets the scene Iscene

• Based on the scene, camera pose is computed.

• When cam pose is known, the virtual object is added in the scene.

Scene image

Iscene

Features extraction

Iscene

Features matching between Iref et Iscene

Camera Pose computation

Adding augmentations

Page 7: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: display technologies

9

Display Technologies

Optical see-through

Optical Glance

Video see through

Hand held display

Projective worn

Projective fixed

Contact lens

Page 8: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: display technologies

10

Ideal configuration from user perspective

Page 9: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: display technologies

Tablet

Intuitive interface

Good resolution

Not hand free

11

Page 10: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: display technologies

12

Optical Glance Google glass-like

Hand free usage

Secure

Interface

Small display

Only glance

Page 11: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: display technologies

Optical see through

Hand free usage

Registered

Security issue

Small display

13

Page 12: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: display technologies

Projective

Natural vision

Limited virtual objects

Indoor usage

14

Page 13: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: display technologies

15

Display Technologies

Optical see-through

Optical Glance

Video see through

Hand held display

Projective worn

Projective fixed

Contact lens

Choice depends on the application

Page 14: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015 16

How to choose the right display technology?

Augmented reality: display technologies

Device Type Characteristics

Contac lens Optical see-

through Optical Glance

Video see through

Hand held display

Projective worn Projective fixed

Mobile + + + + + Limited Limited

Outdoor use + + + + + Limited Limited

User interface Gesture

or Tactile Gesture

or Tactile Gesture

or Tactile Gesture

or Tactile Tactile

Gesture or Tactile

Gesture or Tactile

Free hand usage + + + + + +

Field Of View + Limited Limited Limited + + +

Occlusion of real world by AR object

Yes Yes Limited Yes Limited No No

Unrestricted Augmentation model

Yes Yes Yes Yes Yes Limited Limited

Eyestrain Not determined Yes Yes Yes No No No

Power usage Limited Limited Limited Limited Limited Limited Unlimited

Available on the white paper by the end of March

Page 15: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented Reality: the most relevant frameworks

17

AR Framework Characteristics

Metaio Vuforia Wikitude OpenCV

Marker tracking Yes Yes Yes Dev

Image Target tracking

Yes Yes Yes Dev

3D tracking Yes Limited box and cylinder

No Dev

Face Tracking Yes No No Dev

Unity 3D Yes Yes Yes Need

Integration

iOS Yes Yes Yes Yes

Android Yes Yes Yes Yes

Windows Phone 8 No No No Yes

PC Yes Yes Yes Yes

Accelerometer and Gyroscope sensors

Yes Yes Yes Dev

Developer community

Yes Yes Yes Yes

Official Webinar Yes No No No

Free Yes but

Watermark Yes but

Watermark Yes but

Watermark Yes

Open Source No No No Yes

Page 16: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Expert – operator maintenance problematic

How the expert can quickly create Augmented Reality content for the operator, without development skills?

How can we accelerate AR Player development on current device and future ones?

18

Need to

improve communication

Page 17: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented Reality content production modelling

19

AR Editor SolidWorks

Authoring tool

Plugin

1

3D model

AR player Smart Glasses

or tablet

Standard

XML

file

Export

Ressources

2 3

Expert Opérateur

Page 18: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented Reality: maintenance modelling

• An AR maintenance is represented by a list of Action performed on Entity of the system (For example: Unscrew the screw 1).

• An Action is a simple Animation played on one of the Entity of the system to maintain. • The Action Unscrew means an animation starting from position, orientation

(x,y,z), (rx, ry, rz) = (0,0,0) (0,0,0) => (0,0,-50) (0,0, -720°) in 800 msec.

• An Action may be composed of Illustration = an Animation on an ExternalEntity.

• An ExternalEntity is a virtual object that is not part of the system. • For example display a screwdriver to show which tool use for disassembling an element.

20

Page 19: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented Reality: maintenance modelling

• We can define template Action like Unscrew, Screw,… that are reusable for several maintenance operations.

21

Page 20: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: maintenance configurations

• Guided maintenance 1. Expert prepares the maintenance tasks in AR.

2. Before leaving, operator loads the task he must perform from the DB.

3. Once on site, he performs the maintenance actions.

22

1

2

Operator

Expert

Remote

On site

3

Page 21: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented reality: maintenance configurations

• Assisted maintenance (or supervised maintenance) 1. Operator shares his view with the expert and explain his issue.

2. Expert augments the operator view in order to explain and assist him.

23

1

2

Operator Expert

Remote On site

Page 22: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Works done

• State of the art of augmented reality.

24

• State of the art of Augmented Reality devices and frameworks • We chose Metaio as framework and Moverio BT-200 as a smart glasses.

Page 23: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Works done

• 2 demonstrators of augmented reality

25

AR Demonstrator with marker

created with ARToolKit

(Alexis Barreau EXIA A2)

AR Demonstrator markerless

created with Vuforia and Unity3D

(Gael Lago EXIA A3 and Olivier Massot ECIA A4)

• AR maintenance modelling v1 (Dorian Boulc’h EXIA A4)

Page 24: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Works done

Conferences

• 4 Mer Innovate Conferences : CESI Rouen 21 of May, Exeter University 29 of October, Technopole Cherbourg 9 December, Portsmouth 18 of March.

Papers

• IEEE VR conference demo (25-27 March 2015, Arles) Augmented Reality maintenance demonstrator and associated modelling.

• Mer innovate white paper available by the end of March on www.merinnovateproject.eu

26

Page 25: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Perspectives

• Evolution of the AR maintenance modelling. Adding operator review of the process.

• Based on that model, development of the AR Player on the EPSON Moverio BT-200.

• Creating maintenance scenario.

• Testing the 2 maintenance configurations and evaluate augmented reality for maintenance.

• Study the Gesture User Interface usage for smart glasses.

27

Page 26: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Augmented Reality for maintenance

Thank you

28

Page 27: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Bibliography

(s.d.). ARToolKit. Récupéré sur www.hitl.washington.edu/artoolkit/

Azuma, R. T., & others. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. Dans Computer Vision--ECCV 2006

(pp. 404-417). Springer.

Bottecchia, S. (2010). Système TAC: Télé-Assistance Collaborative. Réalité augmentée et NTIC au service des

opérateurs et des experts dans le cadre d'une tâche de maintenance industrielle supervisée. Ph.D. dissertation,

Université Paul Sabatier-Toulouse III.

Bottecchia, S., Cieutat, J.-M., & Jessel, J.-P. (2010). TAC: augmented reality system for collaborative tele-assistance

in the field of maintenance through internet. Proceedings of the 1st Augmented Human International Conference,

(p. 14).

Caudell, T. P., & Mizell, D. W. (1992). Augmented reality: An application of heads-up display technology to manual

manufacturing processes. System Sciences, 1992. Proceedings of the Twenty-Fifth Hawaii International Conference

on, 2, pp. 659-669.

Corke, P. (2011). Robotics, vision and control: fundamental algorithms in MATLAB (Vol. 73). Springer Science \&

Business Media.

Grasset, R., Gascuel, J.-D., & others. (2003). Réalité Augmentée et environnement collaboratif: Un tour d'horizon.

AFIG'03.

Harris, C., & Stephens, M. (1988). A combined corner and edge detector. Alvey vision conference, 15, p. 50.

Henderson, S., & Feiner, S. (2011). Exploring the benefits of augmented reality documentation for maintenance and

repair. Visualization and Computer Graphics, IEEE Transactions on, 17(10), 1355-1368.

29

Page 28: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Bibliography

icons8. (2014). Icons8. Récupéré sur http://icons8.com

Kalkofen, D., Sandor, C., White, S., & Schmalstieg, D. (2011). Visualization techniques for augmented reality.

Springer.

Koch, C., Neges, M., König, M., & Abramovici, M. (2014). Performance Study on Natural Marker Detection for

Augmented Reality Supported Facility Maintenance. Australasian Journal of Construction Economics and Building-

Conference Series, 2, pp. 23-34.

Lieberknecht, S., Benhimane, S., Meier, P., & Navab, N. (2009). A dataset and evaluation methodology for

template-based tracking algorithms. Mixed and Augmented Reality, 2009. ISMAR 2009. 8th IEEE International

Symposium on, (pp. 145-151).

Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International journal of computer

vision, 60(2), 91-110.

Mallem, M., & Roussel, D. (2008). Réalité augmentée: principes, technologies et applications. Techniques de

l'ingénieur. Télécoms(TE5920).

MerInnovate. (2012). Mer Innovate project. Récupéré sur http://merinnovateproject.eu

Metaio. (s.d.). Tracking configuration. Récupéré sur https://dev.metaio.com/sdk/documentation/tracking-config/

30

Page 29: Augmented Reality maintenance modelling - Mer Innovate · Augmented Reality: definition Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual 2. Interactive

Mer Innovate final event in Technopole at Portsmouth, 18 March 2015

Bibliography

Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information

and Systems, 77(12), 1321-1329.

Nee, A., Ong, S., Chryssolouris, G., & Mourtzis, D. (2012). Augmented reality applications in design and

manufacturing. CIRP Annals-manufacturing technology, 61(2), 657-679.

Rosten, E., & Drummond, T. (2005). Fusing points and lines for high performance tracking. Computer Vision, 2005.

ICCV 2005. Tenth IEEE International Conference on, 2, pp. 1508-1515.

University, W. (2011). CSE/EE 576: Computer Vision. Récupéré sur

http://courses.cs.washington.edu/courses/cse576/11sp/,

http://courses.cs.washington.edu/courses/cse576/11sp/notes/SIFT_white2011.ppt

Van Krevelen, D., & Poelman, R. (2010). A survey of augmented reality technologies, applications and limitations.

International Journal of Virtual Reality, 9(2), 1.

Zhou, F., Duh, H. B.-L., & Billinghurst, M. (2008). Trends in augmented reality tracking, interaction and display: A

review of ten years of ISMAR. Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented

Reality, (pp. 193-202).

31