Laboratory for Manufacturing Systems and Automation - LMS7th CIRP Conference on Assembly Technologies and Systems –

CATS 2018

Dr. George Michalos

(michalos@lms.mech.upatras.gr)

Laboratory for Manufacturing Systems and AutomationUniversity of Patras

Application of Wearable Devices for Supporting Operators in Human-

Robot Cooperative Assembly Tasks

❑ Introduction

❑ Approach

❑ Wearable Device for Supporting Operators Collaboration with Robot

❑ Application Implementation

❑ System Integration

❑ Case Study

▪ Automotive case study

▪ White goods case study

❑ Conclusions

❑ Future Work

❑ Acknowledgments

Contents

Introduction: Challenges

Production environments volatility is not only external (demand):- Products have become modular and are updated on-production- Processes become hybrid depending on who is executing them- Behaviour of resources operating in them

- Humans –unique skills and characteristics- Resources – becoming intelligent and reconfigurable (e.g. robotics)

Enabling Humans for Changeability :- Past: Expert teams acting as assistants – intervening on site- Recent Past: Cognitive support by providing access to information - Lately: Customized information

- at the right place, - on the right time - in the right form

Wearable Technologies: Evolution

New devices with higher resolution and faster response in a compact size.

Augmented Reality DevicesSmartwatches

Wearables are of course not limited to these: smart gloves, mini projectors, exoskeletons and

many other are becoming available.

• Wireless connectivity (Bluetooth, Wifi, 3G/4G)

• Customizable Applications

• Pairing with phones/ PCs

• Input: Touch, voice etc.

• Processing power

• Sensors (Acceleration, gyro, etc.)

• Biosensors (heart rate, sweat etc)

• Lightweight

• All day battery

People have become familiar through every day use

IntroductionThe latest trends in EU manufacturing foster the Human-Robot co-existence in a collaborative environment,

sharing workplaces and tasks.

Introduction

- Robots can handle high payload and

repetitive tasks

- Human workers contribute with their

cognition capabilities and dexterity

• The goal is to provide a robust interaction and communication between Human and Robot resources.

• Enabling changeability by providing customized information at the right place, time and form

• Increase the operator’s experience and acceptance

Approach: Overall design of an HR system

Motion of the robot

Support provision on different organizational

levels

Multiple visualization means can be used

Instructions

The robot stops,moves

Parts & models

Restricted Areas

Wearable Device for Supporting Operators Collaboration with Robot (4/)

Approach: A Human – Robot collaborative assembly cell

Human Operator

Product

High Payload Robot

Applications for supporting the operator Supporting Operators Collaboration

with Robot (4/)

Approach: A Human – Robot collaborative assembly cell

AR glasses application Smartwatch application

Pairing process• Presents a unique ID as QR code

• Enables the connection with AR glasses and Execution System

• Running on various hardware devices

Wearable Device for Supporting Operators Collaboration with Robot (1/5)

AR scans the QR

Smartwatchgenerates QR

AR informs station

controller

Station controller connects these

devices

Task complete

• Pressed by the operator when a human task is done

• Station controller can synchronize the tasks among resources

Wearable Device for Supporting Operators Collaboration with Robot (1/5)

Station controller publish current

Human Task

Smartwatchinforms when this task is completed

Station controller coordinates the rest execution

Audio Commands interface

• Enables the Audio commands functionality

Wearable Device for Supporting Operators Collaboration with Robot (2/5)

• Speech recognition and translation in commands

(Android speech recognition API level 3)

• Increase the usability of the application

Manual guidance interface

Wearable Device for Supporting Operators Collaboration with Robot (3/5)

• Application provides two kinds of manual guiding the robot

1. Physical interaction

Operator using smartwatch enables the closed loop force control system and is able

to hand guide the Robot

Manual guidance interface

Wearable Device for Supporting Operators Collaboration with Robot (4/5)

• Application provides two kinds of manual guiding the robot

2. Arrow key buttons

Operator guides the Robot relatively with its TCP by pressing the arrow buttons

AR operator support application interface

Wearable Device for Supporting Operators Collaboration with Robot (5/5)

• Control the output information at AR glasses

1. Show/hide the production info

2. Show/hide 3D models

3. Show/hide robot’s trajectory

On long click checks the connection with the Station controller

Green indicates that it is connected

Red indicates that it is disconnected

Application Implementation (1/2)

Screen Interfaces

The Smartwatch application consist of:

• 10 stage buttons in total

• divided in 3 levels

Stage Button:

• The main button executes the presented functionality

• The label button presents:

- Current Stage title

- The connectivity with Station controller

Stage Levels:

• Each level contains similar functionalities

On single click sends the command to the Station controllerOn swipe left/right navigate through Stage Buttons on the

same level

On swipe up/down navigate through Stages Levels

Navigation between Stages:

Application Implementation (2/2)

Overall layout of all stages

2nd level

1st level

3rd level

System Integration

System Integration

Smartwatchrunning Android

AR glassesapplication

developed with

WIFI

ETHERNET

Station Controllerhandles the messages using

TCP and Web socket

Robot Controllerreceives commands

from station controller for execution over TCP

Force Torque sensorpublish the measured

data through

Assembly of car’s rear wheel axle

Application Examples (1/4)

▪ 1 High payload Robot

▪ 1 Real axle (25 kg)

▪ 2 Rear wheel group (12 kg)

▪ 4 Clips for fixing cables

▪ 8 screws for wheel groups

Hybrid Assembly Steps:

1. Loading Axle (Robot)

2. Loading Right Wheel Group (Robot)

3. Screwing the Right Wheel Group on the Axle (Human)

4. Assembly of the cable group (Human) – Loading the Left

Wheel Group (Robot)

5. Screwing the Left Wheel Group on the Axle (Human)

6. Assembly of the cable group (Human)

Assembly of car’s rear wheel axle (video)

Application Examples (2/4)

Link to

video

Refrigerator assembly stations

Application Examples (3/4)

▪ 2 Small payload Robot

▪ 1 Preassembly of cabinet

▪ 1 Flexible panel (polionda)

▪ 1 Conveyor system

▪ Foam sponges and pieces of tape

Hybrid Assembly Steps:

1. Loading Polionda (Human)

2. Hold polionda and manual guide Robot (Human - Robot)

3. Sealing the front section (Robot)

4. Assembly and fix the back side of polionda (Human) – Sealing the middle section (Robot)

5. Sealing the back section (Robot)

Refrigerator assembly stations (video)

Application Examples (3/4)

Link to

video

Conclusions

• Added value:

- Integrating humans in the manufacturing execution workflow with a dynamically configurable way

- Improve the worker experience compared to existing approaches (e.g. stationary touch screens

or physical buttons)

- Same application can be installed in various android smartwatches

- The interface with ROS framework makes easier the integration with other robotics application.

• Obstacles:

- Time for a user to get familiar with the application

- Navigation among different buttons could be difficult in case of many features.

- Customization time for adopting each use case requirements

Conclusions

Future work

Based on the end user feedback it is planned to:

• Improve the usability of the application :

- Swiping through buttons

- Touching gestures

- Pairing process with AR glasses

• Use additional sensors that smartwatches usually acquire

- Accelerometer , gyroscope

- Vibrator

• Develop programs for quicker and easier customization of the application from

non experts users

Future work

Acknowledgments The work of this paper has been partially funded by EC

research projects:

• ROBO-PARTNER – Seamless Human-Robot Cooperation for

Intelligent, Flexible and Safe Operations in the Assembly Factories of

the Future”

(Grant Agreement: 608855)

www.robo-partner.eu

• THOMAS - Mobile dual arm robotic workers with embedded

cognition for hybrid and dynamically reconfigurable manufacturing

systems”

(Grant Agreement: 723616)

http://www.thomas-project.eu

Acknowledgments

THANK YOU!

Laboratory for Manufacturing Systems and AutomationUniversity of Patras

www.lms.mech.upatras.gr