d.1.3 system architecture v1 - inclusive · smartandadaptiveinterfacesfor&&...

Smart and adaptive interfaces for INCLUSIVE work environment

Grant Agreement N°723373

________________________________________________________________________________

Deliverable 1.3 – Definition of INCLUSIVE system specifications and architecture

Contractual delivery date: 2017-‐05-‐31

Actual delivery date: 2017-‐05-‐31

Responsible Partners: PROGEA, SOFTWARE FACTORY

WP n°: 1

WP leader: UNIMORE

Project Coordinator: Prof. Cesare Fantuzzi

Project Coordinator Organization: UNIMORE

Dissemination level: Public

The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement N723377.

The author(s) is/are solely responsible for its content, it does not represent the opinion of the European Commission and the Commission is not responsible for any use that might be made of data appearing therein.

Executive Summary

This deliverable describes the overall system architecture and the deduced specification description of the INCLUSIVE Project, based on the Requirements derived from the three industrial use cases described in the deliverable D1.1 of January 31, 2017.

Task description from Annex I of Grant Agreement

This task develops the INCLUSIVE system architecture. Starting from PROGEA experience in industrial HMI systems, with the specifications collected by the industrial and academic partners, the hardware for the HMI implementation is defined from off-‐the-‐shelf systems (touch panel PCs or similar solutions, smartphones, tablets, etc.). The communication protocol with portable and personal input systems (speech synthesizers, wearable sensors, etc.) is then defined as well as the interface with the control system of the machines (WP 5). Finally, the software architecture is defined to allow modular development into three subsystems: measurement module (WP 2), adaptation module (WP 3) and training module (WP 4).

Table of Content

Executive Summary ______________________________________________________________________ i

Table of Contents _______________________________________________________________________ ii

1 Introduction, Concept and Strategies ___________________________________________________ 1

Objectives _____________________________________________________________________ 1

INCLUSIVE Approach _____________________________________________________________ 2

Adaptive User Interface (AUI) ___________________________________________________ 3

Adaptive presentation ____________________________________________________ 3

Adaptive navigation ______________________________________________________ 3

Context-‐Sensitive User Interface (CSUI) ___________________________________________ 3

2 System requirements ________________________________________________________________ 4

3 Description of the use cases ___________________________________________________________ 5

Use case 1: SCM ________________________________________________________________ 5

Use case 2: Gizelis / Silverline ______________________________________________________ 5

Use case 3: KHS _________________________________________________________________ 6

4 INCLUSIVE system architecture ________________________________________________________ 7

Measurements module __________________________________________________________ 7

Adaption module _______________________________________________________________ 8

Support and teaching module _____________________________________________________ 8

Interaction and communication between the three INCLUSIVE modules ____________________ 9

External sensors and additional output devices ______________________________________ 11

Adaptive HMI _________________________________________________________________ 11

Adaptive automation middleware _________________________________________________ 12

ThingWorx _________________________________________________________________ 13

Kepware server (middleware Interface) __________________________________________ 13

OPC UA (Unified Architecture) _________________________________________________ 14

ThingWorx test environment __________________________________________________ 14

Components of the ThingWorx middleware controller ______________________________ 14

5 The INCLUSIVE specification and its implementation ______________________________________ 16

Specification for requirements 1 and 7 _____________________________________________ 16

Use Case 1 (SCM) ___________________________________________________________ 19

Configuration of automatic Tool warehouse and Worktable of woodworking machines 19

Use Case 2 (Gizelis / Silverline) _________________________________________________ 20

Use Case 3 (KHS) ____________________________________________________________ 21

Specification for requirements 2 and 4 _____________________________________________ 21

Use Case 1 (SCM) ___________________________________________________________ 22

Configuration of automatic Tool warehouse and Worktable _____________________ 22

Routine and procedure for maintenance ____________________________________ 22


Use Case 3 (KHS) ____________________________________________________________ 23

Specification for requirement 3 ___________________________________________________ 23

Specification for requirement 5 (Use Cases 1, 2 and 3) _________________________________ 23

Use Case 1 (SCM) ___________________________________________________________ 24

Routine and procedure for maintenance ____________________________________ 24


Use Case 3 (KHS) ____________________________________________________________ 24

Specification for requirement 6 ___________________________________________________ 25

6 Conclusion ________________________________________________________________________ 26

1

1 Introduction, Concept and Strategies

Objectives Starting from the requirements collection and analysis of the three use cases (Figure 1) addressed within the INCLUSIVE project, herein we describe the overall system architecture and system specification for INCLUSIVE.

Figure 1: Structural analysis of INCLUSIVE: from industrial use cases to system specification

The diagram in Figure 2 depicts a simplified concept of the INCLUSIVE strategy that correlates operator’s skills, HMI complexity and demanded support/training to ensure optimal usability for each use case. This strategy combines techniques of adaptive-‐ and context-‐sensitive user interfaces, integrated with off-‐ and/or on-‐line support and teaching strategies.

Figure 2: Adaption diagram of HMI-‐complexity and HMI-‐support related to the skill level of the operators

The overall objective of INCLUSIVE is to develop a new concept of interaction between the user and a manufacturing system in which the behavior of the automation system adapts to human operator capabilities, through implementation of variable HMIs to exploit the assessment of user’s capabilities derived from Work Package 1 (WP1) and Work Package 2 (WP2).

2

INCLUSIVE Approach The results obtained from the measurement module (WP2) will be used to determine adaptation-‐criteria, to adapt HMIs and Interfaces.

Adaptation occurs at both sensorial and cognitive levels (Figure 3); development of universal adaptation patterns will be used in INCLUSIVE to build customized HMIs according to the applications.

Figure 3: Types-‐, criteria-‐ and HMI-‐adaption factors

For each use case, the existing HMIs of the industrial partners will be integrated with additional functionality by means of Adaptive User Interfaces (AUI) and Context-‐Sensitive User Interfaces (CSUI), which are discussed hereafter.

The meta-‐HMI core in Figure 3 can be seen as a set of HMI modules that bring predefined INCLUSIVE functionalities to any application and includes several modules. Amongst the possible modules there are:

- Real-‐Time IO Server Module, responsible for the real-‐time communication between the middleware and the HMI using OPC UA protocols.

- User Management Module, that can be preconfigured in order to include the user profiles defined by the measurement module.

3

- Graphic HMI Templates Module: A set of Template screens with predefined navigation, help buttons etc., it helps to design several types of customized applications depending on the user profiles.

- Online help Module: A set of Template Popup Screens that contains media controls such as web browsers, pdf viewers and mp4 streaming viewers. The material displayed by this module (documents or video) will be archived in the middleware.

- The Alarm Management Module: This module informs the user about problems, events, needed maintenance, etc…

- The Augmented Reality Module: This can support an operator by means of smart glasses in order to access remotely computer configuration tools.

The implementation of these modules is described in Section 5 of this deliverable.

Adaptive User Interface (AUI) AUIs are user interfaces which adapts, i.e. change their layout and elements depending on the skills of the user or the context. The properties of adaptive HMIs allow to show only relevant information based on the current user (adaptive presentation), or how users utilize the application (adaptive navigation).

Adaptive presentation The goal behind adaptive presentation is to display information based on the current user. This may mean that users with basic knowledge of a system will only be shown minimal information. Conversely, a user with advanced knowledge will have access to more detailed information and capabilities.

One way the AUIs can achieve this differentiation is to hide information to be presented based on the user's experience level.

Adaptive navigation This intends to guide users between pages on the GUI by altering the way the system is navigated depending on user’s expertise level within the system and other relevant factors.

Context-‐Sensitive User Interface (CSUI) A CSUI is a help implementation which support a user to automatically choose from a multiplicity of options based on the current or previous state(s) of a program operation. Simplification is achieved for instance by reducing the number of commands and options, number of “clicks” and “keystrokes” required to carry out a given operation.

Context sensitive help is a common implementation of context sensitivity, a single help button is clicked and the help page or menu will open a specific page or a related topic.

4

2 System requirements Figure 4 summarizes the main system requirements, representing the main goals (WHAT) of the INCLUSIVE project.

Figure 4: System requirements, derived from the description of the use cases and of the user needs

A detailed description of system requirements and their relationship with identified user needs of target user groups (elderly, disabled and inexperienced) have been reported in the deliverable D1.1 “Summary of all the requirements”. Therein, the role of three INCLUSIVE modules (Measure, Adapt, and Teach) and how they address each of the system requirements is also fully described.

5

3 Description of the use cases The following figure recalls the use cases that will be addressed in the INCLUSIVE project, as described in D1.1.

Figure 5: Industrial use cases object of the INCLUSIVE Project

Use case 1: SCM Tuning of the components of woodworking machines

INCLUSIVE will deal with a main issue concerning complex configuration procedures on existing HMIs, where little help or guidance is provided to the user.

Currently, for such an operation, the user has to memorize the configuration values done on the HMI, go to the machine and report with the help of a mobile panel, the HMI-‐configuration settings.

Thus, without guidance or help-‐tools, the risk to make mistakes in this operation is high, especially for low-‐skilled operators.

Ordinary maintenance procedures for the woodworking machine

Currently, maintenance procedures for SCM-‐woodworking machines are controlled by the alarm management system of the HMI, by monitoring machine-‐parameters and counters. If threshold-‐values went out of range, alarm warnings are displayed.

Since machine maintenance are usually operations that are performed within a certain time-‐interval, without an efficient interactive help support the risk to forget maintenance procedures is high.

Use case 2: Gizelis / Silverline

Bending parts and ordinary maintenance operations

With the Silverline use case, INCLUSIVE focuses on the bending profile tolerances of bending tools mounted on the press, as well as supporting operators by replacing malfunctioning parts of the machines.

Bending profile tolerance is a critical factor in bending processes, since the set parameter values need to be compatible with robot configuration parameter values, in order to reduce risks of collision with robots.

6

The Gizelis-‐HMIs that will be implemented in Silverline bending machines currently do not guarantee such prerequisites.

Use case 3: KHS Changeover is required when a new format of bottles has to be labelled.

This process is a time-‐consuming and a complex procedure that consists of replacing and correcting the loading of many components of the KHS Innoket Neo labeling machine for bottles.

Currently, the operator receives almost no assistance from the KHS proprietary HMI. If the operator loads machine elements that do not match the format selected on the HMI (e.g., because he/she has selected the wrong format on the HMI), the HMI does not notify the mismatch. The only result displayed to the operator is that the machine did not start and no technical hints are provided.

Considering that a machine can host up to 20 formats, it is very likely that mismatching errors, such as the one reported above, occur frequently.

7

4 INCLUSIVE system architecture

Figure 6: High level system architecture of INCLUSIVE

Figure 6 shows a model of the system architecture of INCLUSIVE at high level, depicting the main components involved in the communication between operators and machines. The detailed description of each component of the diagram is reported in Subsections 4.1 to 4.7.

Within the data obtained from the three INCLUSIVE modules (Measure, Adapt, Teach), the communication between operators and machine will be mediated by a complex set of operative interactions between HMIs and the adaptive middleware.

Measurements module The measurement module consists of three sub modules: a-‐priori measurement, real-‐time measurement and longitudinal measurement. Regarding the first module, relevant limitations of human-‐machine interaction due to the user’s capabilities will be considered a-‐priori. In the second module the actual individual cognitive load will be assessed in real-‐time during user operations. Finally, the third module detects user performance in the long-‐term.

- A-‐priori measurement: relates to problems of the operator-‐interaction with the HMI, most relevant characteristics of the user that would have an influence on interacting with the machine will be screened by offline assessment tools, such as questionnaires, tests for perceptive and cognitive/motoric capabilities.

- Real-‐time measurement: tracks individual cognitive load of the user in real-‐time via analysis of physiological parameters, such as pupil diameter, blinking rate, skin conductance, body temperature,

8

hormonal balance and heart rate variability. The measurement is done by means of proximal and distal measurement techniques.

- Longitudinal measurement: user behavior regarding performance indicators will be tracked, e.g. time to make decisions, execution steps for the task, mistakes, and redundancies. Data can be used for the development of structural knowledge maps, e.g. registering the training evolution to give support to the operator in the future.

The data output of the measurement module provides the teaching and adaption module with data input. Interfaces between the modules have yet to be defined.

Adaption module The adaption module aims at introducing a methodology to change the way the user interacts with the machine. This is obtained by adjusting the way information is presented and modifying the set of operations the user can perform. Such adjustment of the HMI is performed based on the results of the Measurement module, which provides an off-‐line computed user profile that is updated online during interaction. Such operator profile is used to determine the most appropriate level of adaptation in the HMI.

Adaptation is performed to accommodate physical, visual, auditory and dexterity impairments of users. Moreover, the cognitive capabilities of workers, either novice or experienced, are taken into account to provide cognitive adaptation to the HMI.

Specifically, a meta-‐HMI that is general-‐ and dialogue-‐independent from the hardware (Figure 3), is firstly developed, thus devising general design patterns. Then, it is customized to match the application systems of the considered use cases: at this stage, the developed interfaces share the same conceptual core, but are still independent from hardware targets. As a last step, the application-‐matched meta-‐HMIs are linked to the control system of the considered machines and tests are carried out with industrial partners.

Support and teaching module This module develops a system that integrates a HMI, a virtual environment and an industrial social network to support and train low skilled operators to accomplish a complex automation task property.

The teaching module supports the user by providing initial offline training as well as on line training. The provided support should be adapted to the capabilities of the user and his current behavior and state. The module provides two kinds of training:

- offline training is often an initial training that is carried out in a separate environment using virtual reality technology to familiarize a worker with a task.

- online training is offered at the workplace after the user has received an initial offline training. It provides guided procedures or short training units that can be consumed at the workplace using, for instance, mobile devices or smart glasses.

To detect when the user needs training, the teaching module needs to have access to the output of the measurement module. This allows keeping track of the current situation, for instance the knowledge state of the user, his emotional situation or his current task. Based on this data, the system would then be able to provide support if it detects stress or mental strain.

9

Furthermore, the module needs access to data about the users of the system and the responsibilities and capabilities to facilitate the interpretation of the real-‐time measurements.

Figure 7 reports the interfaces that will be implemented within the INCLUSIVE teaching module:

1) a wireless headset and wireless eye tracker devices will be used in the INCLUSIVE teaching module; 2) The measured data will consist mostly of speech and eye tracker (Gaze Point, Pupil size); 3) For the on-‐line support systems, INCLUSIVE will use a mobile device (e.g., Android-‐based tablet or

Smart Glass, Windows tablet) featuring common interfaces (e.g., Bluetooth, USB, Wifi).

Figure 7: Interfaces of the teaching module.

Interaction and communication between the three INCLUSIVE modules The measurement, the adaptive and the teaching module communicate with each other through the middleware controller (e.g. using REST-‐protocol) according to the modular architecture of INCLUSIVE. The middleware will further establish the communication with the HMI. Figure 8 shows some details of the interaction as well as the static profiles (determined a priori by the INCLUSIVE measurement module), stored in a shared database accessible for the HMIs via the middleware.

10

Figure 8. Representation of the Interactions between the three INCLUSIVE modules, of static-‐ and dynamic-‐user profiles and their

communication with the middleware (MW: MiddleWare, TW: ThingWorx, UC: Use Case)

Through the INCLUSIVE measurement module, a set of standard static user profiles will be defined. Depending on their a-‐priori, real-‐time or longitudinal measurements, these will be associated to one of the standard profiles developed in the HMI, by using the user manager module of the Progea Movicon.Next platform (Figure 8).

Figure 9: Diagram showing the communication flow within the Measurement module, the Adaptive module and the derived User Profile implemented by the HMI.

Each HMI User profile will be added to a group where users with predefined skills are associated to define a set of attributes that will determine HMI-‐components visibility and accessibility.

Additionally, incoming Inputs from the adaptive module can promote dynamic reconfiguration of HMI at runtime, in case a user with a different skill profile is recognized by the system.

Identification of users occurs by user name and password or by using biometric sensors interfaced to the HMI.

Adapt Teach Measure

UC1: MW/TW Controller



Dynamic Dynamic

UC-‐specific data

UC1 UC3UC2

External sensorsAdditional Output Devices

Static user profiles Use case data

UC1 UC3UC2

11

External sensors and additional output devices External sensors such as wristbands (https://www.empatica.com/e4-‐wristband) and T-‐shirts with an adrenaline sensor will be worn by study participants. These will allow the measurement of skin temperature, electrodermal activity (EDA), blood volume pulse, acceleration (wristbands) and adrenaline level (T-‐shirts). To measure oculomotoric functions, such as pupil diameter, gaze point and blinking rate, two types of eye tracking devices will be used: The remote eye tracker will be placed in front of the HMI, the wireless eye tracking glasses will be worn by participants. Moreover, a (wireless) headset will be worn that allows the measurement of speech input.

Communication of the wristbands and t-‐shirts with the middleware can take place through a USB port for offline and Bluetooth for real-‐time connectivity as regards the wristbands, and USB and Bluetooth as regards the T-‐shirts. Specifically, the wristbands exploit standard Low Energy Bluetooth to communicate with computers. The eye trackers use USB ports and TCP/IP protocols for data output. Speech sensors are connected via Bluetooth.

Real-‐time monitoring of the above mentioned physiological parameters will be achieved by means of two independent applications to be installed on computers/laptops: one application will be used for measurement with wristbands, and another one will be used with T-‐shirts. While in a first attempt two computers will be used (one per application), the possibility to run both applications on a single computer will be investigate at the moment of building up the whole system. Specifically, an API is available for wristbands and an application for wristbands will be developed using MATLAB for Windows 7, which uses Empatica Ble Server for Windows (http://developer.empatica.com/windows-‐ble-‐server.html). The same approach will be adopted for the T-‐shirts with an adrenaline sensor. Also monitoring oculomotoric functions requires distributed architecture, meaning that data analysis programs will run on external computers.

Special attention should be paid to data security and privacy of study participants. In particular, it must be ensured that other workers, including supervisors and members of the public, have no access to the results and findings of the measurements. Participants’ information and data, including physiological recordings, have to be treated confidentially and analyzed anonymously, as described in Deliverables 10.1 – 10.3 (Ethical issues). Speech input might use cloud recognition service, which requires Internet connection. Problems could occur, if Wifi is not available at the use case providers’ premises.

Adaptive HMI In the system architecture of Figure 6, Progea HMIs (as described in Section 1.2) refer to the INCLUSIVE HMIs that will be developed by implementing the meta-‐HMI functionalities by using the Progea Automation Platform.NExT™, a scalable software architecture for the development of HMI solutions.

The approach that will be developed in INCLUSIVE will be based on the concepts of structural modelling, dialogue independence from hardware and rapid prototyping. In particular, dialogue independence from hardware separates the design of the interface from the design of the computational component of an application system so that modifications in either subsystems does not cause major changes in the other one.

One of the main components of Automation platform.Next is the powerful HMI/SCADA package: Movicon.Next is a SCADA/HMI solution (Figure 10).

12

Figure 10: Communication flow of realtime information with Movicon.Next of Progea

The structure of the product is modular; it is composed by a framework that represents the core of the product and various components or plug-‐in offering specific features. The main module is the I/O Data Server that includes the components responsible for the data acquisition from the field, alarm management and data storage or historian. Additional modules like recipe manager, scheduler or event manager, alarm dispatcher and the graphic interface are available.

As already mentioned, INCLUSIVE user profiles are defined by means of:

o predefined user patterns, o information coming from wearable sensors, o dynamic information (or measurements) coming from the interaction between the user and the HMI.

INCLUSIVE HMIs must be able to recognize user profiles and at the same time dynamically adapt to the connected user, offering HMIs with the right level of complexity. Interactive help can be available anywhere thanks to the media controls, menus, function keys or shortcut, the tooltips and the possibility to create contextual popups. This task is performed from an integrated user manager in the I/O Data Server module of the Progea platform (Figure 10).

Adaptive automation middleware The adaptive automation middleware realizes the interfaces between the different interacting components of the project, allowing the operational modes of the machines/robotic cell adapt to the user.

The middleware that is developed in INCLUSIVE will enable the integration of the machines with the HMI, but will also be open to different IT systems used within the companies, such as MES or ERP. Therefore, the middleware will be modular, allowing the integration of new components as the means to interact with machines evolve.

In INCLUSIVE, the communication between the HMIs and the machines of each use case is mediated by a multilayer middleware, based on Thingworx and Kepserver technologies (Figure 11).

13

Middleware

User Profiles / Use Cases

OPC

HMI

ERPMES

SCADA

OPC UA

Machine / RobotHardware drivers & protocols

Middleware controller Middleware interface

ODBC/JDBC

GUIsO Captain! My Captain! Our fearful trip is done

The ship has weather’d every rack, the prize we sought is won;;The port is near, the bells I hear, the people all exulting,While follow eyes the steady keel, the vessel grim and daring:But O heart! heart! heart!O the bleeding drops of red,Where on the deck my Captain lies,Fallen cold and dead.

O Captain! my Captain! rise up and hear the bells;;Rise up—for you the flag is flung—for you the bugle trills;;For you bouquets and ribbon’d wreaths—for you the shores a-crowding;;For you they call, the swaying mass, their eager faces turning;;Here Captain! dear father!This arm beneath your head;;It is some dream that on the deck,You’ve fallen cold and dead

Walt Whitman1865O! Captain my Captain!

Informationen

Informationen

Informationen

Informationen

O Captain! My Captain! Our fearful trip is done




Informationen

Informationen

Informationen

Informationen

O Captain! My Captain! Our fearful trip is done




Informationen

Informationen

Informationen

Informationen

Integrated with IT-Services and Standards: AutomationML with

ecl@ss, Weihenstephaner with

MES-ML

Figure 11. Components of the INCLUSIVE middleware and the communication level between the components

ThingWorx The ThingWorx® Technology Platform (TW) enables rapid application development and deployment of smart, connected solutions for the Internet of Things, business systems and big data analytics. The platform is based on the concept of the object orientation which allows a modular construction and rebuilding software and hardware components within the middleware. It includes built-‐in functionality to manage and control connected devices including their data and built-‐in interfaces to connect with other systems. ThingWorx can be easily extended by means of existing extensions like DB connectors or protocol implementation. It also offers an DK (Software Development Kit) for developing customized extension and even for building applications for devices to collect data and send them via secure protocol to ThingWorx.

The INCLUSIVE system will make use of this platform in order to:

- manage store and control data - business logic implementation (automated adaption to components of the INCLUSIVE System) - ensure connectivity (through the Kepware server) - analysis (real-‐time anomaly detection, predictive analytics and simulations) - experience (use of augmented reality in order to create experiences that help end users)

Kepware server (middleware Interface) The Kepware Server KEPServerEX® is providing secure, reliable communications between the hardware layer and software application layer. The Kepware System offers drivers for more than 160 different controllers and is OPC/UA-‐capable. In the INCLUSIVE system the Kepware Server will ensure connectivity of software components as well as hardware components. Therefore, all the data and information of the whole infrastructure will be combined in one system and the number of external communications connections is reduced for end-‐use applications (HMIs) and the middleware.

INCLUSIVE will make use of the Kepware server in order to:

14

- have a single source of data - ensure connectivity to all components:

o machines via S7 TCP/IP Ethernet protocol, MQTT, REST, Oracle, etc. o HMIs via OPC UA

- be expandable to client application like MES, SCADA, ERP Systems via OPC

OPC UA (Unified Architecture) This is a protocol for communication between machines (Machine-‐to-‐Machine, M2M) and because of its independence, scalability and in-‐build security (access control, authentication, encryption), it fulfills the needs of the overall INCLUSIVE system. Additionally, both Progea Platform.Next and the ThingWorx based middleware Controller are OPC UA based platforms.

ThingWorx test environment For test purposes, a demonstration example for an adaptive HMI (GUIs in Figure 11) customized for the three INCLUSIVE use cases, using the ThingWorx Platform will be developed. This will be used to:

- test the middleware against the HMIs - obtain information during development phase of the middleware

Components of the ThingWorx middleware controller

EVENTCHANNEL

• ThingWorx Core: Extensions (APIs, Protocols, Functionality)

Use CaseSCM Use Case

SILVERLINEUse CaseKHS

• ThingWorx AlwaysOn Protocols, REST Protocols and APIs

• JVM and Tomcat Server

Common Middleware Services

Distribution Middleware

Host Infrastructure Middleware

Hardware Devices

• KEPServerEx / HW specific communication

Operating SyStems, Protocols& Drivers

Applications

Domain-specific Middleware services

Process automation

Display Management, Sensor Management, Data link

management, ...• KEPServer Ex (OPC UA)

• HMI/SCADA• MES, ERP

Figure 12: Structure of the adaptive automation layers

15

The domain-‐specific middleware services in Figure 12 have the most potential to increase system quality and decrease the cycle-‐time and effort required to develop particular types of networked applications. In the INCLUSIVE automation adaptive middleware this layer represents a part of the functionality of the KepserverEx Platform.

The common middleware services augment distribution middleware by defining higher-‐level domain-‐independent services that allow application developers to concentrate on programming business logic, without the need to write the “plumbing” code required to develop distributed applications by using lower-‐level middleware directly. For example, application developers no longer need to write code that handles transactional behavior, security, database connection pooling or threading, because common middleware service providers bundle these tasks into reusable components. Whereas distribution middleware focuses largely on managing end-‐system resources in support of an object-‐oriented distributed programming model, common middleware services focus on allocating, scheduling, and coordinating various resources throughout a distributed system using a component programming and scripting model. Developers can reuse these component services to manage global resources and perform common distribution tasks that would otherwise be implemented in an ad-‐hoc manner within each application. The form and content of these services will continue to evolve as the requirements on the applications being constructed expand.

The distribution middleware defines higher-‐level distributed programming models whose reusable APIs and components automate and extend the native OS network programming capabilities encapsulated by host infrastructure middleware.

Distribution middleware enables clients to program distributed applications much like stand-‐alone applications, i.e., by invoking operations on target objects without hard-‐coding dependencies on their location, programming language, OS platform, communication protocols and interconnects, and hardware.

The host infrastructure middleware encapsulates and enhances native OS communication and concurrency mechanisms to create reusable network programming components, such as acceptor-‐connectors, monitor objects, active objects, and component configurators. These components abstract away the peculiarities of individual operating systems, and help eliminate many tedious, error-‐prone, and non-‐portable aspects of developing and maintaining networked applications.

As a result, a common modular middleware will be achieved, that will allow for the addition of new data structures and for the development of proprietary interfaces for the realization of the different use cases.

16

5 The INCLUSIVE specification and its implementation Starting from the user profile data obtained from the measurement and adaption modules and from the technical information from the machines considered in INCLUSIVE, a solution that addresses the INCLUSIVE derived requirements (Figure 3 and deliverable D1.1) was developed:

- An adaptive HMI will be developed to exploit the assessment of user’s capabilities derived from WP1 (Measurements) and in WP2 (Adaptation). The derived user profiles will be implemented by the Progea-‐User management module.

- Standardization of input data occurs through APIs of the adaptive automation middleware. - Communication between HMIs and middleware occurs through the OPC UA (Unified Architecture)

Protocol and is mediated by a Real-‐Time I/O Server Module from the Progea Movicon.Next platform. This Module will provide all the mapping of machines, sensors and robotic-‐information sent by the adaptive middleware.

- Control-‐activity of sensors and actuators is mediated through interaction with the middleware, by standardizing information from the SCADA-‐based HMIs.

- Communication between the different components of the INCLUSIVE system is based on OPC UA Protocol. Both client-‐ and server-‐connectors (for Progea Movicon.Next, Kepware Server, Thingworx, etc) will mediate the communication between real-‐time information and additional software (i.e. middleware controller).

The HMI should adapt to a connected user, providing the right level of complexity and allowing a gradual and dynamic increase of functionality, so that the user can get comfortable and familiar by using the HMI. Figure 13 shows schematically this concept of relationship between HMI-‐complexity and operators’ profiles.

Figure 13: HMI’s complexity depending on operators’ skills

Specification for requirements 1 and 7

17

The HMIs that deal with the two requirements addressed in this section, will differ mainly in terms of:

- the amount of information presented, - the amount of functionalities enabled to users, - a guided interaction with the productive system.

Displayed information is mainly adapted according to two factors:

- user experience in the task to accomplish; - experience in the use of the HMI (e.g. novel, occasional or habitual user).

INCLUSIVE HMIs will make use of the Progea user manager tool to provide a user repository that allows the configuration and parameterization of users and groups as well as the customization of Graphical User Interfaces (GUIs) depending on identified user profiles (Figure 14). Personal and sensitive user data are provided to the HMIs from the INCLUSIVE adaptive module and therefore are not directly exposed to the operators or HMIs developers.

s

Figure 14: User Management Module (left) with the properties panel (right)

- User Group & name: users can be grouped or defined as Microsoft active directory users in order to automatically inherit Microsoft users name and password. Each group is characterized by specific HMI attributes that control HMIs-‐access level and HMI-‐access area mask.

- User password: users are univocally identified by the HMI application, by means of identification/authentication procedures based on user-‐specific passwords.

- Access Level (user hierarchical level): this can be assigned to each user based on a hierarchical level from 0 to 1000 so that operation like execution of critical or advanced commands, access to

18

specific project area/screen or the setting of specific parameters, will be showed/hidden based to HMI-‐identified user skills.

- Access Area Mask (user profile). This tool allows to: o define up to 31 profiles, o map groups or users to a specific profile, o for each component within the GUI, there is a possibility to offer the corresponding profile

(in terms of visibility-‐, activation-‐/deactivation-‐, interactions-‐ of each single component in the GUI, etc).

Figure 15: Schematic representation of User Access Control to adaptive HMIs

Based on this information three main modalities for customizing the HMI could be foreseen:

-‐ Static solution: it is represented by a library containing similar HMI template-‐projects, designed and developed following the criteria of user profile dependency. Each identified user profile will match the profile-‐corresponding template.

-‐ Interactive solution: with some help support (online help, media controls, contextual menu, tooltip, shortcut facilities, etc) extracted from existing projects or provided by linked-‐training modules or private social networks.

-‐ Dynamic solution: INCLUSIVE-‐developed HMIs will be able to differentiate up to 31 user profiles and match the appropriate User Interface (GUI) at application startup. Additionally, a user profile can be upgraded at any time.

By implementing such tools and strategies even complex operations can be made simple and intuitive and carried out by people of any educational level, possibly upon a short and comfortable training stage.

19

Use Case 1 (SCM)

Configuration of automatic Tool warehouse and Worktable of woodworking machines

Up to three template-‐projects of different HMI-‐complexity will be created within INCLUSIVE in order to support operators in tool warehouse configuration procedures.

Once a template-‐project is selected and the user profile has been identified from the HMI, several example-‐scenarios can be postulated:

a) HMI for expert users. HMI containing details on the tool warehouse and several GUI-‐components (i.e. command buttons) in one page (Figure 16). All components can be selected and edited.

Figure 16. Configuration tool for expert operators. All Buttons are visible and enabled for the operator

b) HMI for middle-‐expert users: HMI with tool warehouse plan and command buttons in one single

page, but with a minimum guidance. Once an operator has configured the number of tools to be loaded, the HMI will expose in active state (enabled) only the selected configuration parameter (Figure 17).

Figure 17: Configuration tool for middle-‐expert operators. Only some buttons are visible and enabled for the operator

For example if a user plans to load 3 tools, it starts by getting the first position highlighted and only after clicking a validating button, is allowed to continue with the configuration of a second tool.

20

c) HMI for non-‐expert users: HMI with a step-‐by-‐step procedure on more screens, so that only a single operation per screen can be executed. The HMI is very simple and intuitive presenting flowcharts showing configuration-‐steps and guidance information (Figure 18).

Figure 18: Example of configuration tool for non-‐expert operators, containing guidance information.

Following this approach, the measurement module of the HMI (see Figure 9) will:

o Identify and classify the user according to his profile and communicate the information with the adaptive module;

o The adaptive module will interact with the user manager-‐module of the HMI in order to permit a correct matching between related user template-‐project and user-‐profile.

Use Case 2 (Gizelis / Silverline) The main goal of INCLUSIVE within the Gizelis Use Case consists of supporting operators in sensible operations like configuring bending-‐profile-‐tolerance parameters of tools mounted on the bending press.

By implementing the functionality provided by HMI templates (included in the Progea platform) a set of three template screens (Graphical User Interfaces) with predefined components, will be designed and developed in relation to identified user profiles:

1) high resolution Graphical User Interfaces (GUI) for expert users, providing many information and little help;

2) same GUI as in 1 but enhanced with a step by step procedural help; 3) simple GUIs, showing only single operability and a step-‐by-‐step flowchart that comfortably guide

the operators through configuration procedures.

More specifically, the INCLUSIVE HMI will provide additional functionality in terms of:

21

1) 2D-‐profile simulations in order to support users in easily defining the number of segments that compose metallic profiles and all related coordinates.

2) Codification of a library of Tool-‐press profiles. 3) Integration of the HMI with alarm components being activated according to matching-‐/mismatching

criteria of “head profiles” and 2D bending profiles: a. In case of good matching the profile coordinate will be passed to the robot (positive or

negative X positions) using the Yaskawa protocol DX100; b. In case of bad matching an alarm will be generated, sending process will be interrupted and

suggestions/hints (, i.e. through interactive documentation, speech support, etc) will be presented on the HMI to the operator on how to fix the problem.

Use Case 3 (KHS) The changeover operation within the Innoket Neo bottle-‐labeling machine is a complex and a time-‐consuming procedure that requires physical effort and a certain grade of machine know-‐how proceeding.

KHS Innoket Neo is provided with a proprietary, sophisticated and complex HMI. The goal of INCLUSIVE is to integrate the existing functionalities in the KHS-‐HMI with applications that support the operator in operation, such a graphical alert signal on the HMI screen or notification procedures (i.e. sms via smart watches), if a change of the label running containers or a maintenance operation is required.

Specification for requirements 2 and 4

INCLUSIVE Graphical User Interfaces must be intuitive and self-‐explaining in order to efficiently support people with low education/poor computer skills in using the devices addressed by the three use cases.

This objective will be achieved by integrating intuitive-‐designed GUIs (enriched by contextual menu, tooltips, shortcut facilities, etc), and will be further supported by teaching techniques both in terms of help-‐extensions, augmented reality or supplemented by speech explanation.

Additional support will be provided by implementing techniques such as online help or media controls.

Examples of media controls are:

o media control for mp4 video, o pdf viewer, o excel-‐sheet viewer with cells connected to tags, o web browser viewer.

This type of interactive help can be available anywhere combining media controls to HMI-‐components such as contextual popup-‐menus, function keys/shortcut and tooltips.

22

Use Case 1 (SCM)

Configuration of automatic Tool warehouse and Worktable INCLUSIVE will support users at this level by means of wearable devices such as smart glasses. Once the setting-‐procedures for the configuration of the tool warehouse on the HMI have been terminated (see section 5.1.1.1), the operator will be able to:

o take the provided handheld panel, o wear the smart glasses, o perform machine configuration according to the information viewed on the glasses.

The wearable smart glasses (with a webclient application provided by Progea) will:

o show to the operator previously configured HMI-‐GUIs containing tool warehouse configuration values,

o minimize the risk of mistakes since an operator has to report the same configuration he is viewing in the glasses;

Furthermore, the operator will have free hands for the handheld panel.

Additionally INCLUSIVE-‐HMIs will provide interactive help in forms of PDF guide, videos and trainings sessions that can be supplied by HMI components such as shortcuts, buttons and menu-‐items present on the GUI.

Routine and procedure for maintenance Since maintenance operations are usually performed within certain time intervals, sometimes for many months long, operators must read the machine user guide each time to refresh the procedure.

At this level, INCLUSIVE will provide support in terms of interactive help and by integrating the existing HMI with additional functionalities.

More specifically, online maintenance documentation will be provided, through implementation on the HMI of the Progea user-‐management tool that will show electronic documents.

From information contained in SCM-‐counters of the woodworking machine, additional diagnosis for the real-‐time state of the machine -‐ by using a dedicated driver -‐ can be obtained for easy and guided maintenance operation.

Use Case 2 (Gizelis / Silverline) INCLUSIVE will integrate part of the description and configuration values of the robot-‐simulator to the configuration tool of Progea-‐Movicon as well as to the help online module.

The help online module will provide a set of template popup screens containing media controls (web browsers, pdf viewers, mp4 streaming viewers, etc) to be linked, thus allowing any kind of assistance to the user:

o directly to GUIs, that will be developed ex-‐novo within the INCLUSIVE project; o indirectly to events, alarms, popup menus and function keys

23

Popup menus, function keys and other HMI features can also be predefined. The material displayed by this module can be in form of documents/videos (present in the archive in the middleware) or in form of interactive material provided by the INCLUSIVE teaching module.

Use Case 3 (KHS) In changeover operations within the Innoket Neo, INCLUSIVE will support operators through the implementation of vocal commands, augmented reality (in form of smart glasses) or signal warnings technologies.

Additionally for machine maintenance operation, interactive help (use of handheld panel) and alerting management (messaging over smart watch), will be implemented.

Specification for requirement 3

Currently, the HMIs by SCM and Silverline are based mostly on industrial computers or on touch screens computers (KHS), and they cannot be effectively utilized by people with disabilities. For all use cases an example of a solution could be represented by voice recognition/synthesis if the working environment is not very noisy.

In case of colorblind operators, INCLUSIVE will develop apposite standardized and color-‐independent profiles derived from existing libraries following neutral color palettes criteria. These profiles will be stored in the middleware, thus being available to the HMIs.

In case of deaf operators the use of augmented reality tools as smart glasses will be implemented to support them by presenting visual information.

Specification for requirement 5 (Use Cases 1, 2 and 3)

Some operational tools present in the Movicon.Next Platform (Scheduler Manager, Recipe Manager, etc) will be implemented in order to monitor and debug running processes in case of wrong/defect operational activity.

24

Figure 19: Progea Movicon.Next operational tools

Therefore, HMIs will be developed taking into account control-‐parameterization ranges. Each key features within a device will range between threshold values. If these ranges are violated warnings, errors and alarm signals will be enhanced by the HMIs.

INCLUSIVE will support operators within this task by providing some interactive help and guidance.

Use Case 1 (SCM)

Routine and procedure for maintenance Actually, maintenance operations by SCM are managed with the alarm system provided by the HMI, constantly monitoring the threshold machine parameters; if these override the defined, limits alarm warnings will be activated.

1) Monitoring alarm management tools in the HMI with criteria of filtering, severity and alarm coloring depending on severity;

2) Link specific alarms to media: PDF, videos, training sessions so that in case of maintenance, immediate contextual help can be easily provided to the user.

Use Case 2 (Gizelis / Silverline) Within INCLUSIVE the Progea-‐Movicon.Next Alarm Management module (Figure 10) will be implemented within the available control tools.

This module efficiently informs users about problems, events, maintenance-‐needs etc. It is configurable with predefined Alarm Templates and it is linked to an existing online-‐help module in order to assure quick and efficient support.

Alarms can be viewed from any GUI through banners or sliding side screens and operators can be notified from anywhere with SMS, Voice over IP, Emails, etc.

Use Case 3 (KHS) In changeover procedure of the Innoket Neo labelling machine, if the operator loads on the machine elements that do not fit the format selected on the HMI (e.g., because she/he has selected the wrong format on the HMI), the HMI does not notify the mismatch. The only result visible to the operator is that the machine does not start.

INCLUSIVE contribution at this level will be in terms of:

1) Implementing new and efficient help-‐tools to support the user in learning easily and efficiently the complex changeover procedures;

2) Assist the user during the changeover with smart glasses (with the “web-‐client app” of PROGEA) or handheld panels (or smart watches) that reports information on the changeover procedure step by step.

25

Three application templates will be defined to address three different user profiles. The INCLUSIVE adaptive module will dynamically reconfigure the HMI user profile after recognizing extraordinary emotional and physical state of the operator.

Specification for requirement 6

Intuitive and easy to use HMIs improve interaction processes between human operators and the machines.

This requirement in INCLUSIVE will be addressed by combining and integrating well established operational technologies within the HMIs with strategies of different nature (teaching and built-‐in help pages).

In any industrial process, operator’s satisfaction is a key to improve productive efficiency as well as for establishing comfortable atmosphere between workers. INCLUSIVE will integrate HMIs with apposite developed modules to measure the OEE (Overall Equipment Effectiveness) index.

Figure 20 shows some statistical data about production efficiency in a company. These indices can be considered as an important indicator, providing valuable information on the progress of operator efficiency.

Figure 20: Statistical data calculated by using the Progea Pro-‐Lean module

26

6 Conclusion This deliverable defines the system specification for the INCLUSIVE project. These specifications at system and at module level, are defined for the requirements described in the INCLUSIVE D1.1-‐Requirements Deliverable.

Table 1 reports a summary of the strategy and tools that will be implemented in INCLUSIVE to satisfy the requirements for each use case.

Requirements Industrial partner Use cases Specifications / Implementation

R1: The interface adapts to the level of skills of the operator

R7: Interaction with the system generates a low level of stress for the operators

SCM Configuration Warehouse

Three HMIs with different level of complexity; interactive help

Machine maintenance Smart Alarm management with interactive help and guidance

Silverline Bending processes Three HMIs with different level of complexity; 2D-‐profile simulation

Replacement tools Tools library; Alarm management

KHS Format changeover

Fault recovery procedure

Alerting management (notification via sms on smart watch)

R2: The system can be used by low educated operators

R4: The system can be used by people with low computer skills


Wearable devices (smart glasses); interactive help (document, video, training session)

Machine maintenance Smart Alarm management with interactive help and guidance

Silverline Bending processes Template-‐GUIs, media controls, web browsers and pdf viewer

Replacement tools Interactive help (online maintenance documentation)

KHS Format changeover Speech, Augmented reality, signal warning technologies


Interactive help and alert management

27

R3: The system can be used by physically and cognitively impaired operators


Color-‐independent profiles for daltonic user; speech and smart glasses

Machine maintenance Color-‐independent profiles for daltonic user; speech and smart glasses

Silverline Bending processes Color-‐independent profiles for daltonic user; speech and smart glasses

Replacement tools Smart glasses for deaf user; Color-‐independent profiles

KHS Format changeover Color-‐independent profiles for daltonic user; speech technologies


Color-‐independent profiles; speech technologies and smart glasses

R5: The system enforces the correct procedures


Interactive help and guidance

Machine maintenance Monitoring and alarm management; interactive help

Silverline Bending processes Interactive help and guidance

Replacement tools Monitoring and alarm management; GUIs with banners and/or sliding side screens; interactive help

KHS Format changeover Interactive help; smart glasses, handheld panels; different apps for three user profiles


Interactive help

R6: The operator feels satisfied from the interaction experience


Survey forms, monitoring of improvements and production downtime time statistics, OEE calculation.

Machine maintenance Survey forms, monitoring of improvements and production downtime time statistics, OEE calculation.

28

Silverline Bending processes Survey forms, monitoring of improvements and production downtime time statistics, OEE calculation.

Replacement tools Survey forms, monitoring of improvements and production downtime time statistics, OEE calculation.

KHS Format changeover Survey forms, monitoring of improvements and production downtime time statistics, OEE calculation.


Survey forms, monitoring of improvements and production downtime time statistics, OEE calculation.

Table 1: Summary of INCLUSIVE requirements and deduced specifications

As a result, the development of adaptive HMIs or the integration of additional functionalities to existing proprietary-‐HMIs will support users, referring in particular to elderly, disabled and inexperienced operators, during daily working operations.

Implementation of additional tools that use augmented reality (i.e. smart glasses), speech technologies (speech recognition, text-‐to-‐speech) and teaching applications (online and offline) will further assist operators in configuration-‐, monitoring-‐, maintenance-‐ and alarming-‐processes.

d.1.3 system architecture v1 - inclusive · smartandadaptiveinterfacesfor&&...

Documents