new trends of computing platforms: convergence vs...

New Trends of

Computing Platforms:

Convergence

vs.

Diversification

Robert Lovas

Deputy head

Laboratory of Parallel and Distributed Systems

Email: [email protected]

Industry 4.0 in Europe:Policy Initiatives on Digitizing Industry

Source: https://ec.europa.eu/futurium/en/system/files/ged/backwall-booth.pdf

https://www.i40platform.hu/en

• Research division of MTA SZTAKI from 1998

• Head: Peter Kacsuk, Prof.

• editor in chief: Journal of Grid Computing (Springer)

• coordinator of four e-Infrastructure projects in the EU 7th Framework Programme (FP7)

•Deputy head: Robert Lovas, PhD

• coordinator of two projects in EU FP7

• international liaison (EGI)

• secretary of International Desktop Grid Federation

•15 research fellows (full/part time)

• Foundation member

– Hungarian National Grid Initiative (NGI_HU) - 2003

– International Desktop Grid Federation (IDGF) - 2010

– Hungarian Academic Cloud (MTA Cloud) - 2016

Laboratory of Parallel and Distributed Systems

• Participant and/or coordinator in European and national Grid & Cloud research, e-infrastructure, and educationalprojects (from 2000)

• By far the most successful research lab in the country in the field ICT / e-Infrastructures in terms of the number of coordinated FP7 EU projects

Our international impact (past)

WS-PGRADE/gUSE gateway and workflow technology:

– Most used science gateway framework in Europe(see Map 1)

– More than 30 large international projects installed in Europe

– Its integration with clouds using CloudBroker in SCI-BUS was declared as the industrial research with the largest potential after comparing 500 EU projects by the JRC (EU)

The desktop grid & crowd computing technology developed in the EDGeS, EDGI, DEGISCO, IDGF-SP projects has also world wide impact (see Map 2)

4 PERL: 2018. 10. 20.

• Several solutions are already available frompublic cloud providers for Internet of Things (IoT) and Big Data applicationareas.

• Private clouds have significant benefits in terms of security and integrability into theenterprise environment but hybrid and multi-clouds are also widespread.

Growing demand for cloud ochestratorsand brokering tools.

5

Actual trends

(2)

from Google

Source: www.computerzone.org

• Application areas:

• Store and process sensor data

• Historical analysis, simulations, predictions, etc.

• Visulisation

• Industrial users face challenges when they intend to benefit from cloud computing:

• migration of legacy and new applications into clouds

• their orchestrated deployment,

• their on-demand scaling,

• portability, when a cost efficient hybrid cloud or cloud agnostic (vendor independent) solution is needed

emerging new software container and orchestrator solutions

6

Clouds in production systems: some non-trivial problems

Source: Google trends

Occopus hybrid cloud / container orchestrator

511.96 €1 023.96 €

2 710.44 €

4 502.32 €

777.60 €

1 555.20 €

3 427.20 €

7 372.80 €

0.00 €

2 000.00 €

4 000.00 €

6 000.00 €

8 000.00 €

A3 A4 A7 A11

EU

R/M

ON

TH

VM TYPE

M O N THLY F E E : 4 N O D E

HA D O OP CL U S TE R

Occopus HDInsight

– Multi-cloud solution

– Contextualization

– No vendor lock-in

– Portable descriptor file

– Big Data support

– Enable auto-scaling

occopus.lpds.sztaki.hu

MTA CLOUD

8

• Joint project with Wigner Datacenter

• Dedicated for academic research communities

• Operation since Q4/16

• Wigner Datacenter:

• 800 vCPU

• 2.5 TB RAM

• 400 TB HDD

• MTA SZTAKI:

• 360 vCPU

• 832 GB RAM

• 164 TB HDD

• OpenStack based community cloud

• 60+ research project supported by MTA Cloud

www.cloud.mta.hu

Agrodat.hu project

Main objective: knowledge centre and decision support system• based on data gathered by an

innovative, complex sensor system and from international open repositories

• relying on big data, cloud, and HPCtechnologies

to support precision agriculture.

Duration: 2014-2017Budget: appr. 8 MEURURL: www.agrodat.hu

Consortium:

Big Data research center:844 CPU Core5274 GB Memory564 TB SSD/HD

GPGPU:21504 CUDA Core 488 Xeon Phi Core

Network:40 Gb / Infiniband for HPC10 Gb copper 1 Gb copper for mgm.8/16 Gb FC for SANConnected to HBONE

Cloud middleware

Scalable IoT back-end for IaaS clouds

Connected cars: Controller Area Network (CAN) Data Collector and its applications

Three tiers of the IoT back-end architecture for Connected Cars

Cloud and microservice platform forIndustry 4.0 simulations Docker@SZTAKI

Goal: reduce the execution time of EasySIM descreteevent simulations for production systems using cloud and containers technologies

TimeEvent1 Event2 Event3 Event4 Event5

Event1

Event2

Event3

…Eventn

Real system

Sim. model

Flowbster

– quick deployment of the workflow as a pipeline infrastructure in the cloud

– once the pipeline infrastructure is created in the cloud it is activated and data elements of the data set to be processed flow through the pipeline

– as the data set flows through the pipeline its data elements are processed as defined by the Flowbsterworkflow

Aone_file

two_file

out_file

Bone_file

two_file

out_file

Cone_file

two_file

out_file

Done_file

two_file

out_file

Data set

to be

processed

Processed

data set

e.g. EUDAT

servicee.g. EUDAT

servicee.g. EGI FedCloud service

Graphical design layer

Application description layer

Workflow system layer

Cloud deployment and

orchestration layer

Flowbster layers

Occopus layer

• Deployment and configuration of Infrastructure-as-a-service (IaaS) clouds

• Support for cost efficient application on cloud and microservice platforms

• Orchestration of complex processes in clouds and/or on software container based platforms

• Design and development of vendor independent (cloud agnostic) prototypes and pilots

and related R&D activities

2017. október 10.15

Our services and on-going international R&D projects

• EU H2020 CloudiFacturing project: www.cloudifacturing.eu

• EU H2020 COLA project: www.project-cola.eu

• EU H2020 EOSC-hub project: www.eosc-hub.eu

http://www.cloudifacturing.eu/

http://www.project-cola.eu/

http://www.eosc-hub.eu/

Consortium:Fraunhofer IGD (coordinator),DIHs, ISV, SMEs, etc.SZTAKI role: WP leaderDuration: 2017-2021Budget: 9.7 MEUR

www.cloudifacturing.eu

7 Experiments and 4 Digitial Innovation Hubs

1. Optimizing design and

production of electric drives

2. Cloud-based modelling for

improving resin infusion

process

3. Improving quality control and

maintenance at manufacturing

SMEs using big data analytics

4. Numerical modelling and

simulation of heat treating

processes

5. Optimizing solar panel

production

6. Optimizing efficiency of truck

components manufacturing

processes by data analytics

7. Simulating and improving

food packaging

CLOUDIFACTURING: MISSION STATEMENT

•The mission of CloudiFacturing is to optimize production processes and producibility

• using Cloud/HPC-based modelling and simulation,

• leveraging online factory data and advanced data analytics,

• thus contributing to the competitiveness and resource efficiency of manufacturing SMEs,

• ultimately fostering the vision of Factories 4.0 and the circular economy.

•Vision: closing the loop from factory data back to simulation and forward to influencing factory processes (with support in real-time).

LIVE DIGITAL TWIN MODEL

1. Fraunhofer (coordinator)

2. UoW

3. Clesgo (start-up)

4. cloudSME (start-up)

5. CloudBroker

6. CloudSigma (cloud provider)

7. ScaleTools

8. Lunds University

9. SINTEF

•

10. SUPSI

11.MTA SZTAKI

12. UNott

13. Innomine (DIH)

14. University of Bologna

15.IT4I (HPC provider & DIH)

16. Insomnia (DIH)

17. STAM (DIH)

18. DFKI/SmartFactory (tech. partner & DIH)

CORE PARTNERS

CloudiFacturing: 4 layers of ambition

Integrated Cloud/HPC platform

CloudFlowcloudSME

Cross-border application experiments involving

manufacturing SMEs and mid-caps

Digital

Innovation

Hubs

Digital

Marketplace

Business

modelling

and creation

Advanced

security

solutions

Data and

visual

analytics

Real-time

support

Manufacturin

g engineering

CloudiFacturing: Integrated Platformby the end of the 1st year

• CloudiFacturing is part of I4MS III (3rd phase)

• The 3rd phase has brought additional requirements:

o Cross-border application experiments

o Access to regional funds

HIGH LEVEL GOALS

WHAT DOES CLOUDIFACTURING OFFER?

+ Open call for new experiments (in mid 2019)

H2020 COLA: High level concept and facts

Application 1 Application 2 Application N

Service 1 Service 2 Service 3 Service 4 Service 5

Baseline resource consumption

Variable resource consumption

Cloud services

Dynamic demand

Manually adjusted supply

Resource requirements

To be replaced by automatically

adjusted supply

• Project started: 1st January 2017

• Project ends: 30th June 2019

• Overall project cost: 4.2 million Euros

• EC contribution: 3 million Euros

o Further contributions by Switzerland and the participating companies

• 14 project partners from 6 European countries

o 10 companies and 4 academic/research institutions

10/20/2018 www.project-cola.eu

Project objectives

• Define a generic pluggable framework, called MiCADO (Microservices-based Cloud Application-level Dynamic Orchestrator) that supports

• optimal and secure deployment and

• run-time orchestration of cloud applications.

• The project will provide a reference implementation of this framework by

• customising and

• extending existing, typically open source solutions.

• Moreover, it will demonstrate via large scale close to operational level

• SME and

• public sector demonstrators the applicability and impact of the solution.

10/20/2018 www.project-cola.eu

MiCADO: objectives

• cloud deployment orchestrator service by which the application templates can be deployed in the most simple and user-friendly way (typically by one click) in various types of cloud and multicloud systems

• monitoring microservice by which the status and health of the application services can continuously be monitored

• scalability decision service by which MiCADO can decide based on the monitoring information if application services should scaled up or down

• price/performance optimization extension to the scalability decision service by which MiCADO can optimize the execution of running applications both from the point of view of performance and execution price according to QoS parameters provided by the application developer or end-user

10/20/2018 www.project-cola.eu 27

Technology independentarchitecture of MiCADO


Node/container monitor

Node/container monitor

MICADOWORKERNODE

Info onnodes/containers

Container create/destroy/scale up/down, node evacuation, etc.

ContainerOrchestrator

Worker node create/destroy/scale up hor/verCloudOrchestrator

MonitoringSystem

Submitter

Policy Keeper

Register policies

Scale/updateworkernodes

Scale/update containers

description on infrastructureand policies

CreateWorkernodes

MICADOMASTERNODE

container

container

container

Optimiser

AdviceParameters

MICADOWORKERNODE

ContainerExecutor

Create containerinfra

ContainerExecutor

Technology choicesin MiCADO


Node exporter/cadvisor

Node exporter/cadvisor

MICADOWORKERNODE

Info onnodes/containers

Container create/destroy/scale up/down, node evacuation, etc.Swarm

Worker node create/destroy/scale up hor/verOccopus

Prometheus

Submitter(own

developed)

Policy Keeper(own

developed)

Register policies

Scale/updateworkernodes

Scale/update containers

TOSCAdescription on infrastructureand policies

CreateWorkernodes

MICADOMASTERNODE

container

container

container

Optimiser(not decided)

AdviceParameters

MICADOWORKERNODE

Docker

Create containerinfra

Docker

MiCADO software: 0.6.1 (16.10.2018)

• Application description by TOSCA: contains specification of the virtual machines, containers and scaling policy

• Auto-scaling decision is performed on virtual machine and container level based on the scalingpolicy

• REST API (TOSCA submitter) to handle application description and to implement user commands(deploy, query, update, undeploy, etc.)

• Dashboard for inspecting internal behaviour

• Deployment with Ansible playbook

• Security features

• Handles credentials for cloud API, docker registry and accessing the MiCADO service

• Supports Ansible vault mechanism to protect credentials

• Implements internal firewall, proxy, encryption, authorisation

• Authenticate against Dashboard and REST API


• Grafana to show loads/ consumptionsas well as scaling in time

• Customizablevisualisation tool


MiCADODashboard

Docker Visualizer toshow nodes and containers graphically


MiCADODashboard

Prometheus forthe applicationdevelopers toinspect the detailsof the alerts


MiCADODashboard

MiCADO source, images & docs

• MiCADO source: https://github.com/micado-scale

• MiCADO docker images: https://hub.docker.com/u/micado

•

• Documentation source:http://github.com/micado-

scale/docs

• Documentation is hosted athttp://micado-scale.readthedocs.io


https://github.com/micado-scale

https://hub.docker.com/u/micado

http://github.com/micado-scale/docs

http://micado-scale.readthedocs.io/

• European Commission Horizon 2020 Programme

• 100 Partners, 76 beneficiaries (75 funded)

• 3874 PMs, 108 FTEs, more than 200 technical and scientific staff involved

- Budget: €33,331,18, contributed by:

• 36 months: Jan 2018 – Dec 2020

European Open Science Cloud

10/10/2018

• EOSC-hub brings together multiple service

providers to create the Hub:

• a single contact point for European

researchers and innovators

• to discover, access, use and reuse a broad

spectrum of resources

• for advanced data-driven research.

• For researchers, this will mean a broader access

to services supporting their scientific discovery

and collaboration across disciplinary and

geographical boundaries.

Source: EOSC-hub

10/10/2018

The project creates EOSC Hub:

a federated integration and management system for

EOSC• Data

• Applications &

tools

• Baseline services

(storage, compute,

connectivity)…

• Training,

consultants

• Marketplace

• AAI

• Accounting

• Monitoring

• …

Usage according to

Rules of

Participation

From the

consortium AND

from external

contributors

• Lightweight

certification of

providers

• SLA negotiation

• Customer

Relationship

Management

• …

Based on

FitSM

• Security regulations,

• Compliance to standards,

• Terms of use,

• FAIR implementation

guidelines

• …

Source: EOSC-hub

Robert [email protected]

37

Thank you for your attention!

new trends of computing platforms: convergence vs...

Documents