the gridcc instrument element: from prototypes to production environments roberto pugliese on behalf...

The GRIDCC Instrument Element: from prototypes to production environments

Roberto Pugliese On Behalf of the GRIDCC Collaboration

EU FP6 Project

www.gridcc.org (EU FP6 Contract 511382)

1. The GRIDCC - Grid Enabled Remote Instrumentation with Distributed Control and Computation – project has the main aim to bring Instruments to the GRID

2. It is a 3-years EU FP6 project started in September 2004

3. Web site: www.gridcc.org

4. More in particular the project goals are:1. Definition of a “Instrument Element” allowing a standard remote

access to any type of instrumentation

2. Tight integration between instrument grid and classical computational grid

3. Human interaction with Grids via Virtual Control Room (collaborative environment)

4. Enactment of complex workflows

The GRIDCC project: Goals & Objectives


The GRIDCC partners

Participant name Country

Istituto Nazionale di Fisica Nucleare Italy

Institute Of Accelerating Systems and Applications Greece

Brunel University UK

Consorzio Interuniversitario per Telecomunicazioni Italy

Sincrotrone Trieste S.C.P.A Italy

IBM (Haifa Research Lab) Israel

Imperial College of Science, Technology & Medicine UK

Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale delle Ricerche

Italy

Universita degli Studi di Udine Italy

Greek Research and Technology Network S.A. Greece


Execution ServicesCollaborative Environment

GRIDCC: Architecture (I)Instruments Grid Computational Grid

IE

IE

CE

CE

SE

Instrument Element

Problem Solver

InstrumentManager Information &

Monitoring System

DATA

ES

VCR

VCR

VCR

VCR

Workflow

Service Oriented Architecture – SOA

The GRIDCC services are Web Services compliant, according to the WS-I convention.

Any type of client WS-I compliant, can access the GRIDCC services without any specific software library.

X509 based security is used When performances are

required , a kerberos based mechanism is used to access IEs.


GRIDCC Architecture

Web ServiceInterface

Execution

Service

WfMS

WMS

AS

Instrument elements

(IE)StorageElement

(SE)

Instrument elements

(IE)StorageElement

(SE)

Instrument Element

(IE)StorageElement

(SE)

Compute element

(CE)

Compute element

(CE)

Computing Element

(CE)

StorageElement

(SE)

StorageElement

(SE)

StorageElement

(SE)

Global ProblemSolver

Information and Monitoring

Services(IMS)

Information System(BDII)

Security Services

Virtual Control

Room

Virtual Control

Room

CollaborativeServices

(CS)

WMS Work Management System

WfMS Work Flow Mng System

AS Agreement Service

Broker


GridCC PermanentTest Bed

VOMSVOMS

AS, KrbServerAS, KrbServer

VCR, SE, IE, BDIIVCR, SE, IE, BDII

IE, CE/CREAM,SE, LFC IE, CE/CREAM,SE, LFC

WMProxyWMProxy

WfMS WfMS

VCR, IE VCR, IE


GRIDCC main target areas(Large-scale) scientific

experimentsHigh energy particle physics

(Radio-) Telescopes

Remote process control Accelerator control (Tele-) Biomedicine

Robotics Automotive

Electronic microscopes

Widely Sparse Instrumentation Power Grids

Monitoring of the territory Monitoring of the seaGeo-hazard prediction

Distributed laboratories Transportation monitoring

Sensor network

GRIDCCMiddleware


GRIDCC pilot applications

Power Grid

Power Grid

High Energy Physics

CMS

High Energy Physics

CMS

Particle Accelerator

Particle Accelerator

Device Farm

Device Farm

Geohazard MonitoringGeohazard Monitoring

MeteorologyMeteorology


Accessing GRID Enabled Instrumentation: Requirements

1. Interactive access to allow the control and the monitor of the instrumentation. Uniform access to the physical devices.

2. Fast Data Publishing to allow the publishing acquired by the instrumentation

3. Information (logs, errors, etc.) Publishing to track the behaviour of the instrumentation and possibly to correct the malfunctioning

4. GRID integration. The acquired data should be visible by the GRID protocols both to :

1. Move them to large GRID data storage (Storage Element)

2. Process them from the GRID computing power (Computing Element)


A new 4 channels element to fit the requirements: The Instrument Element (IE)

Inst

rum

enta

tion

Con

torl

GR

ID A

cces

s

Instruments Access

Data/Info Pubblishing

InstrumentElement

Web

Ser

vice

SR

M/G

RID

FT

P

Custom (JMS, ..... )

Custom

I’m here

Discovery

QoSAdvance Reservation


The IE permits SOA accessible operations to control and monitor the instruments (via VIGS), such as:

• execute a command• get / set parameters

different data outputs:• data mover to/from a grid Storage Element (via SRM) • high bandwidth channel for data publishing (via IMS).• low bandwidth channel for logs, states etc. (via IMS).

Instrument Element: Versatile I/O multichannel

VIG

SCommands

Status

Data Subscribers

Grid Interaction

SR

M

VCR

StorageElement

(SE)

StorageElement

(SE)

StorageElement

(SE)

Logs, Errors,

States, Monitors

Instruments

IMS

ParametersES

VIGS Virtual Instrument Grid ServiceIMS Information and Monitor ServiceSRM Storage Resource Management

IE


VIG

S

IMSProxy

ControlManager

DataCollector

Real Instruments

Data Flow

Control Flow

State FlowError FlowMonitor Flow

The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments.

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

IMSInstrument Element Architecture

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

Instrument Protocols


Configuration Examples

Top IM

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

Top IM

Top IM

IMIM IM IM IM IM

Resource Service Instruments Pool

A B C

Configuration A, B, C can run concurrentlysharing services like RS, PS, IMS


Instrument Element Framework Characteristics

The IE framework is modular, fully customizable and can be adapted to different environments and different type of instruments to be controlled.

Customizable plug-in modules are used to interface the framework with the communication protocol (both hardware and software) used by the instruments.

the dynamic behaviour of the control that has to be performed is programmable via the built in Finite State Machine (FSM).

relatively fast control feedback loops can be provided using the customizable built in Event Processor (EP).

Error repairing actions due to malfunctioning of the instruments can even be handled by EP or by the Local Problem Solver (LPS).

IE

ResourceService

IMS

LocalProblem

Solver

InstrumentManagerA

cces

s C

ontr

ol M

anag

er

DataMover

InstrumentManager

VIG

S

InstrumentInstrumentInstrumentInstrument

IMSProxy

ControlManager

DataCollector

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager


VIG

S

IMS

Instrument Manager

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

InstrumentManager

IMSProxy

ControlManager

DataCollector


16

Instrument Manager

IM is composed by 3 main components:- Control Manager:

- Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs,errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability

- FSM. A finite state machine is implemented - Resource Proxy. It handles all the outgoing connections with the resources.

- Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen.- IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS

IMSProxy

DataCollector

Instrument Manager

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

Instruments

Data Flow

State Flow

Error Flow

Monitor Flow

Control Flow

Customizable Plug-in modules to interface to the instruments


Status• data collection • data publishing on RMM-JMS • data on file in progress• data on DB to be started• sync commands (enable/disable done trough the “set param” of the IE façade (VIGS)

Data Collector

ResourceService

IMS

LocalProblem

Solver

InstrumentManager

Instrument Element

IMSProxy

ControlManager

DataCollector

Acc

ess

Con

trol

Man

ager

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

DataMover

VIG

SIE File

System

RMM-JMSDB persistency

enable/disabledata stream

InstrumentInstrumentInstrument

Instrument

data stream

Data Subscribers

(Monitor, d

ata analysis, etc.)


VIG

S

IMSProxy

ControlManager

DataCollector

IMS

Resource Service

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

ResourceService


Resource Service Architecture

The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). A resource can be any hardware or software component involved in the IE (instruments, Instrument

Managers, IMS components) RS stores the configuration data of the resources and download them to resource target when

necessary Resources can be discovered, allocated and queried. It is the responsibility of the RS to check resource availability and contention with other active

partitions when a resource is allocated for use. A periodic scan of the registered resources keeps the configuration database up to date.

DiscoveryManager

SubscribeManager

Partition&LockManager

ConfigurationManager

Available Resources

PartitionDefinitions

ConfigurationDefinitions

RS

Dat

a B

ases

Partition/Configurationretrieve methods

Partition and Locksetting methods

Configurationsetting methods

Discoverymethods


VIG

S

ControlManager

DataCollector

IMS

Information and Monitor Service

ResourceService

IMS

ProblemSolver

InstrumentManager

Instrument Element

Data Publisher

Acc

ess

Con

trol

Man

ager

SRM/SEGridFTP

SE

IMS

IMS

IMSProxy


21

Information and Monitor System (IMS)

The Information and Monitor Service (IMS) collects messages and monitor data coming from the IMS Proxy of the IMs.

The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms.

The IMS collects and organizes the incoming information either in a database or a pub/sub system or both. It publishes the data to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level.

IMSProxy

IMSProxy

IMSProxy

PE

RS

IST

EN

CY

Pub

lish

/ S

ubsc

rib

SU

BS

CR

IBE

RS

SU

BS

CR

IBE

RS


IMS Fast Data Publishing: RMM-JMS

IE

IE D

ata

Pro

duc

er

Dat

a S

ubsc

ribe

rs Same data are sent to several subscribers.Multicast protocols can have a benefic impact on performances

• JMS provides a standard set of APIs for the communication• Many commercial and academic JMS implementations both in C/C++ and Java (NaradaBrokering, Sun, IBM)• GRIDCC (IBM Haifa lab) has implemented a Reliable Multicast protocol (RMM) JMS compliant• RMM-JMS works within a LAN but an efficient bridge technology has been developed to allow inter-LAN multicast communication


Status: installed and running• SRM 2.2 • STORM implemetation (INFN)• Bestman implementation (Berkeley))• XFS IE file system

Embedded SRM compliant Grid Storage Element (SE)

ResourceService

IMS

LocalProblem

Solver

InstrumentManager

Instrument Element

Acc

ess

Con

trol

Man

ager

SEBackend

VIG

SIE File

System

SR M GRID

accessibile data


Querring Client

Querring Client

Discovery Instruments (I) The discovery of the instruments or of the IEs is an issue

when the number of elements is high We can have two cases:

Quasi static cases The number of IEs are well defined and the single IE is quite complex with

a good hardware support In this case a register based discovery mechanism can be used. GRIDCC tesbed is using the LCG BDII (Berkely Database Information

Index ) based on LDAP The information collected in the BDII follow a GLUE schema and can be

used for match making querries

IE

IE

IE

BDIILocal BDII

Querring Client


Discovering Instruments (II)

Dynamic cases The number of IEs can change very quickly, they are very simple devices,

often with poor hardware support The discovery is just use to know which are the online IEs A new approach has been developed based on Peer to Peer (P2P)

protocols See demo on this workshop

IE

IE

IEIE

M1

M2

Querring Client

Querring Client

Querring Client


Instrument Elements at Work


The GridCC middleware has been deployed to control the run of the CMS (Compact Muon Solenoid), one of the four high energy experiments in LHC (Large Hadron Collider) at CERN laboratory.

CMS Magnet Test and Cosmic Challenge (MTCC) I and II, a milestone in the CMS construction, positively carried out.

Pilot applications: CMS - I

Top IE

CSCIE

TrackerIE

HCALIE

DAQIE

RPC IE

TriggerIE

DTIE

ECAL IE

DAQIM

FB RB FF

xdaq

xdaqxdaq

DAQRS

DAQIMS

UserInterface

CMS Control Structure

CMS Detector


CMS MTCC phase I and phase II carried out in 2006

Scale MTCC versus CMSFEDs: 20 out of 600 3%EVB RUs: 6 out of 600 1%Filter Farm: 14 out of 2000 0.3%Trigger rate: 100 Hz out of 100 kHz 0.1%Event size: 200 kB out of 1 MB 20%IEs 15 out of 150 10 %

Scale MTCC versus CMSFEDs: 20 out of 600 3%EVB RUs: 6 out of 600 1%Filter Farm: 14 out of 2000 0.3%Trigger rate: 100 Hz out of 100 kHz 0.1%Event size: 200 kB out of 1 MB 20%IEs 15 out of 150 10 %


Pilot applications: Power grid

Instrument Manager

Instrument Element

...

Virtual Control Room

Virtual Control Room

Gas

Solar Panel

Power Grid V.O.

GRIDCC deployed to monitor:• A 50kW generator• A 1 kw Photo-Voltaic array

Power Grid. GRIDCC deployed to monitor a 50kW generator and a 1 kW Photo-Voltaic array

See D

EMO o

n this

work

shop


Pilot applications: Remote Operation of an Accelerator

Elettra Synchrotron

See D

EMO o

n this

work

shop


Meteorology (Ensemble Limited Area Forecasting) Weather forecasting system to detect hazardous weather

Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories The Device Farm consists of a pool of Measurement

Instruments for Telecommunication Experiments

Geo-hazards: Remote Operation of Geophysical Monitoring Network The monitoring net will be characterized by different levels of

activity: stand-by, pre-alert, alert, plus a control modality An event worth to be monitored is for example the evolution

of a “landslip”

GRIDCC: other applications


Instrument Element: Scalable on embedded systems

FPGAPPC405

Xilink Virtex IV

Custom Board

Cus

tom

Ele

ctro

nics

1 Gbps Ethernet

Web

Ser

vice

Grid

CC

IM

Java

VM

Linu

x

USER INTERFACE

Custom Logic

Montavista www.montavista.com

JamVM http://jamvm.sourceforge.net

Standalone Axis

IEInstrument Manager


IE technologies Web Service compliant (WS-I) Tomcat + Axis (and Java) and Axis standalone are the main

technologies of the IE All the services are deployed on a single or multiple instances

of Tomcat, according to the needs of the application Message oriented middleware (Pub/Sub) is based on the

Java Messaging System (JMS). The following implementations are used in the project Sun Narada Brokering RMM - JMS (GridCC IBM)

MySQL and Oracle are used as Data Base for the RS SRM interface version 2.2 used


Conclusion - I The GridCC project is integrating instruments into the

“classic” computational/storage Grids.

One of the novel concepts introduced by GridCC are: The Instrument Element, allowing a virtualisation of the

instruments to be controlled and their insertion in a Grid hard (reservation of IE) and soft (statistical prediction)

guarantees of the IE’s methods execution times Fast Data Publication via a Message Oriented middleware

(RMM-JMS) to distribute data and information from an IE to the world wide Grid.

The IE is highly customizable and can be adapted in different environments.


Conclusion - II 3 pilot applications deployed, one of them in a real

production environment The IE can be shrunk down into a chip allowing grid

enabled embedded control of the instrumentation Several heterogeneous pilot applications are

deploying and running the IE middleware We support and encourage the adoption of our

middleware in other projects/experiments.

the gridcc instrument element: from prototypes to production environments roberto pugliese on behalf...

Documents

eu fp6 contract

gridcc grid

gridcc services

gridcc collaborationeu

years eu fp6 project

gridcc instrument element

gridcc permanenttest

remote instrumentation