Diirt statusdata integration in real time

Gabriele Carcassi

diirt: data integration in real-time

• Connection/data processing layer for general purpose Java clients– http://diirt.org

• Includes pvmanager/graphene/…• Main two clients:– CS-Studio (Java based Eclipse RCP)– WebPODS (Web-socket + JSON “gateway”)

Requ

est/

resp

onse

Publ

ish/

subs

crib

e

pvmanager

Ch A

cces

s

Processing

Visualization

vTypes Data Definition

Aggregation

formula

graphene

Data integration in real time (Diirt) architecture

BOY Web-pods (aka web gateway)

DB

(jdbc

)

Data serialization/conversion

csv

datasources support services support

list

CS-Studio (general purpose clients) Pods (Protocol Oriented Distribution Service)PV

Acc

ess

Web

Pod

s

File

Loca

l

…

PV A

cces

s

Cmd

line

(exe

c)

Chan

nel

Find

er

…

JSON

Probe …

Funding situation

• Funding at BNL was drastically cut– Closed what I could– Cleanly dropped the rest

• I’ll report on each area– Current state– What we wanted to do but won’t get done

Overall project

• General improvements– All projects in a single repository

• Easier to move code where it belongs• Easier to make releases• (Maven lesson learnt: change version in feature branches)

– All elements configurable through a single configuration directory structure DIIRT_HOME

– Documentation review (Wiki for user documentation, javadocs for developer documentation, documented source code examples in the repository)

VTypes

• Review types with Java 8 in mind– Use javax.time– Support for unsigned integers– Reorganize type hierarchy to overcome some

implementation and usage issues– Reorganize utility methods within the type

hierarchy (instead of utility classes)• Work was started, currently unfunded• Frozen in org.diirt.vtype.next package

VTypes

• Extract VType conversions from JCA/PVA support– Allow to use same exact type conversion for those

who do not need/want the whole stack• Currently unfunded

Datasources (pvmanager)

• Final review to the datasource spec– Generalize type conversion for writes (so that all datasources

behave the same) – Use Java 8 standard classes where possible– Generalize caches from sources of external data (i.e. better

support for archive data, better link between services, …)– Finalized documentation and examples for writing datasources– Performance improvements– Streamline error handling (i.e. do more work in the framework,

clients are simpler, error handling more homogeneous)• Currently unfunded

Archiver integration

• Client side temporal cache that can be filled by different archiver sources and real-time updates

• Under development by ITER.

Services

• Final review to the services spec– Uses Java 8 standard classes for callback– Defines order for arguments and results– Allows both synch and asynch execution– Allows both synch and asynch implementation– Automatically matches/wraps execution with

implementation (e.g. asynch execution on synch implementation)

– Finalized documentation and examples• No obvious functionality missing: this part can be

considered done

public static void main(String[] args) { System.out.println("Locating service method..."); ServiceMethod method = ServiceRegistry.getDefault().findServiceMethod("math/add"); System.out.println("Service method found: " + method); System.out.println("Preparing arguments..."); Map<String, Object> arguments = new HashMap<>(); arguments.put("arg1", ValueFactory.newVDouble(1.0)); arguments.put("arg2", ValueFactory.newVDouble(2.0)); System.out.println("Arguments: " + arguments); System.out.println("Executing service..."); Map<String, Object> result = method.executeSync(arguments); System.out.println("Result: " + result);}

Sync

exe

cutio

n ex

ampl

e

Out

putLocating service method...

Service method found: add(VNumber arg1, VNumber arg2): VNumber resultPreparing arguments...Arguments: {arg2=VDouble[2.0, 2015/03/25 09:35:09.243], arg1=VDouble[1.0, 2015/03/25 09:35:09.229]}Executing service...Result: {result=VDouble[3.0, 2015/03/25 09:35:09.321]}

public static void main(String[] args) { System.out.println("Locating service method..."); ServiceMethod method = ServiceRegistry.getDefault().findServiceMethod("math/add"); System.out.println("Service method found: " + method); System.out.println("Preparing arguments..."); Map<String, Object> arguments = new HashMap<>(); arguments.put("arg1", ValueFactory.newVDouble(1.0)); arguments.put("arg2", ValueFactory.newVDouble(2.0)); System.out.println("Arguments: " + arguments); System.out.println("Executing service..."); method.executeAsync(arguments, result -> { System.out.println("Result: " + result); }, ex -> { ex.printStackTrace(); });}

Locating service method...Service method found: add(VNumber arg1, VNumber arg2): VNumber resultPreparing arguments...Arguments: {arg2=VDouble[2.0, 2015/03/25 09:35:09.243], arg1=VDouble[1.0, 2015/03/25 09:35:09.229]}Executing service...Result: {result=VDouble[3.0, 2015/03/25 09:35:09.321]}

Asyn

c ex

ecuti

on e

xam

ple

Out

put

Services

• Supported services:– v4 PVAccess services– ChannelFinder– Database queries– Command-line execution

• Plenty of copy and paste examples

Formula

• Final review to the function spec and parser– Reorganized implementation, more consistent and

reusable– Parser generates Abstract Syntax Tree (AST)• AST can be manipulated and then converted to the

actual function

– Finalized documentation and examples• Current functionality is done

Formula

Formula

• Other enhancements– Writeable formulas– Passive scanning for formula (performance

improvement)– User defined functions (e.g. site specific function

to handle custom v4 PVStructure, naming conventions, …)

• Currently unfunded

Graphene

• Contributions from students– Improvements to intensity graph color map

• Import color map from Matlab• Support for both relative and absolute value color maps (e.g. RED is

at 10% of the range, or at value 25)

– Time axis and plot• Supports automatic time scale (e.g. sec -> min -> hours -> day ->

week -> month -> year)• Support for daylight savings time, different time zones

– Preliminary JavaFX components based on the graphs• Student work finished• Other work needed to integrate new features

HSVRadian.XML:<?xml version="1.0" encoding="UTF-8"?><!-- Varies the hue component of the HSV color model. The map is useful for displaying periodic functions, and expects the value to be in radians. --><colormap position="absolute" colorNaN="BLACK"> <color position="0.0" value="RED"/> <color position="1.04719755119659774" value="YELLOW"/> <color position="2.09439510239319549" value="GREEN"/> <color position="3.14159265358979323" value="CYAN"/> <color position="4.18879020478639098" value="BLUE"/> <color position="5.23598775598298873" value="MAGENTA"/> <color position="6.28318530717958647" value="RED"/></colormap>

JET.XML:<?xml version="1.0" encoding="UTF-8"?><colormap position="relative" colorNaN="BLACK"> <color position="0.0" value="rgb(0,0,138)"/> <color position="0.2" value="BLUE"/> <color position="0.4" value="CYAN"/> <color position="0.6" value="YELLOW"/> <color position="0.8" value="RED"/> <color position="1.0" value="rgb(138,0,0)"/></colormap>

JavaFX line graph integration:

JavaFX

• There is a general desire to develop widgets and tools in JavaFX– Can be used within SWT/Eclipse/CS-Studio– More full featured than SWT– Can be developed/tested/used outside Eclipse– Allows testing contribution to diirt outside of CS-

Studio• Work was started, currently unfunded

JavaFX

• What is done– Almost all basic tools re-implemented and

expanded in JavaFX– Determined best approach for JavaFX integration• general purpose binding between channel updates and

JavaFX properties

JavaFX

• Probe

Browseableevent history

JavaFX

• Service probe

Autogeneratesfield based onservice data

Currently handles only scalar strings and numbers

JavaFX

• Formula Function Browser

JavaFX

• Service Browser

JavaFX

• What needs to be done– Set of general purpose JavaFX widgets that

understand VTypes and channels/formula/services• Can be used in standalone Java UI and in Eclipse/CS-Studio• Can be used to create BOY widgets• Ideally, use the same definitions for JavaFX/BOY and Web

– Finalize Graphene JavaFX widgets– Finalize toolbox

• Work was started, currently unfunded

CS-Studio

• CS-Studio integration (what’s done)– Renaming packages– Jars are OSGi bundles (no extra build step)– Bridge from Java standard dependency injection

(ServiceLoader) to OSGi implemented– Proof of concept for updating CS-Studio codebase

to new version• These items are done

CS-Studio

• CS-Studio integration (what’s not done)– Finish the integration– Testing– Replace old SWT application with JavaFX substitutes

• Items that were on my horizon– Integrate example BOY screens– BOY JavaFX widgets– Simplify error reporting for connection problems (e.g.

collect pvmanager/caj/pva logs to report to developers)• Work currently unfunded

WebPODS

• Web gateway based on WebSocket + JSON and pure HTML/CSS widgets

• Main benefits:– Uses web standards (any web client in any language)– Widgets can be configured through standard CSS– Data access outside of control network (possibly

WAN)– Web pages do not need to live on the same server

where the gateway is

WebPODS

• Specs available on the WIKI:– https://

github.com/diirt/diirt/wiki/WebPods-Protocol-Specification-v1

– https://github.com/diirt/diirt/wiki/JSON-serialization-for-vTypes-v1

Chrome extension: Simple Web Socket Client

WebPODS when installed

HTML Probe (Danielle Connolly, UMich)

Text monitor examples

LED examples

WebPODSSpecify server location

Adding text-monitor

Adding led

Specify color for “LOW” enum value

Using CS-Studio off site through Web Pods

Web Pods server configuration[xxx@diirt ~]# more .diirt/pods/web/mappings.xml<?xml version='1.0' encoding='UTF-8'?><mappings version="1"> <mapping channel="calibration/constants" substitution="file:///path/to/file/table.csv" permission="READ_ONLY"/> <mapping channel="public-.*" permission="READ_WRITE"/> <mapping channel="cf-tag-(.\w)" substitution="=cfQuery($1)" permission="READ_ONLY"/> <mapping channel="sim/(.*)" substitution="sim://$1" permission="READ_ONLY"/></mappings>

Access security planned, not yet implemented• Use wss (like https) for authentication• Use username/role/unix group/host for authorizationBe careful not to expose too much

Web Pods• What’s done:

– Version 1 specification done and implemented– Preliminary Javascript API – Finalized HTML/CSS compliant widgets (text-monitor, led, table, bubble-graph)– Being used in production at UMich for 6 months under light load. No downtime.– Client-side Java API– Client-side and server-side formulas

• What needs to be done:– More widgets– Client-side security (server side is implemented)– Load tests– Review Javascript API– Client-side reconnected and ping/pong

• What could be done:– Investigate binary protocols (for large data or pre-calculated graphs)– Make single widget spec for HTML and JavaFX widgets