diirt status data integration in real time gabriele carcassi
Post on 29-Jan-2016
214 Views
Preview:
TRANSCRIPT
Diirt statusdata integration in real time
Gabriele Carcassi
diirt: data integration in real-time
• Connection/data processing layer for general purpose Java clients– http://diirt.org
• Includes pvmanager/graphene/…• Main two clients:– CS-Studio (Java based Eclipse RCP)– WebPODS (Web-socket + JSON “gateway”)
Requ
est/
resp
onse
Publ
ish/
subs
crib
e
pvmanager
Ch A
cces
s
Processing
Visualization
vTypes Data Definition
Aggregation
formula
graphene
Data integration in real time (Diirt) architecture
BOY Web-pods (aka web gateway)
DB
(jdbc
)
Data serialization/conversion
csv
datasources support services support
list
CS-Studio (general purpose clients) Pods (Protocol Oriented Distribution Service)PV
Acc
ess
Web
Pod
s
File
Loca
l
…
PV A
cces
s
Cmd
line
(exe
c)
Chan
nel
Find
er
…
JSON
Probe …
Funding situation
• Funding at BNL was drastically cut– Closed what I could– Cleanly dropped the rest
• I’ll report on each area– Current state– What we wanted to do but won’t get done
Overall project
• General improvements– All projects in a single repository
• Easier to move code where it belongs• Easier to make releases• (Maven lesson learnt: change version in feature branches)
– All elements configurable through a single configuration directory structure DIIRT_HOME
– Documentation review (Wiki for user documentation, javadocs for developer documentation, documented source code examples in the repository)
VTypes
• Review types with Java 8 in mind– Use javax.time– Support for unsigned integers– Reorganize type hierarchy to overcome some
implementation and usage issues– Reorganize utility methods within the type
hierarchy (instead of utility classes)• Work was started, currently unfunded• Frozen in org.diirt.vtype.next package
VTypes
• Extract VType conversions from JCA/PVA support– Allow to use same exact type conversion for those
who do not need/want the whole stack• Currently unfunded
Datasources (pvmanager)
• Final review to the datasource spec– Generalize type conversion for writes (so that all datasources
behave the same) – Use Java 8 standard classes where possible– Generalize caches from sources of external data (i.e. better
support for archive data, better link between services, …)– Finalized documentation and examples for writing datasources– Performance improvements– Streamline error handling (i.e. do more work in the framework,
clients are simpler, error handling more homogeneous)• Currently unfunded
Archiver integration
• Client side temporal cache that can be filled by different archiver sources and real-time updates
• Under development by ITER.
Services
• Final review to the services spec– Uses Java 8 standard classes for callback– Defines order for arguments and results– Allows both synch and asynch execution– Allows both synch and asynch implementation– Automatically matches/wraps execution with
implementation (e.g. asynch execution on synch implementation)
– Finalized documentation and examples• No obvious functionality missing: this part can be
considered done
public static void main(String[] args) { System.out.println("Locating service method..."); ServiceMethod method = ServiceRegistry.getDefault().findServiceMethod("math/add"); System.out.println("Service method found: " + method); System.out.println("Preparing arguments..."); Map<String, Object> arguments = new HashMap<>(); arguments.put("arg1", ValueFactory.newVDouble(1.0)); arguments.put("arg2", ValueFactory.newVDouble(2.0)); System.out.println("Arguments: " + arguments); System.out.println("Executing service..."); Map<String, Object> result = method.executeSync(arguments); System.out.println("Result: " + result);}
Sync
exe
cutio
n ex
ampl
e
Out
putLocating service method...
Service method found: add(VNumber arg1, VNumber arg2): VNumber resultPreparing arguments...Arguments: {arg2=VDouble[2.0, 2015/03/25 09:35:09.243], arg1=VDouble[1.0, 2015/03/25 09:35:09.229]}Executing service...Result: {result=VDouble[3.0, 2015/03/25 09:35:09.321]}
public static void main(String[] args) { System.out.println("Locating service method..."); ServiceMethod method = ServiceRegistry.getDefault().findServiceMethod("math/add"); System.out.println("Service method found: " + method); System.out.println("Preparing arguments..."); Map<String, Object> arguments = new HashMap<>(); arguments.put("arg1", ValueFactory.newVDouble(1.0)); arguments.put("arg2", ValueFactory.newVDouble(2.0)); System.out.println("Arguments: " + arguments); System.out.println("Executing service..."); method.executeAsync(arguments, result -> { System.out.println("Result: " + result); }, ex -> { ex.printStackTrace(); });}
Locating service method...Service method found: add(VNumber arg1, VNumber arg2): VNumber resultPreparing arguments...Arguments: {arg2=VDouble[2.0, 2015/03/25 09:35:09.243], arg1=VDouble[1.0, 2015/03/25 09:35:09.229]}Executing service...Result: {result=VDouble[3.0, 2015/03/25 09:35:09.321]}
Asyn
c ex
ecuti
on e
xam
ple
Out
put
Services
• Supported services:– v4 PVAccess services– ChannelFinder– Database queries– Command-line execution
• Plenty of copy and paste examples
Formula
• Final review to the function spec and parser– Reorganized implementation, more consistent and
reusable– Parser generates Abstract Syntax Tree (AST)• AST can be manipulated and then converted to the
actual function
– Finalized documentation and examples• Current functionality is done
Formula
Formula
• Other enhancements– Writeable formulas– Passive scanning for formula (performance
improvement)– User defined functions (e.g. site specific function
to handle custom v4 PVStructure, naming conventions, …)
• Currently unfunded
Graphene
• Contributions from students– Improvements to intensity graph color map
• Import color map from Matlab• Support for both relative and absolute value color maps (e.g. RED is
at 10% of the range, or at value 25)
– Time axis and plot• Supports automatic time scale (e.g. sec -> min -> hours -> day ->
week -> month -> year)• Support for daylight savings time, different time zones
– Preliminary JavaFX components based on the graphs• Student work finished• Other work needed to integrate new features
HSVRadian.XML:<?xml version="1.0" encoding="UTF-8"?><!-- Varies the hue component of the HSV color model. The map is useful for displaying periodic functions, and expects the value to be in radians. --><colormap position="absolute" colorNaN="BLACK"> <color position="0.0" value="RED"/> <color position="1.04719755119659774" value="YELLOW"/> <color position="2.09439510239319549" value="GREEN"/> <color position="3.14159265358979323" value="CYAN"/> <color position="4.18879020478639098" value="BLUE"/> <color position="5.23598775598298873" value="MAGENTA"/> <color position="6.28318530717958647" value="RED"/></colormap>
JET.XML:<?xml version="1.0" encoding="UTF-8"?><colormap position="relative" colorNaN="BLACK"> <color position="0.0" value="rgb(0,0,138)"/> <color position="0.2" value="BLUE"/> <color position="0.4" value="CYAN"/> <color position="0.6" value="YELLOW"/> <color position="0.8" value="RED"/> <color position="1.0" value="rgb(138,0,0)"/></colormap>
JavaFX line graph integration:
JavaFX
• There is a general desire to develop widgets and tools in JavaFX– Can be used within SWT/Eclipse/CS-Studio– More full featured than SWT– Can be developed/tested/used outside Eclipse– Allows testing contribution to diirt outside of CS-
Studio• Work was started, currently unfunded
JavaFX
• What is done– Almost all basic tools re-implemented and
expanded in JavaFX– Determined best approach for JavaFX integration• general purpose binding between channel updates and
JavaFX properties
JavaFX
• Probe
Browseableevent history
JavaFX
• Service probe
Autogeneratesfield based onservice data
Currently handles only scalar strings and numbers
JavaFX
• Formula Function Browser
JavaFX
• Service Browser
JavaFX
• What needs to be done– Set of general purpose JavaFX widgets that
understand VTypes and channels/formula/services• Can be used in standalone Java UI and in Eclipse/CS-Studio• Can be used to create BOY widgets• Ideally, use the same definitions for JavaFX/BOY and Web
– Finalize Graphene JavaFX widgets– Finalize toolbox
• Work was started, currently unfunded
CS-Studio
• CS-Studio integration (what’s done)– Renaming packages– Jars are OSGi bundles (no extra build step)– Bridge from Java standard dependency injection
(ServiceLoader) to OSGi implemented– Proof of concept for updating CS-Studio codebase
to new version• These items are done
CS-Studio
• CS-Studio integration (what’s not done)– Finish the integration– Testing– Replace old SWT application with JavaFX substitutes
• Items that were on my horizon– Integrate example BOY screens– BOY JavaFX widgets– Simplify error reporting for connection problems (e.g.
collect pvmanager/caj/pva logs to report to developers)• Work currently unfunded
WebPODS
• Web gateway based on WebSocket + JSON and pure HTML/CSS widgets
• Main benefits:– Uses web standards (any web client in any language)– Widgets can be configured through standard CSS– Data access outside of control network (possibly
WAN)– Web pages do not need to live on the same server
where the gateway is
WebPODS
• Specs available on the WIKI:– https://
github.com/diirt/diirt/wiki/WebPods-Protocol-Specification-v1
– https://github.com/diirt/diirt/wiki/JSON-serialization-for-vTypes-v1
Chrome extension: Simple Web Socket Client
WebPODS when installed
HTML Probe (Danielle Connolly, UMich)
Text monitor examples
LED examples
WebPODSSpecify server location
Adding text-monitor
Adding led
Specify color for “LOW” enum value
Using CS-Studio off site through Web Pods
Web Pods server configuration[xxx@diirt ~]# more .diirt/pods/web/mappings.xml<?xml version='1.0' encoding='UTF-8'?><mappings version="1"> <mapping channel="calibration/constants" substitution="file:///path/to/file/table.csv" permission="READ_ONLY"/> <mapping channel="public-.*" permission="READ_WRITE"/> <mapping channel="cf-tag-(.\w)" substitution="=cfQuery($1)" permission="READ_ONLY"/> <mapping channel="sim/(.*)" substitution="sim://$1" permission="READ_ONLY"/></mappings>
Access security planned, not yet implemented• Use wss (like https) for authentication• Use username/role/unix group/host for authorizationBe careful not to expose too much
Web Pods• What’s done:
– Version 1 specification done and implemented– Preliminary Javascript API – Finalized HTML/CSS compliant widgets (text-monitor, led, table, bubble-graph)– Being used in production at UMich for 6 months under light load. No downtime.– Client-side Java API– Client-side and server-side formulas
• What needs to be done:– More widgets– Client-side security (server side is implemented)– Load tests– Review Javascript API– Client-side reconnected and ping/pong
• What could be done:– Investigate binary protocols (for large data or pre-calculated graphs)– Make single widget spec for HTML and JavaFX widgets
top related