status and future perspectives of the analysis language ( edsl ) for hwc and lhc operation
DESCRIPTION
Status and future perspectives of the analysis language ( eDSL ) for HWC and LHC Operation. Kajetan Fuchsberger TE-MPE TM, 2014-02-27 On behalf of the TE-MPE-MS Software Team: D. Anderson, M . Audrain, M. Dragu , J .-C. Garnier , A. Gorzawski , M. Koza , K.H. Krol , - PowerPoint PPT PresentationTRANSCRIPT
Status and future perspectives of the
analysis language (eDSL) for HWC and LHC Operation
Kajetan FuchsbergerTE-MPE TM, 2014-02-27
On behalf of the TE-MPE-MS Software Team:D. Anderson, M. Audrain, M. Dragu, J.-C. Garnier,
A. Gorzawski, M. Koza, K.H. Krol, A. Moscatelli, K.Stamos,P.C. Turcu
2Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Content
Motivation
Status & Usage
Behind the Scenes
Outlook & Visions
3Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Warning!Vi
siona
rity
Tem
pera
ture
100°
50°
0°
Visionarity Temperature might be High!
(But not unrealistic ;-)
4Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Content
Motivation
Status & Usage
Behind the Scenes
Outlook & Visions
5Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
eDSL ?
eDSLDomain
Specific Language
embedded
Hwc; Systems; Accelerators; Signals;
6Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Why a DSL at all? Currently:
E.g. writing a PM module (or an analysis for a test) is pure coding (Java/LV). A lot of knowledge required, which distracts from the real logic: Where to get the data from? What tools to use to display it? What algorithms to apply?
A lot of code duplication. E.g.: E.g. ArrayUtils in PM modules Every application again connects in its own way to the Logging DB,
PM, reads files....
DSL focuses on the logic of the domain:
7Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Why (Java) embedded? Integration within the Accelerator SW Architecture Java
We are not language designers, but software developers.
Reuse: All the native language features of Java, while providing the
required readability for non-programmers. Tools for writing and verification of analysis scripts (eclipse). Strongly typed; with the help of an external library, it was
easy to enforce the correct usage of physical units.
It provides the necessary possibility to constrain the user input to the available functionality.
8Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Where could it be used? Test Analysis within AccTesting (origin)
PM modules
Pre/Post operational checks
periodic calls + reports: E.g. Splice measurements LHC statistics
…?
Any Java Application!
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
9Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
What about a better name?
ZeldaZuper Easy Language
for Data Analysis
Brainstorming….
GENIAL
Generic Integrated Analysis Language
Analysis Language for Operation and Hardware
Commissioning
10Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Content
Motivation
Status & Usage
Behind the Scenes
Outlook & Visions
11Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Actual Status
Courtesy: M. Audrain
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
12Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Full Automation of PNO.d1 and PCC.1
See Scott & Zinur‘s presentation
Executed ~2000 times within an HWC campaign
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
13Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
What are we missing? More signals from post mortem
(E.g. QPS buffers)
Circuit parameters (deferred scalars)
Calculations:
All logging variables (code generation)
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
14Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
How to test? - Framework On our (SW) side:
http://sonar.cern.ch See also Cristina‘s recent Presentation:https://indico.cern.ch/event/293574/
15Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
How to test? – Module Developer
Use a set of known cases for which the module passes or fails.
To be automated (bamboo)!
16Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Systematic Checks against old data
17Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Integration in AccTestingTo come:Add possibility to display arbitrary signals
18Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
(mis ;-) use for data extraction Do not care, where data comes from!
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
19Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Content
Motivation
Status & Usage
Behind the Scenes
Outlook & Visions
20Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Behind the scenes
request
„compile“
Data
Result
Only extract necessary data!
21Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
The Tree …
Courtesy: Kamil Krol
22Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Resolving….
Main Principle: Ask the Modules, to resolve the biggest subtree possible in one go.
23Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Content
Motivation
Status & Usage
Behind the Scenes
Outlook & Visions
24Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
A more abstract view Requirements for data Analysis:
Data incompleteness (e.g. a beam positions along the ring: added/removed BPMs)
Data invalidity: (e.g BPMs might be misbehaving) Mathematical operations: (e.g.
average/addition/scaling) Physical units: (Different units in storage ->
conversion left to the user) Error propagation: left to the user
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
25Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Anything out there?
Courtesy:K. Stamos, D. Anderson
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
26Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Tensorics® Everything is a tensor.
Common Operations.
Direct and deferred Usage.
Choosable strategies(with useful defaults): Error propagation Data incompleteness Units
To be released as Open Source soon.
Courtesy:A. GorzawskiVi
siona
rity
Tem
pera
ture
100°
50°
0°
27Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
The dispatcher – possible Improvements
Caching
All operations are Stateless (basically mathematical functions) easy to parallelize.
E. Montesnitsalis: UFO search as an example Use Case
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
28Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Blockers & Wishes ;-) Java 8: Closures!
Would allow us to make the language more dynamic and easily extensible.
Will be production released on 18th March. Unfortunately blocked by CO. Will not come
before end of Run 2! Cluster!
To evaluate different options (parallelization, Scalability)
Useful also for different other R&D: PM storage, distributed file systems, test environments, R&D for future protection systems.
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
29Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Only measured data? Circuit Models?
Accelerator Models (MadX)?
Acquisitions (live data)
…
Beam Commissioning?Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
30Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Summary & Outlook Ready (from language point of view) to analyse PNO.d1 and
PCC.1
Next features: More Signals, Circuit parameters (scalars), Calculations.
Switch to Tensorics® when ready.
Optimization (Caching, Parallelization…) to come?… as soon as we can get some cluster!
Can be use in other fields (Models? Big Data analysis? Supervision? Statistics…)
Wishes/dreams: Java8 and a Cluster ;-)
Visio
narit
y Te
mpe
ratu
re
100°
50°
0°Visio
narit
y Te
mpe
ratu
re
100°
50°
0°Visio
narit
y Te
mpe
ratu
re
100°
50°
0°Visio
narit
y Te
mpe
ratu
re
100°
50°
0°Visio
narit
y Te
mpe
ratu
re
100°
50°
0°
31Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Growing a Language http://www.youtube.com/watch?v=_ahvzDzKdB0
Guy Steele ‚Growing a Language‘(Keynote OOPSALA Conference, 1998)
„If one lets the users help do the work, growth can be quick!“
„if many persons work side by side and the best work is added with care and good taste, a great deal can be added in short time.“
32Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Thanks a lot for your Attention!
33Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
Resolving…. Ask each resolver: can resolve ?
If a resolver can resolve the given node remember.
If none of the resolvers can resolve the node, then the same procedure is recursively invoked with all the children of the node (e.g. node B in the example).
Else (at least one potential candidate resolver), the dispatcher stops the iteration through the tree in this branch and does not try to resolve the child nodes anymore (in the initial state of the example, this would happen for node C).
As soon as the iterations through all branches are the dispatcher selects one of the candidate resolvers per node, invokes it and rebuilds the tree with the resolved node.
After all resolvers return, the whole algorithm is looped starting at item (1), until all nodes (including the root node) are resolved.
34Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation
The Tree …
Courtesy: Kamil Krol