setup/runcontrol andrei and giovanna. setup requirements (feb 2002, by sergei) daq active state: daq...
DESCRIPTION
Setup requirements - general UR – Use of Configuration DB for: description of a partition, dependencies, setup/recovery actions and error conditions. UR005,006 – initialize and shutdown a partition UR007 – operate on several partitions in parallel UR008 – initialize and shutdown a single DAQ component UR009 – verify h/w used by the partition UR010 – quite, trace and profiling 'output modes' UR011 – interactive and automatic operational mode UR012,013 – user GUI, reflecting the state of partition UR014 – GUI for browsing log information produced by a partition UR015 – GUI for editing setup/shutdown dependenciesTRANSCRIPT
![Page 1: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/1.jpg)
Setup/RunControl
Andrei and GiovannaAndrei and Giovanna
![Page 2: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/2.jpg)
Setup Requirements Setup Requirements (Feb 2002, by Sergei)(Feb 2002, by Sergei)
• DAQ Active state: DAQ is ready to accept RC commands from Operator
• DAQ Inactive state (assumed that all h/w in installed and properly connected, PCs loaded etc.)
• Setup and Shutdown – procedures to move DAQ from Active to Inactive state and back.
![Page 3: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/3.jpg)
Setup requirements - generalSetup requirements - general• UR001-004 – Use of Configuration DB for: description of a partition,
dependencies, setup/recovery actions and error conditions.• UR005,006 – initialize and shutdown a partition• UR007 – operate on several partitions in parallel• UR008 – initialize and shutdown a single DAQ component• UR009 – verify h/w used by the partition• UR010 – quite, trace and profiling 'output modes'• UR011 – interactive and automatic operational mode• UR012,013 – user GUI, reflecting the state of partition• UR014 – GUI for browsing log information produced by a partition• UR015 – GUI for editing setup/shutdown dependencies
![Page 4: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/4.jpg)
User Requirements - recoverUser Requirements - recover
• UR016 – recognize and report failure with a DAQ component
• UR017 – try to recover from a failure, if recover procedure is defined
• UR018 – use DVS to diagnose the problem if recover procedure is not defined or does not work
• UR019 – in case of unrecoverable failure setup should stop and ask user for shutdown option
![Page 5: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/5.jpg)
User Reqs: logging and sequrityUser Reqs: logging and sequrity• UR020 – log information produced by DAQ components
• UR021 – log all performed setup/shutdown actions, qualified with timestamps
• UR022 – log all failures and recovery actions, including DVS output
• UR023 – use of user access control
• UR024 – setup should deny initialization of resources allocated from another partition
• UR025 – editing the setup configuration must be protected by user access rights
![Page 6: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/6.jpg)
OkCustomizable coreIr004
OkInterface to be imported in all appsIR002
OkSkeletonIR001
Difficult to measure…Performance requirementsPR001-003
Partially okController failure no effect on controlled items
RR001
Limited reconfiguration possible today. Quality of service concept missing
Configuration and configuration changes
UR401-405
Many hooks, no uniform approachError reception and handlingUR301-307
Ok, extra synchronization to be implementedState Transitions and statesUR201-204
Ok, shared on setup, DSA and LCProcess managementUR101-105
Not implemented.Lock resourcesUR012
Partially okRun interactivelyUR011
Ok in LCQueue commandsUR007
To be done (except shutdown…)Interrupt commandsUR006
OkRefuse commandsUR005
OkDistribute commandsUR004
Ok in SetupTest before usingUR002
Ok1-to-1 with SegmentUR001Controller RequirementsController Requirements
![Page 7: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/7.jpg)
Controller RequirementsController Requirements
![Page 8: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/8.jpg)
Setup: implementation
![Page 9: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/9.jpg)
Implemented functionalityImplemented functionality
• replacement of play_daq, based on DVS and ConfDB
• can make partial setup and re-setup (“recover” setup) if a Segment is attached
• restarting of failed applications
• IDL interface, used from IGUI
• IGUI embeds setup panel which reflects status of infrastructure
![Page 10: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/10.jpg)
Setup IDLSetup IDL boolean status () ; void get_components (out Component comp) ; long subscribe (in callback cb) ; void unsubscribe (in long handler); ComponentStatus setup_component (in string component_name); void reset_component (in string component_name); void abort () ; void shutdown_component (in string component_name); ComponentStatus setup_partition (); boolean shutdown_partition ();
![Page 11: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/11.jpg)
Setup logicSetup logic
Setup:setupComponent()
DVS:verifyComponent()
PMG:startApplication()
- shares verification rules with DVS- extends verification with setup/recovery via PMG (DVS core + PMG Client + more rules)
- implemented on the same CLIPS framework as DVS
![Page 12: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/12.jpg)
Setup sequenceSetup sequence• setup script:
• verify database• launch setup_server for partition• launch IGUI for partition
• setup server (implementing IDL):• verify/setup global infrastructure• verify/setup partition IPC server• if Setup is already registered in the partition, exit• register in partition IPC, wait for commands from
IGUI• on command, verify/setup/shutdown infrastructure
![Page 13: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/13.jpg)
The DSA
Diagnostic Supervisor Diagnostic Supervisor Application/Agent… Did you know Application/Agent… Did you know
it? it?
![Page 14: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/14.jpg)
DSA: the historyDSA: the history
Diagnostics System
VerificationComponent
SupervisionComponent
DAQ Supervisor
DSA
?
![Page 15: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/15.jpg)
Scope? Not completely clear but Scope? Not completely clear but that’s what it does…that’s what it does…
• Process management of all controlled applications (NOT online servers) defined in configuration database for a Partition, taking into account:– When to start/stop them– Dependencies in start/stop sequence– If they shall be automatically restarted when
they fail/die
![Page 16: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/16.jpg)
ImplementationImplementation• Basically an “intelligent” PMG client:
– Navigate through Partition configuration– Subscribe to changes in configuration for Partition,
Segments, Applications, Resources, RunControlApplications (reload of DB only when in Idle state)
– Build dependency tree of apps to be started/stopped– Restart apps– If DSA crashes and is restarted it recovers to previous
state (true?)– Publish state in IS and send messages to MRS
![Page 17: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/17.jpg)
The Online Run Control
![Page 18: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/18.jpg)
ScopeScope
• Model system according to a finite state machine
• Distribute commands in the control tree (in sequence or in parallel)
• Propagate error state in control tree• Provide a c++ interface to subsystems to
implement transition commands
![Page 19: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/19.jpg)
ImplementationImplementation
• Basically an intelligent command dispatcher:– Builds up list of children from configuration DB– Keeps track if child left/joined control tree– Publishes state in IS– Refuses illegal transition commands and all
commands while busy– Root Controller extension updates run
parameters, interprets “end of run” message, etc…
![Page 20: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/20.jpg)
Implementation of DSA and RCImplementation of DSA and RC
• Very similar, based on the same core• Mixture of C++/Java and the CLIPS expert
system• 3 layer approach:
– Language bindings– Proxy– Rules
![Page 21: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/21.jpg)
Language BindingsLanguage Bindings
• This layer is concerned with the integration to other Online Software components.
• Functionality provided by the components client API is wrapped and made accessible to the CLIPS language. The language bindings are grouped in external shared libraries. They can be loaded dynamically and provide their features on demand.
• It is foreseen to develop a comprehensive set of language bindings that encompass main Online Software functionality. Development of such language bindings require detailed knowledge of Online Software and the mechanism of CLIPS language bindings.
![Page 22: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/22.jpg)
ProxyProxy• In this layer external entities, such as applications, run-controllers
or other entities of the TDAQ system, are modelled as objects.
• Typically these objects are proxy objects: on one hand they forward commands to an external application and on the other hand they mirror the state of the external application by subscribing some information. This feature makes the state of an external entity available to the expert system.
• It is foreseen to develop a set of prepared proxy objects for common uses in the online software. Development of specialised objects requires knowledge of the online software concepts and some knowledge of the CLIPS language.
![Page 23: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/23.jpg)
![Page 24: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/24.jpg)
RulesRules• In this layer the rules are defined which provide the logic for the
application. • The rules steer the interaction between the objects and drive the
application. • Development of specialised rules requires knowledge on the proxy objects
and some knowledge of the CLIPS language. It is expected that a detector or subsystem expert could customise actions on this layer to perform some system specific tasks.
• Example:1: (defrule do-something-on-controller-error2: (object (is-a RUN-CONTROLLER) (NAME ?name) (STATUS ERROR))3: =>4: (printout t ‘Controller’ ?name ‘is in error state’ crlf)5: )
![Page 25: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/25.jpg)
AssessmentAssessmentGood
• Layering of “service tools” and “intelligence”
• Rules and facts can be changed without (re)-compilation
• IPC communication mechanism
Not very Good• Separation of process
management and run control (-> synchronization between DSA and RC states)
• No clear way of receiving errors and handling them
•Open for discussion:•How many rules can be added before a system becomes unmanageable?•Can we really expect users to play around with an expert system?•If we go for an expert system we will probably need to define a way for users to specify rules and facts which is independent of the expert system syntax. (if this happens, do that…)
![Page 26: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/26.jpg)
Local Controller
![Page 27: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/27.jpg)
ScopeScope
• Extend Online RC (in 2001 there was only a fixed c++ API to implement commands; expert system appeared in 2003) for ROS
• Serve as:– Configuration server– Process manager (very similar to DSA)– Command dispatcher (rc commands, operational monitoring,
event sampling)– Error handler
• Unique Interface between online software and ROS• Minimal constraints on controlled items (simple,
lightweight interfaces)
![Page 28: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/28.jpg)
Historical EvolutionHistorical Evolution
• UR to have apps not using online software tools removed (evolution of platforms, better comprehension of system)
• Local Controller has become general purpose leaf controller (DF, HLT, ROD Crate DAQ)
• UR to control directly some hardware items instead of apps not used (DF, HLT and detectors prefer to work on their own apps, for easier debugging, standalone running, etc…)
![Page 29: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/29.jpg)
ImplementationImplementation• Extends rc interface• C++ based, with runtime plug-in library for error handling• Extended use of RO infrastructure libraries (threads,
factory for dynamic loading of libs, smart pointers, queues, mutexes, conditional variables, error format/catcher/handler, …)
• Uses IPC for communication between apps and controller, PMG for process management, DB for configuration
• One direction for commands• Opposite direction for errors• Provides main loop for controlled apps• Controlled apps can run in interactive mode with special
command-line option
![Page 30: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/30.jpg)
Controlled AppControlled Appmain { MySubSystemItem pippo(string name); ItemCtrl lucy(*pippo, interactiveMode,…); lucy->run(); exit(0);}
• ItemCtrl sets up path to receive commands and to send errors.
• Errors are all DFError objects and carry information for later handling.
• Failure of synchronous commands is reported by throwing an exception (DFError created from exception and handled like other errors).
![Page 31: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/31.jpg)
AssessmentAssessment
Good• Direct process
management of controlled apps
• Easily customizable hooks for error handling
• Interface to controlled apps allows to run them interactively
Bad/Obsolete• Heavier than necessary
(done to satisfy initial requirements)
• Synchronous implementation of communication
• Within one transition command can only be dispatched to all controlled items in parallel
![Page 32: Setup/RunControl Andrei and Giovanna. Setup Requirements (Feb 2002, by Sergei) DAQ Active state: DAQ is ready to accept RC commands from Operator DAQ](https://reader035.vdocument.in/reader035/viewer/2022062523/5a4d1adb7f8b9ab059974d46/html5/thumbnails/32.jpg)
SummarySummary
• Large part of functionality requested is there, though scattered
• Clear need to rationalize – Interface to subsystems for:
• Error reporting and handling• Command distribution• Process management
– Configuration changes– Use of expert system– Use of DVS for tests