slow controls, simulations and daq/data analysis— wbs 1.7
DESCRIPTION
Slow Controls, Simulations and DAQ/Data Analysis— WBS 1.7. Jim Miller Boston University Chicago meeting, June 6-8, 2007. Outline. Subsystem Overview, WBS 1.7 Scope of Work Plans and organization Summary. Scope of Work. This WBS element has four independent parts: - PowerPoint PPT PresentationTRANSCRIPT
1
Jim MillerBoston University
Chicago meeting, June 6-8, 2007
Slow Controls, Simulations and DAQ/Data Analysis—WBS 1.7
2
Outline
• Subsystem Overview, WBS 1.7– Scope of Work– Plans and organization
• Summary
3
Scope of Work
This WBS element has four independent parts:
• Develop a reliable and redundant system to control the operation of the EDM apparatus;
• Develop a data acquisition system for carrying out the EDM measurement program;
• Coordinate simulations to model the apparatus and all aspects of the experiment;
• Coordinate development of analysis codes to process large amounts of data.
4
Planning Assumptions—Slow Controls
• EPICS (Experimental Physics and Industrial Control System) will control and monitor the apparatus, and control the measurement cycle (change valves, HV, magnets etc..)
• Anticipate up to 1000 read-out and control parameters; system must be distributed, reliable, capable of remote operation, and redundant (if the EPICS main/local control fails, hard-wired local control should be readily available)
• Similar sub-systems for neutronics, cryogenics, 3He, inserts, and magnetic fields (N,C,H,I,M)
• Utilize Unix/Linux, VME and VxWorks to capitalize on existing software libraries (Argonne light source web page is a repository of such libraries and a reference source for EPICS info, )
• Safety decisions should be hard-wired locally; monitored and alarmed but NOT controlled by EPICS (quench protection, HV breakdown, vacuum loss…)
EPICS widely used – SNS, APS, JLABEPICS widely used – SNS, APS, JLABMulti-headed: framework, protocol (Channel Access), tools, driversMulti-headed: framework, protocol (Channel Access), tools, drivers
Workstations: *Sun Hp DEC/Alpha Silicon Graphics * PC MacintoshOS: Unix, *Linux, *Windows
I/O Controllers: *VME,VXI PCI, WorkstationsOS: *VxWorks, Unix, Windows RTEMS, RTLinux, L4 linux
Field I/ORemote and Local I/O Buses: Control Net, PCI, CAN-Bus, *Industry Pack, *VME, xVXI,x ISA, CAMAC, *GPIB, Profibus, Bitbus, *Serial, *Allen-Bradley, Modbus, Yokogawa, G-3, USB *Ethernet/IP
Field I/O Field I/O
Site LAN/WAN
Field I/O
Courtesy Bob Dalesio (APS)
6
Likely EPICS slow control baseline
• Choose VME-based : reliable and widely used
• Choose VxWorks- real-time operating system
• VxWorks software development should be done on Sun-Solaris
• VxWorks-based applications downloaded to local computers (Input/Output Controllers, or IOC) in each VME crate
• Local control devices are mounted in VME crates
• Ideally, each sub-system would have a VME crate, IOC, etc., and would develop the slow controls with EPICS from day 1.
• A sub-committee of EPICS experts is being formed to help collaborators select control devices and to assist in EPICS setup.
Please check with this group before purchasing control equipment.
7
EPICS- basic example
Suppose that you want to read the voltage from a thermistor
• A VME module is purchased which senses voltages.
• Or more commonly, a separate module reads the voltage, digitizes it, then sends the data via GPIB, serial, etc. to an interface module in the VME crate.
Either way, we strongly encourage using modules which have EPICS drivers which already exist. (information on supported modules is available at the Argonne Light Source web site: www.aps.anl.gov/epics)
• A VME Input/Output Controller (IOC) (a computer mounted in the VME crate running the VxWorks operating system) reads data in real time.
• The datum is then transferred to your PC (LINUX, Windows,…) for processing.– Simple input or output commands– Mouse-driven PC display which has menus allowing read/write of variables,
strip charts, monitoring, etc.
• The particular variable must have a unique name, up to 26 characters, since your PC has access to all variables on the CA (Channel Access)
• Naming scheme example- cryo:read:upper2:temp:F or similar
.
VME Crate
Single Board Computer (SBC, PPC)
Digital to Analog (DAC), etc to device controllers
Analog to Digital (ADCs),TDC, scalers, etc from detectors
Operator Interface (OPI),Linux Host Computer
CODA HOST,Linux Host Computer
(runs VxWorks)
LAN LAN
IO Modules + SBC + LAN = Input/Output Controller(IOC)
• EPICS Core• IOC Database Management• various development toolkits• Software device drivers, etc• Motif Tools
• CODA Core (coda_roc)• Database Management (SQL)• Software device drivers, etc• Tcl/Tk tools
This link needs to be developed
We need to be mindful of electrical noise causing problems with SQUIDS
EPICS test bench setup at TUNL- M. Ahmed, C. Swank, C .Taylor, S Hartman
VME Crate
VMIVME4132DAC
MVME-5100, SBCAlarms, LEDs, etc
Current Integrator
EPICS control of the test bench devices
Slow Controls R&D, Development of an EPICS based system for nEDMMohammad Ahmed, Duke University
Work at TUNL is focused on developing a prototype EPICSSystem for the control of R&D Cryo System. The EPICS developmentis being done by Mohammad Ahmed (Duke) and the cryo systemdevelopment is being coordinated with David Haase andthe NC-State Group.
Outline of EPICS R&D• Development of slow controls Infra-structure
• Hierarchy of Controls and Menus• Nomenclature of IOCs, Channels, etc
• Development of an outline on standards of hardware purchases to:• maximize compatibility of hardware across groups• reduce software development• ensure cost effectiveness of controls (minimize cost/control channel)
• Development of a prototype system which can be used as an example
What have we acquired ?
• VxWorks ( Mohammad Ahmed got VxWorks viaeducational grant of the software)
• VME Crate• Single Board Computer• Analog Output Controller
What is next ?
Hardware and Software
• Lakeshore Temperature Monitor• VME Analog Input• USB Controller (computer)• Temperature Sensors
We are in the process of making an nEDM distribution of EPICSwith standard set of extensionsEPICS supports all sorts of mouse-driven control screens, e.g Motif,tcl/tk- many modern ones to choose from- perhaps the simplest these dayswould be web-based?
Software/GUI development
A prototype nEDM Main control window
EPICS ControlsFor R&D Cryo
Read out willbe implementedwith Lakeshorecontroller
15
Planning Assumptions – DAQ
• Neutron-3He capture event rate ~ 1kHz , background (and ) rate comparable to event rate.
• Require pmt coincidence trigger to suppress background events in the light guides
• Digitize prompt and afterpulse pmt waveforms for ~ few sec to suppress background events. Issue: how much zero suppression in waveform data stream will be possible?
• Digitize SQUID voltages at 1 ms intervals (3He precession rate ~10-30 Hz)
• Use VME and CODA – developed and supported by JLAB, used at TUNL, ….
• Issue: need to construct the pipeline which carries EPICS data into the CODA DAQ stream
16
Sample event from HMI test setup
17
Mike Hayden et al LT12
Afterpulsing Is a Signature of Good Events
18
Schematic DAQ
5 s pair- coinc. enable
4 ch. 12-bit 1 GHz TD
4 ch. 12-bit 1 GHz TD
Cell B TD’s
Cell A PMT’s 5-8 E,F,G,H
VME crate + CODA computer
Cell A PMT’s 1-4
PMT’s 1-8 discr’s
Squid ADC’s
Event time stamp
Scalers
Also need to add a route for EPICS slow control info to ge to DAQ stream
A Note on R&D for the DAQ development
TUNL has full CODA capability. Complete hardware and softwareInfra-structure exists to do DAQ development. Mohammad Ahmedis a co-lead on CODA based DAQ at TUNL
IDEA Use Flash ADCs to transient digitize the PMT signal. AdditionalADC/QDC/TDCs can be made part of the readout.New issue: Do we need to develop FADCs which are not noisy, and which sitnear the PMTs?
Where do we stand on DAQ R&D for nEDM ?
• We have a VME crate,• We have single board computer,• We have a FADC (100 MHz),• We have ADC/QDC/TDC,• We have Scalers• Noise levels should be measured
A Note on R&D for the DAQ development
Software development is being carried out for the FADC. The planis to readout sample data using the FADC.
What is State-of-the-Art available in the Market for FADC ?
Struck SIS3350 (VME-based)
• 4-channels• 500 MHz• 12-bit resolution• 512 MSamples/Channel• internal/external clocks
Getting close to what we want ?Fine if we have <20 channels or so. Mayneed custom if we need a large numberof channels.
The goal is to do a trigger based, deadtime-free acquisition of multi-event signals.
21
Planning Assumptions—Data Analysis
• Expect 10 GB of waveform data per 1000 sec measurement cycle (10 kbytes of data per event, event rate of 1kHz). At 30% live time ~ 100 TB per year. Potentially large error on this number- depends on the number of PMTs, number of bits in WFD (depends on dynamic range needed, need 12 bits?), sampling rate of WFD (depends on width of pulses), and the decision on the permissible extent of zero suppression.
• Need fast analysis for diagnostic monitoring during commissioning- An online version to monitor system functionality, offline version for detailed analysis
• Need reliable data compaction methods as experiment moves into production mode.
• Slow control data rate is manageable: ~2kB at 1Hz ~ 60GB per year
• Establish programming standards to facilitate independent analyses of the data by collaboration members.
22
Major Planning Assumptions—Simulations
Work is carried out by WBS system groups:
• beam transport into the measurement cell* – How many neutrons get there, what is their momentum, spin, position?
Where are they lost along the way? How much background do they produce?
• 3He preparation, injection and removal,
• magnetic and electric field configurations*,
• neutron cell dynamics (momentum and spin motions) and n- 3He spin interaction
• experiment cycle optimization,
• scintillation light production and propagation*
(* potentially impacts design)
General simulation package• ~3 or 4 major pieces
– UCN in cells– N beam flux– Light production and collection in the cells– He3 cycle and behavior in cell
• Geometries defined or at least compatible as much as possible with GEANT4• Use ROOT-based analysis• C++, C, Fortran• Begin with
– central cell simulation w/ interactions with walls, electrodes, magnets…• flexibility to study different geometries and materials: database driven so that
geometries can be modified by changing parameters rather than re-coding– model where neutrons go en-route to the cell, how many stop in the cell, how
many pass through the cell, where they get captured and cause background, etc.• Model the neutron and 3He trajectories, spin motions in the E and B fields of the cell,
study geometric phase effects, spin coherence times, other systematic issues.• Model capture process: neutrons on 3He (probably not available in GEANT)• Model light collection (get neutron capture coordinates from central cell simulation,
return distribution of PMT responses) (questionable in GEANT)• Simulate effects of cosmics, other backgrounds, effectiveness of suppression via
detection of after-pulses• Once backgrounds are known, optimize the cycle times, estimate nEDM sensitivity• Effects on B field due to current from HV discharges, stray B-fields,…• Build in extensive fail code bits to keep track of where and why particles are lost,
failure modes, etc.
24
Simulations- continued
• Goal: make simulations as compatible as possible, distribute the tasks, and build a more global simulation package which is available for general use
• We will need a bank of CPUs for simulations, and later for data analysis
• A simulations committee will be formed, with knowledgeable members from each subsystem, to coordinate simulations, establish standards
• Draft decision: ROOT and GEANT4 compatibility. Output formats compatible. Common data base and naming scheme for variables where possible.
• Full simulation should be available to deliver ‘real’ data in ‘real’ format for playback in the data analysis.
• Likely scenario: we don’t know final DAQ format yet; set up a flexible intermediate format agreed upon by offline analysis and simulations. Later, convert real data to this format. In practice:there is no offline team yet, simulations team will try to put something together with offline in mind.
• Need volunteers from each subsystem!
• We also need a brief summary from each group describing what simulations they have done, are working on, or are planning.
• Maintain a catalogue of online notes
25
A few examples of simulations
• Betsy Beise has an undergraduate (Joshua Rehak) who will get a lightguide program (LITRANI) running to evaluate light collection in the proposed light guides over the summer (already have design from T. Ito, expect alternative geometry in a few weeks from L. Roberts)
• Bob Golub plans to study n, He3 spin evolution, geometric phases, etc., and would welcome student help
• The Caltech group has been studying spin coherence times, trajectories, geometric phase effects, etc. in addition to B field studies.
• Neutron transport simulations at Kentucky seem to be well-along; we need to get the output parameters of neutrons for input to central cell, also distribution of where neutrons were lost in order to do background studies.
• Goal: build a team to create the central cell simulation in GEANT4 and set up interfaces to other system simulations. BU postdoc Vanya Logashenko will begin the process
26
Project Management
27
Summary
• Slow Control– VME-based, EPICS compatible modules, using VxWorks in the VME
computer– A small committee of experts should be consulted for purchases, setup of
local systems
• DAQ– VME-based CODA, which uses VxWorks OS (unless VME is found to be
too noisy for the SQUIDS!)– FADCs for PMT readout– Slow control data incorporated into DAQ data
• Simulations– GEANT4, ROOT as much as possible– Develop a GEANT4 model of central cell, take input neutrons from
transport simulations, study spin evolution, particle diffusion, geometric effects, effects of leakage currents,…
– Put together a group of simulation experts from each sub-system which will decide standards and work toward compatible code Please volunteer
28
Technical Interfaces
• N,C,H,I,M groups will select hardware compatible as much as possible with existing EPICS software, in consultation with the controls committee
• N,C,H,I,M groups can choose how they want to control their equipment during initial testing, but will need to coordinate closely with controls group to insure compatibility at the integration stage
• Preference: Control systems can be delivered to the N,C,H,I, groups if they prefer to use EPICS at the outset but funds may not be there yet to support this
(neutronics, cryogenics, 3He, inserts, and magnetic fields = N,C,H,I,M)
29
Initial Risk Analysis
Risks Probability Strategy Mitigation
Low light output Medium AvoidSimulations and/or change DAQ
Cosmic ray backgrounds
Low Control Implement a veto system
VME costs too high Medium TransferAlternate hardware (but more expensive software)
3He: flow and non-uniformity
Medium Control Implement an NMR system
Note: Additional Data available in EDM Risk Management Plan
30
Estimated Costs
ODC: other direct costs MP: Major purchasesCont.: Initial Contingency Est. TEC: total estimated costs * Costs do not include escalation
WBS Description Labor ODC MP Cont TEC
1.7.1 slow controls systems 91,242 91,980 134,000 31,722 348,944
1.7.2 Data acquisition system - design procurement and testing 13,666 5,110 75,000 12,455 106,230
1.7.3 Simulations
1.7.4 Data analysis package
1.7 Electronics, Computers, Simulations, Data Analysis 104,907 97,090 209,000 44,177 455,174
Office of Nuclear Physics
Preliminary BA Schedule - Electronics
32
Deliverables
• Subsystem deliverables include:– Five EPICS VME control systems for sub system groups– Integrated EPICS control system at the FNPB– CODA based DAQ and local analysis computer systems – Simulation results that impact design of apparatus– Simulations that model analysis of the data stream from the DAQ
33
Summary
• Subsystem Overview– Scope of Work– Work Packages– Technical Discussion
• Project Management– Technical Interfaces– Risk Analysis– Estimated Costs– Preliminary Schedule– Deliverables
34
Supplementary Slides
35
Major Procurements and Total Cost
All equipment is standard off-the-shelf hardware:
• VME systems for slow controls (crates, controllers, ADC, DAC, stepper motors, scalers, timers, computers, VxWorks..) ~$226K
• VME system for DAQ (Waveform digitizers, SQUID readouts, MCA’s, discriminators, logic modules, computers…) ~$82K
• Total unburdened hardware = $308K
• Total labor (technician and software engineer): $101K
• Total including contingency: $457K
36
STEP file input from CAD to GEANT4
There is NIST-developed code for reading "AP203" STEP files in GEANT through versions 4.5.x.
Starting with 4.6 support for STEP files was removed and no longer supported.
Working now to implement a trial conversion of the central region of the apparatus.
37
Work Packages
WBS Work Package Description WP Manager
1.7.1 Slow controlsJim MillerBoston University
1.7.2 Data acquisition system ``
1.7.3 Simulations ``
1.7.4 Data analysis software ``