gunther haller [email protected] lusi doe review aug. 20, 2008 controls (wbs 1.6) p. 1...

Gunther [email protected]

LUSI DOE Review Aug. 20, 2008Controls (WBS 1.6) p. 1

Breakout Session

G. Haller – Sub-System Manager

August 20, 2008

LUSI LUSI WBS 1.6 Controls and Data SystemsWBS 1.6 Controls and Data Systems



Content

ScopeCost & Schedule

WBS OrganizationCostSchedule

Control and Data System ArchitectureControl System

ArchitectureDevicesController Examples

Data SystemData Systems ArchitectureScience Data Acquisition & ProcessingDAQ ComponentsHigh Level Applications, Online ArchiveOffline File Management, Meta Data

Summary



LUSI Controls & Data Systems Location

XPPLCLS AMO

X-Ray Transport Tunnel (XRT) (200 m)

XCS

HEDS

CXI

SXR imaging

Controls & Data Systems hardware/softwareHutches 3, 4, an 5Control rooms for hutches 3, 4, and 5X-Ray tunnelNEH and FEH server rooms

H: Hutch

NEH (H3 , H3 Control Room & Server Room

FEH (H4/5, H4/5 Control Rooms & Server Room

Common control and data systems design for photon beam-line/instruments (XTOD, AMOS, LUSI, SXR)



Scope – WBS 1.6 Control & Data Systems

Included in W.B.S. 1.6All controls & DAQ, labor and M&S, for XPP, CXI, XCS instrument components with diagnostics/common optics included in baseline

Includes controllers, racks, cables, switches, installationData-storage and processing for FEHInitial offline (more effort will be on operating budget)Input-signals to LCLS machine protection system link-node modules

Provided by LCLS X-Ray End Station controls (CAM is G. Haller)Personnel protection systemMachine protection system (LCLS modules, fibers)Laser safety systemAccelerator timingFemto-second laser timingNetwork architecture & securityData-storage and processing for NEHUser safeguardsLaser controlsCXI 2D detector controls

Interfaces described in1.1-517 ICD between XES and LUSI (released document)



Performance Requirements

From LUSI Performance Execution Plan (PEP)

This presentation will show that the requirements will be fulfilled



1.6 WBS to Level 4

1.6Control & Data

Systems

1.6.1Integration & Management

1.6.2Common Controls

1.6.3XPP

1.6.4CXI

1.6.5XCS

1.6.6Offline

Computing

1.6.3.1Requirements, Design, Setup

1.6.3.2Standard Hutch

Controls

1.6.3.4Specific Controls

Example XPP



W.B.S 1.6.2 Common Controls

W.B.S. 1.6.2 Common ControlsPhoton beam feedbackElectron beam feedbackHutch environmental measurementFEH data storageData processing

Initial level 2 processing

Racks & cablesNon-hutch racks and cables, mainly FEH



W.B.S 1.6.3, 1.6.4, 1.6.5

W.B.S. 1.6.3 XPP, 1.6.4 CXI, 1.6.5 XCSRequirements, design, setupStandard hutch controls

Hutch cables, racks, installationWorkstationsBeamline processorChannel access gatewayMachine protection system Interface

Specific controlsValve/vacuum controlsPop-in profile monitorPop-in intensity monitorIntensity position monitorSlit controlsInstrument specific controls for each section of the instrument



W.B.S 1.6.6 Offline Computing

W.B.S. 1.6.6 Offline ComputingData-formatAPIData-catalogMeta-data managementProcessing frameworkWorkflowPipeline



Cost Methodology

Basis for agreement on what components need to be controlled and how

Detailed Engineering Specification Documents (ESD’s) for each instrumentAll ESD’s are approved and released

Two ESD’s for each instrumentControls ESD

Describing devices to be controlledE.g. motion, vacuum

EPICS processing to be performed E.g scanning

DAQ ESDDescribing devices to be read into DAQ

E.g. 2-D detectors, waveform sampling, some 120-Hz cameras, etcOnline processing to be performed

Plus one ESD for diagnostics

Basis for agreement on who is responsible for what and where the interface is: Interface Control Documents (ICD’s)

ICD’s to all instruments are approved and released



Example XPP Beam-Line

Start from beam-line, itemize controls



ESD’s and ICD’s

XPP SP-391-001-21 XPP Controls ESD SP-391-001-22 XPP Controls & DAQ ICD SP-391-001-23 XPP DAQ ESD

CXISP-391-001-13 CXI Controls ESD SP-391-001-14 CXI Controls & DAQ ICD SP-391-001-18 CXI DAQ ESD

XCSSP-391-001-24 XCS Controls ESD SP-391-001-25 XCS Controls & DAQ ICD SP-391-001-26 XCS DAQ ESD

DiagnosticsSP-391-001-19 LUSI Common Diag. & Optics ESD

All documents athttp://confluence.slac.stanford.edu/display/PCDS/LUSI+Document+Page



Cost Methodology

Bottoms-up: supporting excel spread-sheet organized by WBS created from ESD content (agreement between scientist and controls)

LaborNumber of hours and detailed tasks for each WBSBased on prior experience from previous SLAC experiments

MaterialLists each individual component to be purchased with price under each WBSEach item is labeled with reference numberReference number references component on LUSI Controls Item list spread-sheet

List of every controls item used for LUSI

> 95% of components supported by quotes or purchase ordersAll items on item list supported by quote or purchase order print-out



WBS Spread-Sheet Example

WBS Activity BOE Hours Cost

Item Item # Count $/each Total

Item # references Controls Item list, see next slide



LUSI Controls Item List

Below are first 14 items of LUSI Controls Item listTotal ~70 separate items

Components in WBS spread-sheet refer to this Reference Number

Price support pages containing copies of previous orders or quotes are labeled with this item #



Contingency

Contingency calculated for each element from two factors

Design Maturity 6 levels for labor5 levels for M&S

Judgment FactorRisks, exchange rate, etc

Held at project level



Project Budget

WBSControl

AccountsWork

PackagesValues

WBS 1.1 6 12 $5,461,314

WBS 1.2 14 49 $5,942,486

WBS 1.3 11 45 $9,486,460

WBS 1.4 16 45 $7,715,265

WBS 1.5 10 39 $6,383,995

WBS 1.6 (G. Haller) 20 289 $7,135,691

Total BAC $42,125,211

WBS 1.6

Resource Type Value

Labor $3,409,458

Non-Labor $3,726,233

Total BAC $7,135,691

Detailed bottoms-up cost estimateLabor: number of hours listed for each task All M&S itemized to the component levelAlmost 100% supported by vendor quotes or recent purchase orders



Schedule

All tasks and materials (order, award, receive dates) in P31.6 is internally linked with predecessors and successors“Available” mile-stones for each deliverable identified and enteredLinked to instrument “Need” mile-stonesResources leveled



Milestones

XPP

XPP Controls PDR Dec 08

CD-3A – XPP Instrument Start Construction Jun 09

XPP Controls FDR Sept 09

XPP Controls available Mar 10

CD-4A – XPP Start Operation Dec 10

CXI

CXI Controls PDR Sep 09

CD-3B – CXI – Instrument Start Construction Apr 10

CXI Controls FDR Jun 10

CXI Controls available Nov 10

CD-4B – CXI – Start Operation Dec 11

XCS

XCS Controls PDR Nov 09

CD-3C – XCS – Instrument Start Construction Apr 10

XCS Controls FDR Feb 11

XCS Controls available Jul 11

CD-4C – XCS – Start Operation Aug 12



CDA Schedule Critical Envelope

CDA has multiple deliveries to the instruments and is

heavily driven by their needs. The project will

monitor strings of activities with the least float



Slow Controls Tasks & Hardware

EPICSIn use at BaBar, APS, ALS It is the LCLS control system

Basic EPICS Control and Monitoring Vacuum: Instruments, connecting ‘pipes’Valve controlTiming/triggering (timing strobe from EVR)Motion control (‘stages’)Camera controlBias voltage supplies120-Hz (slow) Analog-Digital Converters Digital IO bits/statesTemperatures

HardwareAs much as feasible chosen from LCLS repertoireAdded new controllers based on instrument requirements



Common Controls Hardware

ExamplesRacksVME CratesMotorola CPUsTiming EVR PMC cardsCameralink PMC cardsVME ISEG HV suppliesAnalog-digital converter modulesSolenoid controllersPLCsNetwork switchesTerminal servers (Ethernet-to-Serial Port)



Example: Motion Systems

Newport XCS controller



Common Diagnostics Readout

• Four-diode design

R2

R121

L

Target

Quad-Detector

FEL

• On-board calibration circuits not shown

E.g. intensity, profile monitor, intensity position monitorsE.g. Canberra PIPS or IRD SXUV large area diodes (single or quad)

Amplifier/shaper/ADC for control/calibration/readout



Interface to LCLS

Interface to LCLS/X-Ray End-Station InfrastructureMachine timing (~ 20 psec jitter)Laser timing (< 100 fsec jitter)120 Hz beam dataMachine protection systemHutch protection systemLaser safety systemNetworkingEPICS server



120-Hz Data Feedback Loop

Low latency 120 Hz beam-line data communicationUse existing second Ethernet port on IOC’s

No custom hardware or additional hardware requiredUDP multi-castRaw Ethernet packages

IOC IOC IOC

Accelerator EO Experiment

Timing

120-Hz network

Realtime per-pulse information can be used for e.g. Vetoing of image samples (using accelerator data)Adjustment of accelerator or photon beamline components based on instrument/diagnostics results

Compensation of drifts, etcTransport of electro-optics timing result to hutch experiments



Data Sub-System

Data Rate/Volume of CXI Experiment(comparable to other LUSI experiments)

LCLS Pulse Rep Rate (Hz) 120

Detector Size (Megapixel) 1.2

Intensity Depth (bit) 14

Success Rate (%) 30%

Ave. Data Rate (Gigabit/s) 0.6

Peak Data Rate (Gigabit/s) 1.9

Daily Duty Cycle (%) 50%

Accu. for 1 station (TB/day) 3.1

Difference to conventional X-Ray experiments: High peak rate & large volume comparable to high-energy physics experiments such as BaBar @ SLAC

Challenge is to perform data-correction and image processing while keeping up with continuous incoming data-stream SLAC Particle Physics and Astro-Physics group involved has advantage since it has substantial experience acquiring and processing large data rates at high rates



Coherent Imaging of Single Molecules

• Diffraction from a single molecule:

single LCLS pulsenoisydiffractionpattern of unknown orientation

• Combine 105 to 107 measurements into 3D dataset:

Classify/sort Average AlignmentReconstruct by Oversampling phase retrieval

Miao, Hodgson, Sayre, PNAS 98 (2001)

unknown orientation

Gösta Huldt, Abraham Szöke, Janos Hajdu (J.Struct Biol, 2003 02-ERD-047)

The highest achievable resolution is limited by the ability to group patterns of similar orientation



Data System Architecture

Photon Control Data Systems (PCDS)Detector specific

Detector + ASIC FEE

Timing L0: Control

L1: Acquisition

L2: Processing L3: Data Cache

DetectorExperiment specificMay be bump-bonded to ASIC or integrated with ASIC

Front-End Electronics (FEE)Provide local configuration registers and state machinesProvide ADC if ASIC has analog outputsFEE uses FPGA to transmit to DAQ system

Beam LineData



Level 0 Nodes

Level 0: ControlDAQ operator consoles

Provide different functionalitiesRun control

Partition management, data-flow

Detector controlConfiguration (modes, biases, thresholds, etc)

Run monitoringData quality

Telemetry monitoringTemperatures, currents, voltages, etc

Manage all L1, L2 and L3 nodes in a given partition (i.e. the set of DAQ nodes used by a specific experiment or test-stand)



Level 1 Nodes

Level 1: AcquisitionReceive 120 Hz timing signals, send trigger to FEE, acquire FEE dataError detection and recovery of the FEE dataControl FEE parametersCalibration

Dark image accumulation and averagingTransfer curve mapping, gain calculationNeighbor pixel cross-talk calculation

Event-build FEE science data with beam-line dataImage processing

Pedestal subtraction using calibration constants, cross-talk correctionsPartial data reduction (compression)Rejection using 120 Hz beam-line dataProcessing envisioned both in software and firmware (VHDL)

Send collected data to Level 2 nodes over 10 Gb/s Ethernet



Level 2 & 3 Nodes

Level 2: ProcessingHigh level data processing:

Learn, pattern recognition, sort, classifye.g. combine 105 – 107 images into 3D data-set

Alignment, reconstruction

Currently evaluating different ATCA blades for L2 nodesSend processed data to L3 over 10 Gb/s Ethernet

Level 3: Data CacheProvide data storage

Located in server room in experimental hall

Off-line system will transfer data from local cache to tape staging system

Tape staging system located in SLAC central computing facilities

Must be able to buffer up data in local storage during downtimes of staging system



ATCA Crate

ATCAAdvanced Telecommunication Computing Architecture Based on backplane serial communication fabric

We use 10-Gigabit Ethernet2 custom boards

Reconfigurable Cluster Element (RCE) Module

Interface to detectorUp to 8 x 2.5 Gbit/sec links to detector modules

Cluster Interconnect Module (CIM)Managed 24-port 10-G Ethernet switching

One ATCA crate can hold up to 14 RCE’s & 2 CIM’s

Essentially 480 Gbit/sec switch capacityNaturally scalableCan also scale up crates

ATCA Crate

RCECIM



Reconfigurable Cluster Element (I)

The RCE is the most challenging among the different Level 1 node types

SLAC custom made ATCA boardUsed in other SLAC experiments

Based on System On Chip (SOC) Technology

Implemented with Xilinx Virtex 4 devices, FX familyXilinx devices provide

Reconfigurable FPGA fabricDSPs (200 for XC4VFX60)Generic CPU (2 PowerPCs 405 running at 450 MHz for XC4VFX60)PPC is choice for IP cores for next generation FPGA’sTEMAC: Xilinx TriMode Ethernet Hard CoresMGT: Xilinx Multi-Gigabit Transceivers 622Mb/s to 6.5Gb/s (16 for XC4VFX60)

RCE with RTM



Reconfigurable Cluster Element (II)

System Memory Subsystem512 MB of RAM Memory controller provides 8 GB/s overall throughput

Uses Micron RLDRAM II

Platform Flash Memory SubsystemStores firmware code for FPGA fabric

Configuration Flash Memory Subsystem

128 MB configuration flashDedicated file system for storing software code and configuration parameters (up to 16 selectable images)

Storage Flash Memory Subsystem (optional)

Up to 1TB per RCE persistent storage flash (currently 256GB per RCE)

Low latency/high bandwidth access through I/O channels using PGPUses Samsung K9NBG08 (32 Gb per chip)



RCE Software

SoftwarePorted open source Real-Time kernel

Adopted RTEMS: Real Time Operating Systems for Multiprocessor Systems

Written BSP mainly in C++Plus some C and assembly

Written 10Gb Ethernet driver and PGP drivers for bulk data1Gb management interface driverBuilt interface to RTEMS TCP/IP network stack Developed specialized network stack for zero-copy Ethernet traffic



Cluster Interconnect Module

ATCA network cardSLAC custom made boardBased on two 24-port 10Gb Ethernet switch ASICs from Fulcrum

Up to 480 Gb/s total bandwidth

Managed via Virtex-4 deviceCurrently XC4VFX12Fully managed layer-2, cut-through switchInterconnect up to 14 in-crate RCE boards (i.e. 28 RCEs)

Interconnect multiple crates for additional scalabilityFully configurable

Designed to optimize crates populated with RCE boards

Ability to use ATCA redundant lanes for additional bandwidth if desiredAbility to use 2.5Gb/s connections in place of standard 1Gb/s Ethernet

At the same time may be configured to connect standard ATCA blades



Experiment Front-End Board

Interfaces to detector ASICControl signalsRow/column clocksBiases/thresholdsAnalog pixel voltage

ContainsCommunication IP coreLocal configuration state machineLocal image readout state machine

Example: SLAC board FPGA with

MGT interfaces, up to 4 x 2.5 Gbit/sec fiber IO~ 200 digital IO VHDL programmedIncludes communication IP core provided by SLAC



CXI 2D-Detector Control and DAQ Chain

Ground-isolationVacuum

Fiber

Cornell detector/ASIC

SLAC FPGA front-end board

ATCA crate with SLAC DAQ Boards

Each Cornell detector has ~36,000 pixelsControlled and read out using Cornell custom ASIC

~36,000 front-end amplifier circuits and analog-to-digital convertersInitially 16 x 32,000-pixel devices, then up to 64 x 32,000-pixel devices

4.6 Gbit/sec average with > 10 Gbit/sec peak



Calibration & Distribution (using SLAC DAQ)



Noise (using SLAC DAQ)



XPP 2D-Detector Control and DAQ Chain

BNL XAMP Detector 1,024 x 1,024 arrayUses 16 each 64-channel FexAmps BNL custom ASICsInstantaneous readout: 4 ch x 20 MHz x 16bit= 20 Gbit/sec into FPGAOutput FPGA: 250 Mbytes/s at 120 Hz (1024x1024x2x120)

FexAmps proto-type ASIC has been received at SLAC and configuration and read out tests using SLAC LCLS DAQ system have begun

Detector

ASIC board with readout ASIC plus ADC’s

SLAC standard front-end board

Fiber

SLAC LCLS DAQ ATCA crate



DAQ Waveform Sampling Digitizer

Agilent Acqiris DC282 high-speed 10-bit cPCI Digitizer4 channels2-8 GS/s sampling rateAcquisition memory from 1024 kpoints to 1024 MpointsLow dead time (350 ns) sequential recording with time stamps6U PXI/CompactPCI standard, 64 bit, 66 MHz PCI bus

Sustained transfer rate up to 400MB/s to host SBC



High-Level Applications

To allow commissioners and users and of each experiment to:

Use a common interface to both the DAQ system and EPICSSpeed up the development cycle by using a high level programming language, but still be able to easily build critical sections in C/C++Easily develop new applicationsProvide a GUI integrated with the programming languageRe-use code developed by other LUSI experimentsPython as high level scripting language

Easy to learn, fast dev cycle, extensible, open-source, powerful, relatively fast

QT as graphical user interfaceFramework and support for scientists provided by PCDS



Online Archive

The online archive has a dual roleStore science and EPICS data for retrieval/monitoring/analysis by the online systemAllow DAQ and controls to keep operating during downtimes of the offline staging system

The archive size depends on average data rate and estimated downtime

Initially assumes:2 MB per image, 120Hz pulse rate, 30% success rate, 50% daily duty cycle: ~3.1 TB/day4 days estimated downtime offline staging system (eventually up to 7 days)

Will start with 12 TB going up to 20 TB before all 3 instruments are operatingMust be able to easily scale size to accommodate for larger detectors

Must be able to store initially > 250 MB/s



Online Archive Data Format

Online acquires data from instruments as C++ objectsEach class represents instrument data type or instrument configurationClasses might also describe processed instrument data or EPICS data needed for data analysis

Data written to disk native DAQ object oriented formatData stored in its memory representation

Classes designed to optimize high performance and self describing features

Minimize read/write operations needed to re-create or store an objectMaximize ability to adapt to changes in the data structures (eg number of pixels for a given detector) without introducing a new class



Archive and Offline System

Interface between the archive and the offline system made of two parts

Files staged on dedicated local disk 10 Gb/s link between NEH and SCCS for bulk data transferReplicated MySQL database used to maintain transfer state MySQL database in PCDS enclave to share meta-data information

Availability of a file, completion of a file copy operation, etc

1.6-526 Online/Offline ICD (Interface Control Document) released

Offline will store the data in HDF5 filesCompatible with NeXus standard for X-ray, neutron and muon data



File Management, Metadata

1.6-118 Offline Data Management System document released

File ManagementCentral file manager tracks all files [iRODS].High-performance parallel filesystem used for disk storage [Lustre].Tape system used for long-term archiving [HPSS].Network-based export interface.Disk-based (e-SATA, USB) export interface.

Meta DataScience metadata database contains user, run, instrument, and pulse attributes from online system.Additional user and run information replicated from electronic logbook.All metadata may be queried to locate files and portions of files of interest.Metadata is exported with data.



Analysis Options

NeXus API

Can use NeXus or HDF5 Tools and Analysis Packages

HDF5File API

Open Genie LAMP

GumTree Nathan

Redas Scilab Amortool More …

Open Source

Cactus Chombo dxhsf5

H5PartRoot HL-HDF

ParaView PyTables

VisAD

Many more …

Open Source

Commercial

IDL-HDF5 Matlab

Mathematica O-Matrix ViTables More …



Scientific Computing

Scientific Computing for LUSI ScienceOpportunities and needs are being evaluatedVery dependent on the detailed nature of the scienceUnprecedented size (for photon science) of data sets to be analyzed Unprecedented computational needs (for photon science)Comparable in scale to a major high-energy physics experiment

Greater need for flexibility than for a major high-energy physics experiment

Main scientific computing effort not part of baseline



Risk

RiskIF there are major changes in the scope, performance, existence or placement of CXI/XPP/XCS instrumentation due to evolving user requirements…THEN, it might be difficult to meet the schedule and budget as specified in P3

MitigationRelease Engineering Requirement documents

Already done

Adhere to BCR processLCLS requirement

Participate in Experimental Area design processAlready participating



Controls/DAQ Team Leaders

1.6. CAM: G. Haller Deputy (P. Anthony)Online (A. Perazzo)Controls (D. Nelson)DAQ (C. O’Grady)Infrastructure (R. Rodriguez)Offline Computing (S. Luitz)

Technical leaders are also responsible for AMO and XES-provided photon area controls/DAQ/infrastructure needed by LUSI

Provides low risk having interface issues, provides high efficiencyEnsures common solutionsNo issue with man-power, plus instruments are time-phased. Could accelerate LUSI controls, all driven by budget availability

ScientistXPP (D. Fritz)CXI (S. Boutet)XCS (A. Robert)Diagnostics/Common Optics (Y. Feng)Detectors (N. Van Bakel)

1.6. CAM: G. Haller Deputy (P. Anthony)Online (A. Perazzo)Controls (D. Nelson)DAQ (C. O’Grady)Infrastructure (R. Rodriguez)Offline Computing (S. Luitz)

Technical leaders are also responsible for AMO and XES-provided photon area controls/DAQ/infrastructure needed by LUSI

Provides low risk having interface issues, provides high efficiencyEnsures common solutionsNo issue with man-power, plus instruments are time-phased. Could accelerate LUSI controls, all driven by budget availability

ScientistXPP (D. Fritz)CXI (S. Boutet)XCS (A. Robert)Diagnostics/Common Optics (Y. Feng)Detectors (N. Van Bakel) CAM: Control Account Manager

CAM: Control Account Manager



Summary

All control and data systems requirements in LUSI Performance Execution Plan will be met with system presented for W.B.S. 1.6Technical, cost, and schedule risks are low

Well documented agreements with instrumentsRe-use of LCLS controls software, hardware where appropriateCost bottoms-up with detailed quotes for each componentSchedule fully linked and resource leveled

Data subsystem concept & architecture are well developedUse standard interface to all detectorsCXI and XPP/XCS detector ASICs are already being configured and read out using the LCLS DAQ systemData management system provides high bandwidth and is scalableLeverage significant expertise at SLAC in data acquisition and management

Ready to be approved for cost and schedule baseline

All control and data systems requirements in LUSI Performance Execution Plan will be met with system presented for W.B.S. 1.6Technical, cost, and schedule risks are low

Well documented agreements with instrumentsRe-use of LCLS controls software, hardware where appropriateCost bottoms-up with detailed quotes for each componentSchedule fully linked and resource leveled

Data subsystem concept & architecture are well developedUse standard interface to all detectorsCXI and XPP/XCS detector ASICs are already being configured and read out using the LCLS DAQ systemData management system provides high bandwidth and is scalableLeverage significant expertise at SLAC in data acquisition and management

Ready to be approved for cost and schedule baseline



END OF PRESENTATION