glast ground software chep 2007, victoria, daniel flath 1 the glast data handling pipeline...
Post on 06-Jan-2018
218 Views
Preview:
DESCRIPTION
TRANSCRIPT
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 1
The GLAST Data The GLAST Data Handling PipelineHandling Pipeline
““Pipeline II”Pipeline II”
Daniel Flath for the GLAST CollaborationStanford Linear Accelerator Center
Gamma-ray Large Gamma-ray Large Area Space Area Space TelescopeTelescope
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 2
OutlineOutline
Introduction to GLAST
Computing and GLAST
Pipeline Components
Processing Task Examples
Conclusion
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 3
GLAST Key FeaturesGLAST Key Features
• Huge field of view– LAT: 20% of the sky at any instant; in sky survey mode, expose all parts of sky for
~30 minutes every 3 hours. GBM: whole unocculted sky at any time.• Huge energy range, including band 10 GeV - 100 GeV• Will transform the HE gamma-ray catalog:
– by > order of magnitude in # point sources– spatially extended sources– sub-arcmin localizations (source-dependent)
Large Area Telescope (LAT)
GLAST Burst Monitor (GBM)
spacecraft partner: General Dynamics
Two GLAST instruments:LAT: 20 MeV – >300 GeVGBM: 10 keV – 25 MeV
Launch: Feb 5, 2008. Cape Kennedy 565 km, circular orbit 5-year mission (10-year goal)
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 4
GN
HEASARCGSFC
DELTA7920H
White Sands
TDRSS SNS & Ku
LAT Instrument Science
Operations Center
GBM Instrument Operations Center
GRB Coordinates Network
Telemetry 1 kbps
AlertsData, Command Loads
Schedules
Schedules
Mission Operations Center (MOC)
GLAST Science Support Center
GLAST Spacecraft
Large Area Telescope& GBM
GPS
GLAST MISSION ELEMENTS
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 5
e+ e–
Overview of LATOverview of LAT
• Precision Si-strip Tracker (TKR) 18 XY tracking planes. Single-sided silicon strip detectors (228 m pitch) Measure the photon direction; gamma ID.
• Hodoscopic CsI Calorimeter (CAL)Array of 1536 CsI(Tl) crystals in 8 layers. Measure the photon energy; image the shower.
• Segmented Anticoincidence Detector (ACD) 89 plastic scintillator tiles. Reject background of charged cosmic rays; segmentation removes self-veto effects at high energy.
• Electronics System Includes flexible, robust hardware trigger and software filters.
Systems work together to identify and measure the flux of cosmic gamma Systems work together to identify and measure the flux of cosmic gamma rays with energy 20 MeV - >300 GeV.rays with energy 20 MeV - >300 GeV.
Calorimeter
Tracker
ACD [surrounds 4x4 array of TKR towers]
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 7
Motivation: General Computing RequirementsMotivation: General Computing Requirements
• Data rate:– 400Hz total (5Hz Signal)– 12GB raw data, total, over 8 daily downlinks
• Prompt Processing:– Reconstruct downlink within 1hr of receipt– Provide results to collaboration, other missions for:
• Analysis, Monitoring • Flaring, bursting source detection• Follow-up observation, confirmation
– Result: 500GB reconstructed data per day
• Monte Carlo:– Ongoing throughout mission, ~100TB/yr
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 8
SLAC
SLAC
Pipeline ComponentsPipeline Components
PipelineServer
OraclePipeline Web Interface
Line ModeClient
Catalog WebInterface
IN2P3Job Control
BatchFarm
INFNJob Control
BatchFarm
Data Portal
CommandClient
Data Catalog
Oracle
SLACJob Control
BatchFarm
Client
Client
Client
ClientClient
Client
Client
read-only
read-only
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 9
Example: A “Simple” TaskExample: A “Simple” Task
getIntervalData
diffuseResponses liveTimeCube
launchRoiAnalysis
DRP_monitoring
roiAnalysisgetRoiData
exposureMap
combineExpMaps
sourceAnalysis
launchEnergyBandAnalysis
fitEnergyBand
energyBandAnalysis
fitEnergyBand
energyBandAnalysis
roiAnalysisgetRoiData
exposureMap
combineExpMaps
sourceAnalysis
launchEnergyBandAnalysis
drpExpMap
exposureSubMap
drpExpMap
exposureSubMap
drpExpMap
exposureSubMap
drpExpMap
exposureSubMap
Script Process Jython; Runs as thread inside Server Process. Launches Subtasks, sets variables, registers datasets with Data Catalog
SubTask Provides parallelism
Execution Condition
Batch Process Compute intensive: Dispatched to farm. Results collected via email.
Success, Failure, (all)-Completed
Sub-SubTask
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 10
The Real Deal: Downlink Prompt ProcessingThe Real Deal: Downlink Prompt Processing
Reconstruction
DigitizationM
erge
Merge
Reg
iste
r
Verify
Clean
Calibration
Monitoring
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 11
Usage Plots: Activity SummaryUsage Plots: Activity Summary
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 12
Usage Plots: Task Activity DetailUsage Plots: Task Activity Detail
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 13
Processing Plots: Single Process StatisticsProcessing Plots: Single Process Statistics
Maintain a record of:
•Total time (Wall Clock)
•Wait (Pending) time
•CPU time used
•Node name (machine class)
Expect this to be ~1.0 if our process is not I/O bound
CPU Factors show up
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 14
Pipeline Front End: Stream ViewPipeline Front End: Stream View
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 15
Technologies UsedTechnologies Used
• Database– Oracle– Java (and PL/SQL) Stored Procedures
• Servers and Client Libraries (Pipeline, Data Catalog)– Java
• Web:– Apache/Tomcat servers– JSP for web pages
• DisplayTag for tabular data• AIDA TLD for plotting• Custom TLDs expose Pipeline client methods
– Java Servlets• Serve GraphViz State diagrams
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 16
ConclusionConclusion
• Pipeline is currently in extensive use:– Monte Carlo Production – Integration and Test
• raw data processing• monitoring
• Software not coupled specifically to GLAST or SLAC– Could be used elsewhere– Additional batch farms incorporated by implementing simple Job
Control (Java)-Interface– Contact <pii-devlist@slac.stanford.edu> if you are interested
• To see GLAST Pipeline II, Data Catalog (and much more!) in action:– http://glast-ground.slac.stanford.edu
• For more on DisplayTag (plotting in web pages):– Victor Serbo: “JAIDA, JAS3, WIRED4 and the AIDA tag library” --
EP7, Carson Hall A, Thursday August 6 17:30
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 17
AcknowledgmentsAcknowledgments
Software Development Team:Daniel FlathCharlotte HeeKaren HeidenreichTony JohnsonIgor PavlinMax Turri
Brave Users:Warren FockeTom Glanzman
GLAST Ground Software
CHEP 2007, Victoria, Daniel Flath 18
Supplemental: GLAST Data CatalogSupplemental: GLAST Data Catalog
top related