1 anasac meeting – may 20, 2015 alma pipeline brian glendenning (for jeff kern)
TRANSCRIPT
1 ANASAC Meeting – May 20, 2015
ALMA Pipeline
Brian Glendenning (for Jeff Kern)
2 ANASAC Meeting – May 20, 2015
ALMA PipelineStatus
• Pipeline provisional acceptance May 2014– Extensive punch list primarily on weblog and
presentation issues
• Calibration pipeline put into operation Oct 2014– Detailed parallel testing by ARC staff prior to
operations
We now have approximately 6 months of operational experience with the calibration pipeline
3 ANASAC Meeting – May 20, 2015
ALMA PipelinePipeline Performance and Use
Cycle 1 Cycle 2
4 ANASAC Meeting – May 20, 2015
ALMA PipelinePipeline Performance
Success
Success w/ Flagging
Success w/ Intervention
Failed
Cycle 1
Cycle 2
5 ANASAC Meeting – May 20, 2015
ALMA PipelineManual Flagging by Reason
Outlier AmplitudeBad AntennaOutlier PhaseNoisy PhaseEdgesAntenna PositionTsysOtherNo PhaseCalNoisy AmplitudeBad RawdataDelayQuack
6 ANASAC Meeting – May 20, 2015
ALMA PipelinePath Forward
Internal Only
• Cycle 3 R1 (Jan 2015):– Cleanup of capabilities and feedback from early cycle2– Infrastructure for automation– Bandwidth mapping for low SNR issues
• Cycle 3 R2 (~May 29) – Imaging Mechanics– Minor flagging changes– Infrastructure for Sessions
• Pipeline Face 2 Face: June 16-18– Pipeline WG (ISOpt) and Developers– Priority for imaging improvements– Total Power Pipeline Commissioning Status and Plans
7 ANASAC Meeting – May 20, 2015
ALMA PipelinePath Forward (Continued)
• Cycle 3-R3 (Sept 2015)– Pipeline for use for cycle 3– Including initial imaging– Increasing levels of testing leading to parallel
testing of imaging and full acceptance.
8 ANASAC Meeting – May 20, 2015
ALMA PipelinePlanning Post Cycle 3– Software engineering issues• Automated testing• Documentation
– Expanded Scope• Long Baselines• Imaging Improvements• Sessions• Band Switching Calibration• Low calibrator SNR Heuristics• Polarization
9 ANASAC Meeting – May 20, 2015
ALMA PipelineSteady State• New capabilities initially offered in non-standard mode
– Provides reasonable sample of data sets– Manual data reduction provides “best practice” heuristics– Comparison to manual reduction for pipeline validation.
• Move to standard mode in following cycle– Depends on how standard the mode is
• VLBI, Solar, Time Domain may never become standard
• Feedback from Data Reduction and Pipeline Operations– Improve heuristics and coverage
10 ANASAC Meeting – May 20, 2015
ALMA PipelineRisks
• Image quality– Diverse expectations – Must deliver some capabilities to relive data
reduction load on ARCs
• Sociology– Pipeline produced science products are new to
radio interferometry, community acceptance will be mixed.
– Errors in early days will have long lasting effects on confidence in the pipeline
11 ANASAC Meeting – May 20, 2015
ALMA Data Mining Toolkit (ADMIT)Update: May 11, 2015
Milestones #1 and #2 successfully completed. Task and data i/o infrastructure Work flow management (allows scripting of ADMIT tasks) ADMIT tasks: ingest, file, cubestats, cube spectrum, line-id, linecube, moment Automated build and regression testing Data browser prototype (url: carma.astro.umd.edu/admit-M2) On-line documentation: design, code docs, and user manuals
(url: carma.astro.umd.edu/admit) Prototype pipeline for deployment on ALMA pipeline machines Tests performed on Cycle 0 and Cycle 1 data
Milestone #3: July 31, 2015 Production of ADMIT pipeline data products on ALMA machines Ingestion of data products into archive Broad testing against Cycle 1 and 2 data Expansion of ADMIT tasks and exploration of new algorithms
Creates data products for the archive which can be viewed and manipulated by the user. Data products can also be generated on local datasets using ADMIT/casa python environment. Data products are compatible for use by future ALMA archive enhancements
12 ANASAC Meeting – May 20, 2015
ALMA Data Mining Toolkit (ADMIT)
Prototype ADMIT Data browser view displaying Cycle 1 NGC 253 data
Chose web browser interface to minimize barrier for user adoption.
Built on same browser resources as ALMA Pipeline Viewer to maintain consistent look and feel for user.
Automatically updates as new data products are created locally.
Allows organization data products by source using tabs
Mitigates cross-platform incompatibilities that could arise from a custom GUI
13 ANASAC Meeting – May 20, 2015
www.nrao.eduscience.nrao.edu
The National Radio Astronomy Observatory is a facility of the National Science Foundation
operated under cooperative agreement by Associated Universities, Inc.