december 16, 2002 1 noao mosaic pipeline codr noao mosaic pipeline technical presentation

December 16, 2002December 16, 2002 11NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR

NOAO Mosaic PipelineNOAO Mosaic Pipeline

Technical PresentationTechnical Presentation

December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 22

Outline of Technical PresentationOutline of Technical Presentation

• Introduction

• Contexts

• Capabilities

• Architecture

• Implementation


Presentation GoalsPresentation Goals

Convince you that:• we understand the

– problem– requirements– resources– components

• and that the project– is feasible– has a solution for the primary application– has a flexible design for expansion and wider application


Guiding PrinciplesGuiding Principles

• Modest project

• Part of Data Products Program

• (NOAO) Mosaic Imaging Data

• Dedicated pipeline


Principles: Modest ProjectPrinciples: Modest Project

• Reuse as much software as possible

• Keep it simple software


Principles: DPPPrinciples: DPP

• MDHS: Mosaic Data Handling System

• IRAF: Image Reduction and Analysis Facility

• NSA: NOAO Science Archive

• DTS: Data Transport System

• OPUS: AURA sister institution (STScI)

• GONG: AURA sister institution (NSO)


Principles: (NOAO) Mosaic DataPrinciples: (NOAO) Mosaic Data

• Use experience of Mosaic Survey Teams

• Need to deal with specific peculiarities– Crosstalk, pupil reflections

• Allow for high performance per exposure (for real-time telescope context) by capitalizing on the inherent data parallel nature of mosaic imaging data


Principles: Dedicated PipelinePrinciples: Dedicated Pipeline

• Network of similar computers

• No competition with general users


What does this project encompass?What does this project encompass?

• Pipeline infrastructure• CCD mosaic data reduction• Data quality assessment• Image differencing• Catalog production• Database entry and querying• Source merging/classification

• Archive ingest and retrieval• Alerts• Monitoring• Data transport• High performance computing• Parallel computing• More …

Algorithms, interfaces, and software for:


ContextsContexts

In what contexts will the pipeline run?In what contexts will the pipeline run?

Can we design a pipeline to satisfy multiple contexts?Can we design a pipeline to satisfy multiple contexts?


ContextsContexts

• NOAO– Telescope/operational context– Archive/NVO context

• Community– NOAO Mosaic surveys and observers– Other mosaic instruments


PrioritiesPriorities

1. NOAO Archive

2. NOAO Mosaic observers− telescope− downtown− home institution

3. NOAO Mosaic observers at home

4. Community


NOAO ContextsNOAO Contexts

• Downtown center fed from telescope

• Mountain at telescope

• Archive on-the-fly reprocessing


Pipeline LocationsPipeline Locations

La SerenaArchive

TucsonArchive

Kitt Peak Cerro Tololo

Pipeline

Pipeline Pipeline

Pipeline


Context: Downtown PipelineContext: Downtown Pipeline

ObserverDCA

Data Spooland Transport

Pipeline

DSC

User @telescope,downtown,

home

Archive

DTS


Context: Mountain PipelineContext: Mountain Pipeline

DCAData Spool

and TransportPipeline

User @telescope

Archive

DTS


Context: Archive PipelineContext: Archive Pipeline

User @home

Pipeline

Archive

DTS


Context: User PipelineContext: User Pipeline

User @home

Pipeline@ home

More


Proposed ContextProposed Context

• Downtown pipeline for NOAO archive

• Observer may subscribe to data products– At telescope, downtown, home– Images, catalogs, alerts, …

• Observer may connect to DQ monitors• Pipeline software available at telescope with

minimal support• DQ task/monitors may run at telescope


Data RequirementsData Requirements

The pipeline design is dependent on the information available about the input data.

• Basically we require data with the current NOAO Mosaic readout format that includes:– identification of exposure type (object, etc)– description of regions (data, overscan)– an approximate world coordinate system


Data RequirementsData Requirements

There may be additional information that the pipeline will use if present.

• Associations: type, ID, total and index SEQUENCE = ‘zero2002-12-16T043244.20.3’

SEQUENCE = ‘dither2002-12-16T043244.5.2’

If not present heuristics will be used based on a requirement that data enters in time order


CapabilitiesCapabilities

• Capabilities

• Major Features and Goals

• Data Products– Basic– Advanced

• Data Quality Assessment

• Instrumental Calibration


CapabilitiesCapabilities

• Calibrate mosaic exposures• Update instrumental calibrations• Identify potential bad data (data quality assessment)• Monitor trends and maintain database• Stack dither sets• Catalog and classify objects and artifacts• Get and subtract reference image and detect sources• Identify interesting sources• Automatically provide data products to subscribers• Keep up with observing given sufficient CPU resources


Major Features and GoalsMajor Features and Goals

• Data products for NOAO archive and NVO node• Data products for observers (by subscription)• Pipeline for NOAO and mosaic community• Basic CCD mosaic calibrations• Advanced time-domain data products• Real-time data quality assessment and monitoring• High performance, data parallel system• LSST testbed• Fairly generic pipeline infrastructure (NEWFIRM, …)• Automated operation• Thorough processing history and data documentation


Data Products: BasicData Products: Basic

• Instrument calibrated mosaic exposures• Rough photometric zero point• Astrometric calibrations• Data quality evaluations• Updated calibrations• Bad pixel, saturated, bleed trail masks• Object catalogs• Object masks• Observing logs• Processing information

– logs– graphs


Data Products: AdvancedData Products: Advanced

• Dither stacks• Exposure masks• Field Catalogs• Difference image detections

– Relative to dither stack– Relative to archive or catalog reference

• Light curves• Variable object detections• Unusual object alerts• Moving object trajectories


Data Quality AssessmentData Quality Assessment

Instrument• Telemetry• Crosstalk• Overscan• Bias, flat• Noise• Focus / Distortions

Sky• Seeing (PSF)• Sky brightness• Approx. zero point• Twilight• Moon up / distance

Data quality measures are monitored against preset and user limits as well as adaptive time series limits. Some quantities include mean, sigma, and spatial variations.


Instrumental CalibrationsInstrumental Calibrations

• Crosstalk [1]

• CCD defects [2,4,5]

• Saturated pixels [2,4,5]

• Bleed trails [2,4,5]

• Cosmic rays [2,4,5]

• WCS update [3]

1. Requires image data from full mosaic (non-parallel)

2. Each image element independent of others (parallel)

3. Global calculation on measurements images (parallel and non-parallel)

4. Interpolate in data

5. Flag in mask

• Overscan [2]

• Bias [2]

• Flat field [2]

• Pupil pattern [3]

• Fringing [3] • Approx. zero point [3]


Instrumental CalibrationsInstrumental Calibrations

Two-pass calibration for telescope context:

1. Nighttime pass for immediate and nearly complete calibrated exposures

2. Daytime pass for calibration update from the full night’s data set


Nighttime PassNighttime Pass

• Perform standard CCD calibrations:– Use afternoon master bias– Use most recent flat field

• Apply pupil and fringe correction– Use most recent pupil and fringe templates

• Apply global coordinate calibration


Daytime PassDaytime Pass

• Determine if night’s data is suitable for deriving updates to library calibrations

• Derive new pupil, fringe, and sky flat calibrations• Evaluate changes and significance of new

calibrations• Update library calibrations for next night• Update night’s exposures with new calibrations• Combine afternoon biases into new master bias• Combine afternoon dome flats if no library flat


Other ContextsOther Contexts

• For archive data will either already have best calibration from library or will be derived by requesting raw data for night

• At home or in the community raw data will be queued as at telescope

• Documentation and support (data ingest applications) will be provided


Data Products SubscriptionData Products Subscription

• Capability of the DPP system– Not necessarily specific to the pipeline but

requires interfacing with DTS

• Allows external software to request notification of new data products

• Allows flexibility and broader access– Has implications for the pipeline context


ArchitectureArchitecture

• What is a pipeline?

• Mosaic Pipeline Architecture Concept

• Pipeline Components– Controls and Monitors– Modules– Calibrations and Database (Rafael Hiriart)– Archive (Robyn Allsman)


What is a Pipeline?What is a Pipeline?

System to transform input data to output data

• Automated

• Composed of processing steps (modules)

• Steps connected by rules (triggers)

• Provides monitoring and alerts

• Error tolerant (continue with next input data)


Mosaic Pipeline Architecture ConceptMosaic Pipeline Architecture Concept

• Multiple CPUs but no dependency on N

• Multiple types of sub-pipelines by function– One for operations over all mosaic elements– One for operations on individual elements– One for cataloging– One for image differencing

• All types on all CPUs: no master!

• Sub-pipelines triggered by files


• All CPUs with identical pipeline software, possibly on common NFS disk

• Assign work by minimum data backlog• Transfer data to local CPU disk: not NFS!

– Optimize by modules writing to next trigger directory

• Controls connected to operator console• Monitors viewed via network by multiple

parties

Mosaic Pipeline Architecture ConceptMosaic Pipeline Architecture Concept


Network of Sub-pipelines and CPUsNetwork of Sub-pipelines and CPUs

Pipeline

CPUCPU

CPU

CPU

CPU

MEF

SIF

SIF

MEF

SIF

MEF

MEFCPU

SIF

MEF

SIF

MEF

SIF

MEF: pipeline for operations over all mosaic extensions; eg crosstalk, global WCS correction

SIF: pipeline for single CCD images; eg ccdproc, masking


Data Flow ConceptData Flow Concept

Last module in one pipeline writes output directly to the data directories of the host for next pipeline, with the host selected by having the minimum number of waiting data files.


Data Flow AlgorithmData Flow Algorithm

• Search list of potential hosts:– Check if host is up– Check number of trigger files– Assign output filename to data directory of

host with least number of data files– Network filenames are used: (eg. host!

directory/filename

• Module runs and writes output files


Data Flow NetworkingData Flow Networking

• Use a daemon automatically spawned the first time data is transferred to a host

• Daemon provides portability across platforms; eg. Unix and VMS


Data Flow Networking: ExampleData Flow Networking: Example

• Crosstalk input is Obj123.fits with 2 extensions• Output names are generated from Host.dat:

– Host1 has two waiting files, Host2 has one, Host3 is down, Host4 has none

– Host2!Obj123.1, Host4!Obj123.2

• Crosstalk module runs and writes output files directly to the hosts

• There are no extra network copy or splitting steps


Data Flow Networking: ExampleData Flow Networking: Example

Host0:

Crosstalk

Host1:

Obj456.1

Obj321.2

Host2:

Obj567.2

Host3:

Obj123Obj123.2

Obj123.1Host3!Obj123.1

Host2!Obj123.2

Host4:

DOWN


Pipeline ComponentsPipeline Components

Data Source(DTS, user)

Pipeline

Controls & Monitors

Calibrations &Databases

Data Sink(DTS, user)

raw

data

data

productsModule Module


Pipeline ModulesPipeline Modules

Pipeline

Module Module Module

CLSH APICSH


Data Parallel ModulesData Parallel Modules

Some algorithms may need to be (re-)implemented specifically for a data parallel pipeline.

One type is where measurements are made across the mosaic for a global calibration.

Rather than requiring all pieces to be in one pipeline arrange for measurements made in parallel to be collected for the global calibration and then apply the global calibration to the pieces in parallel.


Data Parallel ModulesData Parallel ModulesWCS ExampleWCS Example

• Catalog objects in each CCD in parallel• Bring catalogs (not images) together

– Only need x/y coordinates of brighter stars

• Match sources to ref. catalog (eg. USNO)• Compute global correction (shift, scale, etc.)• Return correction coefficients to parallel

pipelines to be applied to each CCD• Cataloging and correction stages can be

separated and run asynchronously with other stages


Data Parallel ModulesData Parallel ModulesFringe/Pupil ExampleFringe/Pupil Example

• Determine best global scaling of pupil and fringe templates to each exposure and then subtract scaled template– Compute statistics over each CCD in parallel– Combine statistics to get global scale factor– Subtract template with global scale from each

CCD in parallel


Pipeline TriggersPipeline Triggers

• Files: trigger on appearance of files• Flags: trigger on particular set of flags• Timers: trigger at times or intervals• File contents: trigger on keywords, etc• Messages: trigger on messages• Resources: trigger on resources

May be more but one type can mimic others


Pipeline TriggersPipeline Triggers

• File triggers useful for initiating a pipeline

• Flag triggers useful within a pipeline to communicate success of previous steps

• Flag triggers also useful for waiting for completion of parallel steps

• Timer triggers useful in telescope pipeline for performing different daytime/nighttime steps


Pipeline

DataDirectory

TriggerDirectory

Module

obj123.fits obj123.trig GO

File TriggersFile Triggers

May containinformation

such asoutput path

More


Pipeline

TriggerDirectory

DataDirectory

obj123a.trigobj123b.trigobj123c.trigobj123a.fitsobj123b.fitsobj123c.fits

--------

d------

dd----

ddd--

ddddobj123d.fits obj123d.trig

Module

GO

Flag Triggers and MergingFlag Triggers and Merging

Module

GO


Timer Triggers and Two-PassesTimer Triggers and Two-Passes

• Nighttime pipeline runs and leaves data in starting directory for daytime pipeline

• Daytime pipeline is triggered at end of night by timer


Controls & MonitorsControls & Monitors

Pipeline

Process Manager

Obs. Manager

Status Monitor

Keyword Monitor

Module

ToDatabase


Data ManagerData Manager• Interacts with the pipeline, operator, and potentially other

parts of the system such as archives or external applications• Record

– New calibrations from pipeline or operator– New parameters from operator– Processing information from pipeline

• Responds to queries for – Calibrations– Parameters– Processing history– Documentation and reports for data products


Data Manager ArchitectureData Manager Architecture


What do we want to store in the What do we want to store in the database?database?


Where is Data Manager?Where is Data Manager?


CalibrationsCalibrations

• The Data Manager responds to requests from pipeline for current calibration for a particular date, filter, etc.

• Updates calibrations produced by pipeline (or externally) for a particular date, filter, etc. Calibration updates may require operator confirmation.

• Calibrations include– Biases and flat fields– Pupil and fringe templates– Standard star data– Astrometry coordinates

• Some queries are satisfied through secondary queries to other databases such as USNO, GSC2, Landolt, etc.


ParametersParameters

• Responds to requests from pipeline for current parameters for– Pipeline module– Observation date, filter, exposure type, etc.– Position on sky

• Updates parameters supplied by operator


Processing InformationProcessing Information

• All information produced by the pipeline is recorded (keyed by a data identifier). This includes all the information provided to the keyword monitor as well as other data processing sources (logs, graphics, etc.)

• Pipeline requests processing information for a pipeline execution packaged as an associated data product for the archive.

• The operator can query processing information for diagnostic purposes.


ReportsReports

• Produces reports for a particular data product• Documentation is created from processing information

according to some template and desired format (eg xml, html)


Pipeline/Archive Ingest InterfacePipeline/Archive Ingest Interface

• Desirable traits

– Independence of database semantics– Use of self-describing data description

standards– Hiding data’s physical location


Archive IngestArchive Ingest

Who, What, WhereAuthority

Payload

DataReceiving

Agent

Data Store

Archive Ingest

Manager


Strawman ImplementationStrawman Implementation


Things We Looked At / Aware OfThings We Looked At / Aware Of

• Macho pipeline• SM/SN pipeline• Sloan pipeline• Pan-Starrs: IMCAT, Vista• IRAF: Core, IMRED pipelines, STSDAS, PYRAF, etc• MIDAS: Mosaic Imager Data Archive System• Linda and descendants• Elixar (CFH), Terapix (CFH), Subaru, ESO WFI, INT WFI• Condor / PVM / NOAO message bus• Opus pipelines: HST. MSSO, GONG• Databases: MySQL, Postgres


Software and SystemsSoftware and Systems(Blue Ribbon)(Blue Ribbon)

• OPUS• IRAF System

– CLSH (enhanced), KI, OBM/GUI

• IRAF Tasks– MSCRED, ACE

• SM/SN Alard/Lupton Algorithm• POSTGRES• DTS• NSA


Software and SystemsSoftware and Systems(Honorable Mention) (Honorable Mention)

• PVM

• Condor

• Other scripting languages and systems– PYRAF and Python– Perl– MLCL


Pipeline ModulesPipeline Modules

Pipeline

Module Module Module

CLSH OAPICSHMSCRED, etc


Controls & MonitorsControls & Monitors

Pipeline

Process Manager

Obs. Manager

Status Monitor

Keyword Monitor

Opus

Opus

IRAF GUI

IRAF GUI

OpusModule


Switchboard ServerSwitchboard Server

CPU

Pipeline

ModuleModule

Pipeline

ModuleModule

CPU

Pipeline

ModuleModule

Pipeline

ModuleModule

Switchboard

Server

Backup

Keyword

Monitor

Status

Monitor

Database

Manager

Other types

Or instances

Switchboard address set by environment variable


TriggersTriggers

OPUS provides:

• Files: trigger on appearance of files– Data entry pipeline initiation

• Flags: trigger on “blackboard” flags– Internal sequencing of modules– Parallel to Global sequencing

• Timers: trigger at certain times or intervals– Nighttime/Daytime Two-Pass Control


Monitoring IRAF TasksMonitoring IRAF Tasks

• IRAF tasks, including scripts, will open a messaging connection and write status and monitor information

• Minimal changes will be required to tasks• If a server is not running or disappears the tasks

will continue to run with output spooled locally



• Initially the broadcasting will be a socket connection with a server that multiple clients may connect to for rebroadcast

• The monitor tasks are IRAF GUI tasks which provide flexibility for changes to the GUI or functionality



• The GUI monitors will include:– Adaptive alarms– Adaptive heartbeat monitoring– Advanced graphics


IRAF Keyword Monitor PrototypeIRAF Keyword Monitor Prototype


IRAF Status Monitor PrototypeIRAF Status Monitor Prototype


NOAO Mosaic PipelineNOAO Mosaic Pipeline

Development Plan

1. Basic Calibration Pipeline

2. Advanced Time-Domain Pipeline


1. Basic Calibration Pipeline1. Basic Calibration Pipeline

• Basic single exposure calibrations• Data quality assessment and monitoring• High-performance pipeline infrastructure• Simple data transport system• Connection to the NOAO Science Archive


2. Advanced Time-Domain Pipeline2. Advanced Time-Domain Pipeline

• Catalogs• Image difference detections• Multiple detection ident. and merging• Time series• Alerts• Archiving of new data products


Timeline TargetsTimeline Targets

• Test version of basic calibration pipeline– July 2003

• Operational– September 2003

• Test version of time-domain pipeline– July 2004

• Operational– September 2004


Work BreakdownWork Breakdown

• Pipeline

• Monitors

• Data Manager

• Input and Output

• Data Products

• Archive


• Pipeline– Define methods for running IRAF tasks in OPUS

• Parameters• Error handling• I/O

– Define and verify data flow balancing method– Define, develop, and implement DQ methods– Develop data parallel algorithm steps for

• WCS• Fringe/pupil removal

– Develop data parallel OPUS architecture– Setup development system of at least two machines

Work Breakdown


• Monitors– Develop status monitor

• Experiment with different GUI formats

– Develop keyword monitor• Experiment with different GUI formats

– Develop switchboard server

Work Breakdown


Work Breakdown

• Data Manager– Define interfaces

• Pipeline• DBMS• NVO/web services• External clients

– Define database structures– Define archive data products– Design processing reports– Design calibration library storage and methods– Design and implement manager application

• Include GUI monitor and operator interface

– Install and configure DBMS


• Input and Output Services– Contribute to DTS– Implement interim data transport, staging, and

queuing

Work Breakdown


• Archive– Contribute to NSA development of automatic

ingest– Adjust data product specification to include

NSA requirements

Work Breakdown


• Specify Data Products– File types– Headers– Documentation

Work Breakdown


Implementation PlanImplementation Plan

• It is important to deliver core functionality quickly• Some technologies are new (to the development

team)• Delivery timeframe is short

This project will proceed using a Phased Delivery lifecycle development model –– i.e., phases of detailed design-develop-test as core functionality is built up. This is especially appropriate when:


Management PlanManagement Plan

Key elements of the management plan are:• Management/staffing• Work Breakdown [covered by FV]• Schedule• Risk Management


PersonnelPersonnelStaff

MemberRole

Allocation

Responsibilities

Dick Shaw Project Manager 5% Schedule development, resource planning

Frank Valdes Team Lead 40%Allocation of work, tracking technical progress, lead designer, documentation

Chris Smith Project Scientist 10%Definition of requirements, use cases, verification & validation, documentation

Rafael Hiriart S/W Engineer 25%Database & infrastructure design & development, use case development

Robyn Allsman S/W Sys. Eng. 5%Archive interface definition, archive system updates, data storage planning, consultant

F. Pierfederici Scientific Progr. 30% Implementation, testing

TBD (U. MD) S/W Engineer 50% Implementation, testing


Staffing ProfileStaffing Profile

0%

20%

40%

60%

80%

100%

120%

140%

160%

180%

2002Q4

2003Q1

2003Q2

2003Q3

2003Q4

2004Q1

2004Q2

U.MD

Pierfederici

Allsman

Hiriart

Smith

Valdes

Shaw


Risk ManagementRisk Management

• Heavily matrixed staff– New staff will also off-load other work from team lead

• Staff distributed across continents & institutions– Project leadership remains in Tucson– Extended visits by new remote staff– Weekly videoconferences

• New staff has limited experience in problem domain– Project leader to work closely with new staff

• Use of new/third-party software– Make effective use of expertise from external partners

december 16, 2002 1 noao mosaic pipeline codr noao mosaic pipeline technical presentation

Documents

data productsat telescope

noao archiveobserver

data transport systemopus

noao science archivedts

realtime telescope context

archive pipelineuser

user pipelineuser

multiple contexts