Distributed Pipeline Distributed Pipeline Programming for MosaicsProgramming for Mosaics

OrOr

Mario Tips’N’TricksMario Tips’N’Tricks

NOAO Mosaic PipelineNOAO Mosaic Pipeline

Major Features and GoalsMajor Features and Goals

Data products for NOAO archive and NVO nodeData products for NOAO archive and NVO node Data products for observersData products for observers Pipeline for NOAO and mosaic communityPipeline for NOAO and mosaic community Basic CCD mosaic calibrationsBasic CCD mosaic calibrations Advanced time-domain data productsAdvanced time-domain data products Real-time data quality assessment and monitoringReal-time data quality assessment and monitoring High performance, data parallel systemHigh performance, data parallel system LSST testbedLSST testbed Fairly generic pipeline infrastructure (NEWFIRM, …)Fairly generic pipeline infrastructure (NEWFIRM, …) Automated operationAutomated operation Thorough processing history and data documentationThorough processing history and data documentation

MARIOMARIOMMosaic osaic AAutomatic utomatic RReduction eduction IInfrastructure and nfrastructure and OOperationsperations

(i.e. a pipeline)(i.e. a pipeline)

Key Concepts (Tips’N’Tricks)Key Concepts (Tips’N’Tricks) sub-pipelines - “meta pipeline sub-pipelines - “meta pipeline

programming”programming” indirect filesindirect files load balancing using trigger filesload balancing using trigger files stay-alive modulestay-alive module parallelization of algorithms over mosaicparallelization of algorithms over mosaic shared monitoringshared monitoring network filenamesnetwork filenames image processing language (CL)image processing language (CL)

What is a pipeline?What is a pipeline?

collection of processing modulescollection of processing modules connected by dependency rulesconnected by dependency rules modules may run concurrently on modules may run concurrently on

different data objectsdifferent data objects

Infrastructure to manage processesInfrastructure to manage processes Infrastructure to manage dependenciesInfrastructure to manage dependencies Infrastructure to monitor processes and Infrastructure to monitor processes and

processingprocessing

OPUSOPUSOOperations perations PPipeline ipeline UUnified nified

SSystemystem

OpusOpus

Triggers (dependency rules)Triggers (dependency rules) file, osf, timefile, osf, time

BlackboardBlackboard PollingPolling

Monitors and ManagersMonitors and Managers

Distributed Pipeline IssuesDistributed Pipeline Issues

data vs. functional parallelismdata vs. functional parallelism shared file system vs. local file systemshared file system vs. local file system heterogeneous vs. homogenous heterogeneous vs. homogenous

processorsprocessors parasitic processingparasitic processing push vs. pull push vs. pull load balancingload balancing master-worker vs. peer-to-peermaster-worker vs. peer-to-peer

MARIO ChoicesMARIO Choices

data parallelismdata parallelism local file system (w/ shared blackboard)local file system (w/ shared blackboard) heterogeneous processorsheterogeneous processors push AND pull push AND pull load balancing by number of data load balancing by number of data

objectsobjects peer-to-peerpeer-to-peer

MARIO Architecture ConceptMARIO Architecture Concept

Multiple CPUs but no dependency on NMultiple CPUs but no dependency on N Multiple types of sub-pipelines by functionMultiple types of sub-pipelines by function

One for operations over all mosaic elementsOne for operations over all mosaic elements One for operations on individual elementsOne for operations on individual elements One for catalogingOne for cataloging One for image differencingOne for image differencing

All types on all CPUs: no master!All types on all CPUs: no master! Sub-pipelines triggered by filesSub-pipelines triggered by files

““Meta Pipeline Meta Pipeline Programming”Programming”

Build a pipeline out of sub-pipelinesBuild a pipeline out of sub-pipelines Form a distributed web of sub-pipelinesForm a distributed web of sub-pipelines Sub-pipelines play role of subroutinesSub-pipelines play role of subroutines

Need equivalents of:Need equivalents of: objectsobjects call and returncall and return node assignmentnode assignment library of standard moduleslibrary of standard modules

start, call, return, done, obs, runstart, call, return, done, obs, run

What is a sub-pipelineWhat is a sub-pipeline

primarily operates on one type of primarily operates on one type of objectobject

operates on one nodeoperates on one node data is maintained locallydata is maintained locally multiple stages but limited multiple stages but limited

functionalityfunctionality

Example of Sub-pipelinesExample of Sub-pipelines

NGTCAL

SCL

MEF

SIF

DTS

multiextension

single images

Sub-pipelinesSub-pipelines

NGT: Nights worth of dataNGT: Nights worth of data Group, Zero, Dome Flat, Objects, DoneGroup, Zero, Dome Flat, Objects, Done

CAL: Calibration sequence (MEF)CAL: Calibration sequence (MEF) Setup, Split, DoneSetup, Split, Done

SCL: Calibration sequence (SIF)SCL: Calibration sequence (SIF) Setup, CCDPROC, Combine, DoneSetup, CCDPROC, Combine, Done

MEF: Process objects (MEF)MEF: Process objects (MEF) Setup, Split, DoneSetup, Split, Done

SIF: Process objects (SIF)SIF: Process objects (SIF) Setup, CCDPROC, DoneSetup, CCDPROC, Done

Network of Sub-pipelines and Network of Sub-pipelines and CPUsCPUs

Pipeline

CPUCPU

CPU

CPU

CPU

MEF

SIF

SIF

MEF

SIF

MEF

MEFCPU

SIF

MEF

SIF

MEF

SIF

MEF: pipeline for operations over all mosaic extensions; eg crosstalk, global WCS correction

SIF: pipeline for single CCD images; eg ccdproc, masking

Example Processing StatusExample Processing StatusOBJECT NAME PIPELINE NODE STAGESOBJECT NAME PIPELINE NODE STAGES

anight1 ngt dhcp-4-152 cccw_ct4m20030102T183424S cal dhcp-4-152 cccd_ct4m20030102T183424S_01 scl archive2 ccccdct4m20030102T183424S_02 scl dhcp-4-152 ccccdct4m20030102T183424S_03 scl archive2 ccccdct4m20030102T183424S_04 scl dhcp-4-152 ccccdct4m20030102T191558S cal dhcp-4-152 cccd_ct4m20030102T191558S_01 scl archive2 ccccdct4m20030102T191558S_02 scl vmware ccccdct4m20030102T191558S_03 scl archive2 ccccdct4m20030102T191558S_04 scl dhcp-4-152 ccccdct4m20030103T084044 mef dhcp-4-152 ccw__ct4m20030103T084044_01 sif archive2 ccd__ct4m20030103T084044_02 sif archive2 cp___ct4m20030103T084044_03 sif vmware p____ct4m20030103T084044_04 sif archive2 _____ct4m20030103T084307 mef dhcp-4-152 cccd_ct4m20030103T084307_01 sif archive2 ccd__ct4m20030103T084307_02 sif vmware ccd__ct4m20030103T084307_03 sif archive2 ccd__ct4m20030103T084307_04 sif archive2 ccd__

Calling a Sub-pipelineCalling a Sub-pipeline

Data is setup either locally or on target Data is setup either locally or on target nodenode

File with path for returned result File with path for returned result written to target pipelinewritten to target pipeline

File with paths of returned results File with paths of returned results written in calling pipelinewritten in calling pipeline

Trigger file written to target pipelineTrigger file written to target pipeline

Returning ResultsReturning Results

Return module in target pipeline Return module in target pipeline looks for return filelooks for return file

Results are written to trigger file for Results are written to trigger file for calling pipeline specified in the return calling pipeline specified in the return filefile

Calling pipeline triggers on return fileCalling pipeline triggers on return file

Call/ReturnCall/Return

A -> HA -> HNN!B/data/abc!B/data/abcNN (derived from abc)(derived from abc)

A -> HA -> HNN!B/return/abc!B/return/abcNN [H!A/abc [H!A/abcNN.btrig].btrig]

A -> H!A/abc.b [abcA -> H!A/abc.b [abc11.btrig,abc.btrig,abc22.btrig,…].btrig,…]

A -> HA -> HNN!B/abc!B/abcNN.btrig [H.btrig [HNN!B/data/abc!B/data/abcNN]]

HHNN!B -> H!A/abc!B -> H!A/abcNN.btrig [results].btrig [results]

return checks H!A/abc.b for all donereturn checks H!A/abc.b for all done

Indirect FilesIndirect Files

anight1.ngttrig: anight1.ngttrig:

anight1.list:anight1.list:

Distribute data files across a networkDistribute data files across a network Move references and only move data as Move references and only move data as

neededneeded Pipeline objects: standard form, variable Pipeline objects: standard form, variable

contentcontent Act as triggers and meta-data containersAct as triggers and meta-data containers

anight1.ngttrig

anight1.list

Pipeline

DataDirectory

TriggerDirectory

Module

obj123.fits obj123.trig GO

File TriggersFile Triggers

Contains reference to

data

Data Trigger(DRA, user, or

pipeline module)

Tape Disk

DTS Process

Data Flow Networking: Data Flow Networking: ExampleExample

Host0:

Crosstalk

Host1:

Obj456.1

Obj321.2

Host2:

Obj567.2

Host3:

Obj123Obj123.2

Obj123.1Host3!Obj123.1

Host2!Obj123.2

Host4:

DOWN

Data Parallel ModulesData Parallel Modules

Some algorithms may need to be re-Some algorithms may need to be re-implemented specifically for a data parallel implemented specifically for a data parallel pipeline.pipeline.

One type is where measurements are made One type is where measurements are made across the mosaic for a global calibration.across the mosaic for a global calibration.

Rather than requiring all pieces to be in one Rather than requiring all pieces to be in one pipeline arrange for measurements made in pipeline arrange for measurements made in parallel to be collected for the global parallel to be collected for the global calibration and then apply the global calibration and then apply the global calibration to the pieces in parallel.calibration to the pieces in parallel.