ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

1

SD Pipeline Heuristics Status Report

ALMA Project, NAOJ

George KOSUGI

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

2

E2E Pipeline Process Flow

Scheduler

Archive

Offline

Telcal

ASDM (with raw data)

MS2

Filler

Result

Engines

Recipe

ScriptsCurrent Development

•Reference to Science Data•Ref. to Calibrator in a necessary condition

•need to search other calibrator for the sufficient condition?

•Pipeline Parameters

?

Pipeline

Pipeline

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

3

Current SD Pipeline Flow ChartSDFITS

Extract Info

Extract Data

Re-Gridding

FlagSpectrum

Pointing PatternAnalysis

TextTable

TextTable

numarrayPointing,StartTime,IntegrationTime,etc.

BaselineFit

Mapnumarray

Integration over Channels

> Single Pointing> Multi Pointing> Raster/OTF

MS for input data AIPS++Table for Info Storage

MS2MS2

MS for output

FITS

FITS for image

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

4

Environment for SD Heuristics Pipeline

Python Packages– casapy with ASAP– numarray: work area for spectrum data analysis– Scipy: Kmean clustering algorism– Numpy: data for Kmean– PyLab: plotting– PyFits: for collecting info from SDFITS header

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

5

What is necessary for SD UT MS input

– Need sample MS

– MSs for casa/ASAP UT in 2007/03 will be adequate Store/Retrieve info into/from AIPS++Table MS and FITS output

----------------- 2~3 FTE-w Calibration script

– Investigate TelCal Engine

– Investigate scripts used in the casa/ASAP UT

----------------- 2~3 FTE-w ? Additional plots

----------------- 1~2 FTE-w ?

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

6

Baseline Fit is done for each spectrum one by one

Determine Line Free Channels (if not specified)

Fit and Subtract Baseline Calculated from the Line Free Channels

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

7

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

8

Determine Line Free Channels (if not specified)

– Subtract DC (bias) and gradient components– Accumulate several spectra in close vicinity in space to e

nhance S/N ratio for line detection– Line Detection by application of ASAP.asaplinefinder– Kmean clustering analysis is applied to the detected lines

(parameters are line center and line width)– Select real (supposed to be) lines by examining continuiti

es in space, distortion of the vector, density of the detection, and cluster size

– Calculate 2-dimensional fit (ra, dec, line center), (ra, dec, line width) to each cluster member

– Apply fit to spectra whose position is within the clustering radius

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

9

Fit and Subtract Baseline Calculated from the Line Free Channels

– Determine fitting order for polynomial• Search gaps in the time sequence• fitting order is determined for each bunch of data (between 2 tim

e gaps)• fitting order is determined from frequencies of major component

s in the Fourier space– Calculate Fit

• to avoid jaggy due to the shortage of the accuracy in the calculation of higher-order power for polynomial, spectrum is divided into sub-spectra, if necessary.

• ASAP.asapfit was used for the fit, however it didn’t converge sometimes. So far python script is used for polynomial fit.

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

10

Spectrum based Flagging (not channel based)

Statistics are measured for each spectrum (emission line channels are excluded):– baseline rms

• before baseline subtraction• after baseline subtraction

– difference from the running mean• before baseline subtraction• after baseline subtraction

– may need other criteria• Tsys, etc...

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

11

Plot statistics

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

12

Gridding (Align to Lattice pattern)

Beam size and Grid spacing are given Spectra at positions inside beam radius from th

e grid center are combined into one spectrum– weight: none, gaussian beam, or cone-shaped beam– combine: median (no weight), weighted mean, Nm

axMmin rejected weighted mean

ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

13

Line Intensity Map

Accumulate flux in given channels Subtract flux outside given channels