the radio interferometric data challenge: from meerkat...

O. Smirnov - MeerKAT to SKA - SIPS2018 - Cape Town, 23 Oct 2018

1

The Radio Interferometric Data Challenge: From MeerKAT Towards The SKA

O. Smirnov (Rhodes U. & SARAO)

+many others

O. Smirnov - MeerKAT to SKA - SIPS2018 - Cape Town, 23 Oct 2018 2

Why Is MeerKAT Like This?

www.ska.ac.za3

www.ska.ac.za4

www.ska.ac.za5

64x13.5m offset gregorian

Design and specification factsheet

~75% within 1 km core, baselines ranging from 7.7km down to 29.3m (resolution 5’’ - ~40’)

L-Band (856 - 1712 MHz) 208 kHz correlator commissioned, 26 kHz correlator under development

www.ska.ac.za6

Array configuration

KAPB

Latest survey antenna coordinates available in configuration simulation package SimMS https://github.com/radio-astro/simms

https://github.com/radio-astro/simms

www.ska.ac.za7

MeerKAT sensitivity

Array average SEFD from recent sensitivity tests on SCP using 20K noise diode(Lower is better!)

www.ska.ac.za8

VLA sensitivity

On par!

www.ska.ac.za9

Recent imaging results


MeerKAT @Karoo, South AfricaNB: radio is not really orange...


The Galactic Centre Image(Ian Heywood, Oxford U. & Rhodes)


The GC Image

www.ska.ac.za13

G332 (Benjamin Hugo, SARAO/Rhodes)

Blue: MeerKAT (May, 2018)Red: NASA Spitzer

www.ska.ac.za14

ATCA SGPS (Haverkorn et al. 2006)

MeerKAT 16 AR 1.5 (Sharmila Goedhart, March 2017)

MeerKAT 64 (Benjamin Hugo,

July 2018 inaugural event)6 pointings (2 hrs each)~50 uJy noise (uniform)

www.ska.ac.za15

1.5deg

LMC

Oleg Smirnov (Jul 2018)

www.ska.ac.za16

30 Doradus

MeerLICHT & MeerKAT (22 uJy, steps of sqrt(2)) Benjamin Hugo, Paul Vreeswijk, Ian Heywood


17

Circinus A (Kshitj Thorat, Gyula Jozsa, SARAO/Rhodes)

• Interesting nearby galaxy

• Left: optical image (Cir A centre), with new MeerKAT detections of 7 previously unknown HI galaxies

• HI is the spectral line corresponding to neutral atomic hydrogen, and has a rest frequency of 1420 MHz

rms = 0.15 mJy (45 km/s)


18

H I total intensity

• HI image (ATCA)


19

H I total intensity

● MeerKAT resolution much better with similar sensitivity


20

Circinus Galaxy optical

UK Schmidt (DSS)


21

Circinus Galaxy infrared

WISE Composite:W1,W2,W3(court. Jarrett)


22

Circinus Galaxy infrared + H I total intesity


23

Circinus Galaxy optical + H I total intesity


24

Circinus Galaxy optical + H I total intesity


25

Circinus Optical + H I + Radio Continuum


26

Circinus H I Velocity field: rotation


Interferometric Imaging Is Simple“A high quality radio map is a lot like a sausage, you might be curious

about how it was made, but trust me you really don't want to know.”– Jack Hickish

data

instrumentalresponse

sky noise

A is somewhat large

For ~ few hours of raw MeerKAT data: ~ 1011x109

A is nasty (AHA is non-invertible): ill-posed inverse problem

We don’t (precisely) know A to begin with (calibration!)

(going to ignore that little problem for today...) So we can’t and don’t really do it this way

But it’s a good equation to keep in mind


Radio Interferometer...

What lay people think I do

(In celebration of the passing of an extremely lame but blissfully short-lived internet meme)



What lay people think I do What funding agenciesthink I do





What cosmologists & astrophysicists think I do





What cosmologists & astrophysicists think I do What my engineers think I do





What cosmologists & astrophysicists think I do What my engineers think I do What I actually do



Why Bother

A key observational limitation of any telescope is its resolution (i.e. pixel size)


Why Bother

A key observational limitation of any telescope is its resolution (i.e. pixel size)

Resolution is determined by wavelength and aperture size D

Radio wavelengths are ~meters (optical: nm)

25m dish observing at 21cm: (full Moon) Going bigger quickly becomes prohibitively expensive

...but two dishes tied together into an interferometer have a combined resolutiondetermined by the baseline B (baseline = distance between dishes)


How To Make An Interferometer 1

Start with a normal reflector telescope....



Then break it up into sections...



Replace the optical path with electronics



Move the electronics outside the dish

...and add cable delays



Why not drop thepieces onto the ground?



...all of them



And now replace them with proper radio dishes.

...and that's all! (?) Well almost, what about

the other pixels?

+


How Does Optical Imaging Do It?

This bit sees signal from all directions in the sky, added up.

This bit sees signal from all parts of the

dish surface, added up.∬ S l ,me i ulvmdl dm


Fourier Transforms

An optical imaging system implicitly performs two Fourier transforms:

1. Aperture electromagnetic field distribution = FT of the sky

2. Focal plane = FT-1 of the aperture EMF A radio interferometer array measures (1)

(Each baseline gives one Fourier mode at a time) Then we do the second FT in software Hence, “aperture synthesis” imaging

“Earth rotation aperture synthesis”

The Earth swings our baselines around and helps sample the Fourier plane more densely


Interferometric Imaging

A pair of antennas (p,q) measures a single Fourier mode (visibility) of the sky brightness B, given by the baseline vector u

(So A is really a kind of a Fourier transform matrix)

Thus, to make an image quickly:

Collect enough visibilities to decently sample the Fourier plane

Do an inverse FT using the FFT algorithm

Release glorious images

But...


Measuring In Fourier Space

As the Earth rotates, a baseline sweeps out an arc in the uv-plane, filling out the uv-coverage

Invariably, gaps remain (incomplete Fourier plane sampling)

Sampling in Fourier domain <=> convolution in image domain(Fourier convolution theorem)


The Dreaded PSF

Response to a point source: Point Spread Function (PSF)

PSF = FT(uv-coverage)

Observed “dirty image” is convolved with the PSF

Structure in the PSF = uncertainty in the flux distribution (corresponding to missing data in the uv-plane)

24

PSF of WSRT. The regular rings are due to the regular spacing of its antennas in theEast-West direction.

PSF of MeerKAT


“Dirty” Images The PSF causes

bright sources to mask faint sources

Deconvolution required to remove the effect

Left: JVLA image of 3C147`

This sciencehas been done

All the new scienceis down here


“Clean” Images

Effect of PSF removed through an algorithm called CLEAN (Högbom 1978)

A.k.a. “the venerable CLEAN algorithm”

Identify brightest peak

Subtract a bit (10%, say) of the PSF centred on that peak

Rinse & repeat

Its various derivations have become the workhorse of radio interferometric imaging


LMC Revisited: Clean


LMC Revisited: Dirty


Calibration Errors & PSF Calibration errors distort the PSF (w.r.t. the nominal

one), making CLEAN fail

(So A is a kind of corrupted Fourier transform matrix)


CLEAN Pros & Cons

The words “venerable CLEAN algorithm” mask 40 years of Stockholm syndrome

Comfortingly familiar: at least we (think we) understand when it does and doesn't work

It is reasonably efficient (N log N)

Has straightforward multiscale, multifrequency extensions

Is a “just so” algorithm (though see CS!)

No error bars

Uncertain convergence

Recovers non-physical models


53

S. Makhathini, R. Perley & RATT 2016 JVLA L-band 640 MHz, BnA+C+D config 2.87 uJy rms, DR: ~8 million


54

Spot 3 differences


MeerKAT @Karoo, South AfricaSpot 3 differences


56

Best-fitting model

“Restored” image

Residual data

“Noise-limited” map


Cyg A: RATT & R. Perley 2016 JVLA S-band, A+B+C config

"This is an image of a supermassive black hole about a billion times more massive than the Sun emitting jets at close to the speed oflight. If that doesn't get you up in the morning, I don't know what will." –Ian Heywood


Good Map, Bad Map

“Good maps are noise-limited maps” Is this a robust statement?


Real or Fake?


Science...

...has developed a rigorous approach to this Theory → hypothesis → prediction Match observed data to prediction Verify, or reject, or alter hypothesis Bayesian reasoning encapsulates this

mathematically:


Bayes' Theorem

P M∣D=P D∣M P M

P DM :modelD : data

PosteriorProbability of the model given this set of data.

We want to find an M that maximizes this.

LikelihoodProbability of this set

of data, given the model.

Marginal probability (Evidence)Normalizing term,

(but see model selection...)

PriorProbability of

the model from prior knowledge

or assumptions


Interpreting Features

Prior based on years of experience, and independent data

Likelihood based on [non] appearance of feature (and is fairly flat)

Why not a proper likelihood?

CLEAN gives no error bars...


The Brutal Bayesian

Bayesian imaging is dead simple in principle (MCMC, nested sampling, etc.):

draw random samples from your prior x feed them forward through A and evaluate the

likelihood of y construct full map of posterior (or at least MAP

estimate + error bars)

In practice, this means evaluating Ax many times (105~106)...

...while CLEAN does it <10 times


Imaging As An Inverse Problem

Imaging is an inverse, ill-posed problem A continuum of possible skies fits the observed

data (if the sky was truly random, we'd be sunk...)

y: observed data x: underlying sky A: instrument response n: Gaussian noise


LSQ & Regularization

(Thanks to Jason McEwen) Because noise is Gaussian, the maximum-

likelihood solution is a least-squares fit:

Infinitely many solutions exist, hence, introduce regularization to pick some preferred solution

Alternatively, solve a constrained problem:


CLEAN Is Regularization

CLEAN can be viewed as a regularization:

Minimizing the L1-norm promotes minimizing the number of non-zero pixels (sparsity)

Another popular algorithm, MAXENT, seeks to maximize the entropy of the solution:

Both represent some prior beliefs about what underlying sky we expect


Compressive Sensing & Sparsity

What if the signal is not sparse in pixels, but sparse in some other representation (“dictionary”) e.g. wavelets?

We can then reformulate the problem as finding a representation with the least number of non-zero coefficients (lowest L0-norm) α:

Or as a constrained optimization problem:

(where the L1-norm is a proxy for the L0-norm...)


Link To Bayes

Philosophically, this represents our attempt to reconcile the observed data (visibilities) with prior beliefs about the underlying sky

Bayes' theorem represents a statistical framework for this


Regularization Is a Poor Man’s Bayes

Likelihood, assuming Gaussian noise:

Consider a Laplacian prior:

Then the MAP estimate is:

=


CLEAN From a Bayesian POV

CLEAN, as well as the newer CS approaches, can be seen as imposing a Bayesian sparsity prior

...and finding the single maximum a-posteriori (MAP) solution

...without information on the posterior likelihood distribution (not even error bars)


Discovery vs Verification Space

A common discourse between Bayesians and (CS) map-makers:

“You have a lump of data sitting there. You point your finger at it and say, 'I declare thou sparse!'”

“You can never discover anything new in the data, because you're always imposing your prior models, so you can only prove or disprove the prior model.”

Map-making is “discovery space” We’ve learned how to do this relatively cheaply

Bayesian reasoning is “verification space” This is where we need urgent progress


Bayesian Approaches

RESOLVE (Juklewitz et al., Ensslin at al.) a.k.a. Information Field Theory (IFT) Sky: random field with a log-normal prior Finds MAP estimate + error bars

BIRO (Lochner, Natarajan et al.) The Brutal Bayesian construct parametric models of the instrument and sky

and then do MCMC or nested sampling gives a handle on degeneracies between sky and

instrument


The Major Cycle

Most (all?) imaging algorithms hinge around evaluating A and/or AH

...and are bottlenecked by it (Model) image↔visibilities

This is what we call the “major cycle” typically, N∙log N (± I/O) thanks to the FFT

The universal currency of algorithmic cost Historically very stable ©


The Dodgy/Barmy Matrixba

rmin

ess

dodginess

dodgybarmy line

Reality Zone

Santa/Moore Zone

Conference Zone(a.k.a. Museum Of Toys,

a.k.a. Unicorn Cemetery)

Kooks'Corner

run for the hills dodgy

meet theparents

close youreyes and hope

for the best dodgy

totallylegit

BIRO

CLEAN

CS

most ofthe time

not dodgy

really not dodgy

I promise

IFT

GPR

BIROSKA

1

10

100

10 3

10 6+

barm

ines

s (V

LBI u

nits

)

1

10 6

1

42


The SKA Challenge

SKA1-MID: (descoped) 10% of the full SKA: 197 dishes

~few hours of raw MeerKAT data: ~ 1011x109

~few hours of raw SKA1-MID data: ~ 1013x1011

Upping the barminess factor by 100x100 MeerKAT data can (just about) be stored and shipped to users

(large LSP teams), no real-time processing requirement

We can process and reprocess it until we get it right SKA1 raw data can’t be moved or stored (at a realistic cost), so...

Throw it away Reduce to science products in real-time, centrally Compress it cleverly

This is your raw data size i.e. what

rolls off the telescope

This is the skyi.e. your

“science product”


Where To From Here?

Push compressive sensing and the like to be less dodgy (Wiaux et al.)

Push Bayesian methods to be less barmy (reduce the cost in )

Inflate the (make it cost less):

New clever approximations Develop data compression techniques

Break the ?

Drown the in CPU?


In ConclusionThe future is bright (and often orange)!

the radio interferometric data challenge: from meerkat...

Documents