LOFAR - AARTFAAC

Gijs Molenaargijs@pythonic.nlhttp://pythonic.nl@gijzelaerr

Agenda

LOFAR / AARTFAAC

The data

Pipeline layout

Software

Disclaimer

I'm no scientist

No background in astronomy

Software engineer

LOFAR

Radio telescope

Low-Frequency

Antenna's, no dishes

Software telescope

Omni-directional

Concurrent observations

7000 antennas

10 to 80 MHz (LBA)

120 to 240 MHz (HBA)

Multiple small stations

Radio interferometry

Fourier Transformation

AARTFAAC

AmsterdamASTRON Radio Transients Facility and Analysis Center

Transient something that changes

Full sky monitor

Real time astro alert

Huge amount of data

+50 Gb/s

only superterp

What to see

Dwarf novae

Neutron star/black hole outbursts

Magnetar outbursts

Supernovae

Gamma-ray bursts

AGN and tidal disruption events

Unknown?

Image data (RSM)

Radio Sky Model

The data

1 datacube per second

12 frequency bands4 different polarization

50 images/second

Future: 500 images/second

Non stop

Pipeline

Performance

Last month: 0.3 image/second

Now: 3.2 image/second

Future: 50 image/second

Distributing computation

All image operations

Source extraction

Database interactions

Source association

T

T+1

T+2

T+3

Light curve

Libraries

Numpy / Scipy

Pyrap (casacore)

Celery

Django

Database

Move calculation to the data

Highly structured

Independent data

Naturally separable by sky coordinates

~100 TB/year

10.000 insert/second

MonetDB

Relational

Column store DB

Fast with huge data

Auto tuning

Developers next door (CWI)

Visualization - banana

Web interface

Django, bootstrap, high charts

Team

2 full time software developers (MSc A.I.)

2 astro software developers (PhD), 1 in UK

1 database expert

Scientists

Github, Jenkins

Unit tests, 80% coverage

Code review, pull requests

Open source

Almost!

Legal issues

https://github.com/aartfaac/

https://github.com/gijzelaerr/banana

https://github.com/gijzelaerr/djonet

More

http://www.transientskp.org/

http://www.lofar.org/

http://www.aartfaac.org/

Questions?