meerkat science processing

Click here to load reader

Post on 14-Jan-2016




0 download

Embed Size (px)


MeerKAT Science Processing. Jasper Horrell, SKA SA Postgrad Bursary Conference, Stellenbosch – Dec 2010. 2014 Vision. Its 2014, imagine …. Science Processing (SP). Back to the present: - PowerPoint PPT Presentation


  • MeerKAT Science Processing

    Jasper Horrell, SKA SA Postgrad Bursary Conference, Stellenbosch Dec 2010

  • Its 2014, imagine .

    2014 Vision

  • Back to the present:Renewed 2010 focus on science processing with a dedicated team fairly early days. 6 people currently plus some part-time inputs.Two main focus areas as reflected in the approved large survey projects: a) imaging; b) non-imaging. Is there anything else? Imaging: SP focus on data processing and archive post correlation. Define, design and build the facility systems to support this over next few years.Non-imaging: Current SP focus on raw data capture, offline correlation, data analysis, pulsars, VLBI capture, SETI. Interface to the approved large survey project teams re data products, formats, pipeline development, etc. (Meeting earlier today!)SIMULATIONS, prototypes, involvement in early science projects, etc.

    A bit more to this than using CASA for data reduction

    Science Processing (SP)

  • MeerKATData Flow

  • KAT-7 our playground

  • Observation Planning/Tracking

  • Getting there - not quite

  • Getting there remote ops!

  • New Cape Town Office (cont.)

  • KAT-7 Early Fringes (2009)

  • KAT-7 Horizon Masks

  • KAT-7 Horizon Masks

  • KAT-7 Images (2010)4-dishes

  • Started archiving visibility data from KAT-7 in Dec 2009To date, 1230 data captures saved to archiveApprox 490 GB visibility data so farOn average 40 data captures per week (March to Oct, excluding RFI campaign)Trivia: Fornax A and Pic A are most observed targets

    KAT-7: Archive To Date

  • Archive will need more and more attention as data requirements evolveAlready seeing the need to support multiple data sources and formats Fringe Finder, KAT-7 and holographyCurrently reviewing and testing Apache OODT components to assess viability for MeerKAT data catalog and archivingASTRON and NRAO also have systems of interest.Early days.

    The threat of raw data!!!

    KAT-7: Archive Future

  • KAT-7 Data Archive

  • First site test of raw data capture (full band 400 MHz for a few secs plus GPU-based digital down-converter for indefinite period)First VLBI data capture run (HartRAO fringe test)First SETI data capture for Project Dorothy300 GB recorded Also, Picture A

    KAT-7: Last week on site

  • 2 x 10 GbE NICs2 x GTX 460 2GB GPUs2 X 6 Core 3.0 GHz Xeons16 x 2 TB HDD24 GB RAM

    The Beast - Big Gulp?

  • Pic A: Hot Off the Press!

  • Pic A: Hot Off the Press!

  • Pic A: Hot Off the Press!

  • Pic A: Hot Off the Press!

  • Pic A: Hot Off the Press!

  • Pic A: Hot Off the Press!

  • Exploring Polarimetric Cal

  • Exploring Polarimetric Cal !!where (even) Oleg fears to tread!!

  • Exploring Polarimetric Cal

  • Exploring Polarimetric Cal

  • Calibrated visibilitiesCertain standard pipelines (spectral line image cubes, continuum images)Storage for visibility data (project disk quotas?)A 10 PB archive on site plus 3 PB in CT, plus likely European mirrorArchive access and facilities to reprocess from archiveFlexible architecture for black belt users plus space in the Array Processor Data Centre for additional user-supplied equipmentSupport for some re-use of existing mature packages where possible (support CASA, MeqTrees).Proposal management etc.

    SP: MeerKAT to provide (?) :

  • Basic data capture framework to HDF5 file archive format and MS export writing capability (currently being extended for full KAT-7 correlator)Simple archive for commissioning purposesSCAPE commissioning packageFirst imaging softwareSimulation and data reduction tests in MeqtreesPrototype online system based on ROACH sampling, followed by GPUs pre-processing and disk storage. Allows for raw data recording for commissioning, VLBI, software correlation, SETI Emerging NRAO and ASTRON collaborations Some great early results on the KAT-7 systemA lot of work to be done over the next few years (we wont be bored).

    SP: In place now

  • Tests of streaming framework (online system) prototype for KAT-7 with new 16-input KAT-7 correlator (KAT-8 ready!) now in lab testing phase, site deployment soonishMore mature imaging on KAT-7 using both CASA and MeqTrees. Aim for 7-antenna hand-crafted test image using both packages and compare. Also try pipeline mode on new data (expect poor initial results).Coarse grain parallelization investigations (for CASA)GPU gridding (for CASA) Simulations for MeerKAT using primary beams (sky de-rotation effects etc). CASA and Meqtrees. Aim for SKA memo.Beam polarization characterization on ATA and KAT-7Continue RFI characterization and mitigation work (RFI database plus GPU flagger)Ongoing HDF5 scalability testingStart of some pulsar tools on KAT-7 (port GUPPI to ROACH2 plus GPU?)More detailed design for MeerKAT (archive, data model etc)Small VLBI and SETI projects on KAT-7Engagement with large survey project teams!!

    SP: Next few months

  • Major joint effort needed over coming years to ensure that the science teams hit the ground runningAlso need to support commissioning and smaller PI proposalsTechnology within reach for MeerKAT scale imagingFine tuning (e.g. imaging pipeline) will take timeTechnology challenges in pulsar search areaKAT-7/8 will continue to be an excellent engineering and science test bedPlan re-use software where possible, but expect some custom developmentsLots to do. Technical team will need to grow a bit. Project teams can help (a lot).




    jasper @ (full SP Team):

    Thomas BennettMattieu de VilliersJasper HorrellTshaks NamaliliSimon RatcliffeLudwig Schwardt


View more