the dark energy survey data management system
DESCRIPTION
The Dark Energy Survey Data Management System. Ignacio Sevilla Noarbe CIEMAT (Madrid) o n behalf of the DES Collaboration. From photons to catalogs. DES in a nutshell. Cosmological survey in visible/near IR light using 4 complementary techniques to characterize dark energy : - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/1.jpg)
From photons to catalogs
![Page 2: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/2.jpg)
• Cosmological survey in visible/near IR light using 4 complementary techniques to characterize dark energy:
I. Cluster Counts
II. Weak Lensing
III. Large-scale Structure
IV. Supernovae• Two multiband (photometric) surveys:
5000 deg2 grizY to 24th mag AB griz
10-30 deg2 repeat (SNe)• Build new 3 deg2 FOV multi-CCD camera,
Data management system, improve Blanco facilities
Blanco 4-meter at CTIO
2/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
Credit:NOAO
![Page 3: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/3.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 3/29
SDSS-II DES
Area 10000 5000
Nb. of CCDs 22 62
Resolution 120 Mpix 570 Mpix
Raw data/night 200 GB/night 300 GB/night
Catalog size 18 TB 100 TB (est.)
Total data volume 60 TB 4 PB (est.)
Credit: Kotwani et al. (2010)
![Page 4: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/4.jpg)
Transfer
Process
Archive
Distribute
4/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
![Page 5: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/5.jpg)
Transfer
CTIO NCSA 300 GB/night in 18 h
Process
Archive
Distribute
5/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
![Page 6: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/6.jpg)
Transfer
Process
Orchestration:
NCSA HPC nodes
Archive
Distribute
6/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
![Page 7: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/7.jpg)
Transfer
Process
Archive
Results Archive nodesOracle DB
Distribute
7/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
![Page 8: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/8.jpg)
Transfer
Process
Archive
Distribute
Through web portals
8/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
![Page 9: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/9.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 9/29
![Page 10: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/10.jpg)
Exposure consists of 62+ CCD images – 570 Mpix - 3deg2
Survey is ~150,000 100-sec exposures over 525 nights
Auxiliary CCDs record images for autoguiding and calibration
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 10/29
X 300 + calib. = RAW DATA
we send this to NCSA
![Page 11: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/11.jpg)
Correct for cross-talk among CCDs.
Correct for bias levels, non-uniformities, other optical and electrical effects.
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 11/29
DETRENDED DATA
Credit:NOAO
![Page 12: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/12.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 12/29
We need reference star catalogs
Full focal plane is fit to single solution
Correct optical distortion at focal plane
Credit: E.Bertin
We use Sextractor, SCAMP software by E.Bertin
![Page 13: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/13.jpg)
REDUCED DATA11/08/2011 APS-DPF 2011 DESDM I.Sevilla 13/29
Use standard star fields at different angles in the sky (X) with known fluxes (m_std and color_std) and relate with instrumental flux (m_inst):
Make big least squares solution for a,b,k; apply results to science images.
€
minst −mstd = a+ bCCDncolorstd + kXinputoutput
![Page 14: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/14.jpg)
At this point we have, for every night, approx. 300 exposures corrected by:
• Instrumental effects
• Absolute position
• Absolute photometry
This is the nightly processing.
We store these in the archive (+ auxiliary images, info).
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 14/29
![Page 15: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/15.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 15/29
COADDED DATA
Go deeper; calibrate better
BUT
Point spread function is inhomogeneous: each exposure has different quality
single exposure
single exposure
single exposure
single exposure
single exposure
single exposure
single exposure
![Page 16: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/16.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 16/29
COADDED DATA
Find PSF in each image
Homogeneize per coadd tile
PSF HOMOGENEIZED
0.77 ’’0.77 ’’
1.32 ’’1.32 ’’
0.94 ’’0.94 ’’
0.94 ’’0.94 ’’
![Page 17: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/17.jpg)
It takes 30x more time for coaddition with respect to nightly:
• PSF has to be extracted• PSF has to be homogeneized• Actual addition of image and recomputation of errors
Additionally:
• Global photometric calibration among all images of the season
We store these in the archive (+ auxiliary images, info).
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 17/29
![Page 18: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/18.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 18/29
CATALOGS
E.Bertin
This step is performed with the SExtractor package
Position, shape and photometry is calculated.
![Page 19: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/19.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 19/29
Hundreds of millions of objects with hundreds of columns each:
![Page 20: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/20.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 20/29
Hundreds of millions of objects with hundreds of columns each:
![Page 21: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/21.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 21/29
Hundreds of millions of objects with hundreds of columns each:
![Page 22: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/22.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 22/29
Photometric redshift pipeline: take fluxes in five bands -> estimate redshift
Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.
Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.
mag_band_g = 20.7mag_band_r = 19.2mag_band_i = 18.5… + errors, other estimates
(this only one kind of photo-z! More estimations foreseen)
DESDM Photoz pipeline(neural network)
![Page 23: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/23.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 23/29
Photometric redshift pipeline: take fluxes in five bands -> estimate redshift
Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.
Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.
Eliminate instrumentalsignature
Obtain true shape (intrinsic galaxy shape+shear)
DESDM
WL pipeline
local PSF
![Page 24: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/24.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 24/29
Photometric redshift pipeline: take fluxes in five bands -> estimate redshift
Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.
Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.
Credit: Pan-STARRS
![Page 25: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/25.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla
Produce cosmological simulations
Process through atmosphere, detectors, include ‘nasty’ stuff
25/29
![Page 26: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/26.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 26/29
Galaxies
Stars
![Page 27: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/27.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 27/29
BCS Images of First SPT Clusters
Credit: Blanco Cosmology Survey
Real data sets from the Blanco Cosmology Survey, SPT, SCS (Mosaic2 camera).
Large scale management with SDSS data.
![Page 28: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/28.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla
The Dark Energy Survey (next talks!) will make use of a large, scalable data management system to process and archive raw images and science ready data products.
Acceptance of the system is underway (results end of year). Tests on real DES data expected for first months of 2012.
Raw and reduced images to be released yearly, catalogs at midpoint and end of survey.
Community pipeline getting ready for usage of the DES camera starting May 2012.
28/29
![Page 29: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/29.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla
![Page 30: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/30.jpg)
![Page 31: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/31.jpg)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla
Development Funding ~$6MDevelopment Funding ~$6M $4 million from NSF$4 million from NSF $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and
MunichMunich $300K from DES collaboration for Community Pipeline$300K from DES collaboration for Community Pipeline
![Page 32: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/32.jpg)
(new simulations with more DES-like systematics coming up by Stanford team)
Carnero et al. 2010
![Page 33: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/33.jpg)
![Page 34: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/34.jpg)
![Page 35: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/35.jpg)
![Page 36: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/36.jpg)
Perform full-depth observations in 100 sq.deg.
Area is off main survey
Overlap existing datasets when possible
Run acceptance tests on data
![Page 37: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/37.jpg)
![Page 38: The Dark Energy Survey Data Management System](https://reader030.vdocument.in/reader030/viewer/2022013012/568159f3550346895dc73e5d/html5/thumbnails/38.jpg)