millennium data dissemination
Post on 23-Feb-2016
35 Views
Preview:
DESCRIPTION
TRANSCRIPT
Millennium Data Dissemination
databases and other services
2012-12-17 Millennium Workshop 1
Raw data:Particles
FOF groups and Subhalos
Density fields
Subhalo merger trees
Synthetic galaxies (SAM)
Mock catalogues
Mock images
2012-12-17 Millennium Workshop 2
SUBFIND Tree Builder
Halo Merger Trees
SAMSSP
FOF GroupFinder
(Sub-)HalosFOF GroupsSimulation Particles
Gadget
MoMaF’
SAM Galaxies
Light-cone Galaxies
SkyMaker’SExtractor
Synthetic Images
SExtractor Catalogues
PSF, Noise,Background
Scaling
IGM
MRObs
2012-12-17 Millennium Workshop 3
Virgo-Millennium Database
• Online access to Millennium simulations etal– portal: http://www/mpa-garching.mpg.de/millennium
• Databases – public: http://gavo.mpa-garching.mpg.de/Millennium – private + MyDB:
http://gavo.mpa-garching.mpg.de/MyMillennium• Improved database access– MillenniumTAP batch queries – http://galformod.mpa-garching.mpg.de/millenniumtap
2012-12-17 Millennium Workshop 4
Millennium data model (old)
5 5
last progenitor main leaf and root
Trees in Database
2012-12-17 Millennium Workshop 6
Merger trees
particles halos
2012-12-17 Millennium Workshop 7
Joining Garching and Durham
8 8
2012-12-17 Millennium Workshop 9
Most recent data sets(@MPA)
• New simulations: – MR-XXL: Angulo etal 2012– Millennium WMAP7 (Thomas etal, MilliGas?)
• Updated semi-analytics, with (scaling to) WMAP7, MCMC– Guo etal 2011, 2012– Henriques etal 2012/3
• New synthetic light cone catalogues– Henriques etal 2012– also some all-sky (SDSS depth)
• Millennium Run Observatory– Overzier etal 2012, images+Sextractor catalogues
• Most loaded in new databases– soon: MR-XXL trees, Millennium WMAP7 (sim +scaled, halos + galaxies)
• New DB functions:– spatial queries– possibly: scaling, …
2012-12-17 Millennium Workshop 10
Millennium Workshop 11
AllSky(Paritosh Verma)
2012-12-17
“Red”“Blue”
“All” “All”smoothed
Most recent data sets(@MPA)
• New simulations: – MR-XXL: Angulo etal 2012– Millennium WMAP7 (Thomas etal, MilliGas?)
• Updated semi-analytics, with (scaling to) WMAP7, MCMC– Guo etal 2011, 2012– Henriques etal 2012/3
• New synthetic light cone catalogues– Henriques etal 2012– also some all-sky (SDSS depth)
• Millennium Run Observatory– Overzier etal 2012, images+Sextractor catalogues
• Most loaded in new databases– soon: MR-XXL trees, Millennium WMAP7 (sim +scaled, halos + galaxies)
• New DB functions:– spatial queries– possibly: scaling, …
2012-12-17 Millennium Workshop 12
Spatial queries in the databaseSpatial3D library (with Tamas Budavari)
• Searches for objects in spatial subsets of simulation box is hard
• C# library for optimising such spatial queries using PH space filling curve
• Soon in Millennium DBs (needs documentation)
2012-12-17 Millennium Workshop 13
with cover as ( -- some details left outselect * from dbo.fSimulationCoverSphere(
Simulation::Millennium,Sphere3D::newInstance(0,0,0,20),6) )
select f.* from cover c , millennium..fof f where f.snapnum=67 and f.phkey between c.keymin and keymax and c.shape.ContainsPoint(f.x, f.y, f.z)=1
14
Query shape syntax• Primitives :
– box[ 0,0,0,10,10,10 ], sphere[10,10,10,20]– cone[0,0,0,1,1,1,.1,600] vertex,direction,angle,min-
depth,max-depth, – conesegment[0,0,0,1,1,1,.1,500,600]
vertex,direction,angle,min-depth,max-depth• Combinations :
– union/intersection/difference [ shape (, shape)* ]
15
courtesy RaulAngulo
used in MXXL browser
In the future: light cones in SQL(requires galaxy catalogues with dM/dz)
2012-12-17 Millennium Workshop 17
Usage statistics• 589 registered users• > 17.3 million queries (7532/day)• > 287 billion rows (137 million/day)queries,rows,users
2012-12-17 Millennium Workshop 18
Distributions over users
2012-12-17 Millennium Workshop 19
Millennium TAP• New DB access service
– All MyMillennium users have access• Batch queue
– longer timeout– capability to kill queries– history
• Implements IVOA’s Table Access Protocol (TAP)– AJAX frontend: TAPFace– TAP aware tools can query it, TOPCAT
• Some differences– schema vs database (Non SQLServer DBs can be served)– no mydb (yet)
• Demo this afternoon
2012-12-17 Millennium Workshop 20
It’s not always as simple as it seems.One should try to understand the data when using the database
thanks to
Kevin Bundy, Tommaso Treu, Richard Ellis
http://astro.berkeley.edu/~kbundy/millennium/
2012-12-17 Millennium Workshop 21
Other Millennium data dissemination services
• portal: http://galformod.mpa-garching.mpg.de • Simulation service(s)– SeSam online (Neistein, Weinmann , Bourges)– L-Galaxies online (see talk by Matthias Egger)
• Image Browsers: – may help define where to point your virtual
telescope.– MR-XXL browser (public | interactive) (Angulo), – MRObs image browser (public | private) (Overzier
etal)
2012-12-17 Millennium Workshop 22
Service vs other data sets• Databases– HydroClusters database (Dolag)
• Virtual telescopes– Smac (Dolag, Reynecke, Bourges)– PHOX (Biffi etal)
• Image Browsers: – Phoxy browser + PHOX
2012-12-17 Millennium Workshop 23
Thank you.
2012-12-17 Millennium Workshop 242012-12-17 Millennium Workshop 24
top related