XLDB -Lyon Aug 2009

Astro-WISE -> TarGetAstro-WISE -> TarGet

OmegaCEN – TarGet Kapteyn Astronomical Institute

University of Groningen

28 August 2009

UU

OmegaCEN – TarGet Kapteyn Astronomical Institute

University of Groningen

28 August 2009

UU

Edwin A. ValentijnEdwin A. Valentijn

Astro-WISE information system – fully datacentricAstro-WISE information

system – fully datacentric

All data beyond pixel data is Metadata

all pixel data <–>data serversall Metadata <–> database

compute clusters / GRIDs all I/O to db

• all components scalable• all components EU distributed

All data beyond pixel data is Metadata

all pixel data <–>data serversall Metadata <–> database

compute clusters / GRIDs all I/O to db

• all components scalable• all components EU distributed

OmegaCEN review 2008

VST - Virtual Survey TelescopeVST - Virtual Survey Telescope

XLDB -Lyon Aug 2009

Centers – satellitesCenters – satellites

Brasil

Astro-WISE paradigmAstro-WISE paradigm

“Classical” paradigmForward chaining

Target processing - AweBackward chaining

waterfall model TIER architecture

User hunts upstream

driven by input raw data Driven by query of user

Process in pipelineworkflow

Process in bits and pieces on the flyBackward chaining

Operators push data User pulls data

Results in releases Provide information system

Static archives – publish Dynamic archives –publish Internet

Raw data - obsolete Raw data is sacred

TARGET diagram TARGET diagram

Central role dbCentral role db

• All I/O via db: params, sources, events• Objects persistent in db• data server access by global filename=key

• Security, proprietary, privileges• parallel processing• Web services• Synchronized real time National Nodes

• All I/O via db: params, sources, events• Objects persistent in db• data server access by global filename=key

• Security, proprietary, privileges• parallel processing• Web services• Synchronized real time National Nodes

XLDB -Lyon Aug 2009

the avalanchethe avalanche

• on-the fly re-processing for everything • 5LS: 5 Lines Script Awe> prompt Python• Trend analysis Awe > prompt• All dependent bits are traced “tell_me_everything_tool• Administration for parallel processing -compute GRID

• Connect to EGEE - Grid

• Global solutions • astrometry/photometry

• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO

• on-the fly re-processing for everything • 5LS: 5 Lines Script Awe> prompt Python• Trend analysis Awe > prompt• All dependent bits are traced “tell_me_everything_tool• Administration for parallel processing -compute GRID

• Connect to EGEE - Grid

• Global solutions • astrometry/photometry

• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO

Example 5LSExample 5LS

#Find ScienceFrames for a ccd named ccd53 and filter

Awe> q = (ReducedScienceFrame.chip.name == 'ccd53‘) and (ReducedScienceFrame.filter == ‘#841’)

# From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q]

# get the rms of the used Masterflat (dependency)Awe> y = [k.flat.imstat.stdev for k in q]

# Make a plot Awe> pylab.scatter(x,y)

#Find ScienceFrames for a ccd named ccd53 and filter

Awe> q = (ReducedScienceFrame.chip.name == 'ccd53‘) and (ReducedScienceFrame.filter == ‘#841’)

# From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q]

# get the rms of the used Masterflat (dependency)Awe> y = [k.flat.imstat.stdev for k in q]

# Make a plot Awe> pylab.scatter(x,y)

XLDB -Lyon Aug 2009

+ WFCAM, MegaCAM, LBC, ISAAC, LOFAR

XLDB -Lyon Aug 2009

www.astro-wise.org

OmegaCEN review 2008

Web servicesTarget processor

Web servicesTarget processor

Quality viewQuality view

XLDB -Lyon Aug 2009

XLDB -Lyon Aug 2009

Extreme data lineage Extreme data lineage

TASK24

TarGet TarGet

XLDB -Lyon Aug 2009

2009-13 32M

UKIDSS workshop 2007

Astro-WISE – LOFAR LTAAstro-WISE – LOFAR LTA

IBM- Blue Gene/P

5 Petabyte/yr

- EoR

- Survey

IBM- Blue Gene/P

5 Petabyte/yr

- EoR

- Survey

XLDB -Lyon Aug 2009

Inte

gra

tion

LifeLines 165.000 patients 30 yr

LifeLines 165.000 patients 30 yr

METABASE

MOLGENIS

Healtcare Transaction Base (HTB/HL7)

Molecular

Care

Physiology

XLDB -Lyon Aug 2009

IJkdijkIJkdijk

XLDB -Lyon Aug 2009