Future Directions of the VO

Alex SzalayThe Johns Hopkins University

Jim GrayMicrosoft Research

Living in an Exponential World

• Astronomers have a few hundred TB now– 1 pixel (byte) / sq arc second ~ 4TB

– Multi-spectral, temporal, … → 1PB

• They mine it looking for new (kinds of) objects or more of interesting ones (quasars), density variations in 400-D space correlations in 400-D space

• Data doubles every year

The Challenges

DataCollection

Discoveryand Analysis

Publishing

Exponential data growth: Distributed collections Soon Petabytes

New analysis paradigm: Data federations, Move analysis to data

New publishing paradigm: Scientists are publishers and Curators

Publishing Data

• Exponential growth:– Projects last at least 3-5 years– Data sent upwards only at the end of the project– Data will never be centralized

• More responsibility on projects– Becoming Publishers and Curators

• Data will reside with projects– Analyses must be close to the data

Roles

Authors

Publishers

Curators

Consumers

Traditional

Scientists

Journals

Libraries

Scientists

Emerging

Collaborations

Project www site

Bigger Archives

Scientists

Accessing Data

• If there is too much data to move around,take the analysis to the data!

• Do all data manipulations at database– Build custom procedures and functions in the database

• Automatic parallelism guaranteed• Easy to build-in custom functionality

– Databases & Procedures being unified– Example temporal and spatial indexing– Pixel processing

• Easy to reorganize the data– Multiple views, each optimal for certain analyses– Building hierarchical summaries are trivial

• Scalable to Petabyte datasets

active databases!

Next-Generation Data Analysis

• Looking for– Needles in haystacks – the Higgs particle– Haystacks: Dark matter, Dark energy

• Needles are easier than haystacks• ‘Optimal’ statistics have poor scaling

– Correlation functions are N2, likelihood techniques N3

– For large data sets main errors are not statistical

• As data and computers grow with Moore’s Law, we can only keep up with N logN

• A way out? – Discard notion of optimal (data is fuzzy, answers are approximate)– Don’t assume infinite computational resources or memory

• Requires combination of statistics & computer science

Why Is Astronomy Special?

• Especially attractive for the wide public• Community is not very large• It has no commercial value

– No privacy concerns, freely share results with others– Great for experimenting with algorithms

• It is real and well documented– High-dimensional (with confidence intervals)– Spatial, temporal

• Diverse and distributed– Many different instruments from

many different places and many different times

• The questions are interesting• There is a lot of it (soon petabytes)

The Virtual Observatory

• Premise: most data is (or could be online)• The Internet is the world’s best telescope:

– It has data on every part of the sky– In every measured spectral band:

optical, x-ray, radio..– As deep as the best instruments (2 years ago).– It is up when you are up– The “seeing” is always great– It’s a smart telescope:

links objects and data to literature on them

• Software became the capital expense– Share, standardize, reuse..

Goal Create the most detailed map of the Northern sky

“The Cosmic Genome Project”Two surveys in one Photometric survey in 5 bands Spectroscopic redshift surveyAutomated data reduction 150 man-years of developmentHigh data volume 40 TB of raw data 5 TB processed catalogs Data is public

Sloan Digital Sky Survey

The University of Chicago Princeton University The Johns Hopkins University The University of Washington New Mexico State University Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg

Sloan Foundation, NSF, DOE, NASA

The University of Chicago Princeton University The Johns Hopkins University The University of Washington New Mexico State University Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg

Sloan Foundation, NSF, DOE, NASA

Drift scan of 10,000 square degrees 24k x 1M pixel “panoramic” images in 5 colors – broad-band filters (u,g,r,i,z)

2.5 Terapixels of images

The Imaging Survey

Precision Cosmology

Power Spectrum of Fluctuationsfew percent accuracy!

Main challenge: with so much data the dominant errors are systematic, not statistical!

Using large simulations to understand significance of detection

SkyServer

• Sloan Digital Sky Survey: Pixels + Objects• About 500 attributes per “object”, 400M objects• Currently 2.4TB fully public• Prototype eScience lab (800 users) CasJobs

– Moving analysis to the data

• Visual tools – Join pixels with objects

Wireless Sensor Networks

• Collaboration with K. Szlavecz, A. Terzis, J. Gray, S. Ozer– Building 200 node network to measure soil moisture for

environmental monitoring– Expect 200 million measurements /yr– Deriving from the SkyServer template we were able to build

and end-to-end system in less than two weeks– Built a OLAP datacube, conditional sums along multiple

dimensional axes

http://lifeunderyourfeet.org/

Sociological Challenges

• How to avoid trying to be everything for everybody?• Rapidly changing “outside world”• Make it simple!!!• Publishing:

– Exponential linear– Data reliability credits and career paths

Where are we going?

• Relatively easy to predict until 2010– Exponential growth continues– Most ground based observatories join the VO– More and more sky surveys in different wavebands– Simulations will have VO interfaces: can be ‘observed’

• Much harder beyond 2010– PetaSurveys are coming on line (PanSTarrs, VISTA, LSST)– Technological predictions much harder– Changing funding climate – Changing sociology

Similarities to HEP

HEP

• Van de Graaf

• Cyclotrons

• National Labs

• International Labs

• SSC vs LHC

Optical Astronomy

• 2.5m telescopes

• 4m telescopes

• 8-10m class telescopes

• Surveys/Time Domain

• 30-100m telescopes

Similar trends with a 20 year delay,fewer and ever bigger projects…increasing fraction of cost is in software…more conservative engineering…

Can the exponential trend continue, or will be logistic?What can astronomy learn from High Energy Physics?

But: Why Is Astronomy Different?

• Especially attractive for the wide public• Data has more dimensions

–Spatial, temporal, cross-correlations• Diverse and distributed

– Many different instruments from many different places and many different times

• A broad distribution of different questions!

Future

How long does the data growth continue?• High end always linear• Exponential comes from technology + economics

rapidly changing generations– like CCD’s replacing plates, and become ever cheaper

• How many new generations of instruments do we have left?

• Software is also an instrument– hierarchical data replication– virtual data– data cloning

Technology+Sociology+Economics

• Neither of them is enough– We have technology changing very rapidly– Google, tags, sensors, Moore's Law– Trend driven by changing generations of technologies

• Sociology is changing in unpredictable ways– In general, people will use a new technology if it is

• Offers something entirely new

• Or substantially cheaper

• Or substantially simpler

• Funding is essentially level

Tale of the Tails

• Long tailed distributions– Pareto: 20% of population holds 80% of wealth– Zipf: word frequency follows a power law– C. Anderson: everything on the web is a power law

• Lognormal vs Gaussian– Multiplicative processes lead to lognormal

Log P = Log p1 + Log p2 + … + Log pn …

– Central limit theorem: Log P is a normal random var– Kapteyn: random fragmentation

• Lognormal resembles a 1/f over large dynamic range• Extremely important in web-based economics

– Amazon, Time-Warner, blogs, etc

Power Laws

• Barabasi: Power laws tend to arise in social systemswhere people are faced with many choices

• The more choices, distribution more extreme– Measured by the distance between #1 and the median

• Most elements in the power law system are below the average

• People’s choices affect one another, they are not random independent events

Examples: the Grid

• The size of computational problems is multiplicative Has to have a lognormal distribution

• Computers bought for the average job will not be large enough in the tail, but the system is still often idel– Need to borrow CPU for large jobs and loan when idle

M. Ripeanu (UC): Top 500 computers

Footprints and Cardinalities

10000

100000

1000000

1E+07

1E+08

1E+09

1E+10

1E+11

1 10 100

rank

nu

mb

er o

f ro

ws

SkyServer tables

S. Lubow (STScI)

Analyzing the SkyServer

• Prototype in data publishing– 200 million web hits in 5 years– 1,000,000 distinct users

vs 10,000 astronomers

http://skyserver.sdss.org/

SkyServer Web and SQL Traffic

Web and SQL Traffic

1

10

100

1000

10000

100000

1000000

10000000

100000000

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61Month

Hit

s

Web

SQL

Expon. (Web)

Vic Singh (Stanford/ MSR)

Skyserver Sessions

Vic Singh (Stanford/ MSR)

Human Pageviews by Organization

Vic Singh (Stanford/ MSR)

SQL Traffic

SDSS SQL Traffic

1.E+00

1.E+01

1.E+02

1.E+03

1.E+04

1.E+05

1.E+06

1.E+07

1.E+08

1.E+09

1.E+10

1.E+11

public astro collab antique runs CASJOBS

rows

elapsed

cpu

queries

Vic Singh (Stanford/ MSR)

Data Sharing in the NVO

• Users are more willing to part with their data if machine obtained

• What is the business model?• Three tiers (power law!!!)

(a) big surveys

(b) value added, refereed products

(c) mode ad-hoc data, images, outreach info

• largely done (a)• need “Journal for Data” to solve (b)• need “VO-Flickr” and an integrated environment

for virtual excursions for (c)

Data Reliability

• Is new data necessary better?– Yes: more of it, better calibrations

– But: always on the edge

– Usage of old data: changing into a power law

– (CNN, Time-Warner)

• Data publishing: once published, must stay

• SDSS: DR1 is still usedEDR

DR1 DR1

DR2 DR2 DR2

DR3 DR3 DR3 DR3

2001/0

7

2001/1

1

2002/0

3

2002/0

7

2002/1

1

2003/0

3

2003/0

7

2003/1

1

2004/0

3

2004/0

7

2004/1

1

2005/0

3

2005/0

7

2005/1

1

2006/0

3

EDR

weblog

0.E+00

5.E+04

1.E+05

2.E+05

2.E+05

3.E+05

3.E+05

4.E+05

4.E+05

5.E+05

5.E+05Human page

views

EDR

DR1

DR2

DR3

DR4

DR5

astro

collab

vo

admin

weblog

VO Trends

• VO is inevitable, a new way of doing science• Present on every physical scale today, not just

astronomy (NEON, Neptune, CERN, MS)• Driven by advances in technology, and economics,

mapped onto society• Boundary conditions: funding will be at best level• Computational methods, algorithmic thinking will

come just as naturally as mathematics today

VO Technology

• We will have Petabytes• We will need to save them, move them

– several big archive centers connected

• Need Journal for Data– curation is the key

• Always will be an open-ended modular system• Archives -- also computational services

– driven by economics: cheaper to process than move

VO Economics

• The Price of Software– 30% from SDSS, 50% for LSST– should there be full reuse vs no reuse today?– neither: we are not systems integrators– risks and benefits are power law– repurpose for other disciplines is an example

• The Price of Data– $100,000 /paper (Norris etal)– Drives new projects

• For SDSS there are 1300 refereed papers for $100M so far

• Level budgets

VO Sociology

• Learn from particle physics– do not for granted that there will be a next one– small is beautiful

• What happens to the rest of astronomy after the world's biggest telescope?

• The impact of power laws: – we need to look at problems in octaves– the astronomers may be the tail of our users– there is never a natural end or an edge

(except for our funding)

The Changing VO

• Boundary conditions change, we need to change every year!

• We must change at least as fast as the outside world or we will be left behind

• We will make mistakes! We need to recognize and recover from them, step back and do it differently

• If we do not make mistakes, we are not taking enough risks

• But: we need to buffer/dampen these changes to the community

The Future of the VO

• Does not have much of a past…• We need to keep running forward• We must take risks• Technology driving Sociology - limited by Economics• Everything is a power law – do not make assumptions!• Enormous potential• May be the only way to do 'small science' in 2020

Summary

• Data growing exponentially• Analyzing so much data requires a new model• More data coming: Petabytes/year by 2010

– Need scalable solutions– Move analysis to the data– Spatial and temporal features essential– 20 queries!

• Data explosion is coming from inexpensive sensors• Same thing happening in all sciences

– High energy physics, genomics, cancer research,medical imaging, oceanography, remote sensing, …

• eScience: an emerging new branch of science