1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 0

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 0

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 0100 1011

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 0

1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 0The National Virtual ObservatoryThe National Virtual Observatory

When and where are discoveries made? Always at the edges and boundaries Going deeper, using more colors….Metcalfe’s law Utility of computer networks grows as the number of possible connections: O(N2)VO: Federation of N archives Possibilities for new discoveries grow as O(N2)Current sky surveys have proven this amplification Very early discoveries from SDSS, 2MASS

Exponential growth Data will be never centralizedMore responsibility on projects Becoming Publishers and Curators Larger fraction of budget spent on software Lot of development duplicated, wastedMore standards are needed Easier data interchange, fewer toolsMore templates are needed Develop less software on your own

Astrophysical data is growing exponentially Doubling every year (Moore’s Law): both data sizes and number of data setsComputational resources scale the same way Constant $$$ will keep up with the dataMain problem is the software component Currently components are not reused Software costs are increasingly larger fraction Aggregate costs are growing exponentially

RolesAuthorsPublishersCuratorsConsumers

TraditionalScientistsJournalsLibrariesScientists

EmergingCollaborationsProject www siteBigger ArchivesScientists

Trends

Discoveries

Data Publishing

Changing Roles

Astrophysics has good track recordFITS: universally used to share low level data Individual images, tables, filesBut: new industry standards emerging XML, SOAPRequired by modern data exchange More dynamic (streams, queries) Merging heterogeneous sourcesGood time to adopt…

Accessing remote data: WWW, FTP Data formatted in certain ways (HTML, FITS)Accessing remote computing: Hard configured local area clusters Remote supercomputers Need to move data to the computing Resources do not always match problem

Standardizing distributed data Web Services, supported on all platforms Custom configure remote data dynamically XML: Extensible Markup Language SOAP: Simple Object Access Protocol WSDL: Web Services Description LanguageStandardizing distributed computing Grid Services Configure remote computing dynamically Build your own remote computer, and discard Virtual Data: new data sets on demand

Standards

Accessing Data Today

Distributed Services

Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.

gamma-ray

x-rayUV Opt

IRradio

Terabytes

No. Collections0

10

20

30

40

50

60

70

80

90

The data sets in astrophysics today span the whole electromagnetic spectrum, with a large sky coverage. The total volume of data today is about 100TB, in over 50 collections. New surveys, like GALEX and PRIME will dramatically improve the all-sky, all-wavelength coverage of the Universe.

Next generation astronomical instruments have always enabled discoveries beyond their original goals. For example, HST’s new ACS camera sees 3000 galaxies where previously only a few hundred were known. NVO can capitalize on the richness of these new multiwavelength surveys.

Building the Framework for the NVO

Funded by NSF ITR (2001), $10M over 5 yearsCollaboration of 20 organizations

Astronomy data centersNational observatoriesSupercomputer centersUniversity departmentsComputer science/IT specialists

PI/Project Director: Alex Szalay (JHU)CoPI: Roy Williams (Caltech/CACR)

Major milestones associated with simple, intermediate and complex science demos

First science demonstrations planned for Jan 2003

External Advisory CommitteeExecutive CommitteeWorking GroupsProject TeamsEducation and Outreach Initiatives

Astrophysical Virtual Observatory funded by European Commission (€3.3 million, three years)AstroGrid, funded by UK e-science

program (£5 million, three years)Canadian Virtual ObservatoryNational initiatives in Germany, Russia, Australia,

Japan, China, India, …International VO roadmap developedInternational VO Alliance has been formed

Metadata information about resources - Waveband - Sky coverage - Translation of names to universal dictionarySimple search patterns on the resources - Cone Search - Image mosaic - Unit conversionsSimple filtering, counting, histogrammingOn-the-fly recalibrations

Built on Atomic ServicesPerform more complex tasks

- Automated resource discovery- Cross-identifications- Photometric redshifts- Outlier detections- Visualization facilities

Expectation:Build custom portals in days from existing

building blocks(like in IRAF or IDL)

Define commonly used `atomic’ servicesBuild higher level toolboxes/portals on topWe do not build `everything for everybody’Use the 90-10 rule: Define the standards and interfaces Build the framework Build the 10% that are used by 90% Let users build the rest from components

The NVO: How Will it Work?

Atomic Services

Higher Level Services

• NSF ITR NVO project is one of three major and numerous other NSF ITR NVO project is one of three major and numerous other small VO-related initiatives now underway world-widesmall VO-related initiatives now underway world-wide

• NVO is adopting, adapting, or developing necessary technology as NVO is adopting, adapting, or developing necessary technology as derived from science requirementsderived from science requirements

• NVO project is dealing with many of the management challenges NVO project is dealing with many of the management challenges that will face the ultimate VO organizationthat will face the ultimate VO organization

• NVO is inevitable—the next logical step in the evolution of the NVO is inevitable—the next logical step in the evolution of the astronomical research environmentastronomical research environment

SummarySummary

International Collaboration

Schedule

Organization

The NVO CollaborationThe NVO Collaboration

Sponsored by the National Science Foundation’s Information Technology Sponsored by the National Science Foundation’s Information Technology Research ProgramResearch Program

USNO

Data Mining Find unusual patterns and features