heasarc status and plans

February 13th, 2004HEASARC Users Group

Nick White

HEASARC Status and Plans

Nick White


Nick White

The HEASARC CharterThe HEASARC CharterEstablished December 1990

• Maintain and disseminate data from previous and concurrent high-energy astrophysics missions

• Provide software and data analysis support for these data sets• Maintain and provide the necessary scientific and technical expertise

for the processing and interpretation of the data holding• Develop and maintain multi-mission analysis and support tools• Provide catalogs of observations and ancillary information for the

data holdings• Coordinate data, software and media standards with other

astrophysics sites• Support education and outreach activities in high energy

astrophysics (not part of original charter)


Nick White

The HEASARC and High Energy Astronomy Missions

HEASARC provides the multi-mission infrastructure that is used by parallel GOFs and science support centers in the LHEA and elsewhere

Swift

Chandra data@CXC

HETE2

2GLAST

Integral


Nick White

HEASARC Science Personnel• Nicholas White: HEASARC Head• Steven Murray: HEASARC Deputy Head (at CXC)• Lorella Angelini: Data Restoration, Image and timing

analysis, BeppoSAX, Swift (inc. AKBAR), Astro-E2• Keith Arnaud: Data selection software, spectral analysis• Michael Corcoran: Calibration, GLAST, HETE-2, ROSAT• Stephen Drake: Catalogs, website & archive quality

assurance, EUVE, RXTE, CGRO• James Lochner: Education and public outreach• Thomas McGlynn: Archives, Virtual Observatory,

SkyView• Bill Pence: FITS, Software, HEASOFT, Hera


Nick White

Recent HEASARC Highlights• Hardware:

– Acquisition of 8.4 TB (12 TB raw) TB network appliance (March 2003)– Upgrade to Gigabit network (Oct 2003)

• Science Archive:– XMM downloads now 2nd in volume to RXTE– HETE-2 archive reaches 150 GB and 51 bursts– Specialized Swift Browse interface

• Software:– HEAsoft 5.3 release (November 2003): Improved fv integration with DS9– Browse 7.0 release (January 2004): new upload & download features– XSPEC 12 testing by Integral SPI team (ongoing)

• Catalogs:– GRBCAT master catalog of gamma-ray bursts (Feb 2004)

• Education and Outreach– SkyView SciAm Top 50 award (June 2003: 2nd year in a row)– Teacher’s Hera beta implementation (July 2003)

• Services:– First VO Tool: Data Inventory Service (July 2003)


Nick White

The Physical ArchiveThe Physical Archive

Active Missions

RXTE (1995- )Chandra (1999- ) [data at CXC]HETE-2 (2000- ) Integral (2001-)XMM-Newton (1999- )

Past Missions

Ariel 5 EXOSATASCA GingaBBXRT BeppoSAX CGRO HEAO 1Copernicus HEAO 3COS B OSO 8DXS ROSAT Einstein SAS 2EUVE SAS 3 Vela 5B • Data from 24

missions currently in the archive

• 355 astronomical catalogs & mission tables

• The archive volume was3250 Gigabytes as of the end of 2003

Upcoming Missions

Swift (2004 Launch)Astro-E2 (2005 Launch)GLAST (2007 Launch)


Nick White

Usage & Data StatisticsUsage & Data Statistics

Gigabytes transferred per year1998 1999 2000 2001 2002 2003

Datasets byftp & http 844 1406 1880 2270 2917 3943

Science WebPages & Images 198 234 371 399 580 872

Main E&PO WebPages & Images 203 358 678 1010 3108 7581


Nick White

Data Transfers by Mission over Time


Nick White

Current Web Service Statistics• Archive volume: 3.3 TB (mostly compressed)

– Current ingest rate: 400 GB/yr– End of 2004 anticipated rate: 1.4 TB/year, with increased or new flows from

Swift, Integral, and XMM

• Missions in archive:Ariel-5, ASCA, BBXRT, BeppoSAX, Chandra, Compton, Copernicus, COS-B, DXS, Einstein, EUVE, EXOSAT,

Ginga, HEAO-1, HEAO-3, HETE-2, OSO-8, ROSAT, RXTE, SAS2, SAS3, Vela 5B, XMM-Newton

• Downloads (annual): 3.9 TB of data (by FTP and HTTP), 0.9 TB of science web pages (by HTTP) and 7.6 TB of E&PO web pages

• Browse: 355 local tables (+ ~3000 VizieR); 2,150,000 queries• SkyView: >60 surveys; >1,000,000 images generated

• 26,000 bibliography entries linking ADS to HEASARC datasets (ROSAT, ASCA, XMM-Newton, and some RXTE datasets)


Nick White

HEASARC Hardware Configuration

HEASARC(load balancer)

HEASARC12 GHz Dual Processor

HEASARC22 GHz Dual Processor

More slots available

12 TBNetwork Appliance

Archive

750 GBNetApp

Static web site and shared disks

Web servers

DLT Jukebox

Backup archive on 150 GB DLT tapes (copy to NSSDC)

Safehouse6 TB

AssortedStorage

Online archive backup

InternalMachines

Hera (x5)1 GHz

SkyView (x2)1 GHz

150 GBSurveys (x2)

240 GBExternal

Survey Cache

400 GBUser

Cache

DBMS1 on line

GSFC Portal

Linux, Solaris, Mac, Windows, …

Gig

ab

it

100 Mb

DBMS2 backup

Database (Sybase)2 GHZ dual Processors


Nick White

Web site maintenance• Main HEASARC web site has 15,000 static pages and

hundreds of dynamic pages (excluding the FTP area and data archive) comprising 7 GB in total

• Ensuring valid and up-to-date links pages requires constant attention– In the past, invalid links left unattended grew to ~500.– Concentrated effort has now eliminated all but a few broken or

out of date links (Newly broken links are fixed weekly)

• HEASARC is now 99% Section 508 compliant– Assuring appropriate access to HEASARC Web pages involved

changes to most Web pages– Compliance changes were often beneficial for all users.– Maintenance cost of 508 compliance is <=0.05 FTE effort


Nick White

Browse Catalog and Archive Interface

• Capable of querying all local and VizieR catalogs• Local and remote archives (e.g., CXC) • HTML, text, VOTable, FITS and Excel ouputs• New capabilities:

– Expressions allowed in search criteria ( fluxX/fluxY > 2)– Better integration with SkyView– Upload files of targets (in Browse 7.0)– Download queries for later re-use (7.0)– Dataset ID info added when available (7.0)– Links between tables


Nick White

Current Active Missions• Chandra

– Browse service fully linked to CXC archive• RXTE

– Dominates current archive downloads in terms of GB– Simple XTEIndex table developed and deployed (more simplification needs to be

done)• HETE2

– Some HEASARC-written FITS products for GRBs– Difficult to use most data delivered to archive

• Similarities to data recovery efforts of early HEASARC

• XMM-Newton– In 2003, second biggest HEASARC mission in archive volume & downloads– Archive contains both US & non-US Guest Observer data

• Integral– Use modified version of HEASARC archive software at ISDC

• HEASARC Browse can link to ISDC Browse tables– Archive at HEASARC will open in July 2004


Nick White

Preparations for Swift• New HEASOFT package• MOU delineating HEASARC responsibilities• End to end testing including archive ingest. • Customized mission interfaces:

– Individualized interface to Browse• HTML wrappers for standard Browse scripts

– Links between Browse tables– AKBAR: new service to import external information from

observers• HEASARC responsible for some high-level data products

– e.g., overall burst evolution• Mission requires rapid dissemination of data


Nick White

Swift Browse Interface


Nick White

Preparations for GLAST

• PDMP review

• Consultations on archive and catalog design

• Review proposed FITS formats


Nick White

Standards• FITS

– Subsumed GSFC FITS Office Web site– Develop and coordinate domain conventions for high energy data

(OFWG))• Organize periodic HEADCC meetings

– FITS Libraries Support• CFITSIO is essentially reference implementation of FITS• HEASARC also supports widely used FITS libraries in Java and IDL.• New C++ object implementation based on CFITSIO

• Standard Dataset IDs• VO Standards

– Browse support for VO Cone search– SkyView support for VO Simple Image Access Protocol– VOTable (and FITS) outputs available through Browse– VO Service Metadata descriptions for HEASARC resources

Publication of metadata using Open Archives Initiative (OAI) protocol


Nick White

Software Tools and Services• HEASARC software is frequently downloaded

– fv (FITS browser and editor): 500 downloads/month– CFITSIO library: 500/month– HEASOFT: 200-300/month

• HEASOFT– New packages for Swift and Astro E2 missions under

active development• Use HEADAS in order to provide a simple platform-

independent procedure for new software builds and updates

– XSPEC New major release (12.0) in alpha test• Complete re-write of Xspec internals in object oriented

fashion• Can support coded aperture mask data sets


Nick White

Hera• Provides a full, on-line analysis environment to users.

– Directly connected to archive, no downloads needed– Up to date versions of HEASOFT tools automatically available– Easily upload user data and download results.

• Uploaded data screened to ensure service is not used inappropriately.– Uses standard HEASARC FV FITS browser tool as user interface.

• Why?– Large volume of data (e.g., Integral, RXTE) downloads and/or software

installation may be infeasible for some users – With Hera, users can start using HEAsoft and other software without the

need for local installation• Status

– Initial system operational and being used actively.– Proposal for continued funding has been submitted– Both teacher and student interfaces in beta testing


Nick White

New and Updated Catalogs• ~24 new catalogs or major catalog updates in

last year.– Major correlative catalogs: VERONCAT, GB6, etc– High energy source catalogs: XMMSSC,

BMWHRICAT, INTREFCAT– Master catalogs updated regularly– GRBCAT: Compilation of all gamma-ray bursts

• VizieR connection for transparent access to massive catalogs (2MASS, USNO B, GSC 2)


Nick White

Education and Outreach• Education Web Sites:

– Imagine the Universe (Grades 6-12): 500K hits/month.

– StarChild (Grades 1-8) [in collaboration with L. Whitlock]: 1.25 M hits/month.

• New poster on origin of the elements• X-ray analysis school

– Highly successful and oversubscribed

• Host for APOD Web site• StarChild/Imagine/APOD CDs


Nick White

Other Projects• SkyView

– Java redesign– New surveys

(DSS2, Halpha, SFD) & SDSS soon.

– Request for continued funding submitted

• ClassX– Automated

classification of entire RASS and WGACAT samples of X-ray sources using supervised learning algorithm

Density of source requests in SkyView


Nick White

Virtual Observatory Involvement• Data Inventory Service• HEASARC metadata inventory

– Standardized metadata describing all HEASARC tables,data file types, surveys and services.

– Registered with VO to enable effective discovery of HEASARC resources.

– Table metadata

• HEASARC support for VO standards


Nick White

Data Inventory Service• Access to distributed services that use VO

protocols.– Many sites contributing data

• Dynamic service lookup using VO registries

• Catalog, observations, Images

http://heasarc.gsfc.nasa.gov/vo/

http://heasarc.gsfc.nasa.gov/vo/


Nick White

Near Term Missions and Deadlines

• Launches– Swift: Fall 2004– Astro-E2: Early 2005– GLAST: Early 2007

• HEASARC Senior Review Proposal Due: March 17, 2004


Nick White

Current and Upcoming IssuesScience

• May need greater emphasis on supporting gamma-ray astronomers as Integral and GLAST Archives come online and become an increasing fraction of the HEASARC’s archive

• Must deal with ever increasing complexity of data and calibration– Coded aperture masks (Swift, HETE2, Integral)– Complex source dependent response (“ + GLAST)– Multi-satellite observations (Con-X, LISA)


Nick White

Current and Upcoming IssuesHardware

• CPU and disk space needs achievable, but bandwidth is a potential limitation– Datasets may be growing faster than bandwidth– Demand for data will spike after GRB detections– Higher speed linkage to Internet possible but costly

• … and doesn’t help users who have slower connections

– Move towards more user processing at HEASARC• Hera, grid computing

• On-line backup for archive is presently in same building as primary HEASARC archive.– Planned move of backup archive to another building when space

and network is ready (Summer 2004)


Nick White

Current and Upcoming Issues

Local Software and Services• Integration of tools and services

– Strengthen SkyView/Browse, Browse/NVODIS, Hera/Browse, … cross-links.

– Enable coupling archive/web services and HEASOFT tools.

• Systematic attention to internal documentation of HEASARC systems

heasarc status and plans

Documents

data holdingscoordinate

elsewhereswiftchandra

data transfers

data restoration

data holdingdevelop

data setsmaintain

data analysis support

data selection software