sdo/hmi data base

SPACEINN -‐-‐-‐-‐-‐-‐

SDO/HMI Data Base

Laurent Gizon, Ray Burston 21 October 2013, Paris

SPACEINN(helioseismology) at MPS

•  Kaori Nagashima (SPACEINN MPS coordinator) >Experience with SDO and Hinode data

•  Raymond Burston (MPS data base) >Experience with netDRMS, SDO formats

•  Science support: Aaron Birch (ex-‐NASA SDO Science Center), Jesper Schou (HMI Project ScienUst), Hannah Schunker (ex-‐HELAS)

SDO/HMI data

•  Global and local helioseismology applicaUons •  All SDO data = 2 TB/day •  SDO/HMI Lev.1 data = ca. 50 TB/year •  Calibrated data produced by JSOC •  SDO/HMI data automaUcally flowing into German Data Center (GDC)

•  Access to SDO/HMI for SPACEINN users: Through SPACEINN local helioseismology website or via netDRMS

Access #1 to SDO/HMI data SPACEINN local helioseismology website

www.mps.mpg.de/projects/seismo/SpaceInn/ •  Data download: sample data sets

> science ready tracked data cubes > more on request

•  InformaUon on data reducUon etc. •  Links: SDO-‐HMI (hmi.stanford.edu/) •  Links: JSOC (jsoc.stanford.edu/) > data export •  Links: Lockheed Heliophysics Knowledgebase (HEK) •  Need to link this work to other SPACEINN sites

Access #2: netDRMS (JSOC Data Record Management System)

•  Ray Burston will explain… •  netDRMS must be installed locally •  Data requests are forwarded to relevant data centers (GDC is Europe’s site for HMI)

•  Upon request (must be jusUfied), GDC can push some data series automaUcally to remote sites (e.g. KIS)

SDO/HMI Data in DRMS

Doppler Velocity

Heliographic Doppler velocity

maps

Tracked Tiles Of Dopplergrams

Stokes I,V

Filtergrams

ConUnuum Brightness

Tracked full-‐disk 1-‐hour averaged ConUnuum maps

Brightness feature maps

Solar limb parameters

Stokes I,Q,U,V

Full-‐disk 10-‐min Averaged maps

Tracked Tiles

Line-‐of-‐sight Magnetograms

Vector Magnetograms Fast algorithm

Vector Magnetograms Inversion algorithm

Egression and Ingression maps

Time-‐distance Cross-‐covariance

funcUon

Ring diagrams

Wave phase shig maps

Wave travel Umes

Local wave frequency shigs

Spherical Harmonic Time series To l=1000

Mode frequencies And splihng

Brightness Images

Line-‐of-‐Sight MagneUc Field Maps

Coronal magneUc Field ExtrapolaUons

Coronal and Solar wind models

Far-‐side acUvity index

Deep-‐focus v and cs maps (0-‐200Mm)

High-‐resoluUon v and cs maps (0-‐30Mm)

Carrington synopUc v and cs maps (0-‐30Mm)

Full-‐disk velocity, v(r,Θ,Φ), And sound speed, cs(r,Θ,Φ),

Maps (0-‐30Mm)

Internal sound speed, cs(r,Θ) (0<r<R)

Internal rotaUon Ω(r,Θ) (0<r<R)

Vector MagneUc Field Maps

HMI Data Data Product Processing

Level-0

Level-1

Something to consider •  netDRMS can potenUally be used as common SPACEINN

database sogware used for sharing all helio-‐ and asteroseismology data sets?

•  Advantage: already in place at MPS, KIS, IAS •  netDRMS excellent to share data among sites (soluUon

chosen for Plato data center) •  However, ingesUng new data series = work

Next: Ray Burston

German Science Center for SDO –  Dedicated data and scienUfic analysis center for SDO/HMI data –  Consists of two components

GDC-‐SDO: German Data Center for SDO –  Funded by the German Aerospace Agency (DLR), 2007 – 2016 –  Collects SDO/HMI data directly from JSOC (USA) and archives locally at MPS –  GDC-‐SDO is a master European data distribuUon center for HMI data –  Storage capacity: 192 TB online + 180 TB in near-‐line tape libraries (224 slots) –  To be expanded to > 500 TB online –  Main data products include all HMI Dopplergrams, l.o.s. magnetograms and

conUnuum intensity images = 125 TBs

MPS compu<ng resources •  Local access to SDO/HMI data via GDC-‐SDO (InfiniBand connecUvity) •  ERC SISI Compute cluster: 256 cores, 2 TB RAM •  To be extended to > 1000 cores •  Pegasus Workflow Managment System

NetDRMS

R Bogart (2007) Astron. Nachr.328, No 3/4, 352-‐355

NetDRMS Site 3

•  The Data Record Management System (NetDRMS) is developed by Stanford University with the aim of sharing SDO data worldwide

•  A PostgreSQL database stores meta-‐data, e.g. keyword name/value pairs in FITS headers

•  Storage Unit Management System stores/retrieves physical data (e.g. data arrays in FITS files)

•  Java Mirroring Daemon transfers the HMI images using high-‐performance scp

•  NetDRMS provides uUliUes for subscribing to a data series

•  SLONY-‐I is required publishing data series

NetDRMS

§  NetDRMS API consists of 250 funcUons to communicate with NetDRMS from C

§  Command line uUliUes are useful for searching your NetDRMS database

§  UUliUes are also useful to obtain info in languages other than C; IDL example below

NetDRMS

GDC-‐SDO Help Desk for SPACEINN

•  Located hxp://www.mps.mpg.de/projects/seismo/GDC-‐SDO/spaceinn_access.htm

•  Contains informaUon for sehng up NetDRMS and receiving relevant data

•  For user support, contact Ray Burston at [email protected]

Three steps for implemen<ng NetDRMS at your site

•  InstrucUons for installing NetDRMS are located jsoc.stanford.edu/netdrms, and includes details on sehng up PostgreSQL DB and subscribing to published data series

•  InstrucUons for installing the Java Mirroring Daemon (to transfer images) are located hxp://vso.tuc.noao.edu/VSO/w/index.php/Main_Page

•  InstrucUons for publishing data series are located jsoc.stanford.edu/jsocwiki/PublishSeries

NetDRMS

Objec<ves

•  Serve the community with HMI observaUons through the German Science Center for the Solar Dynamics Observatory (GSC-‐SDO)

•  Make relevant HMI data derivaUves arising from scienUfic working groups, as well as complementary data (e.g. GONG, MDI, JSOC pipeline science products, GSC-‐SDO pipelines, etc.), available via the GSC-‐SDO

•  Facilitate effortless and transparent data handling, thereby increasing the Ume available for innovaUve scienUfic data analysis.

WP 3.5 HMI-‐SDO Data Handling and DistribuUon

Tasks

•  IdenUfy community requirements

•  Maintain the Data Record Management System (NetDRMS) at GSC-‐SDO

•  Provide technical support for NetDRMS installaUon/maintenance at other sites

•  Monitor HMI data flow; check consistency with JSOC; ensure data integrity

•  Backup and secure HMI data, science derivaUves, and complimentary data

•  Create website for the disseminaUon of informaUon relevant to this project e.g. data access instrucUons

•  Gather relevant informaUon from the scienUfic community and adverUse to a much wider audience through the website

•  Establish help desk and user support infrastructure

Status

•  All tasks have commenced and they will require conUnuous axenUon or updates

•  Provided support to KIS for installing NetDRMS: interfaces are operaUonal


Planning, Needs and Ac<on Items

•  GDC-‐SDO will publish addiUonal relevant data series if required by the community

•  Calculate staUsUcs on data usage and data flow precisely and automaUcally

Relevant Deliverables

•  D3.16 1st report on help-‐desk and user-‐support operaUons (SDO): 12 months

•  D3.19 1st report on HMI data products and usage staUsUcs: 12 months


HMI/AIA Data Volumes

sdo/hmi data base

Documents