page 1jsoc overview august 2007 hmi status hmi is virtually done. –virtually similar to but not...
TRANSCRIPT
Page 1 JSOC Overview August 2007
HMI Status
• HMI is virtually done.
– Virtually similar to but not in fact
– Front window issue resolved, flight window now at LMSAL will be installed next week
– Light leaks to be fixed next week.
– Some electronics boards need conformal coating
– Some software issues remain
• Vibration and EMI/EMC testing done
• Thermal Balance and Thermal Vacuum testing starting week after next.
• In Air and Vacuum Calibration next few weeks.
• Pre-Ship Review set for 25 Sept.
• Ship date estimated 5 October.
• Launch date slip to probably 15 Jan 2009 pending review at HQ.
• Some serious implications from delay
Page 2 JSOC Overview August 2007
JSOC-SDP Status
• Capture system complete, waiting testing with DDS at White Sands
• DRMS and SUMS virtually done
• Level-0 work progressing, to be done by delivery
• Level-1 work to start after delivery
• Getting serious about work on basic pipeline modules
Page 3 JSOC Overview August 2007
HMI & AIA JSOC Architecture
Science TeamForecast Centers
EPOPublic
Catalog
Primary Archive
HMI & AIAOperations
House-keeping
Database
MOCDDS
Redundant Data
Capture System
30-DayArchive
OffsiteArchiv
e
OfflineArchiv
e
HMI JSOC Pipeline Processing System
DataExport& WebService
Stanford
LMSAL
High-LevelData Import
AIA AnalysisSystem
Local Archive
QuicklookViewing
housekeeping GSFCWhite Sands
World
Page 4 JSOC Overview August 2007
JSOC Dataflow Rates
Dataflow (GB/day)
Joint Ops
ScienceArchive440TB/yr(Offiste)
Data Capture
2 processors each
1230
1610
HMI &AIA Science
Hk
0.04
30d cache40TB each
Quick Look
LMSAL secure host
Level 0(HMI & AIA)
2 processors
75
Level 1(HMI)
16 processors
Online Data
325TB+50TB/yr
HMI High LevelProcessingc. 200 processors
HMI Science Analysis Archive 650TB/yr
Redundant data capture system
1210
1210
Data Exports
1200
LMSAL Link(AIA Level 0, HMI Magnetograms)
240
1610
1820
1230
rarelyneeded
1230
2 processorsSDO Scientist &User Interface
Dataflow (GB/day)
Joint Ops
ScienceArchive440TB/yr(Offiste)
Data Capture
2 processors each
1230
1610
HMI &AIA Science
Hk
0.04
30d cache40TB each
Quick Look
LMSAL secure host
Level 0(HMI & AIA)
2 processors
75
Level 1(HMI)
16 processors
Online Data
325TB+50TB/yr
HMI High LevelProcessingc. 200 processors
HMI Science Analysis Archive 650TB/yr
Redundant data capture system
1210
1210
Data Exports
1200
LMSAL Link(AIA Level 0, HMI Magnetograms)
240
1610
1820
1230
rarelyneeded
1230
2 processorsSDO Scientist &User Interface
Page 5 JSOC Overview August 2007
JSOC-SDP Major Components
Page 6 JSOC Overview August 2007
JSOC Pipeline Processing System Components
Database Server
SUMSStorage Unit
Management System
DRMSData Record
Management SystemSUMS Tape Farm
SUMS Disks
Pipeline Program, “module”
Record Manage
ment
Keyword Access
Data Access
DRMS Library
Link Manage
ment
Utility LibrariesJSOC Science
Libraries
Record Cache
PUIPipeline User
Interface
Pipeline processing
plan
Processing script, “mapfile”
List of pipeline modules with needed datasets for input, output
Pipeline Operato
r
Processing History Log
Page 7 JSOC Overview August 2007
Data Record Management Service
(DRMS)
Data Record Management Service
(DRMS)
Pipeline client-server architecture
JSOC Disks
Analysis codeC/Fortran/IDL/Matlab
JSOC Library
Record Cache (Keywords+Links+Data paths)
OpenRecordsCloseRecords
GetKeyword, SetKeywordGetLink, SetLink
OpenDataSegmentCloseDataSegment
Pipeline client process
JSOC DisksJSOC DisksJSOC Disks
Data Record Management Service
(DRMS)
Storage Unit Management Service
(SUMS)
Tape ArchiveService
SeriesTables
Record Catalogs
Storage UnitTables
Record CatalogsRecord
Tables
PostgreSQL DatabaseServer
AllocUnitGetUnitPutUnit
SQL queries
SQL queries
SQL queries
Storage unit transfer
Data Segment I/O
Storage unit transfer
Generic file I/O
DRMS socket protocol
Page 8 JSOC Overview August 2007
Pipeline batch processing
• A pipeline batch is encapsulated in a single database transaction, “DRMS session”:
– If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session
– If failure occurs all data records are deleted and the database rolled back
– It is possible to commit data produced up to intermediate checkpoints during sessions
DRMS Service = Session Master
Input data records
Output data records
DRMS API
Registersession
DRMS API
Module 1
DRMS API
Module 2.2
…
DRMS API
Module N
DRMS API
Commit Data&
Deregister
Pipeline batch = atomic transaction
Record & SeriesDatabase SUMS
DRMS API
Module 2.1
Page 9 JSOC Overview August 2007
HMI module status and MDI heritage
DopplerVelocity
HeliographicDoppler velocity
maps
Tracked TilesOf Dopplergrams
StokesI,V
ContinuumBrightness
Tracked full-disk1-hour averagedContinuum maps
Brightness featuremaps
Solar limb parameters
StokesI,Q,U,V
Full-disk 10-minAveraged maps
Tracked Tiles
Line-of-sightMagnetograms
Vector MagnetogramsFast algorithm
Vector MagnetogramsInversion algorithm
Egression andIngression maps
Time-distanceCross-covariance
function
Ring diagrams
Wave phase shift maps
Wave travel times
Local wave frequency shifts
SphericalHarmonic
Time series
Mode frequenciesAnd splitting
Brightness Images
Line-of-SightMagnetic Field Maps
Coronal magneticField Extrapolations
Coronal andSolar wind models
Far-side activity index
Deep-focus v and cs
maps (0-200Mm)
High-resolution v and cs
maps (0-30Mm)
Carrington synoptic v and cs maps (0-30Mm)
Full-disk velocity, sound speed, Maps (0-30Mm)
Internal sound speed
Internal rotation
Vector MagneticField Maps
MDI pipeline modules exist
Standalone production codes in use at Stanford
Research codes in use by team
Codes to be developed at HAO
Codes being developed in the community
Codes to be developed at Stanford
Primary observables
Intermediate and high level data products
Page 10 JSOC Overview August 2007
AIA Level-2
Page 11 JSOC Overview August 2007
JSOC Data Volumes from Proposal
this version modified to show the links to the hardware plan
img size channels
cadence
compress
HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90 49
AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90 59
HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180 93
AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900 2,004
HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0 0
AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39 0
HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0 0
AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0 0
HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0 0
AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0 0
HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0 0
AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0 0
HMI: tlm SU 553 100% 198 24
AIA: tlm SU 674 100% 241 30
HMI: Lev0, Lev-1, All Higher SU 730 412 93
AIA: Lev0, Lev1a SU 1,296 743 2,004
HMI Totals 68 71 610 118
AIA Totals 146 77 984 2,034
Combined (TB) 214 148 1,594 2,151
Tape shelf size (TB) 7,968
Tape shelf number of tapes - mixed density 11,257
Export
Near-line retain days
Near-line Cache (TB)
Tape Archive Fraction
Totals
2,026Local tape
LMSAL Link 1,193
1,227Offsite tape
Higher level 286
Level-1
Data Path Assumptions Combined (GB/day)
Fixed Disk cache (TB)
Online disk cache days
Perm disk per year (TB)
Level-01,610
Volume (GB/day)
Processed at
In from DDS
Tape per year (TB)
1,227
1,210
797
Page 12 JSOC Overview August 2007
Stanford JSOC effort plan
HMI-SU Effort Distribution
0
100
200
300
400
500
600
700
800
900
1000
FY2006Q1
FY2006Q2
FY2006Q3
FY2006Q4
FY2007Q1
FY2007Q2
FY2007Q3
FY2007Q4
FY2008Q1
FY2008Q2
FY2008Q3
FY2008Q4
FY2009Q1
Pe
rce
nt
FT
E
On-Demand Support
Irradiance
Forecast - farside…
Coronal Inferences
Mag Field - Vector Field
Mag. Field - Line-of-Sight
Level 2 - Local HS Holography
Level 2 - Local HS Ring Diagram
Level 2 - Local HS Time Distance
Level 2 - Global HS
Level 2 - Quick Look
HK & FDS
Level-0
JSOC Verification & Test
Data Quality and Proc. Metadata
General Env.: cvs,os,oracle,compile
Archive h/w arch.
Processing Hardware Arch.
User Tools: API, Data Export, & Browsing
pui
Data Capture
DRMS
SUMS
Support for AIA