dmd data flow operations david silva data management and operations division
DESCRIPTION
DMD Data Flow Operations David Silva Data Management and Operations Division. Data ingestion & delivery Chile sites, WFCAM, surveys Data quality control and processing (Paranal) Archive/database content management Data Interface Control Board (DICB) chairman, members - PowerPoint PPT PresentationTRANSCRIPT
ESO-Wide Review 2005
DMD Data Flow Operations
David SilvaData Management and Operations Division
ESO-Wide Review 2005
http://archive.eso.orghttp://www.eso.org/qc
Mission: ESO Archive Operations
• Data ingestion & delivery– Chile sites, WFCAM, surveys
• Data quality control and processing (Paranal)
• Archive/database content management– Data Interface Control Board
(DICB) chairman, members
• Archive systems administration• Database administration• HST and VO support• Remedy and Web administration
ESO-Wide Review 2005
DFO Personnel
Missing: Mignani (QC),Percheron (QC), Sjøberg
14 ISM, 11 CFTE
ESO-Wide Review 2005
2004 Highlights
• Department re-organization– Good-bye and thanks to Benoît
Pirenne!– Merged OTS and DFO/QC– Absorbed Web/Remedy
administration– Transition to Service Contract
• Before: many 1-on-1 contracts• Now: Data Conversion Services • Duration: 3/2004 - 3/2007 (option:
3/ 2009)• One year summary: going well
• New services– ESO library database interface– Proposal abstracts on-line– Archive query forms for [FILL IN]– Data Interface Control Document 3.0– QC services for MIDI
• Selected numbers– More than 100 hardware servers– Over 8 hours of Venus Transit Web
event, 1.5 TB and 45 million web hits served
– Ops_log database warehouse reached 1 billion rows in March 2004
– Archived ~1 million files, approximately 1/3 of all files in Archive
– Over 150 000 QC processing jobs (VIMOS = 54 000), 5 Tbytes processed, 1364 DVDs delivered
ESO-Wide Review 2005
2005 Outlook
• Archive opening to world– Per endorsement by Council– Monday 4 April 2005– Improved Archive interface(s)
(see VOS presentation)
• WFCAM data ingestion– UK in-kind, raw data, 6 TB/year
• New systems– Database server migration
• New hardware, Sybase ASE 12.5.3
– Web servers to Linux– New compute cluster– New DFO/QC tools– Net-based data transfer (again?)
• New QC Services– Apr: SINFONI, VISIR– Oct: AMBER– Improved science products– Expand to Visitor Mode (?)
• New Archive services– Support for VO
• Collaborate with VOS department
– Better ToO/RRM support– Better SCI/CAL association– Ingest & serve science products
• Prepare for VST/OmegaCam– Date volume/processing
accelerando begins…
ESO-Wide Review 2005
Data Volume Challenge
0.00
200.00
400.00
600.00
800.00
1000.00
1200.00
1400.00
1600.00
1800.00
Tera
Byt
es (
com
pre
ssed
)
2004 2005 2006 2007 2008 2009 2010 2011 2012
Other WFCAM VST VISTA ALMA
HST: 0.2 TB/yearVSTCam: 30 TB/year VISTA: 150 TB/year ALMA: 180 TB/year
ESO-Wide Review 2005
Data Processing Challenge
• Drivers: volume, complexity, demand– More pixels = more FLOPS
• Example: Cam, VISTA
– More processing per pixel = more FLOPS• Example: VIMOS, SINFONI
– More VO re-processing = more FLOPS
• Can be: volume and complexity– In terms of required processing
power: SINFONI:MUSE = 24
FLOPS = floating-point operations per second
• Required DFO processing power – UVES: 1.0 (actual, inc. fibre)– VIMOS: 2.2 (actual)– All VLT/VLTI (2004): ~ 6– Cam: ~ 21– VISTA: ~ 49– Based on Wicenec, Hanuschik,
Knudstrup, & Dolensky, 2005)
• Related issues– Peak loads– VO demand– 2nd generation VLT/VLTI instruments
ESO-Wide Review 2005
ESO Data Center: 2010
Storage volume: 1.2 PetaByte (one copy)Inflow: ~0.25 PB/yearMix of slow-access and fast-access mediaNeed 2nd secure copy
In addition: database server(s), etc.
100+ processors: 2006: 100 UVES equivalent 2010: 1000 UVES equivalent
Floor space: ~100 m2