the prototype tng long-term archive –– interoperability issues and interactions with the italian...
Post on 19-Jan-2016
215 Views
Preview:
TRANSCRIPT
The prototype TNG Long-Term Archive ––
Interoperability issues and interactions with the Italian GRID
project
F.Pasian, R.Smareglia – INAF/O.A.Trieste
L.Benacchio – INAF/O.A.Padova
“TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
Garching, 10-14 June 2002
Towards an Italian Virtual Observatory?
Joining the International Virtual Observatory Alliance
Struggling to convince your funding agencies that it is worth to devote manpower and funds to coordinate efforts within an international VO activity, while they are still not necessarily
convinced it is useful to develop archives
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
2
“Telescopio Nazionale Galileo” (TNG)
• 3.5 m Nasmyth telescope - NTT-based - La Palma• First light in 1998 - AaT since the beginning• Long integration / testing / commissioning phase• First generation instruments completed in 2000:
– optical imager (OIG)– low-resolution imager/spectrograph (DOLORES)– high resolution spectrograph (SARG)– near-IR camera/spectrograph (NICS)
• ADOPT@TNG, DIMM, Meteo• Assisted observations
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
3
The TNG LTA pilot project
• A pilot project is currently being finalized, aiming at demonstrating the feasibility of technical solutions for the construction of the Long-Term Archive (LTA) for the TNG building a prototype LTA
• Interfaces to TNG “archiving at the telescope” (AaT)• End of pilot phase foreseen to be end of June 2002• Financed by CNAA, results to be submitted to INAF
– INAF board on computing, archives and networks to produce a report in early July
4“TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
Garching, 10-14 June 2002
Scope of the pilot project
• Provide suggestions / plans / prototypes to: – allow the long-term storage of scientific and technical data
from the TNG – guarantee accessibility by the TNG Observatory staff and
by the scientific community to original and derived data – test practically the feasibility of deriving value-added
information from the stored data– provide tools supporting life cycle of observing proposals – provide interoperability with a number of existing
international data centers and archives – guarantee harmonization with other Italian projects related
to archiving of data of astrophysical interest
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
5
n a m e re so lverb a s ic in te ro p era b ilityre com m en d a tio ns
In te rop e rab ility
D a ta In te rfa ceD o cum e nt
D a ta M o d e l
R D B M S
O O D B M S
p a ra lle l R D B M S
D a tab a se
o n -line
q u a s i-o n -line
o ff-line
D a ta s to re
A rch ive s tru c tu red e fin it ion
T e chn ica l d a ta
S c ien ce d a ta
In g es tion to o ls
U IF
R e q u es t se rv ic ing
R e trie va l to o ls N e tw o rk
te le m e trysc ie nce fra m es
v isu a liza tion
ca lib ra tion
co n s iste n cy ch e cks
q u a lity co n tro l
su p po rt to T A C
d a ta m in ing
in s tr. tre n d an a lys is
p u b lic ou trea ch
A d d it ion a l se rv ices
T N G L o T A rP ro to type
F in a lR e p o rt
N a tio na l a rch ivesD a ta m in ingp ro je c ts
C o ord ina tion
Product Tree
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
6
Interoperability
• Direct participation in “Interoperability WG” of OPTICON
• Prototype LTA uses standards (e.g. VOTable) defined within “Interoperability WG” aims at being compatible with AVO development
• Catalogue of observations available at CDS for basic-level interoperability with other archives
• More in future LTA
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
7
Lessons (being) learned
• Data quality is an issue:– inconsistencies in data (esp. scientific exposures)– some information missing (documentation, observation
schedule, etc.)– improved observing procedures are possible – AI analysis of technical / engineering data identifies some
possible problems in T&I systems data quality (see G.Longo’s talk on Friday)
• An archive must be developed in synch with the observatory:– late development of LTA (even if prototype) late discovery of
inconsistencies
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
8
Coordination with Italian research Grid
• The Italian astronomical community will not build its own AstroGrid
• It will rather coordinate with nation-wide initiatives to foster a Grid infrastructure:– eScience Grid– IG-BIGEST
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
9
The eScience Grid in Italy (1/2)
• Proposal submitted in October 2001 to Ministry of Education University & Research (MIUR)
• Bid for ~15 M€ of available ~30 M€ funds; answer soon
Title: “Enabling Platforms for High-Performance Computational Grids Oriented to Scalable Virtual Organizations”Strategic Programme on Enabling IC Technologies for Knowledge Society
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
10
The eScience Grid in Italy (2/2)
• Objectives:– R&D Grid technological development project (CNR&INFN)– Research on optical networks (CNIT)– Deployment of an Italian eScience Grid infrastructure
(INFN) for:• Astrophysics• Biology• Computational Chemistry• Geophysics• Earth Observation
– International integration by INFN (following requirements)
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
11
Astrophysics in the Italian eScience Grid
• Four demonstrators:– distributed computing– accessing and processing VST data – accessing archives and databases (inter alia the prototype
TNG LTA)– accessing observational facilities as “data nodes”
• About 1 M€ (1/15 of the total MIUR request)
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
12
The Italian Grid for Business, Industry, Government, EScience&Technology IG-BIGEST
• New initiative leveraging from MIUR proposal and aiming at the common development of the IG-BIGEST
• Italian Participation to the Expression of Interest for an Integrated Project "ENABLING GRIDS AND E-SCIENCE IN EUROPE" (EGEE) in the context of 6th EU Framework
• All major Italian Scientific National Research Institutions, Universities and Computing Centres reached general consensus:
– to take part to the large European Initiative EGEE having CERN as leading partner and INFN as the organization representing the Italian Grid eScience comunity
– to have only one large Integrated Project aiming at the development and deployment of the Grid Infrastructure in Europe as Geant does for the Networking
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
13
The participants and the Italian Grid eScience infrastructure
• High bandwidth network provided by Garr-G Geant• High performance computing infrastructures provided by
National Research Institutions, Universities and Super- Computing Centers. Participants :
– INFN, CNR, INAF, INFM, INGV, ENEA……– Computing Centers: CINECA, CILEA, CASPUR….– Universities
• Università di Lecce, Università di Napoli, Università di Pisa, Università di Roma Tor Vergata, Centro Fermi Roma1, Università di Cosenza, Università di Genova, Università di Padova, Università di Politecnico Torino, Università di Salerno, Università di Messina
• Formal participation has been finalized
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
14
Conclusions (and roadmap?)
• TNG LTA Pilot Project has coordination duties (of national archives and with international activities)
• TNG prototype LTA is VO-compliant; so will be future archives (TNG LTA, VST, LBC, etc.)
• INAF board on computing, archives and networks will provide recommendations
• The goal is to have the Italian community joining AVO Phase-B
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
15
DTWRITEDTWRITEWR
user applicationuser application
TAGTAG
TDBTDB
Message Exchange Network
Data Network
Staging Disk
External Disk
DTINDTIN
DTFLUSHDTFLUSH
DTOUTDTOUT
database Data store
AaT
UIF
AaT
UIF
WSS-std instr. non-std instr.
User Disk
VERIFYVERIFY
commands/messagestelemetryscientific datadata descriptors
on-line archive
External processesExternal processes
AaT processAaT process Storageon-linestorage
off-line storage
observer’s CDsoff-line archives CDs
AS
IWS
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
16
INSTRUME
EXP_ID
OBSERVER
DATE_OBS
EXPSTART
EXPTIME
RA
DEC
AIRMASS
CRDELT1
CRDELT2
DETOFF1
DETOFF2
EXP_ID
DET_ID
EXP_ID
INS_MODE
OBS_MODE
OBS_TYPE
EXP_TYPE
FLT_ID
…
…
DET_ID
BITPIX
PIXSIZE
NAXIS
DETSIZ1
DETSIIZ2
GRM_ID
LMP_ID
SLT_ID
…
…
…
…
…
EXP_ID
$_filename
DATE_CR
NAXIS1
NAXIS2
$_path_fts
CD_ID
wss_sess_id
DATE_CR
TIME_CR
DATE_MD
TIME_MD
$_path
CD_ID
…
SA_FILE MDBT
SA_DET_IDSA_INS SA_DET SA_EXP
FITS database shared with Technical database
CD_DATE
NCOPIES
CD_ID
SA_CD
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
17
CD_DATE
NCOPIES
CD_ID
SESS_ID
DATE_CR
TIME_CR
DATE_END
TIME_END
$path
CD_ID
CD_TAB
MDBT
shared with FITS database
Technical database
VALUE
TAG
TAG
ACRON
TAG
HASHTAB
AAA database
ACRON
TAG
HASHTAB
XYZ database
…
STR
ACRONTAG
HASHTAB
CUB database
…
TIME
VALUE
TAG
TAG
STR
TIME
VALUE
TAG
TAG
STR
TIME
Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”
18
Cyclic operations scheme
from: di Serego et al., 26 Jan 1995
top related