david malon, peter van gemmeren (argonne national laboratory) richard hawkings, (cern)
DESCRIPTION
An Inconvenient Truth: file-level metadata and in-file metadata caching in the (file-agnostic) ATLAS distributed event store. David Malon, Peter van Gemmeren (Argonne National Laboratory) Richard Hawkings, (CERN) R.D. Schaffer (LAL Orsay). 1. Abstract. - PowerPoint PPT PresentationTRANSCRIPT
David Malon, Peter van Gemmeren (Argonne National Laboratory)Richard Hawkings, (CERN)R.D. Schaffer (LAL Orsay)
1. Abstract
In the ATLAS event store, files are sometimes "an inconvenient truth." From the point of view of the
ATLAS distributed data management system, files are too small--datasets are the units of interest.
From the point of view of the ATLAS event store architecture, files are simply a physical clustering
optimization: the units of interest are event collections--sets of events that satisfy common conditions or
selection predicates--and such collections may or may not have been accumulated into files that contain
those events and no others.
2. Architecture
3. In-File Conditions Data Example
4. Provenance Metadata
An Inconvenient Truth: file-level metadata and in-file metadata caching in the (file-agnostic) ATLAS distributed event store
It is nonetheless important to maintain file-level metadata, and to cache metadata in event data files.
When such metadata may or may not be present in files, or when values may have been updated after
files are written and replicated, a clear and transparent model for metadata retrieval from the file itself or
from remote databases is required. In this paper we describe how ATLAS reconciles its file and non-file
paradigms, the machinery for associating metadata with files and event collections, and the
infrastructure for metadata propagation from input to output for provenance record management and
related purposes.
DataFile1
MetaDataStore
CondOutStream
1 1 1 1
DataFile2
2 2
InputMetaDataStore
1 1
InputMetaDataStore
2 2
FileIncident:•EventSelectorsends endFileincident•open next file•sends beginFileincident.
FlushStore
MetaDataTool:•Create MetaData (no Input)•Read and Combine MetaData•Write MetaData.
Write
Read
EventData
EventData
MetaDataSvc:•Retrieves all registered IMetaDataTools•Listens to beginFileand endFileincidents.•Reads next DataFileand provides transient addresses.
EventSelector: Fire FileIncidents
DataFile1
MetaDataStore
CondOutStream
1 11 1 1 11 1
DataFile2
2 22 2
InputMetaDataStore
1 11 1
InputMetaDataStore
2 22 2
FileIncident:•EventSelectorsends endFileincident•open next file•sends beginFileincident.
FlushStore
MetaDataTool:•Create MetaData (no Input)•Read and Combine MetaData•Write MetaData.
Write
Read
EventData
EventData
MetaDataSvc:•Retrieves all registered IMetaDataTools•Listens to beginFileand endFileincidents.•Reads next DataFileand provides transient addresses.
EventSelector: Fire FileIncidents
Use of In-File Metadata
Data Caching: It is convenient and more effective not to have to use (rely upon) a remote Database.E.G.: Trigger information could be stored in the event file
Stream-level Metadata: A summary of the characteristics of the Events within a Stream (File). Data which is common for many or all Events can be made easily accessible by writing them on a Stream (File) level.
E.G.: Number of Events, Run Numbers and Luminosity Blocks, Type of Events and Process Stage Information.
Writing In-File Metadata
In-File Metadata is written using a Conditions Stream “shadowing” the Event Data Output Stream writing Data Objects during finalize() at the end of the job into the Event Data file. Separate POOL container are used for the Metadata DataHeader and the Data Objects.
Reading In-File Metadata
The EventSelectorAthenaPool uses the Gaudi Incident Service to fire beginInputFile / endInputFile FileIncidents before a file is opened and after the end of file was found. The MetaDataSvc will listen to these FileIncidents and act as an AddressProvider for In-File Metadata. In order to access Metadata, the MetaDataSvc will create a POOL Implicit collection over the Metadata DataHeader. Than the MetaDataSvc creates a new InputMetaDataStore into which it reads File-Level Metadata when handling a beginInputFile incident and clears that store on an endInputFile incident. In addition, the MetaDataSvc will retrieve all registered (via jobOptions) IMetaDataTools to allow the Metadata Maker to be initialized to listen to the FileIncidents.
Event collections (sets of events that share common characteristics) are fundamental, and their metadata requirements do not depend upon whether the events of interest have been gathered into their own file set.
For event data files (“implicit” collections), provenance metadata sufficient to support cross-section calculation can be stored within files and propagated correctly through the downstream workflow, even if that workflow further culls the event selection.
Event Collection
List of references to the Events satisfying the selection criteria.
MetaData for collectionprovenance
Event File
= “Implicit” Collection
In-File MetaData for collection provenance
Persistent DataObjectsfor the Events
Tag Database
Query
Event Collection
List of references to the Events satisfying the selection criteria.
MetaData for collectionprovenance
Event File
= “Implicit” Collection
In-File MetaData for collection provenance
Persistent DataObjectsfor the Events
Event File
= “Implicit” Collection
In-File MetaData for collection provenance
Persistent DataObjectsfor the Events
Persistent DataObjectsfor the Events
Tag Database
Query
Event File
EventData
IOV Db Svc•Resolve TimeStamp to IOV.•Provide Proxies for Conditions Data.•Retrieve Conditions DataObjects
Check forIn-File IOV
IOVIn-File
Provide Proxy Information for Conditions Data
Cond DBIOV DB
Reconstruction /Analysis
Check forIn-File Cond
Retrieve Conditions DataObjects
Cond.In-File
Read Conditions Data
Read Event Data
Event File
EventData
IOV Db Svc•Resolve TimeStamp to IOV.•Provide Proxies for Conditions Data.•Retrieve Conditions DataObjects
Check forIn-File IOV
IOVIn-File
Provide Proxy Information for Conditions Data
Cond DBIOV DB
Reconstruction /Analysis
Check forIn-File Cond
Retrieve Conditions DataObjects
Cond.In-File
Read Conditions Data
Read Event Data
CachingMany kinds of interval-of-validity data (e.g., trigger configurations), while authoritatively managed externally, may be cached within event data files for efficiency
Architecture is robust: if such data are not found within the file, fallback to conditions database occurs.
1
DataFile
POOLContainer_DataHeader
CollectionTree
DataHeaderDataHeader
DataHeader
abcabc
abc
StreamColl_DataHeader
Stream
DataHeader 1referencesreferences
1
DataFile
POOLContainer_DataHeader
CollectionTree
DataHeaderDataHeader
DataHeader
DataHeaderDataHeader
DataHeader
abcabc
abc
abcabc
abc
StreamColl_DataHeader
Stream
DataHeader 1referencesreferencesreferences
In the ATLAS event store, files are sometimes "an inconvenient truth." From the point of view of the
ATLAS distributed data management system, files are too small--datasets are the units of interest.
From the point of view of the ATLAS event store architecture, files are simply a physical clustering
optimization: the units of interest are event collections--sets of events that satisfy common conditions or
selection predicates--and such collections may or may not have been accumulated into files that contain
those events and no others.