a tlas m etadata i nterface. portal page. ami2p3.fr
DESCRIPTION
A TLAS M etadata I nterface. Portal page. http://ami.in2p3.fr/ A generic cataloging framework – used in ATLAS for dataset discovery. AMI. Data in AMI is organized in CATALOGUES. Catalogues are assigned to a project and sub-project. mc08-Production (Holds 2008 MonteCarlo datasets) - PowerPoint PPT PresentationTRANSCRIPT
• ATLAS Metadata Interface.• Portal page. http://ami.in2p3.fr/• A generic cataloging framework – used
in ATLAS for dataset discovery
AMI
1
•Data in AMI is organized in CATALOGUES. Catalogues are assigned to a project and sub-project.
• mc08-Production (Holds 2008 MonteCarlo datasets)• data08-RealData (cosmics and first beam)
Dataset Discovery• Means finding the names of valid datasets to use in
your analysis.• An ATLAS (container) dataset is either
– A number of files output of DAQ, and managed by Tier 0. (RAW data and fast reconstruction)
– A number of files produced by one or more production system tasks. Each task makes a set of (TID) datasets. (MC and reprocessing.)
• The distributed data management (DDM) manages datasets as containers of task datasets.
• Physicists doing analysis normally should worry only about the containers, and let the infrastructure take care of the rest.
2
An introduction to datasets and their nomenclature
3
Project.datasetNumber.physicsShort.productionStep.dataType.version[/]
dataNN_* or mcNN_*
ESD, AOD
Concatanation of configurations
Project.runNumber.streamType.productionStep.dataType.version[/]
digit, recon, merge, etc…
mc09.106099.PhojetCdiff.evgen.EVNT.e456
4
5
Where does AMI get its data?• Real data : From the Tier 0 : DAQ data, and first
reconstruction is registered in AMI < 5 minutes after it is declared “ready” (both datasets and files). Config tags for real data.
• Monte Carlo and reprocessing.– From the Task Request DB : Tasks, dataset names, MC and
reprocessing config tags– From the production DB : Finished tasks – files and metadata.
• From physicists.– M.C. GEN datasets– MC Dataset number info, physics group owner,…– DPD tags– …
A precision about REAL DATA registration in AMI
data09_somethingStart of Run. Tier 0 declares empty datasets in DQ2Tier 0 adds files to the dataset as they arrive from SFO (LB by LB)
End of Run. Dataset closed in DQ2. AMI Flag raised. Arrives in AMI within 5 minutes.
Users may find data in DQ2 several hours before AMI sees the dataset.
7
• Overview page :https://ami.in2p3.fr/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?linkId=246– Notice that not all datasets are VALID.
• Invalid because all tasks failed, declared bad, or deleted.• Invalid datasets are hidden by default.
– Project tags are controlled, for official datasets; they follow the nomenclature specification and are declared by data preparation coordination. (data09_7TeV for example)
• If you know the project tag this is perhaps the simplest way to get to the list.
8
The “Simple” search• Simple Search – search on part of the
name.– Example “L1Calo “– Note that “%” is used for wild carding.
• Search on keywords, and comments – a search in many fields – takes longer.
• Tutorial link: http://ami.in2p3.fr:8080/opencms/opencms/AMI/www/Tutorial/Simple_search_interface.html
9
The “Advanced” search
• Examples to try:– Physics group = “phys-beauty”
• Notice that there are results in several catalogues.
• Tutorial link: http://ami.in2p3.fr:8080/opencms/opencms/AMI/www/Tutorial/Advanced_search_interface.html
• This interface will de redesigned in the next few months.
10
Looking at the results.• Use any search interface to get
“physics_MinBias ” data for 2009• Use the “group by” under data_type to find
the AOD datasets. (409 2010-02-08)• “Filter” on the dataset name, or the
runNumber to find 142383.• http://ami.in2p3.fr:8080/opencms/opencms/A
MI/www/Tutorial/Exploring_the_results.html• http://ami.in2p3.fr:8080/opencms/opencms/A
MI/www/Tutorial/Refine_the_Results.html
11
Looking at the results some more• Click on “additional fields” . Notice the prodsysStatus.
Choose one which has “EVENTS_AVAILABLE”. (more)
• Click on “details” – to get the “child” elements.• Click on “event_range” to get the prodsys task details.• Click on “files” to browse the files.• Notice the “history” buttons (“bread crumb trail”). You
can even remove them.• Click on “Provenance” of dataset. You can use this to
navigate along the production chain. For REAL data one gets back to the RAW type. (For MC to EVGEN and even GEN input. From EVGEN you can see the jobOptions.)
12
Getting to the data• Look for an mc09 dataset with
“EVENTS_AVAILABLE” mc09_900GeV.105004.pythia_ddiff.merge.AOD.e466_s667_s668_d258_r1026_
r1051 .• Click on “DQ2”. This page shows an
aggregation of information from DQ2. You can subscribe to the dataset from this page if you like, going through the DDM permission system of course.
• AMI will also make you “dq2-get” commands, but in general you should not do it.
13
Config Tags• Example: e466_s667_s668_d258_r1026_r1051 (last field of
dataset name)• e466 -> event generation parameters
s667, s668 -> simulation parameters (simul.HITS, merge.HITS)
d258 -> digitizationr1026,r1051 -> reconstruction parameters
• Interpretation of Config tagshttp://ami.in2p3.fr/opencms/opencms/AMI/www/ReferenceTables/
• Searching starting from the Config Tag.http://ami.in2p3.fr/opencms/opencms/AMI/www/Tutorial/ConfigTags
• Comparing tags (try r1026 and r1051) from the simple search page.
14
AMI Accounts
• Logging on to AMI.– In general you do not need to log on to read (at the
moment)– You can make an AMI account to access a
personal page.– You must log on for any writing operation.– Once you log on to AMI you can make bookmarks.– Tutorial link :
http://ami.in2p3.fr/opencms/opencms/AMI/www/Tutorial/Other_AMI_basic_functionalities.html
15
Other stuff.• pyAMI. Everything in AMI can be obtained
from the python client. http://ami.in2p3.fr/opencms/opencms/AMI/www/Client/pyAMISecure_and_cmt
• Ad Hoc queries. (For really advanced users!) http://ami.in2p3.fr/opencms/opencms/AMI/www/Tutorial/Refine_the_search.html