metadata progress gridpp18 20 march 2007 mike kenyon
TRANSCRIPT
![Page 1: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/1.jpg)
Metadata Progress
GridPP1820 March 2007Mike Kenyon
![Page 2: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/2.jpg)
Metadata Team
• Glasgow– Mike Kenyon– Caitriana Nicholson– Tom Doherty– Paul Millar
• CERN/Glasgow– Helen McGlone
![Page 3: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/3.jpg)
ATLAS Event-level Metadata
• Navigation from Tag to events in AOD/ESD/RAW format
• Produced during AOD production– Oracle DBs at Tier0, Tier1– MySQL (probably) at Tier2
• ATLAS are implementing a Tag database to allow efficient identification and selection of events for analysis
• First-level cuts will be possible via queries on relational databases
![Page 4: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/4.jpg)
Tag Navigator Toolhttps://twiki.cern.ch/twiki/bin/view/Atlas/TagNavigat
orTool
• Need to integrate Tag DB with Distributed Data Management (DQ2) and Distributed Analysis (Ganga) tools
• Tag Navigator Tool (TNT) developed to plug that gap
• Aims to take user from query on Tag DB to analysis using results of query
![Page 5: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/5.jpg)
TNT Process Summary
• Runs a query on the database and outputs a local Root collection
• Divides this into a number of sub-collections
• Submits user jobs to LCG, one per sub-collection
• Output files can be registered as new DQ2 dataset
![Page 6: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/6.jpg)
TNT Process Summary
![Page 7: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/7.jpg)
Ganga Plugin
• TNT has also been implemented as a plugin to Ganga
• Ganga already allowed use of Tag files – TNT plugin is complementary to this
• Makes use of existing Ganga functionality for job submission, handling etc
• Will be available in Ganga release 4.3
![Page 8: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/8.jpg)
Future Plans
• Continue development of TNT and GangaTnt in response to user feedback
• Performance testing of Tags in TNT• Scalability and performance
measurements on Tag DB• Continue integration of Tags with AMI
![Page 9: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/9.jpg)
AMI-VOMS solution1. The grid user authenticates
themselves to AMI using a grid certificate.
2. AMI allows the user to launch a Java web start application to create a VOMS proxy.
3. The user chooses the x509 certificate to be sent to the VOMS server
4. The VOMS server sends back a VOMS proxy certificate to the Java application.
5. The VOMS proxy file is uploaded to the server, which hosts a proxy delegation service, via a secure web service attachment.
6. The user’s VO, group and role can be extracted from the proxy and used for authorisation purposes.
SourceForge project: Acacia
12
3
4
5
67
![Page 10: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/10.jpg)
DQ2/AMI integration• Use of delegated VOMS proxy certificate - Contact
application on behalf of user, e.g. DQ2.• Register a dataset simultaneously in AMI and DQ2• Other dq2 commands possible
Tags/AMI integration• Preliminary work on prototype database to allow
physics predicate cut on a collection and catalogue the results within AMI
![Page 11: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/11.jpg)
pyAMI
• AMI Web Service Client for Python• AMI tutorial available at:
http://ami3.in2p3.fr:8080/opencms/opencms/AMI/www/Tutorial/pyAMI.html
• Future development and maintenance• Providing AMI command wrappers for groups/tools such as
POOL Conditions, Muon and Tag collector• Code optimisation – replace SAX with DOM parsing of AMI
output• Reformatting output – interface to format XML
![Page 12: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/12.jpg)
The present: MonAMI v0.6
• “Universal sensor framework”– Capable of monitoring many different services– Can send data to any number of information
systems– Plugin structure
• v0.6 deployed at all ScotGrid sites & Grenoble
• Configuration details available at– http://www.gridpp.ac.uk/wiki/MonAMI
![Page 13: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/13.jpg)
Upcoming: MonAMI v0.7
• v0.7– Support for 64-bit (AMD64) architecture– Improved documentation– Major improvements to R-GMA support
• Robust against temporary R-GMA failures• Support for TLS-based secure upload of data• Support for proxy-certificate based authentication• “Drop in” deployment
– Support for event monitoring (e.g. DPM/GridFTP)
• Released pending solution of minor SL4-related issues
![Page 14: Metadata Progress GridPP18 20 March 2007 Mike Kenyon](https://reader036.vdocument.in/reader036/viewer/2022062619/5515c41e55034689058b46a7/html5/thumbnails/14.jpg)
The Future: MonAMI v0.8
• Planned features– New python plugin API– Web & RSS feeds– PostGreSQL monitoring