tracking provenance information with the opus job...
TRANSCRIPT
![Page 1: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/1.jpg)
Tracking provenance information with the OPUS job controller
Mathieu Servillat
Observatoire de Paris - LUTH Paris Astronomical Data Centre
IVOA Santiago - Oct. 2017
![Page 2: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/2.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
OPUS: Observatoire de Paris UWS ServerMain features• IVOA standards
• Universal Worker System (UWS)• Provenance data model
• REST architecture • Python micro-framework: bottle.py
• Collaborative development • Git server at PADC (gitolite)• GitHub: https://github.com/mservillat/OPUS
• Tests and quality• Unit tests with unittest and webtest• Activity history with logging
Prototype available• https://voparis-uws-test.obspm.fr
2
![Page 3: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/3.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Job management at PADC
Available structure• Work cluster (Tycho)• Job scheduler (SLURM)
PADC projects• Web based clients
• Data access• Online data processing• Wrap simulation codes
Need a simple interface to computational resources
3
![Page 4: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/4.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Inner workings
• Separate job controller from work cluster• Wait for work cluster signals• Avoid (too many) status queries to work cluster
4
![Page 5: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/5.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Server main classes
5
• SQLAlchemy•SQLite•PostgreSQL•…
•SLURM•Local
• ActivityDescription•WADL• JSON
![Page 6: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/6.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Core Provenance Data Model
• Core concepts from the W3C PROV recommendations• Entity - Activity - Agent• Relations and roles = provenance information• W3C PROV has many more relations• IVOA Provenance connected to VO concepts and astronomy needs
http://www.w3.org/TR/prov-overview/30 April 2013
http://www.ivoa.net/documents/ProvenanceDM/
6
![Page 7: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/7.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
IVOA Provenance Data Model diagram
Activity ActivityDescription
Entity EntityDescription
Used
WasGeneratedBy
UsedDescription
WasGeneratedByDescription
Parameter ParameterDescription
Agent
WasAssociatedWith
WasAttributedTo
ActivityFlow
HadStep
Collection
HadMember
0..1*
*
1
WasInformedBy
1
*
1*
1*
1
*
1
0..1
1
*
1*
1
*
* 0..1
1
*
1
WasDerivedFrom
*
* 1
1
*
0..1
1
*
1 1
*
* 0..1
* 0..1
0..1*
1
*
1
*
*
1
*
1
Description side
Configuration
Groups ofactivities
Groups ofentities
(same in W3C)
7
![Page 8: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/8.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Serializations - ActivityDescription
—> DataLink Service Descriptor + Used/Generated groups
—> VOTable PARAM + attributes
8
![Page 9: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/9.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Client main features
• Javascript based + Python Flask• UwsLib.js: sends AJAX requests to the server• uws_manager.js: handles and displays responses
• Integration with Bootstrap3
• HTML pages with specified <div> elements(id=joblist, parameters, results…)
• Job definition editor• Interface to create the ActivityDescription file
• parameters, used and generated entities
• bash execution script
• Job manager• list jobs, create jobs, control jobs, view results
9
![Page 10: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/10.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Client - Job definition
10
![Page 11: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/11.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Client - Job list
11
Tracking of Provenance informations
![Page 12: Tracking provenance information with the OPUS job controllerwiki.ivoa.net/internal/IVOA/InterOpOct2017Apps/servillat_IVOA2017CL_OPUS.pdfInner workings • Separate job ... • Core](https://reader034.vdocument.in/reader034/viewer/2022051804/5fecebf1091e917de506e800/html5/thumbnails/12.jpg)
Mathieu Servillat (Observatoire de Paris) IVOA Santiago, Chile - Oct 2017
Serializations - W3C PROV formats
12