solar orbiter sowg 5| richard carr| low latency pipeline hosting and delivery architecture | slide 1...
TRANSCRIPT
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 1
ESA UNCLASSIFIED – For Official Use
Solar Orbiter SOWG 5
Low Latency Pipeline Hosting and Delivery Architecture
Richard Carr,ESAC, July 2014
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 2
ESA UNCLASSIFIED – For Official Use
Objective and Format
• Recap reasoning and proposal from guidelines document
• Address any questions from instrument teams
• Questions from SOC
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 3
ESA UNCLASSIFIED – For Official Use
Background Reasoning
Advantages of Virtual Machines
1. Lifetime of platform.
a. Not a short mission.
2. Greatly simplifies deployment.
a. At SOC.
b. But also multiple copies in development/testing.
3. Symmetrical mechanism across instruments.
4. Cloneable, notably for diagnostics.
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 4
ESA UNCLASSIFIED – For Official Use
Objectives
Essentially attempt to define 3 interfaces
1. How the LL pipelines will retrieve the necessary input data.
a. Including auxiliary inputs.
2. How, where and in what form the LL pipelines will store the generated products.
3. How, and in what form, each instrument team will deliver the pipeline software to the SOC.
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 5
ESA UNCLASSIFIED – For Official Use
Objectives
Further interfaces…
1. External control of pipelines.
a. How to ‘kick’ etc.
2. Diagnostic.
a. Logging
b. Potential remote access to clone.
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 6
ESA UNCLASSIFIED – For Official Use
Specific Objectives
What we (at least the SOC) want from an architecture
1. Uniformity across instruments.
2. Data fetching external to pipeline.
a. Partly derived from 1.
b. Allows simpler control of pipelines, and coordinated SOC/MOC data interactions.
3. Appropriate products. (See further slide).
4. Allow post-processing.
a. Again, as uniformly as possible.
5. Testability.
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 7
ESA UNCLASSIFIED – For Official Use
Specific Objectives
Appropriate products
1. Allow crude checks of instrument performance and science data quality.
2. Allow instruments implementing selective data downlink to choose the telemetry to retrieve.
3. Allow the improvement of pointing profile and/or retargeting when tracking solar features.
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 8
ESA UNCLASSIFIED – For Official Use
Objective/Constraint
Relative Isolation of Pipelines
• Constraint on SOC due to computer infrastructure security concerns.
• But also meets various objectives
• Architectural simplicity.
• Configuration control.
Understood that diagnostic mechanisms needed.
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 9
ESA UNCLASSIFIED – For Official Use
Proposal
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 10
ESA UNCLASSIFIED – For Official Use
1. Virtual Appliances.
a. x86.
2. OVF delivery.
3. OS supported by VMWare.
4. Licencing
a. In principle responsibility of instruments.
b. Possible exceptions for practical reasons.
5. I/O Exclusively to disk.
a. NFS baseline with other support if essential (CIFS/iSCSI)
Proposal
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 11
ESA UNCLASSIFIED – For Official Use
6. Processing Batch = Pass Volume.
a. Preceding pass data also available.
7. Statelessness.
8. Request mechanism.
9. Output.
Proposal
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 12
ESA UNCLASSIFIED – For Official Use
Proposal
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 13
ESA UNCLASSIFIED – For Official Use
Proposal
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 14
ESA UNCLASSIFIED – For Official Use
Proposal
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 15
ESA UNCLASSIFIED – For Official Use
SOC Deployment Context
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 16
ESA UNCLASSIFIED – For Official Use
Testing
Not Yet Addressed
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 17
ESA UNCLASSIFIED – For Official Use
Anything not yet answered?
?
Instrument Team Questions
Solar Orbiter SOWG 5| Richard Carr| Low Latency Pipeline Hosting and Delivery Architecture | Slide 18
ESA UNCLASSIFIED – For Official Use
1. How similar can LL Pipeline be to science pipeline? Just cut-down version?
2. Which OS do you envision using?
3. How much software are you inheriting from previous instruments?
4. Which virtualisation infrastructure are you accustomed to using/is in use at your institutions?
SOC Questions