euclid consortium ou-le3 kom meeting, london 1march 22-23, 20121 ou /sdc proposed development...

EuclidConsortium

OU-LE3 KOM Meeting, London 1March 22-23, 2012 1

OU /SDCProposed

Development StrategyMaurice Poncet (CNES, SDC-FR Lead)

Andrea Zacchei (Inaf – SDC-IT Lead)Keith Noddle (IfA – SDC-GB Lead)

EuclidConsortium

OU-LE3 KOM Meeting, London 2March 22-23, 2012

Euclid – Ground Segment Concepts

The recurring acronyms of the ground segment:

EuclidConsortium


Euclid – Ground Segment Concepts

OUs (Organization Unit) :Responsible for the definition and prototyping of a given

pipeline (i.e transform scientific req. from SWG into algorithms or mathematical methods)

Responsible for the validation of a given pipeline (i.e providing baseline test data)

SDCs (Science Data Centers) :Responsible for the S/W development of pipelines : SDC Dev

Responsible for the H/W processing infrastructure : SDC Prod

Responsible for the pipeline processing operations : SDC Prod

OUOU

SDC DEVSDC DEV

SDC PRODSDC PROD

EuclidConsortium


SDCs - Development Strategy

• SDC-xx DEV Team is a “delocalized” team

– Distributed over several labs

– Integration of other countries contributions

• This team is lead by only one responsible which is at SDC-xx Dev level

• SDC-xx DEV Team works through a collaborative web based development infrastructure

• A given SDC-xx DEV Team may contribute to SDC of other countries (under their responsibility)

• There is one dev team per pipeline

• SOC should be involved as any other SDC

4

EuclidConsortium


SDCs – Implementation Involvement

• L : Lead

• C : Contribution (as corresp. OU Co lead)

• c : other contribution

ImplementationLE1 VIS NIR SIR EXT SIM MER SPE LE3 SHE PHZ

SDC-CH C LSDC-DE L L C cSDC-ES L CSDC-FR C L C c C L L LSDC-GB C C LSDC-IT C L L L C CSDC-NL C LSDC-FI cSOC Coord

EuclidConsortium


SDCs – SDC Generic WPs

WP-4-4-nnSDC-<NAT>WP-4-4-nn

SDC-<NAT>

WP-4-4-nn-1000SDC-<NAT> Management

WP-4-4-nn-1000SDC-<NAT> Management

WP-4-4-nn-2000SDC-<NAT>-DEVWP-4-4-nn-2000

SDC-<NAT>-DEVWP-4-4-nn-3000

SDC-<NAT>-PRODWP-4-4-nn-3000

SDC-<NAT>-PROD

WP-4-4-nn-2100SDC-<NAT>-DEV Infra.

WP-4-4-nn-2100SDC-<NAT>-DEV Infra.

WP-4-4-nn-2200SDC-<NAT>-DEV Implementation

WP-4-4-nn-2200SDC-<NAT>-DEV Implementation

WP-4-4-xx-2300SDC-<NAT>-DEV Integration

WP-4-4-xx-2300SDC-<NAT>-DEV Integration

WP-4-4-nn-22sb<OU> pipeline Implementation

WP-4-4-nn-22sb<OU> pipeline Implementation

WP-4-4-nn-3100SDC-<NAT>-PROD Infra.

WP-4-4-nn-3100SDC-<NAT>-PROD Infra.

WP-4-4-nn-3200SDC-<NAT>-PROD Operation

WP-4-4-nn-3200SDC-<NAT>-PROD Operation

WP-4-4-xx-23sb<OU> pipeline Integration

WP-4-4-xx-23sb<OU> pipeline Integration

WP-4-4-nn-32sb<OU> pipeline operation

WP-4-4-nn-32sb<OU> pipeline operation

WP-4-4-nn-22sb-st<OU> pipeline Implementation subtask

WP-4-4-nn-23sb-st<OU> pipeline Integration subtask

WP-4-4-nn-32sb-st<OU> pipeline operation subtask

6

Feel free to propose any other set up.

EuclidConsortium


SDC-DEV – Dev Environment

Local Infra

DevCompilation

Unit Test

SDC-DEV

Local Infra

DevCompilation

Unit Test

Local Infra

DevCompilation

Unit Test

Local Infra

DevCompilationUnit Tests

S/W int tests

e – Infra

Conf Mangt

ExtractionGeneration

(Preint) testsQuality CheckDash Board

SDC-PROD

IntegrationValidation

DeploymentGenerationInt/Val testsPerf. Test

Quality CheckDash Board

Production

Data IngestionProcessing

Data Distribution

SDC

LabsLocated

ComputingCenter(s)

7

EMAEC level S/W conf mangt)

EuclidConsortium


SDCs - Development Environment & Context• Tailored (i.e. simplified & adapted) ECSS process should be applied

• Linux platform

• Model based Approach : SysML

• Scripting language : Python

• Coding language : C/C++

• Distributed Development (cf. open source best practices)

• Test driven approach

• Iterative & Incremental process (Agile Approach)

• Continuous S/W Integration

• Strong configuration management

• Automation (extraction, build, test, code quality check, …)

• Dashboards (motivating !)

• Online tools

• Collaborative documentation

8

EuclidConsortium


OU / SDC-DEV – Agile Approach (1/2)• Iterative & Incremental development

– Sprints (e.g. monthly or less) : sub functions

– Release (e.g. half year or less)

=> S/W available in early phases, ability to adapt the S/W

EuclidConsortium


OU / SDC-DEV – Agile Approach (2/2)• OU is “Product Owner”:

– Backlog management : Prioritized list (e.g. priority, maturity, …) of pipeline “features”,

– Definition of releases and sprints contents in coordination with SDC-DEV team (e.g. according to SDC-DEV capability),

– Validation of pipeline with SDC-DEV team.

• SDC-DEV team responsible of sprint & release dev.

OUOUSDC DEVSDC DEV

EuclidConsortium


Continuous Integration Principles

* Maintain a code repository

* Automate the build

* Make the build self-testing

* Everyone commits to the baseline every day

* Every commit (to baseline) should be built

* Keep the build fast

* Test in a clone of the production environment

* Make it easy to get the latest deliverables

* Everyone can see the results of the latest build

* Automate deployment

EuclidConsortium


SDCs - Development Phase

• Pipeline Development in respect with SGS and System rules– ECSS tailoring (documentation, dev cycle)

– UML modeling (common look & feel)

– Pipeline I/F

– IAL principles

– Products, Data modeling with modeling tool

– Quality rules (check should be as much as possible integrated and automated inside the collaborative platform)

• Potential common S/W modules should be submitted at system level

12

EuclidConsortium


SDCs – Integration Phase

• “target” pre-integration may be done at SDC-DEV side on virtual machines being representative of target configuration.

• Target Integration should be done on SDC-Prod infra. in a dedicated Integration « sand box »

• Integration should be achieved in close cooperation of SDC-DEV, SDC-PROD and OU teams

• “Technical” tests designed at SDC level

• “Functional” tests designed at OU level

• Integration tests output data should be accessible to above teams for validation and further investigation

13

EuclidConsortium


SDCs – Validation Phase

• Validation should be done on SDC-Prod infra. in a dedicated Validation « sand box »

• Integration should be achieved in close cooperation of SDC-DEV, SDC-PROD, OU teams, and as needed SWG.

• “Scientific” tests designed at OU/SWG level

• Validation tests output data should be accessible to corresponding OU team and SWG for validation and in case of failure for further investigation.

14

EuclidConsortium


SDCs – Collaborative Development Platform

• A Collaborative Development Platform should be set up:

– Aims : verify tools readiness, simulate dev cycle, may be used for SGS mock-up.

– O/S : Linux

– Languages : Python, C++

– Configuration : SVN

– Generation : Maven

– Test s : xUnit

– Integration : Jenkins

– Quality : Sonar (with C++ plug-in)

– IDE : Eclipse, Netbeans, vi, whatever…

– Tracking : Jira, Redmine

– Doc Gen. : Doxygen

– Modeling : Sparx Enterprise Architect

– Deployment : IzPack

15

EuclidConsortium


SDCs - Collaborative Devlopment Platform

Eclipse

Jenkins

Sonar

Jira & co

16

EuclidConsortium


Tools and …. Rules

But, tools are not sufficient, needs for :

• Rules (e.g. naming conventions, formats, directory trees, …)

• Guidelines

• Code, Doc templates

• Good practices

• User support

At SGS level

17

EuclidConsortium


SDCs – Typical Product Tree

18

SDC-xxSDC-xx

SDC-xx-DEVSDC-xx-DEV SDC-xx-PRODSDC-xx-PROD

yyy pipeline (contribution)yyy pipeline (contribution)

InfrastructureInfrastructure

Production InfrastructureProduction Infrastructure

Development InfrastructureDevelopment Infrastructure

TL/IAL Plug-insTL/IAL Plug-ins

TL/IAL ConnectorsTL/IAL Connectors

Integration InfrastructureIntegration Infrastructure

Validation InfrastructureValidation Infrastructure

xxx pipelinexxx pipeline

EuclidConsortium


SDCs – Development Inputs

• OU-xx outputs :

– Pipeline workflow, breakdown, inputs & outputs and algorithms specifications, Use cases,…

– Pipeline V&V kit (test data, expected results, documentation)

– Optionally, prototype code : should be reengineered in order to fit SGS rule

• Formal data modeling of all xx pipeline input (primary and secondary), output and intermediate data.

• Code and Data from OU-SIM

• SGS Level Documentation impacting the software development (RB, TS, DDF, DJF, MGT, PAF) to which the xx pipeline implementation should be compliant and/or should derive its own documentation.:

– Architecture

– Development principles

– Common Rules & Tools

– PA/QA

– …

• Architecture constraints from production infrastructure

EuclidConsortium


SDCs – Development Deliverables outputs

• Pipeline product including Software executables, documented source code, under strong configuration control

• pipeline documentation (RB, TS, DDF, DJF, MGT, PAF), in particular :

– RB : Software system specifications (SSS), Interface requirements document (IRD)

– TS : Software Requirements Specifications (SRS), Interface Control Document (ICD)

– DDF : Software Design Document (SDD), Software Configuration File (SCF), Software Release Document (SRelD), Software User Manual (SUM) including in particular extraction, deployment, installation and generation procedures.

– DJF : Software V&V Plan (SVVP), Software Test Specifications (STS), Software Test Report (STR), Software problem reports and non-conformance reports (SPR & NCR)

– MGT : Software Development Plan (SDP)

– PAF : Software Product Assurance Plan (SPAP), traceability & coverage matrixes (TCM).

20

EuclidConsortium


SDCs - Integration Deliverables Outputs

• Operational VIS pipeline on SDC-FR PROD infra.

• Verification & Validation documentation :

– Verification & Validation Plan (SValP)

– Verification & Validation Tests Specification (SValS)

– Verification & Validation Tests report (SValR)

• Contribution with OU & PO/System team to system 2by2 tests report

• Contribution to system E2E tests report

21

EuclidConsortium


SDCs - Operation Deliverables Outputs

• Data products from pipeline

• Production reports : global metrics ( problems met during exploitation, performance : processing, storage,…)

22

EuclidConsortium


SDCs – Versions & Overall PlanningSGS Global schedule

Phase FDisposal

23

Proposed Schedule

euclid consortium ou-le3 kom meeting, london 1march 22-23, 20121 ou /sdc proposed development...

Documents

nn sdc wp

sdc dev responsible

dev wp

dev level sdc

given sdc

sdc management wp

sdcdev dev environment

sdc prod responsible