distributed development in ssa - introduction › sites › default › files › esaw 2017... ·...
TRANSCRIPT
ESA UNCLASSIFIED - For Official Use
Continous integration and deployment in SSA Data System
The SSA way towards distributed/cooperative development and operation processes (DevOps)
Contineous Integration in SSA Distributed/Federated Development
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 2
Outline
• Background
• The SSA CI Experience
• Starting point
• Toolset
• Pilot Phase objectives
• CI Principles
• Process flow
• Status & Achievement Summary
• Evolution
• Pilot roadmap
• Vision (CI, CD, DevOps)
• Conclusion
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 3
Background
• Three different segments
• Each with different Operational Concepts
• Heterogeneous procurement
• Workplan
• National Programme funded activity
• Studies
• Needs to support upcoming SW Common Core (SST)
• Combine segment specific and common services development and maintenance
• Functional processing fine tuning required to react to operational events
• 10 development contracts X 3 main delivery X 3 component per delivery 90 processes
• Growing along the SSA Data System assets growth
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 4
• Software controlling is done without the support
of any SW versioning tool
• Use of Software tools to support Data System
Engineering lifecycle processes limited to:
• Ticketing system
• SPR management tools (RC, Jira)
In support of the human driven processes
defined in the applicable ECSS/QA practices
Current Status
Delivery
•FTP/DVD
Build/Compile
•Build from scratch
•RPM
Deployment
•PA -> FA -> Under Production
Unit Testing
•PA -> FA
Starting point
•Adequate support to SW Configuration , Change, Versioning •Increase process efficiency via automate compilation and deployment
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 5
Toolset
SW LifecycleProcesses
Starting point Pilot evolution Notes
Ticketing OTRS OTRSJira
OTRS integrated with JiraOr move to Jira
SW Delivery FTP , DVD and human processes
ESA Gitlab
SW versioning Human processes ESA Gitlab
Compilation&Deployment
Human processes JenkinsMavenDocker
GIT/Gitlab perfectly integrates withjenkins, maven etc
SPR Management RC ESA Jira
Project Management none ESA Jira TBD Variety of tools available on market
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 6
• ESA Gitlab as code repository
• And keeping in focus basic CI principles
• stepwise process automation
• Build
• Deploy
• Test …
Pilot phase objectives
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 7
CI Principles
Principle @ ESA SSA @ Industry
1 Maintain code repository
2 Automate the build
3 Make the build self testing
4 Everyone commit to baseline every day
5 Every commit (to baseline) should be built
6 Keep build fast
7 Test in a clone of the production environment
8 Make it easy to get latest deliverables
9 Everyone can see the result of the latest build
10 Automate deployment
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 8
Process flow
CI pipelinescompilationdeployment
GIT ORIGIN @ESA
Software VersioningInternal Repository
NEO SW App 1
NEO SW App 2
NEO SW App n
SST SW App 1
SWE SW App 1
SST SW App 2
SST SW App n
SWE SW App n
IRE OPE
Consortium 2
GitLab
Consortium 1
GitLab
Consortium 3 GitLab
Study 1GitLab
NEO SSTSWE
git://http://https://
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 9
Process flow: CI/CD @ Industry
CI automated processes
Consortium
Git Repository usedfor delivery to SSA
Application 1
Application 2
Application 3
Application 3
Application 1
Application 1
Application 3
Application 2
Clone of ESA OPE
Subco2
Subco1
Subco3
git://http://https://
ConsortiumWorking CI Repository
CI Repository
Build/Compilation
Deployment
Unit Testing
Integration Test
Authorise Delivery
1. Consortium Working Repository (not visible to ESA)
2. Automate processes• Build• Deploy• Test
3. Fast and frequent code change • Feedback to developers• Repair broken links immediately • Capability to revert back
4. Integrate frequently 1. Feedback to SubCo/ Developers2. Repair broken links immediately3. Capability to revert back
5. Test done onto a clone of the Production Environment
6. Check in validated delivery onto the Consortium Master Repository (visible to ESA)
Feedback
success
1
2
3 & 4
5
6
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 10
Process Flow: CI/CD @ESA SSA
automated processes
IRE &Pre-OPE
git://http://https://
ESA SSA ORIGIN
Code Repository
Build/Compilation
Deployment
PA
FA
Authorise Delivery
1. Pull out from Industry Repository 2. Automate processes
• Build• Deploy• Test
3. PA&FA • Feedback to developers• Capability to revert back• Test include also the test of the
automated processes 4. Failed test result is inputs to the Industry CI 5. Test done onto a clone of the Production
Environment 6. Validated delivery get the go-ahead for
undergo Production (the actual step shall be coordinated)
Feedback
1
2
3
ProductionOPE
Consortium
Git Repository usedfor delivery to SSA
65
4
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 11
Process Flow: CI/CD @Industry
FATDev/maint
Team Integration
Test
A B C D E F
By Grégoire Détrez, original by Jez Humble - This file was derived from: Continuous Delivery process diagram.png, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=43977816
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 12
Process Flow: CI/CD @ESA SSA
FA(Pre-OPE)
SSA DS Team
PA(IRE)
A B C D E F
By Grégoire Détrez, original by Jez Humble - This file was derived from: Continuous Delivery process diagram.png, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=43977816
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 13
FA(Pre-OPE)
SSA DS Team PA (IRE)
GIT Mirror Repository Synchronization
ESA Feedback
FATDev/maint
Team Integration
Test
A B C D E F
A B C D E F
Process flow: Combined processes
By Grégoire Détrez, original by Jez Humble - This file was derived from: Continuous Delivery process diagram.png, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=43977816
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 14
Phase 1
• Scope
• Create a SSA repository reference for all the ongoing development/maintenance
• Support to SW Versioning
• Automate compilation and deployment
• Try out and consolidate configuration set up
• Projects/groups trees
• Repository Structures
• Started with
• ESA GIT GITLAB
• Gitlab supports git, http, https connection
• Challenges
• Selection of tools and initial set-up
• What tool are the most appropriate?
• No industry/partner reference
Output:•Capture key aspect into documents defining guidelines and processes•Initial set-up and tools selection •Familiarization
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 15
Phase 2
• Scope
• Complement pilot phase started in Phase 1
• Fully automated compilation and deployment
• Full Validation of Processes within one of the three segment
selected/identified as pilot
• Prepare for a Common Core Software Engineering lifecycle (required by
one of the three segment)
• Challenges
• Capability to continuously evolve
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 16
Summary
Steps Schedule Planned action Objective
1) Pilot Jan-May2017
Configuration of the selected technology (GitLab) Creation of current repository Integration with plug-in technology (Jenkins, maven , RPM) Open (not imposed) to on going development & maintenance Definition of Guidelines to be applied in SSA development
• Support to SW Versioning • Create 80% of the SSA SW Repository• Automate
• Build 80%• Deployment 50%
• Try out and consolidate configuration set up• Make the approach applicable to all future
SSA development • Consolidate requirements• Tool evaluation
2) Consolidation Jun-Sep2017
Fully deployed set-up Finalization of the applicable Guidelines Formal applicability to all the new activity
• 100% SSA SW repository• Fully automated compilation and deployment• Support to SST Common Core Software
Engineering lifecycle • Security issue (needed by SST)
3) Evolution Q4 2017TBC
Automatic unit testing Documentation management Documentation production Extension of the processes to Portal CMS Full integration with
Ticketing management Requirements management Change management Problem management Project management (desireable) Security Management Asset Management IPR Management (i.e. Blackduck Protex plug-in?)
• Further efficiency of the processes
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 17
Roadmap
Feb-May 2017
Support to SW Versioning
Create 80% of the SSA SW Repository
Automate
• Build 50%
• Deployment 50%
Try out and consolidate configuration set up
Make the approach applicable to all futureSSA development
Consolidate requirements
Tool evaluation and adoption
Jun-Sep 2017
Create and maintain 100% SSA SW repository
Automate
• 90% Build
• 90% deployment
Prepare the ground for SST Common Core Software Engineering lifecycle
Security issue (needed by SST)
From October-2017
Applicable in all new SSA Procurement
Automatic unit/integration testing
Documentation management
Documentation production
Extension of the processes to Portal CMS
Full integration with
•Ticketing management
•Requirements management
•Change management
•Problem management
•Project management (desirable)
•Security Management
•Asset Management
•IPR Management (i.e. Blackduck Protex plug-in?)
•System Monitoring
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 18
Status on 20/06/2017Delivery Repository Compilation Deployment
build binaries IRE OPE Testing
SSA-SWE
swe-services SWE-XI gitlab mirror available jenkins maven manual manual manual
swe-resources SWE-XI gitlab mirror available jenkins maven manual manual manual
swe-bindings SWE-XI gitlab mirror available jenkins maven manual manual manual
swe-common-libraries SWE-XI gitlab mirror available jenkins maven manual manual manual
swe-generic-services SWE-XI gitlab mirror available jenkins maven manual manual manual
SWE-ISM SWE-III DVD/FTP available jenkins maven manual manual manual
SWE Registration Portlet SWE-III DVD/FTP available jenkins maven jenkins&gitlab-CI manual manual
Evolution Maintenance SWE-III 3.4 DVD/FTP not available manual manual manual manual manual
Evolution Maintenance SWE-III 3.5 DVD/FTP not available manual manual manual manual manual
Tools
ESA Gitlab
SSA-NEO
Neo Portal COM-V.1 DVD/FTP available manual manual manual manual
SSA Jenkins Neo Tool BE NEO-VI DVD/FTP available jenkins maven manual manual manual
SSA Maven Neo Tool FE NEO-VI DVD/FTP available manual manual manual manual
Docker NEODECS via bridging gitlab available
D3 COM-V.1 DVD/FTP available manual manual manual manual
SSA-SST
SST Cataloguing Tool SST-I WP 1 DVD/FTP available manual manual n/a manual
Analisys Tool SST-I WP 1 DVD/FTP available jenkins manual n/a manual
SST Visualization Tool SST-I WP 2 DVD/FTP available GitLab CI GitLab CI n/a manual
SST Web Portal Applications COM-V.2, SST-XIX DVD/FTP available jenkins manual manual manual
DPC COM-V.2, SST-XIX DVD/FTP not available not available manual manual manual
Planning System COM-V.2, SST-XIX DVD/FTP not available not available manual manual manual
SSIM COM-V.2, SST-XIX DVD/FTP not available not available manual manual manual
General Services COM-V.2, SST-XIX DVD/FTP to be created not available manual manual manual
Gendared FTP to be created not available manual n/a manual
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 19
• Whether the team is building the right thing in the first place
• Whether the operational requirements can be captured as executable specifications
• Whether you can add value by working with the team (ESA Team, Industry Partners, Developers,
Operators) to identify and express acceptance criteria, scenarios and examples
• Whether processes are working or can be improved
• Where the waste is and how it can be removed
• How to approach non-functional testing, performance, security, accessibility etc.
• What data and configuration are required across all of the environments in your build pipeline
• What does make sense to focus upon, what does make sense other teams focus upon
Evolution
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 20
• Critical analysis of current processes
• Avoid the syndrome “we are doing like this
because of our peculiarity and constraint”
• IPR, Security, Quality, Asset, Financial
management are all independent processes
• Practices adopted quite heterogeneously across different
activity /projects/programme
• Last (and worst) relying on human (manual) processes
• Often not linked together and scattered around
• Evolution
• Link the specialised management (Git based)
• IPR Management
• Security Management
• Quality Management
• Corporate asset management
• Continuous screening
Evolution
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 21
DEVOPS https://en.wikipedia.org/wiki/DevOps
Is a cultural shift and a collaboration concepts (between developers and
operators) reflective of key aspects of SW Development and Delivery
processes
• Code — Code development and review, version control tools,
code merging;
• Build — Continuous integration tools, build status;
• Test — Continuous testing tools that provide feedback on
business risks;
• Package — Artifact repository, application pre-deployment
staging;
• Release — Change management, release approvals, release
automation;
• Configure — Infrastructure configuration and
management, Infrastructure as Code tools;
• Monitor — Applications performance monitoring, end–user
experience.
Evolution
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 22
Conclusions
• No Conclusions but we can certainly report our initial feedback
• Piloting Phase successful
•Quick results Further impulse
• Raised the awareness of the process complexity
•Helped to focus on next steps
• Tools selection and their proper usage is a key factor
•But avoid getting paranoid
• Communicate (within the Team, with the Customer, with Industry)
• New approach open, not imposed to the Industrial partners, in the pilot
Phase
• Will became applicable in the upcoming procurement
ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 23
Conclusion
and we are open to questions
… and cooperation