dockerizing the enterprise – jean-paul van deursen wiebe de … · 2018-06-28 · wiebe de roos...
TRANSCRIPT
![Page 1: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/1.jpg)
Jean-Paul van Deursen
Wiebe de Roos
ABN-AMRO
Dockerizing the enterprise –fast and secure
![Page 2: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/2.jpg)
Wiebe de Roos
CI/CD Consultant / IT Wizard
• Studied Communication & Multimedia Design and Master of Management & ICT
• 12+ years of IT expertise • CI/CD Consultant / Engineer implementing
Jenkins Enterprise in AWS at ABN AMRO• Lots of expertise about Docker (security) topics• Speaker at ABN AMRO and other industry
conferences
Who are we?
Jean-Paul van Deursen
IT Wizard
• Studied Electrotechnical Engineering @TU Delft
• 20+ years of experience in IT in various roles• Currently active as Wizard in the Center of
Expertise Software Development and Control• Mission: make ABN AMRO the leading digital
bank
![Page 3: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/3.jpg)
What’s on the menu?
• Docker Use Cases @ ABN AMRO• ABN-AMRO – current status of CI/CD• The existing CI platform• Challenges and limitations• Vision of the future• The new & improved CI platform• Docker containers everywhere• Pipelines to fit all use cases• Security• What’s next?• Questions and answers / discussion
![Page 4: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/4.jpg)
Docker Use Cases @ABN AMRO
• PR like Dev provisioning – Shift Left• Mocking dependencies• Encapsulate technical debt• Checkpointing and versioning• CICD Pipeline components (masters/agents)
![Page 5: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/5.jpg)
Use cases for CI/CD
5
Produce automated builds and detect errors as soon as possible, by integrating and testing all changes on a regular (daily) basis.
High frequency delivery of a tested functional piece of software that can be deployed to production rapidly.
Fully automated process including deployment to production without human interaction.
Continuous Integration
Continuous Deployment
Continuous Delivery
Many manual handovers and approvals
Long lead time for software delivery
Software quality issues found at a late stage
Code merging happening at a late stage
Inefficient cooperation between DEV and OPS
Big non-frequent releases to Production
It is not only about tooling but mainly mindset & behavior, a changed Way of Working and process improvements.
• Increase maturity of teams • Set up the conditions (tooling, pipelines, generic building blocks) for the teams to get
working.• Train the blocks on applying the right mindset, knowledge and appropriate tooling
We know other large companies which need 3 - 8 years, and changed their approach along the way.
Therefore we keep the overall stages in mind, but plan for the coming three months. Focus on learning and improving instead of long term planning.
![Page 6: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/6.jpg)
CI/CD pipeline orchestration midrange
![Page 7: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/7.jpg)
7
Dependency scan
Check out project from
SCM
Developer triggers build
Build project and execute unit
tests
Code quality scan
Secure coding scan
Publish Deployable
artifact
N
Y
ABN AMRO has introduced a set of quality gates and build breakers. The principle is that the Jenkins build is broken once therequired quality or security is not met and the developer needs to fix the defect in order to proceed. The developer has access to software quality in his IDE so defects can be detected and fixed in an early stage
Standard CI pipelines and buildbreakers
![Page 8: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/8.jpg)
Existing CI platform – Jenkins on VMs• Statistics:
• +/-1500 users• 350+ projects• 10000+ Jenkins jobs
• 1 Jenkins Operation Centre• 10 Jenkins Masters
• 30+ Linux build slaves• 30+ Windows build slaves• 4 OSX build slaves• 25+ HP-fortify (secure coding) slaves
70+ (!!!) VMs in on-prem datacentre…and GROWING…
![Page 9: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/9.jpg)
Challenges and limitations – how to…?• Hard to handle growth of DEV teams.• A lot of static VMs, constantly upscaling needed. • Hard to maintain all the servers.• Server configuration out of sync.• No Docker container support.• No true team autonomy.• A mix of tools and versions on
each build slave.• Innovation is slow.
![Page 10: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/10.jpg)
Five major improvements
1. Empower the CI/CD teams: decentralized maintenance.
2. Docker containers instead of static VMs
3. Support flexibility of tech stacks and configuration.
4. Infrastructure as Code & Configuration as Code.
5. Cloudbees Jenkins Enterprise is critical to the CI/CD program
![Page 11: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/11.jpg)
The new and improved CI platform
AWS
CMS CI
Master Slave
CD
CI
![Page 12: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/12.jpg)
The new and improved CI platform - architecture
Teams can create their own Jenkins master and run their own pipelines.This solution prevents interference of teams with each other. Reduces conflicts.
![Page 13: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/13.jpg)
Context of containers in Jenkins Enterprise
1. Platform2. Running Jenkins jobs3. Build containers4. Application containers
Specific
Generic
![Page 14: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/14.jpg)
Use case: Jenkins Build agents (containers)
Have a proper
solution for the configuration
difficulties
A never ending story…
![Page 15: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/15.jpg)
Pipelines - overview• Q1 2017: Birth of the standard pipelines (STPLs)• Lots of benefits but also challenges• Q1 2018: Birth of the new (Dockerized) pipelines:
• A pipeline for Docker images• A Dockerized pipeline for Java applications
• Easy to use, easy to implement & extend• Security is build-in• A reference for other technologies
![Page 16: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/16.jpg)
Docker image pipeline – main building blocks
16
Smoke test
Jenkinsfile + Dockerfile from
SCM
Developer triggers Docker
image build
Build Docker image
Docker lint syntax check
Docker container dependencies
check
Sign + Publish Docker image in trusted registry
N
YDocker container configuration
check
Apply security profiles
![Page 17: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/17.jpg)
Pipelines – Docker image pipeline
• A pipeline which creates Docker images• That are secure• That are versioned and tested• Which are “official” and “approved”• Ready to re-use by DEV teams
![Page 18: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/18.jpg)
Pipelines – Java pipeline Dockerized
• A pipeline which uses Docker images (building blocks from previous pipeline)
• Create Java artefact• Package in Docker image• Security stages in place• Push to registry• Ready to deploy to (Xlrelease/Xldeploy, AWS, Kubernetes)
![Page 19: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/19.jpg)
Docker Security topics on all levels
Security is needed on every level
![Page 20: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/20.jpg)
Security – why all this?
To avoid compromised containers where-ever they are used: secure business continuity
![Page 21: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/21.jpg)
Security (1): Syntax check
v1.6.2-6-gcfb547a: Pulling from hadolint/hadolintStatus: Downloaded newer image for hadolint/hadolint:v1.6.2-6-gcfb547a/dev/stdin:3 DL3005 Do not use apt-get upgrade or dist-upgrade/dev/stdin:3 DL3009 Delete the apt-get lists after installing something/dev/stdin:4 DL3008 Pin versions in apt get install. Instead of `apt-get install <package>` use `apt-get install <package>=<version>`/dev/stdin:4 DL3015 Avoid additional packages by specifying `--no-install-recommends`Docker lint syntax check (just like SonarQube
![Page 22: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/22.jpg)
Security (2): Anchore
![Page 23: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/23.jpg)
Security (3): Sonatype - Nexus Lifecycle
![Page 24: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/24.jpg)
Security (4): Docker benchmark (OSS)
![Page 25: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/25.jpg)
What’s next - roadmap
CJE to PR
Finish Dockerizedpipelines
Onboard 50 teams this year
Docker runtime scanning
Choose a container runtime on AWS
PoC for a small number of innovative teams
Enterprise based solution for all DEV teams
![Page 26: Dockerizing the enterprise – Jean-Paul van Deursen Wiebe de … · 2018-06-28 · Wiebe de Roos CI/CD Consultant / IT Wizard • Studied Communication & Multimedia Design and Master](https://reader035.vdocument.in/reader035/viewer/2022081611/5f0a934a7e708231d42c4ea6/html5/thumbnails/26.jpg)
Questions and Answers
Thank you!
Wiebe de Roos – [email protected] Paul van Deursen – [email protected]