o'reilly 2016: "continuous delivery with containers: the trials and tribulations" by...

Continuous Delivery for Containerized Applications:The Trials and Tribulations

Daniel Bryant @danielbryantuk

02/05/2023 @danielbryantuk

Setting the scene…

• Continuous delivery is a large topic

• Focusing on the process and tooling• No live coding today• Mini-book contains more details• “Building a CD pipeline” by Adrian and Kevin

02/05/2023 @danielbryantuk

TL;DR – Containers and CD

• Container image becomes the build pipeline ‘single binary’

• Adding metadata to containers images is vital

• Must validate container constraints (NFRs)• Cultivate containerised ‘mechanical sympathy’

02/05/2023 @danielbryantuk

@danielbryantuk

• Chief Scientist at OpenCredo, CTO at SpectoLabs

• Agile, architecture, CI/CD, DevOps

• Java, Go, JS, microservices, cloud, containers

• Leading change through the application of technology and teams

02/05/2023 @danielbryantuk

Continuous Delivery

02/05/2023 @danielbryantuk

Continuous Delivery

• Produce valuable and robust software in short cycles

• Optimising for feedback and learning

• Not (necessarily) Continuous Deployment

02/05/2023 @danielbryantuk

Creation of a build pipeline is mandatory for continuous delivery

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

The Impact of containers on CD

02/05/2023 @danielbryantuk

Container technology (and CD)

• OS-level virtualisation• cgroups, namespaces, rootfs

• Package and execute software

• Container image == ‘single binary’

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

Creating a pipeline for containers

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

Make your dev environment like production

• Develop locally or copy/code in container

• Use base images from production

• Must build/test containers locally• Perform (at least) happy path tests • All tests should be runnable locally

02/05/2023 @danielbryantuk

Lesson learned: Dockerfile content is super important

• OS choice

• Configuration

• Build artifacts

• Exposing ports

• Java• JDK vs JRE and Oracle vs OpenJDK

• Golang• Statically compiled binary

• Python• Virtualenv

02/05/2023 @danielbryantuk

Please talk to the sysadmin people:Their operational knowledge is invaluable

02/05/2023 @danielbryantuk

Different prod and test containers?• Create “test” version of container• Full OS (e.g. Ubuntu)• Test tools and data

• Easy to see app/configuration drift

• Use test sidecar containers instead

• ONTEST proposal by Alexi Ledenevhttp://blog.terranillius.com/post/docker_testing/

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

Building images with Jenkins• My report covers this

• Build as usual…

• Build Docker Image• Cloudbees Docker Build and Publish Plugin

• Push image to registry

02/05/2023 @danielbryantuk

Storing in an image registry (DockerHub)

02/05/2023 @danielbryantuk

Lesson learned: Metadata is valuable• Application metadata

• Version / GIT SHA

• Build metadata• Build date• Image name• Vendor

• Quality metadata• QA control• Security audited etc

02/05/2023 @danielbryantuk

Metadata – Beware of “latest” Docker Tag• Beware of the ‘latest’ Docker tag

• “Latest” simply means • the last build/tag that ran without

a specific tag/version specified

• Ignore “latest” tag• Version your tags, every time• Danielbryantuk/test:2.4.1

02/05/2023 @danielbryantuk

Metadata - Adding Labels at build time

• Docker Labels

• Add key/value data to image

02/05/2023 @danielbryantuk

Metadata - Adding Labels at build time• Microscaling Systems’ Makefile

• Labelling automated builds on DockerHub (h/t Ross Fairbanks)• Create file /hooks/build

• label-schema.org • microbadger.com

02/05/2023 @danielbryantuk

Metadata - Adding Labels at runtime$ docker run -d --label uk.co.danielbryant.lbname=frontdoor nginx

• Can ’docker commit’, but creates new image

• Not possible to update running container

• Docker Proposal: Update labels #21721

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

Component testing

02/05/2023 @danielbryantuk

Testing: Jenkins Pipeline (as code)

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

Testing individual containers

02/05/2023 @danielbryantuk

Integration testing

02/05/2023 @danielbryantuk

Introducing Docker Compose

02/05/2023 @danielbryantuk

Docker Compose & Jenkins Pipeline

02/05/2023 @danielbryantuk | @spoole167 35

Mechanical sympathy: Docker and Java• Watch for cgroup limits (and cgroup awareness)• getAvailableProcessors issue (bugs.openjdk.java.net/browse/JDK-8140793)• Default fork/join thread pool sizes (based from host CPU count)

• Set container memory appropriately • JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead• Account for native thread requirements e.g. thread stack size (Xss)

• Entropy • Host entropy can soon be exhausted by crypto operations

02/05/2023 @danielbryantuk

Mechanical sympathy: Docker and security

02/05/2023 @danielbryantuk

Containers are not a silver bullet

02/05/2023 @danielbryantuk

Containers: Expectations versus reality

“DevOps”

02/05/2023 @danielbryantuk

Containerise an existing (monolithic) app?• For

• We know the monolith well

• Allows homogenization of the pipeline and deployment platform

• Can be a demonstrable win for tech and the business

• Against

• Can be difficult (100+ line scripts)

• Often not designed for operation within containers, nor cloud native

• Putting lipstick on a pig?

02/05/2023 @danielbryantuk

Whatever you decide…push it through the pipeline ASAP!

02/05/2023 @danielbryantuk

Key lessons learned• Conduct an architectural review

• Architecture for Developers, by Simon Brown• Architecture Interview, by Susan Fowler

• Look for data ingress/egress• File system access

• Support resource constraints/transience• Optimise for quick startup and shutdown • Evaluate approach to concurrency• Store configuration (secrets) remotely

02/05/2023 @danielbryantuk

New design patterns

bit.ly/2efe0TP

02/05/2023 @danielbryantuk

Microservices…

Containers and microservices are complementary

Testing and deployment change

https://specto.io/blog/recipe-for-designing-building-testing-microservices.html

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

02/05/2023 @danielbryantuk

Microservice architectural impact on CD• Application decomposition• Bounded context• Change cadence• Risk • Performance• Scalability• Team location

h/t Matthew Skelton, Adam Tornhill

• Worth knowing about:• Consumer-based contracts • Service virtualisation• Synthetic transactions and

semantic monitoring

02/05/2023 @danielbryantuk

Using containers does not obviate the need for good architectural practices

02/05/2023 @danielbryantuk

https://speakerdeck.com/caseywest/containercon-north-america-cloud-anti-patterns

02/05/2023 @danielbryantuk

Summary

02/05/2023 @danielbryantuk

In summary• Continuous delivery is vitally important in modern architectures/ops

• Container images must be the (single) source of truth within pipeline

• Mechanical sympathy is important (assert properties in the pipeline)• We’re now bundling more responsibility into our artifact (e.g. an OS)• Not all developers are operationally aware

• The tooling is now becoming stable/mature• We need to re-apply old CD practices with new technologies/tooling

02/05/2023 @danielbryantuk

Bedtime reading

02/05/2023 @danielbryantuk

Thanks for listening

• Any questions?

• Feel free to contact me• @danielbryantuk• daniel.bryant@opencredo.com