securing deployment pipeline
TRANSCRIPT
NICTA Copyright 2014
Securing the
“Last Mile” of
the Software
Supply Chain
Len Bass, An Binh Tran, Paul
Rimba, Ralph Holz, Liming Zhu
NICTA Copyright 2014
Suppliers from all over
the world
The Software Supply Chain has a
Great Deal of Diversity
2
Operational
System
Open Source
In House
Proprietary
3rd party
NICTA Copyright 2014
Deployment pipeline is the “last mile”
of the supply chain
3
Pre-commit tests
Xpromote to normal
production
Build Image and Perform Integration
tests
UAT / staging / performance
tests
Deploy to production
Commit
...
...
Pre-commit tests
Commit
Developers
Suppliers from
all over the
world Operational
System
• The term “Last Mile” comes from telco and logistics.
• It refers to the difficulties in getting calls, goods, software to the
consumer from a distribution center.
• See http://blogs.atlassian.com/2014/04/practical-continuous-
deployment/ for a discussion of the “last mile” as it pertains to
software
Deployment pipeline
NICTA Copyright 2014
What are the security requirements and
threats to those requirements?
4
• The security requirement we will discuss in this
talk: the image deployed into operation is a valid
image.
• Threats:
– The integrity of the specification of the image has
been compromised
– The image built is not the image specified.
– The image deployed is not the image built.
NICTA Copyright 2014
Steps of our process for hardening the
deployment pipeline
5
• Identify security requirement to be satisfied
• Model steps in deployment pipeline that pertain to that
requirement.
• Identify components that implement the steps.
• Repeat until no vulnerabilities found
– Identify trusted and untrusted components
– Formally analyze model to identify vulnerabilities
– Refine and modify the model through adding trusted
components to resist vulnerabilities found,
• Implement necessary modifications to components
NICTA Copyright 2014
Security Requirements for the
Deployment Pipeline
6
Create an image of
the operational
system
Place an image into
production
The deployed
image is the image
that was created
earlier.
The created
image is the
desired image.
Security Requirements:
Pre-commit tests
Xpromote to normal
production
Build Image and Perform Integration
tests
UAT / staging / performance
tests
Deploy to production
Commit
...
...
Pre-commit tests
Commit
Developers
NICTA Copyright 2014
Trusted and untrusted components
7
• Divide components that are implied in the
deployment pipeline into
– Trusted
• Ideally these components are small enough to be verified via
code verification techniques.
• More likely they are trusted because they are widely used
and problems get reported and patched.
– Untrusted
• All other components
• The trusted components prevent the untrusted
components from violating a security
requirement.
NICTA Copyright 2014
Are there vulnerabilities?
8
Has the
specification been
compromised?
Is this the desired
image?
Image is built by Jenkins & Docker.
Do we trust Jenkins & Docker?
Both Jenkins and Docker are large
open source projects. We do not trust
them.
We need to refine the activity of “build
image” and decompose Jenkins/Docker
to identify trusted and untrusted
portions
NICTA Copyright 2014
Refine Step of “Build Image”
9
Jenkins builds a Docker image
Docker, in turn loads additional portions of
the image
Jenkins then deploys the image into an
environment.
What do we see?
• Docker protects user’s credentials
• Formal analysis identifies additional
files to be protected:
• Application artifacts
• Dockerfile (specification for
building image)
We do not trust Jenkins and Docker
NICTA Copyright 2014
Add image verification step
10
Dockerfile is encrypted.
New Step – Compute Docker
image checksum
Can Docker be trusted with this
step?
No. There are several reported
vulnerabilities.
We intend to create an integrity
checker solely to calculate the
checksum. This should be
formally verifiable.
NICTA Copyright 2014
Second security question
11
Deploy to Production
Is this the image
that was created
earlier?
Deployment is managed by
Jenkins
The questions are
“Do we trust Jenkins?”
“Do we trust AWS
security mechanisms.?”
Answers: no and yes
We must refine model and
Jenkins
NICTA Copyright 2014
Refinement of deployment step
12
• Calculate checksum of created image.
• Verify that deployed image has same
checksum
• These operations are performed by
Jenkins.
• Do we trust Jenkins?
• No – must do more refinement
NICTA Copyright 2014
Summary
13
• Our contribution is the creation of an
engineering process to evaluate/modify the
design of a deployment pipeline
• Our process is based on
– identifying trusted components,
– Analyzing a process using formal methods
– Patching vulnerabilities by creating small trusted
components
– Refining until no vulnerabilities remain
• The specifics of what we have done depends on
the technologies we use but the process will
work for any collection of technologies