DevOpsfrom a Christian Perspective

5 months with Pearson Application Security Lead EngineerPrior to Pearson

● Rackspace - Lead Engineer, Product Security● AppSec consulting

o VP Services, Praetoriano Consultant Trustwave’s Spiderlabs

● TEA - Senior Security Engineer● DIR - Penetration Tester● Texas A&M University

o Systems Analyst, Sys Admin, Developer, DBA o Lecturer in MIS department

● Viatel - Internet App Developer

Who am I?

Other professional experience

● OWASP Live CD / OWASP WTE o Project lead 2008 to presento Over 300K downloadso http://appseclive.org

● OWASP Foundation Board of Directorso International charity focused on improving software security

● Multiple speaking engagements internationally at AppSec, DHS, ISC2, SANS… conferences

● Application Security Training internationally● B.S. Economics, M.S. in MIS

o Strong believer in the value of cross-discipline study

Who am I?

A Christian

● Life long Lutheran o Lutheran private schools through 8th Gradeo Pro Bono consulting work for Lutheran Foundation of TXo My kids attend Cross Lutheran School in New Braunfels

● Prefer contemporary to traditional services● I've never had that big “aha” moment

o Many times in my life I had to lean upon my faith● Kids illnesses● Hospice care● Parents illness

Who am I?

DevOps

The old way...

Very early and prescriptive requirements and design

Long development cycles

Waterfall Approach

Groups work in Silos - Dev, SysAdmin, QA, Security

Possible feedback from bug reports but little else

Throwing code over the wall

Traditional Software Dev & Ops

Waterfall Development

Why waterfall can be prone to failure

Very difficult to capture all the requirements...

- before design is complete

- some may not surface until implementation

All features “locked in” during early stages

Operational considerations occur very late

- Ops resorts to 'work arounds' and ad-hoc installs

- Dev assumptions may violate Ops policy

“Worked on my laptop...”

Waterfall Development

Waterfall Development – from this

Waterfall Development – to this

Why DevOps came to be

What's different about DevOps

Web/Cloud companies needed

- high availability

- fast introduction of new features

Easy for users to switch to a competing service + fist mover advantage

No media to ship with SaaS models

Cultural change – not just new cool tech aka CI/CD, Docker...

Focus on clear business objectives

Dev and SysAdmins share responsibility for uptime, deploys, downtime

Emphasize people and process, repeatability

Goal is better uptime and lower operational costs

The DevOps Answer

The Phoenix Project3 Ways of DevOps

Strategies for Improving Operations

Workflow

The 3 Ways of DevOps

1

2

3

Look at your purpose and those process which aid it

From the Bible

Make sure the process is correct from beginning to the end

Then look at ways to speed up that process

Value Stream – the name a the process which provides value to the business

Working from left to right – think of a time line:

business / development => customer / operations

Flow [rate] – the speed work goes through the process

I make known the end from the beginning, from ancient times, what is still to come. I say, 'My purpose will stand, and I will do all that I please.' - Isaiah 46:10

The end of a matter is better than its beginning; Patience of spirit is better than haughtiness of spirit. - Ecclesiastes 7:8

#1 - Workflow

An example workflow

Software release process

● Code written

● Code committed to a code repository

● Unit test the code

● Package the code for deployment

● Integration testing

● Deploy code to production

#1 - Workflow

Making things repeatable

Remove all haphazard and ad hoc work from the process

Repeat until stable, I like doing the first couple times manually with a 'run book'

Scripting languages are your friends

Config Mgmt – Puppet, Chef, Salt, Ansible, Jenkins, CFEngine, …

Creating deployable artifacts from a branch/release aka .rpm / .deb / .msi

Make sure what you do can be done on 1 server or 10,000 servers

Repetition is the mother of skill

- Master Bret Riley, Sa Bom Nim (Master Instructor TSD-MGK)

#1 - WorkflowEach Step Repeatable

Work left to right but don't pass on failures

From the Bible

Test early and often

Increase the rigor of testing as you work left to right

When a failure occurs, end that flow and start a new one after corrections

The further right you are, the more expensive failure is

So whoever knows the right thing to do and fails to do it, for him it is sin. - James 4:17

The King will reply, ‘Truly I tell you, whatever you did for one of the least of these brothers and sisters of mine, you did for me. - Matthew 25:40

#1 - WorkflowNever Pass on Defects

Your fix cannot be my new problem

From the Bible

Ensure no single-step optimizations degrade the overall performance of the workflow Spending time optimizing anything other than the critical resource is an illusion. Find the bottle neck in your workflow and start there - Upstream changes will just back things up - Downstream changes won't manifest since input is limited Each new optimization creates a new bottleneck – iterate on this

So whatever you wish that others would do to you, do also to them, for this is the Law and the Prophets. - Matthew 7:12

#1 - WorkflowLocal optimizations with a global view

Now go faster

From the Bible

Make sure you have a well-defined, repeatable process first

Look for manual steps that can be automated

Look for duplicate work that can be removed/eliminated

Measuring/tracking time taken at each step is crucial

Where does the flow ebb?

Give, and it will be given to you. A good measure, pressed down, shaken together and running over, will be poured into your lap. For with the measure you use, it will be measured to you. - Luke 6:38

#1 - WorkflowIncrease the flow of work

Workflow

Improve Feedback

The 3 Ways of DevOps

1

2

3

Open yourself to upstream and downstream information

From the Bible

Feedback loops occur when information is gathered from

- upstream (business / development)

- downstream (customer / operations)

Make visible problems, concerns, potential improvements – share this publicly

Learn as you move left to right so improvements aren't lost

Requests are opportunities to better fulfill the needs of the business

There is rarely enough feedback, capture and look for more

Feedback collected can be used to optimally improve the system

For there is nothing hidden that will not be disclosed, and nothing concealed that will not be known or brought out into the open. - Luke 8:17

#2 – Improve Feedback

Customers are also inside your business

From the Bible

Customer is more then the 'consumer' at the end of the process

- Each step is the customer of the previous step

- Understand what the next steps need from you to succeed

Remember, feedback isn't guaranteed - encourage it by responding

- Responses are required of external and internal customers

Make feedback & responding quick, easy and readily available

Where there is no guidance, a people falls, but in an abundance of counselors there is safety. - Proverbs 11:14

My dear brothers and sisters, take note of this: Everyone should be quick to listen, slow to speak and slow to become angry. - James 1:19

#2 – Improve FeedbackUnderstand and respond to your customers

Remove any intermediaries and impediments to feedback

From the Bible

Communicate directly as possible, skipping steps/people if possible

- e.g. The person who finds a problems communicates with the person who can fix the problem

The more hands that hold the feedback, the more chance to get garbled

If possible, intermediaries should be software not people

Whispered secret across a classroom, how much change occurs?

A person finds joy in giving an apt reply — and how good is a timely word! - Proverbs 15:23

#2 – Improve FeedbackShorten Feedback loops

Shout it from the mountain tops

From the Bible

No heroes quietly fixing things or applying workarounds.

Open, honest communication of feedback, especially of problems

- File a bug report

- Halting the process at that step (pull the cord to stop the line)

Public feedback == full knowledge to solve the problem in the optimal way

Make having problems OK and hiding problems a fireable offense

Cease to hear instruction, my son, and you will stray from the words of knowledge. - Proverbs 19:27

Let the wilderness and its towns raise their voices; let the settlements where Kedar lives rejoice. Let the people of Sela sing for joy; let them shout from the mountaintops. - Isaiah 42:11

#2 – Improve FeedbackAmplify all feedback

Go all in

From the Bible

Keep specialized knowledge out of people's heads and into the system

- special configurations, business requirements, etc

- Check it into source control – automatically versioned.

- git blame anyone? You can find out where/when regressions occurred

Moving left to right, keep needed info in the stage that requires it

- Docs to build a package stored in the repo for that package

- Deploy automation in repo with configuration templates, etc

Gold there is, and rubies in abundance, but lips that speak knowledge are a rare jewel. - Proverbs 20:15

#2 – Improve FeedbackEmbed knowledge when needed

Workflow

Improve Feedback

Continual Experimentation and Learning

The 3 Ways of DevOps

1

2

3

Create a culture of innovation and experimentation

From the Bible

The fundamentals are now solid, what can your new knowledge buy you?

The business culture must allow for and embrace innovation / experimentation

Two essential things must be understood by the business and all involved

- We can learn from the failed experiments and risks we take

- Mastery comes with repetition and practice

and you won't be a master the first N times you practice

The mind of the discerning acquires knowledge, and the ear of the wise seeks it. – Proverbs 18:15

But be doers of the word and not hearers only, deceiving yourselves.– James 1:22

#3 – Continual Experimentation & Learning

Reward risk + learning

From the Bible

Don't just talk about rewarding risk, walk the walk

Trying new things and failing is OK when you gain knowledge

Consider this creating your own feedback in a very tight loop

Get real about this – failures should be noted positively in annual reviews

if and only if a lesson was learned

Edison invented the lightbulb by running out of things that didn't work

Be very careful, then, how you live—not as unwise but as wise, making the most of every opportunity, because the days are evil. Therefore do not be foolish, but understand what the Lord’s will is. Ephesians 5:15–17

#3 – Continual Experimentation & LearningRituals are created that reward risk taking

Plan to improve or you're planning on stagnation

From the Bible

Invest in improving the system created

- By providing value to the business, it should want to maximize that return

Prune any technical debt – all debt is not bad

- some is good, none has opportunity costs, too much will crush you

Amplifying feedback helps sell this to the business

Can keep mistakes from being repeated

For I know the plans I have for you,” declares the LORD, “plans to prosper you and not to harm you, plans to give you hope and a future.” - Jeremiah 29:11

There is a time for everything, and a season for every activity under the heavens – Ecclesiastes 3:1

#3 – Continual Experimentation & LearningMgmt allocates time for projects to improve the system

Practice emergencies so emergencies feel routine

From the Bible

Fire drills aka Chaos Monkey

You need to be a very mature org to do this

Wonderful feedback loop

- How would your programming change if you knew the DB could go away

at any time?

How else can you check redundancy? Think trying to restore from backups

For God gave us a spirit not of fear but of power and love and self-control. - 2 Timothy 1:7

(Yeah, a bit of a stretch)

#3 – Continual Experimentation & LearningFaults are introduced to increase resilience

Stretch out of your comfort zone

From the Bible

Requires embracing of failures since many of these won't work

Forces out-of-the-box thinking

Provides new perspectives on existing systems

- You may think A will break first, but B falls over instead

Can help find false bottlenecks, bad assumptions, the dreaded unknown unknowns

Yet another source of feedback so make sure and learn from it publicly

Take pains with these things; be absorbed in them, so that your progress will be evident to all. - 1 Timothy 4:15

#3 – Continual Experimentation & LearningTry crazy or audacious things

Everything shouldn't be bigger in Texas

I got nothing for this one...

Do small releases frequently

- Release become ordinary not extraordinary

- Feedback loops are quick, positive changes can happen quicker

- Bugs are easier to find & fix in a smaller code base / diff

Reduction in code latency (code latency is how long written code is idle)

- Customers won't see new features until deployed. Happy Customer == $$

- Start making returns on your coding investment – aka ROI

Feels counter-intuitive but bigger changes == more complexity, less practice,

customer wait for features/bug fixes which means more risk

A pebble every day or a boulder every quarter?

Bonus MaterialSmall Batches Are Better

What Does DevOpsMean For My AppSec

Program?

The AppSec Pipeline

Key Features of AppSec Pipelines● Designed for iterative improvement ● Provides a reusable path for AppSec activities to

follow● Provides a consistent process for both the team and

our constituency● One way flow with well-defined states● Relies heavily on automation ● Has the ability to grow in functionality organically

over time● Gracefully interconnects with the development

process

Spending time optimizing anything

other than the critical resource

is an illusion.

Key Goals of AppSec Pipelines• Optimize the critical resource - AppSec personnel

● Automate all the things that don’t require a human brain

● Drive up consistency● Increase tracking of work status● Increase flow through the system● Increase visibility and metrics● Reduce any dev team friction with

application security

Pipeline - Intake• “First Impression”

• Major categories of Intake

• Existing App

• New App

• Previously tested App

• App to re-test findings

• Key Concepts

• Ask for data about Apps only once

• Have data reviewed when an App returns

• Adapt data collected based on broad categories of Apps

Pipeline – the Middle● Inbound request triage

● Ala Carte App Sec

● Dynamic Testing

● Static Testing

● Re-Testing mitigated findings

● Mix and match based on risk

● Key Concepts

● Activities can be run in parallel

● Automation on setup, configuration, data export

● People focus on customization rather than setup

Pipeline – the End● Source of truth for all AppSec

activities

● ThreadFix is used to

● Dedup / Consolidate findings

● Normalize scanner data

● Generate Metrics

● Push issues to bug trackers

● Report and metrics automation

● REST + tfclient

● Source of many touch points with external teams

Why we like AppSec Pipelines

● Allow us to have visibility into WIP● Better understand/track/optimize flow of

engagements● Average static test takes ...

● Great increase in consistency● Easier re-allocation of engagements between staff● Each step has a well defined interface● Knowing who has what allows for more informed

“cost of switching” conversations● Flexible enough for a range of skills and app maturity

Books to read...

The Phoenix Project The Practice of Cloud SystemAdministration

Gene Kim, Kevin Behr and George Spafford

Books to read

Thomas A. Limoncelli, Strata R. Chalup,

Christina J. Hogan

The Bible The Shack

Books to read

William P. Young

Thank you !