DevOps-ing the Fulfillment Center

Shawn GandhiHead of Solutions ArchitectureAWS Canada

@shawnagram

How to set up an IT organization (processes/people/tools)where Customer Obsession, Agility, and Innovation

are primary components of its culture?

What type of organizations are doing this well?Customer Obsession, Agility, Innovation

Question then becomes:

How can a large-scale organization operate like a start-up?

Back to the Fulfillment Center…Identify and break warehouse processes into fine-grained units

Assign those units to small development teamsAmazon Fulfilment Technologies

Process: Stow products into Bins

microservice(software)

builds uses

Now let each team operate as a start-upAutonomous, full ownership of their domain, freedom to innovate

Stow Team

Web of Loosely-Coupled Micro-services

Store products in the warehouse

Pick products for customers Report defects

Track shipments

Product Details

Amazon Software Development Team4-8 developers (SDEs) + manager (SDM)

Teams should be small and hierarchically flat 2-Pizza Teams

Amazon Software Development Team4-8 developers (SDEs) + manager (SDM)

Agile Rolesscrum master, developers, product owner

Teams adopt an Agile methodology (eg, Scrum, Kanban, etc.)Agile Teams

Teams are flexible and can be quickly re-organized within the orgEg, due to new demands, teams getting too big, maturing products

Team A Team B

Team C Team D

Team A Team B

Team C Team D Team E

new team

quick re-org

Discourage “silos” in the organization It leads to lack of Agility / Ownership

Operations

Database Team

Quality Assurance

Application Security

Software Dev Team

“We’re blocked waiting the Dev environment setup”

“We’re blocked waiting a new table/SQL script”

“We’re blocked waiting performance tests”

”We’re blocked waiting sourcecode manual inspection”

”We’re blocked waiting database security approvals”

”We’re blocked waiting security tests to complete”

”We’re blocked waiting the creation of security certificates”

“We’re blocked waiting thesetup of the database hosts”

“We’re blocked waiting new test hosts”

Instead teams should operate in a DevOps settingBreak the Silos!

Operations

Database Administrator

Quality Assurance(Testers)

Application Security

Software Dev Team

“Working together”

It’s not necessarily about putting different professionals to work together

At Amazon, it’s about empowering developers and letting them fully own the systems they build

What does “working together” mean?The Anatomy of Amazon Dev Teams

Some Responsibilities of an Amazon Developer

• Write application code• Review code of others• Write unit/integration/performance tests• Create database schemas/SQL queries• Create and automate CI/CD pipelines• Provision test and production hosts• Use secure APIs and apply security policies• Monitor production environments• Be on-call on a rotation-basis and carry a pager

Amazon DevOps Model

Builder tools team Application security team

Tools, APIs, standards, best practices,documentation, training, reviews, etc.

Empower developers and let them OWN the system they build!

Thousands of Dev teams

…

push code

CI/CD Pipeline

Code review check / Code standards checkSecurity tests / Unit tests checkInteg tests check / Acceptance tests check…

deployment

monitoring production software

automated quality assurance and packing

Teams automate tasks as much as possibleTeams are encouraged to achieve full CD (continuous deployment)

continuous feedback

Release processes levels

Source Build Test Production

Continuous integration

Continuous delivery

Continuous deployment

Benefits of Micro-services for Amazon

Build Test Release

Build Test Release

• 50 Million Deployments a Year (2014)• Software enhancements delivered every second

Build Test Release

Build Test Release

Build Test Release

Build Test Release

… … … … … …

Gigantic Web of Micro-services at Amazon

Amazon Transformation Into a DevOps Shop

Continuous ImprovementCulture, processes, people, software, tooling

2001

BuildSystemsdashboarding

2009 TodayEmpowermentResizing

2001

2001-2009: Microservices + 2-Pizza Teams

monolithic application + teams microservices + 2 pizza teams

2009

Things went much better under this model

and teams were developing features

faster than ever, but we felt that we could still

improve.

In 2009, we ran a study to find out where inefficiencies might still exist

We were just waiting.

WaitWrite Code WaitBuild

Code WaitDeploy to Test

Deploy to Prod

We were just waiting.

WaitWrite Code WaitBuild

Code WaitDeploy to Test

Deploy to Prod

Mins Days Mins Days Mins Days Mins

We were just waiting.

WaitWrite Code WaitBuild

Code WaitDeploy to Test

Deploy to Prod

Weeks

Mins Days Mins Days Mins Days Mins

We were just waiting.

WaitWrite Code WaitBuild

Code WaitDeploy to Test

Deploy to Prod

Weeks

Mins Days Mins Days Mins Days Mins

2009-2013: Empowerment of Teams + Agile

2009

microservices + 2 pizza teams

BuildSystemsdashboarding

DevOps Tools + Agile

2010-Today

We built tools to automate our software release process

https://secure.flickr.com/photos/lindseygee/5894617854/

Automated actions and transitions; from check-in to production

Development benefits:

• Faster

• Safer

• Consistent & Standardized

• Visualization of the process

Pipelines

This has continued to work out really well:

In 2014:• Thousands of service teams across Amazon• Building microservices• Practicing continuous delivery• Using Agile practices• Many environments (staging, beta, production)

50 million deployments

We continue to survey our software developers every year and in 2014 results found only one development tool/service could be correlated statistically with happier developers:

Our pipelines service!

continuous delivery ==

happier developers!

https://www.flickr.com/photos/cannnela/4614340819/

Amazon-style DevOps on AWS

BuildSystemsdashboarding

https://aws.amazon.com/devops/

Find more about how these work at

Thank you!