devops and continuous deployment @ wwps government, education, and non-profit symposium 2014

AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014


DevOps on AWSContinuous Integration and Deployment Best Practices on AWS

Leo Zhadanovsky, Senior Solutions Architect, AWS | @leozh JP Schneider, DevOps / Internet Jedi, Mozilla Foundation |

@jdotp

AWS Government, Education, and Nonprofits SymposiumWashington, DC | June 24, 2014 – June 26, 2014

Who Am I?

I work for AWS

I worked for the DNC 2009-2012

I was embedded in the DevOps Team @ OFA

AWS does not endorse

political candidates


Who Am I?

JP, DevOps for Mozilla Foundation

Previous gig DevOps at OFA 2012

Before that, Ops at Threadless

@jdotp

Mozilla Foundation does endorse animated cats


CONTINUOUS INTEGRATION


What is Continuous Integration?

Changes to code automatically deployed to mainline branch• After passing unit and mock tests

Makes changes to code, and deployments iterative, not monolithic

Bugs are detected quickly

Allows rapid development

Helps automate deployments


DEVELOPER


SOURCE CODEREPOSITORY



PROJECT MANAGEMENT SERVER




CONTINUOUS INTEGRATION SERVER





PICKTASKS





SUBMITCODE





SCHEDULEBUILD





RECURRENTBUILDS





CODEFETCH





CODE QUALITYTESTS

TESTRESULTS





BUILD OUTPUT





DOCS

BINARIES& PACKAGES

DEV FACING NOTIFICATIONS

SOURCE CODE REPOSITORY

DNS



BUILDS


PAIN POINTS• UNIT TESTS INCOMPLETE• MOCK TESTS MAINTENANCE• EXPENSIVE TEST ENVIRONMENT• TEST ENVIRONMENT ≠ PRODUCTION• DEPLOYMENT CYCLES


ON-DEMAND

PAY AS YOU GO

ELASTIC


=

PROGRAMMABLE PLATFORM


IF YOU CAN PROGRAM ITYOU CAN AUTOMATE IT


A lot of options…

Configuration Management Systems• Puppet• Chef• Saltstack

Deployment Frameworks• Elastic Beanstalk• OpsWorks• Ansible• Fabric• Capistrano

Infrastructure Management• CloudFormation


Bake an AMI Configure dynamically

Time consuming configuration (startup time)

Static configurations (less change management)

Bootstrapping


Continuous deployment (latest code)

Environment specific (dev-test-prod)

Bootstrapping

Bake an AMI Configure dynamically


Obama for America

awsofa.info

CASE STUDY


So here’s the Idea

~30th biggest E-commerce operation, globally

~200 distinct new applications, many mobile

Hundreds of new, untested analytical approaches

Processing hundreds of TB of data on thousands of servers

Spikes of hundreds of thousands of concurrent users


a few constraints…

~30th biggest E-commerce operation, globally

~200 distinct applications, many mobile

Hundreds of new, untested analytical approaches

Processing hundreds of TB of data on thousands of servers

Spikes of hundreds of thousands of concurrent users

Critically compressed budget

Less than a year to execute

Volunteer and near-volunteer development team

Core systems will be used for a single critical day

Constitutionally-mandated completion date


Web-Scale Applications


500k+ IOPS DB Systems


Services API


Business as usual..

…for a technology startup


Election Day – OFA Headquarters


Typical Charts


How?


The old approach, even from Amazon


The old approach.. Might have some problems..


OFA’s Infrastructure

awsofa.info


Ingredients

Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB


Infrastructure, Configuration Management & Monitoring

Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB


Configuration Management: Puppet

In mid-2011, we looked at options for configuration management and chose Puppet

We needed to make it scale, and to get it to work with state-less, horizontally scalable infrastructure

How did we do this?


Bootstrapping Puppet with CloudInit

CloudInit is built into Ubuntu and Amazon Linux• Allows you to

pass bootstrap parameters in Amazon EC2 user-data field, in YAML format



Don’t store creds in puppet manifests, store them in private Amazon S3 buckets

Either pass Amazon S3 creds through CloudInit:

Even better – avoid this by using AWS Identity and Access Management (IAM) roles and AWS Unified CLI’s S3 client



Built-in puppet support

Use certname with %i for instance id to name the node

Puppetmaster must have auto sign turned on• Use security groups and/or NACLs for network-level security

In nodes.pp, use regex to match node names


Puppet Tips

Use a base class to define your standard install


Use runstages

Don’t store credentials in puppet, store them in private Amazon S3 buckets• Use AWS IAM to secure the credentials bucket/folders within that bucket

Puppet Tips


Puppet Tips

Use puppet only for configuration files and what makes your apps unique

For undifferentiated parts of apps, use Amazon S3 backed RPM/Debian repositories• Can be either public or private repos, depending on your needs

• Amazon S3 Private RPM Repos: http://git.io/YAcsbg• Amazon S3 Private Debian Repos: http://git.io/ecCjWQ

http://git.io/YAcsbg

http://git.io/YAcsbg

http://git.io/ecCjWQ

http://git.io/ecCjWQ


Puppet Tips

By using packages for applications deploys, you can set ensure => latest, and just bump the package in the repo to update

Log everything with rsyslog/graylog/loggly/NewRelic/splunk


Scaling the Puppet Masters

Use an Auto Scaling group for puppet masters• Min size => 2, use multiple Availability Zones

Either have them build themselves off of existing puppet masters in the group or off packages stored in Amazon S3 and bootstrapped through user-data

Auto-sign must be on


One thing that is difficult to prepare for…


They had this built for the previous 3 months, all on the East Coast.


They had this built for the previous 3 months, all on the East Coast.

We built this part in 9 hours to be safe.

AWS +Puppet +

Netflix Asgard + WAN Optimization Software +

DevOps =

Cross-Continent Fault-Tolerance On-Demand


If OFA was run on AWS today, what might that look like?

Take advantage of OpsWorks• This would simplify configuration management

Use CloudFormation to recreate environments for prod/dev/test quicklyUtilize PostgreSQL RDS instead of the many manually installed PostgreSQL instancesAnalytics data warehouse took months to build, RedShift would take minutesDocker containers could be used to easily allow developers to replicate their dev environments in prod or stagingUse of immutable infrastructureMore use of CloudFront, resulting in less load on EC2 resources


Mozilla FoundationCASE STUDY


Webmaker.org circa 2012• Included Apps, non-SOA: Thimble, Popcorn, Goggles• ~20 pushes of new software in 2012• Operations and Development interacted mostly through bugzilla

tickets for deploys.• Hosting in physical datacenter at Mozilla

Webmaker.org circa early 2013• Deciding to go 12-factor, SOA in app layer• Weekly pushes of Popcorn on train model• Operations and Development interacted mostly through bugzilla

tickets for deploys.


April 2013

Webmaker begins rebuilding entire platformSOA, 12-factor in node.js exclusivelyMoving apps into AWS and DevOps / CI


Since April 2013….Openbadges, Webmaker combine for: 1339 Pushes

Pushes Per Day to Staging / Prod

Pushes Per Day (Staging and Prod)


Who?• ~30 Paid Developers • Hundreds of Students • Thousands of Contributors• One DevOps / Internet Jedi• Multiple Teams

How?• Puppet, Jenkins, Fabric• Tight feedback loops:

Newrelic, Opsview• Culture Shift

• Staging Envs• Brave devs iterate,

keeping work in-context• Visible Ops• Cross-train developers in

operations


What changed?


1) Know How You Were Doing Before

2) Know What Changed When

3) Know How You Are Now Doing

=

The confidence to try more things and try them faster, with minimum viable planning.


Deployment

Pipeline


AWS CLOUDFORMATION

STACK-BASED DEPLOYMENT SERVICE

CLOUDFORMATIONTEMPLATE


{ "Description" : "Create RDS with username and password", "Resources" : {

"MyDB" : { "Type" : "AWS::RDS::DBInstance", "Properties" : { "AllocatedStorage" : "500", "DBInstanceClass" : "db.m1.small", "Engine" : "MySQL", "EngineVersion" : "5.5", "MasterUsername" : "MyName", "MasterUserPassword" : "MyPassword" } } }}


"AWS::CloudFormation::Init" : { "config" : {

"packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "httpd" : [], "php" : [], "php-mysql" : [] } }, "sources" : { "/var/www/html" : "https://s3.amazonaws.com/my-builds/build-v4.zip" } }


{ "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } },}


CLOUDFORMATIONTEMPLATE

PROCEDURALDEFINITION

Create it programmatically

KNOWN CONFIGURATIONStore stack configuration in source control

PARAMETER DRIVEN

Dynamic and user-driven templates

COLLABORATIONShare templates with ease as just files


APPLICATIONVERSIONS

+INFRASTRUCTURE

VERSIONS

CLOUDFORMATION TEMPLATE


CASE STUDY

NASA/Jet Propulsion LaboratoryCalifornia Institute of Technology


Mars Rover Landing by the #’s

NASA TV = HD stream, 1080p, ~1 mb/s per viewer

Expecting peak of ~1m viewers

All playback devices (iOS, Android, Flash, HTML5, blah)

Once in a lifetime moment in history (no crashy crashy)

FUN FUN FUN


Mars Rover Landing by the #’s

NASA TV = HD stream, 1080p, ~1 mb/s per viewerExpecting peak of ~1m viewersAll playback devices (iOS, Android, Flash, HTML5, blah)Once in a lifetime moment in history (no crashy crashy)

NASA says we can’t use their live stream setupIt’s 6 days before the landingIt’s the same week as the OlympicsAvailable technical resources from JPL: Brett and Khawaja


The sticky wicket:

HD Video streamsize = 4mbnever changeseasy to cache

manifest.f4msize = 4kbNew every 4 seccaching difficult


The plan

Design a solution around our limits• Max connections to origin = 6• Max streams per cache node = 20• Local Latency = critical• US-WEST-1 capacity reserved for S3 static images

Test the snot out of it

Hang on!


TWO GUYS


ok, so one of them is a rocket scientist..


Mars Science Laboratory - Live Video Streaming Architecture

“The 42 pack”


LOAD TESTING


Battle Testing our DeploymentBenchmarking


Dynamic Traffic ScalingUS-East Cache Node Performance25.3 Gbps


Only ~42Mbps

Dynamic Traffic ScalingImpact on US-East FMS Origin Servers


CONTINUOUS DEPLOYMENT

SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO

PRODUCTION.


KEY = ITERATION


ITERATION=

MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF

YOUR USERS


11.6sMean time between

deployments (weekday)

1,079Max number of

deployments in a single hour

10,000Mean number of

hosts simultaneously

receiving a deployment

30,000Max number of

hosts simultaneously

receiving a deployment

DEPLOYMENTS AT AMAZON.COM


SOFTWARE DEPLOY

≠PRODUCT LAUNCH


1.5 BILLION PAGE VIEWS

OCTOBER 2012

$83 MILLION IN TRANSACTIONS4.2 MILLION ITEMS SOLD

CASE STUDY


30 DEPLOYS PER DAY1 DEPLOY EVERY 20 MINUTES


"Production is truly the only place you

can validate your code."


AWS OPSWORKSINTEGRATED APPLICATION

MANAGEMENT

AWS OPSWORKS UNDER THE HOOD


DATA-DRIVENARCHITECTURES


METRICS @ETSY

AWS Government, Education, and Nonprofits SymposiumWashington, DC | June 24, 2014 – June 26, 2014METRICS @OBAMA FOR AMERICA


COST-ORIENTED ARCHITECTURES


PHP+APACHE+VARNISH

NGINX+NODEJS

vs.


CONTINUOUS INTEGRATION




=

CONTINUOUS EXPERIMENTATION



=

CONTINUOUS IMPROVEMENT


INNOVATE


« Want to increase innovation? Lower the cost of failure »

Joi Ito


SPEED AND AGILITY

Experiment Often

Fail quickly at a low cost

More Innovation

Experiment Infrequently

Failure is expensive

Less Innovation

“ON-PREMISE”


Q & A


aws.amazon.com/training

Expand your technical expertise to design, deploy, and operate

scalable, efficient applications on AWS

Training

aws.amazon.com/certification

Certification

20% off AWS Instructor-Led Training Class*• Code MK20PSSYM314 • Expires September 30, 2014

Get Savings on AWS Training!

* Discount only applies to classes delivered by AWS. Discount does not apply to classes delivered by an APN Training Partner.

aws.amazon.com/training/self-paced-labs

Self-Paced Labs

Get hands-on practice working with AWS technologies in a live environment

Validate your proven technical expertise

with the AWS platform

Expand your skills with AWS

http://aws.amazon.com/training/

http://aws.amazon.com/certification

http://aws.amazon.com/training/self-paced-labs/



DevOps on AWSContinuous Integration and Deployment Best Practices on AWS

Thank You

Leo Zhadanovsky, Senior Solutions Architect, AWS | @leozh JP Schneider, DevOps / Internet Jedi, Mozilla Foundation |

@jdotp

devops and continuous deployment @ wwps government, education, and non-profit symposium 2014

Internet

aws government

aws continuous integration

ofa aws

aws leo zhadanovsky

devops team

jdotp mozilla foundation

devops internet jedi

deployment best practices