devops at obama for america(2012) and the dnc (devops days nyc jan 2013)

DevOps @ DNC & OFADevOps @ DNC & OFA

Leo Zhadanovsky@leozhDirector of Systems Engineering @Democratic National Committee

John Schneider@jdotpFormerly Obama for AmericaCurrently DevOps / Internet Jedi @ Mozilla Foundation

DevOps @ DNC - 2009 - 2011

Year 1● Cleanup● Virtualization● Documentation● Version Control● Making everything HA● Replacing all our network gear● Automation

Labs Development Team

● Created first versions of Call Tool, Voter Registration Site, Canvassing iPhone App

Labs Development Team

Start of DevOps Practices at the DNC● Agile Development● Usage of EC2● Continuous Integration● Deploys using Capistrano● Scaling up using custom AMIs● Monitoring using Nagios, Cacti, NewRelic,

Graphite, Pingdom, StatsD

Analytics Team

● Lessons learned from 2010 used in 2012● Big Data● Hardware vs AWS● Limitations of AWS at the time● What would we do differently now?● Developed software for data analysts,

modelers and field staff

Cloud (AWS)

● Unpredictable traffic patterns: No problem! (We had no clue those birther mugs would sell so well.)

● Makes entire org far more agile

● Apples to apples testing envs are a snap

● Cloud rules changes the way we manage systems

Dev + Ops = <3The New Mentality of Devops

SOA / 12 Factor

● Allow for easy migration, deployment● All components independently scaled

without retooling or re-architecting● Critical for building scalable, high-availability

apps in the cloud.● Goes well with Route53.

Dev + Ops = <3The New Mentality of Devops

Agile and Continuous● Tool agnostic (we used Pivatol, Basecamp

Classic, Basecamp Next, spreadsheets, github issues, zendesk tickets, etc)

● Small and fast releases. (we pushed ~40 new versions per day in the last weekend)

● SHIPIT! Build 'good enough' tools and iterate.

Dev + Ops = <3The New Mentality of Devops

Instrument Everything (and trust the data)

● CPU, traffic, response time, etc

● User/System events

● Deployment-related change metrics

● Logs are data too!

Dev + Ops = <3The New Mentality of Devops

Visible Ops (aka, how to turn PMs, Devs, Management into Ops partners)

● They all get Nagios, NewRelic, and Pagerduty accounts and training

● Activity feed of commits, deploys, alerts, changes.

● Cross-pollinate team stands, retros● Give everyone contextual data

Who doesn't love color charts?

Load Testing with NewRelic Deploys

Dev + Ops = <3The New Mentality of Devops

Fail Early, Fail Fast, Fail Well

● It is not if, but when. ● Learn how to detect & mitigate, and then

iterate.● Reduce the time to detect, the time to

resolve. ● Preventing all failure is not the DevOps role,

making failure hurt less or not hurt at all is DevOps role

Yo, mind if we bring the donations db down during the VP debate?

Oh Hai East Virginia Datacenter!

Other AWS Tips-Use EBS-backed instances, AMI-based deploys, and provisioned IOPS. -Use SQS. -Multi-AZ, Multi-Region early-Autoscale from any metric, push your own metrics to Cloudwatch even-Setup ifttt or zapier alerts on AWS status pages-Check out billing broken out by tag, and TrustedAdvisor

S3 microsites are awesome

Other Things That Helped

New Relic (We love you New Relic)Asgard (Thanks Netflix)StatsD/Graphite and CubismLogglyBeesWithMachineGunsAkamai DSA and GTMBotoPalominoDB is awesome!Lots of caffeineLastly....culture, culture, culture.

Election Day