dev ops without the ops

Konstantin GredeskoulCTO, wanelo.com

DevOps without the “Ops” A fallacy? A dream? A ________?

@kig

@kigster

How Wanelo handles thousands of writes per second with 99.97% uptime without an operations team

@kig

Proprietary and Confidential

Wanelo is the digital mall of the future, and a place to find the most amazing products.

What are you running on? No really, what’s your stack?

Are you on Mongo? No!?!?!??

or…

You running ruby? WTF? It’s slow! You are running Erlang? WTF? It’s in Swedish!

etc.

People often ask…

Backend Stack & Key Vendors

■ MRI Ruby, jRuby, Sinatra, Ruby on Rails

■ PostgreSQL, Solr, redis, twemproxymemcached, nginx, haproxy, pgbouncer,elastic search

■ Joyent Cloud, SmartOS, Manta Object StoreZFS, ARC Cache, superb IO, SMF, Zones, dTrace, humans

■ DNSMadeEasy, MessageBus, Chef, SiftScience

■ LeanPlum, MixPanel, Graphite analytics

■ AWS S3 + Fastly CDN for user / product images

■ Circonus, NewRelic, statsd, Boundary, PagerDuty, nagios, SumoLogic monitoring, alerting, error reporting

Proprietary and Confidential

How much traffic does your app get?

• If you are building an internal web-site in Rails you’d be lucky to get 100 RPMs – your users are only a limited set of employees

• Semi-Popular sites with up to a few hundreds of concurrent users can expect about 1K-2K RPM

• When you cross 100K RPM mark, you joined the “small big boys” :)

• When you are Pinterest, Facebook or Twitter… You are probably doing 1-10M RPMs

So what is this talk about?

• Review Operations, DevOps, and the Cloud, and how the new technologies are changing the landscape

• Learn some key points and patterns that dramatically reduce stress and pain associated with running a site, particularly ruby and/or rails

• Discuss if modern startups really need a dedicated operations team, and if so – at what point?

Let’s start with the basics DevOps

Proprietary and Confidential

What the heck is DevOps?

• “Today, many organizations are confused on what DevOps means for them..” [2]

1. WikiPedia article on DevOps

2. FORRESTER: “Eliminate DevOps Myths With Situational-Awareness-Based Performance”. John Rakowski, October 10, 2014

• DevOps is a software development method that stresses communication, collaboration, integration, automation and measurement cooperation between software developers and other information-technology (IT) professionals. [1]

Proprietary and Confidential

“…Efficient teams are deploying code 30 times more frequently with 50 percent fewer failures in 2014…” [3]

“…DevOps practices correlate strongly with high organizational performance” [3]

3. Source: PuppetLabs “State of DevOps Report”, 2014

DevOps however, works…

Traditional “Heavy” Agile

• Traditional Ops responsibilities were often in conflict with product development: stability versus change.

Product Dev QA OperationsProduct Dev QA Operations

Traditional Operations

• Uptime, stability and reliability

• On-call, fixing site at night

• Backups and disaster recovery• Security, patching, OpenSSL :)

• Hardware

• Networking

• Colocation / DC

“The Cloud” changed things

• Uptime, stability and reliability

• On-call, fixing site at night

• Backups and disaster recovery• Security, patching, OpenSSL :)

• Hardware

• Networking

• Colocation / DC

So the Cloud is a big part of what makes DevOps possible

Let’s talk about a simpler and more friendly way to build and deploy

software.

Early Company Goals (based on Wanelo)

• Maximize iteration speed

• Practice “aggro-agile”™

• Scale up as we go, keep the app fast

• Break things, learn, move on

• Enable, empower and inspire our team

• Remain in control of our infrastructure

And while moving really fast…

We just never hired Ops

But we did hire several brilliant engineers who actually enjoyed infrastructure / platform work.

Except they approach it like … code.

Not having Ops meant

• We had to deploy our app to the cloud, and learn how to provision the nodes we needed, as well as:

• How to provision load balancers and app servers

• How to configure new Solr masters and replicas

• How to install and tune PostgreSQL databases

• memcaches, redis shards, twemproxy, haproxy

Fast forward to today

• 100% cloud hosted (Joyent Cloud)

• 100% automated (Chef)

• 10,000% traffic growth in 6 months and survived

• 99.97% uptime (without trying very hard)

• on call engineers get 1-2 pages per week

• 80% of engineers are on call rotation, including iOS & Android developers

Still no “Ops” team, but plenty of Ops work

How?

1. Automation and Deployment

• Infrastructure is a first class citizen

• Pairs deliver user stories which include automation

• Did I mention we pair program? It rocks!

• We run Chef continuously in production

• I want to trust my tools, and if they break, fix them

• Partition staging and production environments

Incremental Deployment

• Roll code out everywhere, restart 2% of servers

• Watch errors, latency, other anomalies

• When satisfied continue rolling all servers

• Ensure old and new code can co-exist

• Ensure no “drop/rename” migrations happen on live tables

• Ensure no exclusive locking migrations (eg. create index concurrently)

2. Fault tolerant infrastructure

• Ensure aggressive client timeouts

• Achieving fault tolerance today is much cheaper than ever before! It’s a crime not to do it :)

• Put haproxy in front of everything, literally

• Stateless services only

• Put makara, twemproxy, Dalli in front of database, redis and memcached

Let’s look at a couple of recipes for resilience

Resilience keeps you sleeping at night

Where is everything? HAProxy + Chef Search + Stateless

App talks tohttp://127.0.0.1:8000 http://127.0.0.1:8001

App HAProxy

Backend 1

Backend 2

Solr

Web ServiceBackend 2

ElasticSearch

Virtual Zone / Server

This pattern allows us to have one place that knows about everything else, in Chef

What the hell Makara?

• Makara is a simple database routing tool for ActiveRecord that has been in production on Wanelo and TaskRabbit for years

• https://github.com/wanelo/makara (PostgreSQL)

• https://github.com/taskrabbit/makara (MySQL)

Proprietary and Confidential

• Was the simplest library to understand, and port to

• Worked in the multi-threaded environment of Sidekiq Background Workers

• automatically retries if replica goes down

• load balances with weights

• Was running in production

Replicate everything that replicates

App

HAProxy

Backend 1

Backend 2

Solr Replica

Backend 2

Solr Replica

Solr Replica

Solr Master

Web / API Requests

Background WorkerQueue

reads

writes

App

HAProxy

Backend 1

Backend 2

Solr Replica

Backend 2

Solr Replica

Solr Replica

Solr Master

Web / API Requests

Background WorkerQueue

Degraded State, but still up!

Many replicas can be down

reads

writes

Replicas are great because they are easy to add and often ok to ignore when they die/reboot/etc.

Don’t buy an expensive load balancer

Load Balancer

haproxy

nginx

Load Balancer

haproxy

nginx

200.200.234.145 200.200.234.146

example.com

App Server App Server App Server App Server App Server App Server

You can build a decent one with DNS

App Server App Server App Server

Load Balancer

haproxy

nginx

App Server App Server App Server

Load Balancer

haproxy

nginx

DNS Provider

pingping

200.200.234.145 200.200.234.146

DNS auto-failover is offered with some enterprise DNS services, e.g. from DNSMadeEasy

When LB goes down, it is removed from the DNS pool

App Server App Server App Server

Load Balancer

haproxy

nginx

App Server App Server App Server

Load Balancer

haproxy

nginx

DNS Provider

pingping

200.200.234.145 200.200.234.146

It works pretty well

It works pretty well

Load Balancer

haproxy

nginx

Dead Load Balancer

200.200.234.145 200.200.234.146

DNS Provider

ping

App Server App Server App Server App Server App Server App Server

example.com

This works best with a short TTL

Configure LBs in pairs, as the others failover, to account for network partitioning

When LB goes down, it is removed from the DNS pool

This pattern allows us to tolerate reboots and maintenance with minimal effect on our users

Failover to the overflow patternTwo queues: large primary, small secondary

The primary distributes jobs to a large set of specialized workers, assigned to specific queues

App

HAProxy

Primary Backend 1

Failover Backend 2

Primary Background Workers

Redis PrimaryQueue

RedisFailover

"Overflow" Workers

The failover queue has only a small number of overflow workers, but they will accept any work

During spikes in traffic, this pattern allows our application to continue enqueuing jobs when the

primary is overwhelmed

This is useful in situations when you can’t easily round robin between multiple shards.

Example: Sidekiq with a “Unique Job” extension.

• Some tools allow alerting on the first derivative of an observed metric.

• This is what we want: rapid drop (or increase) in a key metric to generate an alert.

3. Alert only on what’s important

• Nagios is great for visibility

• Not great for knowing when to drop everything because the site is on fire

• We never page on “host down”Because, who cares?The host is likely redundant, and will be back. …Probably.

Alerting examples

• We only page for things like “sudden drop in product saves per second”, or a spike in error rate, etc.

• Monitoring / alerting tool Circonus supports this

4. Obsessive monitoring

• Modern tools offer unprecedented visibility

• Real time application monitoring

• Real time business stats monitoring

• Real time network monitoring

• Dashboards, TV Monitor, alerts

• Real time, real time, real time.

Systems Status: Dashboard Monitoring & Graphing with Circonus, NewRelic, statsd, nagios

5. Cloud vendor is your partner

• We get phenomenal customer support from Joyent

• Our Cloud Partner, in a way, is our Ops

• Joyent is innovative in that they develop and run their own cloud stack: from the OS layer (SmartOS) to the data center management software

• They offer a unique option to take our “cloud” in-house when that time comes

6. DevOps, really, is just code

• Hire folks who write code, so that they don’t have to repeat the same task twice

• Everyone will be happier that way.

So here is how to reduce stress!

1. Insist on 100% automation

2. Deploy fault tolerant patterns wherever possible

3. Page only on what’s important to the business

4. Monitor everything else obsessively

5. Choose a cloud provider that can be your partner

6. Infrastructure work is software engineering

Thanks! slideshare.net/kigstergithub.com/kigster

github.com/wanelo github.com/wanelo-chef wanelo technical blog

building.wanelo.com

Proprietary and Confidential

@kig

@kig

@kigster