scaling wanelo.com 100x in six months

Proprietary and Confidential

Scaling

100x in six months

by Eric Saxby & Konstantin GredeskoulApril 2013

1Thursday, April 18, 13

What is Wanelo?


■ Wanelo (“Wah-nee-lo” from Want, Need Love) is a global platform for shopping.



■ It’s marketing-free shopping across 100s of thousands of unique stores



Personal Activity Feed...4Thursday, April 18, 13


iOS + Android



Early Decisions



■ Optimize for iteration speed, not performance

Early Decisions




■ Keep scalability in mind, track metrics, and fix as needed

Early Decisions




■ Keep scalability in mind, track metrics, and fix as needed

■ Introduce many levels of caching early

Early Decisions


Technology Timeline

Proprietary and Confidential 7Thursday, April 18, 13

Technology Timeline


■ 2010 - 2011 Wanelo v1 stack is Java, JSP, MySQL, Hibernate90K lines of code, 53+ DB tables, no tests


Technology Timeline



■ May 2012 - June 2012Rewrite from scratch to RoR on PostgreSQL (v2)


Technology Timeline



■ May 2012 - June 2012Rewrite from scratch to RoR on PostgreSQL (v2)

■ Ruby app is 10K LOC, full test coverage, 8 database tables, less features


The “Big” Rewrite




More info here....




More info here....

http://building.wanelo.com/


http://building.wanelo.com/post/42356243899/the-big-switch-how-we-rebuilt-wanelo-from-scratch-and

http://building.wanelo.com/post/42356243899/the-big-switch-how-we-rebuilt-wanelo-from-scratch-and

Growth Timeline


Growth Timeline


■ 06/2012 - RoR App Relaunches


Growth Timeline


■ 2-3K requests per minute (RPM) peak ■ 06/2012 - RoR App Relaunches


Growth Timeline


■ 08/2012 - iOS App is launched

■ 2-3K requests per minute (RPM) peak ■ 06/2012 - RoR App Relaunches


Growth Timeline



■ 2-3K requests per minute (RPM) peak

■ 10-40K RPM peak



Growth Timeline



■ 12/2012 - Android app launched


■ 10-40K RPM peak



Growth Timeline





■ 40-120K RPM peak

■ 10-40K RPM peak



Growth Timeline




■ 03/2013 - #24 top free apps iTunes



■ 10-40K RPM peak



Requests Per Minute (RPM)


Current Numbers...


■ 4M active monthly users

■ 5M products saved 700M times

■ 8M products saved per day

■ 200k stores


Backend Stack & Key Vendors


■ MRI Ruby 1.9.3 & Rails 3.2

■ PostgreSQL 9.2.4, Solr 3.6

■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace

■ Circonus, Chef + OpscodeMonitoring, graphing, alerting, automation

■ Amazon S3 + Fastly CDN

■ NewRelic, statsd, Graphite, nagios


Wanelo Web Architecture


nginx

haproxy

unicorn x 14

haproxy pgbouncer twemproxy

PostgreSQLSolr Redis MemCached

sidekiq


20 x 8GB 4 x 8GB

6 x 2GB


This talk is about:


This talk is about:


1. How much traffic can your database handle?


This talk is about:



2. Special report on counters


This talk is about:



3. Scaling database reads



This talk is about:



3. Scaling database reads

4. Scaling database writes



1.How much traffic can your database handle?


PostgreSQL is Awesome!




■ Does a fantastic job of not corrupting your data





■ Streaming replication in 9.2 is extremely reliable






■ Won’t write to a read-only replica







■ But... No master/master replication







■ But... No master/master replication(good!)


Is the database healthy?


What’s healthy?


What’s healthy?


■ Able to respond quickly to queries from application (< 4ms disk seek time)


What’s healthy?



■ Has enough room to grow


What’s healthy?



■ Has enough room to grow

■ How do we know when we’re approaching a dangerous threshold?


Oops!


NewRelic Latency (yellow = database)


pg_stat_statements


■ Maybe your app is to blame for performance... select query, calls, total_time from pg_stat_statements order by total_time desc limit 12;


pg_stat_statements


■ Maybe your app is to blame for performance... select query, calls, total_time from pg_stat_statements order by total_time desc limit 12;

Similar to Percona Toolkit, but runs all the time collecting stats.



pg_stat_statements



pg_stat_user_indexes

■ Using indexes as much as you think you are?

■ Using indexes at all?



pg_stat_user_tables

■ Full table scans? (seq_scan)


Throw that in a graph


Reads/second for one large table, daily


Non-linear changes


Suspicious spike!


Correlate different data


Deployments! Aha!


Utilization vs Saturation


# of Active PostgreSQL connections




Red line: % of max connections establishedPurple: % of connections in query


Disk reads/writes


green: reads, red: writes


Disk reads/writes


Usage increases, but are the disks saturated?

green: reads, red: writes




[How much are you waiting on disk?


File system cache (ARC)


Watch the right things


Hit ratio of the file system cache (ARC)


Room to grow...


Size (including indexes) of a key table


Working set in RAM?


Adding index increases the size


Collect all the data you can


Once we knew where to look, graphs addedlater could explain behavior we could

only guess at earlier


2.Special report on Counters and Pagination



Problem #1: DB Latency Up...




■ iostat shows 100% disk busy




device r/s w/s Mr/s Mw/s wait actv svc_t %w %b

sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100 sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100 sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100

■ iostat shows 100% disk busy



Problem #1: Diagnostics



■ Database is running very very hot. Initial investigation shows large number of counts.





■ Turns out anytime you page with Kaminari, it always does a count(*)!





■ Turns out anytime you page with Kaminari, it always does a count(*)!


SELECT "stores".* FROM "stores" WHERE (state = 'approved') LIMIT 20 OFFSET 0

SELECT COUNT(*) FROM "stores" WHERE (state = 'approved')



Problem #1: Pagination




■ Doing count(*) is pretty expensive, as DB must scan many rows (either the actual table or an index)



■ We are paginating everything! Even infinite scroll is a paged view behind the scenes.




■ We are paginating everything! Even infinite scroll is a paged view behind the scenes.


■ But we really DON’T want to run count(*) for every paged view.



■ We are showing most popular stores

■ Maybe it’s OK to hard-code the total number to, say, 1000?




■ We are showing most popular stores

■ Maybe it’s OK to hard-code the total number to, say, 1000?

■ How do we tell Kaminari NOT to issue a count query in this case?




Problem #1: Pagination (ctd)



Solution #1: Monkey Patch!!



Solution #1: Pass in the counter



SELECT "stores".* FROM "stores" WHERE (state = 'approved') LIMIT 20 OFFSET 0

Solution #1: Pass in the counter



■ AKA: We still are doing too many counts!

Problem #2: Count Draculas



■ AKA: We still are doing too many counts!

Problem #2: Count Draculas

■ Rails makes it so easy to do it the lazy way.



■ But it just doesn’t scale well

Problem #2: Too Many Counts!



■ But it just doesn’t scale well

Problem #2: Too Many Counts!

■ Fortunately, Rails has just a feature for this...



■ Unfortunately, it has one massive issue:

Counter Caches




Counter Caches

■ It causes database deadlocks at high volume




Counter Caches


■ Because many ruby processes are creating child records concurrently




Counter Caches



■ Each is executing a callback, trying to update counter_cache column on the parent, requiring row-level lock




Counter Caches



■ Each is executing a callback, trying to update counter_cache column on the parent, requiring row-level lock

■ Deadlocks ensue



Possible Solution:Use Background Jobs




■ It works like this:





■ As the record is created, we enqueue a request to recalculate counter_cache on the parent





■ As the record is created, we enqueue a request to recalculate counter_cache on the parent

■ The job performs a complete recalculation of the counter cache and is idempotent



Solution #2: Explained




■ Sidekiq with UniqueJob extension





■ Short wait for “buffering”






■ Serialize updates via small number of workers






■ Serialize updates via small number of workers

■ Can temporarily stop workers (in an emergency) to alleviate DB load



Solution #2: Code



Things are better. BUT...



Things are better. BUT...Still too many fucking counts!




■ Even doing count(*) from workers is too much on the databases

Still too many fucking counts!





■ We need to stop doing count(*) in DB. But keep counter_caches. How?






■ We need to stop doing count(*) in DB. But keep counter_caches. How?

■ We could use Redis for this.




Solution #3: Counts Deltas unicorn

PostgreSQL

sidekiq

save product product_id

RedisCounters

RedisSidekiq

3. Dequeue

2. ProductCountWorker.enqueue product_id

4. GET5. RESET

1. INCR product_id

5. SQL Update INCR by N

counter_cache column



Solution #3: Counts Deltas unicorn

PostgreSQL

sidekiq

save product product_id

RedisCounters

RedisSidekiq

3. Dequeue

2. ProductCountWorker.enqueue product_id

4. GET5. RESET

1. INCR product_id

5. SQL Update INCR by N

counter_cache column

■ Web request increments counter value in Redis

■ Enqueues request to update counter_cache

■ Background Job picks up a few minutes later, reads Redis delta value, and removes it.

■ Updates counter_cache column by incrementing it by delta.



Define counter_cache_on...

■ Internal GEM, will open source soon!



Can now use counter caches in pagination!


3.Scaling reads


Multiple optimization cycles


■ Cachingaction caching, fragment, CDN

■ Personalization via AJAXCache the entire page, then add personalized details

■ 25ms/req memcached time is cheaper than 12ms/req of database time


Cache optimization


40% hit ratio! Woo!Wait... is that even good?


Cache optimization


Increasing your hit ratio means lessqueries against your database


Cache optimization


Caveat: even low hit ratio cachescan save your ass. You’re removing

load from the DB, remember?


Cache saturation


How long before your cachesstart evicting data?

Blue: cache writesRed: automatic evictions


Ajax personalization


Nice!


■ Rails Action CachingRuns before_filters, so A/B experiments can still run

■ Extremely fast pages4ms application time for some of our computationally heaviest pages

■ Could be served via CDN in the future


Sad trombone...


■ Are you actually logged in?Pages don’t know until Ajax successfully runs

■ Selenium AND Jasmine tests!


Read/write splitting


■ Sometime in December 2012...





■ Database reaching 100% saturation






■ Latency starting to increase non-linearly







■ We need to distribute database load







■ We need to distribute database load

■ We need to use read replicas!


DB adapters for read/write


■ Looked at several, including DbCharmer


DB adapters for read/write


■ Features / Configurability / Stability

■ Thread safety? This may be Ruby, but some people do actually use threads.

■ If I tell you it’s a read-only replica, DON’T ISSUE WRITES

■ Failover on errors?

■ Looked at several, including DbCharmer


Chose Makara, by TaskRabbit


■ Used in production

■ We extended it to work with PostgreSQL

■ Works with Sidekiqs (thread-safe!)

■ Failover code is very simple. Simple is sometimes better.

https://github.com/taskrabbit/makara




We rolled out Makara and...


■ 1 master, 3 read-only async replicas


We rolled out Makara and...


Wait, what?

■ 1 master, 3 read-only async replicas


A note about graphs


■ NewRelic is great!

■ Not easy to predict when your systems are about to fall over

■ Use something else to visualize Database and disk saturation


3 days later, in production


■ 3 read replicas distributing load from master

■ app servers and sidekiqs create lots of connections to DB backends


3 days later, in production


■ 3 read replicas distributing load from master

■ app servers and sidekiqs create lots of connections to DB backends

■ Mysterious spikes in errors at high traffic


Replication! Doh!


Replication lag (yellow) correlates with application errors (red)


Replication lag! Doh!


■ Track latency sending xlog to slavesselect client_addr, pg_xlog_location_diff(sent_location, write_location) from pg_stat_replication;

■ Track latency applying xlogs on slavesselect pg_xlog_location_diff( pg_last_xlog_receive_location(), pg_last_xlog_replay_location()), extract(epoch from now()) - extract(epoch from pg_last_xact_replay_timestamp());


Eventual Consistency




■ Some code paths should always go to master for reads (ie, after signup)





■ Application should be resilient to getting RecordNotFound to tolerate replication delays





■ Not enough to scale reads. Writes become the bottleneck.

■ Application should be resilient to getting RecordNotFound to tolerate replication delays


Write load delays replication


Replicas are busy trying to apply XLOGsand serve heavy read traffic


4.Scaling database writes


First, No-Brainers:


■ Move stuff out of the DB. Easiest first.


First, No-Brainers:


■ Tracking user activity is very easy to do with a database table. But slow.



First, No-Brainers:



■ 2000 inserts/sec while also handling site critical data? Not a good idea.



First, No-Brainers:



■ 2000 inserts/sec while also handling site critical data? Not a good idea.

■ Solution:UDP packets to rsyslog, ASCII delimited files, log-rotate, analyze them later



Next: Async Commits


Next: Async Commits


■ PostgreSQL supports delayed (batched) commits


Next: Async Commits



■ Delays fsync for some # of microseconds


Next: Async Commits



■ Delays fsync for some # of microseconds

■ At high volume helps disk IO


PostgreSQL Async Commits


ZFS Block Size


ZFS Block Size


■ Default ZFS block size is 128Kb


ZFS Block Size



■ PostgreSQL block size is 8Kb


ZFS Block Size




■ Small writes require lots of bandwidth


ZFS Block Size



device r/s w/s Mr/s Mw/s wait actv svc_t %w %b

sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100 sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100 sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100


■ Small writes require lots of bandwidth


ZFS Block Size (ctd.)




■ Solution: change ZFS block size to 8K:





device r/s w/s Mr/s Mw/s wait actv svc_t %w %b sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100 sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100 sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100





device r/s w/s Mr/s Mw/s wait actv svc_t %w %b sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100 sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100 sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100

device r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd1 130.3 219.9 1.0 4.4 0.0 0.7 2.1 0 37sd1 329.3 384.1 2.6 11.3 0.0 1.9 2.6 1 78sd1 335.3 357.0 2.6 8.7 0.0 1.8 2.6 1 80sd1 354.0 200.3 2.8 4.9 0.0 1.6 3.0 0 84sd1 465.3 100.7 3.6 1.7 0.0 2.1 3.7 0 91


Next: Vertical Sharding




■ Move out largest table into its own master database (150 inserts/sec)





■ Remove any SQL joins, do them in application, drop foreign keys





■ Remove any SQL joins, do them in application, drop foreign keys

■ Switch model to establish_connection to another DB. Fix many broken tests.


Vertical Sharding


unicorns


PostgreSQLmain master

PostgreSQLsaves masterPostgreSQL

main replicaPostgreSQLmain replica

streaming replication


Vertical Sharding: Results


Vertical Sharding: Results


■ Deploy All Things!


Future: Services Approach


unicorns


PostgreSQLmain master

Shard1

PostgreSQLmain replica

PostgreSQLmain replica

streaming replication

sinatra services app

Shard2 Shard3

http / json


In Conclusion. Tasty gems :)


https://github.com/wanelo/pause

https://github.com/wanelo/spanx

https://github.com/wanelo/redis_with_failover

https://github.com/kigster/ventable
















■ distributed rate limiting using redis

















■ rate-limit-based IP blocker for nginx


















■ attempt another redis server if available


















■ attempt another redis server if available

■ observable pattern with a twist










Thanks.

Comments? Questions?

https://github.com/wanelo

https://github.com/wanelo-chef


@kig & @sax

@kig & @ecdysone

@kigster & @sax






scaling wanelo.com 100x in six months

Technology

nagios proprietary andthursday

neededproprietary andthursday

storesproprietary andthursday

testsproprietary andthursday

postgresql v2 ruby app

wanelo v1 stack

x 8gb4 x

gbhaproxyunicorn x