Download - Scaling the Rails
Scaling the RailsHow a little quiz achieved Olympic Glory
Josh Schwartzman, Newzwag
Nov 19, 2008
Get Ready For:
• Database Enhancements & Gotchas
• Hardware Basics
• Load Testing
• Rails Backends & Web Servers
From Humble Beginnings:
30,000 concurrent users 9 million pageviews/hour
(on limited hardware)
[question]
[answer]
FindYour
BottlenecksMySQL
Analyze Your Logs
• Look for slowest requests
• Verbose Logging
• Detailed DB logging (a small script)Completed in 0.06763 (14 reqs/sec) | Rendering: 0.01377 (20%) | DB: 0.00587 (8%) 2.6k, 54 rows, 7 queries | 200 OK [http://url]
Database Enhancements
• Index all of the keys you are going to use to find your data
• Turn on debugging and check the slowest queries
• Use EXPLAIN on your SQL statements
(Reads)
(Some MySQL-specific enhancements)
• Minimize your # of queries
• Use Eager Loading
Database Enhancements(Reads)
• Only save the data that has changed
• Rails 2.1: 2x faster on some actions, due to:
• Partial Updates (automatic performance)
• Dirty Tracking
Database Enhancements(Writes)
• Check your model validations
Database Enhancements(Writes)
• Transactions w/InnoDB
Database Enhancements
*40x if you don’t use validations
• Bulk Writes with AR-Extensions
• 8-12x faster on InnoDB
• 8-40x faster on MyISAM*
• Master/Slave Replication
• Master DB for writes with replication to read slaves
• Distributes load to multiple machines
(Writes)
Useful Plugins
• ActiveRecord Extensions
• Bulk writes
• Dr. Nic’s magic_multi_connections
• Offload reads/writes to another DB
• Single-Write Multi-Read (SWMR)
• Our plugin to write to one DB & use multiple read slaves
Hardware
• Cache everything, so lots of RAM
• Make your reads/writes fast
• RAID helps:
RAID 5 - faster reads RAID 10 - faster writes
Hardware
• Minimize hits to your Rails Stack
• Cache images or use a fast web server
• Move all your routes.rb routing logic to dedicated hardware (load balancer)
Hardware
• Keep App & DB on separate servers
• Infrastructure specific to it’s purpose
• Easy to isolate & debug bottlenecks:
Graph from Cacti
Load Testing
• Test the path a user will take
• One Tool: JMeter
• Session and full hardware stack testing
• jmeter-server
• Distributed testing
• Test from any computer @ ~1500 threads per box
Load Testing
• You’ll always find new problems under load
• We discovered issues with lots of new users: checking login twice
• Simple queries (especially joins) became slow
• Tweaking how often we wrote scores to the database, and how many at a time
Load Testing• The type and number of web servers*
is critical. Quick comparison:
Mongrel•Stable•RAM & CPU heavy•Slowest per action
Ebb•Some bugs•RAM & CPU light•Fastest per action (uses C for heavy lifting)
Thin•Almost as stable as Mongrel (it uses Mongrel’s parser)•Lightest hardware load•Almost as fast as Ebb
*Yeah, you still need a web server like Lighttpd or Nginx to handle your static files and proxy to your Rails backends
Load Testing#
of q
uest
ions
1 2 3 4 5 6 7 8 9 10minutes
Gruff graphs from our custom user actions
Using 18 Mongrels
Load Testing
949 questions / minute: 40% faster than mongrel!
1 2 3 4 5 6 7 8 9 10
# o
f que
stio
ns
minutes
Using 18 Ebbs
When You Need a Little Help
• After all of that performance work...
• Servers were crashing spontaneously
• Nothing was apparent in the logs
• Sleepless nights...
We Found Stability in GodSimple monitoring for Rails apps:
What Was the Problem?
• Post requests on older browsers (Firefox 1.5, Navigator 7)
• Killed our backends, and as each failed, it passed on the request to another server
• Solution: switch from Nginx to Lighttpd
A Summary
• Achieved by adding plugins, custom Memcached methods, optimizing the DB, changing to ebbs, and numerous other code tweaks (and a slight hardware upgrade)
a 25x performance gain
• 200 (scaling) 5000 simultaneous users
Resources•Active Record Extensions: http://www.continuousthinking.com/are/activerecord-extensions-0-0-5
•Detailed DB Logging: http://gurge.com/blog/2006/11/09/rails-sql-logging-improvements/
•Ebb: http://ebb.rubyforge.org/
•God (Monitoring): http://god.rubyforge.org/
•Gruff Graphs: http://nubyonrails.com/pages/gruff
•Jmeter: http://jakarta.apache.org/jmeter/
•Magic Multi Connections: http://magicmodels.rubyforge.org/magic_multi_connections/
•Seattle.rb’s Memcache Client: http://github.com/fiveruns/memcache-client/tree/master
•Thin: http://code.macournoyer.com/thin/