fun with ruby and redis, arrrrcamp edition, javier_ramirez, teowaki
DESCRIPTION
In this talk I make an introduction to Redis, then I explain how some big names (twitter, pinterest...) are using it, then I describe some pitfalls, then I explain how we are using redis at teowakiTRANSCRIPT
Fun with
&
arrrrcamp 2013javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
javier ramirez@supercoco9
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
friday hug anyone?
hugs and hooks:
arrrr yousure ?
In the beginner's mind there are many possibilities, in the expert's mind there are few.
Shunryu Suzuki
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
myth: the bottleneck
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (with pipelining)
$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 552,028 requests per secondGET: 707,463 requests per secondLPUSH: 767,459 requests per secondLPOP: 770,119 requests per second
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (without pipelining)$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q
SET: 122,556 requests per secondGET: 123,601 requests per secondLPUSH: 136,752 requests per secondLPOP: 132,424 requests per second
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
myth: arrrrRedis is just like memcached
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
myth: rrrrRedis is for queues
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.
http://redis.io
started in 2009 by Salvatore Sanfilippo @antirez
100 contributors at https://github.com/antirez/redis
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
hacker meets thinker*
*@janogonzalez and @advi inspired slide
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
The arrrrRedis Manifesto1.A DSL for Abstract Data Types
2.Memory storage is #1
3.Fundamental data structures for a fundamental API
4.Two levels of API
5.Code is like a poem; it's not just something we write to reach some practical result
6.We're against complexity
7.We optimize for joyjavier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
basics: setting and getting
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
basics: counterrrrrs
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
basics: sets
cool example about sunionstore
http://robots.thoughtbot.com/post/46335890055/redis-set-intersection-using-sets-to-filter-data
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
Redis keeps everrrrrything in memorrrrry all the time
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
Perrrrrsistance: RDB Compact binary formatSaves snapshots every few minutesGood for backups and synchronizing
If Redis crashes, a few minutes worth of data will be lost
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
Durrrrrability: AOF
Log text formatConfigurable durabilityLarge file, can slow down startup
If Redis crashes, typically one second of data could be lost
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
rrrrreplication
one master, several read-only slaves
sharding
Twemproxy & redis clusterjavier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
atomicity
single threaded, so no concurrency problems
transactions and lua scripts to run multiple operations atomically
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
Scrrrrripting with luaYou can use Lua for scripting Redis when you need to atomically execute a sequence of commands in which the output of a command is used as input for another
It reduces the need to use complex lock mechanisms and simplifies dependencies between clients
You can even extend the functionality of Redis by using Lua scripts
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
what's being used forrrrr
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
twitterrrrr
Every time line (800 tweets per user) is on redis
5000 writes per second avg300K reads per second
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
user info from gizmoduck(memcached)
user id tweet id metadata
write API (from browser or client app)
rpushx to Redis
tweet info from tweetypie (memcached + mysql)
your twitter timeline
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
fanout (flockDB)one per follower
pinterest object grrrrraph
from mysql+memcached to redis
30% save on IOps
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
per userrrrrRedis SortedSet, with timestamp as the score, is used to store the users followed explicitly. Redis SortedSet, with timestamp as the score, is used to store the users followed implicitlyRedis SortedSet, with timestamp as the score, is used to store the user’s explicit followersRedis SortedSet, with timestamp as the score, is used to store the user’s implicit followersRedis Set is used to store boards followed explicitlyRedis Set is used to store boards unfollowed explicitly
per boarrrrrdRedis Hash is used to store a board’s explicit followersRedis Set is used to store a board’s explicit unfollowers
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
viacomObject dependency graph. Cache on steroids
Redis as a queue for background jobs
Activity tracking and view counts buffering before saving to mysql
Lua script working on a slave to recalculate ranking and popularity of contents, then send the data to master. The new process takes 1/60th less than the old one in mysql
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
openredis
redis as a service on AWS
serving over 1000 customers with a single machine
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
when things go wrong
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
the instagrrrrram case
moving from redis to cassandra: 75% savings on servers
lesson learnt: know when redis is not appropriate
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
the twilio case
credit card hell
lesson learnt: know what you are doing. Don't change config on the fly
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
how teowaki is using redis
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
Abusing sidekiq/resque
keep track of every activity in the system, even if you don't need them all right now:- every page view- every API request- every time a record is created/updated/deleted
benefits:- highly decoupled system- easier to divide into services- you can add behaviour without changing your app
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
Some of our uses of LuaExpiring attributes inside a Redis hash
Inserting notifications into a list only if there are notpending notifications from the same user for the same scope
Paginating a list by an attribute
Manipulating JSON directly at the Redis layer
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
counterrrrrs
Atomic counters can be safely invoked concurrently from anywhere, so you can implement “like” features, global sequences or usage monitoring systems in highly concurrent applications for free.
You can share your counters with any other internal application and still be sure they won't collide.
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
Temporarrrrry dataRedis allows us to self expire keys after a time has passed. You can use this mechanism for a simple cache
If you operate on a key with expiration time, you can change its value and still keep the expiration going. By combining a decrementing counter with an expiration time, implementint usage quotas is trivial
We are controlling our API usage quota with a lua script following this pattern
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
bloom filterrrrrsbloom filter: space-efficient probabilistic data structure that is used to test whether an element is a member of a set. False positive matches are possible, but false negatives are not.
Redis bit operations make easy to implement bloom filters
We are using bloom filters for checking uniqueness of user names without going to postgresql
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
nginx + lua + redisapache + mruby + redis
Multiple levels of cache by using Redis on the webserver/middleware layer
http://wiki.nginx.org/HttpRedishttps://github.com/jodosha/redis-store
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
summarrrrrizing* Redis is more powerful than it seems
* Very fast, easy to use, simple, good documentation
* In-memory data structures, distributed, shared and persisted
* Good as data store, intermediate data store, cache or queue
* Lots of use cases, both in huge and smaller systems
You should probably use it a lot more
javier ramirez @supercoco9 http://teowaki.com arrrrcamp 2013
If you enjoyed this presentation, please thank
me by registering on http://teowaki.com
It's a site for developers, you can hang around for free, and I think it's quite cool
<3 <3 <3Javier Ramírez
@supercoco9
arrrrcamp 2013