redis: remote dictionary server
DESCRIPTION
TRANSCRIPT
![Page 1: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/1.jpg)
Redis
REmote DIctionary Server
Ezra Zygmuntowicztwitter: @ezmobius#redis
![Page 2: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/2.jpg)
• Fast, in memory key/value store
• STRING, LIST, SET & ZSET data types
• Persistence via async snapshots or AOF
• Perfect Data Structure/State/Cache Server
![Page 3: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/3.jpg)
Data Structure Server
![Page 4: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/4.jpg)
Data Structure ServerKey:String => Value:String
![Page 5: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/5.jpg)
Data Structure ServerKey:String => Value:String
Key:String => Value:List
![Page 6: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/6.jpg)
Data Structure ServerKey:String => Value:String
Key:String => Value:List
Key:String => Value:Set
![Page 7: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/7.jpg)
Data Structure ServerKey:String => Value:String
Key:String => Value:List
Key:String => Value:Set
Key:String => Value:Zset
![Page 8: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/8.jpg)
Operations on any Type
• Exists, Del, Type
• Keys, Randomkey
• Rename, RenameNX
• Dbsize, Select, Move, Flushdb, Flushall
• TTL, Expire
![Page 9: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/9.jpg)
Operations on STRING’s
• Get, Set, GetSet, SetNX
• Mget, Mset, MgetNX, MsetNX
• Incr, Incrby
• Decr, Decrby
![Page 10: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/10.jpg)
Operations on LISTS’s
• Atomic Operations:
• Push/Pop
• Index (array indexing)
• Lrange, Ltrim, Llen
• Blpop, Brpop
• RpopLpush, BRpopLpush
![Page 11: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/11.jpg)
Operations on SET’s
• Sadd, Srem, Spop, Smove
• Scard, Sismember, Smembers, Srandmember
• Sinter, Sinterstore, Sunion, Sunionstore
• Sdiff, Sdiffstore
![Page 12: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/12.jpg)
Operations on ZSET’s
• Zadd, Zrem, Zincrby
• Zrange, Zrevrange
• Zrangebyscore, Zcard
• Zscore, Zremrangebyscore
![Page 13: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/13.jpg)
Seems cool, but what are the use cases?
• Memcached on Steroids
• Tag Clouds, Leaderboards
• Stat collections, circular log buffers
• Share state between processes
• A/B testing
• REDIStribute your load
![Page 14: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/14.jpg)
Example: Tagging
![Page 15: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/15.jpg)
Example: Tagging
SADD article:42 magickSADD article:42 unicornsSADD article:42 rainbows
![Page 16: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/16.jpg)
Example: Tagging
SADD article:42 magickSADD article:42 unicornsSADD article:42 rainbows
SADD article:12 magickSADD article:12 barSADD article:12 quxSADD article:12 foo
![Page 17: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/17.jpg)
Example: TaggingSADD article:42 magick
SADD article:42 unicornsSADD article:42 rainbows
SADD article:12 magickSADD article:12 barSADD article:12 quxSADD article:12 foo
SINTER article:42 article:12#=> magick
![Page 18: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/18.jpg)
Example: Fair Work Scheduler
![Page 19: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/19.jpg)
Example: Fair Work Scheduler
![Page 20: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/20.jpg)
Example: Fair Work Scheduler
Each worker node periodically issues: ZADD worker:nodes 1.5 hostname
1.5 is the load ave of the host
![Page 21: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/21.jpg)
Example: Fair Work Scheduler
Each worker node listens for work with: BLPOP hostname 10
![Page 22: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/22.jpg)
Example: Fair Work Scheduler
When a producer wants to issue a work request:ZRANGE worker:nodes 0 2
returns top 3 least loaded nodes
![Page 23: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/23.jpg)
Example: Fair Work Scheduler
Pick a random node out of the 3 least loaded nodesin order to add jitter so we don’t overload hosts
![Page 24: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/24.jpg)
Example: Fair Work Scheduler
Then issue the work request:LPUSH hostname {“work”: {“foo”: “bar”}}
![Page 25: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/25.jpg)
Example: Fair Work Scheduler
This setup uses ZSETS and the load average as a scorein order to evenly distribute load across a farm ofworker instances listening for jobs with BLPOP
![Page 26: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/26.jpg)
Example: AMQP message de-dupe
![Page 27: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/27.jpg)
Example: AMQP message de-dupe
Route messages through 2 separate rabbitmq brokers for redundancy
Use redis for message de-dupe by tagging messages with a guid
Use guid as key into redis
Use SETNX when setting the guid key so first one wins and second messages gets discarded as already received
![Page 28: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/28.jpg)
Other Ideas
• Distributed Lock Manager for process coordination
• Full Text Inverted Index Lookups
• Caching with extra smarts using lists, sets and atomic ops on said structures
• Share data structures between multiple processes like a blackboard/tuplespace
![Page 29: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/29.jpg)
Speed
• 110k GETS/SETS/second on average linux box
• Mostly faster then memcached for same ops
• Event Driven, can handle thousands of clients with epoll/kqueue support
• Ops happen in memory, persistence is async so everything is very fast
![Page 30: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/30.jpg)
Persistence
• Async Snapshots with configurable triggers
• Append Only File: AOF
• Redis 2.0 Virtual Memory
![Page 31: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/31.jpg)
Replication
• Built in Master -> Slave Async replication
• Create replication chains as needed
![Page 32: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/32.jpg)
Sharding
• Client side sharding (like memcached)
• Consistent ring hashing
• Special case keys “foo{tags}” for operations on keys that must live on the same server in ruby client
![Page 33: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/33.jpg)
Redactor• Simple Actor Library based around Redis
![Page 34: Redis: REmote DIctionary Server](https://reader031.vdocument.in/reader031/viewer/2022020206/5491bbe0ac79592f288b45ee/html5/thumbnails/34.jpg)
Questions?
http://code.google.com/p/redis/http://github.com/ezmobius/redis-rb