redis use patterns (devcontlv june 2014)
DESCRIPTION
An introduction to Redis for the SQL practitioner, covering data types and common use cases. The video of this session can be found at: https://www.youtube.com/watch?v=8Unaug_vmFITRANSCRIPT
![Page 1: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/1.jpg)
Redis Use PatternsAn Introduction to the SQL
Practitioner@ItamarHaber #DevConTLV 2014
![Page 2: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/2.jpg)
About
A Redis Geek and Chief Developers Advocate at .com
We provide Redis Cloud – an enterprise-class Redis service for developers (infinitely scalable, highly-available with auto failover, top performing hosted Redis off AWS, Google, Azure & IBM SoftLayer)
Get your free t-shirt and
sticker outside!
![Page 3: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/3.jpg)
What’s Redis? (REmote DIctionary Server)• Open-source (BSD), in-memory, persist-able key-value advanced datastore• Key-value means something like
CREATE TABLE redis (k VARCHAR(512MB) NOT NULL,v VARCHAR(512MB),PRIMARY KEY (k)
);
• 6 data types, 160 commands, blazing fast• Created in 2009 by @antirez
(a.k.a Salvatore Sanfilippo)• Source: https://github.com/antirez/redis• Website: http://redis.io
![Page 4: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/4.jpg)
Why Redis? Because It Is Fun!
• Simplicity rich functionality, great flexibility• Performance easily serves 100K’s of ops/sec• Lightweight ~ 2MB footprint• Production proven (name dropping)
Twitter Pintrest StackOverflow teowaki many more…
![Page 5: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/5.jpg)
RedisMakesYou....
THINK!
• about how data is stored• about how data is accessed• about efficiency• about performance• about the network• …
• Redis is a database construction kit
• Beware of Maslow's "Golden" Gavel/Law of Instrument:
"If all you have is a hammer, everything looks like a nail"
![Page 6: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/6.jpg)
Pattern: Caching Calls to the DB
Motivation: quick responses, reduce load on DBMSHow: keep the statement's results using the Redis STRING data type
def get_results(sql):
hash = md5.new(sql).digest()
result = redis.get(hash)
if result is None:
result = db.execute(sql)
redis.set(hash, result)
# or use redis.setex to set a TTL for the key
return result
![Page 7: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/7.jpg)
STRINGs
• Are the most basic data type• Are binary-safe• Is used for storing:• Strings (duh) – APPEND, GETRANGE,
SETRANGE, STRLEN• Integers – INCR, INCRBY, DECR, DECRBY• Floats – INCRBYFLOAT• Bits – SETBIT, GETBIT, BITPOS, BITCOUNT,
BITOP
http://xkcd.com/171/
![Page 8: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/8.jpg)
Pattern: Avoiding Calls to the DB
Motivation: server-side storage and sharing of data that doesn't need a full-fledged RDBMS, e.g. sessions and shopping cartsHow: depending on the case, use STRING or HASH to store data in Redis
def add_to_cart(session, product, quantity):
if quantity > 0:
redis.hset('cart:' + session, product, quantity)
else:
redis.hrem('cart:' + session, product)
def get_cart_contents(session):
return redis.hgetall('cart:' + session)
![Page 9: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/9.jpg)
The HASH Data Type
• Acts as a Redis-within-Redis contains key-value pairs• Have their own commands: HINCRBY, HINCRBYFLOAT, HLEN, HKEYS, HVALS
…• Usually used for aggregation, i.e. keeping related data together for easy
fetching/updating (remember that Redis is not a relational database). Example:
Using separate keys Using hash aggregationuser:1:id 1 user:1 id 1user:1:fname Foo fname Foouser:1:lname Bar lname Baruser:1:email [email protected] email [email protected]
![Page 10: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/10.jpg)
Denormalization
• Non relational no foreign keys, no referential integrity constraints• Thus, data normalization isn't practical• Be prepared to have duplicated data, e.g.:> HSET user:1 country Mordor
> HSET user:2 country Mordor
…• Tradeoff:
Processing Complexity ↔ Data Volume
![Page 11: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/11.jpg)
Pattern: Lists of Items
Motivation: keeping track of a sequence, e.g. last viewed profilesHow: use Redis' LIST data type
def view_product(uid, product):
redis.lpush('user:' + uid + ':viewed', product)
redis.ltrim('user:' + uid + ':viewed', 0, 9)
…
def get_last_viewed_products(uid):
return redis.lrange('user:' + uid + ':viewed', 0, -1)
![Page 12: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/12.jpg)
Key Points About Key Names
• Key names are "limited" to 512MB (also the values btw)• To conserve RAM & CPU, try avoid using
unnecessarily_longish_names_for_your_redis_keys because they are more expensive to store and compare (unlike an RDBMS's column names, key names are saved for each key-value pair)• On the other hand, don't be too stringent (e.g 'u:<uid>:r')• Although not mandatory, the convention is to use colons
(':') to separate the parts of the key's name• Your schema is your keys' names so keep them in order
![Page 13: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/13.jpg)
Pattern: Queues (apropos the list data type)Motivation: a producer-consumer use case, asynchronous job management, e.g. processing photo uploads
def enqueue(queue, item):
redis.lpush(queue, item)
def dequeue(queue):
return redis.rpop(queue)
# or use brpop for blocking pop
![Page 14: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/14.jpg)
Is Redis ACID? (mostly) Yes!
• Redis is (mostly) single threaded, hence every operation is• Atomic• Consistent• Isolated
• WATCH/MULTI/EXEC allow something like transactions (no rollbacks)• Server-side Lua scripts ("stored procedures")
also behave like transactions• Durability is configurable and is a tradeoff
between efficiency and safety
![Page 15: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/15.jpg)
Pattern: Searching
Motivation: finding keys in the database, for example all the usersHow #1: use a LIST to store key namesHow #2: the *SCAN commands
def do_something_with_all_users():
first = True
cursor = 0
while cursor != 0 or first:
first = False
cursor, data = redis.scan(cursor, 'user:*')
do_something(data)
![Page 16: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/16.jpg)
Pattern: Indexing
Motivation: Redis doesn't have indices, you need to maintain themHow: the SET data type (a collection of unordered unique members)
def update_country_idx(country, uid):
redis.sadd('country:' + country, uid)
def get_users_in_country(country):
return redis.smembers('country:' + country)
![Page 17: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/17.jpg)
Pattern: Relationships
Motivation: Redis doesn't have foreign keys, you need to maintain them
> SADD user:1:friends 3 4 5 // Foo is social and makes friends
> SCARD user:1:friends // How many friends does Foo have?
> SINTER user:1:friends user:2:friends // Common friends
> SDIFF user:1:friends user:2:friends // Exclusive friends
> SUNION user:1:friends user:2:friends // All the friends
![Page 18: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/18.jpg)
ZSETs (Sorted Sets)I HAVE
CDOIT'S LIKE
OCDBUT ALL THE LETTERS ARE
IN ALPHABETICAL ORDERAS THEY SHOULD BE
• Are just like SETs:• Members are unique• ZADD, ZCARD, ZINCRBY, …
• ZSET members have a score that's used for sorting• ZCOUNT, ZRANGE, ZRANGEBYSCORE
• When the scores are identical, members are sorted alphabetically• Lexicographical ranges are also supported:
• ZLEXCOUNT, ZRANGEBYLEX
![Page 19: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/19.jpg)
Pattern: Sorting
Motivation: anything that needs to be sortedHow: ZSETs
> ZADD friends_count 3 1 1 2 999 3
> ZREVRANGE friends_count 0 -1
3
1
2
Set members (uids)
Scores (friends count)
![Page 20: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/20.jpg)
The SORT Command
• A command that sorts LISTs, SETs and SORTED SETs• SORT's syntax is the most complex (comparatively) but SQLers should
feel right at home with it:SORT key [BY pattern] [LIMIT offset count][GET pattern [GET pattern ...]][ASC|DESC] [ALPHA][STORE destination]
• SORT is also expensive in terms of complexity O(N+M*log(M))• BTW, SORT is perhaps the only ad-hoc-like command in Redis
![Page 21: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/21.jpg)
Pattern: Counting Things
Motivation: statistics, real-time analytics, dashboards, throttlingHow #1: use the *INCR commandsHow #2: use a little bit of BIT*def user_log_login(uid):
joined = redis.hget('user:' + uid, 'joined')
d0 = datetime.strptime(joined, '%Y-%m-$d')
d1 = datetime.date.today()
delta = d1 – d0
redis.setbit('user:' + uid + ':logins', delta, 1)
def user_logins_count(uid):
return redis.bitcount(
'user:' + uid + ':logins', 0, -1)
I love to COUNT(*) data!
One datum,two data,
three data!HA HA HA HA!
![Page 22: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/22.jpg)
Pattern: Counting Unique Items
How #1: SADD items and SCARD for the count Problem: more unique items more RAM
How #2: the HyperLogLog data structure> PFADD counter item1 item2 item3 …• HLL is a probabilistic data structure that counts (PFCOUNT) unique items• Sacrifices accuracy: standard error of 0.81%• Gains: constant complexity and memory – 12KB per counter• Bonus: HLLs are merge-able with PFMERGE
![Page 23: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/23.jpg)
Wait, There's More!
• There are 107 additional commands that we didn't cover • Expiration and eviction policies• Publish/Subscribe• Data persistency and durability• Server-side scripting with Lua• Master-Slave(s) replication• High availability with Sentinel• Redis v3 == Cluster (currently in beta)• …
![Page 24: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/24.jpg)
Where Next?
• Try the interactive demo and get a free 25MB Redis database in the cloud at http://redislabs.com• Need help?• RTFM: http://redis.io/documentation• Ask the redis-db mailing list• Visit #redis on Freenode IRC• Email me: itamar@ .com
![Page 25: Redis Use Patterns (DevconTLV June 2014)](https://reader033.vdocument.in/reader033/viewer/2022042814/54c676e34a7959b6298b45a0/html5/thumbnails/25.jpg)