Redis And Python

PyCon India, 2011Sunil Arora

Raising Hands...

How many of you have used Redis before ?

How many of you have got a laptop ?

About Me

I tweet at Sunil Arora / @_sunil_

I work at ShopSocially

I blog at http://sunilarora.org

Today's talk

What is Redis

How it works and what you can do with it

Real life use-casesReal life use-cases

Some hand-on with Redis

Redis - Brief History

Initially written to improve performance of Web Analytics product LLOOGG out of his startup

Salvatore Sanfilippo

@antirez

http://antirez.com

Redis – Brief History

Released in March 2009 (Open Source, BSD licensed)

VMWare hired Salvatore in March, 2010

Then Pieter Noordhuis (key contributor) was hired

What is Redis ?

Its between lot of stuff, so difficult to categorize it precisely

What is Redis ?

I see Redis definitely more as a flexible tool that as a

solution specialized to solve a specific problem: his mixed soul of cache,

store, and messaging server shows this very

well

-Salvatore Sanfilippo

Picture by herzogbrhttp://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/

Redis is...

Remote Data Structure Server

Redis is...

Supports rich data types of computer science

- Strings, Lists, Sets, Sorted Sets, Hashes...

Rich sets of primitives (commands) to manipulate these types

Predictive complexity measurements

A few fundamentals

Written in C (no external dependency) Uses memory as main storage Uses disk for persistence Single Threaded Every command performs atomic execution

Performance

Screamingly fast performance ~50K read/write operations per seconds 100K+ read/write ops per second on a regular

EC2 instance

Installation

$ git clone http://github.com/antirez/redis

$ cd redis

$ make

$ ./src/redis-server

..........

..........

$./src/redis-cli

Redis> PING

PONG

Python Libraries

Redis-py - Python client library

Txredisapi - An asynchronous Python client for the Redis database, based on Twisted.

Redisco - ORM for redis along the lines Ohm for Ruby

redis-py

The most popular python client library Andy McCurdy (sedrik@gmail.com) Github: http://github.com/andymccurdy/redis-py

$easy_install redis OR

$pip install redis Optional

$easy_install hiredis

$pip install hiredis

Lets get started...

$ cd redis

$ ./src/redis-server

.........

>>> from redis import Redis

>>> redis_client = Redis()

>>> redis_client.keys()

>>> help(redis_client)

Redis Keys

Not binary safe. Should not contain space or newline character A few rules about keys:

Too long keys are not a good idea Too short keys is also not a good idea “object-type:id:field” can be a nice idea, i.e.

“user:1001:name”

Operations on Keys

KEYS EXISTS DEL EXPIRE OBJECT PERSIST

RANDOMKEY RENAME TYPE TTL EXPIREAT MOVE

Lets play with keys

>>>redis_client.keys()

>>>redis_client.exists('key')

>>>redis_client.delete('key')

>>>redis_client.type('key')

>>>......

>>>......

Data Structures

Strings Lists Sets Sorted Sets Hashes

Strings

SET GET MSET MGET SETEX GETSET SETNX

INCR INCRBY DECR DECRBY

Strings – with redis client

>>> redis_client.set('key', 'value')

>>> redis_client.get('key')

>>> redis_client.delete('key')

Fetch multiple keys at once

mget/mset redis_client.mset({'key1': 'val1', 'key2': 'val2'}) redis_client.mget('key1', 'key2',......)

Expiration

Set a value with expire

>>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs

>>>redis_client.expire('key', 2)

>>>redis_client.get('key')

>>>None

Expire Semantics

Key with expiry known as volatile keys

Whenever a volatile key is modified, its expiry is reset

- To avoid inconsistency in following cases• Replication

• Append Only Log

Uses

To store transient states in your web application

Uses

Who is online?

Uses

Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)

Atomic Increments

>>>help(redis_client.incr)

>>>help(redis_client.decr)

>>>

>>> redis_client.incr('counter', 1)

>>>1

>>> redis_client.incr('counter')

>>>2

>>> redis_client.incr('counter')

>>>3

Uses

High Speed counters (views/clicks/votes/likes..)

Uses

API Rate Limiting

Uses

Generating unique IDs

Lists

Ordered list of binarysafe strings Doubly linked list Memory footprint optimized for smaller list O(1) insertion/deletion at both ends

Lists - operations

LPUSH RPUSH LSET LRANGE LPOP BLPOP BRPOP BRPOPLPUSH

LINSERT RPOP RPOPLPUSH LPUSHX RPUSHX

Uses

Web apps are full of lists :)

Uses

Capped List

Uses

Real time message QueueBackground Worker queues (Resque, Celery)

https://github.com/blog/542-introducing-resque

Uses

Social Activity Streams or notifications

Sets

An unordered collection of distinct byte strings

Nothing different from the data type in python

Sets - Operations

SADD SCARD SREM SISMEMBER SMEMBERS SPOP SRANDMEMBER

Operations between Sets

SMOVE SUNION SDIFF SINTER

Sets - Operations

SDIFFSTORE SINTERSTORE SUNIONSTORE

Sets - Uses

Picking random items from a set using SRANDMEMBER

Ex.

– Picking a random article from daily news

– Pick a random Ad for serving

– Pick a random option for A/B

Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”

Sets - Uses

To model Relations in social graph Ex.

>>>redis_client.sadd('friends:john', 'jenny', 'maria')

>>>redis_client.sadd('friends:ben', 'maria', 'kate')

>>>redis_client.sinter('friends:john', 'friends:ben')

Relations (friend/followers)

Common followlist for A and B

>>>sinter('users:A:follows', 'users:B:follows')

Unique to B compared to C

>>>sdiff('users:B:follows', 'users:C:follows') Mutual relationship (friend as well as follower)

>>>sinter('users:B:followers', 'users:B:friends')

Who does not follow me back ?>>>sdiff('users:B:friends', 'users:B:followers')

Who am I not following back?>>>sdiff('users:B:followers', 'user:B:friends')

Which of my friend's are online right now?

http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/

SINTER online_people my_friends

Which of my friends are online right now?

>>>RENAME online:fresh online:stale #every minute

>>>SUNIONSTORE online:fresh online:stale

>>>#friend jack connects

>>>SADD online:fresh 'jack'

>>>SUNIONSTORE online online:fresh online:stale

>>>#who is online ?

>>>SINTER online friends

Sorted Sets

Ordered sets on the basis of score

Sorted Sets

ZADD ZCARD ZCOUNT ZINCRBY ZINTERSTORE ZRANGE ZRANGEBYSCORE ZRANK

ZREM ZREMRANGEBYRA

NK ZREMRANGEBYSC

ORE ZREVRANGE ZREVRANGEBYSCO

RE ZSCORE ZUNIONSTORE

Sorted Sets – Use Case

To build index on your dataset

Uses - Realtime Leaderboards

>>>zincrby('leaderboard', 'john', 2)

>>>zincrby('leaderboard', 'jack', 5)

>>>zincrby('leaderboard', 'kate', 1)

>>>zincrby('leaderboard', 'kate', 10)

....

>>>zrange('leaderboard', 0, -1, withscores = True)

Uses - Realtime Leaderboards

Most download resources

Most popular articles on the website

Weekly popular list

Most downloaded stuff between date1 and date2

.....

.....

AutoComplete with Sored Sets

http://antirez.com/post/autocomplete-with-redis.html

Hashes

Equivalent to Python dictionary or Ruby hash or Java hashmap

Operations– hmset users:1 {'username': 'jim', 'score': 23}

– hgetall users:1

– Hincrby Useful for storing structured data

– hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'}

– Expire user:1:preferences <duration>

– Hmgetall user:1:preferences #returns entire dict

Redis Transactions

Using Multi/Watch

>>>p = redis_client.pipeline()

>>>p.lpush('a', 1)

>>>p.ltrim('a', 0, 100)

>>>p.execute Limitation

– Since commands are queued, Can't read values

– No conditional execution

– If not high write contention scenario, WATCH can be used

Use Redis Scripting to write your own primitive

Designing with Redis

Data layout design on basis of Query

No Query Optimizer

Manually build indexes

Durability

Snapshotting mode

– Binary dump every x secs or y ops Append Only File (AOF)

– Every command is written to a file

– On restart/crash, commands replayed

– Fsync on every new command

– Fsync every second

– OS decide to Replication

Publish/Subscribe

A simple and efficient implementation of publish/subscribe messaging paradigm

Client can subscribe/psubscribe to receive messages on channels (key patterns)

Publish/Subscribe

PSUBSCRIBE PUBLISH PUNSUBSCRIBE SUBSCRIBE UNSUBSCRIBE

Uses

Many to Many message passing

Uses

Web Chat

References

Redis Website (http://redis.io) SimonWillison Tutorial (

http://simonwillison.net/static/2010/redis-tutorial/) Redis from ground up (

http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19)

Redis under the hood (http://pauladamsmith.com/articles/redis-under-the-hood.html)

Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications)

References

Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/)

Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/

Redis at Bump

http://devblog.bu.mp/how-we-use-redis-at-bump

Redis: AK 47 of postrelational database

http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases

Questions

Thanks

Contact details

Twitter: @_sunil_Email: arora.sunil AT gmail.com