redis love at 8tracks

@nettofarah

@nettofarah

8tracks.com/nettofarah netto@8tracks.com

full stack developer at 8tracks.com

8tracks

8tracks

stats

• users: 11,351,961!

• public playlists: 1,545,586 (3,5 mi total)!

• uploaded tracks: 28,185,644

not a sysadmin talk

getting sudo

http://devopsreactions.tumblr.com/post/83704083404/getting-sudo

the right tool for the job

http://peopletakingpictureswithipads.tumblr.com/

RIGHT TOOL

http://peopletakingpictureswithipads.tumblr.com/

FOR THE JOB!

http://peopletakingpictureswithipads.tumblr.com/

engineering effort machine effort

engineering effort machine effort

engineering effort machine effort

engineering effort machine effort

When development tries to blame operations for the outage

http://devopsreactions.tumblr.com/post/84309007419/when-development-tries-to-blame-operations-for-the

R.T.F.J. = engineer <3 + machine <3

key-value store

• open source

• in memory (most of the times)

• data-structure server

data types• strings

• lists

• sets

• sorted sets

• hashes

just like that data structures class

except it is COOL!

you can turn this

into this

next next next next

head

tail

or this

next next next

things to keep in mind

• all in memory (but not restricted to)

• no joins or iterating over data (but not restricted to)

• you should be able to rebuild data

why?

• sometimes basic data structures is all you need

• some problems are hard to fit in a relational way

redis @ 8tracks = <3sidekiq

feature toggles

caching

player (moved to couchdb, then to mysql)

autocomplete (moved to SOLR, then to ElasticSearch)

tag browsing!

news feed

listening sessions

rate limiter

awesome documentation

sidekiq

what’s indie?tag

artistplaylist

1 SELECT name !! ! FROM tags !! ! WHERE name !! ! LIKE 'ind%' LIMIT 10;!!2 SELECT name !! ! FROM mixes !! ! WHERE name !! ! LIKE 'ind%' LIMIT 10;!!3 SELECT login !! ! FROM users !! ! WHERE login !! ! LIKE 'ind%' LIMIT 10;!!4 SELECT name !! FROM artists !! WHERE name !! LIKE 'ind%' LIMIT 10;!

you could do

1 (SELECT name FROM tags WHERE name LIKE 'ind%' LIMIT 10)!2 UNION ALL!3 (SELECT name FROM mixes WHERE name LIKE 'ind%' LIMIT 10);!

the problems

• cross-type search

• I don’t want 10 of each, I want the 10 most relevant ones

• can’t select different column counts

not a fair comparison

people don’t use sql databases for search. they use SOLR (or elasticsearch)

sometimes basic data structures!

is all you need

sorted sets

• score

• add, remove, or update log(n)

• elements in order, fast existence test, fast access to elements in the middle

breaking it down

indie =>

probably irrelevant

relevant

i !! ! ! ! ! ! ! in !!

! ! ! ! ! ! ! ind!! ! ! ! ! ! ! indi!! ! ! ! ! ! ! indie

our collection

The indie summer Indiana Industrial

one zset per n-gram

ind [ ][ ][ ]

indi

indie

which is the same asZADD ind, <score>,

ZADD indi, <score>,

ZADD indie, <score>,

searching

1 ind = $r.zevrange('ind', 0, 10)!!

2 indie = $r.zevrange('indie', 0, 10)!

tag browsing

how?

tags mixes

taggings

a single tag1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 WHERE tg.tag_id = 28!5 LIMIT 10;!

chill

two tags1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 INNER JOIN taggings tg2!5 ON tg2.mix_id = m.id!6 WHERE tg.tag_id = 28!7 AND tg2.tag_id = 12!8 LIMIT 10;! chill

acoustic

adding sort1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 INNER JOIN taggings tg2!5 ON tg2.mix_id = m.id!6 WHERE tg.tag_id = 28!7 AND tg2.tag_id = 12!8 ORDER BY m.first_published_at!9 LIMIT 10;!

ZINTERSTORE

http://redis.io/commands/zinterstore

zinterstore

Time complexity: O(N*K)+O(M*log(M))

intersects two sets, then store them in a new different set

we’ll get to this

chill

acoustic

chill acoustic

+

chill acoustic

1 sets = ['chill', 'acoustic']!2 cache_key = 'chill:acoustic:cache'!3 $r.zinterstore(cache_key, sets, :weights => [1, 1])!

chill

chill acoustic

popular

1 sets = ['popular', 'chill', 'acoustic']!2 cache_key = 'popular:chill:acoustic:cache'!3 $r.zinterstore(cache_key, sets, :weights => [1, 0, 0])!

back to the BIG O thing

O(N*K)+O(M*log(M))

- N: smallest input - K: number of sorted sets - M: number of elements of the resulting set

cool stuff worth checking out

• pipelining

• lua scripting

• replication + persistence

lessons

• it consumes a LOT of memory (duuuh)

• sentinel for failure recovery

• DO NOT USE “keys” in production

• duplication is tricky

gracias!

thank you!

obrigado!