nosql at gumtree
DESCRIPTION
Presentation on the usage of NoSQL technologies at Gumtree.com Covers experiences of using Riak, Redis and MongoDBTRANSCRIPT
![Page 1: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/1.jpg)
Andy Summers, Software Engineer
@NoSQL (or Not Only SQL)
![Page 2: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/2.jpg)
![Page 3: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/3.jpg)
![Page 4: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/4.jpg)
We are the UK’s #1 classifieds website
and in the UK top 30 websites
with over 7 million visitors a month*
Which means almost 1 in 5 UK internet users visit us every month
* Average unique visitors according to ComScore data in 2013
![Page 5: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/5.jpg)
A place which connects people to the things they need…
![Page 6: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/6.jpg)
• 1.7 million live adverts
• 150 million adverts posted in all time
• 100,000 new adverts per day
• Approx 300,000 replies per day
Source: Gumtree data, November 2013
![Page 7: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/7.jpg)
Gumtree Tech
![Page 8: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/8.jpg)
Gumtree Tech
• 15 Back End devs• 6 Front End devs• 3 Mobile App devs• 5 Site Ops• 5 Product Managers• 6 QA• 3 Designers
![Page 9: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/9.jpg)
Redis
RiakCassandra
mongoDBCouchDB
Neo4j
![Page 10: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/10.jpg)
•Replies
•Sessions
•A few other things
![Page 11: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/11.jpg)
Replies
![Page 12: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/12.jpg)
v.1Mail
ServerMail
Server
REST API
Anony-mizer
CassandraMy
SQL
MobileApps
![Page 13: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/13.jpg)
• OSGI
• Two sources of truth?!
![Page 14: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/14.jpg)
MySQL MySQL
Scalability?
![Page 15: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/15.jpg)
Cassandra
?
![Page 16: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/16.jpg)
‘We need something better’
![Page 17: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/17.jpg)
v.2Mail
ServerMail
Server
REST API
Anony-mizer
ElasticSearch
RIAK
MobileApps
![Page 18: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/18.jpg)
‘Just a Key-Value Store’
• Real-time Querying?
• Reporting?
• My manager wants to know how many of x happened in y?
![Page 19: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/19.jpg)
“But all of this
works fine in
MySQL…”
![Page 20: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/20.jpg)
Use the Right Tool
for the Right Job
![Page 21: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/21.jpg)
The German Story
![Page 22: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/22.jpg)
Using iak for ReplyTS
15TB
18Nodes
2Data
centers
![Page 23: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/23.jpg)
And then we Launched.
And everything was fine!
![Page 24: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/24.jpg)
Well, at least 3 Months…
![Page 25: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/25.jpg)
![Page 26: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/26.jpg)
![Page 27: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/27.jpg)
alwaysNew Technology
fails!
![Page 28: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/28.jpg)
Backup... and if you can’t: have a backup plan
Have a
![Page 29: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/29.jpg)
Dear Basho Devs,
![Page 30: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/30.jpg)
So where are Gumtree in all this?
• Keeping a close eye on eBay Kleinanzeigen’s experiences
• Acknowledging that staying with the MySQL and Cassandra solution, is not suitable for Gumtree’s long term needs
![Page 31: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/31.jpg)
Sessions
![Page 32: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/32.jpg)
Richard
Gumtree servers
Interweb
Improving support for regular software releases
![Page 33: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/33.jpg)
Oh… That’s not good.
• Off-peak release times only
• Sticky sessions have a lot to answer for
• Have to restart instance => Kill everyone on it
![Page 34: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/34.jpg)
redis
![Page 35: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/35.jpg)
Redis
• ‘Just another key-value store’
• Data typing
• TTL support
• Support for master-slave, monitoring, publish-subscribe
![Page 36: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/36.jpg)
Richard
Gumtree servers
Interweb
Success!
Redis
![Page 37: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/37.jpg)
Redis – all good?• Single master = single point of failure
• Sentinel – potential 66 seconds of downtime when master fails?
• Sharding is never easy
• Not tested dual data centre setup
![Page 38: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/38.jpg)
The Other Stuff
![Page 39: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/39.jpg)
mongoDB
![Page 40: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/40.jpg)
• Popular Search Tracking
– High realtime write-traffic
– Regretting this a lot
• Some low traffic use cases
– Low traffic, low read
• Some batch event analysis
– Batched reads and writes
![Page 41: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/41.jpg)
In Conclusion
• Data consistency vs. High availability
• NoSQL comes in many different guises, technologies suited for different use cases
• No technology is perfect
![Page 42: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/42.jpg)
If you enjoyed this talk and like Gumtree
• eBay Tech Meetups across Europe– http://www.meetup.com/eBay-Europe-Technology
• Gumtree Careers via LinkedIn– http://linkd.in/1eu4rK5
![Page 43: NoSQL at Gumtree](https://reader035.vdocument.in/reader035/viewer/2022062313/557d76dcd8b42a30708b5459/html5/thumbnails/43.jpg)
Thank You!
…Questions or comments?