couchbase_uk_2013_skyscanner_and_couchbase
TRANSCRIPT
12 April 2023
Couchbase at Skyscanner
Who are Skyscanner?
Edinburgh
website apps
traditional
mobile site
OTAsbudgetbudget
How does Skyscanner work?
budget
Live Pricing Service
Browse Cache
Service
A route-date specific search
The end result
Powering a live pricing search
Prices +Timetables
Data Collection Services
1) Which websites should we show?
2) What prices do we already have
cached?
3) Live update what we still
need. 4) Clean up and save the new data
5) Return the prices to the user.
Live Pricing Service
Scaling the live pricing cache
The original solution
Price Cache
Live Pricing Service
Solution 1 – Split up Reads and Writes
Read Only Cache, writing queues of new data
Quote Processors
Replication
Live Pricing Service
Solution 2 – A temporary local cache
Live Pricing Service
Cached Prices (key/value)
At this scale
2 bn itineraries and quotes
270 GB
table
250 GB
indices
2000 quotes per second
What we really needed
Consistency
Persistence
Speed
Resilience
Horizontal Scaling
Elasticity
Simplicity
Solution 3 – Introduce Couchbase
Live Pricing Service
Legacy read-only cache
Legacy quote Processors
The Documents
struct TransportQuoteSet { 1: string MarketingAgentId, 2: common.PassengerGroup PassengerGroup, 3: i32 OutboundDate, 4: bool OneWay, 5: optional i32 InboundDate, 6: common.Place OriginPlace, 7: common.Place DestinationPlace, 8: common.Combinability Combinability, 9: bool Comprehensive 10: list<TransportQuote> Quotes, 11: list<ChildTransportQuoteSet> ChildTransportQuoteSets, 12: optional i32 UtcQuoteDateTime 13: optional bool Additional 14: optional string CabinClass}
The actual cluster
The buckets
The quotes bucket
Some specifics on the setup
Dedicated 6 servers:
SSDs
1 replica of data
Taking nodes in and out of the cluster
Performance with multiple buckets
Monitoring and reporting in one place
2 x Quad core Intel i7 2.0ghz
256GB of 1600mhz RAM
Dell R620 1U.
Centos 6.2
3 x 600gb MLC SSD each (Raid1 + Hot spare)
Going beyond a cache.
Running a car hire search
Prices + Reference Data
Data Collection Services
1) Which websites should we show?
2) What prices do we already have
cached?
3) Live update what we still
need. 4) Save and enrich the new data
5) Return the prices to the user.
Car Hire Live Pricing Service
Reference Data
Supplier-Location
Map reduce views
python Couchbase 2.0 client lib
Car Codes { "website": { "published": true, "id": "affd", ... }, "office_id": "1", "city_id": "AUHA", "raw_data": [...] "address": "closing_time": "00:00", "routenodeid": "9618", "type": "office“}
Price Data
Python Smart Client Isn’t Ready, so…
Cluster Info
3 nodes
Virtual
Benched to 1000+ ops/sec
SASL Auth.
What’s next?
A flexible search
The results.
What has Couchbase brought us?
quartermile one15 lauriston placeedinburgh eh3 9en
tel: +44 (0)131 252 5353fax: +44 (0)131 252 5354
twitter: @skyscannerfacebook: skyscanner
Thank you for listening.
Alistair [email protected]