couchbase_uk_2013_skyscanner_and_couchbase

12 April 2023

Couchbase at Skyscanner

Who are Skyscanner?

Edinburgh

website apps

traditional

mobile site

OTAsbudgetbudget

How does Skyscanner work?

budget

Live Pricing Service

Browse Cache

Service

A route-date specific search

The end result

Powering a live pricing search

Prices +Timetables

Data Collection Services

1) Which websites should we show?

2) What prices do we already have

cached?

3) Live update what we still

need. 4) Clean up and save the new data

5) Return the prices to the user.

Live Pricing Service

Scaling the live pricing cache

The original solution

Price Cache

Live Pricing Service

Solution 1 – Split up Reads and Writes

Read Only Cache, writing queues of new data

Quote Processors

Replication

Live Pricing Service

Solution 2 – A temporary local cache

Live Pricing Service

Cached Prices (key/value)

At this scale

2 bn itineraries and quotes

270 GB

table

250 GB

indices

2000 quotes per second

What we really needed

Consistency

Persistence

Speed

Resilience

Horizontal Scaling

Elasticity

Simplicity

Solution 3 – Introduce Couchbase

Live Pricing Service

Legacy read-only cache

Legacy quote Processors

The Documents

struct TransportQuoteSet {    1: string MarketingAgentId,    2: common.PassengerGroup PassengerGroup,    3: i32 OutboundDate,    4: bool OneWay,    5: optional i32 InboundDate,    6: common.Place OriginPlace,    7: common.Place DestinationPlace,    8: common.Combinability Combinability,    9: bool Comprehensive    10: list<TransportQuote> Quotes,    11: list<ChildTransportQuoteSet> ChildTransportQuoteSets,    12: optional i32 UtcQuoteDateTime    13: optional bool Additional     14: optional string CabinClass}

The actual cluster

The buckets

The quotes bucket

Some specifics on the setup

Dedicated 6 servers:

SSDs

1 replica of data

Taking nodes in and out of the cluster

Performance with multiple buckets

Monitoring and reporting in one place

2 x Quad core Intel i7 2.0ghz

256GB of 1600mhz RAM

Dell R620 1U.

Centos 6.2

3 x 600gb MLC SSD each (Raid1 + Hot spare)

Going beyond a cache.

Running a car hire search

Prices + Reference Data

Data Collection Services

1) Which websites should we show?

2) What prices do we already have

cached?

3) Live update what we still

need. 4) Save and enrich the new data

5) Return the prices to the user.

Car Hire Live Pricing Service

Reference Data

Supplier-Location

Map reduce views

python Couchbase 2.0 client lib

Car Codes {  "website": {    "published": true,    "id": "affd", ...  },  "office_id": "1",  "city_id": "AUHA",  "raw_data": [...]  "address":   "closing_time": "00:00",  "routenodeid": "9618",  "type": "office“}

Price Data

Python Smart Client Isn’t Ready, so…

Cluster Info

3 nodes

Virtual

Benched to 1000+ ops/sec

SASL Auth.

What’s next?

A flexible search

The results.

What has Couchbase brought us?

quartermile one15 lauriston placeedinburgh eh3 9en

tel: +44 (0)131 252 5353fax: +44 (0)131 252 5354

twitter: @skyscannerfacebook: skyscanner

Thank you for listening.

Alistair Hannalistair.hann@skyscanner.net