hippo gettogether: the architecture behind hippos relevance platform

Building a relevance platform with Couchbase and

Elasticsearch

Hippo GetTogether, 21 June 2013Jeroen Reijn | @jreijn | #hgt2013

Hippo GetTogether 2013

follow the Hippo trail

About me

• Architect @ Hippo

• DevOps guy

• Blogger @ http://blog.jeroenreijn.com

OneHippo @ Goto

Relevance?

OneHippo @ Goto

“The capability of a search engine or function to

retrieve data appropriate to a user's needs.”

http://www.thefreedictionary.com/relevance

OneHippo @ Goto

How we deliver relevant content

@Hippo

Registration

Visitor - entity making HTTP requests

Collector - records data about a visitor or his behavior

Example: location collector (GeoIPCollector)

Targeting Data - all data about a specific visitor

Example: IP address is located in Amsterdam

MatchingCharacteristic - a type of fact about visitors

Example: "comes from a city", "experiences a type of weather"

Target Group - the specification of a Characteristic

Example: "comes from a European city", "comes from Amsterdam"

Persona - one or more target groups that describe a certain type of visitor

Example: "Jim, the European urban consumer",

"Alice, the Pet owner"

What do we store?Request log

Targeting data

Statistics

Averages, e.g. how many visitors became which persona

OneHippo @ Goto

BIG DATA !!

Real-time analysis

OneHippo @ GotoArchitecture

Hippo Delivery Tier

Hippo Repository

App server

XMLJSON (X)HTML

Delivery Tier

URL Matching

Fetch content

Compose output

Request

Response

Request

Delivery Tier

URL Matching

Targeting Data Collection

Compose output

Request

Response

Request

Fetch content

Scoring

OneHippo @ GotoScaling

Hippo Delivery Tier

Hippo Repository

App server

Hippo Delivery Tier

Hippo Repository

App server

Scaling out

Delivery Tier

Repository

App server

Delivery Tier

Repository

App server

Scaling out

TargetingDatastore

OneHippo @ GotoWhat kind of ‘storage’?

OneHippo @ GotoQuestion?

Distributed Cache?

We have a winner!

OneHippo @ Goto

Requirements change!

OneHippo @ GotoNoSQL to the rescue

Suitable types• Key-value store

• Document database

Assessment Criteria

Maturity Data model

Consistency model

PerformanceReplication

Caching model Query model

Monitoring

Scalability

Reliability

Support

Selection Criteria• Performance

• Scalability

• Schema flexibility

• Simplicity

• Monitoring

• Support

OneHippo @ Goto

Performance !!

Performance !!!!

OneHippo @ GotoScalability

OneHippo @ GotoSchema flexibility

{ "visitorId": "7a1c7e75-8539-40", "pageUrl": "http://localhost:8080/site/news", "pathInfo": "/news", "remoteAddr": "127.0.0.1", "referer": "http://localhost:8080/site/", "timestamp": 1371419505909, "collectorData": { "geo": { "country": "", "city": "", "latitude": 0, "longitude": 0 }, "returningvisitor": false, "channel": "English Website" }, "personaIdScores": [], "globalPersonaIdScores": []}

Request log document

{ "geo": { "collectorId": "geo", "city": "", "country": "", "latitude": 0, "longitude": 0 }, "channel": { "collectorId": "channel", "channels": [ "English Website" ], "lastVisitedChannel": "English Website" }}

Visitor document

OneHippo @ GotoSimplicity

OneHippo @ GotoMonitoring

OneHippo @ GotoSupport

OneHippo @ GotoCouchbase

Why Couchbase?

• Drop-in replacement for memcached

• Read/Write-through cache

• High throughput

• Easy scalability

• Schema flexibility

• Low latency

Couchbase

• Open Source

• Document-oriented

• Easy Scalable

• Consistent High Performance

• Apache license

Performance

• Object managed cache

• Write Queue to disk

• Avoids Cold Cache

Easy scalable

• Auto sharding

• Cross cluster replication (XDCR)

• Master - Master replication

Flexible data model

• Native JSON support

• Incremental Map Reduce

• Gives power to the developer

OneHippo @ Goto

How we run Couchbase @Hippo

Load Balancer

Database cluster

Hippo Delivery Tier Couchbase cluster

•Request log data•Targeting data•Statistics data

Query capabilities• Querying via views

• Secondary indexes via views

• Views based on Map - Reduce

• Lacks some advanced query capabilities

Elasticsearch

• Apache Lucene

• Designed to be distributed

• Schema free

• Apache license

• RESTful API

Added value of ES• Full text search

• Faceted search

• Geo spatial search

• All in (near) real-time

Couchbase Server Cluster Elasticsearch Server Cluster

Hippo Delivery Tier

Java API

XDCR Couchbase ES Transport plugin

Replicating to ES

OneHippo @ GotoWhat’s Next?

Advanced analytics

OneHippo @ GotoDemo time!

OneHippo @ Goto

Thank you!

Questions?

j.reijn@onehippo.com | @jreijn

ps. We’re hiring!

hippo gettogether: the architecture behind hippos relevance platform

hippo trailhippo gettogether

hgt2013hippo gettogether

hippo devops guy blogger

gotobig data

english website

visitor document

european city

cold cache

Technology

bigger hippos & happier humans : pete trainor

hermetic hippo

hippos and river ecosystems

an apple- apples a tomato- tomatoes radio-radios...

hippo therapy

review of the crevalle jacks, caranx hippos …...caranx...

hippos reconstruction lcg · title: hippos reconstruction...

angus mac donald amazing hippos

hippos - sussita of the decapolis

hippo cms for the enterprise - bloomreach...

elephants can dance and hippos can limbo

hippo pages

shpslmp^-lbi^patrh spirituality when good gettogether

hippos innovation research

hip hop hippos - origo education › wp-content › uploads...

up to a challenge? help us roll 1000 hippos ·...

is your club up to a challenge? help us roll 1000 hippos ·...

hippo hamburgers!!!

hippos (hippopotamus amphibius): the animal silicon...

hippo current