with cloud bigtable personalization engine driving a ... · dbs302: driving a realtime...
TRANSCRIPT
![Page 1: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/1.jpg)
DBS302:Driving a Realtime Personalization Enginewith Cloud Bigtable
Calvin French-Owen, Co-Founder & CTO, Segment
![Page 2: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/2.jpg)
You’re making a hard choice...
![Page 3: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/3.jpg)
Our roadmap
- A bit of background- Personas architecture- BigQuery + Cloud Bigtable- Making hard choices
![Page 4: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/4.jpg)
A bit of background
![Page 5: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/5.jpg)
![Page 6: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/6.jpg)
![Page 7: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/7.jpg)
- 19,000 users- 300B monthly events- 450B outbound API calls- TB of data per day
Segment by the numbers
![Page 8: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/8.jpg)
Under the hood...
![Page 9: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/9.jpg)
API Kafka Consumer
DB
api.google.com
api.salesforce.com
api.intercom.io
api.mixpanel.com
![Page 10: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/10.jpg)
The biggest advantage of this system
![Page 11: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/11.jpg)
The biggest advantage of this system
It’s stateless
![Page 12: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/12.jpg)
API Kafka Consumer
DB
api.google.com
api.salesforce.com
api.intercom.io
api.mixpanel.comAPI Kafka Consumer
![Page 13: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/13.jpg)
In 2018... we started getting a new set of requirements
![Page 14: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/14.jpg)
![Page 15: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/15.jpg)
Personas brought some decidedly stateful use cases
![Page 16: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/16.jpg)
The use cases of personas
![Page 17: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/17.jpg)
1) Profile API
![Page 18: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/18.jpg)
![Page 19: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/19.jpg)
![Page 20: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/20.jpg)
2) Identity resolution
![Page 21: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/21.jpg)
![Page 22: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/22.jpg)
3) Audience computation
![Page 23: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/23.jpg)
![Page 24: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/24.jpg)
- Query profiles in real-time- Match users by identity- Create audiences of users
Personas
![Page 25: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/25.jpg)
Personas architecture
![Page 26: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/26.jpg)
Let’s first talk about lambda architectures...
![Page 27: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/27.jpg)
![Page 28: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/28.jpg)
- Data is sent to the batch and speed layers- Batch layers runs bigger computations- Speed layer serves real-time updates (+ diffs)
Lambda architecture
![Page 29: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/29.jpg)
- Query profiles in real-time (speed)- Match users by identity (speed)- Create audiences of users (batch)
Personas
![Page 30: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/30.jpg)
Different pipelines, different datastores
![Page 31: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/31.jpg)
Kafka Pubsub
BigQuery
Cloud Bigtable
(batch)
(speed)
Worker
Worker
![Page 32: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/32.jpg)
Kafka Pubsub
BigQuery
Cloud Bigtable
(batch)
(speed)
Worker
Worker
![Page 33: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/33.jpg)
Kafka -> Pub/Sub
![Page 34: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/34.jpg)
Segment messages
- Tracking things like pageviews, user events, etc
- Semi-structured JSON- Typically ~1kb
![Page 35: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/35.jpg)
![Page 36: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/36.jpg)
- Hundreds of thousands of 1kb messages- Published from Kafka to Cloud PubSub- Writes data twice, once for realtime, once for batch- Audience computation in BigQuery- Real-time reads in Cloud Bigtable
Personas architecture
![Page 37: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/37.jpg)
BigQuery + Cloud Bigtable
![Page 38: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/38.jpg)
- Use case- Architecture- Data model- Query patterns
BigQuery + Cloud Bigtable
![Page 39: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/39.jpg)
BigQuery: Use case
![Page 40: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/40.jpg)
BigQuery
Cloud Bigtable
computeservice
Kafka Pubsub
Worker
Worker
![Page 41: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/41.jpg)
- Want to find users who meet arbitrary criteria- Terabytes of data within a few minutes- Tables have billions of rows- We rarely care about all of the columns - Real-time reads are not a big deal- Tens of concurrent queries
BigQuery: Use case
BigQuery
Cloud Bigtable
computeservice
PubSub
Worker
Worker
![Page 42: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/42.jpg)
BigQuery: Architecture
![Page 43: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/43.jpg)
2004: MapReduce
![Page 44: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/44.jpg)
2010: Dremel (built in 2006)
![Page 45: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/45.jpg)
BigQuery: architecture
- Designed to interactively query datasets (seconds-minutes)- Nested, structured data- Uses SQL, no programming- Private version: Dremel
![Page 46: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/46.jpg)
BigQuery Architecture: four good ideas
![Page 47: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/47.jpg)
BigQuery idea #1:Column-oriented
![Page 48: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/48.jpg)
Suppose we want to build a database...
![Page 49: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/49.jpg)
A row-oriented database
![Page 50: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/50.jpg)
What if my database has billions of rows...
...and I only need location?
![Page 51: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/51.jpg)
What if my database has billions of rows...
...and I only need location?
Store columns, not rows!
![Page 52: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/52.jpg)
What if we invert the rows?
![Page 53: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/53.jpg)
![Page 54: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/54.jpg)
![Page 55: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/55.jpg)
BigQuery idea #2: Compression
![Page 56: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/56.jpg)
BigQuery idea #2: Compression
![Page 57: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/57.jpg)
Columns on disk
- We have a lot of repeated data- Run-length-encoding (RLE)- Let’s compress it...
![Page 58: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/58.jpg)
Columns on disk
- We have a lot of repeated data- Run-length-encoding (RLE)- Let’s compress it...
![Page 59: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/59.jpg)
BigQuery idea #3: Efficient nested decoding
![Page 60: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/60.jpg)
BigQuery idea #3: Efficient nested decoding
![Page 61: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/61.jpg)
![Page 62: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/62.jpg)
![Page 63: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/63.jpg)
What happens when I select *?
![Page 64: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/64.jpg)
FSM
![Page 65: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/65.jpg)
BigQuery idea #4: More servers, more efficiency
![Page 66: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/66.jpg)
BigQuery idea #4: More servers, more efficiency
![Page 67: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/67.jpg)
![Page 68: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/68.jpg)
Root
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
query
![Page 69: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/69.jpg)
Root
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
MERGE!
query
![Page 70: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/70.jpg)
Root
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
Level 1
Level 1
Level 1
query
![Page 71: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/71.jpg)
Root
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
leaflet
Level 1
Level 1
Level 1
query
MERGE!
MERGE!
MERGE!
MERGE!
![Page 72: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/72.jpg)
More servers == more distributed work
![Page 73: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/73.jpg)
BigQuery’s good ideas
1. Column-oriented2. Compression3. Fast, nested, data encoding4. Distribute the work (separate data + compute)
![Page 74: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/74.jpg)
BigQuery: Data model
![Page 75: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/75.jpg)
![Page 76: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/76.jpg)
We want to take user-supplied criteria…
…and turn it into query parameters
![Page 77: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/77.jpg)
UI JSON
![Page 78: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/78.jpg)
SQLJSON
![Page 79: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/79.jpg)
BigQuery: Data Model
- Dataset per customer- Table per {collection,event}- Additional tables for traits,
identity, merges
![Page 80: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/80.jpg)
BigQuery: Data Model
- Dataset per customer- Table per {collection,event}- Additional tables for traits,
identity, merges- Repeated fields for
external_ids
![Page 81: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/81.jpg)
BigQuery: Data Model
- Dataset per customer- Table per {collection,event}- Additional tables for traits,
identity, merges- Repeated fields for
external_ids- Explode arbitrary nested
properties
![Page 82: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/82.jpg)
BigQuery:Query patterns
![Page 83: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/83.jpg)
BigQuery
Cloud Bigtable
computeservice
Kafka Pubsub
Worker
Worker
![Page 84: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/84.jpg)
Compute service runs queries every minute
![Page 85: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/85.jpg)
Scan gigabytes in seconds
![Page 86: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/86.jpg)
![Page 87: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/87.jpg)
2GB/s scanned(170T/day)
800 slots
![Page 88: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/88.jpg)
- Tens of concurrent queries- Scans terabytes of data independently- Partitioned by customer- Query by arrays of external_ids- Stored AST as JSON and converted to SQL
Batch computations in BigQuery
![Page 89: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/89.jpg)
Cloud Bigtable: Use case
![Page 90: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/90.jpg)
![Page 91: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/91.jpg)
BigQuery
Cloud Bigtable
profileAPI
Kafka Pubsub
Worker
Worker
![Page 92: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/92.jpg)
Cloud Bigtable: Use case
- Small amounts of data (kb to mb)- Able to be indexed for a single user- A high read and write rate (tens of thousands of qps)- Data should be reflected in real-time
![Page 93: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/93.jpg)
Cloud Bigtable: Use case
- Small amounts of data (kb to mb)- Able to be indexed for a single user- A high read and write rate (tens of thousands of qps)- Data should be reflected in real-time
(Not a new idea)
![Page 94: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/94.jpg)
Cloud Bigtable: Architecture
![Page 95: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/95.jpg)
Bigtable (published in 2006)
![Page 96: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/96.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
![Page 97: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/97.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
write: <k, v>
![Page 98: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/98.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
write: <k, v>
memtable.append(k, v)
![Page 99: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/99.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
write: <k, v>
memtable.append(k, v)
append(k, v)
![Page 100: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/100.jpg)
Writes are fast appends
![Page 101: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/101.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)
![Page 102: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/102.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)
memtable[k]
![Page 103: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/103.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
<value>
memtable[k]
![Page 104: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/104.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k) fetch(offset)
![Page 105: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/105.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
<value><data>
![Page 106: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/106.jpg)
Reads first cache,then merge
![Page 107: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/107.jpg)
What about failures?
![Page 108: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/108.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k) fetch(offset)
![Page 109: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/109.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k) fetch(offset)
![Page 110: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/110.jpg)
Cloud Bigtable: Architecture
Client
BT Node
GFS Tablet
GFS Tablet
GFS Tablet
memtable
BT Node
memtable
GFS Tablet
read(k)
fetch(
offset
)
![Page 111: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/111.jpg)
Cloud Bigtable: Architecture
- Multi-tenant- Row-oriented- Log-structured merge tree- Immutable, with in-memory caching- Bloom filters save on reads- Lock service maps nodes to keyspace
![Page 112: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/112.jpg)
Cloud Bigtable: Data model
![Page 113: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/113.jpg)
- Separate tables for different datatypes- Records- Properties- Events
- Keys are ID and time-ordered- Values are snappy-encoded
Cloud Bigtable: Data Model
![Page 114: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/114.jpg)
Cloud Bigtable: Data Model
- Records provide metadata to stitch together the full record
- User properties power the profile API
- Events are sorted to query the last range of events
![Page 115: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/115.jpg)
Cloud Bigtable + BigQuery:In production
![Page 116: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/116.jpg)
In production
- Cloud Bigtable- 55,000 rows written per second- 175,000 rows read per second- 10 TB of data- 16 nodes
- BigQuery- Hundreds of queries per minute- Scanning hundreds of GB/minute- 500TB worth of data stored
![Page 117: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/117.jpg)
Back to that hard choice...
![Page 118: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/118.jpg)
BigQuery is hard to compare
![Page 119: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/119.jpg)
A few placesCloud Bigtable shines
![Page 120: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/120.jpg)
1. Identification of hot keys
![Page 121: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/121.jpg)
![Page 122: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/122.jpg)
2. Write-heavy workloads
![Page 123: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/123.jpg)
Split compute
- Compute is separated from storage
- Writes can be spread across many nodes
![Page 124: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/124.jpg)
In summary...
![Page 125: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/125.jpg)
Segment Personas
- Powered by Cloud Bigtable and BigQuery- Cloud Bigtable for small, random reads- BigQuery for batch aggregations
- Processes billions of events- Large, multi-tenant architecture- SQL for flexible feature development- Favorable read/write costs- Millions of dollars in revenue- Scales to Google-levels
![Page 126: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/126.jpg)
Fin
![Page 127: with Cloud Bigtable Personalization Engine Driving a ... · DBS302: Driving a Realtime Personalization Engine with Cloud Bigtable Calvin French-Owen, Co-Founder & CTO, Segment](https://reader033.vdocument.in/reader033/viewer/2022041415/5e1ac6489808ed222b5f3f67/html5/thumbnails/127.jpg)
Your Feedback is Greatly Appreciated!
Complete the session survey in mobile app
1-5 star rating system
Open field for comments
Rate icon in status bar