cloud spanner - goto conference · what is cloud spanner? traditional relational semantics:...
TRANSCRIPT
![Page 1: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/1.jpg)
Cloud Spanner
Joe IntrakamhangSolutions Engineer@joe_int
![Page 2: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/2.jpg)
$29.4 Billion3 Year Trailing CAPEX Investment
2
![Page 3: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/3.jpg)
FASTER (US, JP, TW) 2016
Unity (US, JP) 2010SJC (JP, HK, SG) 2013
Edge points of presence >100
Monet (US, BR) 2017
Network
Network sea cable investments
PLCN Unity (HK, LA) 2018
Indigo (SG, ID, AU) 2019Tannat (BR, UY, AR) 2017
Junior (Rio, Santos) 2017
Google global cache edge nodes (>800)
Google Network
![Page 4: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/4.jpg)
BackgroundWhy build Spanner?
![Page 5: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/5.jpg)
Technical infrastructure at Google
It’s 2005...
![Page 6: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/6.jpg)
Google’s needs
Horizontally Scaling Database
ACID Transactions with global consistency
No downtime!
![Page 7: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/7.jpg)
Existing options
Nonrelational Relational
![Page 8: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/8.jpg)
OverviewWhat is Cloud Spanner?
![Page 9: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/9.jpg)
What is Cloud Spanner?
Traditional relational semantics: schemas, ACID transactions, SQL
Fully managed, database service with global scale
Automatic, synchronous replication within and across regions for availability
Battle-tested within Google for 5+ yrs (AdWords, GooglePlay)
Google’s mission-critical scalable relational Database Service
![Page 10: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/10.jpg)
How does it compare?
![Page 11: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/11.jpg)
What alternatives are there?
C
A
P
PickTwo
APCassandraRiakCouchDB
CPBigTable Berkeley DBMongoDB MemcacheDBHbase Redis
Always able to read and write
Always see the same data as others at same point in time
works even in the case of network partitions
CA?
![Page 12: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/12.jpg)
Spanner tries to be both, SQL and distributed, how?
Spanner = NoSQL horizontal scalability + SQL relational semantics
+ consistent transactions
![Page 13: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/13.jpg)
JDBC driver
Standard SQL (ANSI 2011)
Client libraries in popular languages (Java, C#, Python, Go, Node.js, etc.)
Encryption, Audit logging, Identity and Access Management
Open standards
![Page 14: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/14.jpg)
How Spanner works
![Page 15: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/15.jpg)
Traditional Database layout
Zone 1
Table 1
Table 2
![Page 16: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/16.jpg)
Traditional Database layout -- sharded
Zone 1 Zone 2 Zone 3
Table 1a
Table 2a
Table 1c
Table 2c
Table 1b
Table 2b
![Page 17: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/17.jpg)
Traditional Database layout
Zone 1 Zone 2 Zone 3
Table 1a
Table 2a
Table 1c
Table 2c
Table 1b
Table 2b
My awesome app
![Page 18: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/18.jpg)
Traditional Database layout
Zone 1 Zone 2 Zone 3
Table 1a
Table 2a
Table 1c
Table 2c
Table 1b
Table 2b
My awesome app
![Page 19: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/19.jpg)
Traditional Database layout
Zone 1 Zone 2 Zone 3
Table 1a
Table 2a
Table 1c
Table 2c
Table 1b
Table 2b
My awesome app
![Page 20: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/20.jpg)
Spanner Database layout
Zone 1 Zone 2 Zone 3
Table 1
Table 2
Table 1
Table 2
Table 1
Table 2
![Page 21: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/21.jpg)
Transactions
Zone 1 Zone 2 Zone 3
Table 1
Table 2
Table 1
Table 2
Table 1
Table 2
My awesome app
![Page 22: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/22.jpg)
Transactions
Zone 1 Zone 2 Zone 3
Table 1
Table 2
Table 1
Table 2
Table 1
Table 2
My awesome app
![Page 23: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/23.jpg)
Transactions
Zone 1 Zone 2 Zone 3
Table 1
Table 2
Table 1
Table 2
Table 1
Table 2
![Page 24: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/24.jpg)
TrueTime Infrastructure
GPS master GPS master GPS master
Atomic master Atomic master Atomic master
Compute Node Compute Node Compute Node
sync every 30 sec
![Page 25: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/25.jpg)
TrueTime - virtual global clock
● Quantifies the “worst” possible error / drift between clocks in all datacenters around the world (global clock)
● TrueTime.Now() gives you an interval [t1,t2]; t2 = t1 + 2ɛ
![Page 26: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/26.jpg)
Regional Instance
Zone 3Zone 2
Spanner Internals
Zone 1
Sto
rage
Com
pute
DB 1
DB n
DB 1
DB n
DB 1
DB n
![Page 27: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/27.jpg)
Tablet 1
Spanner Internals
Split SP
Instance DB G
DB OO
DB G
DB L
DB E
Table C
Table L
Table O
Index U
Index D
Split A
Split NN
Split R
Split E
![Page 28: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/28.jpg)
* TrueTime used for leader leases → making sure there is only one leader for a tablet at any given time
PaxosGroup
forTablet 1
Zone 3Zone 2
Spanner Internals
Zone 1
Tablet 1
Tablet 2
Sto
rage
Com
pute
Tablet 3Tablet 1
Tablet 2
Tablet 3 Tablet 1
Tablet 2
Tablet 3
![Page 29: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/29.jpg)
Life of a QueryConsistent Read
Zone 1
Tablet 1
Tablet 2
Tablet 3
Zone 2
Tablet 1
Tablet 2
Tablet 3
Zone 3
Tablet 1
Tablet 2
Tablet 3
Follower Leader Follower1) Request 2) Ok to read?
3) yep!4) Response
SELECT * FROM Company WHERE Name = ‘Google’;
![Page 30: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/30.jpg)
Life of a QueryConsistent Read
Zone 1
Tablet 1
Tablet 2
Tablet 3
Zone 2
Tablet 1
Tablet 2
Tablet 3
Zone 3
Tablet 1
Tablet 2
Tablet 3
Follower Leader Follower1) Request 2) Ok to read?
3) nope! You’ll have the data soon, wait!
5) Response
SELECT * FROM Company WHERE Name = ‘Google’;
4) Wait
![Page 31: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/31.jpg)
Life of a QueryTime-bounded (stale) reads
Zone 1
Tablet 1
Tablet 2
Tablet 3
Zone 2
Tablet 1
Tablet 2
Tablet 3
Zone 3
Tablet 1
Tablet 2
Tablet 3
Follower Leader Follower
SELECT * FROM Company WHERE Name = ‘Google’;
1) Request (max 15s old) 2) am I
up-to-date enough? yes!3) Response
![Page 32: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/32.jpg)
dbClient.ReadWriteTransaction(ctx, func(txn *spanner.ReadWriteTransaction) error { txn.Query(ctx, stmt).Do(func(r *spanner.Row) error {...}) m = append(m, spanner.InsertOrUpdate("TableName", []string{"ColA", "ColB"}, []interface{}{"93bc3d", "Lorem Ipsum"})) txn.BufferWrite(m)})
Life of a QueryRead-Write Transaction
Zone 1
Tablet 1
Tablet 2
Tablet 3
Zone 2
Tablet 1
Tablet 2
Tablet 3
Zone 3
Tablet 1
Tablet 2
Tablet 3
Follower Leader Follower1) txn.Query()
6) ack
5) write xyz
3) read result
5) write xyz
6) ack 2) acq. locks
4) txn.BufferWrite()
8) txn success 7) rel. locks
![Page 33: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/33.jpg)
Relational Data Layout
SingerId SingerName
1 Beatles
2 U2
3 Pink Floyd
SingerId AlbumId AlbumName
1 1 Help!
1 2 Abbey Road
3 1 The Wall
![Page 34: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/34.jpg)
Interleave Data Layout
1 Beatles
1 1 Help!
1 2 Abbey Road
2 U2
3 Pink Floyd
3 1 The Wall
![Page 35: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/35.jpg)
Relational data model
CREATE TABLE Singers ( SingerId INT64 NOT NULL, SingerName STRING(MAX),) PRIMARY KEY(SingerId);
CREATE TABLE Albums ( SingerId INT64 NOT NULL, AlbumId INT64 NOT NULL, AlbumName STRING(MAX),) PRIMARY KEY(SingerId, AlbumId) INTERLEAVE IN Singers;
ALTER TABLE Singers ADD COLUMN Age INT64;
No Downtime Schema Migrations
![Page 36: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/36.jpg)
Schema Migration
![Page 37: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/37.jpg)
Anti-Pattern: Timestamp Ordering
![Page 38: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/38.jpg)
Anti-Pattern: Sequences
![Page 39: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/39.jpg)
Median Latency
![Page 40: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/40.jpg)
Demo
![Page 41: Cloud Spanner - GOTO Conference · What is Cloud Spanner? Traditional relational semantics: schemas, ACID transactions, SQL Fully managed, database service with global scale Automatic,](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd173c01a15d235523544db/html5/thumbnails/41.jpg)
Thank You
@joe_int