Upload File

bigtable - university of washingtonbigtable in the early 2000s, google had way more data than...

  • Home
  • Documents
  • BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem
37
BigTable CS 452

Upload: others

Post on 05-Jan-2020

1 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTableCS 452

Page 2: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable

In the early 2000s, Google had way more data than anybody else did

Traditional databases couldn’t scale

Want something better than a filesystem (GFS)

BigTable optimized for:

- Lots of data, large infrastructure

- Relatively simple queries

Relies on Chubby, GFS

Page 3: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Chubby

Page 4: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Chubby

Distributed coordination service

Goal: allow client applications to synchronize and manage dynamic configuration state

Intuition: only some parts of an app need consensus!

- Lab 2: Highly available view service

- Master election in a distributed FS (e.g. GFS)

- Metadata for sharded services

Implementation: (Multi-)Paxos SMR

Page 5: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Why Chubby?

Many applications need coordination (locking, metadata, etc).

Every sufficiently complicated distributed system contains an ad-hoc, informally-specified, bug-ridden, slow implementation of Paxos

Paxos is a known good solution

(Multi-)Paxos is hard to implement and use

Page 6: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

How to do consensus as a serviceChubby provides:

- Small files

- Locking

- “Sequencers”

Filesystem-like API

- Open, Close, Poison

- GetContents, SetContents, Delete

- Acquire, TryAcquire, Release

- GetSequencer, SetSequencer, CheckSequencer

Page 7: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Back to BigTable

Page 8: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Uninterpreted strings in rows and columns

(r : string) -> (c : string) -> (t : int64) -> string

Mostly schema-less; column “families” for access

Data sorted by row name

- lexicographically close names likely to be nearby

Each piece of data versioned via timestamps

- Either user- or server-generated

- Control garbage-collection

Page 9: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 10: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 11: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Tablets

Each table composed of one or more tablets

Starts at one, splits once it’s big enough

- Split at row boundaries

Tablets ~100MB-200MB

a datab datac datad data

Page 12: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Tablets

Each table composed of one or more tablets

Starts at one, splits once it’s big enough

- Split at row boundaries

Tablets ~100MB-200MB

a datab datac datad datae data

Page 13: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Tablets

Each table composed of one or more tablets

Starts at one, splits once it’s big enough

- Split at row boundaries

Tablets ~100MB-200MB

a datab data

c datad datae data

Page 14: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Tablets

A tablet is indexed by its range of keys

- <START> - “c”

- “c” - <END>

Each tablet lives on at most one tablet server

Master coordinates assignments of tablets to servers

Page 15: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Tablets

Tablet locations stored in METADATA table

Root tablet stores locations of METADATA tablets

Root tablet location stored in Chubby

Page 16: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Tablet serving

Tablet data persisted to GFS

- GFS writes replicated to 3 nodes

- One of these nodes should be the tablet server!

Three important data structures:

- memtable: in-memory map

- SSTable: immutable, on-disk map

- Commit log: operation log used for recovery

Page 17: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Tablet serving

Writes go to the commit log, then to the memtable

Reads see a merged view of memtable + SSTables

- Data could be in memtable or on disk

- Or, some columns in each

Page 18: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Compaction and compression

Memtables spilled to disk once they grow too big

- “minor compaction”: converted to SSTable

Periodically, all SSTables for a tablet compacted

- “major compaction”: many SSTables -> one

Compression: each block of an SSTable compressed

- Can get enormous ratios with text data

- Locality helps—similar web pages in same block

Page 19: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 20: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 21: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Master

Tracks tablet servers (using Chubby)

Assigns tablets to servers

Handles tablet server failures

Page 22: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Master startup

- Acquire master lock in Chubby

- Find live tablet servers (each tablet server writes its identity to a directory in Chubby)

- Communicate with live servers to find out who has which tablet

- Scan METADATA tablets to find unassigned tablets

Page 23: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Master operation

Detect tablet server failures

- Assign tablets to other servers

Merge tablets (if they fall below a size threshold)

Handle split tablets

- Splits initiated by tablet servers

- Master responsible for assigning new tablet

Clients never read from master

Page 24: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 25: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 26: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Where is the root tablet?

Page 27: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Tablet server 2

Page 28: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 29: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Where is the METADATA tablet for table T row R?

Page 30: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Tablet server 1

Page 31: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 32: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Where is table T row R?

Page 33: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Tablet server 3

Page 34: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Page 35: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Read table T row R

Page 36: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

BigTable components

Client

Master

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

Tablet Server

GFS

Row

Page 37: BigTable - University of WashingtonBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a filesystem

Optimizations

Clients cache tablet locations

Tablet servers only respond if Chubby session active, so this is safe

Locality groups

Put column families that are infrequently accessed together in separate SSTables

Smart caching on tablet servers

Bloom filters on SSTables


What's Next for Google's BigTable

Bigtable - Rutgers University · Bigtable Author: Paul Krzyzanowski Subject: Distributed Systems Created Date: 11/13/2017 10:23:33 PM

Sentry K300 SafeConsole Deployment Guide exFAT, or NTFS. By default, the K300 ships with an NTFS filesystem. If the current filesystem is not natively compatible with Windows, such

Bigtable Uw Presentation

Google Cloud Bigtable Integrating time series database with · PDF fileOpenTSDB + Bigtable Integrating time series database with Google Cloud Bigtable Danil Zburivsky, Big Data Practice

GOOGLE BIGTABLE

A portable implementation of the FAT16 filesystem to TinyOS ...jmfriedt.free.fr/gwen_tos.pdf · A portable implementation of the FAT16 filesystem to TinyOS-2.x: non-volatile mass

Cassandra: Open Source Bigtable + Dynamo

HBase and Bigtable Storage

Google Cloud Bigtable · 2016/02/03  · Google Research Publications. Google Research Publications. Managed Cloud Versions Bigtable Flume Dremel. Managed Cloud Versions Bigtable

Google File System BigTable

Bigtable and Boxwood

BigTable: A Distributed Storage System for Structured Datacourse/DBMS/class/bigtable/bigtable.pdf · • BigTable is a working system catering the needs for a wide variety of applications

Dynamo and BigTable - Review and Comparison

Cross-Site BigTable using HBase

BigTable Search Presentation to Austin PUG

BigTable A System for Distributed Structured Storage

MicroPython Documentation - Read the Docs · 2019-04-02 · MicroPython Documentation, Release 1.9.2 The boot filesystem is used for 2 things: it is the filesystem from which the

Distributed Systems - Computer Sciencepxk/417/notes/content/17-bigtable-slides.pdf · A big table •Bigtable is NOT a relational database •Bigtable appears as a large table –“A

HBaseConEast2016: OpenTSDB+BigTable

Bigtable Raw

Security approaches in BigTable-like storage systems

Storage of Structured Data: BigTable and HBaselsd.ls.fi.upm.es/nuevas-tendencias-en-sistemas-distribuidos/HBase_2.pdf · Storage of Structured Data: BigTable and HBase HBase/BigTable

Key-Value stores (BigTable)ocw.upc.edu/.../2015/1/55027/05-bigtable-5693.pdf · Based on Google’s Bigtable Ta ble(Key Based on Googles Designed to meet the following requirements

Accumulo Extensions to Google's Bigtable Designpeople.apache.org/~afuchs/slides/morgan_state_talk.pdf · Accumulo { Extensions to Google’s Bigtable Design Adam Fuchs National Security

Bigtable: A Distributed Storage System for Structured Data · Bigtable is designed to reliably scale to petabytes of data and thousands of machines. Bigtable has achieved several

Bigtable: A Distributed Storage System for Structured Data · 2016-01-09 · Introduction •Bigtable is a distributed storage system for managing structured data. •Goals of Bigtable

BigTable: A System for Distributed Structured Storagepages.cs.wisc.edu/~remzi/Classes/739/Fall2017/Papers/bigtable-slides-05.pdf · Bigtable master Bigtable tablet server Bigtable

Lecture 8 gooble bigtable

Centralized vs. DistributedDynamo Voldemort TokyoCabinet KAI Cassandra SimpleDB CouchDB Riak BigTable Hypertable HBase MongoDB Terrastore Scalaris BerkeleyDB MemcacheDB Redis ... BigTable

BIGTABLE - Massachusetts Institute of Technologydb.lcs.mit.edu/6.830/lectures/bigtable-slides.pdf · BigTable System Architecture Read/write Cluster Scheduling Master handles failover,

Bigtable a distributed storage system

Distributed Systems 18. BigTable

BigTable - courses.cs.washington.eduBigTable In the early 2000s, Google had way more data than anybody else did Traditional databases couldn’t scale Want something better than a

Lecture 7 – Bigtable