PowerPoint Presentation

Jason Baker, Chris Bond, James C. Corbett, JJ Furman, Andrey Khorlin, James Larson,JeanMichel Leon, Yawei Li, Alexander Lloyd, Vadim YushprakhMegastoreScalable Highly Available Storage for Interactive Systems

Presented By:

Hamid SeyedmoradiAyoub HamidiEhsan Mohamad Nezamian

Advanced Database Systems SRBIAU, Kurdistan Campus10May2012

MegastoreMotivationIntroductionNoSQL & RDBMSMegastorePaxos

SRBIAU, Kurdistan Campus Advanced Database Systems2

Megastorewow! more than 3 billion write and 20 billion read daily

key ContributionData Model and Storage SystemPaxos ReplicationReport on Experience

SRBIAU, Kurdistan Campus Advanced Database Systems3

AVAILABILITY & SCALEReplication For Availability, we implemented a synchronous, fault tolerant log replicator optimized for long distance links Partitioning and Locality for scale, we partitioned data into a vast space of small databases

SRBIAU, Kurdistan Campus Advanced Database Systems4

AVAILABILITY & SCALEReplicationStrategiesAsynchronous Master/SlaveSynchronous Master/SlaveOptimistic ReplicationWe decided to use Paxos

SRBIAU, Kurdistan Campus Advanced Database Systems5

Technology OptionsSRBIAU, Kurdistan Campus Advanced Database Systems

6

SRBIAU, Kurdistan Campus Advanced Database Systems

7

Technology OptionsAVAILABILITY & SCALEPartitioning and LocalityReplication

SRBIAU, Kurdistan Campus Advanced Database Systems8

DatacentersEntity Groups Partition the datastoreEach entity group is synchronously replicated across datacentersACID semantics within an entity group Looser consistency across entity groups Entity group data and replication metadata stored in scalable NoSQL datastores8AVAILABILITY & SCALEPartitioning and LocalityOperations:

SRBIAU, Kurdistan Campus Advanced Database Systems9

Entities (Units of data)Entity Group 1Most transactions are within a single entity group Cross Entity group transactions supported via Two Phase CommitAsynch communication between entity groups supported by QueuesGlobal Indexes span entity groups but have weaker consistencyEntity Group 2Local IndexLocal IndexSendqueuereceive9AVAILABILITY & SCALEPartitioning and LocalityEntity GroupsSelecting Entity Group BoundariesExampleEmailBlogsPhysical Layout

SRBIAU, Kurdistan Campus Advanced Database Systems10

MegastoreAPI Design PhilosophyData ModelPre-Joining with KeysSCATTERIndexesStoring ClauseRepeated Indexes.Inline IndexesMapping to Bigtable

SRBIAU, Kurdistan Campus Advanced Database Systems11

Megastore

SRBIAU, Kurdistan Campus Advanced Database Systems12

MegastoreTransactions and Concurrency ControlReadcurrent snapshot inconsistentTransaction Lifecycle1-Read 3-Commit 5-Clean up2-Application logic 4-ApplySRBIAU, Kurdistan Campus Advanced Database Systems13

MegastoreQueues

Two Phase Commit

SRBIAU, Kurdistan Campus Advanced Database Systems14

REPLICATIONBrief Summary of PaxosMegastores ApproachFast ReadsFast WritesReplica Types Witness ReplicaArchitectureSRBIAU, Kurdistan Campus Advanced Database Systems15

ArchitectureSRBIAU, Kurdistan Campus Advanced Database Systems16

Data Structures and AlgorithmsReplicated LogsSRBIAU, Kurdistan Campus Advanced Database Systems17

Data Structures and AlgorithmsReadsQuery LocalFind PositionLocal readMajority readCatchupValidateQuery DataSRBIAU, Kurdistan Campus Advanced Database Systems18

Data Structures and AlgorithmsSRBIAU, Kurdistan Campus Advanced Database Systems19

Data Structures and AlgorithmsWritesAccept LeaderPrepareAcceptInvalidateApplySRBIAU, Kurdistan Campus Advanced Database Systems20

FeedbackSRBIAU, Kurdistan Campus Advanced Database Systems21

ENDWith ThanksQuestion?SRBIAU, Kurdistan Campus Advanced Database Systems22