for the social graph - · center for research in intelligent storage 1. complex distributed write...

Center for Research in Intelligent Storage Authors: Nathan Bronson, et. al Presentors: Zhichao Cao, Fenggang Wu 10/01/2018 TAO: Facebook’s Distributed Data Store for the Social Graph

Upload: others

Post on 21-May-2020

4 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Center for Research in Intelligent Storage

Authors: Nathan Bronson, et. al

Presentors: Zhichao Cao, Fenggang Wu

10/01/2018

TAO: Facebook’s Distributed Data Store

for the Social Graph

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 3: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 4: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 5: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

Stage 1: Relational Database Supported

MySQL

WebServers

……

1. Low write latency

2. High read latency, difficult

to achieve load balancing

3. Low scalability

Center for Research in Intelligent Storage

Stage 2: Cache Supported (memcache)

Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

Stage 2: Cache Supported (cache in between)

MySQL

WebServers

……

1. Higher write latency (clean

the cache)

2. Lower read latency

3. Higher scalability

Cache Layer

Center for Research in Intelligent Storage

Stage 3: TAO (The Associations and Objects)

Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

TAO Design 1: One Tier with Sharding

Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

TAO Design 2: Multiple Caching Tiers

Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

1. Complex distributed write

logic to handle to ensure

data correctness,

consistency

2. A lot of concurrent read

or write go to the

MySQL server, for the

same data

3. Make tradeoffs between

limiting the maximum

tier size and scaling the

cache

Source: https://www.usenix.org/node/174510

Page 13: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 14: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 15: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 16: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

Master regionSlave region 1

Slave region 2

Master DBSlave DB

Slave DB

How to Scale out

Page 18: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 19: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 20: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 21: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

MySQL Architecture

Source: https://techsoftcompute.wordpress.com/2013/08/19/mysql-internal-architecture/

Page 23: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: http://data-base-recovery.blogspot.com/2015/10/storage-engines-in-mysql.html

Page 24: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.slideshare.net/morgo/inno-db-presentation

Center for Research in Intelligent Storage

RocksDB

MySQL Architecture in FaceBook

Source: https://techsoftcompute.wordpress.com/2013/08/19/mysql-internal-architecture/

Page 27: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.slideshare.net/meeeejin/rocksdb-detail

Page 28: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

Thanks

Q/A

Correctness Correctness. Quality Perceptions The perception of quality associated with your code is typically bound to: Correctness Efficiency (speed

Waters Quality System Documentation...AutoTest performs automated testing to evaluate such characteristics as correctness, completeness, consistency, traceability, testability and

OOSE UNIT 2. REQUIREMENT ELICITATION CONCEPTS FUNCTIONAL REQUIREMENTS NONFUNCTIONAL REQUIREMENTS COMPLETENESS,CONSISTENCY,CLARITY AND CORRECTNESS REALISM,VERIFIABILITY

Replication of Data CIS 505: Software Systems Lecture Note ...lee/07cis505/Lec/lec... · o The notion of correctness: consistency models Data-centric consistency Client-centric consistency

Preserving correctness during business process model ...Preserving correctness during business process model conﬁguration 461 2. Background Petri nets are a formal model of concurrent

Supporting Data Consistency in Concurrent Process Execution with Assurance Points and Invariants* Andrew Courter Masters Thesis Defense Department of Computer

Introduction to weak memory consistencyviktor/wmc/intro.pdf · 2017-09-01 · Weak memory consistency isaboutthe semantics of concurrent programs takingintoaccounttheeﬀectsof: I

Inferring Models of Concurrent Systems from Logs of their ...bestchai/papers/csight_ubc_tr.pdfthe correctness of their implementations. CSight’s only requirement is that the logged

Correctness of Speculative Optimizations with Dynamic ...aviral.io/publications/correctness-of-speculative-optimizations-with... · 49 Correctness of Speculative Optimizations with

Ordered Sequential Consistency: Composition and Correctness...2. K r Lev-Ari, Gregory Chockler, and Idit Keidar. A constructive approach for prov-ing data structures’ linearizability

Integrity Constraint Management CS2312. The correctness and consistency of the data and its information Implicit of the data model specified and

Knowledge-Based Engineering · Technical Specification Statement of Work Logistics Support ISO 29148 Completeness Consistency Correctness Completeness Consistency ... • SCM VIEWS

Political Correctness

Verifying Concurrent Programs against Sequential Speci cationslara.epfl.ch/~hamza/esop2013.pdf · 1 Introduction A key class of correctness criteria for concurrent systems is adherence

Supporting Data Consistency in Concurrent Process Execution with Assurance Points and Invariants*

Linearizability: A Correctness Condition for …christoff/files/Linearizability...Linearizability: A Correctness Condition for Concurrent Objects MAURICE P. HERLIHY and JEANNETTE M

On Library Correctness under Weak Memory Consistencyplv.mpi-sws.org/yacovet/paper.pdf · 68 On Library Correctness under Weak Memory Consistency Specifying and Verifying Concurrent

Linearizability: A Correctness Condition for Concurrent ...cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf · message-passing architectures in which the shared ... for J. M. Wing was

Eran Yahav 1. Previously… An algorithmic view Abstract data types (ADT) Correctness Conditions Sequential consistency Linearizability Treiber’s

On Library Correctness under Weak Memory Consistencyorilahav/papers/popl19libraries.pdf · On Library Correctness under Weak Memory Consistency Specifying and Verifying Concurrent

Algorithms for Synchronization and Consistency in Concurrent System Services Anders Gidenstam Distributed Computing and Systems group, Department of Computer

Case Study...Challenge #3 - Consistency with Other Assessments Maintaining consistency with concurrent risk assessments, MRSPP updates, or previous MEC HAs • The frameworks have

A Library of Concurrent Objects and Their Proofs of ...wing/publications/CMU-CS-90-151.pdf · A Library of Concurrent Objects and Their Proofs of Correctness Chun Gong and Jeannette

On Library Correctness under Weak Memory Consistencyorilahav/papers/popl19libraries.pdf · and verify concurrent libraries in a general fashion, agnostic to the underlying memory

Hoare-Style Speciﬁcations as Correctness Conditions for ... · Hoare-Style Speciﬁcations as Correctness Conditions for Non-linearizable Concurrent Objects Ilya Sergeyy Aleksandar

CS492B Analysis of Concurrent Programs Consistency Jaehyuk Huh Computer Science, KAIST Part of slides are based on CS:App from CMU

Correctness-Preserving Derivation of Concurrent Garbage Collection Algorithms

Chap 6 Synchronization. Background Concurrent access to shared data may result in data inconsistency Maintaining data consistency requires mechanisms

Global-Local View: Scalable Consistency for Concurrent ... · concurrent data structures, its implementation can be prohibitively expensive. As the number of cores increases in a

On Library Correctness under Weak Memory Consistency · 2018-12-15 · 68 On Library Correctness under Weak Memory Consistency Specifying and Verifying Concurrent Libraries under

Linearizability: A Correctness Condition for Concurrent Objects

Plan Synchronizing without Locks and Concurrent Data ...mmorenom/HPC-Slides/... · Synchronization of Concurrent Programs Memory consistency model (2/4) This is acontract between

Behavioural Types: from Theory to Tools - Enlighteneprints.gla.ac.uk/146884/2/146884.pdfproperty is correctness of communication in concurrent ... 2 Contract-Oriented Programming with

Think - State Institute for Urban Development · Communication Skills –7C’s Credibility Courtesy Clarity Correctness Consistency Concreteness Conciseness. Effective Communication

Correctness-Preserving Derivation of Concurrent Garbage Collection Algorithms Martin T. Vechev Eran Yahav David F. Bacon University of Cambridge IBM T.J