for the social graph - · center for research in intelligent storage 1. complex distributed write...

Center for Research in Intelligent Storage Authors: Nathan Bronson, et. al Presentors: Zhichao Cao, Fenggang Wu 10/01/2018 TAO: Facebook’s Distributed Data Store for the Social Graph

Upload: others

Post on 21-May-2020

4 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Center for Research in Intelligent Storage

Authors: Nathan Bronson, et. al

Presentors: Zhichao Cao, Fenggang Wu

10/01/2018

TAO: Facebook’s Distributed Data Store

for the Social Graph

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 3: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 4: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 5: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

Stage 1: Relational Database Supported

MySQL

WebServers

……

1. Low write latency

2. High read latency, difficult

to achieve load balancing

3. Low scalability

Center for Research in Intelligent Storage

Stage 2: Cache Supported (memcache)

Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

Stage 2: Cache Supported (cache in between)

MySQL

WebServers

……

1. Higher write latency (clean

the cache)

2. Lower read latency

3. Higher scalability

Cache Layer

Center for Research in Intelligent Storage

Stage 3: TAO (The Associations and Objects)

Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

TAO Design 1: One Tier with Sharding

Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

TAO Design 2: Multiple Caching Tiers

Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

1. Complex distributed write

logic to handle to ensure

data correctness,

consistency

2. A lot of concurrent read

or write go to the

MySQL server, for the

same data

3. Make tradeoffs between

limiting the maximum

tier size and scaling the

cache

Source: https://www.usenix.org/node/174510

Page 13: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 14: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 15: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 16: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

Master regionSlave region 1

Slave region 2

Master DBSlave DB

Slave DB

How to Scale out

Page 18: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 19: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 20: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Page 21: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

MySQL Architecture

Source: https://techsoftcompute.wordpress.com/2013/08/19/mysql-internal-architecture/

Page 23: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: http://data-base-recovery.blogspot.com/2015/10/storage-engines-in-mysql.html

Page 24: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.slideshare.net/morgo/inno-db-presentation

Center for Research in Intelligent Storage

RocksDB

MySQL Architecture in FaceBook

Source: https://techsoftcompute.wordpress.com/2013/08/19/mysql-internal-architecture/

Page 27: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.slideshare.net/meeeejin/rocksdb-detail

Page 28: for the Social Graph - · Center for Research in Intelligent Storage 1. Complex distributed write logic to handle to ensure data correctness, consistency 2.A lot of concurrent read

Center for Research in Intelligent Storage Source: https://www.usenix.org/node/174510

Center for Research in Intelligent Storage

Thanks

Q/A

Detecting and Eliminating Potential Violation of Sequential Consistency for concurrent C/C++ program Duan Yuelu, Feng Xiaobing, Pen-chung Yew

Linearizability: A Correctness Condition for …christoff/files/Linearizability...Linearizability: A Correctness Condition for Concurrent Objects MAURICE P. HERLIHY and JEANNETTE M

Linearizability Linearizability is a correctness criterion for concurrent object (Herlihy & Wing ACM TOPLAS 1990). It provides the illusion that each operation

t has not been edited for consistency of correctness ...vdl.sci.utah.edu/CVOWorkshops/assets/documents/...Contextual inquiry, interviews, … Workshop. Create tasks and data abstraction

Verifying Concurrent Programs against Sequential Speci cationslara.epfl.ch/~hamza/esop2013.pdf · 1 Introduction A key class of correctness criteria for concurrent systems is adherence

Algorithms for Synchronization and Consistency in Concurrent System Services

Linearizability: A Correctness Condition for Concurrent ...cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf · message-passing architectures in which the shared ... for J. M. Wing was

Think - State Institute for Urban Development · Communication Skills –7C’s Credibility Courtesy Clarity Correctness Consistency Concreteness Conciseness. Effective Communication

Supporting Data Consistency in Concurrent Process Execution with Assurance Points and Invariants* Andrew Courter Masters Thesis Defense Department of Computer

Replication of Data CIS 505: Software Systems Lecture Note ...lee/07cis505/Lec/lec... · o The notion of correctness: consistency models Data-centric consistency Client-centric consistency

Correctness Proofs

A Consistency Framework for Iteration Operations in Concurrent Data Structures Yiannis Nikolakopoulos A. Gidenstam M. Papatriantafilou P. Tsigas Distributed

Correctness-Preserving Derivation of Concurrent Garbage Collection Algorithms

Knowledge-Based Engineering · Technical Specification Statement of Work Logistics Support ISO 29148 Completeness Consistency Correctness Completeness Consistency ... • SCM VIEWS

Correctness of Speculative Optimizations with Dynamic ...aviral.io/publications/correctness-of-speculative-optimizations-with... · 49 Correctness of Speculative Optimizations with

Eran Yahav 1. Previously… An algorithmic view Abstract data types (ADT) Correctness Conditions Sequential consistency Linearizability Treiber’s

Concepts of Software Quality Yonglei Tao 1. Software Quality Attributes Reliability correctness, completeness, consistency, robustness Testability

Linearizability: A Correctness Condition for Concurrent ...mph/HerlihyW90/p463-herlihy.pdf · Linearizability: A Correctness Condition for Concurrent Objects MAURICE P. HERLIHY and

Integrity Constraint Management CS2312. The correctness and consistency of the data and its information Implicit of the data model specified and

Behavioural Types: from Theory to Tools - Enlighteneprints.gla.ac.uk/146884/2/146884.pdfproperty is correctness of communication in concurrent ... 2 Contract-Oriented Programming with

Correctness-Preserving Derivation of Concurrent Garbage Collection Algorithms Martin T. Vechev Eran Yahav David F. Bacon University of Cambridge IBM T.J

Linearizability with Ownership Transfersoftware.imdea.org/~gotsman/papers/linown-concur12-ext.pdfappropriate notion of library correctness. Correctness of concurrent libraries is commonly

Supporting Data Consistency in Concurrent Process Execution with Assurance Points and Invariants*

Ordered Sequential Consistency: Composition and Correctness...2. K r Lev-Ari, Gregory Chockler, and Idit Keidar. A constructive approach for prov-ing data structures’ linearizability

Case Study...Challenge #3 - Consistency with Other Assessments Maintaining consistency with concurrent risk assessments, MRSPP updates, or previous MEC HAs • The frameworks have

On Library Correctness under Weak Memory Consistencyplv.mpi-sws.org/yacovet/paper.pdf · 68 On Library Correctness under Weak Memory Consistency Specifying and Verifying Concurrent

A Library of Concurrent Objects and Their Proofs of ...wing/publications/CMU-CS-90-151.pdf · A Library of Concurrent Objects and Their Proofs of Correctness Chun Gong and Jeannette

Global-Local View: Scalable Consistency for Concurrent ... · concurrent data structures, its implementation can be prohibitively expensive. As the number of cores increases in a

Correctness Correctness. Quality Perceptions The perception of quality associated with your code is typically bound to: Correctness Efficiency (speed

Chap 6 Synchronization. Background Concurrent access to shared data may result in data inconsistency Maintaining data consistency requires mechanisms

ParallelJava 2012 8 ConditionDeadlockWR · Serial*versus*concurrent Sequential correctness is mostly concerned with safety properties: – ensuing that a program transforms each before-state

Sciencecs237/lectures/old/dsmlecture1old.pdfConsider the requirements of transaction based systems Atomicity - either all effects take place or none Consistency - correctness of data

Introduction to weak memory consistencyviktor/wmc/intro.pdf · 2017-09-01 · Weak memory consistency isaboutthe semantics of concurrent programs takingintoaccounttheeﬀectsof: I

Lecture 11 Distributed Transaction Management. Concurrency Control The problem of synchronizing concurrent transactions such that the consistency of the

Linearizability: A Correctness Condition for Concurrent Objects