l3: ddbs design (tailored from the slides by prof. hector garcia-molina )

L3-1 DDBS Design -- 1

L3: DDBS Design (Tailored from L3: DDBS Design (Tailored from the Slides by Prof. Hector the Slides by Prof. Hector Garcia-Molina)Garcia-Molina)

1. Introduction2. Fragmentation

1) Horizontal fragmentation

2) Vertical fragmentation

3. Allocation


Distributed Database Design Distributed Database Design

Top-down approach: - have DB…

- how to split and

allocate the sites

Multi-DBs (or bottom-up): no design

issues!


Two issues in DDB design:Two issues in DDB design:

Fragmentation Allocation

Note: issues not independent, but will cover separately


ExampleExample

Employee relation E (eno,name,add,sal,…)

40% of queries: 40% of queries:

Qa: select * Qb: select *from E from Ewhere add=Sa where

add=Sband… and ...


ExampleExample

Employee relation E (eno,name,add,sal,…)

40% of queries: 40% of queries:

Qa: select * Qb: select *from E from Ewhere add=Sa where

add=Sband… and ...

Motivation: Two sites: Sa, Sb

Qa QbSa Sb


Eno NM Add Sal E

578

Sa 10Sally Sb 25Tom Sa 15

Joe

Eno NM Add Sal Eno NM Add Sal58

Sa 10Tom Sa 15Joe 7 Sb 25Sally

..

..

....

F

At SaAt Sb


F = { F1, F2 }

F1 = add=Sa E F2 = add=Sb E


F = { F1, F2 }

F1 = loc=Sa E F2 = loc=Sb

E called primary horizontal fragmentation


FragmentationFragmentation

Horizontal Primary depends on local attributesR Derived

depends on foreign relation

Vertical

RFragmentation also called

Sharding


Three common horizontal Three common horizontal partitioning techniquespartitioning techniques

Round robin Hash partitioning Range partitioning


• Round robinRound robin

R D0 D1 D2

t1 t1t2 t2t3

t3t4 t4... t5

Evenly distributes data Good for scanning full relation Not good for point or range queries


• Hash partitioningHash partitioning

R D0 D1 D2

t1h(k1)=2 t1

t2h(k2)=0 t2t3h(k3)=0 t3t4h(k4)=1 t4...

Good for point queries on key; also for joins

Not good for range queries; point queries not on key

If hash function good, even distribution


• Range partitioningRange partitioning

R D0 D1 D2

t1: A=5 t1t2: A=8 t2t3: A=2 t3t4: A=3 t4...

Good for some range queries on A Need to select good vector: else

unbalance data skew execution

skew

4 7

partitioningvector

V0 V1


Which are good Which are good fragmentations?fragmentations?Example:

F = { F1, F2 }

F1 = sal<10 E F2 = sal>20 E

Problem: Some tuples lost!


Which are good Which are good fragmentations?fragmentations?Second example:

F = { F3, F4 }F3 = sal<10 E F4 = sal>5 E

Tuples with 5 < sal < 10 are duplicated...


Prefer to deal with replication explicitly

Example: F = { F5, F6, F7 }

F5 = sal 5 E F6 = 5< sal <10

E

F7 = sal 10 E

Then replicate F6 if convenient (part of allocation problem)


Desired properties for Desired properties for horizontal fragmentationhorizontal fragmentation

R F ={ F1, F2, … }

(1) Completeness

t R, Fi F such that t Fi


(2) Disjointness t Fi, Fj such that

tFj, i j, Fi, Fj F

(3) Reconstruction - ignore


How do we get completeness How do we get completeness and disjointness?and disjointness?

(1) Check it “manually”!

e.g., F1 = sal<10 E ; F2 = sal10 E

(2) “Automatically” generate fragments with these properties

Desired simple predicates Fragments


Example of generationExample of generation

Say queries use predicates:A<10, A>5, Add = SA, Add = SB

Next: - generate “minterm” predicates

- eliminate useless ones


Minterm predicates (part I)Minterm predicates (part I)(1) A<10 A>5 Add=SA Add=SB

(2) A<10 A>5 Add=SA ¬(Add=SB)(3) A<10 A>5 ¬(Add=SA) Add=SB

(4) A<10 A>5 ¬(Add=SA) ¬(Add=SB)(5) A<10 ¬(A>5) Add=SA Add=SB

(6) A<10 ¬(A>5) Add=SA ¬(Add=SB)(7) A<10 ¬(A>5) ¬(Add=SA) Add=SB

(8) A<10 ¬(A>5) ¬(Add=SA) ¬(Add=SB)






(8) A<10 ¬(A>5) ¬(Add=SA) ¬(Add=SB)






(8) A<10 ¬(A>5) ¬(Add=SA) ¬(Add=SB)

A 5

5 < A < 10


Minterm predicates (part II)Minterm predicates (part II)

(9) ¬(A<10) A>5 Add=SA Add=SB

(10) ¬(A<10) A>5 Add=SA ¬(Add=SB)(11) ¬(A<10) A>5 ¬(Add=SA) Add=SB

(12) ¬(A<10) A>5 ¬(Add=SA) ¬(Add=SB)(13) ¬(A<10) ¬(A>5) Add=SA Add=SB

(14) ¬(A<10) ¬(A>5) Add=SA ¬(Add=SB)(15) ¬(A<10) ¬(A>5) ¬(Add=SA) Add=SB

(16) ¬(A<10) ¬(A>5) ¬(Add=SA) ¬(Add=SB)


Minterm predicates (part II)Minterm predicates (part II)

(9) ¬(A<10) A>5 Add=SA Add=SB

(10) ¬(A<10) A>5 Add=SA ¬(Add=SB)(11) ¬(A<10) A>5 ¬(Add=SA) Add=SB

(12) ¬(A<10) A>5 ¬(Add=SA) ¬(Add=SB)(13) ¬(A<10) ¬(A>5) Add=SA Add=SB

(14) ¬(A<10) ¬(A>5) Add=SA ¬(Add=SB)(15) ¬(A<10) ¬(A>5) ¬(Add=SA) Add=SB

(16) ¬(A<10) ¬(A>5) ¬(Add=SA) ¬(Add=SB)

A 10


Final fragments:Final fragments:

F2: 5 < A < 10 Add=SA F3: 5 < A < 10 Add=SB F6: A 5 Add=SA F7: A 5 Add=SB F10: A 10 Add=SA F11: A 10 Add=SB


Note:Note: elimination of useless elimination of useless fragments depends on fragments depends on application application semantics:semantics:

e.g.: if LOC could be SA, SB, we need to add fragments

F4: 5 <A <10 Add SA Add SB

F8: A 5 Add SA Add SB

F12: A 10 Add SA Add SB


Why does this work?Why does this work?

Predicates: p1 p2 p3 p4 p1 p2 p3 ¬ p4

¬ p1 ¬ p2 ¬ p3 ¬ p4

...


(1) Completeness: Take t Rpi(t) must be T or F!

Say p1(t) =T p2(t) = T p3(t) =F p4(t) =F

Then t is in fragment with predicate p1 p2 ¬ p3 ¬ p4


(2) DisjointnessSay t Fragment p1 p2 ¬ p3 ¬ p4

Then: p1(t) = T, p2(t) = T, p3(t) = F, p4(t)= F

t cannot be in any other fragment!


SummarySummary Given simple predicates Pr= { p1, p2,..

pm }minterm predicates are

M={m | m = pk*, 1 k m }

where pk* is pk or is ¬ pk

pkPr

• Fragments m R for all m M are

complete and disjoint


Another Desired Fragmentation Property:Another Desired Fragmentation Property:

Match Access PatternsMatch Access Patterns

data A

data B

data C

frequentlyaccessedtogether

try toplace in

samefragment


Return to example:Return to example:

E(Eno, NM, Add, SAL,…)Common queries:

Qa: select * Qb: select * from E from E where Add=Sa where

Add=Sb and … and ...


Three choices:Three choices:

(1) Pr = { } F1 ={ E }

(2) Pr = {Add=Sa, Add=Sb}

F2={ add=Sa E, add=Sb E }

(3) Pr = {Add=Sa, Add=Sb, Sal<10}

F3={ add=Sa sal<10 E, add=Sa sal10 E,

add=Sb sal<10E, add=Sb sal10 E }


In other words:In other words:

Add=Sa sal < 10

Add=Sa

sal 10

Add=Sb sal < 10

Add=Sb

sal 10

F1 F3F2

Qa: Select … add = Sa ...

Qb: Select … add = Sb ...

F2 is good…(not F1 , F3 )


HF Example Database HF Example Database InformationInformation

Title, Sal

PAY

Eno, Ename, Title

EMP

Jno, Jname, Budget. Loc

PROJ

Eno, Jno, Resp, Dur

ASG

The global schema

Owner and member relations

Cardinality of each relation

L1

L2 L3


PHF Desired Properties: PHF Desired Properties: Completeness Completeness

A set of simple predicates Pr is said to be complete if and only if there is an equal probability of access by every application to any tuple belonging to any minterm fragment that is defined according to Pr.

Example:

PNO PNAME BUDGET LOCP1 Instrumentation 150000 MontrealP2 Database 135000 New YorkP3 CAD/CAM 250000 New YorkP4 Maintenance 310000 ParisApplications:

Q1: Find the projects at each locationQ2: Find projects with budget less than $200,000 Predicates:Pr={ LOC=“Montreal”, LOC=“New York”, LOC=“Paris”}Pr={ LOC=“Montreal”, LOC=“New York”, LOC=“Paris”,

BUDGET 200000, BUDGET >200000}


PHF Example: PAYPHF Example: PAY

TITLE SALMech. Eng. 27000Programmer 24000Elec. Eng. 40000Syst. Anal. 34000

PAYApplication: Employee records kept at two sites , one site handling records with salary>30,000, another site handling records with salary <=30,000.

Simple predicates: Pr = {SAL 30000, SAL > 30000}

Minterm predicates: m1: (SAL 30000); m2: (SAL > 30000)

TITLE SALElec. Eng. 40000Syst. Anal. 34000

PAY2

TITLE SALMech. Eng. 27000Programmer 24000

PAY1


PHF Example: PROJPHF Example: PROJ

Applications: Find the name and budget of

projects given their location.– Issued at three sites

Access project information according to budget– One site access 200000 other

accesses >200000

p1: LOC = “Montreal”

p2 : LOC = “New York”

p3 : LOC = “Paris”

p4: BUDGET 200000

p5 : BUDGET > 200000

m1: LOC = “Montreal” BUDGET 200000

m2 : LOC = “Montreal” BUDGET > 200000

m3 : LOC = “New York” BUDGET 200000

m4 : LOC = “New York” BUDGET > 200000

m5 : LOC = “Paris” BUDGET 200000

M6 : LOC = “Paris” BUDGET > 200000


PHF Example: PROJ - ResultPHF Example: PROJ - Result

PNO PNAME BUDGET LOCP1 Instrumentation 150000 MntrealP2 Database 135000 New YorkP3 CAD/CAM 250000 New YorkP4 Maintenance 310000 Paris

PROJ

m1: LOC = “Montreal” BUDGET 200000

m2 : LOC = “Montreal” BUDGET > 200000

m3 : LOC = “New York” BUDGET 200000

m4 : LOC = “New York” BUDGET > 200000

m5 : LOC = “Paris” BUDGET 200000

M6 : LOC = “Paris” BUDGET > 200000

PNO PNAME BUDGET LOCP1 Instrumentation 150000 Mntreal

PROJ1

PNO PNAME BUDGET LOCP2 Database 135000 New York

PNO PNAME BUDGET LOCP3 CAD/CAM 250000 New York

PNO PNAME BUDGET LOCP4 Maintenance 310000 Paris

PROJ2

PROJ4

PROJ6


PHF - CorrectnessPHF - Correctness

Completeness Since Pr is complete and minimal, the

selection predicates are complete Reconstruction

If relation R is fragmented into FR = {R1, R2, …, Rr}

R = Ri FR Ri

Disjointness Minterm predicates that form the basis of

fragmentation should be mutually exclusive


DHF: Derived Horizontal DHF: Derived Horizontal FragmentationFragmentation

Title, Sal

PAY

Eno, Ename, Title

EMP

Jno, Jname, Budget. Loc

PROJ

Eno, Jno, Resp, Dur

ASG

L1

L2 L3

Each link is an equijoin

Owner relation

Member relation

DHF: Defined on a member relation according to a selection operation on its owner


DHF – ExampleDHF – Example

TITLE SALMech. Eng. 27000Programmer 24000

PAY1= SAL 30000 PAY

ENO ENAME TITLEE1 J. Doe Elec.Eng.E2 M. Smith Syst. Anal.E3 A. Lee Mech. Eng.E4 J. Miller ProgrammerE5 B. Casey Syst. Anal.E6 L. Chu Elec.Eng.E7 R. Davis Mech. Eng.E8 J. Jones Syst. Anal.

EMP ENO ENAME TITLEE3 A. Lee Mech. Eng.E4 J. Miller ProgrammerE7 R. Davis Mech. Eng.

EMP1 = EMP PAY1

ENO ENAME TITLEE1 J. Doe Elec.Eng.E2 M. Smith Syst. Anal.E5 B. Casey Syst. Anal.E6 L. Chu Elec.Eng.E8 J. Jones Syst. Anal.

EMP2 = EMP PAY2

TITLE SALElec. Eng. 40000Syst. Anal. 34000

PAY2= SAL> 30000 PAY

DHF

Title, Sal

PAY

Eno, Ename, Title

EMP L1


DHF: Derived Horizontal DHF: Derived Horizontal Fragmentation Fragmentation Let S be horizontally fragmented and let

there be a link L with owner(L) = S, and member(L) = R, the derived horizontal fragments of R are defined as

Ri = R Si , 1 i w

where Si is the horizontal fragment of S, is the semi join operator, and w is the maximum number of fragments

Inputs to derived horizontal fragmentation: partitions of owner relation member relation the semi join condition

The algorithm is straight forward.


DHF: Correctness DHF: Correctness

Completeness: primary horizontal fragmentation based on

completeness of selection predicates. For derived horizontal fragmentation based on referential integrity

Reconstruction: union

Disjointedness: primary horizontal fragmentation based on

mutually exclusive simple predicates. For derived horizontal fragmentation, based on whether the link between owner and member is 1:1 or 1:m relationship


DHF: IssuesDHF: Issues

Multiple owners for a member relation; how should we derived horizontally fragment a member relation.

There can be a chain of derived horizontal fragmentation.


L3.2.2: DDBS DesignL3.2.2: DDBS Design

Introduction Fragmentation

Horizontal fragmentation

Vertical fragmentation Allocation


Vertical fragmentationVertical fragmentation

E1

Eno NM Add Sal5 Joe Sa 107 Sally Sb 258 Fred Sa 15…

Eno NM Add5 Joe Sa7 Sally Sb8 Fred Sa…

Eno Sal5 107 258 15…

E

E2

Example:


R[T] R1[T1] Ti T

Rn[Tn]

Just like normalization of relations

...


Properties:Properties: R[T] R[T] RRii[T[Tii] ]

(1) Completeness

U Ti = T

all i


(2) DisjointnessTi Tj = for all i,j ij

E(#,LOC,SAL) E1(#,LOC)

E2(SAL)


(2) DisjointnessTi Tj = for all i,j ij

E(#,LOC,SAL) E1(#,LOC)

E2(SAL)

Not a desirable property!!(could not reconstruct R!)


(3) Lossless join

Ri = R

all i

One way to achieve lossless join: Repeat key in all fragments, i.e.,

Key Ti for all i


How do we decide what How do we decide what attributesattributes are grouped with which? are grouped with which?

E1(Eno,NM,Add)E2(Eno,SAL)

Example:E(Eno,NM,LOC,SAL)

E1(Eno,NM)E2(Eno,Add)E3(Eno,SAL)

?


A1 A2 A3 A4 A5A1 - - - - -A2 50 - - - -A3 45 48 - - -A4 1 2 0 - -A5 0 0 4 75 -

Attribute affinity matrixAttribute affinity matrix


A1 A2 A3 A4 A5A1 - - - - -A2 50 - - - -A3 45 48 - - -A4 1 2 0 - -A5 0 0 4 75 -

Attribute affinity matrixAttribute affinity matrix

R1[K,A1,A2,A3] R2[K,A4,A5]


Textbook (Ozsu & Valduriez) discusses How to build affinity matrix How to identify attribute clusters How to partition relation

You are not responsible for Clustering and partitioning algorithms (i.e., Skip pages 135-145)


AllocationAllocation

Example: E(Eno,NM,Add,SAL) F1 = add=Sa E ; F2 = add=Sb E

Qa: select … where add=Sa...Qb: select … where add=Sb…

Site a Site b

Where do F1,F2 go?

?


IssuesIssues Where do queries originate What is communication cost?

and size of answers, relations,…

What is storage capacity, cost at sites? and size of fragments?

What is processing power at sites?


What is query processing strategy?

How are joins done? Where are answers collected?

More IssuesMore Issues


Cost of updating copies? Writes and concurrency control? ...

Do we replicate fragments?Do we replicate fragments?


Optimization problem:Optimization problem: What is best placement of

fragments and/or best number of copies to: minimize query response time maximize throughput minimize “some cost” ...

Subject to constraints? Available storage Available bandwidth, power,… Keep 90% of response time below X ...


Optimization problem:Optimization problem: What is best placement of

fragments and/or best number of copies to: minimize query response time maximize throughput minimize “some cost” ...

Subject to constraints? Available storage Available bandwidth, power,… Keep 90% of response time below X ...

This is an incredibly hard problem


Example:Example: Single fragment F Single fragment F

Read cost: [ti MIN Cij]

i: Originating site of requestti: Read traffic at Si

Cij: Retrieval costAccessing fragment F at Sj

from Si

ji=1

m


Scenario Scenario - Read cost- Read cost

1 2

.

. .

.

.

.

3

i

C=inf

ci,3

ci,1 ci,2

Stream of readrequests for F ti REQ/SEC

C=inf

C=infF

F F


Write costWrite cost

Xj ui C’ij

i: Originating site of requestj: Site being updatedXj: 0 if F not stored at Sj

1 if F stored at Sjui: Write traffic at Si C’ij: Write cost

Updating F at Sj from Si

i=1 j=1

m m


ScenarioScenario - write cost - write cost

Updates

ui updates/sec

. .

. .

. .i

F

F F


Storage cost:Storage cost:

Xi di

Xi: 0 if F not stored at Si

1 if F stored at Si

di: storage cost at Si

i=1

m


Target functionTarget function::

min [ti MIN Cij + Xj ui

C’ij ]

+ Xi di

ji=1 j=1

i=1

m m

m


Can add more complications:Can add more complications:

Examples: - Multiple fragments- Fragment sizes- Concurrency control cost


Solution MethodsSolution Methods

NP-complete Heuristics approaches Reduce search space

l3: ddbs design (tailored from the slides by prof. hector garcia-molina )

Documents