Efficient Skyline Computation on Vertically Partitioned Datasets

Dimitris Papadias, David Yang, Georgios TrimponiasCSE Department, HKUST, Hong Kong

Outline Problem Statement Skyline Computation on Vertically Partitioned

Datasets using Balke’s Algorithm Algorithms for Top-k Query Processing FM Sketches Putting Everything Together

Outline Problem Statement Skyline Computation on Vertically Partitioned

Datasets using Balke’s Algorithm Algorithms for Top-k Query Processing FM Sketches Putting Everything Together

A Motivating Example Consider a database containing information

about hotels. The y-dimension represents the price of the room, whereas the x-dimension captures the distance of the room from the beach.

Distance

Price Skyline objects

Hotel rooms

p

Dominance Region of p

Borders of p’sDominance Region

Skyline Preliminaries[ICDE, 2001] Skylines constitute a very useful tool in numerous

disciplines, such as for multidimensional decision making and data mining.

Given a set of d-dimensional objects p1, …, pN, the skyline operator retrieves all these objects that are nor dominated by any other object in the set.

An object pi dominates another point pj, if it is not worse than pj in all dimensions and better than it in at least one dimension.

Properties:• The top-1 tuple according to any preference function

that assigns scores to tuples is in the skyline tuple.• Conversely, for any skyline tuple, there exists a

preference function according to which it is the top-1.

4 Common Data Distributions

Problem Definition Compute the skyline when the dataset is

vertically decomposed among a set of N servers.

Goal: minimize the data that must be retrieved from each server.

We assume wireless environments, where communication overhead constitutes the dominant factor in battery consumption.

Consider mobile phone applications as real world examples.

(a) Subspace D1 at server N1 (b) Subspace D2 at Server N2

d2

d1

AF

BC

D

E

GH

local skyline

d3

d4

A F

B

C

D

E

GH

local skyline

First Observations

The global skyline may contain points that do not appear in the local skylines.

Instead of transmitting all records over the network, avoid sending out points that are guaranteed to be dominated globally by an anchor point.

Outline Problem Statement Skyline Computation on Vertically

Partitioned Datasets using Balke’s Algorithm

Algorithms for Top-k Query Processing FM Sketches Putting Everything Together

Balke’s Algorithm[EDBT, 2004] Assume that the d-dimensional database is

vertically partitioned into d lists, one for each dimension, assigned to different servers. The lists contain values in ascending order.

Idea: perform sorted accesses on the d lists in a round-robin manner, until a point p (anchor), is reached in every list.

Points that have not showed up at this moment in any list can be safely pruned, since they are dominated by the anchor.

Example Let a 2-dimensional database with the

following two lists: L1

L2

Point a b d m g c

Value 1 2 3 5 5 6

Point c d e k a b

Value 1 2 3 4 5 7

…

…

Example (cont.) Let a 2-dimensional database with the

following two lists: L1

L2

Point a b d m g c

Value 1 2 3 5 5 6

Point c d e k a b

Value 1 2 3 4 5 7

…

…

The first point to be retrieved from both lists.

Example (cont.) Let a 2-dimensional database with the

following two lists: L1

L2

Point a b d m g c

Value 1 2 3 5 5 6

Point c d e k a b

Value 1 2 3 4 5 7

…

…

The first point to be retrieved from both lists.These points cannot be part of the skyline.

Further Improvement Efficiency can be improved, if instead of

visiting the lists in a round-robin manner, we access the most promising list with random accesses.

As a result, only the least expansion is performed on each list.

∙ ∙ ∙ P ∙ ∙ ∙L1

∙ ∙ ∙ P ∙ ∙ ∙L2

avoid visiting these points

Outline Problem Statement Skyline Computation on Vertically Partitioned

Datasets using Balke’s Algorithm Algorithms for Top-k Query Processing FM Sketches Putting Everything Together

Setting Let N1, .., Nm be m servers storing the same

dataset DB. For each record P DB every server Ni maintains

a local score si(P), and sorts all records in decreasing order of their local scores.

A client wishes to obtain the k records of DB with the maximum global score s.

The score is computed using a monotonic function f on the local scores, i.e., s(P) = f(s1(P), .., sm(P)).

Goal: minimize the required number of accesses.

Fagin’s Algorithm[PODS, 2001] Each server Ni performs sorted round-robin

accesses and sends to the client the next record and its local score.

When the first common record Panc is encountered by all servers, the client terminates the sorted accesses.

Then, it obtains the missing local scores of the other encountered points through random accesses.

The candidate with the highest global score is the top-1 result.

Threshold Algorithm[PODS, 2001] It utilizes an upper bound TA on the global

score to terminate earlier than FA. The client retrieves the local scores of newly

encountered points with random accesses at the remaining servers and computes their global scores, and picks the best score sbest.

The threshold TA is equal to the sum of the local thresholds at each server.

As long TA > sbest, TA continues the sorted accesses, while it keeps updating TA.

Eventually, the top-1 point will be returned.

Example for FA and TA

Best Position Algorithm[VLDB, 2007] It further improves TA by utilizing a tighter

threshold. Let bpi be the position at server Ni such that

all points up to bpi have been encountered through sorted or random accesses.

The global threshold BP is equal to the sum of the local thresholds at bpi.

Outline Problem Statement Skyline Computation on Vertically Partitioned

Datasets using Balke’s Algorithm Algorithms for Top-k Query Processing FM Sketches Putting Everything Together

Flajolet / Martin sketches[JCSS ’85]

Goal: Estimate the distinct number of objects from a small-space representation of a set.

Sketch of a union of items is the OR of their sketches Insertion order and duplicates don’t matter!

Prerequisite: Let h be a random, binary hash function.

Sketch of an item

For each unique item with ID x,

For each integer 1 ≤ i ≤ k in turn,

Compute h (x, i).

Stop when h (x, i) = 1, and set bit i.

X 0 0 1 0 0

Z 1 0 0 0 0

X Z 1 0 1 0 0

∩

Flajolet / Martin sketches (cont.)

Estimating COUNT

Take the sketch of a set of N items.

Let j be the position of the leftmost zero in the sketch.

j is an estimator of log2 (0.77 N)

Fixable drawbacks:

• Estimate has faint bias

• Variance in the estimate is large.

1 1 01S 1

Best guess: COUNT ~ 11

j = 3

Flajolet / Martin sketches (cont.) Standard variance reduction methods apply.

Compute m independent sketches in parallel. Compute m independent estimates of N. Take the mean of the estimates.

Provable tradeoffs between m and variance of the estimator.

Application to COUNT in Sensor Databases• Each sensor computes k independent sketches of

itself (using unique ID x)– sensor computes a sketch of its value.

• Use a robust routing algorithm to route sketches up to the sink.

• Aggregate the k sketches via union en-route. (OR)• The sink then estimates the count.

sink

S1

S3

S2

S4

S1S2

S1∪S2∪S3 S4

S1∪S2∪S3∪S3

Outline Problem Statement Skyline Computation on Vertically Partitioned

Datasets using Balke’s Algorithm Algorithms for Top-k Query Processing FM Sketches Putting Everything Together

Problem Characteristics Each vertical decomposition has arbitrary

dimensionality, contrary to Balke’s setting. Anchor selection substantially determines the

total number of transmitted data. VPS adopts sorting on the local dominance. In

particular, the local dominance count domi(P) of a point P with respect to subspace Di is the number of points dominated by P in Di.

Balke selects as the anchor, the data point P with the maximal domSUM(P).

We utilize a tighter upper bound for dom(P) is the minimum domMIN among all local dominance counts.

Anchor Selection

(a) Subspace D1 at server N1 (b) Subspace D2 at Server N2

d2

d1

AF

BC

D

E

GH

local skyline

d3

d4

A F

B

C

D

E

GH

local skyline

C: optimal anchor pointA: has maximal domMIN

B: has maximal domSUM

Our algorithm on the previous example

1st Optimization: Multiple Anchor Points The previous algorithm performs pruning with

a single anchor Panc. Specifically, a point P that is locally dominated by Panc in all subspaces is not sent to the client.

On the other hand, if P is incomparable with Panc even in a single subspace Di, it will be transmitted by the corresponding server Ni.

We suggest that multiple points can often achieve more effective pruning.

Pruning with 2 points

2nd Optimization: Integration of Sketches So far, we have estimated the (expected) global

dominance dom(P) of a point P using domMIN(P). This approach is biased towards points that have

high local dominance counts in all subspaces, but dominate few records globally (A).

Thus, we propose an unbiased approach that directly estimates the global dominance counts using sketches that count the number of distinct objects approximately.

We assume that each Ni server has a local dominance sketch ski(P) for every point P, which aggregates all points that P dominates locally in Di.

Experiments

Thank you!