Euro-Par, 2006 1

Resource Allocation for Distributed Streaming Applications

ICPP 2008 Conference

Sept. 10th, 2008 Portland, Oregon

Qian Zhu and Gagan Agrawal

Department of Computer Science and Engineering

The Ohio State University

ICPP 2008

Euro-Par, 2006 2

Data Streaming Applications

• Computational Steering– Interactively control scientific simulations

• Computer Vision Based Surveillance– Track people and monitor critical infrastructure– Images captured by multiple cameras

• Online Network Intrusion Detection– Analyze connection request logs – Identify unusual patterns

ICPP 2008

Euro-Par, 2006

Streaming Applications in Wide Area Environments

• Distributed high-volume data sources • Increasing WAN bandwidths

– Better than secondary storage bandwidths

• Geographically distributed users / consumers of data

• Exploit flexibility in resource usage in Grid Environments

3

Euro-Par, 2006

Our Previous Work

• A middleware system GATES – Grid-based AdapTive Executions on Streams

• Integration with Grid Standards• Support for self-adaptation• Dynamic allocation and fault-tolerance

4

Euro-Par, 2006 5

Resource Allocation in Streaming Grid Applications

• Challenges– Pipeline of processing stages

• Computation and communication requirements

– Long running nature• Dynamic grid resources

• Current Approach– Ad Hoc and Heuristics-based

– Not considering both bandwidth and computing power

ICPP 2008

Euro-Par, 2006

Overview of Our Research

• Static Resource Allocation– Subgraph isomorphism based

– Handle Network bandwidth and Computing power

– Effectiveness value

• Goal– To minimize the execution time of the data

streaming applications

ICPP 2008

Euro-Par, 2006 7

Outline

• Motivation and Introduction

• Resource Allocation in Data Stream Processing

• Resource Allocation Algorithm

• Experimental Evaluation

• Related Work

• Conclusion

ICPP 2008

Euro-Par, 2006

Data Stream Processing Model

• Directed Acyclic Graph (DAG) – Gp(Vp, Ep)

1800

2300

3200

41000

100

100

120

200

GP

sink

source

Processing nodes

Bandwidth

Requirement

Computing Power

Requirement

ICPP 2008

Euro-Par, 2006

Resource Model

• Directed Acyclic Graph (DAG) – GR(VR, ER)

A1000

B400

C200

D100

E2000 F

600G

800

120400

150 200

30

200

100

100

200

G2

Bandwidth

Computing power

ICPP 2008

Euro-Par, 2006

Problem Description

• To Allocate Resources to the Data Stream Application– A mapping from Gp(Vp, Ep) to GR(VR, ER)

• Modified Subgraph Isomorphism Based– To choose an isomorphic subgraph of GR

– Transporters

• Optimal Mapping– Effectiveness value

– To minimize the execution time

ICPP 2008

Euro-Par, 2006

Example

transporter

A1000

B400

C200

E2000

D100

ICPP 2008

Euro-Par, 2006

Effectiveness Value

• Bandwidth only

• Including Computing Power

A sigmoid function

Number of transporters

Overhead of adding transporters

Computing power match

ICPP 2008

Euro-Par, 2006 13

Outline

• Motivation and Introduction

• Resource Allocation in Data Stream Processing

• Resource Allocation Algorithm

• Experimental Evaluation

• Related Work

• Conclusion

ICPP 2008

Euro-Par, 2006

Proposed Algorithm

• Background: VF algorithm (L.P.Cordella et al.)

– State Space Representation (SSR)– Feasibility rules– Depth-First Search

• Pros and Cons– Efficient with small graphs (<200 nodes)– A large number of candidate partial mappings

ICPP 2008

Euro-Par, 2006

Proposed Algorithm – Step 1• Prune Candidate Partial Mappings

– Candidate node list– Reduce potential matches– Multiple Partial Mapping set

1 2 3 4

B

C

D

E

F

G

C

D

G

A

E

F

Cand(3)={C,D,G}

3200

A1000

B400

C200

E2000

D100

ICPP 2008

Euro-Par, 2006

Proposed Algorithm – Step 2

• Modified Subgraph Isomorphism Mapping– Transporters

A1000

B400

C200

E2000

D100

3200

1 2 3 4

B

C

D

E

F

G

C

D

G

A

E

F

Candidate pair: (3,C)

Candidate pair: (3,D)

transporter

ICPP 2008

Euro-Par, 2006

Handle Computing Power

• Computing Node Network Link– Computing power Network bandwidth

• Effectiveness Value Calculation • Possible Issues: high bandwidth and low

computing power– Map one node onto a cluster of network nodes

ICPP 2008

Euro-Par, 2006 18

Outline

• Motivation and Introduction

• Resource Allocation in Data Stream Processing

• Resource Allocation Algorithm

• Experimental Evaluation

• Related Work

• Conclusion

ICPP 2008

Euro-Par, 2006

Goals for the Experiments

• Demonstrate the Scalability of Our Resource Allocation Algorithm

• Demonstrate the High Performance of the Applications

ICPP 2008

Euro-Par, 2006

Experiment Setup

• Algorithms Compared– Optimal– Streamline

• Streaming Applications– Volume Rendering Application– A Synthetic Application

ICPP 2008

Euro-Par, 2006

Scalability of the Resource Allocation Algorithm

ICPP 2008

Euro-Par, 2006

Application Performance

• Volume Rendering

Within 4%

33%

29%

27%

ICPP 2008

Euro-Par, 2006

Application Performance

• A Synthetic Application

Within 3%

40%

36%

34%

ICPP 2008

Euro-Par, 2006 24

Outline

• Motivation and Introduction

• Resource Allocation in Data Stream Processing

• Resource Allocation Algorithm

• Experimental Evaluation

• Related Work

• Conclusion

ICPP 2008

Euro-Par, 2006

Related Work

• Resource Allocation for Stream Processing– Tang et al. (HPCC 06), Ali et al. (PDPTA 02)

• Resource Allocation for Grid Computing– Abdu et al. (IPDPS 01), Bhat et al. (Grid 07),

Hong et al. (ICPP 03)• Subgraph Isomorphism Algorithms and

Applications– Bioinformatics (Online Information 90), VLSI design

(ISCAS 95), Mobile robot design (JPR 95)

ICPP 2008

Euro-Par, 2006

Conclusion

• Modified Subgraph Isomorphism Algorithm for Resource Allocation in Grid Streaming Applications

• Handling Network Bandwidth and Computing Power

• Comparable Overhead with Streamline• Improved Application Performance

ICPP 2008

Euro-Par, 2006 27

Thank you!

ICPP 2008