1

Data Aggregation Through Web Service Composition in Smart Camera Networks

Rajapaksage Jayampathi

Advisor : Dr Sushil K Prasad

Master Thesis DefenseNovember 19, 2010

2

Outline

• Introduction

• Web Service Enabled Smart Camera Network Architecture

• Data Aggregation through Web Service Composition

• SQL Query Primitives For Smart Camera Network

• Producer/Consumer Pattern

• Results and Analysis

• Conclusion

3

Introduction [Contd.]

SensingUnit

ProcessingUnit

CommunicationUnit

Se

nso

r

AbstractData

Figure : Generic Architecture of a Smart Camera

• Smart Camera– Smart camera is a powerful sensor that can process complex information

such as image and video streams

− Output can range from abstract data to video streams

Figure : CMUCam3 Smart Camera

4

Introduction

• Smart Camera Networks− Real-Time embedded systems− Perform computer vision using multiple cameras − Central processing and distributed processing− Also called Distributed Smart Camera Networks (DSC Networks)

Figure : Distributed Processing in a Smart Camera

5

Introduction [Contd.]

• Specific Requirements of Smart Camera Networks:– Resource availability requirement– Image processing requirement– Communication bandwidth requirement– Automated load distribution requirement

• Challenges in Smart Camera Networks:– Complex image processing algorithm development– Collaborative image analysis/decision making– Image/Video Data Aggregation – High level query API development

6

Introduction [Contd.]

• Applications:• interactive• dynamic• stream based• computationally demanding and • need real time or near real time guarantees

• Applied Areas:• law enforcement• habitat monitoring• security surveillance – gang activity detection etc.

• Designing, implementing, and deploying applications for DSCs is a complex and challenging task due to the multiple and heterogeneous nature of the DSC networks

7

Scope of the Thesis

• Problem Statement

Our thesis focuses on following research problems in Smart Camera Networks:

– Problem 1 :

Providing data aggregation for complex image and video streams

– Problem 2:

Providing scalable operation by exposing functionality and data produced by the nodes in highly interoperable manner

– Problem 3:

Providing High Level Language Primitives for querying DSC networks

8

Scope of the Thesis

• Research Contributions

We develop a novel SOA based middleware framework on a DSC network.

– Contribution1:Scalable operation of DSC through web service enablement using Distributed OSGi

– Contribution2:Data aggregation through web service composition

– Contribution3:Develop an SQL API for Image Data Analysis

– Contribution4:Efficient Resource Management in DSCs using OSGI Life Cycle

To best of our knowledge this is the first time above has been done for a DSC environment

9

Web Service Enabled Smart Camera Network Architecture

• Environmental Description – Hierarchical network

architecture

– Server receives queries from user, send queries to camera nodes and sends results back to the user

– Camera nodes with higher number of children are more powerful in terms of processing capability, memory capacity etc

Figure: The hierarchical DSC network Architecture

Server

Root Smart Camera Node

10

Web Service Enabled Smart Camera Network Architecture [Contd.]

• Proposed Layered Middleware Architecture of a Camera Node:

Figure: Layered Middleware Architecture of a Smart Camera

Applications

Domain-specificmiddleware

servicesCommon

middlewareservices

Distribution layer

Host infrastructurelayer

Operatingsystems and

protocols

Hardware device

General PurposeMiddleware LayeredArchitecture

Web servicecomposition

Web services

Distributed OSGi

OSGi

Java VirtualMachine (JVM)

Operatingsystems and

protocols

Hardware device

Layered MiddlewareArchitecture of aSmart Camera

11

Web Service Enabled Smart Camera Network Architecture [Contd.]

• OSGi and Distributed OSGi:– Third party components that are commercially available– Relatively mature to serve our purpose– Used to model dynamic behaviour of the network– Every Smart Camera node is an OSGi Container

• OSGi– Component based framework specification– Brings modularity to the Java platform – Enables creating highly cohesive, loosely coupled deployable

modules called Bundles– Bundles offer services– Bundles offer lifecycle management

12

Web Service Enabled Smart Camera Network Architecture [Contd.]

• OSGi framework enable SOA by :– Every Smart Camera node is an OSGi

Container– Service producer bundles register

services– Consumer bundles consume services. – Services are published and consumed

within the same JVM. (SOA in a JVM)

• Distributed OSGi– Allow OSGI service bundles to be

published as web services (distribution capability)

– We use Apache cxf-dosgi as D-OSGi framework

OSGiServiceRegistry

ServiceBundle

ConsumerBundle

1. RegisterService

2. FindService

3. Bind/UseService

Figure: OSGi SOA in JVM

UDDIRegistry

ServiceProvider

ServiceConsumer

2. FindService

1. RegisterService

3. Bind/UseService

Figure: Service Oriented Architecture (SOA)

13

Web Services Local Services

Image and Video Raw Data Collection Web Services

RawDataSerivce

Operations:

- getNodeId()

- getData() Request service

- putData() Send results

Face Detection Services

DetectFacesService

Operations:

- getNumFaces()

Image and Video Data Aggregation Web Services

AverageService

MinService

MaxService

WinMaxService

WinMinService

WinAverageService

Neighbor Node Communication Services

ChildService

Operations:

- callParentService()

ParentService

Operations:

- callChildService()

Dynamic Loading and Unloading Web Services

LoadUnloadService

Operations:

- startService(serviceAPI,serviceImpl)

- stopService(serviceAPI,serviceImpl)

Dynamic Loading and Unloading Local Services

LoadUnloadService

Operations:

- startService(serviceAPI,serviceImpl)

- stopService(serviceAPI,serviceImpl)

Services Offered by a Smart Camera NodeServices offered for a monitoring application based on face detection:

14

Sample WSDL file (WinMax Service).....

15Figure: Smart Camera Architecture

Start/Stop

SS

AP

IC

hild

Se

rvic

eIm

pl

Ch

ildS

erv

ice

Imp

lW

ebS

ervi

ceIm

pl

Fa

ceD

ete

ctor

AP

I

Face DetectorImpl

ImportWebService

API

ImportWebService

API

OSGiContainer

D-OSGi

Web

Ser

vice

AP

I

Service API

Local Service

Web Service

Export API

Import API

Notations

Import/ExportDependancy

(Uses)C

hild

Clie

nt (

AP

I+ Im

pl)

Pa

ren

tClie

nt (

AP

I+Im

pl)

Import/ExportDependancy(Implements)

Proposed Smart Camera Architecture

WS

LS

LS

LS

WS - Web ServicesLS - Local Service

16

We

bS

erv

ice

Imp

l

We

bS

erv

ice

AP

I

Fa

ceD

ete

cto

rA

PI

FaceDetectorImpl

Ch

ildC

lien

t

Pa

ren

tClie

nt

OSGiContainer

D-OSGi

We

nS

erv

ice

Imp

l

We

bS

erv

ice

AP

I

Fa

ceD

ete

cto

rA

PI

FaceDetectorImpl

Ch

ildC

lien

t

Pa

ren

tClie

nt

OSGiContainer

D-OSGi

PARENT NODE CHILD NODE

1.1. WebService Request

1.2. Dummy Acknowledgement

2.1. WebService Results

2.2. Confirmation

Figure: Producer/Consumer Asynchronous Communication Architecture

Producer/Consumer Pattern for Asynchronous Communication

Asynchronous Service Invoke

Dummy Reply

17

Data Aggregation through Web Service Composition

Query

QueryQuery

QueryQuery Query

SELECT MAX(node_id, num_faces, epoch)FROM CAMERAS SAMPLE PERIOD 5s FOR 10s

Aggregate Query:

0

1 2

3 4 5

<0,10,1>, 0<0, 6,2>, 0<0,13,3>, 1

Max =10Max =13

<1,8,1>, 0Max =8<1,6,2>, 0<1,7,3>, 1

<3,5,1>,0Max =5<3,4,2>,0<3,20,3>,1

Max =20 <4,6,1>, 0 Max =6<4,4,2>, 0<4,4,3>, 1

<2,12,1>,0 Max =12<2,10,2>,0<2,12,3>,1

<5,3,1>, 0 Max =3<5,3,2>, 0<5,2,3>, 1

Max=<3, 20,3>, 1 Max= <4,5,1>, 1Max= <5,3,2>, 1

Max= <3,20,3>,1 Max=<2,12,1>, 1

Max= <3,20,3>

18

CHILD CAMERA NODE (Node ID: 2)

Start/Stop

SS

AP

IC

hild

Ser

vice

Impl

Chi

ldS

ervi

ceIm

plW

ebS

ervi

ceIm

plF

ace

Det

ecto

rAP

I

Face DetectorImpl

ImportWebServiceA

PI

ImportWebService

API

OSGiContainer

D-OSGi

Web

Ser

vice

AP

I

Chi

ldC

lien

t (A

PI+

Imp

l)

Par

entC

lient

(A

PI+

Impl

)

PARENT CAMERA NODE (Node ID: 1)

Start/StopS

SA

PI

Chi

ldS

ervi

ceIm

plC

hild

Ser

vice

Impl

Max

Ser

vice

Impl

Fa

ceD

etec

torA

PI

Face DetectorImpl

ImportWebServiceA

PI

ImportWebService

API

OSGiContainer

D-OSGi

Max

Ser

vice

AP

I

Chi

ldC

lien

t (A

PI+

Imp

l)

Par

entC

lient

(A

PI+

Impl

)

Data Aggregation through Web Service Composition [Contd.]

1. MaxService.getData(sample period, duration, parentID)

Figure: Example Web Service Composition

2. FaceDetector.getData()

2.MaxService.putData()

2. MaxService.getData(sample period, duration, 1)

3.MaxService.putData()

19

SQL Query Primitives For Smart Camera Network

Basic Select Query

Primitive

Aggregate Query Primitive Temporal Aggregate Query Primitive

SyntaxSELECT nodeId,numFaces,epoch

FROM Cameras

SAMPLE PERIOD 5s FOR 20s

SyntaxSELECT MAX(nodeId,numFaces,epoch)

FROM Cameras

SAMPLE PERIOD 5s FOR 20s

SyntaxSELECT WINMAX(nodeId,numFaces,step) FROM Cameras

SAMPLE PERIOD 1s FOR 10s RES 2s

WIN SIZE 5s

Demo –Single Node Demo –Single Node Demo –Single Node

0s 5s 10s 15s 20s

CS

CS

CS

CS

CS

C - CollectS - Send

0s 5s 10s 15s 20s

C C C C CS

•SQL Query Primitives for Smart Camera Network:

0 1 2 3 4 5 6 7 8 9 10

C C CS

CS

CS

CS

CS

C C C C

20

Dynamic Loading/Unloading of Services in Smart Camera Network

• Problem:– In a resource constrained

environment it is a waste of energy to run all available services

• Solution:– Dynamically load/start

relevant services when user issues a query

– Dynamically unload/stop started services once user gets results

– Saves energy of services running and listening to ports all the time

– Maximize network lifetime

Start Identified Services

Execute Query

Send Results

Stop Started Services

Identify Services Relevant to the Query

21Figure: Dynamic Loading of Services

Start/Stop

SS

AP

IC

hild

Se

rvic

eIm

pl

Ch

ildS

erv

ice

Imp

lW

eb

Se

rvic

eIm

pl

Lo

ad

Un

loa

dS

erv

ice

AP

I

LoadUnloadServiceImpl

ImportWebService

API

ImportWebService

API

OSGiContainer

D-OSGi

We

bS

erv

ice

AP

I

Ch

ildC

lien

t (A

PI+

Im

pl)

Pa

ren

tClie

nt

(AP

I+Im

pl)

Dynamic Loading of Services in Smart Camera Network

1.LoadService(ServiceAPI, ServiceImpl)

3. finishedLoadServiceConfirmation()

3.finishedLoadSerViceConfirmation()

2.LoadService(ServiceAPI, ServiceImpl)

2. LoadService(ServiceAPI, ServiceImpl)

2. finishedLoadSerViceConfirmation()

22

Implementation and Integration

Bundle Development Eclipse Plug-in Development Environment

Root Node Container Eclipse Equinox OSGi Container

Camera Node Container Apache Felix OSGi Container

HTTP Server Jetty

Web Service Development Environment Apache cxf-dosgi D-OSGi

Face Detection Algorithm Viola Jones Face Detection Algorithm

System Setup Environment

Network Architecture Hierarchical

Network Size Depth from root (max 3), No of nodes (max 15)

Average Degree of Camera Node 2 -3

Smart Camera Network Parameters

23

Result and Analysis

• Performance metrics:

– Turnaround Time• Time taken between the query issue at root and the arrival of the last response

message for that query back to the root. • Measures the communication cost in terms of time.

– Bytes Transferred• Total bytes transferred within the network for a given query. • Measures the communication cost in terms of bandwidth consumption.

– Service Startup Time• Time taken between the service startup request triggered by the root node and

the arrival of the confirmation of successful startup of all camera nodes in the network at the root.

– Memory Consumption• Memory consumption of a single camera node as a result of running services.

24

Results and Analysis [Contd.]

• Turnaround Time – SELECT

• Duration =20s, Sample interval =1s,2s,3s,4s,5s , Depth = 1,2,3

• Turnaround time increases with a lower sample interval

• Increase is more apparent in larger network sizes

• Turnaround Time – MAX

• Same parameters as previous

• Same pattern of behaviour observed for SELECT

• Turnaround time is less for MAX than for SELECT ( due to in-node processing)

• Turnaround Time – WINMAX

• Duration=20s, Sample interval=1s, Response time=1s,2s,3s,4s,5s, Depth =1,2,3

• Higher turnaround times when response interval is lowest

Turnaround Time - SELECT

20000

40000

60000

80000

100000

120000

1 2 3depth from root

turn

arou

nd

time

(ms)

Sample Interval:1s

Sample Interval:2s

Sample Interval:3s

Sample Interval:4s

Sample Interval:5s

Turnaround Time - MAX

2000030000400005000060000700008000090000

1 2 3

depth from root

turn

aro

un

d ti

me

(m

s)

Sample Interval:1s

Sample Interval:2s

Sample Interval:3s

Sample Interval:4s

Sample Interval:5s

Turnaround Time - WINMAX

20000

40000

60000

80000

100000

1 2 3

depth from root

turn

aro

un

d t

ime

(m

s)

Response Interval:1s

Response Interval:2s

Response Interval:3s

Response Interval:4s

Response Interval:5s

25

Total Bytes Transferred

0

1000

2000

3000

4000

5000

6000

7000

Flat Network (Data centricrouting)

Hierarchical Network(hierarchical aggregation)

Byt

es t

ran

sfer

red

per

ep

och

55.40% Improvem

Memory Consumption of Web Services

44000

44500

45000

45500

46000

46500

47000

1 2 3 4 5 6 7 8

Number of Web Services Running

Mem

ory

use

d (

KB

)

1.67% 2.00%2.39%

2.84%3.28%

4.45% 4.56%

Results and Analysis [Contd.]

Service Starup Time

0

2000

4000

6000

8000

10000

12000

1 2 3

Depth from root

Sta

rtu

p t

ime

(ms)

Memory Consumption of Local Services

20000

25000

30000

35000

40000

45000

50000

1 2 3 4

Number of Local Service Running

Mem

ory

Use

d (

KB

)

37.22% 38.53%

72.88%

0 1 2 3

• Total Bytes Transferred – MAX

• Duration =20s, Sample rate =1s Depth = 3

• 55.40% reduction in total bytes transferred

• Service Startup Time

• Depth =1,2,3

• More nodes resulted in higher startup times

• Memory Consumption – Local Services

• Single node, number of local services running=0,1,2,3

• OSGi container consumes 25224 kilo Bytes without any services running.

• Memory consumption increase w.r.t deployed local services

• Memory Consumption – Web Services

• Single node, number of web services running=

1-8

• Memory consumption increase w.r.t deployed web services

26

Related WorkData Aggregation Web Service Modeling SQL API

Network architecture plays an important role in aggregation performance

Flat Network-Excessive communication-Ex: SPIN

Hierarchical Network-Better than flat network-Ex: LEACH (cluster aggregation)-EADAT (tree aggregation)

DSC Networks

We have not found any data aggregation strategies proposed for complex data such as videos or images for smart camera networks

We provide image data aggregation using hierarchical network

Fundamental way of offering extensible and scalable service

Ex: More

DSC Networks

Neither web service modeling nor composition has been studied for DSC networks to date.

In this thesis work we provide both of these for a DSC network for the first time

Adaptations of SQL queries for traditional computer networks

Ex: TinyDB, Couger Approach

DSC Networks

No prior work has addressed this issue for Smart Camera Networks

We provide basic SQL Query API primitives for the first time for image data analysis

27

Conclusion

• In this thesis, we develop a novel SOA based middleware framework on a DSC network that uses Distributed OSGi to expose DSC network services as web services.

• We also develop a novel web service composition scheme that aid in data aggregation and a SQL query interface for DSC networks that allow sophisticated query processing.

• Future Work• Event based queries, network health queries • Resource monitoring algorithms and queries• Collaborative Decision Making• New algorithms for on-the-fly creation of the hierarchical

aggregation tree.

28

Bibliography

• Constantin Timm, Jens Schmutzler Peter Marwedel, Christian Wietfeld, Dynamic Web Service Orchestration applied to the Device Profile for Web Services in Hierarchical Networks, 2009.

• Samuel Madden , Michael J. Franklin , Joseph M. Hellerstein , Wei Hong ,TAG: a Tiny AGgregation service for ad-hoc sensor networks, 5th Symposium on Operating Systems Design and Implementation, 2002

• Rinner, B.; Jovanovic, M.; Quaritsch, M., Inst. for Tech. Inf., Graz Univ. of Technol.Embedded Middleware on Distributed Smart Cameras, Acoustics, Speech and Signal Processing,ICASSP 2007.

• Samuel R. Madden et al, Tinydb: An Acquisitional Query Processing System For Sensor Networks, 2005.

29

Questions/Comments?