fjording the stream an architecture for queries over streaming sensor data

1

Fjording The StreamFjording The StreamAn Architecture for Queries over An Architecture for Queries over

Streaming Sensor DataStreaming Sensor Data

Samuel Madden, Michael Samuel Madden, Michael FranklinFranklin

UC BerkeleyUC Berkeley

Feb 28, 2002 Fjording ICDE 2

IntroductionIntroduction

Telegraph Sensor Query Processing Telegraph Sensor Query Processing ArchitectureArchitecture FjordsFjords

Enable push and pull in query plansEnable push and pull in query plans Operators for streaming dataOperators for streaming data Sensor proxySensor proxy

Sensor - Query MediatorSensor - Query MediatorQuery processor

Web

Sensors

User

Proxy


RoadmapRoadmap BackgroundBackground

SensorsSensors RequirementsRequirements Traffic ScenarioTraffic Scenario

FjordsFjords Continuous QueriesContinuous Queries Stream-sensitive operatorsStream-sensitive operators Sensor ProxiesSensor Proxies Querying a Sensor NetworkQuerying a Sensor Network ResultsResults

GraphsGraphs Related WorkRelated Work


Sensor ChallengesSensor Challenges

Battery Powered Battery Powered 2AA Cells (2800 mAh; 1.01x102AA Cells (2800 mAh; 1.01x104 4 J), Coin Cell J), Coin Cell

(100mAh)(100mAh) Communication Dominates Power CostCommunication Dominates Power Cost Can be exhausted - tens to hundreds of MBs of data Can be exhausted - tens to hundreds of MBs of data

/ sensor/ sensor WirelessWireless

High loss rates (20% @ 5meters , typical)High loss rates (20% @ 5meters , typical) Low bandwith (10kbps)Low bandwith (10kbps)

Near Real TimeNear Real Time Streaming DataStreaming Data

Pushed at (user defined) regular intervalsPushed at (user defined) regular intervals

TinyOS “Mote”TinyOS “Mote”


Requirements of Sensor Requirements of Sensor Query ProcessingQuery Processing

Power SensitivityPower Sensitivity Tolerance to unbounded streams of Tolerance to unbounded streams of

datadata Tolerance to pushTolerance to push Tolerance to intermittent, lossy Tolerance to intermittent, lossy

connectionsconnections Tolerance to failed sensorsTolerance to failed sensors


Traffic ScenarioTraffic Scenario

CA Department of Transportation (CalTrans) CA Department of Transportation (CalTrans) has sensors all over bay area freewayshas sensors all over bay area freeways Inductive loop sensors give speed, flow, vehicle sizeInductive loop sensors give speed, flow, vehicle size Motes could collect this data cheaply Motes could collect this data cheaply

Many possible queriesMany possible queries Commuters want to find congestion Commuters want to find congestion California Highway Patrol (CHP) wants to find California Highway Patrol (CHP) wants to find

accidentsaccidents


FjordsFjords

Query plan implementationQuery plan implementation Useful for streams and distributed environmentsUseful for streams and distributed environments

Combine push (streaming) data and pull Combine push (streaming) data and pull (static) data(static) data E.g. traffic sensors with CHP accident reportsE.g. traffic sensors with CHP accident reports


Push vs. PullPush vs. Pull

Problem: Need an API to combine push & Problem: Need an API to combine push & pullpull

Operators (e.g. join) data-direction agnosticOperators (e.g. join) data-direction agnostic Push vs. pull implemented in queues Push vs. pull implemented in queues

((connectorsconnectors)) Contrast withContrast with

Iterator modelIterator model Exchange operatorExchange operator

Allows arbitrary combinations of push and Allows arbitrary combinations of push and pullpull


Pull ExamplePull Example

OperatorOperatorQueue q;…Queue q;…

Tuple Tuple process()process() { {

Tuple t = q.Tuple t = q.get(),get(), outt = null; outt = null;

If (t != null) { If (t != null) {

<process t> <process t>

} else { … do something else … }} else { … do something else … }

return outt;return outt;

}}

Pull QueuePull QueueOperator parent, child;Operator parent, child;Tuple Tuple get()get() { { Tuple t = null;Tuple t = null; while (t == null) {while (t == null) { t = child.t = child.process()process();; }} return t;return t;}}

get()Pull

Connection

Scan

process()

Notice:

•Iterator semantics by making get() blocking

•Get() can return null

•Process() can return null


Push ExamplePush ExampleOperatorOperatorQueue q;…Queue q;…

Tuple Tuple process()process() { {

Tuple t = q.Tuple t = q.get()get(), outt = null;, outt = null;

If (t != null) { If (t != null) {

<process t> <process t>

} else { … do something else … }} else { … do something else … }

return outt;return outt;

}}

ThreadThreadwhile(true) {while(true) {

Tuple t = op.Tuple t = op.process()process();;

if (t != null) op.outq.if (t != null) op.outq.enqueue(t)enqueue(t);;

}}

Push QueuePush QueueOperator parent, child;Operator parent, child;Vector v = new Vector();Vector v = new Vector();Tuple Tuple get()get() { {

if (v.size() > 0) return v.removeFirst();if (v.size() > 0) return v.removeFirst();else return null;else return null;

}}Tuple Tuple enqueueenqueue(Tuple t) {(Tuple t) {

v.put(t);v.put(t);}}

Push Connection

Scan

get()


Fjord ExampleFjord Example

Push

Push

Pull

Query processor


Continuous QueriesContinuous Queries

Given user queries over current sensor dataGiven user queries over current sensor data Expect that many queries will be over the same data Expect that many queries will be over the same data

sources (e.g. traffic sensors)sources (e.g. traffic sensors) Queries over current data always looking at same Queries over current data always looking at same

tuplestuples Those queries can share Those queries can share

Current tuples Current tuples Work (e.g. selections)Work (e.g. selections)

Sharing reduces messages, thereby power!Sharing reduces messages, thereby power! A new query can be “folded” into an existing A new query can be “folded” into an existing

queryquery Use old instances of scans and selectionsUse old instances of scans and selections


Relational Operators And Relational Operators And StreamsStreams

Selection and Projection Apply NaturallySelection and Projection Apply Naturally No Blocking Operators No Blocking Operators

Sorts and aggregates over the entire streamSorts and aggregates over the entire stream Nested loops and sort-merge joinNested loops and sort-merge join

Windowed OperatorsWindowed Operators Sorts, aggregates, etc.Sorts, aggregates, etc.

Online, Interactive QP TechniquesOnline, Interactive QP Techniques In memory symmetric hash joinIn memory symmetric hash join Alternatives: ripple-join, Xjoin, etc.Alternatives: ripple-join, Xjoin, etc. Partial ResultsPartial Results


Query Processor

Sensor ProxiesSensor Proxies Mediate between Sensors and FjordsMediate between Sensors and Fjords

Push operators out to sensorsPush operators out to sensors Hide query processing, knowledge of multiple queries Hide query processing, knowledge of multiple queries

from sensorsfrom sensors Hide details of sensors from query processorHide details of sensors from query processor Enable power-sensitivityEnable power-sensitivity

Sensor Proxy

Operators

Query

Query

Parsed Queries[sources, ops][fields, filters,

aggregates, rates]

[tuples]

Registration


What runs where?What runs where?

(Multi-query) Distributed Optimization (Multi-query) Distributed Optimization Challenge:Challenge: Given set of operators, proxy must choose:Given set of operators, proxy must choose:

Run on sensor or,Run on sensor or, Run on local query processorRun on local query processor

Running on sensors saves powerRunning on sensors saves power Simple computations cheaper than messagesSimple computations cheaper than messages Selection, aggregation reduce communication costSelection, aggregation reduce communication cost

Sensors have limited resourcesSensors have limited resources All queries can’t run in all sensors simultaneouslyAll queries can’t run in all sensors simultaneously Limited state precludes big joins or lots of groupsLimited state precludes big joins or lots of groups

Queries share operatorsQueries share operators Operators vary in selectivityOperators vary in selectivity


Building a QueryBuilding a Query How to translate a declarative query into a Fjord?How to translate a declarative query into a Fjord? Just like traditional query processing, except:Just like traditional query processing, except:

Branches originating in sensors connected by push connectorsBranches originating in sensors connected by push connectors Sensor proxy handles scans, selections over sensorsSensor proxy handles scans, selections over sensors

Proxy delivers tuples from sensorsProxy delivers tuples from sensors Proxy pushes-down selections transparently Proxy pushes-down selections transparently

Output of join is push if one or both inputs is pushOutput of join is push if one or both inputs is push Join carefully chosenJoin carefully chosen

Sensor Proxyaccident = true

Pull: Data Request

Data Request

Data

Index NL Join

Push PushSpeed < 30


Query & FjordQuery & Fjord

Simple Test Query:Simple Test Query:

SELECT AVG(s.speed, SELECT AVG(s.speed, ww))

FROM sensors AS sFROM sensors AS s

WHERE s.loc in WHERE s.loc in {{userLocsuserLocs}}

Loc {userLocs}

BHL Server

Telegraph Server

Average


Multiple Queries in a FjordMultiple Queries in a Fjord

Loc {userLocs}

BHL Server

Telegraph Server

Loc {userLocs}Loc {userLocs}


Fjord Performance Fjord Performance (In Telegraph)(In Telegraph)

Running Time Per Query

0

500

1000

1500

2000

2500

3000

3500

4000

1 2 5 10

Number of Queries

Runnin

g T

ime (

seconds)

Single Fjord Multiple Fjords


Sensor Proxy for TrafficSensor Proxy for Traffic Measure benefit of pushing Measure benefit of pushing

computation into sensors; in this computation into sensors; in this case, vehicle identification.case, vehicle identification. Simple “aggregation” dramatically Simple “aggregation” dramatically

reduces power costsreduces power costsw

w w

t0: Vehicle Covers Su t1: Vehicle Covers Su and Sd

t2: Vehicle Covers Sd t3: Vehicle Covers Neither Sensor


Pushing Aggregates Saves Pushing Aggregates Saves PowerPower

Atmel (TinyOS CPU) Simulator

100 samples / sec5 vehicles / sec

7x power savings

Power Drain (W) vs. Sample Method

00.0010.0020.0030.0040.0050.0060.0070.008

Every Sample Every Vehicle

Sampling Method

Pow

er (W

)


Related WorkRelated Work

CougarCougar Interactive & Adaptive Query Interactive & Adaptive Query

ProcessorsProcessors Tukwila, Xjoin, Eddy, CONTROLTukwila, Xjoin, Eddy, CONTROL

Continuous Query ProcessorsContinuous Query Processors NiagaraCQ, Xfilter, CACQNiagaraCQ, Xfilter, CACQ

Directed DiffusionDirected Diffusion Volcano / Exchange OperatorVolcano / Exchange Operator Temporal DatabasesTemporal Databases


Conclusions & Future WorkConclusions & Future Work Required for sensor query processing:Required for sensor query processing:

Fjords : API for combining push & pullFjords : API for combining push & pull Sensor Proxy: Mediator between sensors and QPSensor Proxy: Mediator between sensors and QP Streaming OperatorsStreaming Operators

Big benefit from pushing selections, aggregates Big benefit from pushing selections, aggregates into networkinto network

Combining multiple queries is a winCombining multiple queries is a win Extensions & Future Work:Extensions & Future Work:

Multi-query optimization & adaptivity (CACQ, SIGMOD Multi-query optimization & adaptivity (CACQ, SIGMOD 2002)2002)

Push down selections & aggregates (Submitted to VLDB Push down selections & aggregates (Submitted to VLDB 2002)2002)

Sensor proxy policiesSensor proxy policies

fjording the stream an architecture for queries over streaming sensor data

Documents