Computer Science

Computer Science @ Boston University

SensoriumManaging Spatio-Temporal Objects in Video Sensor Networks

NSF 2003 RI/MII Workshop

Azer Bestavros, PI/PD

August 18, 2003

Computer Science

BU/CS: By the NumbersUniversity

Founded in 1839 and Chartered in 1869 as independent, private institution17 Schools and Colleges + 57 Centers and Institutes3,300+ tenured or tenure-track faculty members15,000+ Undergraduates, 10,000+ Graduates, and 230,000+ AlumniMetro campus over 132 acres, 346 buildings, 17 Libraries, 11,000+ dorm rooms $670M Endowment, $1.2B annual operating budget, and $2.3B in total assets$250M in grants and contracts

DepartmentFounded in 1981 (out of the Math Dept) in the College of Arts and Sciences17 tenured/tenure-track faculty members + 4 adjunts + 2 instructors4 administrative staff + 3 system administrators325 Undergraduates, 92 MA and PhD students, and 1,500+ alumni$2.2M in sponsored research in AY’01; $4.8M in AY’0210 NSF CAREERs + 5 ITRs + 2 ANI SPNs + 2 ONR + GAANN + …Nine (> 50%) of faculty rank in top 1% of most cited CS authors on Citeseer

Computer Science

What is a Sensorium?Sensorium:A common space equipped with video sensors (VS) for ubiquitous recognition and tracking of activities therein

Infrastructure:Range of VS ElementsProgrammable VS NetworkBackend compute engines Backend TByte storageMobile/wireless query unitsResearch Engineer

Computer Science

Why Acquire a Sensorium? The proliferation of networked, embedded, and mobile digital video sensors requires a paradigm shift in many areas of CS to address:

1. The unique spatio-temporal aspects of sensory (video) data acquisition, processing, representation, communication, storage,real-time indexing/retrieval, data mining

2. The challenges of Quality of Service (QoS) management and coordinated resource arbitration of sensory networks, which are both embedded and mobile

The other extreme in sensor networks research!

Computer Science

Sensoria: DeploymentAssistive Environments

e.g. for home/hospice/elder care/…

Safety Monitoringe.g. in factories/pre-schools/hospitals/…

Intelligent Spacese.g. for classrooms/meeting rooms/theaters/farms…

Secure Facilities and Homeland Security Usese.g. at airports/embassies/prisons/…

People Flow/Activity Studiese.g. at retail stores/museums/…

Computer Science

Driver App: Video-based HCIGoals:

Increase human/machine communication bandwidth Make HCI more natural for humansAssist people with disabilities — Universal Access

“What you look at is what you get” (Jacobs ‘93)Use gaze direction

Midas Touch Problem: Everything user looks at is activatedIntentional fixations versus saccadic motionSelection: blink, dwell-time 200ms, button-press combinationsPassive observation of user

Computer Science

HCI for the Severely Disabled: The CameraMouse

§ Supported by NSF CAREER Award and by other NSF and Whitaker Foundation grants

Computer Science

Video HCI: The CameraMouse

§ Supported by NSF CAREER Award and by other NSF and Whitaker Foundation grants

Computer Science

Video HCI: The “Blink Link”Goal: Eye blink detection and classification of blinks as voluntary or involuntary in real time

Applications: HCI for users with disabilitiesIntelligent vehicle systems detecting drowsy driversHCI systems requiring awareness of visual cues

§ Supported by NSF CAREER Award and by other NSF and Whitaker Foundation grants

Computer Science

Video HCI: Eyebrow ClickerGoal: Detect eyebrow motion and interpret raised eyebrows as a selection (“mouse click”) command

Requirements: Self-initializing, self-correcting, real-time process

§ Supported by NSF CAREER Award and by other NSF and Whitaker Foundation grants

Computer Science

Video HCI: ASL Interpretation

Face Close-up

Stereo Pair

Upper Body

Side View

Goals:(1) Collect/annotate video DB of simultaneous ASL streams(2) Automatic annotation of head, face, body and hand gestures(3) Algorithms for indexing/retrieval of ASL video(4) Linguistic research regarding non-manual cues in ASL

§ Supported by 3 NSF grants and an ONR grant

Computer Science

Facial Feature Tracking and Occlusion Recovery for ASL

Goal: Detect, track, and interpret facial features in real time and recover from occlusion. Why?

Facial features complement ASL interpretation- Raised eyebrows: yes/no or rhetorical question- Furrowed eyebrows: wh-question (who, where, …)- Headshake: negation

§ Supported by 3 NSF grants and an ONR grant

Computer Science

HCI: The Next Generation“What you see (on the screen) is what you get”

“What you look at (on the screen) is what you get”

“What you look at (in the world) is what you get”

Many teams are working towards HCI++Our goal is:

To develop systems/networking video HCI++ service supportTo adapt/re-engineer vision applications to use these servicesTo tackle challenges of freeing users from slavery to I/O seats

Computer Science

Taking HCI to the Next LevelRealityWeb

Bridging the divide between

cyber & physical spaces should

allow interaction with physical

spaces!

Computer Science

Challenges: VSE Handoff

Fuse

Handoff

RealityWeb

Connection handoff among

sensors is necessary for

tracking mobile objects

Computer Science

Challenges: In-Net Processing

Fuse Fuse

RealityWeb

Fuse multiple video streams into a single model of the object and its

motion

Computer Science

Aggregation and SketchingApproximate Aggregation for Sensor Networks

Use efficient (log-space) in-network aggregation to tolerate limited capacity, range and failuresExtends [Flajolet & Martin:85] counting sketches to handle “sum” (and mean, variance, …)Use smart flooding; if any path works, data is included in aggregate

Dealing with Motion in Monitored SpacesEfficient (log n) approximate counting of moving objects without tracking them individuallyUsing duplicate sketches allow ST indexing techniques to be applied efficientlyResulting working set and actual database sizes are smaller (highway vs parking lot)

§ Supported by 2 NSF CAREER Awards and a NSF ANI special projects grant

Computer Science

Challenges: Resource Mgmt

CC

Sched

RealityWeb

Control and schedule VS resources for efficient and stable QoS-

aware tracking

Computer Science

Tracking Tasks: QoS Needs

15-23

2-34

1-4

2-22

1-32

Time (msec)

50

200

100

66

33

Period (msec)

QoS(%)Task

70%Lane Detection

90%Template Matching

90%Cropping Template

80%Feature Search

85%Searching for Cars

Real App NeedsDifferent periodsHighly-variable periodic exec timesVaried QoS

Computer Science

Spatio-Temporal SchedulingProblem

“p1 must execute on (r1 & r3) or on (r1 & r4) …” where pi are tasks & rj are camerasp1 must meet 90% of its deadlines (or miss no more than 2 in a row)

SolutionUse techniques from computational geometry, optimization & RT scheduling at three timescales:

Offline: Recursively partition the space into octreePeriodically: Assign activities to octree leaves and complete octree to optimize mapping of pi to rjOnline: Use Statistical RMS (or similar) to schedule for individual resources

90%

90%

80%

85%

QoS(%)

2-34

1-4

2-22

1-32

Time (msec)

200

100

66

33

Period (msec) WhereTask

VSE 3,4

Template Matching

VSE 3,4

Cropping Template

VSE 1,2,3,4

Feature Extraction

VSE 1,2,3,4

Searching for Person

90%

90%

80%

85%

QoS(%)

2-34

1-4

2-22

1-32

Time (msec)

200

100

66

33

Period (msec) WhereTask

VSE 3,4

Template Matching

VSE 3,4

Cropping Template

VSE 1,2,3,4

Feature Extraction

VSE 1,2,3,4

Searching for Person

§ Supported by NSF CCR and ITR Awards

Computer Science

QoS-aware Network ControlResourcesStreams Classes ResourcesStreams Classes ResourcesStreams Classes

How to adjust network controls to manage QoS while ensuring

efficiency and stability?

Model sharing of PID controllers)]()[()( 11 −− −−−+−+= kk

ik

iki

ki qBqBqBww βα

∑=

−− −+=N

i

ki

kk Cwqq1

11

Use Control theory to evaluate stability/QoS

)(max),(max#:

iiii

BCNββαα = =

buffers allocated : capacity, allocated : flows, competing

N4

< 2 + βα NwCwwN

jjisi /)()(

1

00 ∑=

−+=

§ Supported by 2 NSF ANI special projects grants

Computer Science

Central Research QuestionsComputer Vision:

What new tools are needed for estimating and encoding humans and their motion at multiple levels of detail, and for determining correspondence of the same person seen in different camera views?

Databases:What new data indexing, access and mining methods are needed for spatio-temporal data?

Computer Science

Central Research QuestionsNetworks:

What network protocols are needed for efficient transport of sensor data subject to quality of service constraints imposed by mobile clients?

Real-time/Embedded Operating Systems:What new resource management and scheduling algorithms are needed to efficiently satisfy statistical guarantees in a highly dynamic environment?

Computer Science

Central Research QuestionsProgramming Systems:

What compiler-time and run-time support could be provided to facilitate the development of efficient and safe Sensorium services and applications?

Theory and Algorithms:What algorithmic and optimization techniques can be brought to bear to support Sensorium services and applications?

Computer Science

Sensorium: CollaborationsSensorium Infrastructure

Bestavros BetkeWest

Matta Byers

Itkis Kollios SclaroffCrovella

Homer

Kfoury

Snyder

Reyzin Xi

Teng

RTMulticast

MASSServers

RTTracking

ChurchXanadu

DionysysSafeX

DWCSSRMS

ImageSearch

ShapeMorph

Surge BriteGismo

SmartCards

S-T Indexing

ShapeShifter

SecureMulticast

ScalableWeb

ExtensibleOS Services

ResilientCrypto

ProgrammingWith Flows

MotionMining

AmericanSign Lang

Traffic Managers

DatabasesNetworks Security Vision SoftwareSystems

Gacs

Levin

Computer Science

Back-End SystemA Terabyte storage system for efficient indexing, retrieval, and mining

Tape backup system

PC-based database engines for online indexing, real-time query services, and data mining

High-performance compute engines for real-time service monitoring, arbitration, and coordination to satisfy spatio-temporal constraints

A Terabyte storage system for efficient indexing, retrieval, and mining

Tape backup system

PC-based database engines for online indexing, real-time query services, and data mining

High-performance compute engines for real-time service monitoring, arbitration, and coordination to satisfy spatio-temporal constraints

Computer Science

Distributed Video Sensing Elements (VSE’s)

Independent, wall-mounted pan/tilt/zoom video cameras provide coverage of spatial observable domains

Dual-processor PC-based engines for tracking, prediction, and coding of moving objects

Gigabit ethernet to provide VSE LAN segment and fast connection to database and compute engines in Backend System

Independent, wall-mounted pan/tilt/zoom video cameras provide coverage of spatial observable domains

Dual-processor PC-based engines for tracking, prediction, and coding of moving objects

Gigabit ethernet to provide VSE LAN segment and fast connection to database and compute engines in Backend System

Computer Science

Reconfigurable, Programmable Network

PC-class programmable routers for experiments in routing and processing functionality for sensory data

PC-class programmable routers for experiments in routing and processing functionality for sensory data

Computer Science

Mobile Network

Mobile access pointsforming ad hoc networked clusters to retrieve, monitor and track objects in spatial domains

Mobile query units allowing direct queries to the Sensorium

Mobile access pointsforming ad hoc networked clusters to retrieve, monitor and track objects in spatial domains

Mobile query units allowing direct queries to the Sensorium

Computer Science

HCI++: Putting it All Together

Computer Science

A Fantastic Outreach Tool!

Computer Science

Computer Science @ Boston University

SensoriumManaging Spatio-Temporal Objects in Video Sensor Networks

NSF 2003 RI/MII Workshop

Azer Bestavros, PI/PD

August 18, 2003

Computer Science

Project RequirementsRequested Equipment

Vision Services

Database Services

Network Services

OS ResourceManagement

Backend System • Database Engines • Compute Engines • Terabyte RAID + backup • Network monitoring

Video Sensing Elements • Video sensors • Video capture/computation PC's

Programmable Routers • PC-based routers • OmniView 16-port KVM switch • Gigabit ethernet

Mobile Access • Wireless network • Hand-held computers • Notebook computers