Smart Vision System:Design of Application Algorithm-ArchitectureAlgorithm-Architecture

Eric Delfosse

IBBT-NES-IMEC

What are Smart Vision Systems?

� Systems that embed intelligence through advanced image processing to:

� Enhance visual user experience

� Improve natural interaction

� Facilitate decision making for complex events

� …

Enhance visual user experience

� Multi-camera 3D image reconstruction for advanced surveillance

� 3D GPS� 3D GPS(IBBT GBO ISYSS)

(IBBT GBO URBAN)

Increase natural interaction

� Natural human-machine interface through gesture recognition

(IBBT GBO Hi-Masquerade)� 3D (immersive) video

conferencing

Facilitate decision making for complex events

� Event detection for surveillance applications

(IBBT GBO ISYSS)

� Traffic sign recognition for driver assistance

(IBBT GBO URBAN)

Different applications, different requirements

AccuracyReduce false negatives and false positives

ThroughputHigh resolutionHigh framerate

Functionality

Low PowerSolar energy

Avoid active cooling...

…Size

Cost

These applications require increasingly complex algorithms

Exponential algorithmic complexity increase

1500

2000

2500

3000

Co

mp

lex

ity

(O

ps

/pix

el)

> 9000

0

500

1000

Ed

ge

de

tectio

n

AV

C e

nco

din

g

Ima

ge

en

ha

nce

me

nt

Fe

atu

re

extr

actio

n

De

pth

extr

actio

n

Ob

ject

de

tectio

n

Ima

ge

un

de

rsta

nd

ing

Algorithm

Co

mp

lex

ity

(O

ps

/pix

el)

Diversity of platform architectures with different characteristics

High performance

High power

Limited portability (laptop)

PC + GPU

Server rack

Very high performance

Very high Power

Non-portable

…

Limited portability (laptop)

…

Medium performance

Low power

High portability

…

Embedded systems

Smart Vision Systems design =Matching Application – Algorithm - Architecture

Application

(Requirements)

Architecture

(Constraints)

Algorithm

(Complexity)

Smart Vision Systems design =Matching Application – Algorithm - Architecture

Application

(Requirements)

Architecture

(Constraints)

Algorithm

(Complexity)

Smart Vision Systems design =Matching Application – Algorithm - Architecture

Application

(Requirements)

Architecture

(Constraints)

Algorithm

(Complexity)Introduce parallelism

Processor optimizations

Complexity - Quality trade-off: reduce complexity with limited (visual) quality loss

Matching Algorithm and Architecture: the DCT example

f0

f1

f2

f3

f4

f5

F0

F1

F2

F3

F4

F5

f0

f1

f2

f3

f4

f5

F0

F1

F2

F3

F4

F5

f6

f7

F6

F7

f6

f7

F6

F7

Classical DCT Butterfly DCT

64 multiplications

64 additions

High regularity

Ideal for:

20 multiplications

26 additions

Low regularity

Ideal for:

Smart Vision Systems design =Matching Application – Algorithm - Architecture

Application

(Requirements)

Companies

Architecture

(Constraints)

Algorithm

(Complexity)

IBBT

Conclusions

� Smart Vision Systems:� Enable new applications

� Require new complex algorithms

� Use diverse platform architectures

� Successful design requires competences on these 3 aspectsaspects

� IBBT brings these competences together

Demo’sDemo’s

Demo 1: 3D video through real-time viewpoint interpolation

Viewpoint interpolation:a) stereo capturingb) depth extractionc) interpolation

Autostereoscopic displays:require multiple

(interpolated) views

Demo 2: Eye-gaze corrected video chatting

Webcam

ComputerDisplay

Demo 3: Novel 3D Camera prototype and monitoring application in elderly environment