administrative details · 2017. 11. 6. · • start: 1/10/2015, project duration: 2 year eavise:...

19/11/2015

1

IWT-Tetra project

User Group Kickoff meeting19 November 2015

Program

• 12:00 Reception with sandwiches• 13:00 Welcome and presentation of research team• 13:15 presentation from IWT: what is a tetra project?• 13:30 Project idea and planning• 14:00 Research carried out by Vision sub-team and

outlook• 14:30 Research carried out by AI sub-team and outlook• 15:00 Open discussion on project scope and possible

application cases• 16:00 Wrap-up and administrative aspects

Administrative details- Tetra project: TEchnologieTRAnsfer

- Goal: transfer of knowledge/technology to industry- IWT 150165:

VIPER – Visual Person Detection made Reliable- Research carried out by knowledge partner

- KU Leuven – EAVISEsub-groups Vision and AI

- Industrial users commission - Advising and steering project- First access to project results

- 92,5% financed by IWT- Rest (7,5%) via co-financing users commission

• Start: 1/10/2015, project duration: 2 year

EAVISE:Embedded & Artificially intelligent Vision Engineering

Toon Goedemé

Joost Vennekens

Valley of death

industry

Embedded & Artificially Intelligent Vision Engineering• Research goal:

o Translating state-of-the-art image processingalgorithms and artificial intelligence techniques tosolutions for industry-specific application problems

o Optimizing vision algorithms to real-time performanceo Increasing robustness of experimental algorithms to

industry standardso Implementing advanced image processing

applications on embedded systems :FPGA, DSP, GPU, multicore CPU, cluster

19/11/2015

2

People@EAVISE

Prof. Toon Goedeméresearch leaderComputer Vision

Prof. Joost Vennekensresearch leader

AI & KR

Kristof Van Beeckblind spot cam

Wim Abbeloos3D bin picking

Dries Hulensrobotic UAV

Sander BeckersActual causality

Stijn De BeugherEyetracking

Steven Puttemansobject detection

Wiebe Van RanstPerson detection startup

Bram AertsKR for cinematography

Sander Grielensrobot-vision calibration

Andy WarrensIR person detection

Shana Van Desseltimetabling

Kristof Van EngelandAI surveillance

Floris De SmedtPerson detection startup

Inge Coudronvisual navigation

STARTUP

AIVISION

People@EAVISE



AI & KR















VIPER

STARTUP

People@EAVISE



AI & KR















Exampleprojects

EAVISE: some statistics• Personnel:

o 2 research leaderso 11 internal researchers

• 30+ projects since 2008:o 9 IWT-tetrao 10 KMO-portefeuilleo 9 IWT O&Oo 1 KUL-GOA, 1 IWT-SB

• 84 international publications since 2008o 9 book chapterso 12 journal articleso 55 conference papers

• 18 publications in 2015• Awards:

o Best paper award at CVPR Embedded Vision Workshop, 2015o Best poster award at TC ESAT/CW Research Day, 2015

(potential) industrial users group

19/11/2015

3

Program




Gebruikersgroep VIPER

rol en werking van de gebruikersgroepTom Heiremans – 19/11/2015

Doelstelling Collectieve programma’sDe belangrijkste doelstelling van onze collectieve programma’s is het verhogen van de kennis en competitiviteit van bedrijven. De resultaten die voortvloeien uit collectieve projecten moeten op korte of lange termijn leiden tot een economische meerwaarde bij de bedrijven.

TETRA

= TETRA project

= voor TETRA project

= na TETRA project

Rol van de Gebruikersgroep

19/11/2015

4

Rol van de Gebruikersgroep

• Bevestigen valorisatiepotentieel van het project voor hun bedrijf of organisatie via cofinanciering en valorisatie-intenties

• Ondersteunen projectaanvragers � bij het opmaken van de aanvraag; werkplan; … (input)� nemen actief deel aan gebruikersgroep � hebben een actieve rol bij valorisatie-activiteiten

• Zijn een klankbord voor de projectuitvoerders• Verwerven kennis tijdens het project en zorgen dat projectresultaten

achteraf nuttig kunnen ingezet worden in hun onderneming of een andere onderneming

Rol van de Gebruikersgroep• Eigenaar van de projectresultaten = projectconsortium

(hoofdaanvrager + partner(s))• Algemene inzichten � worden ruim verspreid• Economisch valoriseerbare resultaten� transfer van eigendoms- of gebruiksrechten (licenties)

- niet-exclusief- aan marktconforme voorwaarden- elk geïnteresseerd bedrijf/organisatie in de EU

• Leden van de gebruikersgroep - hebben geen preferente toegang tot de projectresultaten- kunnen bijdrage in cofinanciering in mindering brengen

- indien nodig (tijdelijke) geheimhoudingsovereenkomst

Rol van de GebruikersgroepOn-line bevraging van de leden van de gebruikersgroep<GebruikersPoll>

• IWT heeft een elektronische tool uitgewerkt voor de- unieke registratie van de leden van de gebruikersgroep- bevraging van de (aanwezige) leden na elke vergadering

• Het gaat in eerste instantie om de doelgroepbedrijven

• De bedoeling is om in overleg met de gebruikersgroep en binnen de doelstellingen, het project nog beter af te stemmen op de verwachtingen

• Feedback wordt besproken op de volgende vergadering

Rol van de Gebruikersgroep• In welke mate is het onderwerp van het project nog steeds relevant voor uw onderneming

{scoren op schaal (i) zeer relevant; (ii) relevant, (iii) neutraal, (iv) minder prioritair, (v) niet langer relevant }

• Hoe scoort u het projectverloop en de tot nu toe behaalde projectresultaten in functie van de te bereiken doelstellingen? { scoren op schaal (i) minder dan voorzien , (ii) zoals voorzien, (iii) beter dan voorzien

• Hoe tevreden bent u over de ruimte voor interactie en sturing door de ondernemingen (gebruikers) binnen het project?{ scoren op schaal : (i) zeer tevreden, (ii) tevreden, (iii) neutraal, (iv) ontevreden, (v) zeer ontvreden}

• Hoe tevreden bent u met de behandelde punten in de gebruikersgroepvergaderi ngen ? { scoren op schaal : (i) zeer tevreden, (ii) tevreden, (iii) neutraal, (iv) ontevreden, (v) zeer ontvreden; Indien (iv) of (v) korte toelichting geven}

• Hoe hoog scoort u de (mogelijke) toepassing van concrete, bruikbare resultaten bij uw eigen onderneming (of uw ledenbedrijven) op korte termijn ?[KT is tijdens of < 1 jr na afloop]{ scoren op schaal: (i) de resultaten worden al toegepast in de onderneming (ii) de resultaten zullen wellicht binnen het jaar na afloop van het project worden toegepast in de onderneming, (iii) de resultaten zijn tot op heden op kt niet bruikbaar voor de onderneming, maar mits bedrijfsspecifiek vervolgonderzoek wel nog mogelijk in de toekomst (iv) de resultaten blijken niet bruikbaar voor de onderneming }

AIO AlgemeenWat kan AIO nog voor u doen?

O&O-bedrijfs-projecten

& ICON

Ba

ek

ela

nd

ma

nd

ate

n

onderzoek ontwikkeling engineering

AIO (IWT) subsidies voor innovatie

ideeën-

generatie

bedrijfstype

O&O-

H-

studies

kmo-

innovatie-

projecten

kmo

grote

onderneming (of kmo met groot

project)

technologieverkenning

fase van innovatie

haalbaar-

heidsstudie

financiële steun voor bedrijven

projectkos

t

€ 10.000

€ 100.000

€ 1.000.000

Ba

ek

ela

nd

ma

nd

ate

n

kmo-haalbaarheid-

studies

19/11/2015

5

25

meer informatie

• via het VIN-netwerk : zie www.innovatienetwerk.be

• op onze website : www.iwt.be onder ‘subsidies’

• Kijk ook eens naar : http://www.iwt.be/nieuws/cases

• Website AIO: http://www.vlaio.be/

• rechtstreeks bij het IWT : voorbespreking ([email protected] of [email protected] )

Koning Albert II-laan 35, bus 16

B-1030 Brussel

Tel.: +32 (0)2 432 42 00

Fax.: +32 (0)2 432 43 99

E-mail: [email protected]

www.iwt.be

agentschap voor Innovatiedoor Wetenschap en Technologie

Contact: Tom [email protected] 432 43 04

Program




Project abstract• Camera-based safety and security

systems• Real-time reaction on incidents?

o Manual monitoringo Automatic processing and incident

detection• Needed components:

1. Very reliable detection of persons in camera images

2. Reasoning system that can decideif an alarm must be generated

Enabling factors

• State-of-the-art person detection algorithms show astonishing resultso Accuracy great on standard benchmark data setso EAVISE succeeded in running these in real-time on

limited hardware

o Both open source and commercial-grade implementations available

• Price of LWIR-cameras descends steeply, with increasing resolution

• Knowledge-representation based probabilistic reasoning offers potential to analyse each situation

Project idea

• Making people detection reliable, also in difficult circumstances (fog, smoke, rain, dust, motion blur, …):o Combine RGB and LWIR

camera

o Adapt state-of-the-art person detection algorithms for thissensor combination

• Use probabilistic KR for analysis of situation: must an alarm begenerated?

19/11/2015

6

Project goalso Developing a sensor combination and software for ultra-

reliably detecting people in real-timeo Composing a real-life reference image database for

evaluating person detection techniques in difficultcircumstances

o Studying techniques for automatic analysis of the observedsituation and classification as normal or abnormal

o Studying the certification procedure for camera-based safetyand security systems

o The demonstration and dissemination of the project resultsvia 5 real-life user cases

o Supporting industrial companies to adopt the developedtechniques in their products and services

Work packagesWP1: Hardware

WP4: Evaluation and dissemination

1.A Study on sensors

1.B Hardware imple-mentation & calibration

1.C Benchmark database

WP2: Person detection

2.A Study on algorithms

2.B Person detectionSW implementation

2.C Evaluation on Benchmark database

WP3: Alarm system

3.A Study on AI

3.B Learning of ranking

3.C Online learning

3.D Evaluation

4.A User Cases

4.B Evaluation and documentation

4.C Study on certification and legal aspects

4.D Broad dissimination

Planning

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8WP1.A: Study on sensorsWP1.B: Hardware realisation and calibrationWP1.C: Benchmark database MP8

WP2.A: Study on algoriths for person detectionWP2.B: Implementation algorithm person detection MP6

WP2.C: Evaluation on benchmark database MP7

WP3.A: Study on learning alarm systemWP3.B: Learning of rankingWP3.C: Online learningWP3.D: Evaluation on benchmark database MP7

WP4.A: User cases MP1 MP2 MP3 MP4 MP5

WP4.B: Evaluation and documentation MP7

WP4.C: Certification & legal aspects MP9

WP4.D: Broad dissemination and networking MP9

Program




Research Vision part

Overview:1. Person detection algorithms overview2. Results on RGB person detection3. IR camera market study4. How to combine RGB and IR for person detection?

What is pedestrian detection?

• Localize pedestrian appearances in an image/dataset• pedestrian detection pedestrian recognition

Floris

≠

19/11/2015

7

Pedestrian detection approach

37

• Create a model for pedestrianso Examples of positives (pedestrians)

o Examples of negatives (non-pedestrians)o Convert to feature representation

• Good distinction between pedestrians and background

• Robust for scene changes (e.g. illumination)

o Train a model• Machine Learning: Adaboost, Support Vector

Machines, Neural networks, …

• Distinction between “Pedestrian” and “Background”• Intra-class variation: pedestrians can have many

appearances

Pedestrian detection approach

At every location…and multiple scales � sliding window

38

Search the model in the image features (Sliding Win dow):

…

• Calculate features at multiple scales � Feature pyramid

• Similarity between the model and the features forms the certainty of a pedestrian at that location

• A threshold defines the boundary between “background” and “detection”

• Non-Maximum-Suppressiono Sliding window results in

clusters of detections around pedestrians

o NMS reduces this to only the highest scoring detection of each cluster

87.81

68.71

68.46 26.89

8.405

Pedestrian detection approach State-of-the-art detectors

40

• Histogram of Oriented Gradients [Dalal&Triggs, CVPR2005]

o Uses gradient information• Deformable Part Models

[Felzenszwalb, CVPR2008]

o Allows deformation of the parts relative to the root-model

State-of-the-art detectors

41

Channel based detectorso Use both gradient and color informationo Feature values are calculated as the sum of pixel values in

rectangles• Integral Channel Features [Dollár, BMVC2009]

o 30 000 random rectangles inside model window• Aggregate Channel Features [Dollár, PAMI2014]

o Approximation of the features at most scaleso All possible squares of a specific size inside the model window

• Squared Channel Features [Benenson, CVPR2014]

o All possible squares of which the side is a multiple of that specific scale

Measure accuracy

Precision vs. Recall Miss rate vs. False Positives per Image

42

Miss Rate: The share of pedestrians that is not foundFPPI: Average number of false detections (non-pedestrian) per image

Best point: bottom left

Recall: share of pedestrians foundPrecision: share of detections that is a pedestrian

Best point: top right

19/11/2015

8

Result using our combination technique

43

[Floris De Smedt*, Kristof Van Beeck*, Tinne Tuytelaars and Toon Goedemé; The Combinator: optimal combination of multiple pedestrian detectors; ICPR 2014.]

HOGICFDPM

Accuracy

44

Reduction of 77% in false detections

Ground plane integration

• For each scale we determine the boundaries on the ground• Extend this with pedestrians height

45

Ground plane constraint

46

Twofold advantage• Accuracy improvement by

avoiding false detections• Speed improvement by

reducing the search space

Application on mobile mapping blurring

[Steven Puttemans, Stef Van Wolputte and Toon Goedemé; Safeguarding privacy byreliable automatic blurring in mobile mapping images, submitted to VISAPP2016]

Detection of vulnerable road users in a truck’s blind spot

• Critical real-time constraint!o 15 fpso (+ Response time)

• Difficult view-pointo Scale variationo Rotation

• High accuracy �GPU-accelerated DPM

• High speed

48

Generalisedground constraint

Each position has a predictable scale & rotation� Warping Window technique

[A warping window approach to real-time vision-based pedestrian detection in a truck’s blind spot zone, Van Beeck et al . (2012)]

19/11/2015

9

Integration with Warping-Window approach

49

• Search-space reductiono Kalman-based tracking-

by-detection frameworko Warping-Window

ground plane constraint

• Each track will be evaluated individually by the hybrid framework

0 5 10 15 20 250

100

200

300

400

500

600

700Detection speed

Number of pedestrians per image

Spe

ed

FPS (frames per second) 140x75Hz (detections per second) 140x75

Application evaluation

50

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall

Pre

cisi

on

Precision vs Recall

± 12.9x faster than Matlabimplementation

� Speed-up by hybrid GPU-CPU implementation

20 pedestrians � 25fps

500Hz

Demo movies blind spot detection Let a UAV follow a person

• Goal: Steer autonomous towards the pedestrian

• On-board processingo Pedestrian detectiono Pedestrian tracking

o Controlling the UAV• Low computational power• Real-time constraint

o ACF detector + adaptive ground constraint

52

Experiments & results

[De Smedt, Floris, Hulens, Dries, Goedemé, Toon; On-Board Real-Time Tracking of Pedestrians on a UAV; Embedded VisionWorkshop; CVPR2015.]

Study on IR sensors� Advantages:

� Better people detection through body heat sensing

� Sees clearly in complete darkness without any illumination

� Works in bright sunlight, through smoke, dust or even light fog.

� Disadvantages:� Expensive� Low image resolution

19/11/2015

10

Study on IR sensors

heat waveradiation

Study on IR sensors

SWIR/NIR

LWIR = heat radiation

Types of IR sensors

Micro bolometer(greater resolution)

Thermopile(16x4, 1x8 pixels)

Other properties of IR sensorsNETD • Noise equivalent

temperature difference• Standard deviation of pixel

value• 50 – 150mK for uncooled

(at 30°C)

Field of view

Market study (1/2)Manufacturer Name Price Resolution

Omron D6T-8L-06 41.38 € 1x8

Omron D6T-44L-06 42.59 € 4x4

Melexis MLX90621 (obsolete) 50.43 € 16x4

Melexis MLX90620 59.55 € 16x4

FLIR Lepton 500-0643-00 160.93 € 80x60

FLIR Lepton 500-0659-01 168.29 € 80x60

FLIR Lepton 500-0690-00 168.29 € 80x60

Seek Thermal Compact 299.00 € 206x156

Seek Thermal Compact XR 349.00 € 206x156

Fluke Ti90 1,195.99 € 80x60

FLIR Tau2 160 1,400.00 € 160x128

FLIR Tau2 168 1,400.00 € 168x128

FLIR Tau2 162 1,400.00 € 162x128

FLIR FLIR Vue 336 1,499.00 € 336x256

DRS Technology Tamarisk 320 1,894.43 € 320x240

Fluke Ti100 1,995.99 € 160x120

Market study (2/2)Manufacturer Name Price Resolution

FLIR Quark 336 2,500.00 € 336x256

FLIR Quark 640 2,500.00 € 640x512

FLIR Tau2 324 2,500.00 € 324x256

FLIR Tau2 336 2,500.00 € 336x256

FLIR Tau2 640 2,500.00 € 640x512

FLIR FLIR Vue 640 2,999.00 € 640x512

Mobotix FlexMount S15 3,634.00 € 336x252

Acal Tamarisk 320 3,674.82 € 320x240

COX CX320 3,877.14 € 320x240

COX CX320 3,877.14 € 384x288

Mobotix AllroundDual M15 3,998.00 € 336x252

Fluke Ti200 5,495.99 € 200x150

COX CX640 5,503.03 € 640x480

Fluke Ti300 6,195.99 € 240x180

Xenics Gobi-384 7,500.00 € 384x288

Fluke Ti400 7,995.99 € 320x240

19/11/2015

11

Market study� 30-40 uncooled IR sensors� 50% with a price under 2500,00 EUR

� 66% with a resolution larger than 350 px (in X direction)

0-5001,0-1,5

1,5-2,0

2,0-2,5

2,5-3, 0

3,0-3,5

3,5-4,0

4, 0-4,5

4,5-5,0

5,0-5,5

5,5-6,0

6,0-6, 5

6,5-8,0

8000+

0

1

2

3

4

5

6

7

8

9

10

Aantal sensoren per prijsklasse

Prijsklasse (1000 euro)

n

<100 100-150 150-200 200-250 250-300 300-350 350-400 400-450 450-500 500-550 550-600 600-650

0

2

4

6

8

10

12

14

Aantal sensoren per x-resolutie

Resolutie in x-richting

n

Market study� Selectie

Naam Prijs Resolutie

FLIR Lepton 2 168,29 € 80x60

SEEK Thermal Compact 299,00 € 206x156

FLIR AX8 840 € 80x60 IR (640x480 RGB)

FLIR Vue 336 1499,00 € 336x256

How to combine RGB person detectionwith IR cameras?

Multiple options:1. Person detection algorithms on IR data

2. IR preprocessing + person detection on RGB images3. Person detection on RGB + IR verification4. Integrated IR+RGB person detection

IR RGB

Integrating IR in RGB person detection

• Example: ICF detector

IR

Program




Research AI part

Overview:• Typical approach• Our own research focus• Use cases and needs of application?

19/11/2015

12

Casesnr Test case Industrial

partnerComplexityimageprocessing

Safetycritical

ComplexityAI

Responsa-bility of thesystem

1 Security fromfixed cameras

Seris, Flir ITS, Melexis…

* * ** *

2 Security fromUAVs

DroneMatrix, Xtendit, …

*** * ** *

3 Safety from fixedcameras

Havenbedrijf Antwerpen, WenZ, ..

* *** * **

4 Safety frommanned drivingvehicles

CNHi, GrootJebbink, Dana, …

** *** * *

4bis Safety fromunmanneddriving vehicles

MABO, Octinion, …

** *** * ***

5 Patient/sportermonitoring

Sensolid,Alphatronics, Brenso, …

* ** *** **

CNHi use case proposal19th November 2015 68

� Person detection in dust for agricultural (larger crops particles in Ag dust) and construction vehicles

� Person detection for commercial vehicles in difficult environmental condition with high reliability (>99,xx %?) in:

• Blind spot all-around vehicle,

• Vehicle path.

� Data acquisition equipment and sensors from EAVISE. CNHi acquire data in real condition.

VIPER project: interesting use cases for CNHi

Spin-off“Pedestrian Detection for real-life applications” (PhD F. De Smedt)

Applications

Program




Practical issues

• All feedback is always welcome via mail/tel/…• Website: www.eavise.be/viper (in preparation)• IWT e-tool “gebruikerspoll”

o Collects feedback after every user group meeting• Meeting frequence?• IP-rights

• “Regelement van Orde”• Co-financing

Thank you for your attention!

Contact:[email protected]@kuleuven.be

administrative details · 2017. 11. 6. · • start: 1/10/2015, project duration: 2 year eavise:...

Documents