OpenCV IntroductionHang Xiao

Oct 26, 2012

History 1999 Jan : lanched by Intel, real time machine vision library for UI, optimized code

for intel

2000 Jun : OpenCV alpha 3。

2000 Dec : OpenCV beta 1 for linux

2006 : the first 1.0 version supports Mac OS

2008 mid : obtain corporate support from Willow Garage

2009 Sep : OpenCV 1.2（ beta2.0

2009 Oct : Version 2.0 released。

2010 Dec : OpenCV 2.2。

2011 Aug : OpenCV 2.3。

2012 Apr : OpenCV 2.4.

Overview

Goals Develop a universal toolbox for research and development in the

field of Computer Vision

Algorithms More than 350 algorithms, 500 API

Programming language C/C++, C#, Ch , Python, Ruby, Matlab, and Java (using JavaCV)

OS support Windows, Android, Maemo, FreeBSD, OpenBSD, iOS, Linux and Mac

OS.

Licence BSDlisence, free for commercial and non-commmercial

Overview - Applications

2D and 3D feature toolkits

Egomotion estimation

Facial recognition system

Gesture recognition

Human–computer interaction (HCI)

Mobile robotics

Motion understanding

Object identification

Segmentation and Recognition

Stereopsis Stereo vision: depth perception from 2 cameras

Structure from motion (SFM)Motion tracking

Overview - A statistical machine learning library

Boosting (meta-algorithm)

Decision tree learning

Gradient boosting trees

Expectation-maximization algorithm

k-nearest neighbor algorithm

Naive Bayes classifier

Artificial neural networks

Random forest

Support vector machine (SVM)

Outline

Image Analysis

Structural Analysis

Object Recognition

Motion Analysis and Object Tracking

3D Reconstruction

Outline

Image Analysis

Structural Analysis

Object Recognition

Motion Analysis and Object Tracking

3D Reconstruction

Image Analysis

Thresholds

Statistics

Pyramids

Morphology

Distance transform

Flood fill

Feature detection

Contours retrieving

Image Thresholding

Fixed threshold;

Adaptive threshold;

Image Thresholding Examples

Source picture Fixed threshold Adaptive threshold

Statistics

min, max, mean value, standard deviation over the image

Norms C, L1, L2

Multidimensional histograms

Spatial moments up to order 3 (central, normalized, Hu)

Multidimensional Histograms

Histogram operations calculation, normalization, comparison, back project

Histograms types: Dense histograms

Signatures (balanced tree)

EMD algorithm The EMD computes the distance between two distributions, which are

represented by signatures.

The signatures are sets of weighted features that capture the distributions. The features can be of any type and in any number of dimensions, and are defined by the user.

The EMD is defined as the minimum amount of work needed to change one signature into the other

EMD – a method for the histograms comparison

.),(

,

,

),(

),(

,1,,1,

,

,

jiji

ij

jiij

jijiij

ji

qandpelementsthebetweendistancetheqpd

tscoefficienweightf

f

qpdf

QPEMD

historamstwoQjQqPiPp

Image Pyramids

Gaussian and Laplacian pyramids

Image segmentation by pyramids

Image Pyramids

Gaussian and Laplacian

Pyramid-based color segmentation

On still pictures And on movies

Morphological Operations

Two basic morphology operations using structuring element: erosion

dilation

More complex morphology operations: opening : erosion + dilation

closing : dilation + erosion

morphological gradient : the difference between the dilation and the erosion of

an image

top hat : the difference between an input image and its opening

black hat : the difference between the closing and its input image

Morphological Operations Examples

Morphology - applying Min-Max. Filters and its combinations

Opening IoB= (IB)BDilatation IBErosion IBImage I

Closing I•B= (IB)B TopHat(I)= I - (IB) BlackHat(I)= (IB) - IGrad(I)= (IB)-(IB)

Distance Transform

Calculate the distance for all non-feature points to the closest feature point

Two-pass algorithm, 3x3 and 5x5 masks, various metrics predefined

Flood Filling

Simple

Gradient

Feature Detection

Fixed filters (Sobel operator, Laplacian);

Optimal filter kernels with floating point coefficients (first, second derivatives, Laplacian)

Special feature detection (corners)

Canny operator

Hough transform (find lines and line segments)

Gradient runs

Canny Edge Detector

Hough TransformDetects lines in a binary image

• Probabilistic Hough Transform• Standard Hough

Transform

Another Sample of the Hough Transform Using

Source picture Result

Contour Retrieving

The contour representation: Chain code (Freeman code) Polygonal representation

Initial Point

Chain code for the curve: 34445670007654443

Contour representation

Hierarchical representation of contours

Image Boundary

(W1) (W2) (W3)

(B2) (B3) (B4)

(W5) (W6)

Contours Examples

Source Picture(300x600 = 180000 pts total)

Retrieved Contours (<1800 pts total)

After Approximation(<180 pts total)

And it is rather fast: ~70 FPS for 640x480 on complex scenes

Outline

Image Analysis

Structural Analysis

Object Recognition

Motion Analysis and Object Tracking

3D Reconstruction

Structural Analysis

Contours processing Approximation Hierarchical representation Shape characteristics Matching

Geometry Contour properties Fitting with primitives PGH: pair-wise geometrical histogram for the

contour.

Contour Processing

Approximation: RLE algorithm (chain code) Teh-Chin approximation (polygonal) Douglas-Peucker approximation (polygonal);

Contour moments (central and normalized up to order 3)

Hierarchical representation of contours

Matching of contours

Hierarchical Representation of Contours

A contour is represented with a binary tree

Given the binary tree, the contour can be retrieved with arbitrary precision

The binary tree is quasi invariant to translations, rotations and scaling

Contours matching

Matching based on hierarchical representation of contours

Geometry

Properties of contours: (perimeter, area, convex hull, convexity defects, rectangle of minimum area)

Fitting: (2D line, 3D line, circle, ellipse)

Pair-wise geometrical histogram

Pair-wise geometrical histogram (PGH)

),( jip

.),(/),()(

,),(/),()(

,)](),2(),1(),(),2(),1([

iic

jjr

TcccrrrPGH

jipjipijE

jipjipjiE

MEEENEEEf

Outline

Image Analysis

Structural Analysis

Object Recognition

Motion Analysis and Object Tracking

3D Reconstruction

Object Recognition

Eigen objects

Hidden Markov Models

Eigen Objects

.

.},,,{

,,1},,,,{

),(/1

,/1),()(

,cov}{

,,1,},,{

21

111

1

21

vectorseigenandvalueseigenand

basiseigenmmieeee

uuve

umuuuuuc

matrixariancecC

nmmiRuuuu

im

iiii

in

iii

l

k

kl

iki

il

m

k

klll

jl

l

lilij

ij

nin

iii

Eigen objects (continued)

Hidden Markov Model

Definitions

},...,,{ 21 NsssS },...,,{ 21 MoooO

Oqt )}|(\{ 1 itjtijij sqsqPaaA

)}|(\{ jiijij soPbbB

)}(\{ 0 iii sqP

- The set of states

- The set of measurements

- The state at time t

- The transition probability matrix

- The conditional probability matrix

- The starting states distribution

Embedded HMM for Face Recognition

Model-

- Face ROI partition

Face recognition using Hidden Markov Models

One person – one HMMStage 1 – Train every HMM

Stage 2 – Recognition

Pi - probability

Choose max(Pi)

…1

n

i

Outline

Image Analysis

Structural Analysis

Object Recognition

Motion Analysis and Object Tracking

3D Reconstruction

Motion Analysis and Object Tracking

Background subtraction

Motion templates

Optical flow

Active contours

Estimators

Background Subtraction

Background model (normal distribution)

Background statistics functions: Average Standard deviation Running average

Motion Templates

Object silhouette

Motion history images

Motion history gradients

Motion segmentation algorithmsilhouette MHI

MHG

Motion Segmentation Algorithm

Two-pass algorithm labeling all motion segments

Motion Templates Example

Motion templates allow to retrieve the dynamic characteristics of the moving object

Optical Flow

Block matching technique

Horn & Schunck technique

Lucas & Kanade technique

Pyramidal LK algorithm

6DOF (6 degree of freedom) algorithm

y

xt

yyx

yxx

I

IIb

III

IIIGyxX

bXG

dtdyyIdtdxxItI

tyxIdttdyydxxI

,,

,),,(

,

);/(/)/(//

);,,(),,(

2

2

Optical flow equations:

Pyramidal Implementation of the optical flow algorithm

J image I image

Image Pyramid Representation

Iterative Lucas – Kanade Scheme

Generic Image

(L-1)-th Level

L-th Level

Location of point u on image uL=u/2L

Spatial gradient matrix

Standard Lucas – Kanade scheme for optical flow computation at level L dL

Guess for next pyramid level L – 1

Finally,

Image pyramid building

Optical flow computation

2

2

,

,

yyx

yxx

III

IIIG

)(21 LLL dgg

00 gdd

dUV

6DOF Algorithm

N

i ROI

Tit

N

i ROIi

Ti IIdsII

sXIsII

i

11

///

).(sX

Parametrical optical flow equations:

Active Contours

Snake energy:

Internal energy:

External energy:

Two external energy types:

extEEE int

curvcont EEE int

conimgext EEE

min

,)(

,

imgcurvcont

img

img

EEEE

IgradE

IE

Estimators

Kalman filter

ConDensation filter

Kalman object tracker

Outline

Image Analysis

Structural Analysis

Object Recognition

Motion Analysis and Object Tracking

3D Reconstruction

3D reconstruction

Camera Calibration

View Morphing

POSIT

Camera Calibration

Define intrinsic and extrinsic camera parameters.

Define Distortion parameters

],[],,,[,,,

100

0

0

,][

3

2

1

333231

232221

131211

vupZYXP

t

t

t

T

rrr

rrr

rrr

Rcf

cf

A

PRTAp

yy

xx

.

)],2/(2[)(~)],2/(2[)(~

222

212

42

21

221

42

21

yxr

yyrpxprkrkcvvv

xxrpyprkrkcuuu

y

x

Camera Calibration

Now, camera calibration can be done by holding checkerboard in front of the camera for a few seconds.

And after that you’ll get:3D view of etalon Un-distorted image

View Morphing

POSIT Algorithm

Perspective projection:

Weak-perspective projection:

iiiiii YZfyXZfx )/(,)/(

./,, ZfsYsyXsx iiii

OpenCV Websites

http://opencv.org OpenCV official webpage.

http://opencvlibrary.sourceforge.net/ OpenCV documentation and FAQs.

OpenCV Examples

adaptiveskindetector : detect skin area

fback_c : dense Franeback optical flow

contours : calculate contours on different levels

delaunay : delaunay triangle

find_obj : SURF Detector and Descriptor using either FLANN or brute force matching on planar objects

morphology : open/close, erode/dilate

motempl ：motion templates

mser_sample : Maximal Extremal Region interest point detector

polar_transforms : illustrates Linear-Polar and Log-Polar image transforms

pyramid_segmentation : color pyramid segmentation

Thanks !!!