OpenCV IntroductionHang Xiao

Oct 26, 2012

History 1999 Jan : lanched by Intel, real time machine vision library for UI, optimized code

for intel

2000 Jun : OpenCV alpha 3。 2000 Dec : OpenCV beta 1 for linux

2006 : the first 1.0 version supports Mac OS

2008 mid : obtain corporate support from Willow Garage

2009 Sep : OpenCV 1.2（ beta2.0

2009 Oct : Version 2.0 released。 2010 Dec : OpenCV 2.2。 2011 Aug : OpenCV 2.3。 2012 Apr : OpenCV 2.4.

Overview Goals

Develop a universal toolbox for research and development in the field of Computer Vision

Algorithms More than 350 algorithms, 500 API

Programming language C/C++, C#, Ch , Python, Ruby, Matlab, and Java (using JavaCV)

OS support Windows, Android, Maemo, FreeBSD, OpenBSD, iOS, Linux and Mac

OS.

Licence BSDlisence, free for commercial and non-commmercial

Overview - Applications 2D and 3D feature toolkits Egomotion estimation Facial recognition system Gesture recognition Human–computer interaction (HCI) Mobile robotics Motion understanding Object identification Segmentation and Recognition Stereopsis Stereo vision: depth perception from 2 cameras Structure from motion (SFM)Motion tracking

Overview - A statistical machine learning library Boosting (meta-algorithm)

Decision tree learning

Gradient boosting trees

Expectation-maximization algorithm

k-nearest neighbor algorithm

Naive Bayes classifier

Artificial neural networks

Random forest

Support vector machine (SVM)

Outline

Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction

Outline

Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction

Image Analysis Thresholds Statistics Pyramids Morphology Distance transform Flood fill Feature detection Contours retrieving

Image Thresholding

Fixed threshold; Adaptive

threshold;

Image Thresholding Examples

Source picture Fixed threshold Adaptive threshold

Statistics

min, max, mean value, standard deviation over the image

Norms C, L1, L2 Multidimensional histograms Spatial moments up to order 3 (central,

normalized, Hu)

Multidimensional Histograms Histogram operations calculation, normalization, comparison, back project

Histograms types: Dense histograms Signatures (balanced tree)

EMD algorithm The EMD computes the distance between two distributions, which are

represented by signatures. The signatures are sets of weighted features that capture the distributions.

The features can be of any type and in any number of dimensions, and are defined by the user.

The EMD is defined as the minimum amount of work needed to change one signature into the other

EMD – a method for the histograms comparison

.),(

,

,),(

),(

,1,,1,

,

,

jiji

ij

jiij

jijiij

ji

qandpelementsthebetweendistancetheqpd

tscoefficienweightf

f

qpdfQPEMD

historamstwoQjQqPiPp

Image Pyramids

Gaussian and Laplacian pyramids

Image segmentation by pyramids

Image Pyramids

Gaussian and Laplacian

Pyramid-based color segmentation

On still pictures And on movies

Morphological Operations Two basic morphology operations using structuring element:

erosion dilation

More complex morphology operations: opening : erosion + dilation closing : dilation + erosion morphological gradient : the difference between the dilation and the erosion of an

image top hat : the difference between an input image and its opening black hat : the difference between the closing and its input image

Morphological Operations Examples Morphology - applying Min-Max. Filters and its combinations

Opening IoB= (IB)BDilatation IBErosion IBImage I

Closing I•B= (IB)B TopHat(I)= I - (IB) BlackHat(I)= (IB) - IGrad(I)= (IB)-(IB)

Distance Transform

Calculate the distance for all non-feature points to the closest feature point

Two-pass algorithm, 3x3 and 5x5 masks, various metrics predefined

Flood Filling

Simple Gradient

Feature Detection Fixed filters (Sobel operator, Laplacian); Optimal filter kernels with floating point coefficients

(first, second derivatives, Laplacian) Special feature detection (corners) Canny operator Hough transform (find lines and line segments) Gradient runs

Canny Edge Detector

Hough TransformDetects lines in a binary image

• Probabilistic Hough Transform• Standard Hough

Transform

Another Sample of the Hough Transform Using

Source picture Result

Contour Retrieving

The contour representation: Chain code (Freeman code) Polygonal representation

Initial PointChain code for the curve: 34445670007654443

Contour representation

Hierarchical representation of contours

Image Boundary

(W1) (W2) (W3)

(B2) (B3) (B4)

(W5) (W6)

Contours Examples

Source Picture(300x600 = 180000 pts total)

Retrieved Contours (<1800 pts total)

After Approximation(<180 pts total)

And it is rather fast: ~70 FPS for 640x480 on complex scenes

Outline

Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction

Structural Analysis Contours processing Approximation Hierarchical representation Shape characteristics Matching

Geometry Contour properties Fitting with primitives PGH: pair-wise geometrical histogram for the

contour.

Contour Processing

Approximation: RLE algorithm (chain code) Teh-Chin approximation (polygonal) Douglas-Peucker approximation (polygonal);

Contour moments (central and normalized up to order 3) Hierarchical representation of contours Matching of contours

Hierarchical Representation of Contours

A contour is represented with a binary tree

Given the binary tree, the contour can be retrieved with arbitrary precision

The binary tree is quasi invariant to translations, rotations and scaling

Contours matching

Matching based on hierarchical representation of contours

Geometry

Properties of contours: (perimeter, area, convex hull, convexity defects, rectangle of minimum area)

Fitting: (2D line, 3D line, circle, ellipse) Pair-wise geometrical histogram

Pair-wise geometrical histogram (PGH)

),( jip

.),(/),()(

,),(/),()(

,)](),2(),1(),(),2(),1([

iic

jjr

TcccrrrPGH

jipjipijE

jipjipjiE

MEEENEEEf

Outline

Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction

Object Recognition Eigen objects Hidden Markov Models

Eigen Objects

..},,,{

,,1},,,,{

),(/1

,/1),()(

,cov}{,,1,},,{

21

111

1

21

vectorseigenandvalueseigenand

basiseigenmmieeee

uuve

umuuuuuc

matrixariancecCnmmiRuuuu

im

iiii

in

iii

lk

kl

iki

il

m

k

klll

jl

ll

ilij

ij

nin

iii

Eigen objects (continued)

Hidden Markov Model

Definitions

},...,,{ 21 NsssS },...,,{ 21 MoooO

Oqt )}|(\{ 1 itjtijij sqsqPaaA

)}|(\{ jiijij soPbbB

)}(\{ 0 iii sqP

- The set of states

- The set of measurements

- The state at time t

- The transition probability matrix

- The conditional probability matrix

- The starting states distribution

Embedded HMM for Face Recognition

Model-

- Face ROI partition

Face recognition using Hidden Markov Models

One person – one HMMStage 1 – Train every HMM

Stage 2 – Recognition

Pi - probability

Choose max(Pi)

…1

n

i

Outline

Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction

Motion Analysis and Object Tracking

Background subtraction Motion templates Optical flow Active contours Estimators

Background Subtraction

Background model (normal distribution) Background statistics functions:

Average Standard deviation Running average

Motion Templates

Object silhouette Motion history images Motion history gradients Motion segmentation algorithm

silhouette MHIMHG

Motion Segmentation Algorithm

Two-pass algorithm labeling all motion segments

Motion Templates Example

Motion templates allow to retrieve the dynamic characteristics of the moving object

Optical Flow Block matching technique Horn & Schunck technique Lucas & Kanade technique Pyramidal LK algorithm 6DOF (6 degree of freedom) algorithm

y

xt

yyx

yxx

II

IbIII

IIIGyxX

bXG

dtdyyIdtdxxItItyxIdttdyydxxI

,,

,),,(

,

);/(/)/(//);,,(),,(

2

2

Optical flow equations:

Pyramidal Implementation of the optical flow algorithm

J image I image

Image Pyramid Representation

Iterative Lucas – Kanade Scheme

Generic Image

(L-1)-th Level

L-th Level

Location of point u on image uL=u/2L

Spatial gradient matrix

Standard Lucas – Kanade scheme for optical flow computation at level L dL

Guess for next pyramid level L – 1

Finally,

Image pyramid building

Optical flow computation

2

2

,

,

yyx

yxx

III

IIIG

)(21 LLL dgg

00 gdd

dUV

6DOF Algorithm

N

i ROI

Tit

N

i ROIi

Ti IIdsII

sXIsII

i11

///

).(sX

Parametrical optical flow equations:

Active Contours

Snake energy: Internal energy: External energy: Two external energy types:

extEEE int

curvcont EEE int

conimgext EEE

min

,)(

,

imgcurvcont

img

img

EEEE

IgradE

IE

Estimators

Kalman filter ConDensation filter

Kalman object tracker

Outline

Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction

3D reconstruction

Camera Calibration View Morphing POSIT

Camera Calibration Define intrinsic and extrinsic camera parameters. Define Distortion parameters

],[],,,[,,,

100

00

,][

3

2

1

333231

232221

131211

vupZYXPttt

Trrrrrrrrr

Rcfcf

A

PRTAp

yy

xx

.

)],2/(2[)(~)],2/(2[)(~

222

212

42

21

221

42

21

yxr

yyrpxprkrkcvvv

xxrpyprkrkcuuu

y

x

Camera CalibrationNow, camera calibration can be done by holding checkerboard in front of the camera for a few seconds.

And after that you’ll get:3D view of etalon Un-distorted image

View Morphing

POSIT Algorithm

Perspective projection:

Weak-perspective projection:

iiiiii YZfyXZfx )/(,)/(

./,, ZfsYsyXsx iiii

OpenCV Websites http://opencv.org

OpenCV official webpage.

http://opencvlibrary.sourceforge.net/ OpenCV documentation and FAQs.

OpenCV Examples adaptiveskindetector : detect skin area fback_c : dense Franeback optical flow contours : calculate contours on different levels delaunay : delaunay triangle find_obj : SURF Detector and Descriptor using either FLANN or brute

force matching on planar objects morphology : open/close, erode/dilate motempl ：motion templates mser_sample : Maximal Extremal Region interest point detector polar_transforms : illustrates Linear-Polar and Log-Polar image transforms pyramid_segmentation : color pyramid segmentation

Thanks !!!