Machine Vision and Image Processing Algorithm –Machine Vision and Image Processing Algorithm Fall 2009

Mario

1

Introducion to OpenCVOpenCV structure and classesOpenCV functionsImage acquisition using webcam

2

OpenCV stands for Open Source Computer Vision LibraryyBeing developed at Intel since 1999Written in C/C++; Contains over 500 functions.Available on Windows, Linux and MacOSX.So far is extensively used in many companies and research centersHome page: www intel com/technology/computing/opencv/www.intel.com/technology/computing/opencv/

3

Computer Vision Market is large and continues to grow

There is no standard API (like OpenGL and DirectX in graphics, or OpenSSL in cryptography), most of CV software is of 3 kinds:◦ Research code (slow, unstable, independent/incompatible data types for every

library/toolbox) b a y/too bo )◦ Very expensive commercial toolkits (like Halcon, MATLAB+Simulink, …)◦ Specialized solutions bundled with hardware (Video surveillance, Manufacturing

control systems, Medical equipment …)

Standard library would simplify development of new applications and solutions much easier.

Special optimization for Intel Architectures:Special optimization for Intel Architectures:◦ Creates new usage models by achieving real-time performance for quite “heavy”

algorithms (like face detection)◦ Makes Intel platforms attractive for CV developers

4

The library is actively used by a large number of companies (such as Intel, IBM, Microsoft, SONY, Siemens, Google,…) and research centers (Stanford MIT CMU Cambridge INRIA etc )(Stanford, MIT, CMU, Cambridge, INRIA etc.)>14000 members of the forum OpenCV@yahoogroups com with average dailyOpenCV@yahoogroups.com, with average daily traffic ~10-20 messages.Community contributes to the project: bug reports, y p j g ppatches, new features (video acquisition, 3d tracking, textures, Python interface)

5

Extract from license:[Copyright Clause][Copyright Clause][Copyright Clause][Copyright Clause]Redistribution and use in source and binary forms, with or without modification,Redistribution and use in source and binary forms, with or without modification,are permitted provided that the following conditions are met:are permitted provided that the following conditions are met:

* Redistribution's of source code must retain the above copyright notice,* Redistribution's of source code must retain the above copyright notice,ed st but o s o sou ce code ust eta t e abo e copy g t ot ce,ed st but o s o sou ce code ust eta t e abo e copy g t ot ce,this list of conditions and the following disclaimer.this list of conditions and the following disclaimer.

* Redistribution's in binary form must reproduce the above copyright notice,* Redistribution's in binary form must reproduce the above copyright notice,this list of conditions and the following disclaimer in the documentationthis list of conditions and the following disclaimer in the documentationggand/or other materials provided with the distribution.and/or other materials provided with the distribution.

* The name of Intel Corporation may not be used to endorse or promote products * The name of Intel Corporation may not be used to endorse or promote products derived from this software without specific prior written permission.derived from this software without specific prior written permission.p p pp p p[Disclaimer][Disclaimer]

I b i f S !I b i f S !In brief: Sure!In brief: Sure!

6

7

CVImage processing

and vision algorithms HighGUIGUI, Image and Video I/Og

CXCOREbasic structures and algoritms,g ,XML support, drawing functions We will mostly

focus on these twoin this presentation

8

IplImage* cvLoadImage(image_path, colorness_flag);loads image from file, converts to color or grayscle, if need, and returns it (or g , g y , , (returns NULL).image format is determined by the file contents.

S I (i th i )cvSaveImage(image_path, image);saves image to file, image format is determined from extension.

BMP JPEG ( i libj ) PNG ( i lib ) TIFF ( i libtiff) PPM/PGM f tBMP, JPEG (via libjpeg), PNG (via libpng), TIFF (via libtiff), PPM/PGM formats are supported.

IplImage* img = cvLoadImage(“picture.jpeg”,-1);

if( img ) cvSaveImage( “picture.png”, img););

9

Windows GUI

Scanline()…

Timage(Bitmap)

Windows GUI

I lI T TBit ()

Image Processing(Conversion, Transform etc )

OpenCV-based Image Processing(ConversionIplImageToTBitmap() Transform, etc.)(Conversion, transform, etc.)

IplImage *cvLoadImage()

Image1->Picture->LoadFromFile()

Image File(bmp, jpg,

)cvSaveImage()

Other image Sourcepng) Source(e.g. Camera)

10

typedef struct CvMat {int type; // data type: CV_8UC1, CV 8UC3, …

widthData pointer

_ ,// … CV_16UC1, …, CV_64FC1int step;union { // aliased pointers to the dataunsigned char* ptr;float* fl; double* db;} data;int rows; // ~width

Pixel/elementheight

int cols; // ~height…} CvMat;

// create large color image as a matrixCvMat* image = cvCreateMat( 1024, 768, CV_8UC3 );// fill the image with red color (blue=0, green=0, red = 255)

Example:

255)cvSet( image, cvScalar(0,0,255));

11

cvGEMM(A,B,a,C,b,D,flags); // generalized matrix product: D=a*Aop(A) *Bop(B) + b*Cop(C)cvSolve(A,b,x,method); // solve Ax=b (linear system or least squares problem)cvSVD, cvSVDBkSb // singular value decomposition and back substitution

// some other functions to look atcvDet, cvTrace, cvInvert, cvDotProduct, cvCrossProduct, cvEigenVVcvDFT(A,B,flags); cvDCT(A,B,flags) // forward/inverse discrete Fourier andCosine transformsCosine transforms

// fast vector transcendental functions (5-20x faster than exp(),log() etc.!)cvExp(A,B); cvLog(A,B); cvPow(A,B,a); /* B=pow(A,a); */cvCartToPolar(X,Y,mag,phase); cvPolarToCart(mag,phase,X,Y);

12

The sample task: collect the positions of all non-zero pixels in the image.Where to store the coordinates?Where to store the coordinates?Possible solutions:◦ Allocate array of maximum size (2*image_width*image_height – a huge value)◦ Use two-pass algorithm◦ Or use a list or similar data structure

// create a sequence of non-zero pixel positionsCvSeq* get non zeros( const IplImage* img, CvMemStorage* storage )q g _ _ ( p g g, g g ){CvSeq* seq = cvCreateSeq( CV_32SC2, sizeof(CvSeq),

sizeof(CvPoint), storage );

for( int i = 0; i < img->height; i++ ) for( int j = 0; j < img->width; j++ )

if( CV_IMAGE_ELEM(img,uchar,i,j) ){

CvPoint pt={j,i}; cvSeqPush( seq, &pt ); }return seq;

}13

Memory storage is a linked list f bl kof memory blocks.

It acts like stack:◦ new data is allocated always at y

the end◦ data is not deallocated until

storage is deleted or cleared.gKey functions:◦ cvCreateMemStorage(block_size),◦ cvReleaseMemStorage(storage);◦ cvReleaseMemStorage(storage);◦ cvClearMemStorage(storage);◦ cvMemStorageAlloc(storage,size);

14

Need a config file?Want to save results of your program to pass it to another one, or look at it another day?Use cvSave, cvLoad and other XML I/O functions from cxcore.

// somewhere deep in your code… you get 5x5// matrix that you’d want to save{ CvMat A = cvMat( 5, 5, CV_32F, the_matrix_data);cvSave( “my matrix xml” &A );

<?xml version="1.0"?><opencv_storage><my_matrix type_id="opencv-matrix"><rows>5</rows>cvSave( “my_matrix.xml”, &A );

}// to load it then in some other program use …CvMat* A1 = (CvMat*)cvLoad( “my_matrix.xml” );

rows 5 /rows<cols>5</cols><dt>f</dt><data>1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.0 1 0 0 0 0 0 1 0 0 0 00. 1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.0. 1.</data></my_matrix></opencv_storage>

15

16

Create an IplImage object:◦ IplImage * srcImage;IplImage srcImage;Load Image◦ srcImage = cvLoadImage(FileName, -1););

Show Image◦ bitmap = new Graphics::TBitmap();

◦ IplImageToTBitmap(srcImap g p(ge, bitmap);

◦ Image1->Picture->Bitmap = bitmap;

Or◦ cvNamedWindow(“image”);◦ cvShowImage(“image”, srcImage);

17

src

Color conversion can f RGB Color space

src

be performed using cvCvtColor() function

RGB Color space

function.Void cvCvtColor(const dst

CV_RGB_HSV,

CvArr* src, const CvArr* dst, int code);◦ Src: source image HSV Color space

dst

g◦ Dst: destination image◦ Code: color conversion operation

E.g. CV_RGB_XYZ, CV_RGB_HSV, CV_RGB_YCrCb,

HSV Color space

18

19

We can set ROI in an i b i imageimage by using cvSetImageROI function.void

image

widthvoidcvSetImageROI(IplImage* image, CvRect rect);

Image: the source imageROI

x, y width

height◦ Image: the source image◦ Rect : the roi rectangleROI is defined by one point O s de ed by o e po t(x,y), width and the height.

CvRect r = cvRect(int x, int y, int width, intheight);

20

We can use cvCopy to copy selected elements from input

wsrc

ROIse ected e e e ts o putarray to output array.If the passed input array is IplImage, then its ROI fields is used

srchsrcROI

used.Both array must have the same type.void cvCopy( const CvArr*

C A * d t t C A * wdstsrc, CvArr* dst, const CvArr* mask=NULL );◦ src The source array. ◦ dst The destination array.

k O i k 8 bi dst

wdst

hdst◦ mask Operation mask, 8-bit

single channel array; specifies elements of destination array to be changed.

dsthdst

If there is ROI wdst = wroi and hdst = hroiIf there is ROI, wdst wroi and hdst hroiElse wdst = wsrc and hdst = hsrc

21

ROI

22

The simplest way to

jimg

access image pixel is by using a CvScalart t d

i

structure and cvGet2D() functionExample:

cvGet2DS[0]Example:

CvScalar s;s=cvGet2D(img,i,j);

s

[ ]

S[1]

S[2]

S[3]

23

Example of source code for reading an image:

image = cvLoadImage(filenames.at(k).c_str(), 1);

for(int i=0;i<image->height;i++){for(int j=0;j<image->width;j++){CvScalar n;CvScalar n;n = cvGet2D(image, i, j);intensity[j+roiX*i] = n.val[0];

}}

The value of each pixel is stored in array intensity[width*height]intensity[width*height]

24

We can use cvSampleLine() to imgcvSampleLine() to sample one line in the image.int cvSampleLine(

pt1

img

IplImage* img, CvPoint pt1, CvPointpt2, void* buffer );◦ img Image. pt2img Image.◦ pt1 Starting line point.◦ pt2 Ending line point.◦ buffer Buffer to store the

line points; must have enough

pt2

cvSampleLineline points; must have enough size to store MAX(|pt2.x -pt1.x| + 1,|pt2.y - pt1.y|+1) points. buffer

…

25

cvSampleLine(image, cvPoint(0, 120), cvPoint(319, 120), lineProfile->data.ptr);

Start point End point

for(int i=0;i<lineProfile->cols;i++)

{

int height = lineProfile->data.ptr[i];

int posX = i;

cvCircle(procImage, cvPoint(posX, height), 1, whiteLine);

}

Sh I ("P d I " I )cvShowImage("Processed Image", procImage);

26

We can make a binary image by applying threshold to an image.I O CV th h ldIn OpenCV, we can use cvThreshold()Types of threshold:◦ Binary (CV_THRESH_BINARY)

if src(x,y)>threshold d ( ) l 0

threshold

dst(x,y)=max, else 0;◦ Inverted Binary

(CV_THRESH_BINARY_INV)if src(x,y)>threshold dst(x,y)=0, else max;

◦ Truncated (CV THRESH TRUNC)_ _if src(x,y)>threshold dst(x,y)= threshold, else 0;

◦ Threshold to zero (CV_THRESH_TOZERO)

if src(x,y)>threshold dst(x,y)= src(x,y), else 0;( ,y), ;

◦ Inverted Threshold to zero (CV_THRESH_TOZERO_INV)

if src(x,y)>threshold dst(x,y)=0, else src(x,y);

27

IplImage *thresImage = cvCreateImage(cvSize(image->width, image->height), image->depth, 1);

thresImage->origin = 1;cvCvtColor(image, thresImage, CV_BGR2GRAY);cvThreshold(thresImage, thresImage, 125, 255, CV_THRESH_BINARY_INV);

Sh I ("Th h ld" th I )cvShowImage("Threshold", thresImage);cvReleaseImage(&thresImage);

28

There are several image filter that is available in OpenCV such as

CV_BLURp

gaussian filter, median filter, bilateral filter, and blur.We can use this function:void cvSmooth( const CV_GAUSSIAN, CvArr* src, CvArr* dst, intsmoothtype=CV_GAUSSIAN, int param1=3, intparam2=0);param2=0);◦ Src: The source image. ◦ Dst: The destination image. ◦ Smoothtype: CV_BLUR_NO_SCALE, CV_BLUR, CV_GAUSSIAN, CV_MEDIAN, CV_BILATERAL

◦ Param1 and param2 are size of the kernel

29

Draw a box◦ cvRectangle(img, cvPoint(100,100),

(100,100)g g, , ,

cvPoint(200,200), cvScalar(255,0,0), 1);

Draw a circle◦ cvCircle(img, cvPoint(100,100), 20,

cvScalar(0,255,0), 1); (200,200)Draw a line segment◦ cvLine(img, cvPoint(100,100),

cvPoint(200,200), cvScalar(0,255,0), 1);

Draw a set of polylines

( , )

(100,100)20

p y◦ cvPolyLine(img,curveArr,nCurvePts,nCurve

s,isCurveClosed,cvScalar(0,255,255),lineWidth);

Draw a set of filled polygons

(100,100)

◦ cvFillPoly(img,curveArr,nCurvePts,nCurves,cvScalar(0,255,255)); (200,200)

30

Rectangle CircleRectangle Circle

31Line

Opencv has a capability of acquiring images from a camera such as webcam.This is implemented in cvcam class, in the newer

i f th lib it i i l d d i hi h iversion of the library, it is included in highgui.Therefore, we can implement and try our developed algorithm in real time by writing ourdeveloped algorithm in real time by writing our code in the callback function of the image acquisition.q

32

A function that will be automatically called by a generated event.

camera

For example in a camera, a callback function is called Callback()function is called every time a new image is captured by

Callback()

image is captured by the camera Algorithm

33

Include files:◦ #include <cvcam.h>#include <cvcam.h>◦ #include <cv.h>◦ #include <highgui.h>Check the number of camera in the system// R t th b f il bl i th◦ // Returns the number of available cameras in the system.

◦ Int ncams = cvcamGetCamerasCount();◦ // Exit the program when the video camera is not a ailableavailable.

◦ if( ncams < 1 )◦ {◦ Application->MessageBox("Can not find a video

!" " " MB OK)camera!", "error", MB_OK);◦ ShowMessage("Can not find a video camera!");◦ exit(1);◦ }

34

Set the property of the camera◦ cvcamSetProperty(cameraSelected◦ cvcamSetProperty(cameraSelected, CVCAM_RNDWIDTH , &width);

◦ cvcamSetProperty(cameraSelected, CVCAM_RNDHEIGHT , &height);

◦ cvcamSetProperty(cameraSelected, CVCAM_PROP_ENABLE, &t);

◦ cvcamSetProperty(cameraSelected, CVCAM PROP RENDER &t);CVCAM_PROP_RENDER, &t);

◦ cvcamSetProperty(cameraSelected, CVCAM_PROP_CALLBACK, myCallBack);

Start the camera◦ cvcamInit();◦ cvcamStart();

35

36

37

Install the OpenCV Library into your computer.Create a new application with Image, OpenDialog and two Button components.Create an IplImage object;Create an IplImage object;◦ IplImage * image;

When the Button is pressed, initialize the IplImage to load the image from file.◦ Image = cvLoadImage(…); //read from the referencereference

When the second button is pressed, call the function to save the image into another file◦ if( img ) cvSaveImage( “picture.png”, img );

38

Using the same project file, create another button.When this button is pressed, convert the image from Color to Grayscale using this function:◦ cvCvtColor(bgr image, grayscale image,cvCvtColor(bgr_image, grayscale_image, CV_BGR2GRAY);

Add another button to perform filtering of an image. Use this function:◦ cvSmooth(image, image, CV_GAUSSIAN, 7, 7, 1., 1.); // inplace Gaussian7, 1., 1.); // inplace Gaussian smoothing// with 7x7 kernel and σ=1.

39

Create Histogram of the Image by accessing each pixel of the image.◦ Red◦ Green◦ Green◦ Blue◦ Grayscale

40

Create a program to extract ROI from an imageUse cvSetImageROI(IplImage *image, CvRectrect);

Image x,y width

ROIheight

41

Create two IplImage objects.Create an ROI from the first imageCopy the image into the second image

ROIcopy

ROI

Image1Image2

42

Create an Application withCreate an Application with◦ One Timage◦ One Tchart◦ TButtons and TEdit◦ TButtons and TEditDefine these variable:◦ unsigned char * buffer;◦ CvPoint start end;

start end

◦ CvPoint start, end;

Use this function:◦ cvSampleLine( IplImage* img CvPoint pt1 CvPointimg, CvPoint pt1, CvPointpt2, void* buffer );

Assign the value to the Tchart

43

Draw a rectangle in an image◦ First click sets the first point (use CvPoint)First click sets the first point (use CvPoint)◦ Second click sets the second point (use CvPoint)◦ Then draw the rectangle from first point to second point (use

cvRectangle())Draw a circleDraw a circle◦ First click sets the center of the image (use CvPoint)◦ Second click sets the radius of the circle (use CvPoint and calculate the

radius)◦ Then draw the circle using Ci l ( )◦ Then draw the circle using cvCircle( … )Draw a line◦ First click sets the first point (use CvPoint)◦ Second click sets the second point (use CvPoint)◦ Then draw the rectangle from first point to second point (use cvLine( … ))

Next, use the mouse to draw on the image, draw rectangle, line and circleand circle

44

Use the sample code that can be obtained from this url:http://140.124.201.17/opencv/Th fil W b O CV iThe filename: WebcamOpenCV.zipOpen the project using your C++ Builder.Implement these function:Implement these function:◦ color conversion◦ Thresholdingg◦ Drawing histogram of the image

45

Getting Started with OpenCV by VadimPisarevsky (vadim.pisarevsky@intel.com)http://www710.univ-lyon1.fr/~bouakaz/OpenCV-0 9 5/docs/ref/OpenCVRef ImageProcessing htm0.9.5/docs/ref/OpenCVRef_ImageProcessing.htmIntroduction to programming with OpenCV, GadyAgam, http://www.cs.iit.edu/~agam/cs512/lect-g , p gnotes/opencv-intro/opencv-intro.html

46

t6669036@ntut.edu.tw

47