*Lips-Control Assistive System Presenter : Deng-Shing Yang Advisor: Dr. Shih-Chung Chen

Date: 2012.11.14

OUTLINEMotivation

Introduction

Paper Reviews

Material and Methods

Discussion

Future works

References*

MOTIVATION*Many computers input devices are designed for normal persons, so these devices are unsuitable for the disabled probably. Many researchers develop many auxiliary devices for the disabled, but these auxiliary devices still have many defects when the disabled use them in real life. Efficiency and recognition rate are very important factors of image processing algorithm of the communication system based on image processing.

INTRODUCTIONEssential property*I. AdaptabilityFace Size Face Posture Background ComplexityLight Intensity VariationII. EfficiencySize of Input ImagePC ClockThe Efficiency of Algorithm

OBJECTIVES* We hope to implement a set of lips-control assistive communication system for the person with cerebral palsy by LabVIEW. The disabled can use the assistive communication system without wearing any auxiliary devices.To integrate the Human-Machine interface software system with the hardware system to accomplish the home appliance control system and the so called McTin system.To improve the efficiency and recognition rate of image processing algorithm.

SCHEMATIC DIAGRAM OF SYSTEM STRUCTURE6 Schematic Diagram of System Structuremorse codeuser

PAPER REVIEW * Ming-Hsuan Yang, Member, IEEE, David J. Kriegman, Senior Member, IEEE, and Narendra Ahuja, Fellow, IEEE, Detecting Faces in Images: A Survey, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, JANUARY 2002

SOFTWARE SYSTEM ARCHITECTURE*Block DiagramStage 1Stage 2Stage 3Stage 4

FACE DETECTION AND TRACKING ALGORITHM*Architecture

RGB ImagesInput

Color Space transformation

Convert the Extracted Skin Color Area toa Binary Image

Convert the Extracted Non-Skin Color Area to A Binary Image

Inverse Binary Image

MorphologyOPEN Operation

Particles Area Analysis

Morphology Convex HullOperation

Coordinates of the Skin Color Area

TemplateMatching

Lips TemplateInput

Coordinates of Lips Images

Lips Image in the Skin Color Area

Drawing aBoundary Rectangle

AND Operation

BoundaryRectangle Cancellation

Yes

No

Stage 3

Skin Color Detection

Face Verification and Tracking

Saturation

Hue

Reducing Image Size

MorphologyCLOSE Operation

MorphologyOPEN Operation

MorphologyCLOSE Operation

ALLOCATION AND EXTRACTION OF LIPS IMAGES *Choice of Multiple Templates of Lips AreaTarget- The wrong allocation must be avoided Step2- Allocation and extraction of lips Images

Step1- Calculation of rectangle boundary parametersR1(x,y)R2(x,y)Face RangeThe Center Coordinate of Rectangle Boundary The Center Coordinate of Rectangle Boundary :Lip area:Length= x 30, Wide: y 20 (Pixels) (1)Lips location in face area: High= y + 30, Wide= x (Pixels) ..(2)

PROCESSING AND RECOGNITION OF LIPS IMAGES *Architecture

Lips Images Input

RGB Multi-Threshold(dark)

MorphologyCLOSEOperation

Particles AreaAnalysis

MorphologyDILATEOperation

Taking particle with maximum area

Greater Thanthe Threshold ?

Logical Signal1Output

Logical Signal 0Output

Lips ImagesProcessing

Lips Images Recognition

Yes

No

Windows API What is Windows API? The Microsoft Windows application programming interface (API) provides building blocks used by applications written for Windows .

You can provide your application with a graphical user interface; display graphics and formatted text; and manage system objects such as memory, files, and processes. 17

*

Transformation FormulaH: Hue S: Saturation L: LuminanceInput Image(RGB)HueSaturationLuminanceColor Space Transformation (RGB HSL)Experimental Results of Face Photo in Lab

*Skin Color Detection-Binarization of H S Panel (1)Experimental Results of Face Photo in LabDecision Formula Binary Images of the Skin Color ObjectsThreshold Values of the Hue and the Saturation PanelBinary Image and Threshold ValuesRange of the Hue PanelBinary Image and Threshold ValuesRange of the Saturation Panel

SKIN COLOR DETECTION-MORPHOLOGY CLOSE OPERATION * Target- To fill background with non-skin color area Formula Skin color image area after CLOSE operation Original image Structure element of the erosion and dilation operation Erosion Operator Experimental ResultsDilation Operator Original Binary Image of the Hue PanelBinary Image of the Hue Panel After Dilation OperationBinary Image of the Hue PanelAfter Erosion Operation

SKIN COLOR DETECTION-MORPHOLOGY OPEN OPERATION * Target- Expanding the Face Area Formula Skin color image area after OPEN operation Original image Structure element of the erosion and dilation operation Erosion Operator Experimental ResultsDilation Operator Binary Image of the Hue PanelAfter Close OperationBinary Image of the Hue Panel After Erosion OperationBinary Image of the Hue PanelAfter Dilation Operation

*Skin Color Detection-Binarization of H S Panel (2)Experimental Results of Face Photo in LabBinary Image of the Hue PanelBinary Image of the Saturation Panel Obtaining the Skin-Color AreaCLOSEOPENCLOSE

*Skin Color Detection-Binarization of H S Panel (2)Experimental Results of Face Photo in LabLogicalDifferenceBinary Image of the Hue PanelSkin Color Area ImageBinary Image of the Saturation Panelinverse Obtaining the Skin-Color Area

SKIN COLOR DETECTION-SKIN COLOR PARTICLES AREA ANALYSIS * Target- Eliminating the bigger particles Algorithm Experimental ResultsThe smallest skin color particle area(Pixels)ith area base value after the skin colorparticles area analysis3nd IterationBinary Image after OPEN Operation 2nd Iteration10th Iteration

SKIN COLOR DETECTION-MORPHOLOGY CONVEX HULL OPERATION * Experimental ResultSkin Color Area Image AfterParticles Area Analysis The Result After Convex Hull Operation

EXPERIMENTAL RESULTS AND ALGORITHMS VERIFICATION*Result of Software SystemThe Software System Characteristics Lips Image Area Adjustable

Renewable Lips Template Image

Display Windows Management

Images Browser

System Status & Message Display

CONCLUSIONS* We utilize the Open/Close status of lips images successfully to real time control auxiliary devices for the disabled.

To decrease the interference of face detection because of light variation. Therefore, we can detect face more effectively.

Open/Close status of lips can be detected more stably .

System complexity and cost can be reduced because of our new software architecture developed.

System efficiency can be improved by updating hardware of computer.

FUTURE WORKS*Lab VIEW2012

(To control mouse)

REFERENCES (1)*[1] Ming-Hsuan Yang, Member, IEEE, David J. Kriegman, Senior Member, IEEE, and Narendra Ahuja, Fellow, IEEE, Detecting Faces in Images: A Survey, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, JANUARY 2002

[2] Yuemin Li, Jie Chen, Wen Gao, Baocai Yin, Face Detection: a Survey, 2004

[3] Vladimir Vezhnevets, Vassili Sazonov, Alla Andreeva, A survey on pixel-based skin color detection techniques, Graphicon-2003, Moscow, Russia, September 2003.

[4] IUT Informatique de Bayonne, Univ. de Pau et des Pays de l'Adour, Bayonne .,Real Time Tracking for 3D Realistic Lip Animation, Proceedings of the 18th International Conference on Pattern Recognition (ICPR'06), 2006

[5] Ching-Hsing Luo, Chung-Min Wu, Shu-Wen Lin, Tsan-Hsun Huang, Cheng-Hong Yang, Ming-Che Hsieh, Shih-Chung Chen and Chih-Kuo Liang, Mouth-Controlled Text Input Device with Sliding Fuzzy Algorithm for Individuals with Disabilities, IEEE instruement and measurement 2005 (submitted).

[ 6] fuzzy-, , , 2001

[7] LabVIEWTM PID Control Toolset User Manual

[8] LabVIEWTM Fuzzy Logic for G Toolkit Reference Manual

[9] LabVIEW & Microsoft (I), /, , 2004

[10] LabVIEWTM Using External Code in LabVIEW User Manual

REFERENCES (2)*[11] , , , , 2004

[12] (916)

[13] (926)

[14] (936)

[15] http://labview360.com/Default.asp labview

[16] http://msdn.microsoft.com/ msdn

THANKS FOR YOUR ATTENTION!

*

**outline,,,,, ,.*,,,,,,,,,,.* ,,,,,,,,,.*LABVIEW,,,MCTIN,,MCTIN,,mctin.*,system Accd,mctin,mctin,,,systemBmctin, pc,,api,ccdusbwebcam*survey4 1. Knowledge-based methods:2. Feature invariant approaches:3. Template matching methods:4. Appearance-based methods:*,webcam,,*,rgb,,,hcloseopen, ,close,open,,sclose,,and,,

,?,,,,,,*,,,,,,,60*40..,,,*,,,close,,,,1.0

*,window api,windows api?windows api,,,,labviewAPI,*RGB,hs*

*hclose,close,,.*open,close,*,hcloseopen,sclose,*hsand,,,open,,

*,, ,2310,10,,,

*,,,,,*,,,,,,,,,,, *