wrapping snakes for improved lip segmentation

Wrapping SnakesFor Improved Lip Segmentation

Matthew Ramage

Dr Euan Lindsay (Supervisor)

Department of Mechanical Engineering

Presentation Overview

Brief overview of Visual Speech Recognition

Requirements of a good Lip SegmenterOne approach: Traditional SnakesAn improved approach: “Wrapping

Snakes”

What is Visual Speech Recognition?

Uses video footage to identify spoken wordsDoes not use any audio information

Audio-only speech recognition accuracy falls dramatically as audio signal degraded

Basic Principle:Mouth shape is based on the underlying

structure of the word being spoken

What makes a good Lip Segmenter?

Accurately represents lip shapeRobust against poor initialisation

Can handle multiple enclosed regionsReliably handles noisy images

Can handle weak lip features near strong noise

Quickly finds the lip shape

One Approach: Traditional Snakes

A form of Active Contour Model

Series of connected points Internal forces:

Tension Rigidity

External forces: Constraints

Image Forces: Based on image features Using snakes to track the outer

lip boundary (Kass et al. 1988)

Kass, M, Witkin, A & Terzopoulos, D, 1988, 'Snakes: Active contour models', International Journal of Computer Vision, vol. 1, no. 4, pp. 321-31.

Traditional Snakes

Problems with Traditional Snakes

Not robust against poor initialisationCannot handle multiple enclosed regions

Cannot reliably handle noisy imagesCannot handle weak target features with

strong noise

Multiple Enclosed Regions

(Traditional Snake)

Weak Features & Strong Noise

(Traditional Snake)

The Problem?

Traditional Snakes don’t continue along features

The Problem?

Traditional Snakes don’t continue along features

The Solution: A Wrapping Force

A substitute for the Image ForceBased on the Image Force

Modified by the snake’s shape and position

When a snake curves away from a feature, apply a force along the featurePushes snake along the feature

Calculating the Wrapping Force

Component of Image Force in the direction of the Snake’s Normal

Wrapping Behaviour

Cutting the Snake

Cut the snake when separate sections come into contact

Benefits:Distinguishes between multiple fully

enclosed regions Ignores areas of noise near the lipsAllows the lip shape to be accurately found Increased robustness of lip segmentation

Pinching Forces

A subset of points a made “Pinch Points”

Non-adjacent pinch points attract when close enough

Pulls sections of the snake together when close enoughHelps with cutting

Multiple Enclosed Regions

(Wrapping Snake)

Weak Features & Strong Noise

(Wrapping Snake)

Properties of Wrapping Snakes

Accurately represents lip shapeRobust against poor initialisation

Can handle multiple enclosed regionsReliably handles noisy images

Can handle weak lip features near strong noise

Quickly finds the lip shape

Wrapping SnakesFor Improved Lip Segmentation

Matthew Ramage

Dr Euan Lindsay (Supervisor)

Department of Mechanical Engineering

Handles Very Poor Initial Position

(Wrapping Snake)

Multiple Enclosed Regions

(Traditional Snake)

Multiple Enclosed Regions

(Wrapping Snake)