seungchan lee intelligent electronic systems human and systems engineering department of electrical...

Seungchan LeeIntelligent Electronic Systems

Human and Systems EngineeringDepartment of Electrical and Computer Engineering

Software Release and Support Vector Machine

Research Presentation:

ISIP: Research Presentation of 24

Overview

• Software Release Isip_lm_tester Isip_network_builder Debugging utility : Purify

• Verification System Isip_verify Support Vector Machine

• Audio File Generation

• Next Plans


Isip_lm_tester, Isip_network_builder

• Dummy Symbol generation problem Sentence generation terminated when met with dummy symbol

at the highest level. Dummy Symbol should not show at the output sentence.

Include Dummy Symbol check routine

• Exclude Symbol generation problem When turn on exclude symbol flag, lm_tester should not generate exclude

symbol. It need to modify HierarchicalSearch class.

Isip_network_buider Add save option for ABNF, BNF Need to correct dummy symbol generation problem

When generating dummy symbol without any subgraph, it generates error message.


Debugger Utility

• Purify What is problem?

Compilation error : When instrumenting purify, it generates error message. It is not easy to figure out the reason because we have so many linking

process when compiling.

How to resolve? Simple program without IFC class works fine Narrow down which classes are problem.

Exclude all linking process, and then add one class repeatedly.

Solution : After track down the linking process, I can figure out the problem is

originated from sphere utility. How can correct it?

I’m currently doing this.


Isip_verify

• When doing HMM training, it generates segmentation fault. This happens at the end of the program related to HierarchicalDigraph

object. Recently, we have many changes in IFC classes, but this problem might exists

sometimes ago.

• When doing SVM training, it generates checksum error. This error did not happen before I was recompiling whole repository.

isip_verify utilty also need to be throughly investigated using

purify utility


Audio File Generation

• Load one or two SWB CDs

• Select 100 conversations

• For each conversation, strip the NIST header

• Grab every other byte starting with the first byte (first channel) and put that into a raw audio file; PRESERVE the 8-bit ulaw data (do not use or convert to 16-bit)

• Convert this file to Sun ".au" using Sox


What to Learn?

• Audio File format .au file format

Widely used in UNIX machine and originated by SUN. Header + Variable length information + audio data Support various encoding types

NIST SPHERE file Raw format

PERL Programming Language It is simple programming language which performs extracting and printing out

information from a text file. Interpreted Language ( not compiled)

Conversion Utilities w_decode Sox “od” command


Why Support Vector Machine ?

• This is new learning technology to be noticed recently.

• Even though it has been situated as a subfield of machine learning, it still have many issues about theory and algorithm.

• To be more familiar with verification system, it is required to review one field for the next step.


How it works?

• Suppose we have low dimensional feature space. It is consist of positive examples and negative examples

0x

0x

• How about the following case?

• How can we classify this?

0x


How it works?

Simple idea : Low dimensional feature space map into high dimensional feature space using kernel function.

0x

2 ( , )k k k kx z x x


How can we determine maximum margin?

• To explain this, we need to know the following concepts.

Margin concepts

Lagrange multiplier

Primal and dual representation

Karush-Kuhn-Tucker Conditions (KKT)

Risk Bounds and Minimization

Maximal Margin classifier


Hyperplane

• Linear classification

function. thecontrolthat

parameters theare b)(w, , offunction linear a is )( where

)(

1

xx

xwx

f

bxw

bf

i

n

ii

0 bxw

• Input space X is split into two

parts by the hyperplane defined

by the equation

b

w

x xx

x

x

xx

Objective Function


Margin

x xx

x

x

xx

xxx

x

xx

x1

functional margin of an example (x,y) with respect to a hyperplane

.i i iy b

w

w x

Geometric margin of two points The margin of training set


Maximal Margin Classifier

• The Simplest model, but works only for data which are linearly separable in the feature space.

easy to understand and main building block for more complex SVMs

Margin

w

H1H2

• Plus-plane =

• Minus-plane =

• Separating hyperplane =

Classify as.. +1 if

-1 if

}1:{ bxwx}1:{ bxwx

1 bxw1 bxw

1 bxw0 bxw

1 bxw

{ : 0}b x w x



x

Margin

w

x

• Computing the margin width

2

1

2

1

w wx x

w w

w x w x

w

1 bxw0 bxw

1 bxw



2

2 M

w

x

Margin

w

x

• Computing the margin width

,...,l, i

,by

,

ii

1

1 subject to

minimize bw,

xw

ww

problemon optimizati thesolve that b)(w, hyperplane the

)),(),...,,((xS

sample trainingseparablelinearly aGiven

11 ll yxy

2w1/margin geometric with hyperplane maximal therealises

1 bxw0 bxw

1 bxw



x

Margin

w

x

• How to transform this optimization problem into dual problem?

l

iii

l

iiii

y

y

1

1

0

,

xw

, and respect to with atingdifferentiby found is dual ingcorrespond The

s.multiplier Lagange theare 0 where

12

1),,(

1

b

bybL

i

l

iii

w

xwwwαw

Hypothesis can be described as a linear combination of the training points.

1 bxw0 bxw

1 bxw

2

2 M

wLagrange



x

1 bxw

Margin

w

0 bxw1 bxw

x


.2

1

2

1

12

1),,(

1,1

11,1,

1

l

jijijiji

l

ii

l

ii

l

jijijiji

l

jijijiji

l

iii

yy

yyyy

bybL

xx

xxxx

xwwwαw

2

2 M

w



* *

1

*

2

Then the weight vector w realises the maximal margin hyperplane with

geometric margin 1/

l

i i ii

y

x

w

x

1 bxw

Margin

w

0 bxw1 bxw

x


*

1 , 1

Suppose the parameter solve the following quadratic optimization problem:

1maximize W( ) ,

2

l l

i i j i j i ji i j

y y

x x

1

subject to 0, 0, 1,..., .l

i i ii

y i l



x

1 bxw

Margin

w

0 bxw1 bxw

x


2

minmax *

1

*1

*iyiy

ii

bxwxw

* * * 1 0, 1,..., .i i iy b i l w x zero.-non

ingcorrespond

theare hyperplane

thecloset to lie

efore that therand one

ismargin functional

thefor which iput

:ctor Support ve

*

ix

Only these points are involved for the weight vector.*

SVi

*

*

1

***

),,(

by

bybf

ii

l

iii

xx

xxx



x

1 bxw

Margin

w

0 bxw1 bxw

x


zero.-non

ingcorrespond

theare hyperplane

thecloset to lie


ismargin functional

thefor which iput

:ctor Support ve

*

ix

,1),,( ****

SViiiii byybfy xxx

SVj

***

SVj

*

SVj

*

1,

**

.)1(

SViiij

jiiijj

l

jijijiji

by

yy

yy

xx

xxww **



x

1 bxw

Margin

w

0 bxw1 bxw

x


zero.-non

ingcorrespond

theare hyperplane

thecloset to lie


ismargin functional

thefor which iput

:ctor Support ve

*

ix

.1/

margin geometric with hyperplanemargin

maximal therealises tor weight vec

2/1*

2

1

*

SVii

l

iiiiy

w

xw


Review Maximal Margin SVMs

• Can be slow in practice

• Dose not control the number of support vector (Sparseness)

• Only one degree of freedom is the choice of kernel model selection

• Cannot be used non linear separable feature space

many real world problems deal with nonlinear, noisy data.

• However, it is a starting point for the more sophisticated SVMs.


Next Plan

• Software Release Resolve purify compilation problem Examine memory leak problem using purify utility Track down remaining bugs Test several cases

• Verification System Do NIST 2003 Experiment using new isip_verify Implemenation techniques of support vector machine Algorithm comparison between several SVM softwares Resolve memory leak problems


Reference

• An introduction to Support Vector Machines and other kernel-based learning methods by “Nello Cristianini and John Shawe-Taylor”, 2000, Cambridge Press

• Support Vector Machines Tutorial Slides by Andrew W. Moore

http://www.autonlab.org/tutorials/svm15.pdf

• Practical Perl Programming

http://www.cs.cf.ac.uk/Dave/PERL/

seungchan lee intelligent electronic systems human and systems engineering department of electrical...

Documents