Generaliza)ons,of,support,vector,machines,

Elinor,Velasquez,,Bioinforma)cs,Department,

University,of,California,,Santa,Cruz,

Outline,

•  The,problem,that,we,are,trying,to,solve,•  An,example,

•  Support,vector,machines,defined,

•  Rela)on,to,convolu)onal,neural,nets,•  SVMs,generalized,to,Gelfand,pairs,

•  Some,new,geometrical,generaliza)ons,for,learning,new,features,and,for,classifica)on,

The,problem,we,are,trying,to,solve,

•  Given,data,from,a,cancer,pa)ent,,classify,the,subtype,of,the,cancer.,

The,problem,we,are,trying,to,solve,

•  What,is,the,data?,Suppose,we,have,gene,expression,intensity,data.,

•  The,gene,expression,intensity,increases,when,the,gene,is,“on”,(upPregulated),and,decreases,when,the,gene,is,“off”,(downPregulated).,

Microarray,datachip,(Ohio,State,CompBio,Lab),

Example,

•  Suppose,there,are,only,two,types,of,lung,cancer,,named,AC,and,SCC.,

•  Data,=,{(xi,,yi)}i=1,,…,,N,with,xi,a,MPdimensional,vector,of,gene,expression,intensity,values,(M,genes),

•  yi,=,Dirac,delta,func)on,(1,if,AC,,0,if,SCC),for,,,,,,,i,=,1,,…,,N,pa)ent,samples.,

We,want,to,classify,the,data,samples,

•  How,to,do,this?,•  Use,supervised,learning,method:,Support,vector,machines,

•  Supervised,learning,means,to,predict/classify,

•  Unsupervised,learning,is,used,to,uncover,structure,in,the,data,,

A,support,vector,machine,(SVM),is,a,type,of,graphical,model,(Vapnik,,1995),

x1,

x2,

xM,

+1,

Output,=,f(wTx,+,b),,,,,,,,,,,,,,,=,ypredicted(

Loss,func)on,=,max,(0,,1,–,yf(x)),

Input,

Supervised,learning,method,

Generalize,SVMs,to,convolu)onal,neural,nets,

•  Why?,,•  Convolu)onal,neural,nets,are,performing,beeer,than,SVMs,for,a,variety,of,data,types.,

Convolu)onal,neural,net,is,a,graphical,learning,model,(LeCun,,2006),

x1,

x2,

xN,

+1,

h1,

h1,

h1,

h2,

h2,

h2,

Input,Output,

Supervised,learning,method,

We,want,our,data,to,be,invariant,under,rigid,mo)ons,and,small,deforma)ons,

•  Mallat,(2010),solved,this,problem,for,data,learned,in,convolu)onal,neural,nets:,He,used,a,scaeering,operator,,S,,on,f,ε,L2(RM),which,was,composed,of,a,wavelet,transform,,convolu)on,operator,and,modulus,operator.,,

•  But,data,lies,on,a,more,complicated,manifold.,

Construc)ng,data,invariant,to,rigid,mo)ons,and,small,deforma)ons,

•  Let,G,be,a,locally,compact,group,and,K,a,compact,subgroup,(Gelfand,pair),

•  G,approximates,a,complicated,manifold.,•  Example:,Let,G,=,group,of,inver)ble,matrices,,K,=,orthogonal,group,

•  Suppose,we,can,map,the,data,to,G,(True,for,Lie,groups),•  Then,,build,a,convolu)onal,neural,net,for,the,data,on,G/K,,an,“invariant”,space.,

•  Will,extend,Mallat’s,work,(work,in,progress),

An,ideal,data,space,

•  The,data,actually,lives,on,a,manifold,X.,•  Map,the,data,to,X/G,,with,X,a,manifold,and,G,,its,group,of,invariances,,using,geometric,invariant,theory,tools.,

•  Build,a,convolu)onal,neural,network,on,this,data,space,,X/G.,,

•  (Work,in,progress),

Can,we,improve,upon,convolu)onal,neural,nets?,

x1,

x2,

xN,

h1,

h1,

h1,

h2,

h2,

h2,

h3,

A,cylindrical,neural,net,is,called,a,Recurrent,Neural,Net,(Graves,1980s),

We,can,make,a,convolu)onal,recurrent,neural,net,for,classifica)on,

x1,

x2,

xN,

h1,

h1,

h1,

h2,

h2,

h2,

h3, Aeach,a,recursive,neural,net,to,the,recurrent,neural,net:,

h1,

h2,

Cylinder,(Recurrent,neural,net),

Triangle,(Recursive,neural,net),

Output,

We,can,pinch,the,cylinder,on,each,end,to,make,a,convolu)onal,neural,net,

sphere,minus,the,south,pole,

h3,

h2,

h1,

x1,

h3,

h2,

h1,

x2,

h1, h1,

h2, h2,

h3,

Output,

We,can,construct,a,convolu)onal,neural,net,on,a,smooth,manifold,of,

genus,G,

h9,

h2,

h3,

h4,

h5,

h6,

h7,h8,

h1,

Output,

h1,

h2,

h3,

h4,

h5,h9,

h6,

h7,

h8,

Summary,

•  Convolu)onal,neural,nets,are,learning,models,which,classify,data.,

•  Data,actually,lives,on,a,complicated,manifold.,

•  We,can,remove,data,invariances,before,learning,to,remove,redundancy.,

•  The,future,holds,many,possibili)es,for,learning,models.,

Thank,you!,