guiding semi- supervision with constraint-driven learning ming-wei chang,lev ratinow, dan roth
TRANSCRIPT
![Page 1: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/1.jpg)
Guiding Semi-Supervision with
Constraint-Driven Learning
Ming-Wei Chang ,Lev Ratinow , Dan Roth
![Page 2: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/2.jpg)
• Semi -super vised Learning ? • Scarcity of Training Data • What are constraints ?• How/why do they help ?
![Page 3: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/3.jpg)
Supervised learning
( X1Y1) Labelled Data
(X2-Y2)
(X3 Y3).. ……(XnYn) .
What if n is less ? .. Obtaining training data is Costly and it could be inefficient . Example : (Fraud detection / Anamoly detection)
Domain expertise helps……
![Page 4: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/4.jpg)
Definitions • X = (X1,X2,X3,X4…………Xn)• Y = (Y1,Y2,Y3,Y4…………Yn)
• H : XY is a classifier .
f : (Cross product of X and Y ) -R set of real numbers
• The out-put of the classifier will be such y which maximizes the value of function f
![Page 5: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/5.jpg)
• Classification function .. • It’s a linear sum of feature
functions
![Page 6: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/6.jpg)
Motivational Interviewing
Labels : Support,Reflection,Cofrontation,Facilitate, Question
![Page 7: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/7.jpg)
Can we exploit knowledge of constraints in Inference Phase? • Lets assume n items (observations) in sequence and p labels.. i.e., n tokens and p parts of speech or n tokens and p tags in an NER task
Brute Force : O(n power p )
Viterbi : O( N power P)
Can we go down further ? Can we further reduce our search space
Further down ?
![Page 8: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/8.jpg)
Introducing constraints into Model• Let C1, C2 ……….CK be the constraints
• C: (Cross product of X and Y) {0,1}
• Constraints are of two types . • Hard (MUST be satisfied)• Soft (Can be relaxed)
• 1Cx is the set of sequence labels that DON’T violate the constraints
![Page 9: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/9.jpg)
Constraints come to rescue • Lets say x out of X possible tag sequences violate the constraints .
• Search space comes from X to X-x .• How do we infer ? • Does Viterbi help us ?
![Page 10: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/10.jpg)
Example
A B C D E F G
S1 X1 X1 X1 X1 X1 X1 X1
S2 X10 X10 X10 X10 X10 X10 X10 S3 X11 X11 X11 X11 X11 X1I X11
Motivational Interviewing :
At least ONE reflection
![Page 11: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/11.jpg)
Soft constraints
How do we calculate distance here ?
How do we learn the parameters ?
![Page 12: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/12.jpg)
Lars Ole Andersen. Program Analysis and Specialization for the C programming Language . PhD Thesis , DIKU , University of Copenhagen, May 1994.This is Ground Truth .
But HMM gives this. Lars Ole Andersen. Program Analysis and Specialization for the C Programming Language . PhD Thesis , DIKU , University of Copenhagen, May 1994.
![Page 13: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/13.jpg)
![Page 14: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/14.jpg)
![Page 15: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/15.jpg)
Top-k inference
We only chose the few top possible sequences and add ALL of of them to training data.
The author used beam search decoding, but this can be done with any inference procedure.
From the Unlabeled sample, we label them and include them in the training data.
Choice : We may include only the high confident samples.
PitFall : Then we don’t really learn properly and miss-out some characteristics
![Page 16: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/16.jpg)
Algorithm:
![Page 17: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/17.jpg)
![Page 18: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/18.jpg)
![Page 19: Guiding Semi- Supervision with Constraint-Driven Learning Ming-Wei Chang,Lev Ratinow, Dan Roth](https://reader035.vdocument.in/reader035/viewer/2022062421/56649c765503460f94929e89/html5/thumbnails/19.jpg)