seminar(pattern recognition)
TRANSCRIPT
Pattern Recognition
with
Semi-Supervised Learning
Algorithm
Presented By:-
Anurodh Kumar Sinha
2ND Year MSLIS Student
DRTC,ISI Bangalore
2012-2013
12/3/2012 1
Agenda
• What is Pattern Recognition?
• What is Machine Learning n why we
need..?
• Types of Learning Algorithm
• Need for Semi-Supervised Learning
• Conclusion
2 12/3/2012
What is a Pattern…. ?
• An entity, vaguely defined, that could be
given a name,
• e.g.:
– handwritten word,
– human face,
– fingerprint image,
– speech signal, 3 12/3/2012
What is Feature….?
• A Feature is an individual measurable heuristic property
of a phenomenon being observed
• Examples
• In speech recognition, features for recognizing
phonemes can include noise ratios, length of sounds,
relative power, filter matches and many others.
• In spam detection algorithms, features may include
whether certain email headers are present or absent,
whether they are well formed, what language the email
appears to be, the grammatical correctness of the text
4 12/3/2012
What is Pattern Recognition.. ?
• Pattern recognition is the study of how
machines can:
– observe the environment,
– learn to distinguish patterns of interest,
– make sound and reasonable decisions about
the categories of the patterns.
“The assignment of a physical object or event to
one of several prespecified categories” -- Duda
& Hart
5 12/3/2012
Motivation For The Study
of
Pattern Recognition It is threefold.
• In Artificial Intelligence, which is concerned with techniques, that enable
computers to do things, that seem intelligent when done by people.
• It is an important aspect of applying computers to do analysis and
classification of measurements, from its data observation.
• Pattern Recognition techniques provide a unified frame work to study a
variety of techniques with use of mathematics and computer science, which
helps the machine to make decision
7 12/3/2012
Methodology
of
Pattern Recognitions
It consists of the following:
1.We observe patterns
2.We study the relationships between the various
patterns.
3.We study the relationships between patterns and
ourselves and thus arrive at situations
4.We study the changes in situations and come to know
about the events.
5.We study events and thus find rule behind the events.
6. Using the rule, we can predict future events.
8 12/3/2012
An Example
• Suppose that: – A fish packing plant
wants to automate the process of sorting incoming fish on a conveyor belt according to species,
– There are two species: • Sea bass,
• Salmon.
9 12/3/2012
An Example
How to distinguish one specie from the other ?
(length, width, weight, number and shape of fins,
tail shape,etc.)
11 12/3/2012
An Example
• Suppose we also know that:
– Sea bass are typically wider than salmon.
– But it may happen that decision can‟t be
made on single feature
• We can use more than one feature for our
decision:
– Lightness (x1) and width (x2)
12 12/3/2012
Examples of applications
• Optical Character
Recognition (OCR)
• Biometrics
• Diagnostic systems
• Military applications
• Handwritten: sorting letters by postal code, input device for PDA‘s.
• Printed texts: reading machines for blind people, digitalization of text documents.
• Face recognition, verification, retrieval.
• Finger prints recognition.
• Speech recognition.
• Medical diagnosis: X-Ray, EKG analysis.
• Machine diagnostics, waster detection.
• Automated Target Recognition (ATR).
• Image segmentation and analysis (recognition from aerial or satelite photographs).
14 12/3/2012
What is Machine Learning….?
• Machine Learning algorithms discover the relationships
between the variables of a system (input, output and
hidden) from direct samples of the system
• These algorithms originate form many fields:
– Statistics, mathematics, theoretical computer science,
physics, neuroscience, etc
15 12/3/2012
16
Why Learning algorithms needed….?
• When the relationships between all system variables (input,
output, and hidden) is completely understood!
• This is NOT the case for almost any real system!
• Growing flood of online data
• Computational power is available
• progress in algorithms and theory
12/3/2012
Learning Algorithm Application
• Data mining: using historical data to improve decision
– medical records ⇒ medical knowledge
– log data to model user
• Software applications we can‟t program by hand
– autonomous driving
– speech recognition
• Self customizing programs
– Newsreader that learns user interests
17 12/3/2012
Typical Example
• 9714 patient records, each describing a pregnancy and birth
• Each patient record contains 215 features
• Classes of future patients at high risk for Emergency Cesarean Section
Learn to predict:
Given:
18 12/3/2012
19
The Sub-Fields
of
Machine Learning
• Supervised Learning
• Unsupervised Learning
• Semi-Supervsed Learning
12/3/2012
Supervised Learning
• Supervised learning is the machine learning task of inferring a function from labeled training data.
• In training data each pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal).
• A supervised learning algorithm analyzes the training data and produces an inferred function, which is called a classifier (if the output is discrete) or a regression function (if the output is continuous).
• The inferred function should predict the correct output value for any valid input object. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way.
21 12/3/2012
Supervised Learning Process: two
Steps Learning (training): Learn a model using the training data
Testing: Test the model using unseen test data to assess the model accuracy
,cases test ofnumber Total
tionsclassificacorrect ofNumber Accuracy
12/3/2012 22
Example • A credit card company receives thousands of
applications for new cards. Each application contains information about an applicant, – age
– Job
– House
– credit rating
– etc.
• Problem: to decide whether an application should approved, or to classify applications into two categories, approved and not approved.
12/3/2012 23
25
An example: The Learning Task
• Learn a classification model from the data
• Use the model to classify future loan applications
into
– Yes (approved) and
– No (not approved)
• What is the class for following case/instance?
Bayesian Classifier
• The Simple Bayesian Classifier (SBC) uses probabilistic
methods for classification
• The basis of bayesian classifier is: The probability of document
„d‟ being in class „c‟ is computed as-
where P(tk|c) is the conditional probability of term occurring in a
document of class c .Where,
26 12/3/2012
• Organizing data into classes such that there is Inter-clusters distance maximized Intra-clusters distance minimized
• Finding the class labels and the number of classes directly from the data
(in contrast to classification).
• More informally, finding natural groupings among objects.
What is Unsupervised
Learning….? • Unsupervised learning refers to the problem of trying to
find hidden structure in unlabeled data
• Sometimes it is also referred as Clustering
30 12/3/2012
School Employees Simpson's Family Males Females
Clustering is subjective
What is a natural grouping among these objects?
32 12/3/2012
What is clustering for….? Let us see some real-life examples
• Example 1: Groups people of similar sizes together to
make “small”, “medium” and “large” T-Shirts.
– Tailor-made for each person: too expensive
– One-size-fits-all: does not fit all.
• Example 2: Given a collection of text documents, we want to organize them according to their content similarities,
– To produce a topic hierarchy
12/3/2012 33
What is clustering for? (cont…) In fact, clustering is one of the most utilized data mining techniques
– It has a long history, and used in almost every field,
e.g., medicine, psychology, botany, sociology, biology, archeology, marketing, insurance, libraries, etc.
– In recent years, due to the rapid increase of online documents, text clustering becomes important.
12/3/2012 34
Supervised Learning
versus
Unsupervised Learning
• Unsupervised clustering Group similar objects together
to find clusters • Minimize intra-class distance
• Maximize inter-class distance
• Supervised classification Class label for each training
sample is given
– Build a model from the training data
– Predict class label on unseen future data points
39 12/3/2012
However, for many problems, labeled
data can be rare or expensive.
Unlabeled data is much cheaper.
Speech
Images
Medical outcomes
Customer modeling
Protein sequences
Web pages
Need to pay someone to do it, requires special testing,…
40 12/3/2012
Why Semi-Supervised Learning…?
• Why not clustering?
– The clusters produced may not be the ones required.
– Sometimes there are multiple possible groupings.
• Why not classification?
– Sometimes there are insufficient labeled data.
41 12/3/2012
Semi-Supervised Learning
• Combines labeled and unlabeled data
during training to improve performance: – Semi-supervised classification: Training on labeled data exploits
additional unlabeled data, frequently resulting in a more accurate
classifier.
– Semi-supervised clustering: Uses small amount of labeled data to
aid and bias the clustering of unlabeled data.
Unsupervised clustering
Semi-supervised learning
Supervised classification
42 12/3/2012
Semi-Supervised Classification
• An initial classifier is designed using the labeled data set D(l).
This classifier is then used to assign class labels to examples
in D(u). Then the classifier is re-trained using D(l) U D(u).
• The last two steps are usually repeated for a given number of
times or until some criterion is satisfied
43 12/3/2012
Semi-Supervised Classification
• Algorithms:
– Semisupervised EM
[Ghahramani:NIPS94,Nigam:ML00].
– Co-training [Blum:COLT98].
– Transductive SVM‟s [Vapnik:98,Joachims:ICML99].
– Graph based algorithms
• Assumptions:
– Known, fixed set of categories given in the labeled
data.
– Goal is to improve classification of examples into
these known categories. 46 12/3/2012
Semi-Supervised clustering • Input:
– A set of unlabeled objects, each described by a set of attributes
(numeric and/or categorical)
– A small amount of domain knowledge
• Output:
– A partitioning of the objects into k clusters (possibly with some
discarded as outliers)
• Objective:
– Maximum intra-cluster similarity
– Minimum inter-cluster similarity
– High consistency between the partitioning and the domain
knowledge
47 12/3/2012
How Semi-Supervised Clustering done?
• In addition to the similarity information used by unsupervised
clustering, in many cases a small amount of knowledge is available
concerning either pairwise (must-link or cannot-link) constraints
between data items or class labels for some items.
• Instead of simply using this knowledge for the external validation of
the results of clustering, one can imagine letting it “guide” or “adjust”
the clustering process, i.e. provide a limited form of supervision. The
resulting approach is called semi-supervised clustering
48 12/3/2012
• According to different given domain knowledge: – Users provide class labels (seeded points) a priori to
some of the documents
-Users know about which few documents are related
(must-link) or unrelated (cannot-link)
Semi-Supervised Clustering
Seeded points
Must-link
Cannot-link
51 12/3/2012
Semi-supervised Clustering Algorithm
• Semi-supervised Clustering with labels (Partial label information is given ) : – SS-Seeded-Kmeans ( Sugato Basu, et al. ICML 2002)
- SS-Constraint-Kmeans ( Sugato Basu, et al. ICML 2002)
• Semi-supervised Clustering with Constraints (Pairwise Constraints (Must-link, Cannot-link) is given): – SS-COP-Kmeans (Wagstaff et al. ICML01)
– SS-HMRF-Kmeans (Sugato Basu, et al. ACM SIGKDD 2004)
– SS-Kernel-Kmeans (Brian Kulis, et al. ICML 2005)
– SS-Spectral-Normalized-Cuts (X. Ji, et al. ACM SIGIR 2006)
52 12/3/2012
Conclusions
• Semi-supervised learning is an area of increasing
importance in Machine Learning.
• Automatic methods of collecting data make it more
important than ever to develop methods to make use
of unlabeled data.
• Several promising algorithms (only discussed a few).
Also new theoretical framework to help guide further
development.
54 12/3/2012
Reference
• Duda, Heart: Pattern Classification and Scene Analysis. J. Wiley &
Sons, New York, 1982. (2nd edition 2000).
• Fukunaga: Introduction to Statistical Pattern Recognition. Academic
Press, 1990.
• Sergios Theodoridis, Konstantinos Koutroumbas , pattern recognition
, Pattern Recognition ,Elsevier(USA)) ,1982
• K. Nigam and R. Ghani. Analyzing the effectiveness and applicability
of co-training. In Proceedings of the ninth international conference on
Information and knowledge management, pages 86{93. ACM, 2000.
• http://nlp.stanford.edu/IR-book/html/htmledition/naive-bayes-text-
classification-1.html
• http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/kmeans.htm
l
12/3/2012 55