spoken dialog systems and voice xml pattern recognition spoken dialog systems and voice xml : intro...
TRANSCRIPT
![Page 1: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/1.jpg)
Spoken Dialog Systems and Voice Spoken Dialog Systems and Voice XMLXML :
Intro to Pattern RecognitionPattern Recognition
Esther Levin
Dept of Computer Science
CCNY
Some materials used in this course were taken from the textbook “Pattern Classification” by Duda et al., John Wiley & Sons, 2001 with the permission of the authors and the publisher
![Page 2: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/2.jpg)
Credits and AcknowledgmentsMaterials used in this course were taken from the textbook “Pattern Classification” by Duda et al., John Wiley & Sons, 2001 with the permission of the authors and the publisher; and also fromOther material on the web:
Dr. A. Aydin Atalan, Middle East Technical University, Turkey Dr. Djamel Bouchaffra, Oakland University Dr. Adam Krzyzak, Concordia University Dr. Joseph Picone, Mississippi State University Dr. Robi Polikar, Rowan University Dr. Stefan A. Robila, University of New Orleans Dr. Sargur N. Srihari, State University of New York at Buffalo David G. Stork, Stanford University Dr. Godfried Toussaint, McGill University Dr. Chris Wyatt, Virginia Tech Dr. Alan L. Yuille, University of California, Los Angeles Dr. Song-Chun Zhu, University of California, Los Angeles
![Page 3: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/3.jpg)
Outline
IntroductionWhat is this pattern recogntiion
Background MaterialProbability theory
![Page 4: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/4.jpg)
![Page 5: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/5.jpg)
PATTERN RECOGNITION AREASOptical Character Recognition ( OCR)
Sorting letters by postal code. Reconstructing text from printed materials (such as reading machines for blind
people).Analysis and identification of human patterns
Speech and voice recognition. Finger prints and DNA mapping.
Banking and insurance applications Credit cards applicants classified by income, credit worthiness, mortgage amount, # of
dependents, etc. Car insurance (pattern including make of car, #of accidents, age, sex, driving habits,
location, etc).Diagnosis systems
Medical diagnosis (disease vs. symptoms classification, X-Ray, EKG and tests analysis, etc).
Diagnosis of automotive malfunctioning Prediction systems
Weather forecasting (based on satellite data). Analysis of seismic patterns
Dating services (where pattern includes age, sex, race, hobbies, income, etc).
![Page 6: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/6.jpg)
More Pattern Recognition Applications
SENSORYVision Face/Handwriting/Hand
Speech Speaker/Speech
Olfaction Apple Ripe?
DATA
Text Categorization
Information Retrieval
Data Mining
Genome Sequence Matching
![Page 7: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/7.jpg)
What is a pattern?What is a pattern?“A pattern is the opposite of a chaos; it is an entity
vaguely defined, that could be given a name.”
![Page 8: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/8.jpg)
PR Definitions
Theory, Algorithms, Systems to Put
Patterns into Categories
Classification of Noisy or Complex Data
Relate Perceived Pattern to Previously
Perceived Patterns
![Page 9: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/9.jpg)
Characters
Ç ş ğ İ ü Ü Ö Ğچك٤٧ع
К Ц Д
ζ ω Ψ Ω ξ θ
א ם ש ת ד נ
A v t u I h D U w K
![Page 10: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/10.jpg)
Handwriting
![Page 11: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/11.jpg)
![Page 12: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/12.jpg)
Terminology
Features, feature vector
Decision boundary
Error
Cost of error
Generalization
![Page 13: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/13.jpg)
A Fishy Example I
“Sorting incoming Fish on a conveyor according to species using optical sensing”
Salmon or Sea Bass?
![Page 14: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/14.jpg)
Problem Analysis
Set up a camera and take some sample images to extract features
Length Lightness Width Number and shape of fins Position of the mouth, etc…
This is the set of all suggested features to explore for use in our classifier!
![Page 15: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/15.jpg)
Solution by Stages
Preprocess raw data from camera
Segment isolated fish
Extract features from each fish (length,width, brightness, etc.)
Classify each fish
![Page 16: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/16.jpg)
PreprocessingUse a segmentation operation to isolate fishes
from one another and from the background
Information from a single fish is sent to a feature extractor whose purpose is to reduce the data by measuring certain features
The features are passed to a classifier
2
![Page 17: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/17.jpg)
2
![Page 18: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/18.jpg)
Classification
Select the length of the fish as a possible feature for discrimination
2
![Page 19: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/19.jpg)
2
![Page 20: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/20.jpg)
The length is a poor feature alone!
Select the lightness as a possible feature.
2
![Page 21: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/21.jpg)
2
![Page 22: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/22.jpg)
Threshold decision boundary and cost relationship Move our decision boundary toward smaller values
of lightness in order to minimize the cost (reduce the number of sea bass that are classified salmon!)
Task of decision theory
2
“Customers do not want sea bass in their cans of salmon”
![Page 23: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/23.jpg)
Adopt the lightness and add the width of the fish
Fish x = [x1, x2]
Lightness Width
2
![Page 24: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/24.jpg)
2
![Page 25: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/25.jpg)
We might add other features that are not correlated with the ones we already have. A precaution should be taken not to reduce the performance by adding such “noisy features”
Ideally, the best decision boundary should be the one which provides an optimal performance such as in the following figure:
2
![Page 26: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/26.jpg)
2
![Page 27: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/27.jpg)
However, our satisfaction is premature because the central aim of designing a classifier is to correctly classify novel input
Issue of generalization!
2
![Page 28: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/28.jpg)
2
![Page 29: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/29.jpg)
Decision BoundariesObserve: Can do much better with two features
Caveat: overfitting!
![Page 30: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/30.jpg)
Occam’s Razor
Entities are not to be multiplied without necessity
William of Occam (1284-1347)
![Page 31: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/31.jpg)
A Complete PR System
![Page 32: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/32.jpg)
Problem Formulation
Measurements &
PreprocessingClassificationFeatures
Inputobject
ClassLabel
Basic ingredients:•Measurement space (e.g., image intensity, pressure)•Features (e.g., corners, spectral energy)•Classifier - soft and hard•Decision boundary•Training sample•Probability of error
![Page 33: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/33.jpg)
Pattern Recognition Systems
SensingUse of a transducer (camera or microphone)PR system depends of the bandwidth, the
resolution, sensitivity, distortion of the transducer
Segmentation and groupingPatterns should be well separated and
should not overlap
3
![Page 34: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/34.jpg)
3
![Page 35: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/35.jpg)
Feature extraction Discriminative features Invariant features with respect to translation, rotation and
scale.
Classification Use a feature vector provided by a feature extractor to
assign the object to a category
Post Processing Exploit context dependent information other than from the
target pattern itself to improve performance
![Page 36: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/36.jpg)
The Design Cycle
Data collection
Feature Choice
Model Choice
Training
Evaluation
Computational Complexity
4
![Page 37: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/37.jpg)
4
![Page 38: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/38.jpg)
Data Collection
How do we know when we have collected an adequately large and representative set of examples for training and testing the system?
4
![Page 39: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/39.jpg)
Feature Choice
Depends on the characteristics of the problem domain. Simple to extract, invariant to irrelevant transformation insensitive to noise.
4
![Page 40: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/40.jpg)
Model Choice
Unsatisfied with the performance of our linear fish classifier and want to jump to another class of model
4
![Page 41: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/41.jpg)
Training
Use data to determine the classifier. Many different procedures for training classifiers and choosing models
4
![Page 42: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/42.jpg)
Evaluation
Measure the error rate (or performance) and switch from one set of features & models to another one.
4
![Page 43: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/43.jpg)
Computational Complexity
What is the trade off between computational ease and performance?
(How an algorithm scales as a function of the number of features, number or training examples, number patterns or categories?)
4
![Page 44: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/44.jpg)
Learning and AdaptationLearning: Any method that combines empirical information from the environment with prior knowledge into the design of a classifier, attempting to improve performance with time.Empirical information: Usually in the form of training examples.Prior knowledge: Invariances, correlations
Supervised learning A teacher provides a category label or cost for each pattern in the
training set
Unsupervised learning The system forms clusters or “natural groupings” of the input patterns
5
![Page 45: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/45.jpg)
Syntactic Versus Statistical PR
Basic assumption: There is an underlying regularity behind the observed phenomena.Question: Based on noisy observations, what is the underlying regularity?Syntactic: Structure through common generative mechanism. For example, all different manifestations of English, share a common underlying set of grammatical rules.Statistical: Objects characterized through statistical similarity. For example, all possible digits `2' share some common underlying statistical relationship.
![Page 46: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/46.jpg)
Difficulties
Segmentation
Context
Temporal structure
Missing features
Aberrant data
Noise
Do all these images represent an `A'?
![Page 47: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/47.jpg)
Design Cycle
How do we know what features to select, and how do we select them…?
What type of classifier shall we use. Is there best classifier…?
How do we train…?How do we combine prior knowledge withempirical data?
How do we evaluate our performanceValidate the results. Confidence in decision?
![Page 48: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/48.jpg)
Conclusion
I expect you are overwhelmed by the number, complexity and magnitude of the sub-problems of Pattern Recognition
Many of these sub-problems can indeed be solved
Many fascinating unsolved problems still remain
6
![Page 49: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/49.jpg)
Toolkit for PRStatisticsDecision TheoryOptimizationSignal ProcessingNeural NetworksFuzzy LogicDecision TreesClusteringGenetic AlgorithmsAI SearchFormal Grammars….
![Page 50: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/50.jpg)
Linear algebra
Matrix A:
Matrix Transpose
Vector a
mnmm
n
n
nmij
aaa
aaa
aaa
aA
...
............
...
...
][
21
22221
11211
mjniabAbB jiijT
mnij 1,1;][
],...,[;... 1
1
nT
n
aaa
a
a
a
![Page 51: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/51.jpg)
Matrix and vector multiplication
Matrix multiplication
Outer vector product
Vector-matrix product
)()(,][
;][;][
BcolArowcwherecCAB
bBaA
jiijnmij
npijpmij
matrixnmanABbac
bBbaAa nijT
mij
,
;][;][ 11
mlengthofvectormatrixmanAbC
bBbaA nijnmij
1
;][;][ 1
![Page 52: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/52.jpg)
Inner ProductInner (dot) product:
Length (Eucledian norm) of a vectora is normalized iff ||a|| = 1
The angle between two n-dimesional vectorsAn inner product is a measure of collinearity: a and b are orthogonal iff
a and b are collinear iff
A set of vectors is linearly independent if no vector is a linear combination of other vectors.
n
iii
T baba1
n
ii
T aaaa1
2
||||||||cos
ba
baT
0baT
|||||||| babaT
![Page 53: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/53.jpg)
Determinant and Trace
Determinant
det(AB)= det(A)det(B)
Trace
)det()1(
;,....1;)det(
;][
1
ijji
ij
n
jijij
nnij
MA
niAaA
aA
n
jjjnnij aAtraA
1
][;][
![Page 54: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/54.jpg)
Matrix Inversion
A (n x n) is nonsingular if there exists B
A=[2 3; 2 2], B=[-1 3/2; 1 -1]
A is nonsingular iff
Pseudo-inverse for a non square matrix, provided
is not singular
1; ABIBAAB n
0|||| A
TT AAAA 1# ][ AAT
IAA #
![Page 55: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/55.jpg)
Eigenvectors and Eigenvalues
1||||;,...,1, jjjj enjeAe
0]det[ nIA
n
jjAtr
1
][
Characteristic equation:n-th order polynomial, with n roots.
n
jjA
1
]det[
![Page 56: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/56.jpg)
Probability Theory
Primary references: Any Probability and Statistics text book (Papoulis) Appendix A.4 in “Pattern Classification” by Duda
et al
The principles of probability theory, describing the behavior of systems with random characteristics, are of fundamental importance to pattern recognition.
![Page 57: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/57.jpg)
![Page 58: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/58.jpg)
![Page 59: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/59.jpg)
![Page 60: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/60.jpg)
![Page 61: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/61.jpg)
![Page 62: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/62.jpg)
![Page 63: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/63.jpg)
![Page 64: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/64.jpg)
![Page 65: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/65.jpg)
![Page 66: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/66.jpg)
![Page 67: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/67.jpg)
![Page 68: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/68.jpg)
![Page 69: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/69.jpg)
![Page 70: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/70.jpg)
![Page 71: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/71.jpg)
![Page 72: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/72.jpg)
![Page 73: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/73.jpg)
![Page 74: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/74.jpg)
Example 1 ( wikipedia)•two bowls full of cookies.
•Bowl #1 has 10 chocolate chip cookies and 30 plain cookies,•bowl #2 has 20 of each.
•Fred picks a bowl at random, and then picks a cookie at random.
•The cookie turns out to be a plain one. •How probable is it that Fred picked it out of bowl •what’s the probability that Fred picked bowl #1, given that he has a plain cookie?”
•event A is that Fred picked bowl #1, •event B is that Fred picked a plain cookie. •Pr(A|B) ?
![Page 75: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/75.jpg)
Example1 - cpntinuedTables of occurrences and relative frequenciesIt is often helpful when calculating conditional probabilities to create a simple table containing the number of occurrences of each outcome, or the relative frequencies of each outcome, for each of the independent variables. The tables below illustrate the use of this method for the cookies.
Number of cookies in each bowlby type of cookie
Relative frequency of cookies in each bowl
by type of cookie
The table on the right is derived from the table on the left by dividing each entry by the total number of cookies under consideration, or 80 cookies.
Bowl 1 Bowl 2 Totals
Chocolate Chip 10 20 30
Plain 30 20 50
Total 40 40 80
Bowl #1 Bowl #2 Totals
Chocolate Chip 0.125 0.250 0.375
Plain 0.375 0.250 0.625
Total 0.500 0.500 1.000
![Page 76: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/76.jpg)
Example 2
1. Power Plant Operation. The variables X, Y, Z describe
the state of 3 power plants (X=0 means plant X is idle).
Denote by A an event that a plant X is idle, and by B an event that 2 out of three plants are working.
What’s P(A) and P(A|B), the probability that X is idle given that at least 2 out of three are working?
X Y Z P(x,y,z)
0 0 0 0.07
0 0 1 0.04
0 1 0 0.03
0 1 1 0.18
1 0 0 0.16
1 0 1 0.18
1 1 0 0.21
1 1 1 0.13
![Page 77: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/77.jpg)
P(A) = P(0,0,0) + P(0,0,1) + P(0,1,0) + P(0, 1, 1) = 0.07+0.04 +0.03 +0.18 =0.32
P(B) = P(0,1,1) +P(1,0,1) + P(1,1,0)+ P(1,1,1)= 0.18+ 0.18+0.21+0.13=0.7
P(A and B) = P(0,1,1) = 0.18
P(A|B) = P(A and B)/P(B) = 0.18/0.7 =0.257
![Page 78: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/78.jpg)
2. Cars are assembled in four possible locations. Plant I supplies 20% of the cars; plant II, 24%; plant III, 25%; and plant IV, 31%. There is 1 year warrantee on every car.
The company collected data that shows
P(claim| plant I) = 0.05; P(claim|Plant II)=0.11;
P(claim|plant III) = 0.03; P(claim|Plant IV)=0.18;
Cars are sold at random.
An owned just submitted a claim for her car. What are the posterior probabilities that this car was made in plant I, II, III and IV?
![Page 79: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/79.jpg)
P(claim) = P(claim|plant I)P(plant I) +
P(claim|plant II)P(plant II) +
P(claim|plant III)P(plant III) +
P(claim|plant IV)P(plant IV) =0.0687
P(plant1|claim) =
= P(claim|plant I) * P(plant I)/P(claim) = 0.146
P(plantII|claim) =
= P(claim|plant II) * P(plant II)/P(claim) = 0.384
P(plantIII|claim) =
= P(claim|plant III) * P(plant III)/P(claim) = 0.109
P(plantIV|claim) =
= P(claim|plant IV) * P(plant IV)/P(claim) = 0.361
![Page 80: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/80.jpg)
Example 33. It is known that 1% of population suffers from a
particular disease. A blood test has a 97% chance to identify the disease for a diseased individual, by also has a 6% chance of falsely indicating that a healthy person has a disease.
a. What is the probability that a random person has a positive blood test.
b. If a blood test is positive, what’s the probability that the person has the disease?
c. If a blood test is negative, what’s the probability that the person does not have the disease?
![Page 81: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/81.jpg)
A is the event that a person has a disease. P(A) = 0.01; P(A’) = 0.99.
B is the event that the test result is positive. P(B|A) = 0.97; P(B’|A) = 0.03; P(B|A’) = 0.06; P(B’|A’) = 0.94;
(a) P(B) = P(A) P(B|A) + P(A’)P(B|A’) = 0.01*0.97 +0.99 * 0.06 = 0.0691
(b) P(A|B)=P(B|A)*P(A)/P(B) = 0.97* 0.01/0.0691 = 0.1403
(c) P(A’|B’) = P(B’|A’)P(A’)/P(B’)= P(B’|A’)P(A’)/(1-P(B))= 0.94*0.99/(1-.0691)=0.9997
![Page 82: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/82.jpg)
Sums of Random Variables
z = x + y
Var(z) = Var(x) + Var(y) + 2Cov(x,y)
If x,y independent: Var(z) = Var(x) + Var(y)
Distribution of z:
yxz
dxxzpxpypxpzp yxyx
)()()()()(
![Page 83: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/83.jpg)
Examples:
x and y are uniform on [0,1]Find p(z=x+y), E(z), Var(z);
x is uniform on [-1,1], and P(y)= 0.5 for y =0, y=10; and 0 elsewhere.Find p(z=x+y), E(z), Var(z);
![Page 84: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/84.jpg)
![Page 85: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/85.jpg)
![Page 86: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/86.jpg)
![Page 87: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/87.jpg)
![Page 88: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/88.jpg)
Normal Distributions
Gaussian distribution
Mean
Variance
Central Limit Theorem says sums of random variables tend toward a Normal distribution.
Mahalanobis Distance:
xxE )(
22/2)(
2
1),()( xxx
x
eNxp xx
22])[(xx
xE
x
xxr
![Page 89: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/89.jpg)
![Page 90: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/90.jpg)
Multivariate Normal Densityx is a vector of d Gaussian variables
Mahalanobis Distance
All conditionals and marginals are also Gaussian
dxxpxxxxE
dxxxpxE
xTxe
dNxp
TT )())((]))([(
)(][
)(1)(21
2/1||2/2
1),()(
)()( 12 xxr T
![Page 91: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/91.jpg)
![Page 92: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/92.jpg)
![Page 93: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/93.jpg)
Bivariate Normal Densities
Level curves - elliplses.x and y width are determined by the
variances, and the eccentricity by correlation coefficient
Principal axes are the eigenvectors, and the width in these direction is the root of the corresponding eigenvalue.
![Page 94: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/94.jpg)
Information theoryKey principles: What is the information contained in a
random event? Less probable event contains more information For two independent event, the information is a sum
What is the average information or entropy of a distribution?
)(log)( 2 xPxI
)(log)()( 2 xPxPxHx
![Page 95: Spoken Dialog Systems and Voice XML Pattern Recognition Spoken Dialog Systems and Voice XML : Intro to Pattern Recognition Esther Levin Dept of Computer](https://reader036.vdocument.in/reader036/viewer/2022062518/56649e7e5503460f94b81acd/html5/thumbnails/95.jpg)
Examples: uniform distribution, dirac distribution;
Mutual information: reduction in uncertainty about one variable due to knowledge of other variable.
yx
yxypxp
yxpyxpyxHxHI
,2,
)()(
),(log),()|()(