introduction to machine learning
TRANSCRIPT
1/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Machine Learning using PythonTheory and Implementation
Vishal Kumar Jaiswal
M.TechCST Department , IIEST Shibpur
June 18, 2016
Summer Internship 2016
2/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Table of Contents
1 IntroductionMachine Learning
2 FundamentalsInductive learningSupervised LearningUnsupervised LearningReinforcement Learning
3 Survey Method4 Mathematical Preliminaries5 Learning Algorithms6 Entropy7 Ensemble Learning Method
3/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
What to expect
You’ll learn by doing,Bring machine learning to life using specified use case,Identify financial fraud by mining email datasets toidentify writer of email.
4/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Why machine learning?
Automating automationGetting computers to program themselvesWriting software is the bottleneckLet the data do the work instead!Can help in solve the most practical society impactingproblems.
5/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Comparison with traditional programming
ComputerData
Program
Output
Traditional Programming:-
ComputerData
Output
Program
Machine Learning:-
6/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Machine learning
Building computational artifacts that learn over timebased on experience.Employed in computing tasks where designing andprogramming explicit algorithms is infeasible.Build a model from example inputs in order to makedata-driven predictions or decisions expressed as outputsrather than following strictly static program instructions.
7/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Inductive Learning
Process of learning by example - where a system tries toinduce a general rule from a set of observed instances i.e.specific to general rule.
1 Discrete:- Classification2 Continuous:- Regression
Constructing class definitions is called Inductive learning.Note:Artificial Intelligence is based on deductive learning.
7/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Inductive Learning
Process of learning by example - where a system tries toinduce a general rule from a set of observed instances i.e.specific to general rule.
1 Discrete:- Classification2 Continuous:- Regression
Constructing class definitions is called Inductive learning.Note:Artificial Intelligence is based on deductive learning.
8/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Classification of Machine Learning
Based on feedback available to learning system, we categorizemachine learning algorithms in following categories:-
Supervised learningUnsupervised learningReinforcement learning
9/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Machine Learning: Type 1
Supervised learningFunction approximation from input and output pairs (trainingdataset) i.e. generalization.
10/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Supervised Learning
ClassificationAssign,to a particular input, the name of class to which itbelongs.
11/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Classification: An Example
Spam filtering and Object detection
12/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Supervised learning
RegressionPredicting a numerical value for input dataset.
13/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Regression
stock market prediction and weather forcasting
14/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Machine learning: Type 2
Unsupervised learningWithout any examples, summerize or describe data just bylooking at the input.
e.g. naming all animals dog by a child. Describe people basedon ethinicity like Punjabi, Bangali, South Indian etc.
14/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Machine learning: Type 2
Unsupervised learningWithout any examples, summerize or describe data just bylooking at the input.
e.g. naming all animals dog by a child. Describe people basedon ethinicity like Punjabi, Bangali, South Indian etc.
15/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Unsupervised Learning
ClusteringDiscovering structure in data e.g. finding similiar images ingoogle images, Collaborative filtering of news feed content byfacebook.
16/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Clustering: An Example
Google news
17/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Machine learning: Type 3
Reinforcement learningLearn how to behave based on delayed feedback from theenvironment.
e.g. Figuring out mistakes after wrong answer in test seriesduring preparation and correcting your bias afterwards.
Note:1 All are supplement of each other, not an alternative.2 Fundamentally different, but trying to achieve the same
goal.
17/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Machine learning: Type 3
Reinforcement learningLearn how to behave based on delayed feedback from theenvironment.
e.g. Figuring out mistakes after wrong answer in test seriesduring preparation and correcting your bias afterwards.
Note:1 All are supplement of each other, not an alternative.2 Fundamentally different, but trying to achieve the same
goal.
17/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Machine learning: Type 3
Reinforcement learningLearn how to behave based on delayed feedback from theenvironment.
e.g. Figuring out mistakes after wrong answer in test seriesduring preparation and correcting your bias afterwards.
Note:1 All are supplement of each other, not an alternative.2 Fundamentally different, but trying to achieve the same
goal.
18/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Survey Methodology
Collect data for machine learning by conducting survey orstoring sensor output data stream.Preprocess data to remove inconsistencies in it.Before believing a survey result, check:
1 How many people or situations are surveyed.2 Who is surveyed.3 How survey is conducted.
Train and test machine learning algortihm on different setsof data, otherwise overfitting will happen.
19/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Bayes’ Theorem
Conditional ProbabilityProbability of event obtained with additional information thatsome other event has already occurred.
P(B|A) = P(A ∩ B)
P(A)
Assumption: We are dealing with sequential events and the newinformation is used to revise the probability of previous events.
20/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Bayes’ Theorem
Prior probabilityInitial probability value obtained before any additionalinformation is provided.
Posterior ProbabilityThe probability value that has been revised by using additionalinformation that is obtained later.
21/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Bayes’ Theorem
DefinitionThe probability of event A, given that event B hassubsequently occurred, is
P(A|B) =P(A) · P(B|A)
[P(A) · P(B|A)] + [P(A) · P(B|A)]
22/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Bayes’ Rule: An example
A specific cancer, C , incurs in 1% of population, P(C)=0.01Pathological Test:Probability of positive test result, if you have C ,P(Pos|C)=90% (Sensitivity)Probability of negative test result, if you don’t have C ,P(Pos|C)=90% (Specificity) So, find out
Probability of having cancer if test positive
22/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Bayes’ Rule: An example
A specific cancer, C , incurs in 1% of population, P(C)=0.01Pathological Test:Probability of positive test result, if you have C ,P(Pos|C)=90% (Sensitivity)Probability of negative test result, if you don’t have C ,P(Pos|C)=90% (Specificity) So, find out
Probability of having cancer if test positive
23/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = 0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1
All People
C
C+Pos Pos but not C
No C Test Neg.
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
24/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Prior Probabilities:P(C) = 0.01 P(¬C) = 0.99P(Pos|C) = ‘0.9P(Neg |¬C) = 0.9 P(Pos|¬C) = 0.1Joint Probability:P(C ,Pos) = P(C)× P(Pos|C) = 0.009P(¬C ,Pos) = P(¬C)× P(Pos|¬C) = 0.099Normalize:P(Pos) = P(¬C ,Pos) + P(C ,Pos) = 0.108
Posterior: P(C |Pos) = P(C ,Pos)P(Pos) = 0.083
P(¬C |Pos) = P(¬C ,Pos)P(Pos) = 0.917
So, Total Probability=1
25/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Naive Bayes Method
Supervised learning algorithm based on Bayes’ theorem.The ”naive” assumption of independence between everypair of features.works quite well for many real-world applications. e.g.document classification and spam filtering.extremely fastdecent classifier, but bad estimator.
25/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Naive Bayes Method
Supervised learning algorithm based on Bayes’ theorem.The ”naive” assumption of independence between everypair of features.works quite well for many real-world applications. e.g.document classification and spam filtering.extremely fastdecent classifier, but bad estimator.
25/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Naive Bayes Method
Supervised learning algorithm based on Bayes’ theorem.The ”naive” assumption of independence between everypair of features.works quite well for many real-world applications. e.g.document classification and spam filtering.extremely fastdecent classifier, but bad estimator.
25/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Naive Bayes Method
Supervised learning algorithm based on Bayes’ theorem.The ”naive” assumption of independence between everypair of features.works quite well for many real-world applications. e.g.document classification and spam filtering.extremely fastdecent classifier, but bad estimator.
25/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Naive Bayes Method
Supervised learning algorithm based on Bayes’ theorem.The ”naive” assumption of independence between everypair of features.works quite well for many real-world applications. e.g.document classification and spam filtering.extremely fastdecent classifier, but bad estimator.
25/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Naive Bayes Method
Supervised learning algorithm based on Bayes’ theorem.The ”naive” assumption of independence between everypair of features.works quite well for many real-world applications. e.g.document classification and spam filtering.extremely fastdecent classifier, but bad estimator.
26/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Gaussian Naive Bayes
DefinitionThe liklihood of features is:
P(xi |y) =1√
2πσ2y
exp(−(xi − µ)2
2σ2y
)
27/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Anupam Rahul
.1.8
.1 .5.2
.3
LOVEDEAL LIFE LIFEDEALLOVE
LOVE LIFE!
P (Anupam) = 0.5
P (Rahul) = 0.5
LOVE LIFE!
LIFE DEAL
Guess: (Owner of email)
CalculateP( ” LIFE DEAL ”| Anupam) 0.8× 0.1× 0.5P( ”LIFE DEAL” | Rahul) 0.2× 0.3× 0.5
27/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Anupam Rahul
.1.8
.1 .5.2
.3
LOVEDEAL LIFE LIFEDEALLOVE
LOVE LIFE!
P (Anupam) = 0.5
P (Rahul) = 0.5
LOVE LIFE!
LIFE DEAL
Guess: (Owner of email)
CalculateP( ” LIFE DEAL ”| Anupam) 0.8× 0.1× 0.5P( ”LIFE DEAL” | Rahul) 0.2× 0.3× 0.5
27/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Anupam Rahul
.1.8
.1 .5.2
.3
LOVEDEAL LIFE LIFEDEALLOVE
LOVE LIFE!
P (Anupam) = 0.5
P (Rahul) = 0.5
LOVE LIFE!
LIFE DEAL
Guess: (Owner of email)
CalculateP( ” LIFE DEAL ”| Anupam) 0.8× 0.1× 0.5P( ”LIFE DEAL” | Rahul) 0.2× 0.3× 0.5
27/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Anupam Rahul
.1.8
.1 .5.2
.3
LOVEDEAL LIFE LIFEDEALLOVE
LOVE LIFE!
P (Anupam) = 0.5
P (Rahul) = 0.5
LOVE LIFE!
LIFE DEAL
Guess: (Owner of email)
CalculateP( ” LIFE DEAL ”| Anupam) 0.8× 0.1× 0.5P( ”LIFE DEAL” | Rahul) 0.2× 0.3× 0.5
28/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Support Vector Machine
Supervised learning used for classification, regressionanalysis and outlier detection.Often used as a black box and for comparative study.Support Vectors are set of frontier datapoints of trainingdataset (may be linear).Effective in high dimensional space.
29/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Support Vectors
Maximum margin
Optimal Hyperplane
Pictorial representation of support vector machine
30/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
# Support vector classification from sklearn modulefrom sklearn.svm import SVC# Kernel: default: ’rbf’, alternativess: ’linear’,’poly’,’rbf’ etc.clf=SVC(kernel="linear",gamma=1000.0,C=1)#Fit SVM model to according to the training dataclf.fit(features_train,labels_train)# Perform classification on test setpred=clf.predict(features_test)
31/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Decision Trees
Supervised algorithmMost popular, simple and extremely robust.Non-linear decision making with simple linear surfaces.
Disadvantage:over-complex trees that do not generalize well ondatapoints(overfitting). Pruning is required.Learning optimal decision tree is NP-complete.
32/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Wind
Sun
Weather pattern for Windsurf
Decision trees allows us to use multiple linear boundaries toseperate two non-linearly seperable regions.
32/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Wind
Sun
Weather pattern for Windsurf
Decision trees allows us to use multiple linear boundaries toseperate two non-linearly seperable regions.
33/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Wind
Sun
Multiple linear decision boundary for weather pattern
Windy
Yes No
Sunny
Yes No
Symbolic representation of decision tree for weather data
34/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Classification
DecisionTreeClassifier is a class capable of performingmulti-class classification on a dataset.from sklearn import treeX=[[0,0],[1,1]]Y=[0,1]clf=tree.DecisionTreeClassifier(min_samples_split=50)clf=clf.fit(X,Y)clf.predict([[2.,2.]])
35/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Entropy
DefinitionMeasure of impurity in a bunch of examples.
Entropy,H(P) =∑−Pi log2Pi
Controls how a decision tree decides to split the data, e.g.
36/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Information Gain
DefinitionThe change in information entropy H from a prior state to astate that takes some information as given:
IG(T , a) = H(T )− H(T |a)
Decision trees tries to maximize information gain.
37/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
K Nearest Neighbors Classification Algorithm
k closest training examples in the feature space areprovided as input.Classification is done by a majority vote of neighbors.The object is assigned to the class most common amongits k neighbors.
Illustration of K nearest neighbors
38/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
K Nearest Neighbors Classification Example
# Support vector classification from sklearn modulefrom sklearn.svm import SVC# Kernel: default: ’rbf’, alternativess: ’linear’,’poly’,’rbf’ etc.clf=SVC(kernel="linear",gamma=1000.0,C=1)#Fit SVM model to according to the training dataclf.fit(features_train,labels_train)# Perform classification on test setpred=clf.predict(features_test)
39/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Ensemble Learning Method
Supervised Learning MethodologyUses multiple learning methods to obtain betterperformance.Prediction requires more computational overhead.Some unsupervised learning methods include consensusclustering and anamoly detection.
40/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Random Forest
Ensemble learning methodAvoids overfitting of decision trees by building multipledeep decision treesParameters include number of estimators.
Ensemble Learning Methodology
41/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Adaptive Boosting, Adaboost
Best known classifierAdaptive in the sense that subsequent weak learners aretweaked in favor of those instances misclassified byprevious classifiers.Sensitive to noisy data and outliersAdaboostClassifier
42/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Numerical e.g. Salary info.Categorical e.g. Job TitleTime Series e.g. time stamps on emailsText e.g. Contents of emails
43/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
References
Bayes’ Theorem.
UD262: Machine Learning.
Ali Farhadi.Cse446: Machine learning.University of Washington.
Sebastian Thrun Katie Malone.Ud102: Intro to machine learning.Udacity.
44/ 44
Vishal KumarJaiswal
IntroductionMachine Learning
FundamentalsInductive learning
Supervised Learning
UnsupervisedLearning
ReinforcementLearning
SurveyMethod
MathematicalPreliminaries
LearningAlgorithms
Entropy
EnsembleLearningMethod
Thank You