Download - ROC curve estimation
![Page 1: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/1.jpg)
ROC curve estimation
![Page 2: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/2.jpg)
Index
• Introduction to ROC• ROC curve• Area under ROC curve• Visualization using ROC curve
![Page 3: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/3.jpg)
ROC curve
• Originally stands for Receiver Operating Characteristic curve.
• It is used widely in biomedical applications like radiology and imaging.
• An important utility here is to assess classifiers in machine learning.
![Page 4: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/4.jpg)
Example situation
• Consider diagnostic test for a disease
• Test has 2 possible outcomes:
• Positive or negative.
• Now based on this we will explain the various notations used in ROC curves in the next slide.
![Page 5: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/5.jpg)
Data distribution available
Test Result
Pts Pts with with diseasdiseasee
Pts Pts without without the the diseasedisease
![Page 6: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/6.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
Threshold
![Page 7: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/7.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
True Positives
Some definitions ...
![Page 8: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/8.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
False Positives
![Page 9: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/9.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
True negatives
![Page 10: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/10.jpg)
Test Result
Call these patients “negative”
Call these patients “positive”
without the diseasewith the disease
False negatives
![Page 11: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/11.jpg)
Confusion Matrix
• Confusion matrix is defined as a matrix consisting of two rows and two columns.
• The orientation of entries in the confusion matrix is as follows if say the confusion matrix is called CMat.
• Then CMat[1][1]=True Positives CMat[1][2]=False Positives.
• Similarly CMat[2][1]=False Negatives and CMat[2][2]=True Negatives.
![Page 12: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/12.jpg)
2-class Confusion Matrix
• Reduce the 4 numbers to two ratestrue positive rate = TP = (#TP)/(#P)false positive rate = FP = (#FP)/(#N)
• Rates are independent of class ratio*
True class
Predicted class
positive negative
positive (#P) #TP #P - #TP
negative (#N) #FP #N - #FP
![Page 13: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/13.jpg)
Comparing classifiers using Confusion Matrix
True
Predicted
pos neg
pos 60 40
neg 20 80
True
Predicted
pos neg
pos 70 30
neg 50 50
True
Predicted
pos neg
pos 40 60
neg 30 70
Classifier 1TP = 0.4FP = 0.3
Classifier 2TP = 0.7FP = 0.5
Classifier 3TP = 0.6FP = 0.2
![Page 14: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/14.jpg)
Interpretations from the Confusion matrix
• The following metrics for a classifier can be calculated using the confusion matrix. These can be used for evaluating the classifier.
• Accuracy = (TP+TN)• Precision = TP/(TP+FP)• Recall = TP/(TP+FN)• F-Score = 2*recall*precision/(recall +
precision)
![Page 15: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/15.jpg)
Tru
e P
osi
tive R
ate
(s
en
siti
vit
y)
0%
100%
False Positive Rate (1-specificity)
0%
100%
ROC curve
![Page 16: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/16.jpg)
Tru
e P
osi
tive
Ra
te
0%
100%
False Positive Rate0%
100%
Tru
e P
osi
tive
Ra
te
0%
100%
False Positive Rate0%
100%
A good test: A poor test:
ROC curve comparison
![Page 17: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/17.jpg)
Area under ROC curve (AUC)
• Overall measure of test performance
• Comparisons between two tests based on differences between (estimated) AUC
• For continuous data, AUC equivalent to Mann-Whitney U-statistic (nonparametric test of difference in location between two populations)
• Determines the accuracy of a classifier in machine learning.
![Page 18: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/18.jpg)
Tru
e P
osi
tive
Ra
te
0%
100%
False Positive Rate
0%
100%
Tru
e P
osi
tive
R
ate
0%
100%
False Positive Rate
0%
100%
Tru
e P
osi
tive
R
ate
0%
100%
False Positive Rate
0%
100%
AUC = 50%
AUC = 90% AUC =
65%
AUC = 100%
Tru
e P
osi
tive
R
ate
0%
100%
False Positive Rate
0%
100%
AUC for ROC curves
![Page 19: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/19.jpg)
Further Evaluation methods
• ROC curve based visualization• The visualization of the ROC curve is
a very good method of evaluating the classifier.
• Tools like Matlab, Weka and Orange provide facilities to support visualization of the ROC curve.
![Page 20: ROC curve estimation](https://reader036.vdocument.in/reader036/viewer/2022072016/56813152550346895d97cde4/html5/thumbnails/20.jpg)
• ROCR is one such tool which provides effective visualization.