classification metrics
TRANSCRIPT
Binary Classification Error Measurement1) AUC2) RIG 3) LogLoss4) Precision/Recall5) F16) PE, MSE, MAE
AUC1) ignores the predicted probability values2) usually we are interested in parts of
roc curve3) considers Type 1 error and Type 2
error weights equivalently4) dependent on the underlying distribution
of data
RIG
1) bad to compare two model performances with different distributions
2) can be used to compare the relative performance of multiple models trained and tested on the same data
3) is not informative, because score also depends on the data distribution
Thanks!
J. Yi, Y. Chen, J. Li, S. Sett, and T. W. Yan. Predictive model performance: Offline and online evaluations. In KDD, pages 1294–1302,
2013.http://chbrown.github.io/kdd-2013-usb/kdd/p1294.pdf