learning classifiers for non-iid data

Learning Classifiers For Non-IID Data Balaji Krishnapuram, Computer-Aided Diagnosis and TherapySiemens Medical Solutions, Inc.

Collaborators: Volkan Vural, Jennifer Dy [North Eastern], Ya Xue [Duke], Murat Dundar, Glenn Fung, Bharat Rao [Siemens]

Jun 27, 2006

*2005 Siemens Medical Solutions. All rights reserved. Siemens confidential Computer-aided Diagnosis & Therapy

OutlineImplicit IID assumption in traditional classifier design

Often, not valid in real life. Motivating CAD problems

Convex algorithms for Multiple Instance Learning (MIL)

Bayesian algorithms for Batch-wise classification Faster, approximate algorithms via mathematical programming

Summary / Conclusions


IID assumption in classifier designTraining data D={(xi,yi)i=1N: xi 2 Rd, yi 2 {+1,-1}}, Testing data T ={(xi,yi)i=1M: xi 2 Rd, yi 2 {+1,-1}},

Assume each training/testing sample drawn independently from identical distribution:(xi,yi) ~ PXY(x,y)

This is why we can classify one test sample at a time, ignoring the features of the other test samplesEg. Logistic Regression: P(yi=1|xi,w)=1/(1+exp(-wT xi))


Evaluating classifiers: learning-theoryBinomial test set bounds: With high probability over the random draw of M samples in testing set T, if M large and a classifier w is observed to be accurate on T, with high probability its expected accuracy over a random draw of a sample from PXY(x,y) will be high

If the IID assumption fails, all bets are off !Thought experiment: repeat same test sample M times


Training classifiers: learning theoryWith high probability over the random draw of N samples in training set D, the expected accuracy on a random sample from PXY(x,y) for the learnt classifier w will be high iffaccurate on the training set D; and N largesatisfies intuition before seeing data (prior, large margin etc)

PAC-Bayes, VC-theory etc rely on iid assumptionRelaxation to exchangeability being explored


CAD: Correlations among candidate ROI


Hierarchical Correlation Among Samples


Additive Random Effect ModelsThe classification is treated as iid, but only if given bothFixed effects (unique to sample) Random effects (shared among samples)

Simple additive model to explain the correlationsP(yi|xi,w,ri,v)=1/(1+exp(-wT xi vT ri)) P(yi|xi,w,ri)=s P(yi|xi,w,ri,v) p(v|D) dv Sharing vT ri among many samples correlated prediction

But only small improvements in real-life applications


CAD detects early stage colon cancer


Candidate Specific Random Effects Model: PolypsSpecificitySensitivity


CAD algorithms: domain-specific issuesMultiple (correlated) views: one detection is sufficient

Systemic treatment of diseases: e.g. detecting one PE sufficient

Modeling the data acquisition mechanismErrors in guessing class labels for training set.


The Multiple Instance Learning ProblemA bag is a collection of many instances (samples)

The class label is provided for bags, not instances

Positive bag has at least one +ve instance in it

Examples of bag definition for CAD applications:Bag=samples from multiple views, for the same regionBag=all candidates referring to same underlying structureBag=all candidates from a patient


CH-MIL Algorithm: 2-D illustration


CH-MIL Algorithm for Fishers DiscriminantEasy implementation via Alternating OptimizationScales well to very large datasetsConvex problem with unique optima


Lung CADDR CADLung Nodules& Pulmonary EmboliComputed TomographyAX*Pending FDA Approval


CH-MIL: Pulmonary Embolisms



CH-MIL: Polyps in Colon


Classifying a Correlated Batch of SamplesLet classification of individual samples xi be based on uiEg. Linear ui = wT xi ; or kernel-predictor ui= j=1N j k(xi,xj)

Instead of basing the classification on ui, we will base it on an unobserved (latent) random variable zi

Prior: Even before observing any features xi (thus before ui), zi are known to be correlated a-priori, p(z)=N(z|0,)

Eg. due to spatial adjacency = exp(-D), Matrix D=pair-wise dist. between samples


Classifying a Correlated Batch of SamplesPrior: Even before observing any features xi (thus before ui), zi are known to be correlated a-priori, p(z)=N(z|0,)

Likelihood: Let us claim that ui is really a noisy observation of a random variable zi :p(ui|zi)=N(ui|zi, 2)

Posterior: remains correlated, even after observing the features xiP(z|u)=N(z|(-12+I)-1u, (-1+2I)-1)Intuition: E[zi]=j=1N Aij uj ; A=(-12+I)-1


SVM-like Approximate AlgorithmIntuition: classify using E[zi]=j=1N Aij uj ; A=(-12+I)-1What if we used A=( + I) instead?Reduces computation by avoiding inversion.Not principled, but a heuristic for speed.Yields an SVM-like mathematical programming algorithm:


Detecting Polyps in Colon


Detecting Pulmonary Embolisms


Detecting Nodules in the Lung


ConclusionsIID assumption is universal in MLOften violated in real life, but ignoredExplicit modeling can substantially improve accuracyDescribed 3 models in this talk, utilizing varying levels of informationAdditive Random Effects Models: weak correlation informationMultiple Instance Learning: stronger correlations enforcedBatch-wise classification models: explicit information Statistically significant improvement in accuracyOnly starting to scratch the surface, lots to improve!


learning classifiers for non-iid data

Documents

random sample

random effects

random draw of n samples

expwt xi

learnt classifier w

learning classifiers

iid assumptionrelaxation

trainingtesting sample