![Page 1: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/1.jpg)
Classification and Machine Learningtechniques for CBIR: introduction to the
RETIN system
Matthieu Cord
ETIS CNRS UMR 8051
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.1/81
![Page 2: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/2.jpg)
Content-Based Image Retrieval
• Retrieve large categories of pictures in generalistimage database
• Vector-based description of images• User interaction• Statistical learning approach→ Multimodality (category retrieval)→ Efficient strategies in text retrieval→ Interactive strategies (active learning)
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.2/81
![Page 3: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/3.jpg)
Outline
1. Binary Classification for CBIR
2. Active learning:(a) Error Reduction and Uncertainly-Based
strategies(b) RETIN scheme: Boundary Correction and
diversity
3. Semi-supervised classification
4. Long Term Learning
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.3/81
![Page 4: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/4.jpg)
Supervised Classification for CBIR
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.4/81
![Page 5: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/5.jpg)
Introduction
• Vector-based description of images;
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.5/81
![Page 6: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/6.jpg)
Introduction
• binary classification
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.6/81
![Page 7: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/7.jpg)
Supervised Classification
Three representative methods for CBIR:• Bayes Classifiers (Vasconcelos)• k-Nearest Neighbors• Support Vector Machines (Chapelle)
Specific characteristics [Chang ICIP’03]:
(c1) High dimension and non-linearity of input space
(c2) Few training data
(c3) Many unlabelled data
(c4) Interactive learning (Relevance feedback)
(c5) Unbalanced training dataClassification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.7/81
![Page 8: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/8.jpg)
Support Vector Machines (1/4)
Classification by an hyperplan:
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.8/81
![Page 9: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/9.jpg)
Support Vector Machines (2/4)
Choose the hyperplan which maximizes the margin:
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.9/81
![Page 10: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/10.jpg)
Support Vector Machines (3/4)
Quadratic problem:
α⋆ = argmax
α
n∑
i=1
αi −1
2
n∑
i,j=1
αiαjyiyj < xi,xj >
with
n∑
i=1
αiyi = 0
∀i ∈ [1, n] 0 ≤ αi ≤ C
Decision function:
f(x) =
n∑
i=1
yiα⋆i < x,xi > +b
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.10/81
![Page 11: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/11.jpg)
Support Vector Machines (3/4)
Support Vectors:
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.11/81
![Page 12: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/12.jpg)
"Kernelization"
Kernelization of SVM:• SVM decision function:
f(x) =
N∑
i=1
yiα⋆i < x,xi > +b (1)
• "Kernelized" version:
f(x) =N∑
i=1
yiα⋆i k(x,xi) + b (2)
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.12/81
![Page 13: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/13.jpg)
Kernels
Dealing with the class of kernels k corresponding todot product in an induced space H via a map Φ:
Φ : Rp → H
x 7→ Φ(x)
that is k(x,x′) =< Φ(x),Φ(x′) >
Initial: X Induit: Φ(X)Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.13/81
![Page 14: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/14.jpg)
Kernels
• Usual kernels: Lin., Polynomial, Sigmoid, RBF ...• Choice of a kernel depends on the database and
its usage:
→ Different levels of performances for two differentkernels;
• In our experiments: Gaussian kernels give thebest results→ The most adapted to CBIR;→ In the following experiments: Gaussian kernels
with χ2 distance, because feature vector aredistributions.
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.14/81
![Page 15: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/15.jpg)
Spectral analysis of kernel matrices
0 200 400 600 800 10000
0.2
0.4
0.6
0.8
1
1.2
Valeur propre
Ene
rgie
Gaussien Chi2TriangulairePolynomiale 3Gaussien L2Linéaire
Performances
Large distribution ⇒ high performances;Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.15/81
![Page 16: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/16.jpg)
SVM and Kernels
Deal with (c1) high dimension and non-linear inputspace:
→ Use of a kernel function to induce a feature space
→ Relevance function f using Kernel in SVM:
f(x) =
N∑
i=1
yiα⋆i k(x,xi) + b
When a method cannot be directly "kernelized":KPCA.
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.16/81
![Page 17: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/17.jpg)
Experiments
Protocol:• COREL Photo database (6,000 images);• 50 categories, 100-300 size;• Training set of 200 points (unbalanced).• Statistical measure: Mean Average Precision MAP
Methods MAP(%) TimeNo learning 8 -
Bayes/Parzen 18 0.09sk-NN 16 0.20sSVM 20 0.13s
• SVM selected [Gosselin CVDB04]Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.17/81
![Page 18: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/18.jpg)
Experiments
Training with 10 examples => poor top-similarityranking results
→ User interaction (c4) to enhance the retrieval
2 components: the parameter tuning of f and theoptimization of the set of examples
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.18/81
![Page 19: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/19.jpg)
Active learning for CBIR
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.19/81
![Page 20: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/20.jpg)
Active Learning
Deal with the few training data (c2) and interactivelearning (c4) characteristics
→ optimize training data to get the best classificationwith as few as possible user labeling
Strategies of selective sampling :• Relevance-Based (RB):→ Select the most relevant image• Uncertainly-Based (UB)• Error Reduction (ER)→ Priority to the classification error minimization
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.20/81
![Page 21: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/21.jpg)
Labelling the most relevant (RB)
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.21/81
![Page 22: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/22.jpg)
Labelling the most relevant (RB)
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.22/81
![Page 23: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/23.jpg)
Labelling the most relevant (RB)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.23/81
![Page 24: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/24.jpg)
Labelling the most relevant (RB)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.24/81
![Page 25: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/25.jpg)
Labelling the most difficult toclassify (UB)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.25/81
![Page 26: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/26.jpg)
Labelling the most difficult toclassify (UB)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.26/81
![Page 27: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/27.jpg)
Active Learning
The aim of an active learner is to select the mostinteresting picture x⋆
→ We propose to express the following methods asthe minimization of a cost function g(x):
x⋆ = argminx
g(x)
For Relevance-Based active learning: g(x) = −f(x)where f(x) is the relevance function
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.27/81
![Page 28: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/28.jpg)
Uncertainly-based (UB)
Active learner:x⋆ = argmin
x
g(x)
• UB strategy selects the picture which is the mostdifficult to classify:
g(x) = |f(x)|
• Method:• SVMactive (Tong):→ Works in the version space→ Needs an accurate estimation of the
boundary
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.28/81
![Page 29: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/29.jpg)
Error Reduction (ER)
• ER strategy (Roy and McCallum): select thepicture which will minimize the new expected testerror:
g(x) =∑
c∈{−1,1}
EP̂A+(x,c)
P̂A(c|x)
with:• P̂A(c|x) the estimation of the probability of
class c given x, with the training set A• E
P̂A+(x,c)the estimation of the expectation of the
test error, with training set A+ (x, c)
• Require an accurate estimation of P̂A(c|x)
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.29/81
![Page 30: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/30.jpg)
RETIN
• Active learners select only one example• In image retrieval, several images labeled during
each feedback step = batch processingHow to select other ones ?• Iteration of the active selection:→ Problem: close images may be selected.• Diversity: select different images:→ Clustering or using angle Diversity AD
(Brinker):I⋆ set of selected images
gI⋆(xi) = λ g(xi)︸ ︷︷ ︸
active criteria
+(1− λ) maxj∈I⋆
|k(xi,xj)|√
k(xi,xi)k(xj ,xj)︸ ︷︷ ︸
angle criteria
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.30/81
![Page 31: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/31.jpg)
RETIN
• Second: having a good estimation of f near theboundary• Exploit again the batch processing:
user labels 6= expected labels (balanced set)• too many positive labels => go further (and vice
versa)• Boundary Correction (BC):• O = argsort f , and s the index of the current
threshold: f⋆(x) = f(x)− f(xOs)
→ Update: s(t + 1) = s(t) + 2(pos(t)− neg(t)) with:• pos(t) (resp. neg(t)) = number of relevant
(resp. irrelevant) labels• t = feedback iteration number
• Efficient during the first feedback stepsClassification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.31/81
![Page 32: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/32.jpg)
An example of retrieval session (1/8)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.32/81
![Page 33: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/33.jpg)
An example of retrieval session (2/8)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.33/81
![Page 34: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/34.jpg)
An example of retrieval session (3/8)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.34/81
![Page 35: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/35.jpg)
An example of retrieval session (4/8)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.35/81
![Page 36: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/36.jpg)
An example of retrieval session (5/8)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.36/81
![Page 37: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/37.jpg)
An example of retrieval session (6/8)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.37/81
![Page 38: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/38.jpg)
An example of retrieval session (7/8)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.38/81
![Page 39: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/39.jpg)
An example of retrieval session (8/8)
Revelant labelled pictureRelevant unlabelled picture
Irrelevant labelled pictureIrrelevant unlabelled picture
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.39/81
![Page 40: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/40.jpg)
Experiments
Methods Top-100(%) Timeno AL 16 0.07s
UB 28 0.41sER 30 600s
UB+AD 31 60sER+AD 34 700s
BC+UB+AD 36 60sBC+ER+AD 35 700s
Protocol:• COREL Photo database (6,000 images);• 50 categories, 100-300 size;• Training set of 50 points (10 feedback steps, 5
labels per step). Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.40/81
![Page 41: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/41.jpg)
Experiments
Methods MAP(%)no AL 20
UB 31ER 32
UB+AD 37ER+AD 37
BC+UB+AD 39BC+ER+AD 38
Protocol:• COREL Photo database (6,000 images);• 50 categories, 100-300 size;• Training set of 200 points (20 feedback steps, 10
labels per step). Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.41/81
![Page 42: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/42.jpg)
Experiments
10 20 30 40 50 60 70 80 90 10010
15
20
25
30
35
Size of the SVM training set
Mea
n A
vera
ge P
reci
sion
UB With Correction and DiversityUBNo active learning
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.42/81
![Page 43: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/43.jpg)
Experiments
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.43/81
![Page 44: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/44.jpg)
Experiments
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.44/81
![Page 45: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/45.jpg)
Conclusion for Active Learning
• Active learning gives a theoretical framework forrelevance feedback
• Boundary correction efficient for uncertainly-basedtechniques
• Adding diversity improves the performances
• Active learning framework can be used for otheroptimization problems in CBIR:→ Kernel parameters→ Long-term, similarity matrix learning
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.45/81
![Page 46: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/46.jpg)
Semi-supervised classification for CBIR
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.46/81
![Page 47: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/47.jpg)
Semi-supervised classification
Deal with the few training data (c2) and manyunlabelled data (c3) characteristics
→ use unlabelled data to compensate scarcity oftraining data.
Three representative methods:• Transductive SVM (Joachims):→ Maximize the margin considering all data.• Gaussian Mixture (Najjar):→ Estimate the gaussians using all data.• Gaussian Fields (Zhu):→ Estimate the densities using harmonic
functions.Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.47/81
![Page 48: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/48.jpg)
Transductive SVM (Joachims)
Inductive SVM boundaryRevelant unlabeled dataRevelant labelled data
Irrevelant unlabelled dataIrrevelant labelled data
Transductive SVM boundary
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.48/81
![Page 49: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/49.jpg)
Experiments
Methods Error(%) MAP(%) TimeSVM 2.29 20 0.13s
TSVM 2.29 20 10.7sGM 20.2 9 12.1sGF ? ? >10min
Protocol:• COREL Photo database (6,000 images);• 50 categories, 100-300 size;• Training set of 200 point (unbalanced).
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.49/81
![Page 50: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/50.jpg)
Long Term or Semantic learning forinteractive image retrieval
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.50/81
![Page 51: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/51.jpg)
Challenge
• Active research: Interactive learning techniques ofimage categories using relevance feedback;
• Limitation: Knowledge lost at the end of eachretrieval session;
• Proposition: Learn from past retrieval sessionsClassification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.51/81
![Page 52: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/52.jpg)
Retrieval sessions
• Interactive learning (relevance feedback, SVM,active learning) to retrieve an image category , asubset of the database
• In our framework, binary labels:
• At the end of a retrieval session: Collect all theselabels in a vector y
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.52/81
![Page 53: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/53.jpg)
Training set
• All vectors y in a matrix Y:
Y :
y(1) y(2) y(3) y(4) y(5) y(6) y(7) . . . y(M)
x1 1 1 0 0 −1 1 0 . . . 0
x2 1 1 1 1 −1 0 1 . . . 0
x3 1 0 1 −1 0 0 0 . . . 1
x4 0 −1 1 0 0 −1 0 . . . 1
x5 −1 0 0 1 1 −1 0 . . . 0
x6 0 0 −1 0 1 0 −1 . . . −1...
......
......
......
......
xN 0 1 0 0 0 −1 0 . . . 0
• This matrix Y is the training set
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.53/81
![Page 54: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/54.jpg)
Exotic learning problem
• Partial knowledge→ Not possible to identify 1 category from 1
retrieval session• Unknown category for each retrieval session→ Not possible to easily rebuild categories
• Mixed categoriesProposition:
• Do not work on learning some explicit categoriesbut on the learning of the database similarities
• Long-term learning or semantic (from Y) learning• Assumption: search for a finite nb of categories
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.54/81
![Page 55: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/55.jpg)
A Gram matrix approach
• Aim: Optimize the matrix K of similarities k(xi,xj)
between the N images of the database• Naïve approach: increase (resp. decrease) the
similarity between two images in the same (resp.different) category using an heuristic function
• Problems:• No more guaranty on metric properties• Relevance feedback techniques can not be
used anymore
• Proposition: consider a definite positive matrix ofsimilarities K (a kernel matrix)→ Update K under constraints to keep dp
propertiesClassification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.55/81
![Page 56: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/56.jpg)
Adaptive Method RETIN SL
• Reinforce kernel values corresponding to positivevalues in y, statistical accumulation→ Gram matrix updating:
K(t + 1) = update(K(t),y(t), ρ(t))
= (1− ρ(t))K(t) + ρ(t)×merge(K(t),y(t))
with K(t) the Gram matrix at iteration t, and y(t) arandomly selected vector of Y, andmerge(K(t),y(t)) an operator to merge knowledgein K(t) and y(t);
• Merging strategy: the resulting form with twocomponents:
merge(K,y) = a× (TKTt + bKu)Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.56/81
![Page 57: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/57.jpg)
First condition: unbalance updating
• Two class problem:∀(xi,xj), yiyj > 0 => k(xi,xj)ր
• Unbalance updating:
XClass handling
{
If yi > 0 then ∆k(xi,xj) highow ∆k(xi,xj) small
• ∀(xi,xj), yiyj < 0 => k(xi,xj)ց
• Merging strategy: Ku = uut
where uk = 1 if yk > 0, uk = −γ if yk < 0, otherwise0
• dp properties clearly preserved
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.57/81
![Page 58: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/58.jpg)
Second condition
• Homogenize similarities in one shot y:
(1) Cst values inside
{
∀(yi,yj) = +1 in y
k(xi,xj) → cst (+1)
(2) Cst V outside ∀yi = +1 ∀xq ∈ db, k(xq,xi)→ cstq
• Algebraic trick:
• K← TKTt, T =
1 1
1 1
1
1
1
• Averaging of similarities and (1) and (2)performed
• dp also preservedClassification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.58/81
![Page 59: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/59.jpg)
Comments
• Good news:• Fast evolution of the K similarity matrix because
all the similarities between database imagesand labeled images are updated
• Nice control of the matrix rank
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.59/81
![Page 60: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/60.jpg)
Similarity matrix
• Similarity/kernel matrix:
Before optimization After optimization
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.60/81
![Page 61: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/61.jpg)
Vector-based method
• Operator TKT⊤ equivalent to:
∀i ∈ I1 xi ←1n1
∑
j∈I1xj
with I1 = {relevant labeled images}• Idea: move feature vectors;• General scheme:• Group together images in clusters
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.61/81
![Page 62: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/62.jpg)
second method: Vector-basedmethod
−1.5 −1 −0.5 0 0.5 1 1.5 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
+ +
+ −
−−
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.62/81
![Page 63: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/63.jpg)
Vector-based method
−1.5 −1 −0.5 0 0.5 1 1.5 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
+ +
+ −
−−
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.63/81
![Page 64: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/64.jpg)
Vector-based method
−1.5 −1 −0.5 0 0.5 1 1.5 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.64/81
![Page 65: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/65.jpg)
Vector-based method
−1.5 −1 −0.5 0 0.5 1 1.5 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.65/81
![Page 66: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/66.jpg)
Vector-based method
−1.5 −1 −0.5 0 0.5 1 1.5 2−1.5
−1
−0.5
0
0.5
1
1.5
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.66/81
![Page 67: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/67.jpg)
Vector-based method
−1 −0.5 0 0.5 1 1.5 2−1.5
−1
−0.5
0
0.5
1
1.5
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.67/81
![Page 68: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/68.jpg)
Vector-based method
−1 −0.5 0 0.5 1 1.5−1.5
−1
−0.5
0
0.5
1
1.5
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.68/81
![Page 69: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/69.jpg)
Vector-based method
−1 −0.5 0 0.5 1 1.5−1.5
−1
−0.5
0
0.5
1
1.5
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.69/81
![Page 70: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/70.jpg)
Vector-based method
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.70/81
![Page 71: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/71.jpg)
Experiments
Generalist image database:
• 6,000 from COREL photo database;• Features: L⋆a⋆b⋆ and Gabor filters;• 50 mixed categories, with size from 50 to 300
images, from simple (monomodal) to complex(multimodal);Learning set:
• Vectors y with 100 non-zero values;• From 0 to 300 vectors y.
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.71/81
![Page 72: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/72.jpg)
Experiments
Mean performance for each active learner:
0 50 100 150 200 250 30035
40
45
50
55
60
65
70
75
80
Number of past retrieval sessions
Mea
n A
vera
ge P
reci
sion
RETIN ALSVMactiveBasic AL
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.72/81
![Page 73: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/73.jpg)
Experiments
Precision/Recall curve for the ’savana’ category:
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
onRETIN ALBasic ALSVMactive
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.73/81
![Page 74: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/74.jpg)
Experiments
Example : before optimisation:
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.74/81
![Page 75: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/75.jpg)
Experiments
After optimisation:
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.75/81
![Page 76: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/76.jpg)
Experiments
After optimisation (second screen):
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.76/81
![Page 77: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/77.jpg)
Experiments
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.77/81
![Page 78: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/78.jpg)
Experiments
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.78/81
![Page 79: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/79.jpg)
Experiments
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.79/81
![Page 80: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/80.jpg)
ETIS Lab.
Feature ComputationIndexation
Concept LearningInteractive Learning
Data Analysis
Evaluation
EROSText
C2RMF
Artwork Database
Local Descriptors
Fuzzy Regions
RETIN
Generalist Image Database
CORELANN
Video Retrieval
Object Class Recognition
Pertimm
Internet
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.80/81
![Page 81: Classification and Machine Learning techniques for CBIR](https://reader033.vdocument.in/reader033/viewer/2022051319/5895ad421a28ab43538bd5e6/html5/thumbnails/81.jpg)
ETIS Lab
ETISCNRS UMR 8051 (ENSEA / Université de Cergy)
http://www-etis.ensea.fr/
Matthieu Cord ([email protected]), Philippe-HenriGosselin, Sylvie Philipp Foliguethttp://perso-etis.ensea.fr/˜cord/
RETIN demo : http://dupont.ensea.fr/ ruven/start.php
Classification and Machine Learning techniques for CBIR: introduction to the RETIN system – p.81/81