media genre inference for predicting media interestingness
Post on 23-Jan-2018
360 Views
Preview:
TRANSCRIPT
-
EURECOM @MediaEval 2017: Media Genre Inference for
Predicting Media Interestingness O. Ben-Ahmed, J. Wacker, A. Gaballo, B. Huet
EURECOM Sophia Antipolis, France
-
Introduction
Predicting Media Interestingness (PMI) automatically analyze media data identify the most attractive content
Content based approaches
gap between low-level features and high-level human perception
Our proposal Address PMI in association with
Media Genre
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 2
http://www.dailyherald.com/article/20110627/entlife/706279989/
-
Why Genre Inference for PMI
Motivation Interestingness is highly correlated with data emotional content Affective representation of data content
Hypothesis
Emotional impact of movie genre can be a factor for interestingness of a video fragment
Method Mid-level representation based on media genre prediction for PMI
Represent each video fragment/image as a distribution of genres
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 3
-
Our Framework
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 4
-
Media Genre Prediction
Visual Branch Middle frame selection Deep CNN for features extraction DNN classifier
Audio Branch
Audio extraction : OpenSmile Deep features extraction : Soundnet SVM classifier
VGG Architecture
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 5
| X
| X
-
Media Genre Prediction Example
Visual Audio Audio-Visual Action
Drama
Horror
Romance
Sci-fi
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 6
https://fr.wikipedia.org/wiki/After_Earth
-
Media Genre Prediction Example
Visual Audio Audio-Visual Action
Drama
Horror
Romance
Sci-fi
2.5% 33,34% 17.92%
0% 17,78% 8.89%
0.37% 2.61% 1.49%
0% 3.87% 1.93%
97.12% 42,40% 69,76%
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 7
https://fr.wikipedia.org/wiki/After_Earth
-
Media Genre Prediction Example
Visual Audio Audio-Visual Action
Drama
Horror
Romance
Sci-fi
Interestingness : 1 Rank : 1
2.5% 33,34% 17.92%
0% 17,78% 8.89%
0.37% 2.61% 1.49%
0% 3.87% 1.93%
97.12% 42,40% 69,76%
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 8
https://fr.wikipedia.org/wiki/After_Earth
-
Interestingness Classification
Features vectors probability vector for the genre distribution for the video/image
Classifier Binary SVM, Taking into account the confidence score in training
Image subtask Visual genre vector
Video subtask Visual genre vector Audio-Visual genre vector
Mean of visual- and audio-based genre vectors probabilities
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 9
-
Experiments and Results TASK RUN SVM CLASSIFIER MAP MAP@10
IMAGE 1 Sigmoid kernel 0.2029 0.0587
2 Linear kernel 0.2016 0.0579
VIDEO 1 Sigmoid kernel gamma=0.5, C=100 0.2034 0.0717
2 Polynomial kernel degree=3 0.1960 0.0732
3 Polynomial kernel degree=2 0.1964 0.0640
4 Sigmoid kernel gamma=0.2, C=100 0.2094 0.0827
5 Sigmoid kernel gamma=0.3 , C=100 0.2002 0.0774
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 10
-
Experiments and Results TASK RUN SVM CLASSIFIER MAP MAP@10
IMAGE 1 Sigmoid kernel 0.2029 0.0587
2 Linear kernel 0.2016 0.0579
VIDEO 1 Sigmoid kernel gamma=0.5, C=100 0.2034 0.0717
2 Polynomial kernel degree=3 0.1960 0.0732
3 Polynomial kernel degree=2 0.1964 0.0640
4 Sigmoid kernel gamma=0.2, C=100 0.2094 0.0827
5 Sigmoid kernel gamma=0.3 , C=100 0.2002 0.0774
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 11
-
Conclusions and Future Work
We proposed a Genre Recognition System as a mid-level representation for Predicting Media Interestingness Deep Audio and Visual Features for Genre Recognition SVM Classifier for Predicting Media Interestingness Best Results:
20,29 MAP for Image and 20,94 MAP for Video (on test set). Audio brings limited additional information for PMI
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 12
-
Conclusions and Future Work
We proposed a Genre Recognition System as a mid-level representation for Predicting Media Interestingness Deep Audio and Visual Features for Genre Recognition SVM Classifier for Predicting Media Interestingness Best Results:
20,29 MAP for Image and 20,94 MAP for Video (on test set). Audio brings limited additional information for PMI Joint learning of audio-visual features Integration of temporal information
13/09/2017 MediaEval2017 Dublin - B. HUET, EURECOM - p 13
-
Questions?
Benoit Huet. MediaEval2017 Dublin - B. HUET, EURECOM - p 14 13/09/2017
Media Genre Inference for Predicting Media Interestingness
@ 2017
Thank you,
Slide Number 1Introduction Why Genre Inference for PMIOur FrameworkMedia Genre PredictionMedia Genre Prediction ExampleMedia Genre Prediction ExampleMedia Genre Prediction ExampleInterestingness ClassificationExperiments and ResultsExperiments and ResultsConclusions and Future WorkConclusions and Future WorkQuestions?
top related