an approach to personalized video summarization based on … · 2016. 9. 24. · an approach to...

An Approachto Personalized Video SummarizationBased on User Preferences Analysis

Maria Miniakhmetova, Mikhail Zymblerminiakhmetovams@susu.ru, mzym@susu.ru

South Ural State University, Chelyabinsk, Russia

9th International Conference on Application of Informationand Communication Technologies

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 1 / 21

Video Summarization

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 2 / 21

Personalized Video Summary

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 3 / 21

Using Extra Devices

Eye movements tracking Facial activity recognition

Analyzing physiological responses Using data from personal cameras

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 4 / 21

Without Extra Devices

Analyzing text description Specifying preferences manually

Manual key frames selection

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 5 / 21

Approach Proposed in This Work

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 6 / 21

User’s Evaluations

Formal definition

E = {e+, e0, e−}

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 7 / 21

Video Structuring

Formal definition

V = {si}ni=1; 0 < n ≤ F

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 8 / 21

Video Summary

Formal definition

r(V ) = {sj}tj=1; sj ⊆ V

d(r(V )) =

t∑j=1

d(sj) ≤ dmax

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 9 / 21

Object Detection

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 10 / 21

Object Indexing

Formal definition

O = {oi}Mi=1

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 11 / 21

Importance of Objects

Formal definition

P (Ai) = P (Ai|e+) · P (e+) + P (Ai|e0) · P (e0) + P (Ai|e−) · P (e−)

P (Ai) = P (Ai ∩ e+) + P (Ai ∩ e0) + P (Ai ∩ e−)

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 12 / 21

Importance of Objects

Formal definition

P (Ai|e+) =Li

L; P (Ai|e0) =

Ni

N; P (Ai|e−) =

Di

D

P (e+) =L

W; P (e0) =

N

W; P (e−) =

D

W

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 13 / 21

Importance of Objects

Formal definition

Imp(oi) = func(P (Ai), Li, Di, Ni)

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 14 / 21

Impact of shots

Importance of object Imp(oi)

Imp(oi) = P (Ai) ·Li −Di

max(1, Ni)

Impact of shot

Imp(sj) = sgn(argmax

oi∈sj

(|Imp(oi)|

))· maxoi∈sj

|Imp(oi)| ·∑oi∈sj

|Imp(oi)|

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 15 / 21

Personalized Video Summary

Formal definition

pr(V ) =

t⋃i=1

si|si ⊆ V ;

t∑i=1

d(si) ≤ dmax;

∀si ⊆ pr(V ), sj 6⊆ pr(V ) : |Imp(si)| ≥ |Imp(sj)|

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 16 / 21

Algorithm of Summary Construction

1 Detect shots on video files stored in database.

2 Detect objects O on shots of the video files stored in database.

3 Calculate importance Imp(oi) of each object for the particular user.

4 Calculate the impact Imp(sj) of each shot of the video file that hasn’t beenwatched by the user yet.

5 Rank shots of the video file on the basis of the absolute values of their’simpact.

6 Select top t shots, the total duration of which doesn’t exceed the predefinedthreshold value dmax.

7 Join all of the selected shots to construct personalized video summary pr(V ).

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 17 / 21

Summary Construction

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 18 / 21

Evaluation of the Personalized Video Summary

Forecast evaluation of the original video file

Imp(V ) = func(Imp(sj))

Variants of the function

1

Imp(V ) = sgn(∑sj∈V

Imp(sj))

2

Imp(V ) = sgn(∑sj∈V

Imp(sj) ·d(sj)

d(V ))

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 19 / 21

Evaluation of the Personalized Video Summary

Actual evaluation of the original video file

E = {e+, e0, e−}

Adequacy of the constructed video summary

Ad(pr(V )) =

{1, (E = Imp(V ))

−1, (E 6= Imp(V ))

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 20 / 21

Future Work

M. Miniakhmetova, M. Zymbler (SUSU) Video Summarization via Preferences Analysis AICT2015 21 / 21