simpsons @ work
DESCRIPTION
Simpsons @ work. Why temporal segmentation?. Terms. A Shot. smallest piece of information elementary building block of a video. A Cut. Easy Most common 95% of all transitions are cuts. A Fade-in. Often used at the start of a chapter From Black Color to Image Linear intensity change. - PowerPoint PPT PresentationTRANSCRIPT
Simpsons @ work
Why temporal segmentation?
Terms
A Shot
• smallest piece of information
• elementary building block of a video
A Cut
• Easy
• Most common
• 95% of all transitions are cuts
A Fade-in
• Often used at the start of a chapter
• From Black Color to Image
• Linear intensity change
A Fade-out
• Often used at the end of a chapter
• From Image to Black Color
• Linear intensity change
A Dissolve
• Used to show that time has passed
• Often used as a subchapter
• From one Shot to another Shot
• Linear intensity change
Signals
Human vs. Computer
• Human: Knows what happens.• Computer: No cognitive understanding.
Segmentation has to rely on features or signals.
SAD
• Sum of absolute differences• Substract each pixel by each pixel • Very easy approach
SAD = -
¢ SAD (n)k =P width
i
P heightj jI magei ;j (n) ¡ I magei ;j (n+k)j¢ SAD (n)k =
P widthi
P heightj jI magei ;j (n) ¡ I magei ;j (n+k)j
RGB-Histogram
• Find and count all the different shades of red, green and blue and order them from dark to bright.
brightdark
amou
nt
Histogram Difference =
• For red,green and blue: Substract the histograms bin by bin and sum up the difference.
-256 bins
1 Frame Difference
No cut No cut No cut CUT No cut No cut No cut
Result:
How to find the cuts
• Find Cuts with a threshold• Works on SAD and Histogram Difference
frames
sign
al
cut cut cut cutcut cut cut cut
Problems with Motion
• SAD and Histogram are very sensitive to camera or object movement.
frames
sign
alcutcutcutcutcutcutcutcut
SAD:
cut
Problems with Flash
• SAD and Histogram are very sensitive to flash.
frames
sign
al
cutcutcutcutcutcutcutcutcutcut cutcutcutcutcut cut
Problems with dissolves & fades
• So far we have no features that would indicate a fade-in, fade-out or a dissolve
Improvment needed.
Sliding Window
So far: 1 Frame Difference
No cut No cut No cut CUT No cut No cut No cut
Result:
Now: n - Frame difference
No Cut CUT CUT CUT No cut No cut No cut
Result:cut cut cut cutcut cut cut
Motion & Flash
• Not sensitive to flash anymore
• Not sensitive to motion anymore
frames
sign
al
frames
sign
al
Dissolves & Fades
• Finally we are able to detect dissolves
frames
sign
al
fade
Problems with long dissolves
• With this feature we miss the long and slow dissolves.
frames
sign
al
dissolve start
dissolve end
Mean and Variance
Mean
• Mean is the arithmetic average of the histogram values.
• When the histogram changes, the mean moves to left or right.
meanred
meangreen
meanblue
Variance
• Variance indicates how much the histogram values are spread around the mean.
• When the histogram mean changes, the variance changes, too.
vs
small big
Linear Mean Change in Dissolves
Result:
frames
sign
al
Parabolic Variance Change in Dissolves
Result:
frames
sign
al
Process of Detection
• Finds slow dissolves
• Finds fade-in / fade-out
Search for candidates
Approximate mean
Approximatevariance
Calculate theerror made
If small enough:Add disolve
Evaluation
• ~ 250.000 frames of video material (ca. 3h)~ 1800 cuts, 90 dissolves / fades
• Genres: Sport, Cartoons, News, Music, Commericals, Movies, Documentary
• Focus on News ( ABC Lateline )
• 3 Categories: Match, Miss and False Positive
Results
Cuts Fades
Matched out of total
False Positive Matched out of total
False positive
News 290 | 295 5 27 | 29 2
Cartoons 298 | 313 6 12 | 13 4
Sport 130 | 160 3 15 | 17 6
Demo
Questions?