by m. nithya
DESCRIPTION
Learning Techniques for Video Shot Detection. by M. Nithya. Under the guidance of Prof. Sharat Chandran. Outline. Introduction Types of Shot-break Previous approaches to Shot Detection General Approach - pixel comparison, histogram comparison… - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/1.jpg)
Learning Techniques for Video Shot Detection
Under the guidance ofProf. Sharat Chandran
by M. Nithya
![Page 2: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/2.jpg)
Outline
• Introduction• Types of Shot-break• Previous approaches to Shot Detection
General Approach - pixel comparison, histogram comparison… Recent Work – Temporal slice analysis, Cue Video
• Our Proposed approaches Supervised Learning using AdaBoost algorithm Unsupervised Learning using clustering Semi-supervised Learning combining AdaBoost &
clustering
• Conclusion
![Page 3: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/3.jpg)
Introduction 9,000 hours of motion pictures are produced around the world every year. 3,000 television stations broadcasting for twenty-four hours a day produce
eight million hours of video per year.
Problems:• Searching the video
• Retrieving the relevant information
Solution: Break down the video into smaller manageable parts
called “Shots”
![Page 4: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/4.jpg)
What is Shot?
Shot is the result of uninterrupted camera work
Shot-break is the transition from one shot
to the next
![Page 5: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/5.jpg)
Types of Shot-Break
![Page 6: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/6.jpg)
Shot-Break
Hard Cut Fade Dissolve Wipe
![Page 7: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/7.jpg)
Hard Cut
![Page 8: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/8.jpg)
Fade
![Page 9: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/9.jpg)
Dissolve
![Page 10: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/10.jpg)
Wipe
![Page 11: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/11.jpg)
Shot Detection Methods
![Page 12: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/12.jpg)
Shot Detection Methods
Goal:
To segment video into shots
Two ways:• Cluster the similar frames to identify shots• Find the shots that differ and declare it as shot-break
![Page 13: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/13.jpg)
Pervious Approaches to Shot Detection
• General Approaches– Pixel Comparison– Block-based approach– Histogram Comparison– Edge Change Ratio
• Recent Work– Temporal Slice Analysis– Cue Video
![Page 14: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/14.jpg)
Pixel Comparison
Frame N Frame N + 1
x=1 y=1 | Pi(x,y) – Pi+1(x,y) |D(i,i+1)=
X Y
XY
![Page 15: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/15.jpg)
Block – Based Approach
Frame N Frame N + 1
Compares statistics of the corresponding blocks
Counts the number of significantly different blocks
![Page 16: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/16.jpg)
Histogram Comparison
![Page 17: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/17.jpg)
Edge Change Ratio
![Page 18: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/18.jpg)
Comparison…
Method Advantages Disadvantages
Pixel-Comparison Simple, easy to implement
Computationally heavy,
Very sensitive to moving object or camera motion
Block based Performs better than pixel
Can’t identify dissolve, fade, fast moving objects
Histogram comparison Performance is better
Detects hard-cut, fade, wipe and dissolve
Fails if the two successive shots have same histogram. Can’t distinguish fast object or camera motion
Edge Change Ratios Detects hard-cut, fade, wipe and dissolve
Computationally heavy
Fails when there is large amount of motion
![Page 19: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/19.jpg)
Problems with previous approaches
Can’t distinguish shot-breaks with• Fast object motion or Camera motion• Fast Illumination changes• Reflections from glass, water• Flash photography
Fails to detect long and short gradual transitions
![Page 20: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/20.jpg)
Temporal – Slice Analysis
![Page 21: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/21.jpg)
Temporal – Slice Analysis
![Page 22: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/22.jpg)
Cue Video
![Page 23: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/23.jpg)
Temporal – Slice Analysis
![Page 24: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/24.jpg)
Cue Video
• Graph based approach• Each frame maps to a node• Connected upto 1, 3 or 7 frames apart• Each node is associated with
– color Histogram– Edge Histogram
• Weights of the edges represent similarity measure between the two frames
• Graph partitioning will segment the video into shots
![Page 25: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/25.jpg)
Proposed Approaches
![Page 26: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/26.jpg)
Proposed Approaches
Use learning techniques to distinguish between
shot-break and • Fast object motion or Camera motion• Fast Illumination changes• Reflections from glass, water• Flash photography
![Page 27: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/27.jpg)
Supervised Learning
![Page 28: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/28.jpg)
Feature Extraction
• 25 Primitive features like edge, color are extracted directly from the image
• These 25 features are used as input to next round of feature extraction yielding
25 x 25 = 625 features
• This 625 features can be used as input to compute 625 x 625 = 15, 625 features
![Page 29: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/29.jpg)
How these features can be used to classify images?
![Page 30: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/30.jpg)
Solution : Use AdaBoost to select these features.
Oops!! There are 15, 625 features!
Applying them to red, green and blue separately will result in 46, 875 features!
Can we find few important features that will help to distinguish the images?
![Page 31: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/31.jpg)
Input: (x1,y1) (x2,y2) …(xm,ym) where x1,x2,…xm are the images
yi = 0,1 for negative and positive examples
Let n and p be the number of positive and negative examples
Initial weight w1,i = 1/2n if yi= 0 and
w1,I = 1/2p if yi = 1
For t= 1,…T:
Train one hypothesis hi(x) for each feature and find the error
Choose the hypothesis with low error value
update the weight:wt+1,i = wt,i * t
1-et
where ei=0,1for xi classified incorrectly or correctly
t=et/(1-et)
Normalize wt+1,I so that it is a distribution
Final hypothesis is calculated as
AdaBoost Algorithm
![Page 32: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/32.jpg)
Supervised Learning
• Extract Highly selective features
• AdaBoost algorithm to select few important features
• Train the method to detect different shot-breaks
![Page 33: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/33.jpg)
Unsupervised techniques Clustering
![Page 34: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/34.jpg)
Unsupervised technique - clustering
![Page 35: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/35.jpg)
Unsupervised technique - clustering
Hard Cut Dissolve
![Page 36: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/36.jpg)
Unsupervised technique
• Clustering method to cluster into shots
• Relevance Feedback
![Page 37: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/37.jpg)
Semi-supervised Learning
![Page 38: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/38.jpg)
Semi-supervised Learning
Combination of Supervised and Unsupervised
Few labeled data are available, using which it works on large unlabeled video
Steps:
• AdaBoost algorithm to select features
• Clustering method to cluster into shots
• Relevance Feedback
![Page 39: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/39.jpg)
Conclusion…
![Page 40: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/40.jpg)
Conclusion…
Problems with previous approaches:• Can’t distinguish shot-breaks with
– Fast object motion or Camera motion – Fast Illumination changes– Reflections from glass, water– Flash photography
Fails to detect long and short gradual transitions
Planning to use AdaBoost learning based clustering scheme for shot-detection
![Page 41: by M. Nithya](https://reader033.vdocument.in/reader033/viewer/2022061612/56814f31550346895dbcc2e6/html5/thumbnails/41.jpg)
Thank you…