deep learning features and similarity of movies based on their video content
TRANSCRIPT
![Page 1: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/1.jpg)
Deep learning features and similarity of movies based on their
video content
Summer Camp - Show Max - Lukáš Lopatovský
![Page 2: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/2.jpg)
Assignment
● Deep learning allows extracting useful features from video frames. Your task is to apply new deep learning frameworks to extract features from video frames of selected movies available in the ShowMax streaming platform.
● Goals:● Extract deep features from video frames. Explore similar
movies in the space of latent features and adjust the extraction process in order to create clusters of video assets (eg. TV episodes).
![Page 3: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/3.jpg)
Residual Networks
● Enable to build deeper (convolutional) neural network. (State of the art method for the image recognition.)
![Page 4: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/4.jpg)
Residual Networks
● To enable to build deeper network, the residual nets use the simple trick. They maintain the residuum from the previous layer ( so do not loose the previously known information )
![Page 5: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/5.jpg)
Torch
- Efficient Tensor library (like NumPy) with an efficient CUDA backend
- Neural Networks package -- build arbitrary acyclic computation graphs with automatic differentiation
- fast CUDA and CPU backends
- Good community and industry support - several hundred community-built and maintained packages.
![Page 6: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/6.jpg)
● Torch example of ‘nn’ library
![Page 7: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/7.jpg)
What has been done
● The movies were classified using arbitrary number of picture frames.
● We have used already trained ImageNet FB- resnet network and own data set trained and fine-tuned networks to classify movies.
● To detect the object in the image, we have classify the whole image, as well as we have made the various crops to get more accurate predictions. (Cropping showed better results)
![Page 8: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/8.jpg)
Classification output
● By classification of frames in the movie, the special file is produced (.res). It is in a form to contain all the important data. It can be later post-process according to the special needs of the user:
- To create Object detection .srt file.
- To get various cumulative classification results.
- To trace the appearance of the object at the time-line.
![Page 9: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/9.jpg)
Object classification exmple
![Page 10: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/10.jpg)
Object classification exmple
![Page 11: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/11.jpg)
Object classification exmple
![Page 12: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/12.jpg)
False positive
![Page 13: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/13.jpg)
False positive
![Page 14: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/14.jpg)
False positive
![Page 15: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/15.jpg)
Own datasets
● The network was successfully trained and fine-tuned from the ResNet network
● However, it showed some problems based from improper dataset. - Some categories contain many irrelevant pictures in second half of the search. (Special case: “The doctor House”)
- The style of the images in the search is often very different to the style found in the movie. (kitchen, car)
- Movies mostly contain images full of people, so the categories containing people make false positive prediction. (cinema, theater)
![Page 16: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/16.jpg)
Classification example
⊙ω⊙
![Page 17: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/17.jpg)
Classification example
⊙ω⊙
![Page 18: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/18.jpg)
Object detection
![Page 19: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/19.jpg)
Object detection
![Page 20: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/20.jpg)
Object detection
![Page 21: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/21.jpg)
Object detection
![Page 22: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/22.jpg)
Object detection
![Page 23: Deep learning features and similarity of movies based on their video content](https://reader031.vdocument.in/reader031/viewer/2022030302/587e09cb1a28abe11a8b6871/html5/thumbnails/23.jpg)
Next step
● After the discussion in the company, the programs were transformed to the easily usable form.
● The feature vectors of the classification will be used to find similarities among movies. Compared to the existing algorithms and if successful, incorporated into the current recommendation system.