![Page 1: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/1.jpg)
FrameFree Video Compression
Critical Points versus Block Matching
FrameFree Technologieshttp://framefree.com
Igor Borovikov
![Page 2: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/2.jpg)
Overview
• Motion compensation for video compression,• Critical Points Filter: brief introduction,• FrameFree Video: a bird eye view and side by
side comparison of schematics with MPEG,• Occlusions and CPF,• Side by side comparison CPF3 and H.264,• FFV features overview,• Future work.
![Page 3: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/3.jpg)
Motion Compensation
• Block Matching Algorithm – dominant motion predictor.
• Advantages: – Conceptually simple yet effective algorithm,– Low CPU requirement on decoder side,– Motion vectors data is well suited for entropy
compression.
![Page 4: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/4.jpg)
BMC Downsides
Inherently discrete nature of blocks leads to:• Resolution dependence
(partially addressed by adaptive block size),
• Block artifacts (addressed by deblocking filters),
• Optimal motion vectors (exhaustive search) are usually prohibitively expensive,
• No simple way to reconstruct good quality non-integer frames,
• BMC has no direct relation to Human Visual System.
![Page 5: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/5.jpg)
Alternatives
• Optical flow and related methods. A large group of pixel matching algorithms, yet not directly suitable for motion compensation in video compression.
• Critical Points Filter. The exceptional robustness of CPF pixel matching and adaptive mesh brings CPF to practical applications.
![Page 6: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/6.jpg)
Critical Points Filter (CPF)
• Y. Shinagawa, and T. L. Kunii, Unconstrained Automatic Image Matching Using Multi-resolutional Critical-Point Filters, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.20, no.9, pp994-1001, Sep 1998
• K. Habuka and Y. Shinagawa, Image Interpolation Using Enhanced Multi-resolutional Critical-Point Filters, International Journal of Computer Vision, vol.58, no.1, pp19–35, 2004
![Page 7: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/7.jpg)
CPF mimics HVS
HVS can reconstruct transition between these two images
<====>So can CPF.
![Page 8: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/8.jpg)
Motion Compensation with CPF
“I Frame” “B-Frame” “P-Frame”Original frame compressed with, say, JPEG.
CPF- Interpolated between I and P frames: information from both sides of interval is used.
Forward motion compensated and corrected frame.
![Page 9: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/9.jpg)
Motion Compensation with CPF
![Page 10: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/10.jpg)
BMC vs. CPF
Motion compensationwith CPF (uncorrected)
Destination image Block motion compensation,Search radius 16 (uncorrected)
Source image
![Page 11: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/11.jpg)
CPF vs. BMC
BMC Metaphor:
- “moving around tiles”;
CPF Metaphor:
- “deforming flexible film fixed on a rectangular frame, no tearing or folding allowed”.
![Page 12: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/12.jpg)
• Bijectivity – no “holes” left after compensation,• Adaptive mesh - resolution independence,• Continuous time - no discretization
whatsoever,• “Key features” – critical points – are matched
with high precision, thus: • Remaining artifacts are not too conspicuous for
HVS,• Residual error of motion compensation is
relatively well suited for combination of custom JPEG-like and entropy coding.
Motion Compensation with CPF
![Page 13: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/13.jpg)
MPEG vs. FFV encoding comparison
![Page 14: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/14.jpg)
MPEG vs. FFV encoding comparison
![Page 15: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/15.jpg)
“Practical Scheme”
TRANS: Image Translator TRANS: Image Translator -- : : Differential Encoder & Noise FilterDifferential Encoder & Noise FilterINTERP: Interpolation Unit INTERP: Interpolation Unit ++ : Differential Decoder: Differential Decoder
![Page 16: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/16.jpg)
Occlusions and CPF“Boy and Toys” (NHK test sequence) with ball bouncing off the floor:
CP
F3 O
riginal C
PF
“CPF3” is the latest modification of FFV Practical Scheme that addresses occlusion problem.
![Page 17: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/17.jpg)
Compression ratio and PSNR
Sequence FFV “Practical Scheme”,
12.2005FFV “CPF3”
12.2006
H.264 Med H.264 Low
Yachting 313Kb,27.93
328Kb,30.12
301Kb,34.82
94Kb,30.71
Intersection 257Kb,30.48
206Kb,31.47
253Kb,33.91
63Kb,29.96
Horse race 653Kb,28.35
807Kb,31.13
582Kb,35.25
215Kb,32.07
Boy and Toys
265Kb,31.14
162Kb,32.80
209Kb,33.94
54Kb,30.44
FFV is approaching H.264: many sequences are in between Med and Low settings in terms of PSNR and compression ratio. The table represents work in progress.
![Page 18: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/18.jpg)
Side by Side Comparison
Original Original CPF* CPF3 H.264Low H.264Med
*) The “worst” frames (occlusion artifacts are apparent) were picked for original CPF to show CPF3 improvements.
![Page 19: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/19.jpg)
FFV/CPF3 Features Overview
1. Continuous time – arbitrary playback speed,2. Mesh based rendering – eliminates resolution
dependence.Combination of 1 and 2 “liberates video from frames”, hence “FrameFree video” ;
3. Playback fits well modern graphics hardware – it is nothing else but rendering of textured meshes with simple pixel shader enhancements (ps_1_1),
4. CPF3 allows fallback Practical Scheme playback in case of dropped packets or CPU-bounded client.
![Page 20: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/20.jpg)
Future work• Simplify encoding and decoding by better occlusion handling,• Coming soon:
– DirectShow filters and QT components,– CODEC,– Integration with popular players on both Mac and PC,– Parallel and GPU-based encoding,– Hardware support,– Mobile platforms support,– Lightweight decoder/player.
• Some applications:– Sports video: arbitrary speed playback and web streaming,– Security video archiving,– Integration with FrameFree Studio,– Much more.
![Page 21: FrameFree Video Compression Critical Points versus Block Matching FrameFree Technologies Igor Borovikov](https://reader036.vdocument.in/reader036/viewer/2022062312/551790f45503460e6e8b5923/html5/thumbnails/21.jpg)
•Thank you!
•Q&A
•Business contact: Tom Randolph [email protected]
•Engineering contact: Igor Borovikov [email protected]
•General information: www.framefree.com