high frame rate up conversion ana bertran. problem statement original frame (30 fps) dwnsmpld frame...
TRANSCRIPT
![Page 1: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/1.jpg)
High Frame Rate Up Conversion
Ana Bertran
![Page 2: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/2.jpg)
Problem Statement
Original Frame
(30 fps)
Dwnsmpld Frame
(5 fps)
Recovered Frame
(30 fps)
S & H
LI
MCLI
MCwA
Hypothesis: if we have acceleration MCwA should provide better results
X=a+b*sin(*t), vx=b**cos(*t), ax=-(b*2*sin(*t)
![Page 3: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/3.jpg)
MotivationUses of frame rate up conversion: Converting between standards (PAL to NTSC). Using MCwA less critical
since frame rates not too far from each other. Low bit rate compression for video-confrncng, video-phone and video
games. MCwA becomes more critical (from 10 fps to 30 fps)
Problems with traditional methods: S&H motion looks jerky, not smooth, very choppy Linear interpolation without motion: image looks blurry where motion has
occurred, we will see ghosts due to avg. btwn. frames.
![Page 4: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/4.jpg)
Motivation
Original
Linear Interpolation Sample & Hold
![Page 5: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/5.jpg)
MCLI vs. MCwA
ME Catalog Occsns
LMC
AMC
![Page 6: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/6.jpg)
ME
For forward and backward MVs
Overlapping BM
16x16 blcks, move by 8
Finer MV Selection
• Top Left block use its MV• 1st row blocks min SAD(L, current or next frame)• 1st colmn blocks min SAD(T, current or next frame)• others min SAD(T, L, current or next frame)
½ pixel MV zoom in
Take out illegal MVs
![Page 7: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/7.jpg)
Cataloging Occlusions
1 2 3 4
For covered pixels we need to use frame 2 as the predicted image, one whose blocks we serch for.
For uncovered pixels we need to use frame 3 as the predicted image, one whose blocks we serch for.
Need to track block from 3 to 2
Criteria: compare SADs dependingwhich one minimum classify.
![Page 8: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/8.jpg)
Cataloging Occlusions Results
blue=uncovred=covGreen=no occsn
blue=uncovred=no occsn
![Page 9: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/9.jpg)
LMC
3 cases
No occlusions
Uncovered pixels
Covered pixels
No occlusions Occlusions
For each block along motion trajectory:MVx_est=w*MVx_3_2
MVy_est=w*MVy_3_2
Motion will be non-integer: interpolate
Assumed motion in xDepending on +ve/-ve x mtn, unvov/cov:
3 2
![Page 10: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/10.jpg)
AMCTo solve distancex=vx*t+0.5*ax*t2 you need two MVs but you can get moreaccurate motion trajectories by solving a LMS problem on 3 MVs.
For each MV between two frames take the previous and next MVs to estimatethe trajectory.
Need to solve:
![Page 11: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/11.jpg)
Frame rate up conversion a difficult problemNeed to use true motion vector fields but block matching does
poorly with this, specially across object boundaries and if the moving object is too small. Can´t cope with discontinuities in the velocity plane.
Some motion can fall in between pixels
Dealing with occlusions
Motion can change dramatically between frames and we won´t realize it. (Critical vels)
Not to talk about scene changes! – we no longer will have enough MVs to track motion trajectory.
![Page 12: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/12.jpg)
Results/Demo
Will have the demo in the website by the end of
the week.
For now you can see the demo for the other types of FR up-conversion
![Page 13: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/13.jpg)
Current temporary results
![Page 14: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/14.jpg)
Future Work
Need better MV estimation of the true MVs
Implement ½ pixel resoltn or Use hierarchical block matching to
find true motion MVs or Object based interpretation of the
video to smooth out MVs Use 3D Recursive Search Block
Matching
Solve occlusion problem for both x and y movement
![Page 15: High Frame Rate Up Conversion Ana Bertran. Problem Statement Original Frame (30 fps) Dwnsmpld Frame (5 fps) Recovered Frame (30 fps) S & H LI MCLI MCwA](https://reader030.vdocument.in/reader030/viewer/2022032702/56649cc55503460f9498e8ee/html5/thumbnails/15.jpg)
References
Using Motion-Compensated Frame-Rate Conversion for the Correction of 3:2 Pulldown Artifacts in lman, Video Sequences, Kevin Hilman, Hyuon Wook Park, and Yongmin Kim
True-Motion Estimation with 3-D Recursive Search Block MatchingGerard de Haan, Paul W.A.C. Biezen, Henk Huijgen and Olukayode A. Ojo
Digital Video Standards Conversion in the presence of accelerated motionAmdrew J. Patti, M. Ibrahim Sezan and M.Murat Tekalp
Framee Rate Up- Conversion using transmitted true motion vectorsYen-Kuang Chen, Anthony Vetro, Huifang Sun, and S.Y. Kung