360 video stabilization - university of novi...

360 video stabilization PÉTER BODNÁR

DEPARTMENT OF IMAGE PROCESSING AND COMPUTER GRAPHICS

UNIVERSITY OF SZEGED, HUNGARY

360 VIDEO STABILIZATION, PÉTER BODNÁR, SSIP 2017, NOVI SAD

Outline About 360 video technology

Stabilization: motivation and overview

The video stabilization process ◦ 2D/3D/Hybrid models

◦ Keyframe selection

◦ Tracking

◦ Rotation estimation

◦ Warping

Implementation, evaluation

Limitations, outlook


About 360 degree videos Also known as spherical video, immersive video

Recorded with one (omnidirectional) or more cameras (camera rig)

Viewer can control the viewing direction during playback

Good for action sports (biking, skiing), virtual tours, adult videos


Demo (Project Luna)


Creation Device options:

◦ One camera with multiple lenses

◦ More cameras with overlapping angles

Most commonly in 4K resolution

Output: one spherical video (camera feeds stitched and synchronized)


Devices GoPro: since 2002, action cameras, video editing software

◦ Omni: rig of 6 HERO cameras (€ 5 400) ◦ Fusion: 5.2K fully spherical videos, release: 2017. september

Nokia OZO ◦ 2015., € 40 000 (professional fimmakers) ◦ 8 lenses, FOV: 195 degrees

Samsung Gear ◦ 2016., $ 350 ◦ 2 fisheye lenses

Facebook Surround 360 ◦ 14 wide angle lens, 3 fisheye, global shutter ◦ N/A

Others: ◦ LUNA, Ricoh Theta


Stitching Multiple (overlapping) images (video frames) panorama

Done by camera or software: ◦ VideoStitch

◦ Kolor: Autopano

◦ SkyStitch

Steps: ◦ Feature detection (corners, blobs, Difference of Gaussian of Harris corners)

◦ Registration of images

◦ Compositing (single shot, surface)

◦ Blending (color/exposure correction, etc)


Playback Monoscopic (2D) video

◦ Direction typically controlled with mouse/touch

Stereoscopic (3D) video ◦ Google Cardboard (smartphone holder)

◦ Oculus


Publishing Platforms

◦ Youtube (Google), 2015. March

◦ Oculus (Facebook), 2015. September

Video making ◦ Affordable devices (Samsung Gear 360)

◦ Availability of stitching software

◦ Video editing made easy

◦ Platforms for sharing


Comparison: 3D, 360, VR 3D video

◦ Depth of field

◦ Limited viewpoint

360 video ◦ No depth of field

◦ Control of viewing direction

VR: ◦ Mobility

◦ Interaction


VR sickness 1978. flight simulator sickness

General discomfort, headache, nausea, vomiting, disorientation

Causes ◦ Video shaking (hand-held cameras)

◦ Too much time spent in VR

◦ Personal health conditions

Solution ◦ Reduce camera shaking more comfortable to watch

◦ Make videos smoother in time


Stabilization Estimate camera position (x,y,z) during the video

Calculate rotation for each video frame

Recognize undesired rotations (e.g. shaking)

Calculate an inverse model to remove undesired rotations

Produce the stabilized video


2D video stabilization Feature detection in 2D, narrow Field of View (FOV)

Feature matching, low-dimensional similarity, homography transformation estimation

Inverse transformation


3D stabilization Feature detection on image or in 3D space

Estimation of 3D transformation (transformation matrix or relaxed model)

Inverse transformation


2D: solutions Homography transformation: 1998, Morimoto and Chelappa

◦ Cannot handle parallax

Mesh warps: 2011, Liu et al.

Smoothed flow fields: 2014, Liu et al. ◦ Not easy to adapt to 3D spherical video


3D: solutions 3D model trajectory and scene estimation

◦ Buehler et al, 2001.

◦ Bhat et al, 2007.

◦ Liu et al, 2009.

◦ Kopf et al, 2014.

Drawbacks

◦ Complex process, slow

◦ Too specific to some situations


Facebook: Hybrid model 3D-2D algorithm

◦ Uses deformed rotational model

◦ Handles parallax

◦ 3D can distinguish between translation and rotation movement

◦ 2D fast execution

◦ Preserves desired rotations in scene

◦ 10-20% file size reduction


Hybrid algorithm: overview Feature tracking

Keyframe selection

Keyframe rotation estimation

Keyframe rotation correction

Inner frame optimization

Relaxation of the model

Warping


Video frame representation for tracking


Feature tracking


Implemented in cube faces

Luminance channel (HSV or YCbCr/Yuv)

Tracking by Lukas-Kanade method

Points that leave a cube face are dropped

Tracked points are converted from 2D locations

to 3D unit vectors

Further steps are processed on those vectors

Lukas-Kanade method Optical flow estimation

Sparse features

Basic equations for selected pixels and neighborhood

Least squares criterion

Implemented in OpenCV, MATLAB


OpenCV library CV: Computer Vision

Open-source

Up-to-date

Available for many platforms (C++, Java, Python, Android)

Modular (Core, GUI, Image Processing, Feature extraction, Machine learning, video)

Supports CUDA, OpenGL

Well-documented (http://docs.opencv.org/)


http://docs.opencv.org/

Shi-Tomasi corner detection Moravec corner detection Harris corner detection Shi-Tomasi (Kanade-Tomasi)

Image I, patch (u,v) shifted by (x,y), weighted sum of squared differences S:

Taylor expansion of I(u+x, v+y) using partial derivatives:

Matrix form:


Shi-Tomasi corner detection

Eigenvalues show presence of corners as:


Keyframe selection Detect features (corners) for each frame

First and last frame of the video are keyframes

For each frame: ◦ Detect features

◦ Match features for previous keyframe

◦ Count matched points

◦ If matched points are less than 50%, mark frame as new keyframe

◦ If „some” time is passed, mark frame as keyframe


Rotation estimation Obtain the previously matched point pairs for consecutive key frames

Estimate the rotation using Nister’s five-point correspondance

Use RANSAC procedure

Implemented in OpenCV and OpenGV libraries


About rotation matrices A matrix that rotates a point in Eucledean space

Express rotation around individual axes

Consecutive rotations matrix multiplication

Rotation between keyframes = rotations for each frame multiplied consecutively

Correction of keyframe rotation = inverse of the rotation matrix


RANSAC Random Sample Consensus

Iterative method for mathematical model creation

Inliers: a set of data that makes a model

Outliers: data not fitting on the model


RANSAC - Algorithm Select a subset of the original data (hypothetical inliers)

Fit the model

Evaluate the fitted model on all data, create the set of inliers (consensus set)

Check the size of the consensus set

Repeat fixed number of times

Re-estimate the model with best fit


RANSAC – Advantages and disadvantages

Advantages ◦ Robust estimation

◦ Works with contaminated sets (inliers less than 50%)

◦ Has solution for disadvantages (optimal RANSAC)

Disadvantages ◦ No upper bound on time (exhaustive search)

◦ May not produce optimal result


Point correspondance, Nister Camera calibration

Finding the Essential matrix between corresponding points in images ◦ Defines a constraint, assumes pinhole camera model

◦ Can be used for pose recovery of calibrated cameras

◦ Included in OpenCV

Pose recovery ◦ Essential matrix decomposition

◦ Multiple solutions, cheirality check (positive depth of the triangulated points)

◦ Included in OpenCV


The OpenGV library Geometric Vision

C++ library for calibrated camera pose computation

Includes many algorithms + benchmark

MATLAB, Python support


Inner frame trajectories Linear interpolation between keyframes good initial solution

Optimization of the length and smoothness of features


The Ceres library Non-linear optimalization and modeling

Open-source, C++

Users: ◦ Google

◦ Microsoft

◦ Blender


Deformed rotation model


About barycentric coordinates Introduced in 1827.

Also known as areal coordinates

Positive within the convex hull

X1…Xn: vertices of a simplex in affine space

(a1+ … +an)p = a1x1 + … + anxn

a1 + … + an = 1


Deformed rotation model


Summary


Advantages Accuracy: 3D analysis estimates keyframe rotation, separates rotation and translation

Robustness: 3D estimation is reliable between keyframes

Regularization: ◦ Keyframes provide the backbone

◦ Deformed rotation model gives smoother result

◦ Free from artifacts

Speed: ◦ Rapid convergence of the 2D inner frame optimization

◦ Faster than normal playback


Image warping 360 video no cropping

Equirectangular equirectangular

Rotation estimation: the new position of the pixel

Inverting the model: get the value to each position (x,y) from (u,v)


Rotation filtering Rotation estimation and removal camera orientation does not change

Not all rotations need removal


Demo (GoPro Hero5 camera)


Two-phase tracking Some patches have high distortion on shaky videos

Some features are leaving a specific cube face from the cube map

Proposal: repeat tracking on stabilized video further boost of smoothness


Hyperlapse Increased speed of videos (typically 3-5x)

Rotation estimation and correction is highly recommended

Simplest speedup solution: frame dropping

More sophisticated solution: ◦ Estimate camera velocity

◦ Make it smooth: estimate motion vector magnitudes, median + low pass filter, timestamp remap for video frames


Demo (Microsoft Hyperlapse)


Effect on bitrate Original video can be recovered from modified

360 viewers also apply transformation „no” additional computation required

Bitrate is reduced less rotation between encoded frames


Quantitative evaluation (Facebook)


Limitations High frequency rolling shutter

Wobbling due to the deformed model

„Shots” are not handled, unpredictable

Static overlays, logos

Person filming the scene


Space for improvement Sparse tracking dense tracking

Keyframe selection

Other methods for rotation estimation and filtering

Other representation for frames

Using color information


References Facebook video stabilization:

https://code.facebook.com/posts/697469023742261/360-video-stabilization-a-new-algorithm-for-smoother-360-video-viewing/

Devices:

https://code.facebook.com/posts/1755691291326688/introducing-facebook-surround-360-an-open-high-quality-3d-360-video-capture-system/

http://www.samsung.com/global/galaxy/gear-360/

http://luna.camera/

https://theta360.com/

Image processing and optimalization libraries:

http://opencv.org/

https://laurentkneip.github.io/opengv/

http://ceres-solver.org/

Johannes Kopf: 360° video stabilization: http://dl.acm.org/citation.cfm?id=2982405






















































http://luna.camera/

http://luna.camera/



http://opencv.org/

http://opencv.org/







http://dl.acm.org/citation.cfm?id=2982405

Review 1. Which devices are used for 360 video recording?

2. What is video stitching and how it is performed?

3. What is the difference between 3D video, 360 video and VR?

4. What is video stabilization and what are the steps?

5. What are the main features of OpenCV?


360 video stabilization - university of novi...

Documents