Learning Structure-And-Motion-Aware Rolling Shutter CorrectionBingbing Zhuang1, Quoc-Huy Tran2, Pan Ji2, Loong Fah Cheong1, Manmohan Chandraker2,3

1National University of Singapore, 2NEC Labs, 3UC San Diego

Problem: Rolling shutter (RS) cameras are very popular due to its

low-cost advantage. However, they adopt a sequential exposure

mode, causing image distortion when captured during camera

motion.

1. Introduction

2. Contributions

3. Critical configuration: RS two-view SfM in pure translation

4. Structure-and-motion-aware rolling shutter correction

5. ExperimentsProposition. RS two-view geometry for pure translational camera motion is degenerate.

➢ one cannot tell if the two images are captured with a RS or GS camera based on 2D

correspondences only.

➢Even RS is known a priori, there are infinitely many solutions for the per-scanline camera position

and depth.

𝑐1

𝑐2

𝑠1

𝑠2𝑠2

𝑒

𝑒

Confounding of Depth & Translation

𝒔2 − 𝒆 =𝑍

𝑍 − 𝑇𝑖𝑗(𝒔1 − 𝒆)

See paper for a proof!Radiating Pattern

Overview Network architecture

Image resizing to get training data with same size? No, image cropping.

wx rotation vertical resizing

Training data generation

Sequential exposure mode

3D point

Camera motion between scanlines

Our correction

Rolling shutter distortion

❑ RS two-view SfM in pure translation is degenerate.

❑ A single-view structure-and-motion-aware RS correction

method.

Difficulties: removing such distortion exactly needs

recovering both the intra-frame camera motion and 3D

structure.

• Complexity of RS geometry

Two-view approach → [Dai et al. CVPR17], [Zhuang et

al. ICCV17]

• Degeneracy of RS geometry

Baseline:

(1) MH, [Purkait et al. ICCV17]

(2) 2DCNN, [Rengarajan et al. CVPR17]

Input MH 2DCNN Ours

Bas-relief like ambiguity

Result: Synthetic KITTI

Result: Real data

Contributions:

❑ A geometrically meaningful way to synthesize large-scale

training data

❑ Identify a geometric ambiguity that arises for training.

[Albl et al. ECCV16], [Ait-Aider et al. ICCV09]

— Data-driven with CNN to overcome the geometrical complexity

and degeneracy.

GS image

RS image

RS depth

GS depth

Inp

ut

Ou

rsM

H2

DC

NN

Cropping

VS

Resizing

6-DoF motion

Pure rotation

Example image

Input

Ours

MH

2DCNN

SfM