Interactive Images: Cuboid Proxies for Smart Image Manipulation

Youyi Zheng, Xiang Chen, Mingming Cheng, Kun Zhou, Shi-Min Hu, Niloy J. Mitra

Presenter: Jiapei ZHANG

Outline

• Coupled 2D-3D Analysis 2D pixels 3D proxies• Image Decomposition Background + Textured Proxies • Interactive Editing

Coupled 2D-3D Analysis

Initial camera calibration( )i iR CH R Grab Cut Six corners

iR

Corner points[ | ]j j

ip K R t P

6 correspondence pairsj j

iP pLinear program

1 2, , , ,K R t l lCamera calibration

Initial Scene EstimationTriplet of adjacent edges Candidate cuboid

For each candidate cuboid in the set {Pi}

Joint Optimization minimize the total fitting error

Stacking & Occlusion

BAA B

Resolving Ambiguities

( , )

( ) ( , )l i i i i j ji i j

E u R l c R l R l

MRF formulation

Non-convex objectsCutting plane identification Redundancy removal

Greedy Algorithm

Choose the one covers the largest

area of Ri

Image Decomposition

Light & Shadow

User-marked shadow correspondence

Estimate a single light position (in a least squares sense)such that

,i i ifor each R l S

( , )i p if P l S

Decomposition• Remove projected proxy and shadows• Image completion• For each proxy: (i) shadow effects based on light position (ii) symmetric regions to fill in missing texture for invisible faces

Proxy-Based Edits

Basic Operation

• Extracted proxies + Relations• Relations:

• Operations:

placement, repetition, alignment, coplanarity, repetition, parallelism

translate, rotate, scale, stack

User-Indicated Relation

hinge joint

slide inside mark

Repeated Edit

Virtual Shopping

User Study

Original Images Editing ResultsV.S.

Users recognized: 63.2% real images 44.5% fake images

Conclusion

2D 3D 2D

Thank You