Seamless image-based texture atlases using multi-band

blendingCédric Allène Jean-Philippe Pons Renaud Keriven

Université Paris-Est – Ecole des Ponts – CERTIS

Seamless image-based 3D textures

Outline

1) Context: 3D reconstruction

2) Image regions choice

3) Multi-view blending

4) Atlas building and results

Seamless image-based 3D textures

- 1 -Context:

3D reconstruction

Seamless image-based 3D textures1

– C

on

text:

3D

recon

str

ucti

on

3D reconstructionfrom multi-view

First step: Get a set of pictures or the model to reconstruct.

Seamless image-based 3D textures1

– C

on

text:

3D

recon

str

ucti

on

3D reconstructionfrom multi-view

Second step: Autocalibration from your set of pictures.

Seamless image-based 3D textures1

– C

on

text:

3D

recon

str

ucti

on

3D reconstructionfrom multi-view

Third step: 3D mesh building.

Seamless image-based 3D textures1

– C

on

text:

3D

recon

str

ucti

on

3D reconstructionfrom multi-view

Fourth step: Add colored textures on the mesh from multi-view images.

Problems: Which image should

the texture be from? How to minimize color

discontinuities at the frontiers of regions from different images?

This is here our method comes into

play!

Seamless image-based 3D textures

- 2 -Image regions choice

Seamless image-based 3D textures

Surface partition

We note: the input calibrated images, and the projection from 3D space to image

We have to assign each face of the mesh in to one of the input views in which it is visible.

We obtain a labeling vector

What do we want? good visual detail, and color continuity at region boundaries

2 –

Im

ag

e r

eg

ion

s c

hoic

e

Seamless image-based 3D textures

Good visual detail:

Color continuity at region boundaries:

where is a dissimilarity term between the neighbour faces and with labels and .

Energy terms

2 –

Im

ag

e r

eg

ion

s c

hoic

e

Seamless image-based 3D textures

Energy minimization

Global energy to minimize:

Since is a sum of regular functions, we can use the MRF/min-cut minimization method to solve our problem.

Problem:Even if minimized, color discontinuities remain between regions of different labels…

2 –

Im

ag

e r

eg

ion

s c

hoic

e

Seamless image-based 3D textures

- 3 -Multi-view blending

Seamless image-based 3D textures

Multi-frequencies blending

3 –

Mu

lti-

vie

w b

len

din

g

Method proposed in:P. J. Burt and E. H. Adelson. A multiresolution spline with application to image mosaics. ACM Trans. on Graphics, 2(4), 1983.

Two main functions: Reduce function (RED):

Input: image I Output: image I’ which dimensions are half of those of input

image I reduced through a gaussian kernel

Expand function (EXP): Input: image I Output: image I’ which dimensions are twice those of input

image I expanded through the same gaussian kernel

Seamless image-based 3D textures

Gaussian & laplacian pyramids

3 –

Mu

lti-

vie

w b

len

din

g

Gaussian pyramid (left), obtained by successive reductions of input image.

Laplacian pyramid (right), obtained by substractions of two gaussian pyramid levels.

As a consequence, the laplacian pyramid is a multi-frequencies decomposition.

Seamless image-based 3D textures

Summing laplacian pyramid

3 –

Mu

lti-

vie

w b

len

din

g

Important remark:Summing through successive expansions the laplacian pyramid allows to get back the gaussian pyramid and, so, the original image.

Seamless image-based 3D textures

?2D blending

Second step:Create a new laplacian pyramid from the image ones through the gaussian pyramids of their masks.

3 –

Mu

lti-

vie

w b

len

din

g

2D blending:Merging in a “natural” way the pictures following their respective masks.

First step:Build the laplacian pyramid for each image, andBuild the gaussian pyramid for each mask.

Third step:Get the final blended image by summing the newly created laplacian pyramid.

Seamless image-based 3D textures

Multi-view blending

Straightforward from 2D blending using the projection functions for each image .

So we obtain the final color at point x of the surface:

with

3 –

Mu

lti-

vie

w b

len

din

g

Seamless image-based 3D textures

- 4 -Atlas building and

results

Seamless image-based 3D textures

Results – Aiguille du Midi (France)

4 –

Atl

as b

uild

ing

an

d r

esu

lts

"Naive"  approach Optimized patchwork

Optimized patchwork +

blending

Successive approaches

Copyright Bernard Vallet (www.bvallet.com)

Seamless image-based 3D textures

Atlas – Aiguille du Midi (France)

4 –

Atl

as b

uild

ing

an

d r

esu

lts

Seamless image-based 3D textures

Results – Ettlingen castle (Germany)

4 –

Atl

as b

uild

ing

an

d r

esu

lts

Optimized patchwork +

blending

Successive approaches

Courtesy Christoph Strecha, EPFL (http://cvlab.epfl.ch/strecha/multiview/)

"Naive"  approach Optimized patchwork

Seamless image-based 3D textures

Atlas – Ettlingen castle (Germany)

4 –

Atl

as b

uild

ing

an

d r

esu

lts

Seamless image-based 3D textures

Conclusion

Seamless image-based 3D textures: Graph-based combinatorial optimization for texture partition Multi-view blending on the partition

Advantages: Good results

Drawbacks: Computing time dependant of the number of levels in the

pyramids

Future work: GPU acceleration of the blending part

4 –

Atl

as b

uild

ing

an

d r

esu

lts

Seamless image-based 3D textures

Thank you for your attention!

Any question?