the ibm pieta project: a historical perspective gabriel ...allen/s10/ibmpieta-abridged.pdfpieta`...
TRANSCRIPT
![Page 1: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/1.jpg)
The IBM Pieta Project: A Historical Perspective
Gabriel TaubinBrown University
![Page 2: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/2.jpg)
IBM Pietà 3D Scanning Project : 1998-2000
http://www.research.ibm.com/pieta
Shape
Appearance
![Page 3: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/3.jpg)
Data Capture: Range + multi-texture
“Virtuoso” Multi-baselineStereo camera
5 point light sources
![Page 4: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/4.jpg)
Photometric capture
Same viewpoint, different lightingResolution of .5mm with Virtuoso built-in cameraCompute reflectance and normals per pixel
Our Addition
![Page 5: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/5.jpg)
Capturing ~800 scans (1998)
![Page 6: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/6.jpg)
Design Considerations: Length Scales
2. 25m
O. 15m
Examine on the scale ofMeters to study
proportion, design
Examine on the scale ofmillimeters to study
Tool marks
![Page 7: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/7.jpg)
Controlled Views
How was sculpture supposed to be view?
![Page 8: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/8.jpg)
Orthographic and Impossible Views
How was sculpture constructed?
![Page 9: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/9.jpg)
Representation for Interactive Viewing
I nt er act i ve Vi ewer
Si mpl i f i ed, t ext ur ed
model
Li ght edi t or
Ful l - r es model ,
al bedo, nor mal s
Camer a par ams Al bedo, nor mal s ( per pi xel )Polygon MeshBump mapColor map
![Page 10: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/10.jpg)
Reconstruction Pipeline
![Page 11: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/11.jpg)
![Page 12: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/12.jpg)
Ball Pivoting
To create a meshfrom a point cloudA ball "walks" over the point cloud, creating a triangle for every three points it touches
Pieta` Project
![Page 13: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/13.jpg)
Improving Registration:Using Textures to Refine Alignment
Pieta` Project
![Page 14: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/14.jpg)
color images forfive light positions
details
color
Photometric Processing
Computing colors and normals consistent with underlying geometry and each other
Pieta` Project
![Page 15: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/15.jpg)
SHAPE FROM SHADINGWe can infer shape from a monocular image if the ilumination and surface properties are known. If multiple
images are available under different illumination directions, we can compute the surface orientation at each pointin the image.
Reflectance Functions: First, we need to know how a surface reflects incident light energy. A commonmodel is a diffuse or Lambertian surface. This is a surface that reflects incident light energy equally in all viewingdirections. The amount of this light energy is proportional to the cosine of the angle between the surface normaland the light source direction.
Φ(n, s, v) = ρ cos(i)
where i is angle between surface normal n and light souce direction s, and v is viewing (camera direction).ρ is the albedo of the surface; it is a gain that determines what percentage of the light is transmitted by thesurface versus absorbed. ρ will be high for smooth, glossy objects and lower for rough objects. We will assumeit to be a constant over the surface; i.e. a homogeneous surface material.
If we examine the equation below, we see that if the lighting direction s and viewing direction v are known,then the brightness at a point in the image is simply a function of the surface normal and the albedo (which weassume is constant over the surface).
Φ(n, s, v) = ρ cos(i)
PHOTOMETRIC STEREOIf we have a Lambertian surface, and we shine a known light source on it, we generate a reflectance map
that can be characterized for each pixel in the image as:
Ik(u, v) = ρ cos(ik)= ρ (sk · n)
The intensity at a pixel is proportional to the incident angle. Given three light source directions, s1, s2 s3,we can create a matrix equation I = ρ S n:
⎡⎢⎣
I1(u, v)I2(u, v)I3(u, v)
⎤⎥⎦ = ρ
⎡⎢⎣
s11 s12 s13
s21 s22 s23
s31 s32 s33
⎤⎥⎦
⎡⎢⎣
n1
n2
n3
⎤⎥⎦
Since n is a unit vector, we can solve for ρ and n:
ρ = ‖ S−1 I ‖ ; n =1ρ
S−1 I
1
![Page 16: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/16.jpg)
Data Capture: Range + multi-texture
“Virtuoso” Multi-baselineStereo camera
5 point light sources
![Page 17: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/17.jpg)
surface normal
light source
L
Δω
θ
lightreceivedbycamera
Lr = ρLcosθΔω/ π
n
Pieta` Project
![Page 18: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/18.jpg)
Pieta` Project
![Page 19: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/19.jpg)
Pieta` Project
![Page 20: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/20.jpg)
Remapping Unique Texture
Pieta` Project
![Page 21: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/21.jpg)
for each patch for each camera pos compute tex coords init z-buffer with depth map render weights for tex in {alb, np, nm} render textured patch acctex += rendered*weight accwgt += weight end end normalize save the three imagesend
Blend textures with weights based on data reliability
Pieta` Project
![Page 22: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/22.jpg)
Pieta` Project
![Page 23: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/23.jpg)
Pieta` Project
CapturedSinglephotometric
Geometric registration
Texture registration
![Page 24: The IBM Pieta Project: A Historical Perspective Gabriel ...allen/S10/ibmPieta-abridged.pdfPieta` Project. SHAPE FROM SHADING Wecan infer shape from amonocular image ifthe ilumination](https://reader035.vdocument.in/reader035/viewer/2022081400/60a5b811a874c912746dc944/html5/thumbnails/24.jpg)
Pieta` Project