non conventional imaging systems : application to 3d ... filenon conventional imaging systems :...
TRANSCRIPT
Non Conventional Imaging Systems :
Application to 3D scanning of Transparent Objects
Fabrice MERIAUDEAU
Laboratory Le2i UMR 5158
Jakarta, Indonesia
06- 01-2012
Contributors:
M. Ferraton, R. Rantoson, G. Eren, L. Sanchez-Sécades (PhD Students)
C. Stolz, O. Aubreton, F. Truchetet, D. Fofi, R. Seulin (Academic collaborators)
3D Digitization
Contact
Touching Probe
…..
Non Contact
Reflection Transmission Emission
Outlines
Introduction
State of the Art
Scanning From Heating
Background
Method
Application to Glass
Implementation & Experimental Results
Shape from polarization
Polarization Imaging
Applications
« Shape from polarization »
Experimental System and result
Shape from UV
Background
Implemenation and results
Conclusion
Outlines
Introduction
State of the Art
Scanning From Heating
Background
Method
Application to Glass
Implementation & Experimental Results
Shape from polarization
Polarization Imaging
Applications
« Shape from polarization »
Experimental System and result
Shape from UV
Background
Implemenation and results
Conclusion
Passive 3D Scanning Stereo Vision Shape From Focus
Active 3D Scanning
Laser Triangulation Pattern Projection Time of Flight
Manual Scanning
• Long
• Tedious
• Highly skilled operator
• Results
=> tied to operator’s expertise
Design Solution for 3D Digitization
◦ Results => Independent of User Skills
◦ Automatic & Fast Acquisition
◦ Optimized View Planning
for Maximal Surface Coverage
◦ Post-Processing of Delivered 3D Model
Object 3D Model
Manual
Teaching
Model Based
Non Model Based
Automation Robot CMM Arm => Internal 7-axis articulated arm with an external skeleton driven by electromotors Drives laser line scanner => programmed motion path
Automation => Manual Teaching
http://www.metris.com
Automation Fringe Projection Scanner Head Mounted on Industrial Robot Arm
Automation => Manual Teaching
http://www.steinbichler.de
Model Based – Offline View Planning
Sensor & Robot Modeling
1 Face = 1 ViewPoint
Visibility Study : Binary Table
Optimization : Set Covering Problem
Automation => Model Based
Non Model Based – Online View
Planning Surface
•Mass Vector Chains (MVC)
•Sum of Surface Normal’s
•Weighted by Surface‘s Area
Automation => Non Model Based
Non Model Based – Online View
Planning Surface
•Mass Vector Chains (MVC)
•Sum of Surface Normal’s
•Weighted by Surface‘s Area
Automation => Non Model Based
Non Model Based – Online View
Planning Surface
•Mass Vector Chains (MVC)
•Sum of Surface Normal’s
•Weighted by Surface‘s Area
Automation => Non Model Based
Components
Fringes Projection
3D scanner head
6 DOF
Robot Turntable
Automation => Implementation
3D Digitization Cell
Automation => Implementation
Types of Surfaces
Diffuse (Lambertian) Glossy Specular
Translucent Transparent
Type de surface
19 Diffuse Surface Specular Surface Transparent Surface
Type de surface
20 Diffuse Surface Specular Surface Transparent Surface
Problem to be solved
« Laser scanning »
21 Surface transparente
Outlines
Introduction
State of the Art
Scanning From Heating
Background
Method
Application to Glass
Implementation & Experimental Results
Shape from polarization
Polarization Imaging
Applications
« Shape from polarization »
Experimental System and result
Shape from UV
Background
Implemenation and results
Conclusion
➡ New methods:
➡“Scanning From Heating”
➡Shape from polarization
➡Shape from UV
‣ Capable of scanning different transparent materials and type of surfaces
Increasing demand for three-dimensional (3D) applications
◦ object modeling, preservation of historic artifacts, quality control,...
Transparent objects have not received much attention
High demand for in-line 3D quality control of transparent products
For complex geometric forms: touch probe scanners
◦ too slow for inline inspection
◦ statistical sampling
INTRODUCTION
State of the Art Methods
Structured LightS. Hata, Y. Saitoh, S. Kumamura, and
K. Kaida, “Shape extraction of
transparent object using genetic
algorithm," in Pattern Recognition,
Proceedings of the 13th International
Conference on, vol. 4, 1996.
Shape From MotionM. Ben-Ezra and S. Nayar, "What
does motion reveal about
transparency?" in Proc. IEEE Int'l
Conf. Computer Vision, 2003, pp.
1025-1032.
Optical FlowS. Agarwal, S. Mallick, D. Kriegman,
and S. Belongie, "On refractive optical
flow," in Proc. ECCV'04, 2004, pp.
483-494.
Structured LightS. Hata, Y. Saitoh, S. Kumamura, and
K. Kaida, “Shape extraction of
transparent object using genetic
algorithm," in Pattern Recognition,
Proceedings of the 13th International
Conference on, vol. 4, 1996.
Shape From MotionM. Ben-Ezra and S. Nayar, "What
does motion reveal about
transparency?" in Proc. IEEE Int'l
Conf. Computer Vision, 2003, pp.
1025-1032.
Optical FlowS. Agarwal, S. Mallick, D. Kriegman,
and S. Belongie, "On refractive optical
flow," in Proc. ECCV'04, 2004, pp.
483-494.
State of the Art Methods
FluorescenceM. B. Hullin, M. Fuchs, I. Ihrke, H.-P.
Seidel, and H. P. A. Lensch,
"Fluorescent immersion range
scanning," ACM Trans. Graph., vol.
27, no. 3, pp. 1-10, 2008.
PolarizationD. Miyazaki and K. Ikeuchi, "Inverse
polarization raytracing: estimating
surface shapes of transparent
objects," in IEEE Computer Society
Conference on Computer Vision and
Pattern Recognition, vol. 2, 2005, p.
910.
Shape From
DistortionM. Tarini, H. Lensch, M. Goesele, and
H. Seidel, "3d acquisition of mirroring
objects using striped patterns,"
Graphical Models, vol. 67, no. 4, pp.
233-259, 2005.
FluorescenceM. B. Hullin, M. Fuchs, I. Ihrke, H.-P.
Seidel, and H. P. A. Lensch,
"Fluorescent immersion range
scanning," ACM Trans. Graph., vol.
27, no. 3, pp. 1-10, 2008.
PolarizationD. Miyazaki and K. Ikeuchi, "Inverse
polarization raytracing: estimating
surface shapes of transparent
objects," in IEEE Computer Society
Conference on Computer Vision and
Pattern Recognition, vol. 2, 2005, p.
910.
Shape From
DistortionM. Tarini, H. Lensch, M. Goesele, and
H. Seidel, "3d acquisition of mirroring
objects using striped patterns,"
Graphical Models, vol. 67, no. 4, pp.
233-259, 2005.
Outlines
Introduction
State of the Art
Scanning From Heating
Background
Method
Application to Glass
Implementation & Experimental Results
Shape from polarization
Polarization Imaging
Applications
« Shape from polarization »
Experimental System and result
Shape from UV
Background
Implemenation and results
Conclusion
Transparency
Human flesh is transparent to X-Ray
while bone is not
Electromagnetic Spectrum
Transparency - Example Case of
Glass
Transparent glass bottle in front of an infrared heat sourceand Image
taken with a long wave infrared camera
Evolution of the refraction and absorption index
of glass depending on the wavelength
Glass is Opaque in UV and in IR
We have developed methods making use of this property
Thermal Radiation
Energy distribution of a blackbody Thermal Image of an house
Thermal Radiation
*>47.7°C
*<-11.6°C
-10.0
-5.0
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
Application to Glass - Emissivity
Emissivity of a glass sphere
๏ G. Gaussorgues and S. Chomet, Infrared thermography. Kluwer Academic Publishers,
1994.
A new approach:
Scanning From Heating G. Eren, F. Meriaudeau and al., Optics Express 2009.
Scanning From Heating
For the method to work properly:
The object surface should be opaque to the laser source i.e. the laser should be absorbed by the
surface
The object has to emit Once the surface is heated, the thermal emission should be omnidirectional
so that the thermal camera can capture accurately the heated point on different curvatures of the
surface
Homogenous, Isotropic Uniform in structure and composition
Physical properties of the material are independent of direction
Application to Glass - Heating
Model
P0, power of the CO2 laser beamr, radius of the CO2 laser beam∂(z), impulse function
๏ Jiao, J., Wang, X.: A numerical simulation of machining glass by
dual
CO2-laser beams. Optics and Laser Technology 40(2) (2008) 297-301
Application to Glass - Heating
Model • To bring the surface form 20 to 80 degrees with v=10mm/s, r=1.5mm
➡Laser power : 3W
Experimental results Experimental results compared to the heating model
Application to Glass - Selection of
the Camera
Transmission of light as a percentage in the infrared
domain
Transmission of light as a percentage in the
infrared domain of commonly used glasses
The Scanner
Back View
Camera Calibration
Custom calibration plate
Calibration image set
Results - Glass Plate
Results obtained on a 10x5 cm glass plate
• The reconstruction is compared to a perfect plane
• Average deviation is 150 µm
Results - Glass Cup
Transparent glass cup
3D reconstruction
by SFH Scanner 3D reconstruction (after being powdered)BY Minolta 3D Laser Scanner
Comparison of the reconstructions
(average deviation is 210 µm)
Results - Automotive Glass
Automotive Glass
Reconstruction by a Touch Probe Scanner Comparison of the reconstructions
(average deviation is 360 µm)
Results - Plastic
Transparent plastic bottle 3D reconstruction by SFH Scanner
Comparison to Minolta VI-910
(average deviation is 540 µm)
Line Projection : an extension
Experimental Setup
Meriaudeau and al., IEEE TIM, 2010
Object Reconstruction Object is Powdered
for Comparison
Comparison to Minolta VI-910
(average deviation is 210 µm)
46
Specular object: an extension, A. Bajard, PhD Thesis, 2009/2012
Outlines
Introduction
State of the Art
Scanning From Heating
Background
Method
Application to Glass
Implementation & Experimental Results
Shape from polarization
Polarization Imaging
Applications
« Shape from polarization »
Experimental System and result
Shape from UV
Background
Implemenation and results
Conclusion
Outlines
Introduction
State of the Art
Scanning From Heating
Background
Method
Application to Glass
Implementation & Experimental Results
Shape from polarization
Polarization Imaging
Applications
« Shape from polarization »
Experimental System and result
Shape from UV
Background
Implemenation and results
Conclusion
Electromagnetic Plane waves
Non polarized wave : random phase
Polarized wave :
◦ Linear polarization
◦ Circular polarisation
rktieEE
0
22
11
cos
cos
aE
aE
y
x
rktavec 12: shiftphase
t
0dt
d
mwithm
mwithm
aa
212
21
49
Stokes Vector :
Total Intensity :
Degree of Polarization :
Angle of polarization :
1
2tana
a
0
2
3
2
2
2
1
s
sss
I
I
tot
pol
2
3
2
2
2
1
2
0 ssss
sin2sin
cos2sin
2cos
sin2
cos2
21
21
2
2
2
1
2
2
2
1
3
2
1
0
tot
tot
tot
tot
I
I
I
I
aa
aa
aa
aa
s
s
s
s
s
50
Mueller Matrix : describes the effects of
linear optical systems (polarizers, wave
plates, reflecting surfaces…)
Example : Linear polarizer inclined with an
angle α
51
0000
02sin2sin2cos2sin
02sin2cos2cos2cos
02sin2cos1
22
2
polM
52
Polarization imaging
Polarization imaging in industrial vision:
◦ Avoid specular reflections
◦ Distinguish dielectrics from metallics (Wolff)
◦ …
> Shape from polarization:
Polarization Images Information on the surface normals
Physical principle: after reflection on a surface, an unpolarized light
wave becomes partially linearly polarized.
Studying the state of polarization of the reflected light enables to get
information on the surface normals (Fresnel coefficients).
53
Polarization imaging
Polarization state of a light wave:
◦ Goal: study the state of a partially linearly polarized light
No need to have a stokes polarimeter
Just a rotating polarizer in front of the camera
Imi
n
Imax
180°
minmax
minmax
II
II
• Degree of polarization:
• Magnitude of the light: minmax III
• Angle of polarization: 3 parameters:
Partial Stokes’ polarimeters
Measure of S0, S1, S2
Set-up with a linear polarizer
54
Set-up with elliptical nematics liquid crystal
Set-up with LVCR
55 Wolff et al., 1997
Bigue & Cheney, 2007
Partial Stokes’ polarimeters
Polarization imaging applications
Active polarimetry
◦ Depolarization
56 Morel et al., 2006
Alouini, 2005
« Shape from polarization »
57
Study of the polarization state
of the reflected wave
Normals Determination
Normal field integration
so as to obtain the surface
« Shape from polarization »
58
Principle:
◦ Unpolarized light
◦ Reflective surface
x
y
z
n
i
r
1
sintan
costan
r
r
q
p
n
( , r ) ?
◦ Angle of polarization
◦ Degree of polarization
r
Fresnel reflection coefficients
Angle of polarization
)²(tan
)²(tan
)²(sin
)²(sin
//
ti
ti
ti
ti
F
F
//FF
x
y
z
i
r
n
The linearly polarized component is orthogonal to the plan of incidence
x
y
2
« Shape from polarization »
1st ambiguïty
Degree of polarization r
n z
i
r
t
)²(tan
)²(tan
)²(sin
)²(sin
//
ti
ti
ti
ti
F
F
ti n sinsin
Fresnel reflection coefficients Snell-Descartes Law
)(//
//rf
FF
FF
x
y
z
i
r
n
« Shape from polarization »
« Shape from polarization »
Relation between the degree of
polarisation ρ and the zenithal angle θ
61
2222
22
tansinsin
sintansin2
n
n
2nd ambiguïty
62
Previous approaches of the « Shape from
polarization » for transparent objects
Previous approaches of the « Shape from
polarization » for transparent objects
63 Miyazaki et al., 2002
64 Miyazaki et al., 2004
Previous approaches of the « Shape from
polarization » for transparent objects
Solutions
« Shape from polarization »
65 [Ferraton, 2009] Ferraton, M., Stolz, C. and Meriaudeau, F., "Optimization of a polarization imaging system for 3D measurements
of transparent objects", Optics Express, Optical Society of America, vol. 17 (23), pp. 21077-21082, 2009.
ρ = f(θ,n)
n = f(λ)
λ1 ρ1
λ2 ρ2
ρ = f(θ,λ)
θ
ρ
Solutions
« Shape from polarization »
66 Ferraton & Meriaudeau, 2009
θB θ
ρ
θ1
ρλ1
ρλ2
ρλ2 - ρλ1 > 0 0° < θ1 < θB
Solutions
« Shape from polarization »
67 Ferraton & Meriaudeau, 2009
θB θ
ρ
θ2
ρλ1 ρλ2
ρλ2 - ρλ1 < 0 θB < θ2 < 90°
Solutions
« Shape from polarization »
68 Ferraton & Meriaudeau, 2009
θB θ
ρ
θ2
ρλ2 - ρλ1 < 0 ρλ2 - ρλ1 > 0
θ1
0° < θ1 < θB θB < θ2 < 90°
ρλ1
ρλ2
ρλ1
Outlines
Introduction
State of the Art
Scanning From Heating
Background
Method
Application to Glass
Implementation & Experimental Results
Shape from polarization
Polarization Imaging
Applications
« Shape from polarization »
Experimental System and result
Shape from UV
Background
Implemenation and results
Conclusion
Experimental System
70
12 bit-depth Camera
Orthographic projection Rotating polarizer
Experimental system
71
0° 10°
180°
Acquisition
I
Approximation
Normal extraction
1
sintan
costan
q
p
n
Time < 1sec
Liquid Crystal Variable Retarder (LCVR)
72
λ = 632.8 nm
Experimental system
Optical calibration
73
Rotating polarizer
74
Optical calibration
75
Angle (°) Tension (V)
0 1.565
-5 1.590
-10 1.620
-15 1.650
-20 1.675
-25 1.705
-30 1.745
-35 1.775
-40 1.805
-45 1.835
-50 1.865
-55 1.900
-60 1.935
-65 1.975
-70 2.010
-75 2.055
-80 2.095
-85 2.145
… …
Optical calibration
◦ Saliency operator
76 Walter, N., Aubreton, O. and Laligant, O., "Salient point characterization for low resolution meshes", IEEE International Conference
on Image Processing , pp. 1512-1515, 2008
Optical calibration
77
λB = 472 nm
Inte
nsité lu
min
eu
se
Optical calibration
Optical Calibration Zenithal Angle Ambiguity
78
79
Optical Calibration Zenithal Angle Ambiguity
80
Optical Calibration Zenithal Angle Ambiguity
81
2222 1tansin
tansin2
kn
n
Optical Calibration Zenithal Angle Ambiguity
82
2222 1tansin
tansin2
kn
n
11ˆ 222 knn
n̂
Optical Calibration Zenithal Angle Ambiguity
Pseudo-index
3D Reconstruction
83
0° 10°
180°
Acquisition
I
Approximation
Normal extraction
3D Reconstruction
Multispectral approach to relieve the ambiguity on the Zenithal
Angle
84
Active lighting approach to relieve the ambiguity on the azimutal
angle
85
2/.1
,03100.2 quadquadquad IIIsi
Morel et al., 2005
3D Reconstruction
Active lighting approach to relieve the ambiguity on the azimutal
angle
86
2/.1
,03100.2 quadquadquad IIIsi
Morel et al., 2005
3D Reconstruction
Normal Evaluation
Integration
87
1
sintan
costan
1
,
,
q
p
y
yxfx
yxf
n
22
~~,
~,0,0,
vu
qjvpjuvufvu
Frankot, R. and Chellappa, R., "A method for enforcing integrability in shape from shading algorithms", IEEE Transactions on
Pattern analysis and Machine Intelligence, vol. 10, pp. 439-451, 1988
yxgyxfyxf ,,, 0
kqypxyxg 0,0~0,0~,
88
Std: 0.070 mm , Mean. : -0.015 mm, Min : -0.226 mm, Max : 0.150 mm
89
90
Shape from polarisation in the IR (results from september 2011)
Expérimenta Set-upl: IR camera Flir 3µm - 5 µm, polariser ZnSe 1µm -15µm, « IR dome »
made of 4x14 resistors (12Ω et 0.25W) with a 9V power generator. One resistor 60° ~
maximal radiation around 8.7µm according to de Wien’s Law), a piece of glass of complexe
shape
91
« Two ambiguities solved»
Azimutal angle Iquad in the IR
Shape from polarisation in the IR
92
Zénithal Angle complex refraction index for the glass
Shape from polarisation in the IR
« Two ambiguities solved»
93
Camera Calibration Bouguet ‘s Toolbox (z0 ~ 300mm)
Shape from polarisation in the IR
94
α=0°
α=0° α=90°
Without polariseur
Shape from polarisation in the IR
95
Reconstruction par polarisation dans l’IR
Méthode de validation : Carte binaire
Black pixel(110,325) White Pixel (322,292)
70% of the points are useful
)(I
)(I
96
Reconstruction par polarisation dans l’IR
Implémentation et résultats dans l’IR
Polarisation angle Azimutal Angle
97
Reconstruction par polarisation dans l’IR
Degree of polarisation Zenithal Angle
Integration
Outlines
Introduction
State of the Art
Scanning From Heating
Background
Method
Application to Glass
Implementation & Experimental Results
Shape from polarization
Polarization Imaging
Applications
« Shape from polarization »
Experimental System and result
Shape from UV
Background
Implemenation and results
Conclusion
99
Shape From UV
Principle:
Under the UV irradiations, transparent surface reemit fluorescence in the Visible
Optimization of the excitation wavelegnth
Absorbance
Emission Spectra
100
Principle : Strutured lighting system
Reconstruction by active triangulation
Calibration
Matching
Structure lighting: point or line (or pattern)
Sensors: one or two cameras
3D
Shape From UV
101
Shape From UV
102
Results
Internationale Conferences (2)
• 3D Reconstruction of Transparent Objects Exploiting Surface Fluorescence caused by
UV Irradiation, Rindra Rantoson, Christophe Stolz, David Fofi, Fabrice Meriaudeau, IEEE
International Conference on Image Processing (ICIP), Hong Kong, September 2010.
• Non Contact 3D Measurement Scheme for Transparent Objects using UV Structured
Light, Rindra Rantoson, David Fofi, Christophe Stolz, Fabrice Meriaudeau, IEEE International
Conference on Pattern Recognition (ICPR), Istanbul, Turkey, August 2010
Workshop (1)
• Triangulation par stéréovision basée sur l'exploitation des images de fluorescence d'une
surface transparente, Rindra Rantoson, Christophe Stolz, David Fofi, Fabrice Meriaudeau,
Journées imagerie optique non conventionnelle, Paris, France, GDR ISIS, 22 March 2010
Book (1)
• "Scanning from Heating" and "Shape from Fluorescence" two Non Conventional
Imaging Systems for 3D Digitization of transparent objects, Fabrice Mériaudeau, R.
Rantoson, G. Eren, L. Sanchez-Sécades, O.Aubreton, A. Bajard, D. Fofi, I. Mohammed, O.
Morel, C. Stolz, F. Truchetet, IGI Global, Novembre 2011
Active Triangulation by stereovision : Publications
Shape From UV
103
Active Triangulation by monocular vision
two experimental set-ups
Point structured lighting system Line structured lighting system
Shape From UV
104
Shape From UV
Experimental set-up
Active Triangulation by monocular vision
105
Implementation and results : Expérimental Set-up
- UV Laser (266m, 10mW, spot elliptical shape of 2mm)
- CCD RGB (Guppy F-080C, 480x640, 1/3 inch, 8mm focal, 1.4 f-number
- Displacement table
Shape From UV
106
Shape From UV
107
Shape From UV
108
µ = 0.07mm, σ=0.07mm
µ = 0.08mm, σ=0.1mm
Shape From UV
109
µ = 0.08mm, σ= 0.09mm
Shape From UV
110
Experimental set-up
- A UV lasre (266m, 10mW, spot elliptique de 2mm)
- A semi-cylindrical « UV proof » lens
- CCD RGB (Guppy F-080C, 480x640, 1/3 inch, 8mm focal,1.4 f-number
- Displacement table
Line structured lighting system
Shape From UV
111
µ = 0.14mm
σ = 0.12mm
Shape From UV
112
Shape From UV
113
Potential Extensions
Specular Surfaces
Optimizing the tracking
Light pre-heating of the surface
Modelisation of the shape of the UV spot or line
Shape From UV
114
Scanning from Heating
High accuracy
No a priori for the
object
Adapted for online
processes
High Cost
High accuracy
No a priori for the object
Adapted for online processes
Low cost
Pre/post-
processing needed
Multispectral « shape
from polarization »
High accuracy
Tedious Calibrations
Post-processing needed
Scanning from UV
Thank you for your attention
Terima Kasih
115
Non Conventional Imaging Systems :
Application to 3D scanning of Transparent Objects
Fabrice MERIAUDEAU
Laboratory Le2i UMR 5158
Jakarta, Indonesia
06- 01-2012
Contributors:
M. Ferraton, R. Rantoson, G. Eren, L. Sanchez-Sécades (PhD Students)
C. Stolz, O. Aubreton, F. Truchetet, D. Fofi, R. Seulin (Academic collaborators)