automatic high dynamic range image generation for dynamic ... · instance for ghost free method...
TRANSCRIPT
![Page 1: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/1.jpg)
IEEE Computer Graphics and Applications Vol. 28, Issue. 2, April 2008
Katrien Jacobs, Celine Loscos, and Greg Ward
Presented by Yuan Xi
School of Electrical Engineering and Computer Science Kyungpook National Univ.
Automatic High Dynamic Range Image Generation for Dynamic Scenes
![Page 2: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/2.jpg)
Abstract Requirement of HDRI generation
– Multi-exposure LDR images • Static scene throughout LDR images
Proposed method – Ghost-free
• LDRI alignment − Focus on camera
• Deleting moving objects − Without camera curve − Independent from contrast between background and moving object
2/26
![Page 3: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/3.jpg)
Introduction High dynamic range
– Weakness of LDR images • Loss of information in under or over exposed area
– High dynamic range image • Combining details in multi-exposure LDR images
Problems of High dynamic range image processing – Acquirement of exposure time
• Changing exposure time manually – Ghost
• Camera movement • Object movement
3/26
![Page 4: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/4.jpg)
Instance for ghost free method
Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people walk through the viewing window. (b) An HDRI created from the sequence shown in (a) using conventional methods, showing ghosting effects (black squares). (c) Uncertainty image (UI) shows regions of high uncertainty (bright) due to the dynamic behaviour of those pixels. (d) HDRI of the same scene after applying movement removal using UI.
4/26
![Page 5: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/5.jpg)
Background Debevec’s method
– Depend on good alignment between LDRIs
Rescal’s method – Providing manual image alignment method
Reinhard’s method – Generating binary-map with weighted variance between LDRIs
Kang’s method – Using camera response function to normalize LDRIs – Performing local image alignment using gradient-based optical flow
5/26
![Page 6: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/6.jpg)
MTB method – Median threshold bitmap
• Threshold − Median pixels value
– Ghost detection • Getting summation of all bitmap image
− Classifying pixel neither 0 or N(number of LDR images) as movement
Sand’s method – Basis on feature-based method – Without camera response curve
Khan’s method(segmentation method) – Using kernel density estimation method to estimation motion
area
6/26
![Page 7: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/7.jpg)
HDRI Generation: An Overview Flowchart of proposed method
Fig. 2. HDRI generation methodology: the rounded white and grey boxes are processes that operate on input data and produce output data. The rounded grey boxes are modules developed for this paper.
7/26
![Page 8: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/8.jpg)
Camera Alignment Generate alignment method
– Calculating camera transformation with scene features • Euclidean transformation
− Rotation and translation • Scene features
− Different intensity − Different color − Edge
– Weakness • Impossible find edges perfectly in every image
8/26
![Page 9: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/9.jpg)
• Failing instance of edge finding
Fig. 3. (a,b) Two LDRIs captured with different exposures. (c,d) Edge images of two LDRIs. (e,f) Bitmap images of two LDRIs after applying MTB transformation. 9/26
![Page 10: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/10.jpg)
Proposed alignment method – Using MTB method instead of canny edge detection
• Threshold set as median intensity of each image – Using optimization method
XOR E N D
Minima?
NO!
YES! Euclidean transformation
10/26
![Page 11: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/11.jpg)
Movement Detection Generate alignment method
– Detecting movement clusters • Clusters of pixels affected by movement in any of LDRIs
– Using two evaluation method • Movement detection based on variance • Contrast-independent movement detection
11/26
![Page 12: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/12.jpg)
Movement detection based on variance – Focus on variance over irradiance images
• Pixels affected by movement showing large irradiance variation • Movement derived from variance image
– Achieving variance image • Definition of weighted variance:
− Weighted sum of squares at each pixel over the square of weighted average, the quantity minus 1
• Aim of weight-processing
− Decreasing effect of redundancy information stored in distortion pixels
2
0 0
2 2
0 0
( , ) ( , ) / ( , )( , ) 1
( ( , ) ( , )) / ( ( , ))
N N
i i ii i
N N
i i ii i
W k l E k l W k lVI k l
W k l E k l W k l
= =
= =
= −∑ ∑
∑ ∑(1)
where is weight used during HDRI generation.
( , )iW k l
12/26
![Page 13: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/13.jpg)
• Weight-function − Hat function
13/26
![Page 14: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/14.jpg)
• Binary image making with threshold − Setting threshold to
– Flowchart of proposed method 0.18
Fig. 4. An adaptation of figure 2 illustrates where movement detector based on variance fits inside general HDRI generation framework. Variance detector requires knowledge of camera curve, and therefore movement detector takes place after camera curve calibration.
14/26
![Page 15: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/15.jpg)
– Weakness of movement detection based on variance • Other influences existing, besides remaining camera
misalignment and movement object − Camera curve
» Fail to convert intensity values to irradiance values − Weighting factors − Inaccuracies in exposure speed and aperture width
15/26
![Page 16: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/16.jpg)
Contrast-independent movement detection – Definition of entropy
• In information theory − Uncertainty that remains about a system, after having taken into
account observable properties • Entropy of variable is given by: ( )H X X
( ) ( ) log( ( ))x
H X P X x P X x= − = =∑ (2)
where is random variable with probability function . ranging over a certain interval, for instance [0,255].
X ( ) ( )p x P X x= =x
16/26
![Page 17: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/17.jpg)
– Entropy providing information • Entropy of image has positive value between [0, ]
− The lower entropy, the more less different intensity values • Actual order or organization of pixel intensities in image does not
influence entropy − Only focus on entirety, ignore specific distribution
• Applying scaling factor on intensity values of image does not change its entropy, if intensity values do not saturate
• Entropy of image gives measure of uncertainty of pixels in image − All intensity values are equal: entropy is zero − All intensity values are different: entropy is one
log( )M
17/26
![Page 18: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/18.jpg)
– Calculating UI(uncertainty image) with 2D window • Generating entropy image from LDR image
− One pixel value in entropy image corresponding to entropy value of around region(2D window) at same position 1
0( , ) ( ) log( ( ))
M
ix
H k l P X x P X x−
=
= − = =∑ (3)
where is derived from normalized histogram constructed from the intensity values of pixels within 2D window. and, over all pixels in:
( )P X x=
p
{ ( : , : }ip L k k l lω ω ω ω∈ − + − + (4)
18/26
![Page 19: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/19.jpg)
– Uncertainty image • Local weighted entropy difference
1,
,10 0
,0 0
( , ) ( , )j iN
i ji jj iN
i ji j
i j
UI k l h k lυ
υ
<−
<−= =
= =
=∑∑∑∑
(5)
, ( , ) | ( , ) ( , ) |i j i jh k l H k l H k l= − (6)
, min( ( , ), ( , ))i j i jW k l W k lυ = (7)
19/26
![Page 20: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/20.jpg)
HDRI Generation Final flowchart for proposed method
Fig. 5. An adaptation of figure 2 illustrates where contrast-independent movement detector, explained in section V-B, fits inside general HDRI generation framework. Movement detector does not require knowledge about camera curve, therefore movement detector can take place before camera calibration.
20/26
![Page 21: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/21.jpg)
Results Comparison of resulting aligned or not
(a) (d)
Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f).
21/26
![Page 22: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/22.jpg)
(b) (e)
Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f).
22/26
![Page 23: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/23.jpg)
Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f).
(c) (f)
23/26
![Page 24: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/24.jpg)
HDRI generation and movement(general) removal
(d)
(a) (b)
(c)
Fig. 7. HDRI generation and movement removal for the exposure sequence shown in figure 1 (a). (a) HDRI after object movement removal using variance detector discussed in section V-A. (b) HDRI after object removal using uncertainty detector discussed in section V-B. (c) Variance image V I used to generate (a). (d) Uncertainty image UI used to generate (b).
24/26
![Page 25: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/25.jpg)
HDRI generation and movement(fluid) removal
(d)
(a) (b)
(c)
Fig. 8.(a) HDRI without movement removal: leaves on left hand side show considerable ghosting. (b) HDRI after movement removal using uncertainty image UI shown in (d). (c) Variance image VI. (d) Uncertainty image UI used to generate (b).
25/26
![Page 26: Automatic High Dynamic Range Image Generation for Dynamic ... · Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people](https://reader033.vdocument.in/reader033/viewer/2022052012/6027d48f1d059c27b264cd93/html5/thumbnails/26.jpg)
Proposed method – Camera alignment
• Using MTB method − More better performance than simple edge based methods
– Novel criterion : Entropy • Contrast-independent movement detection • Proposed definition of UI(uncertainty image)
Experimental results – More stable than existing method – Pleasing resulting for fluid ghosts
Future work – Research for more better alignment method
Conclusion And Future Work
26/26