noise - graphics.cs.cmu.edugraphics.cs.cmu.edu/courses/15-463/2019_fall/lectures/lecture6.pdf- which...
TRANSCRIPT
Noise
15-463, 15-663, 15-862Computational Photography
Fall 2019, Lecture 6http://graphics.cs.cmu.edu/courses/15-463
Course announcementsโข Homework 2 is out.
- Due September 27th. - Requires camera and tripod.- Start early! Substantially larger programming and imaging components than in Homework 1.- Generous bonus component, up to 50% extra credit.- No really: start early!
โข Any issues with Homework 1?- How did you find homework 1 in general? - Any issues with Matlab?- Any issues using the cameras?- Which part of homework 1 did you enjoy the most?- How many of you did the room-sized camera obscura bonus?
โข New policy regarding deadlines and submission times: deadlines will be strictly enforced.- Policy applies starting with Homework 2.- Miss deadline by one second, and it will cost you a late day. No excuses (e.g., server lagged, laptop crashed).- This is how things work in real academic deadlines (e.g., conference deadlines).- Many homeworks require submitting large files, so be mindful of that.- Do not leave uploading your Homework 2 solution for the last minute!
Overview of todayโs lecture
โข Leftover from lecture 5: other aspects of HDR imaging.
โข Leftover from lecture 5: tonemapping.
โข Leftover from lecture 5: Some notes about HDR imaging and tonemapping.
โข A few motivating examples.
โข Sensor noise.
โข Noise calibration.
โข Optimal weights for HDR.
Slide credits
Many of these slides were inspired or adapted from:
โข James Hays (Georgia Tech).โข Fredo Durand (MIT).โข Gordon Wetzstein (Stanford).โข Marc Levoy (Stanford, Google).โข Sylvain Paris (Adobe).โข Sam Hasinoff (Google).
A few motivating questions from things weโve seen
Side-effects of increasing ISOImage becomes very grainy because noise is amplified.โข Why does increasing ISO increase noise?
Tonemapping for a single image
Careful not to โtonemapโ noise.โข Why is this not a problem
with multi-exposure HDR?
Modern DSLR sensors capture about 3 stops of dynamic range.โข Tonemap single RAW file instead of using cameraโs default rendering.
Merging non-linear exposure stacks
1. Calibrate response curve
2. Linearize images
For each pixel:
3. Find โvalidโ images
4. Weight valid pixel values appropriately
5. Form a new pixel value as the weighted average of valid pixel values
(noise) 0.05 < pixel < 0.95 (clipping)
(pixel value) / ti
Same steps as in the RAW case.
Note: many possible weighting schemes
Many possible weighting schemesโConfidenceโ that pixel is noisy/clipped
โข What are the optimal weights for merging an exposure stack?
Sensor noise
A quick note
โข We will only consider per-pixel noise.
โข We will not consider cross-pixel noise effects (blooming, smearing, cross-talk, and so on).
Noise in images
Results in โgrainyโ appearance.
The (in-camera) image processing pipeline
Which part introduces noise?
analog front-end
RAW image (mosaiced,
linear, 12-bit)white
balanceCFA
demosaicingdenoising
color transforms
tone reproduction
compressionfinal RGB
image (non-linear, 8-bit)
The (in-camera) image processing pipeline
Which part introduces noise?โข Noise is introduced in the green part.
analog front-end
RAW image (mosaiced,
linear, 12-bit)white
balanceCFA
demosaicingdenoising
color transforms
tone reproduction
compressionfinal RGB
image (non-linear, 8-bit)
The noisy image formation process
What are the various parts?
The noisy image formation process
analog-to-digital converter (ADC)
analog voltage L
analog voltage G
discrete signal I
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
The noisy image formation process
analog-to-digital converter (ADC)
analog voltage L
analog voltage G
discrete signal I
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
introduces photon noise and dark noise
introduces read noise
introduces ADC noise
โข We will be ignoring saturation, but it can be modeled using a clipping operation.
simulated mean #photons/pixel
0.001 0.01 0.1
1 10 100
Photon noiseA consequence of the discrete (quantum) nature of light. โข Photon detections are independent random events.โข Total number of detections is Poisson distributed.โข Also known as shot noise and Schott noise.
Ndetections โผ Poisson[t โ ฮฑ โ ฮฆ]
photon noise depends on scene flux and exposure
Can you think of examples when dark noise is important?
Dark noiseA consequence of โphantom detectionsโ by the sensor. โข Electrons are randomly released without any photons.โข Total number of detections is Poisson distributed.โข Increases exponentially with sensor temperature (+6ยฐC โ doubling).
Ndetections โผ Poisson[t โ D]
dark noise depends on exposure but not on scene
Can you think of examples when dark noise is important?
โข Very long exposures (astrophotography, pinhole camera).
Dark noiseA consequence of โphantom detectionsโ by the sensor. โข Electrons are randomly released without any photons.โข Total number of detections is Poisson distributed.โข Increases exponentially with sensor temperature (+6ยฐC โ doubling).
Ndetections โผ Poisson[t โ D]
dark noise depends on exposure but not on scene
Can you think of ways to mitigate dark noise?
Can you think of examples when dark noise is important?
โข Very long exposures (astrophotography, pinhole camera).
Dark noiseA consequence of โphantom detectionsโ by the sensor. โข Electrons are randomly released without any photons.โข Total number of detections is Poisson distributed.โข Increases exponentially with sensor temperature (+6ยฐC โ doubling).
Ndetections โผ Poisson[t โ D]
dark noise depends on exposure but not on scene
Can you think of ways to mitigate dark noise?
โข Cool the sensor.
โข Average multiple shorter exposures.
Poisson distributionIs it a continuous or discrete probability distribution?
Poisson distributionIs it a continuous or discrete probability distribution?
โข It is discrete.
How many parameters does it depend on?
Poisson distributionIs it a continuous or discrete probability distribution?
โข It is discrete.
What is its probability mass function?
How many parameters does it depend on?
โข One parameter, the rate ฮป.
Poisson distributionIs it a continuous or discrete probability distribution?
โข It is discrete.
What is its probability mass function?
๐ โผ Poisson(๐) โ ๐(๐ = ๐; ๐) =๐๐๐โ๐
๐!
How many parameters does it depend on?
โข One parameter, the rate ฮป.
What is its mean and variance?
Poisson distributionIs it a continuous or discrete probability distribution?
โข It is discrete.
What is its probability mass function?
๐ โผ Poisson(๐) โ ๐(๐ = ๐; ๐) =๐๐๐โ๐
๐!
How many parameters does it depend on?
โข One parameter, the rate ฮป.
What is its mean and variance?
โข Mean: ฮผ(N) = ฮป
โข Variance: ฯ(N)2 = ฮป
The mean and variance of a Poisson random variable both equal the rate ฮป.
What is the distribution of the sum of two Poisson random variables?
Poisson distributionIs it a continuous or discrete probability distribution?
โข It is discrete.
What is its probability mass function?
๐ โผ Poisson(๐) โ ๐(๐ = ๐; ๐) =๐๐๐โ๐
๐!
How many parameters does it depend on?
โข One parameter, the rate ฮป.
What is its mean and variance?
โข Mean: ฮผ(N) = ฮป
โข Variance: ฯ(N)2 = ฮป
The mean and variance of a Poisson random variable both equal the rate ฮป.
What is the distribution of the sum of two Poisson random variables?
๐1 โผ Poisson ๐1 , ๐2 โผ Poisson ๐2 โ ๐1 + ๐2 โผ Poisson ๐1 + ๐2
Fundamental question
Why are photon noise and dark noise Poisson random variables?
The noisy image formation process
analog-to-digital converter (ADC)
analog voltage L
analog voltage G
discrete signal I
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
introduces photon noise and dark noise
introduces read noise
introduces ADC noise
โข What is the distribution of the sensor readout L?
The distribution of the sensor readoutWe know that the sensor readout is the sum of all released electrons:
๐ฟ = ๐photon_detections + ๐phantom_detections
What is the distribution of photon detections?
The distribution of the sensor readoutWe know that the sensor readout is the sum of all released electrons:
๐ฟ = ๐photon_detections + ๐phantom_detections
What is the distribution of photon detections?
๐photon_detections โผ Poisson(๐ก โ ๐ผ โ ฮฆ)
What is the distribution of phantom detections?
The distribution of the sensor readoutWe know that the sensor readout is the sum of all released electrons:
๐ฟ = ๐photon_detections + ๐phantom_detections
What is the distribution of photon detections?
๐photon_detections โผ Poisson(๐ก โ ๐ผ โ ฮฆ)
What is the distribution of phantom detections?
๐phantom_detections โผ Poisson(๐ก โ ๐ท)
What is the distribution of the sensor readout?
The distribution of the sensor readoutWe know that the sensor readout is the sum of all released electrons:
๐ฟ = ๐photon_detections + ๐phantom_detections
What is the distribution of photon detections?
๐photon_detections โผ Poisson(๐ก โ ๐ผ โ ฮฆ)
What is the distribution of phantom detections?
๐phantom_detections โผ Poisson(๐ก โ ๐ท)
What is the distribution of the sensor readout?
๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ + ๐ท))
The noisy image formation process
analog-to-digital converter (ADC)
analog voltage G
discrete signal I
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
introduces photon noise and dark noise
introduces read noise
introduces ADC noise
analog voltage ๐ฟ,๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ + ๐ท))
Read and ADC noiseA consequence of random voltage fluctuations before and after amplifier. โข Both are independent of scene and exposure.โข Both are normally (zero-mean Guassian) distributed. โข ADC noise includes quantization errors.
nread โผ Normal(0, ฯread)
Very important for dark pixels.
nADC โผ Normal(0, ฯADC)
The noisy image formation process
analog-to-digital converter (ADC)
analog voltage G
discrete signal I
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
introduces photon noise and dark noise
introduces read noise
introduces ADC noise
โข How can we express the voltage G and discrete intensity I?
analog voltage ๐ฟ,๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ + ๐ท))
Expressions for the amplifier and ADC outputs
Both read noise and ADC noise are additive and zero-mean.
โข How can we express the output of the amplifier?
Expressions for the amplifier and ADC outputs
Both read noise and ADC noise are additive and zero-mean.
โข How can we express the output of the amplifier?
๐บ = ๐ฟ โ ๐ + ๐read โ ๐
โข How can we express the output of the ADC?
donโt forget to account for the ISO-dependent gain
Expressions for the amplifier and ADC outputs
Both read noise and ADC noise are additive and zero-mean.
โข How can we express the output of the amplifier?
๐บ = ๐ฟ โ ๐ + ๐read โ ๐
โข How can we express the output of the ADC?
๐ผ = ๐บ + ๐ADC
donโt forget to account for the ISO-dependent gain
The noisy image formation process
analog-to-digital converter (ADC)
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
introduces photon noise and dark noise
introduces read noise
introduces ADC noise
analog voltage ๐ฟ,๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ + ๐ท))
analog voltage ๐บ = ๐ฟ โ ๐ + ๐read โ ๐,๐read โผ Normal(0, ๐read)
discrete signal ๐ผ = ๐บ + ๐ADC,๐ADC โผ Normal(0, ๐ADC)
Putting it all togetherWithout saturation, the digital intensity equals:
๐ผ = ๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC
๐ฟ โผ Poisson ๐ก โ ๐ โ ฮฆ + ๐ท
๐read โผ Normal(0, ๐read)๐ADC โผ Normal(0, ๐ADC)
where
What is the mean of the digital intensity (assuming no saturation)?
๐ธ ๐ผ =
Putting it all togetherWithout saturation, the digital intensity equals:
๐ผ = ๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC
๐ฟ โผ Poisson ๐ก โ ๐ โ ฮฆ + ๐ท
๐read โผ Normal(0, ๐read)๐ADC โผ Normal(0, ๐ADC)
where
What is the mean of the digital intensity (assuming no saturation)?
๐ธ ๐ผ = ๐ธ ๐ฟ โ ๐ + ๐ธ ๐read โ ๐ + ๐ธ ๐ADC
=
Putting it all togetherWithout saturation, the digital intensity equals:
๐ผ = ๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC
๐ฟ โผ Poisson ๐ก โ ๐ โ ฮฆ + ๐ท
๐read โผ Normal(0, ๐read)๐ADC โผ Normal(0, ๐ADC)
where
What is the mean of the digital intensity (assuming no saturation)?
๐ธ ๐ผ = ๐ธ ๐ฟ โ ๐ + ๐ธ ๐read โ ๐ + ๐ธ ๐ADC
= ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐
What is the variance of the digital intensity (assuming no saturation)?
๐ ๐ผ 2 =
Putting it all togetherWithout saturation, the digital intensity equals:
๐ผ = ๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC
๐ฟ โผ Poisson ๐ก โ ๐ โ ฮฆ + ๐ท
๐read โผ Normal(0, ๐read)๐ADC โผ Normal(0, ๐ADC)
where
What is the mean of the digital intensity (assuming no saturation)?
๐ธ ๐ผ = ๐ธ ๐ฟ โ ๐ + ๐ธ ๐read โ ๐ + ๐ธ ๐ADC
= ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐
What is the variance of the digital intensity (assuming no saturation)?
๐ ๐ผ 2 = ๐ ๐ฟ โ ๐ 2 + ๐ ๐read โ ๐ 2 + ๐ ๐ADC2
=
Putting it all togetherWithout saturation, the digital intensity equals:
๐ผ = ๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC
๐ฟ โผ Poisson ๐ก โ ๐ โ ฮฆ + ๐ท
๐read โผ Normal(0, ๐read)๐ADC โผ Normal(0, ๐ADC)
where
What is the mean of the digital intensity (assuming no saturation)?
๐ธ ๐ผ = ๐ธ ๐ฟ โ ๐ + ๐ธ ๐read โ ๐ + ๐ธ ๐ADC
= ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐
What is the variance of the digital intensity (assuming no saturation)?
๐ ๐ผ 2 = ๐ ๐ฟ โ ๐ 2 + ๐ ๐read โ ๐ 2 + ๐ ๐ADC2
= ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐2 + ๐read2 โ ๐2 + ๐ADC
2
The noisy image formation process
analog-to-digital converter (ADC)
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
analog voltage ๐ฟ,๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ + ๐ท))
analog voltage ๐บ = ๐ฟ โ ๐ + ๐read โ ๐,๐read โผ Normal(0, ๐read)
discrete signal ๐ผ = ๐บ + ๐ADC,๐ADC โผ Normal(0, ๐ADC)
๐ผ = min(๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC, ๐ผmax) ๐ธ ๐ผ = ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐
๐ ๐ผ 2 = ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐2 + ๐read2 โ ๐2 + ๐ADC
2
discrete image intensity (with saturation): intensity mean and variance:
saturation level
Affine noise modelCombine read and ADC noise into a single additive noise term:
What is the distribution of the additive noise term?
๐ผ = ๐ฟ โ ๐ + ๐add ๐add = ๐read โ ๐ + ๐ADCwhere
Affine noise modelCombine read and ADC noise into a single additive noise term:
๐add โผ Normal(0, ๐read2 โ ๐2 + ๐ADC
2 )
What is the distribution of the additive noise term?
โข Sum of two independent, normal random variables.
๐ผ = ๐ฟ โ ๐ + ๐add ๐add = ๐read โ ๐ + ๐ADCwhere
Affine noise model
analog-to-digital converter (ADC)
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
analog voltage ๐ฟ,๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ + ๐ท))
discrete signal ๐ผ = ๐บ + ๐add,๐add โผ Normal(0, ๐add)
๐ผ = min(๐ฟ โ ๐ + ๐add, ๐ผmax) ๐ธ ๐ผ = ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐
๐ ๐ผ 2 = ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐2 + ๐add2
discrete image intensity (with saturation): intensity mean and variance:
Some observationsIs image intensity an unbiased estimator of (scaled) scene radiant flux?
Some observationsIs image intensity an unbiased estimator of (scaled) scene radiant flux?
โข No, because of dark noise (term ๐ก โ ๐ท โ ๐ in the mean).
When are photon noise and additive noise dominant?
Some observationsIs image intensity an unbiased estimator of (scaled) scene radiant flux?
โข No, because of dark noise (term ๐ก โ ๐ท โ ๐ in the mean).
When are photon noise and additive noise dominant?
โข Photon noise is dominant in very bright scenes (photon-limited scenaria).
โข Additive noise is dominant in very dark scenes.
Can we ever completely remove noise?
Some observationsIs image intensity an unbiased estimator of (scaled) scene radiant flux?
โข No, because of dark noise (term ๐ก โ ๐ท โ ๐ in the mean).
Can we ever completely remove noise?
โข We cannot eliminate photon noise.
โข Super-sensitive detectors have pure Poisson photon noise.
single-photon avalanche photodiode (SPAD)
When are photon noise and additive noise dominant?
โข Photon noise is dominant in very bright scenes (photon-limited scenaria).
โข Additive noise is dominant in very dark scenes.
Summary: noise regimes
regime dominant noise notes
bright pixels photon noise scene-dependentdark pixels read and ADC noise scene-independent
low ISO ADC noise post-gainhigh ISO photon and read noise pre-gain
long exposures dark noise thermal dependence
๐ผ = min(๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC, ๐ผmax) ๐ธ ๐ผ = ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐
๐ ๐ผ 2 = ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐2 + ๐read2 โ ๐2 + ๐ADC
2
discrete image intensity (with saturation): intensity mean and variance:
Noise calibration
How can we estimate the various parameters?
analog-to-digital converter (ADC)
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
dark current D
analog voltage ๐ฟ,๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ + ๐ท))
analog voltage ๐บ = ๐ฟ โ ๐ + ๐read โ ๐,๐read โผ Normal(0, ๐read)
discrete signal ๐ผ = ๐บ + ๐ADC,๐ADC โผ Normal(0, ๐ADC)
๐ผ = min(๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC, ๐ผmax) ๐ธ ๐ผ = ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐
๐ ๐ผ 2 = ๐ก โ ๐ โ ฮฆ + ๐ท โ ๐2 + ๐read2 โ ๐2 + ๐ADC
2
discrete image intensity (with saturation): intensity mean and variance:
saturation level
Estimating the dark currentCan you think of a procedure for estimating the dark current D?
Estimating the dark currentCan you think of a procedure for estimating the dark current D?
โข Capture multiple images with the sensor completely blocked and average to form the dark frame.
Why is the dark frame a valid estimator of the dark current D?
Estimating the dark currentCan you think of a procedure for estimating the dark current D?
โข Capture multiple images with the sensor completely blocked and average to form the dark frame.
Why is the dark frame a valid estimator of the dark current D?
โข By blocking the sensor, we effectively set ฮฆ = 0.
โข Average intensity becomes:
๐ธ ๐ผ = ๐ก โ ๐ โ 0 + ๐ท โ ๐ = ๐ก โ ๐ท โ ๐
โข The dark frame needs to be computed separately for each ISO setting, unless we can also calibrate the gain g.
For the rest of these slides, we assume that we have calibrated D and removed it from captured images (by subtracting from them the dark frame).
Noise model after dark frame subtraction
analog-to-digital converter (ADC)
scene radiant flux ฮฆ
analog amplifier (gain g = k โ ISO)
sensor (exposure t, quantum efficiency ฮฑ)
analog voltage ๐ฟ,๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ))
analog voltage ๐บ = ๐ฟ โ ๐ + ๐read โ ๐,๐read โผ Normal(0, ๐read)
discrete signal ๐ผ = ๐บ + ๐ADC,๐ADC โผ Normal(0, ๐ADC)
๐ผ = min(๐ฟ โ ๐ + ๐read โ ๐ + ๐ADC, ๐ผmax) ๐ธ ๐ผ = ๐ก โ ๐ โ ฮฆ โ ๐
๐ ๐ผ 2 = ๐ก โ ๐ โ ฮฆ โ ๐2 + ๐read2 โ ๐2 + ๐ADC
2
discrete image intensity (with saturation): intensity mean and variance:
saturation level
analog-to-digital converter (ADC)
analog amplifier (gain g = k โ ISO)
Affine noise model after dark frame subtraction
scene radiant flux ฮฆ
sensor (exposure t, quantum efficiency ฮฑ)
analog voltage ๐ฟ,๐ฟ โผ Poisson(๐ก โ (๐ โ ฮฆ))
discrete signal ๐ผ = ๐บ + ๐add,๐add โผ Normal(0, ๐add)
๐ผ = min(๐ฟ โ ๐ + ๐add, ๐ผmax) ๐ธ ๐ผ = ๐ก โ ๐ โ ฮฆ โ ๐
๐ ๐ผ 2 = ๐ก โ ๐ โ ฮฆ โ ๐2 + ๐add2
discrete image intensity (with saturation): intensity mean and variance:
Estimating the gain and additive noise variance
Can you think of a procedure for estimating these quantities?
Estimating the gain and additive noise variance
1. Capture a large number of images of a grayscale target.
Estimating the gain and additive noise variance
What do you expect the measurements
to look like?
1. Capture a large number of images of a grayscale target.
2. Compute the empirical mean and variance for each pixel, then form a mean-variance plot.
mean
vari
ance
Estimating the gain and additive noise variance
1. Capture a large number of images of a grayscale target.
2. Compute the empirical mean and variance for each pixel, then form a mean-variance plot.
mean
vari
ance
๐ธ ๐ผ = ๐ก โ ๐ โ ฮฆ โ ๐
๐ ๐ผ 2 = ๐ก โ ๐ โ ฮฆ โ ๐2 + ๐add2
โ ๐ ๐ผ 2= ๐ธ ๐ผ โ ๐ + ๐add2
Estimating the gain and additive noise variance
1. Capture a large number of images of a grayscale target.
2. Compute the empirical mean and variance for each pixel, then form a mean-variance plot.
mean
vari
ance ๐ ๐ผ 2 = ๐ธ ๐ผ โ ๐ + ๐add
2
3. Fit a line and use slope and intercept to estimate the gain and variance.
equal to line slope
equal to line intercept๐add
2
๐
How would you modify this procedure to separately estimate read and ADC noise?
Estimating the gain and additive noise variance
1. Capture a large number of images of a grayscale target.
2. Compute the empirical mean and variance for each pixel, then form a mean-variance plot.
mean
vari
ance ๐ ๐ผ 2 = ๐ธ ๐ผ โ ๐ + ๐add
2
3. Fit a line and use slope and intercept to estimate the gain and variance.
equal to line slope
equal to line intercept๐add
2
๐
How would you modify this procedure to separately estimate read and ADC noise?โข Perform it for a few different ISO settings (i.e., gains g).
Important note
Noise calibration should be performed with RAW images!
Optimal weights for HDR merging
Merging non-linear exposure stacks
1. Calibrate response curve
2. Linearize images
For each pixel:
3. Find โvalidโ images
4. Weight valid pixel values appropriately
5. Form a new pixel value as the weighted average of valid pixel values
(noise) 0.05 < pixel < 0.95 (clipping)
(pixel value) / ti
Same steps as in the RAW case.
Note: many possible weighting schemes
Many possible weighting schemesโConfidenceโ that pixel is noisy/clipped
โข What are the optimal weights for merging an exposure stack?
RAW (linear) image formation model
Exposure time:t5 t4 t3 t2 t1
(Weighted) radiant flux for image pixel (x,y): ฮฑ โ ฮฆ(x, y)
What weights should we use to merge these images, so that the resulting HDR image is an optimal estimator of the weighted radiant flux?
Different images in the exposure stack will have
different noise characteristics
Simple estimation exampleWe have two noisy unbiased estimators x and y of the same quantity (e.g., a pixelโs intensity).โข The two estimators have variance ฯ[x]2 and ฯ[y]2.
Simple estimation exampleWe have two noisy unbiased estimators x and y of the same quantity (e.g., a pixelโs intensity).โข The two estimators have variance ฯ[x]2 and ฯ[y]2.
Assume we form a new estimator from the convex combination of the other two:
z = a โ x + (1 โ a) โ y
Simple estimation example
Assume we form a new estimator from the convex combination of the other two:
z = a โ x + (1 โ a) โ y
What criterion should we use in selecting the mixing weight a?
We have two noisy unbiased estimators x and y of the same quantity (e.g., a pixelโs intensity).โข The two estimators have variance ฯ[x]2 and ฯ[y]2.
Simple estimation example
Assume we form a new estimator from the convex combination of the other two:
z = a โ x + (1 โ a) โ y
What criterion should we use in selecting the mixing weight a?โข We should select a to minimize the variance of estimator z. (Why?)
What is the variance of z as a function of a?
We have two noisy unbiased estimators x and y of the same quantity (e.g., a pixelโs intensity).โข The two estimators have variance ฯ[x]2 and ฯ[y]2.
Simple estimation exampleAssume we have two noisy estimators x and y of the same quantity (e.g., intensity at a pixel).โข The two estimators have variance ฯ[x]2 and ฯ[y]2.
Assume we form a new estimator from the convex combination of the other two:
z = a โ x + (1 โ a) โ y
What criterion should we use in selecting the mixing weight a?โข We should select a to minimize the variance of estimator z.
What is the variance of z as a function of a?
ฯ[z]2 = a2 โ ฯ[x]2 + (1 โ a)2 โ ฯ[y]2
What value of a minimizes ฯ[z]2?
Simple estimation exampleSimple optimization problem:
๐ฯ[z]2
๐๐= 0
๐(๐2 โ ฯ[x]2 + (1 โ๐)2 โ ฯ[y]2)
๐๐= 0
2 โ ๐โ ฯ[x]2 โ 2 โ (1 โ๐) โ ฯ[y]2 = 0
๐ =ฯ[y]2
ฯ[x]2 + ฯ[y]21 โ ๐ =
ฯ[x]2
ฯ[x]2 + ฯ[y]2and
โ
โ
โ
Simple estimation examplePutting it all together, the optimal linear combination of the two estimators is
๐ง =ฯ[x]2ฯ[y]2
ฯ[x]2 + ฯ[y]2โ
1
ฯ[x]2๐ฅ +
1
ฯ[y]2๐ฆ
normalization factor
weights inversely proportional to variance
Simple estimation examplePutting it all together, the optimal linear combination of the two estimators is
๐ง =ฯ[x]2ฯ[y]2
ฯ[x]2 + ฯ[y]2โ
1
ฯ[x]2๐ฅ +
1
ฯ[y]2๐ฆ
normalization factor
weights inversely proportional to variance
This is weighting scheme is called Fisher weighting and is a BLUE estimator.
More generally, for more than two estimators,
๐ง =1
ฯ๐=1๐ 1
ฯ[๐ฅ๐]2
โ
๐=1
๐1
ฯ[๐ฅ๐]2๐ฅ๐
Back to HDRGiven unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
๐ผ[๐ฅ, ๐ฆ] =1
ฯ๐=1๐ ๐ค๐ [๐ฅ, ๐ฆ]
โ
๐=1
๐
๐ค๐ ๐ฅ, ๐ฆ1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]
The per-pixel weights ๐ค๐ ๐ฅ, ๐ฆ should be selected to be inversely proportional to that pixelโs variance at each image in the exposure stack.โข How do we compute this variance?
Back to HDR
๐ผ[๐ฅ, ๐ฆ] =1
ฯ๐=1๐ ๐ค๐ [๐ฅ, ๐ฆ]
โ
๐=1
๐
๐ค๐ ๐ฅ, ๐ฆ1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]
The per-pixel weights ๐ค๐ ๐ฅ, ๐ฆ should be selected to be inversely proportional to that pixelโs variance at each image in the exposure stack.โข How do we compute this variance? โ Use affine noise model.
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 = ?
Back to HDR
๐ผ[๐ฅ, ๐ฆ] =1
ฯ๐=1๐ ๐ค๐ [๐ฅ, ๐ฆ]
โ
๐=1
๐
๐ค๐ ๐ฅ, ๐ฆ1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]
The per-pixel weights ๐ค๐ ๐ฅ, ๐ฆ should be selected to be inversely proportional to that pixelโs variance at each image in the exposure stack.โข How do we compute this variance? โ Use affine noise model.
ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 =1
๐กi2 ฯ[๐ผ๐[๐ฅ, ๐ฆ]]2
โ ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 = ?
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
Back to HDR
๐ผ[๐ฅ, ๐ฆ] =1
ฯ๐=1๐ ๐ค๐ [๐ฅ, ๐ฆ]
โ
๐=1
๐
๐ค๐ ๐ฅ, ๐ฆ1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]
The per-pixel weights ๐ค๐ ๐ฅ, ๐ฆ should be selected to be inversely proportional to that pixelโs variance at each image in the exposure stack.โข How do we compute this variance? โ Use affine noise model.
ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 =1
๐กi2 ฯ[๐ผ๐[๐ฅ, ๐ฆ]]2
โ ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 =1
๐กi2 ๐กi โ ๐ผ โ ฮฆ[๐ฅ, ๐ฆ] โ ๐2+ ฯadd
2
Computing the optimal weights requires:1. calibrated noise characteristics.2. knowing the radiant flux ๐ผ โ ฮฆ[๐ฅ, ๐ฆ].
Given unclipped and dark-frame-corrected intensity measurements Ii[x, y] at pixel [x, y] and exposures ti, we can merge them optimally into a single HDR intensity I[x, y] as
This is what we wanted to estimate!
Simplification: only photon noise
If we assume that our measurements are dominated by photon noise, the variance becomes:
ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 =1
๐กi2 ๐กi โ ๐ผ โ ฮฆ[๐ฅ, ๐ฆ] โ ๐2+ ฯadd
2 โ?
Simplification: only photon noise
If we assume that our measurements are dominated by photon noise, the variance becomes:
๐ผ[๐ฅ, ๐ฆ] =1
ฯ๐=1๐ ๐ก๐
โ
๐=1
๐
๐ผ๐[๐ฅ, ๐ฆ]
By replacing in the HDR merging formula and assuming only valid pixels, the HDR estimate simplifies to:
Notice that we no longer weight each image in the exposure stack by its exposure time!
ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 =1
๐กi2 ๐กi โ ๐ผ โ ฮฆ[๐ฅ, ๐ฆ] โ ๐2+ ฯadd
2 โ1
๐กi
๐ผ โ ฮฆ[๐ฅ, ๐ฆ] โ ๐2
Some comparisons
original weights optimal weights assuming only photon noise
More general case
Where does this approximation come from?
If we cannot assume that our measurements are dominated by photon noise, we can approximate the variance as:
ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 =1
๐กi2 ๐กi โ ๐ผ โ ฮฆ[๐ฅ, ๐ฆ] โ ๐2+ ฯadd
2 โ1
๐กi2 ๐ผ๐[๐ฅ, ๐ฆ] โ ๐ + ฯadd
2
More general case
Where does this approximation come from?โข We use the fact that each pixel intensity is an unbiased estimate of the radiant flux,
weighted by exposure and gain:
E[๐ผ๐[๐ฅ, ๐ฆ]] = ๐กi โ ๐ผ โ ฮฆ[๐ฅ, ๐ฆ] โ g
If we cannot assume that our measurements are dominated by photon noise, we can approximate the variance as:
ฯ[1
๐กi
๐ผ๐[๐ฅ, ๐ฆ]]2 =1
๐กi2 ๐กi โ ๐ผ โ ฮฆ[๐ฅ, ๐ฆ] โ ๐2+ ฯadd
2 โ1
๐กi2 ๐ผ๐[๐ฅ, ๐ฆ] โ ๐ + ฯadd
2
2.8 dB 14.6 dB
standard weights optimal weights
ton
e-m
app
ed m
erge
d H
DR
ground-truth
Some comparisons
What about ISO?
โข We need to separately account for read and ADC noise, as read noise is gain-dependent.
โข We can optimize our exposure bracket by varying both shutter speed and ISO
Bonus part of Homework 2 (+ 50%!)
Real capture results
5.9 dB
16.2 dB
โgro
un
d t
ruth
โst
and
ard
exp
osu
re
bra
cket
ing
SNR
-op
tim
al
seq
uen
ce
โฆโฆ+15 more
input tone-mapped merged HDR bright region dark region
ISO
64
00
I
SO 6
40
0
IS
O 8
00
1/1
25
s
1
/16
00
s
1/6
40
0 s
ISO
10
0
IS
O 1
00
ISO
10
01
/20
0 s
1/4
00
s
1
/80
0 s
92
ReferencesBasic reading:โข Szeliski textbook, Sections 10.1, 10.2. โข Hasinoff et al., โNoise-Optimal Capture for High Dynamic Range Photography,โ CVPR 2010.
A paper on weighting different exposures based on a very detailed sensor noise model, additionally discussing combining shutter speed and ISO changes.
โข Healey and Kondepudy, โRadiometric CCD camera calibration and noise estimation,โ PAMI 1994.A detailed paper on radiometric and noise calibration based on the noise model we discussed.
โข Martinec, โNoise, Dynamic Range and Bit Depth in Digital SLRs,โ 2008, http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.htmlA very detailed discussion of noise characteristics and other performance aspects of digital sensors.
Additional reading:โข Kirk and Andersen, โNoise characterization of weighting schemes for combination of multiple exposures,โ BMVC 2006.
A great paper on the variance characteristics of most common HDR weighting schemes.โข Granados et al., โOptimal HDR Reconstruction with Linear Digital Cameras,โ CVPR 2010.
This paper extends the analysis of optimal HDR weights to consider spatially-varying noise effects.โข Hasinoff, โFundamentals of Computational Photography: Sensors and Noise,โ ICCP 2010 tutorial,
https://people.csail.mit.edu/hasinoff/hdrnoise/hasinoff-sensornoise-tutorial-iccp10.pptxA detailed tutorial on sensors and noise.
โข Hasinoff et al., โTime-constrained photography,โ ICCV 2009.โข Hasinoff and Kutulakos, โLight-efficient photography,โ PAMI 2011.
These two papers examine noise-optimal acquisition and merging schemes for focal and aperture stacks, rather than exposure stacks.โข Ratner et al., โOptimal multiplexed sensing: bounds, conditions and a graph theory link,โ Optics Express 2007.โข Ratner and Schechner, โIllumination Multiplexing within Fundamental Limits,โ CVPR 2007.
These two papers discuss the effect of different types of noise when fusing multiple images in the context of illumination multiplexing.โข Gupta et al., โPhoton-Flooded Single-Photon 3D Cameras,โ CVPR 2019.
A paper on the noise characteristics of single-photon-sensitive cameras.