mad hvei 2009
DESCRIPTION
My thesis topic presented at Human Vision and Electronic Imaging. It is Also appears in the Journal of Electronic Imaging 2010TRANSCRIPT
Most Apparent DistortionA dual strategy for full reference image quality assessment
Most Apparent DistortionA dual strategy for full reference image quality assessment
Eric Larson Cuong Vu, Damon Chandler Image Coding and Analysis Lab (ICAN)
Motivating example…Motivating example…
Motivating example…Motivating example…
Motivating example…Motivating example…
Most Apparent Distortion
MADMost Apparent Distortion
MAD
OutlineOutline
Introduction/Motivation Current Methods An example of two strategies
Methodology of MAD Detection modeling Appearance modeling Adaptation
Results Conclusion
MotivationMethods
Results
Three Types of IndicesThree Types of Indices
Mathematical efficiency Peak Signal-to-Noise Ratio (PSNR), Mean-Squared
Error (MSE)
Low level properties of Human Visual System (HVS) Visual Difference Predictor (VDP)[Daly,1992], Perceptual
Structure[Carnec, et al., 2003] ,Visual Signal-to-Noise Ratio (VSNR)[Chandler,Hemami,2007], Wavelet-based quality assessment (WQA)[Ninnassi, et al. 2008] ,
MotivationMethods
Results
Three Types of IndicesThree Types of Indices
Mathematical efficiency Peak Signal-to-Noise Ratio (PSNR), Mean-Squared
Error (MSE)
Low level properties of Human Visual System (HVS) Visual Difference Predictor (VDP)[Daly,1992], Perceptual
Structure[Carnec, et al., 2003] ,Visual Signal-to-Noise Ratio (VSNR)[Chandler,Hemami,2007], Wavelet-based quality assessment (WQA)[Ninnassi, et al. 2008] ,
Overarching principles of human vision Structural SIMilarity (SSIM)[Wang,2004], Visual Information
Fidelity (VIF)[Sheikh,2006], VSNR, Perceptual Structure
MotivationMethods
Results
Three Types of IndicesThree Types of Indices
Mathematical efficiency Peak Signal-to-Noise Ratio (PSNR), Mean-Squared
Error (MSE)
Low level properties of Human Visual System (HVS) Visual Difference Predictor (VDP)[Daly,1992], Perceptual
Structure[Carnec, et al., 2003] ,Visual Signal-to-Noise Ratio (VSNR)[Chandler,Hemami,2007], Wavelet-based quality assessment (WQA)[Ninnassi, et al. 2008] ,
Overarching principles of human vision Structural SIMilarity (SSIM)[Wang,2004], Visual Information
Fidelity (VIF)[Sheikh,2006], VSNR, Perceptual Structure
Single most relevant strategy is modeled
MotivationMethods
Results
A Task for High QualityA Task for High Quality
Can we see the distortion? How intense is the distortion?Motivatio
nMethods
Results
A Task for High QualityA Task for High Quality
Can we see the distortion? How intense is the distortion?Motivatio
nMethods
Results
Original JPEG Noise
A Task for High QualityA Task for High Quality
Can we see the distortion? How intense is the distortion?Motivatio
nMethods
Results
17.7 23.50.0
Original JPEG Noise
A Task for Low QualityA Task for Low Quality
How much does the image look like the original, given that there are so many visible distortions?
MotivationMethods
Results
A Task for Low QualityA Task for Low Quality
How much does the image look like the original, given that there are so many visible distortions?
MotivationMethods
Results
[9]
A Task for Low QualityA Task for Low Quality
How much does the image look like the original, given that there are so many visible distortions?
MotivationMethods
Results
74.64
59.0
67.09
82.74
[9]
A Task for Low QualityA Task for Low Quality
How much does the image look like the original, given that there are so many visible distortions?
MotivationMethods
Results
Motivation SummaryMotivation Summary
Approximate high quality task: Visibility Intensity
Approximate low quality task: Preserve content (appearance)
Adaptively change strategies
MotivationMethods
Results
MethodologyHigh Quality
A Strategy for High QualityA Strategy for High Quality Conversion to perceived
brightness Pixel to luminance luminance to L*
MotivationMethods
Results
A Strategy for High QualityA Strategy for High Quality Conversion to perceived
brightness Pixel to luminance luminance to L*
Contrast sensitivity
MotivationMethods
Results
)*,(' CSFLfilterI refref
)*,(' CSFLfilterI errerr
A Strategy for High QualityA Strategy for High Quality Contrast and luminance
maskingMotivationMethods
Results
)(
)()(
p
ppC
ref
refref
A Strategy for High QualityA Strategy for High Quality Spatial frequency and
luminance maskingMotivationMethods
Results
)(
)()(
p
ppC
ref
refref
)(),(),(),(min)( 22211211 ppppp refrefrefrefref
p11
p22
p12
p21
A Strategy for High QualityA Strategy for High Quality Contrast and luminance
maskingMotivationMethods
Results
)(
)()(
p
ppC
ref
refref
otherwise ,0
9.0)( if , )(
)()( ref p
p
ppC ref
err
diff
A Strategy for High QualityA Strategy for High Quality Contrast and luminance
maskingMotivationMethods
Results
)(
)()(
p
ppC
ref
refref
otherwise ,0
9.0)( if , )(
)()( ref p
p
ppC ref
err
diff
otherwise ,0
)()( if 1, )( 4
3 pCpCpVisib referr
A Strategy for High QualityA Strategy for High Quality Contrast and luminance
maskingMotivationMethods
Results
)(
)()(
p
ppC
ref
refref
otherwise ,0
9.0)( if , )(
)()( ref p
p
ppC ref
err
diff
otherwise ,0
)()( if 1, )( 4
3 pCpCpVisib referr
pNji
err jiIpLMSE,
2
2),('
16
1)(
A Strategy for High QualityA Strategy for High Quality Contrast and luminance
maskingMotivationMethods
Results
)(
)()(
p
ppC
ref
refref
NPp
high pVisibpLMSEN
PD 2)()(1
otherwise ,0
9.0)( if , )(
)()( ref p
p
ppC ref
err
diff
otherwise ,0
)()( if 1, )( 4
3 pCpCpVisib referr
pNji
err jiIpLMSE,
2
2),('
16
1)(
A Strategy for High QualityA Strategy for High Quality
MotivationMethods
Results
Visibility Map
Visibility Map
LMSEMap
LMSEMap
CombineMaps
CombineMaps
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
A Strategy for High QualityA Strategy for High Quality
MotivationMethods
Results
Visibility Map
Visibility Map
LMSEMap
LMSEMap
CombineMaps
CombineMaps
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
A Strategy for High QualityA Strategy for High Quality
MotivationMethods
Results
Visibility Map
Visibility Map
LMSEMap
LMSEMap
CombineMaps
CombineMaps
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
A Strategy for High QualityA Strategy for High Quality
MotivationMethods
Results
Visibility Map
Visibility Map
LMSEMap
LMSEMap
CombineMaps
CombineMaps
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
A Strategy for High QualityA Strategy for High Quality
MotivationMethods
Results
Visibility Map
Visibility Map
LMSEMap
LMSEMap
CombineMaps
CombineMaps
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
PDhigh
MethodologyLow Quality
A Strategy for Low QualityA Strategy for Low Quality Defining appearance:
Biological motivation: the log-Gabor filter bank[Field 1987, Kovesi]Motivatio
nMethods
Results
A Strategy for Low QualityA Strategy for Low Quality Defining appearance:
Biological motivation: the log-Gabor filter bank[Field 1987, Kovesi]
Five Scales Four Orientations
MotivationMethods
Results
2
2
2
2
2
)(exp
)/log(2
)/log(exp),(
os
o
f
fffG
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gso
Statistics have been used to model Animal Camouflage [Larson, Chandler 2007]
Texture Appearance[Kingdom, et al. 2003]
Variance, Skewness, and Kurtosis
MotivationMethods
Results
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gso
where ws = [0.5, 0.75, 1, 5, 6]
MotivationMethods
Results
pp OoSs dst
sorefso
dstso
refso
dstso
refso
swp,
||
||2
||
)(
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gso
where ws = [0.5, 0.75, 1, 5, 6]
MotivationMethods
Results
NPp
low pN
PD 2)(1
pp OoSs
dstso
refso
dstso
refso
dstso
refsoswp
,
|]|||2|[|)(
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gmag,so
where ws = [1, 2, 6, 10, 12]
MotivationMethods
Results
pp OoSs
dstso
dstso
dstso
refso
refso
refsoswp
,
)]()[()( 22
NPp
low pN
Q 21)(
Gabor FilteringGabor
Filtering
PatchStatistics
PatchStatistics
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gmag,so
where ws = [1, 2, 6, 10, 12]
MotivationMethods
Results
pp OoSs
dstso
dstso
dstso
refso
refso
refsoswp
,
)]()[()( 22
NPp
low pN
Q 21)(
Gabor FilteringGabor
Filtering
PatchStatistics
PatchStatistics
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gmag,so
where ws = [1, 2, 6, 10, 12]
MotivationMethods
Results
pp OoSs
dstso
dstso
dstso
refso
refso
refsoswp
,
)]()[()( 22
NPp
low pN
Q 21)(
Gabor FilteringGabor
Filtering
PatchStatistics
PatchStatistics
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gmag,so
where ws = [1, 2, 6, 10, 12]
MotivationMethods
Results
pp OoSs
dstso
dstso
dstso
refso
refso
refsoswp
,
)]()[()( 22
NPp
low pN
Q 21)(
Gabor FilteringGabor
Filtering
PatchStatistics
PatchStatistics
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gmag,so
where ws = [1, 2, 6, 10, 12]
MotivationMethods
Results
pp OoSs
dstso
dstso
dstso
refso
refso
refsoswp
,
)]()[()( 22
NPp
low pN
Q 21)(
Gabor FilteringGabor
Filtering
PatchStatistics
PatchStatistics
Collapse with two-
norm
Collapse with two-
norm
Read Images
Read Images
PDlow
AdaptationAdaptation
We can adaptively model their interaction based upon PDhigh
The final index is a weighted geometric mean
MotivationMethods
Results
2)(1
1
1
highPD
)1()()( lowhigh PDPDMAD
Results
ResultsResults
LIVE[9] Quality Database: 779 Distorted Images, 29 Original 29 Observers JPEG, JPEG2000, Blurring, AWGN, and
simulated packet loss
CSIQ (Categorical Subjective Image Quality) Database: 10 original images, 300 distorted
versions 10 observers Blur, contrast, AWGN, JPEG, JPEG2000,
APGN (1/f noise)
MotivationMethods
Results
ResultsResults
LIVE Performance, All images
MotivationMethods
Results
ALL PSNR SSIM VSNR VIF MAD
CC 0.8707 0.9378 0.9233 0.9595 0.9695
SROCC 0.8763 0.9473 0.9278 0.9633 0.9703
Rout 68.16% 59.18% 58.79% 54.56% 42.40%
ResultsResults
LIVE Performance, All images
MotivationMethods
Results
ALL PSNR SSIM VSNR VIF MAD
CC 0.8707 0.9378 0.9233 0.9595 0.9695
SROCC 0.8763 0.9473 0.9278 0.9633 0.9703
Rout 68.16% 59.18% 58.79% 54.56% 42.40%
Statistical SignificanceStatistical Significance
LIVE Database, 99% confidence
1 = better, 0 = same, -1 = worse
MotivationMethods
Results
PSNR SSIM VSNR VIF MADPSNR 0 - - - -SSIM 1 0 - - -VSNR 1 -1 0 - -
VIF 1 1 1 0 -
MAD 1 1 1 1 0
ResultsResults
CSIQ Overall Performance
MotivationMethods
Results
ALL PSNR SSIM VSNR VIF MAD
CC 0.8455 0.8893 0.8472 0.9079 0.9487
SROCC
0.8428 0.9019 0.8577 0.9063 0.9469
Rout 35.6% 30.5% 28.2% 33.9% 23.5%
ResultsResults
CSIQ Overall Performance
MotivationMethods
Results
Logistic MAD
CC = 0.9487
DM
OS
Statistical SignificanceStatistical Significance
CSIQ Database, 99% Confidence
1 = better, 0 = same, -1 = worse
MotivationMethods
Results
PSNR SSIM VSNR VIF MADPSNR 0 - - - -SSIM 1 0 - - -VSNR 0 -1 0 - -
VIF 1 0 1 0 -
MAD 1 1 1 1 0
ConclusionConclusion
Quality prediction algorithms can enhance performance by adaptively changing strategy
MAD performs significantly better than any other existing index on two databases
MAD shows promise in generalizing to a range of distortions
Multiple strategies
MotivationMethods
Results
Thank YouThank You
Questions?
MotivationMethods
Results
Thank YouThank You
MotivationMethods
Results
ReferencesReferences1. B. Girod, What’s worng with the mean squared error?, pp207-240. MIT Press, 2nd ed., 19932. T. Chen, Invited Lecture, Carnegie Mellon University, 2008 IEEE Southwest Symposium on Image Analysis and
Interpretation. 3. S. Daly, “Visible differences predictor: an algorithm for the assessment of image fidelity,” in Proc. SPIE Vol. 1666, p. 2-
15, Human Vision, Visual Processing, and Digital Display III, Bernice E. Rogowitz; Ed. (B. E. Rogowitz, ed.), vol. 1666 of Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference, pp. 2–15, Aug. 1992.
4. D. Chandler and S. Hemami, “Vsnr: A wavelet based visual signal to noise ratio for natural images,” IEEE Transactions on Image Processing, vol. 16, pp. 2284 -2298, Sept. 2007.
5. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, pp. 600–612, April 2004.
6. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, vol. 15, pp. 430–444, Feb. 2006.
7. E. Peli, L. E. Arend, G. M. Young, and R. B. Goldstein, “Contrast sensitivity to patch stimuli: Effects of spatial bandwidth and temporal presentation,” Spatial Vision, vol. 7, pp. 1–14, 1993.
8. G. E. Legge and J. M. Foley, “Contrast masking in human vision,” J. of Opt. Soc. Am., vol. 70, pp. 1458–1470, 1980.9. Z. W. H. R. Sheikh, A. C. Bovik, and L. K. Cormack. Image and Video Quality Assessment Research at LIVE [Online].
Available: http://live.ece.utexas.edu/research/quality/.10.B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: A strategy employed by v1?,” Vision
Research, vol. 37, pp. 3311–3325, Dec. 1997.11.P. D. Kovesi. MATLAB and Octave Functions for Computer Vision and Image Processing. School of Computer Science &
Software Engineering, The University of Western Australia. Available from: http://www.csse.uwa.edu.au/~pk/research/matlabfns/
12.Kingdom, F. A. A., Hayes, A. & Field, D. J. (2001) Sensitivity to contrast histogram differences in synthetic wavelet-textures. Vision Research, 41, 585-598.
13.N. P. S. D. I. A. [Online].14.“Vqeg, final report from the video quality experts group on the validation of objective models of video quality
assessment, phase ii,” August 2003 [Online]. Available: http://www.vqeg.org.15.H. Sheikh, M. Sabir, and A. Bovik, “A statistical evaluation of recent full reference image quality assessment
algorithms,” IEEE Transactions on Image Processing, vol. 15, pp. 1349–1364, Nov. 2006.16.Correlation, Wikipedia, http://en.wikipedia.org/wiki/Correlation17. Regression Analysis, Wikipedia, http://en.wikipedia.org/wiki/Regression_analysis
ResultsResults
CSIQ (Categorical Subjective Image Quality) Database
Preliminary Numbers: 4 observers 10 original images 300 distorted versions Six distortion types:
Blur, contrast, AWGN, JPEG, JPEG2000, APGN (1/f noise)
MotivationMethods
Results
Camp Two: StructureCamp Two: Structure
SSIM captures Gaussian windowed spatial statistics
Collapse quality by taking mean of map
MotivationMethods
Results
))((
))((),(
222
122
21 22
CC
CCyxSSIM
yxyx
xyyx
Camp Two: InformationCamp Two: Information VIF models mutual information by
Analyzing in wavelet domain[6]
Applying reference and distorted images to HVS model
Where CU is the principle image covariance of sub-band i,
σv is the distortion noise, σn is the Visual noise variance,
s and g are sub-band scaling constants
MotivationMethods
Results IIC 2
21
2222
1n
N
jnUji sREFI loglog)(
N
jnvnvUjji sgDSTI
1
222
222222
1IIC )(log)(log)(
subbandsi
isubbandsi
i )I(REF)I(DSTVIF
A Task for High QualityA Task for High Quality
Can we see the distortion? How intense is the distortion?Motivatio
nMethods
Results
[9]
A Task for High QualityA Task for High Quality
Can we see the distortion? How intense is the distortion?Motivatio
nMethods
Results
20.52
8.06
13.03
20.30
[9]
A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon
statistics of Gmag,so
where ws = [1, 2, 6, 10, 12]
MotivationMethods
Results
pp OoSs
dstso
dstso
dstso
refso
refso
refsoswp
,
)]||()||[()( 22
Performance MeasuresPerformance Measures
LIVE[9] Quality Database: 779 Distorted Images, 29 Original 29 Observers Over 20,000 ratings of image
fidelity, in terms of Differential Mean opinion Score (DMOS)
Five categories of distortion: JPEG, JPEG2000, Blurring, AWGN, and
simulated packet loss
MotivationMethods
Results
ResultsResults
LIVE Performance figures
MotivationMethods
Results
ResultsResults
LIVE Performance figures
MotivationMethods
Results
Statistical SignificanceStatistical Significance
In terms of Regression…
MotivationMethods
Results
[16]
Statistical SignificanceStatistical Significance
Gaussian Residuals
MotivationMethods
Results
[16]
Statistical SignificanceStatistical Significance
LIVE Database, 99% confidence
1 = better, 0 = same, -1 = worse
MotivationMethods
Results
ALL PSNR SSIM VSNR VIF MADPSNR 0 -1 -1 -1 -1SSIM 1 0 1 -1 -1VSNR 1 -1 0 -1 -1
VIF 1 1 1 0 -1MAD 1 1 1 1 0
Gaussian 1 0 1 0 1Conf. 0.007 0.257 0.001 0.081 0.001
JB Stat 11.768 2.583 20.011 4.843 246.610Skew 0.292 -0.139 0.091 0.170 -0.518Kurt 3.143 2.957 3.764 2.818 5.554
Statistical SignificanceStatistical Significance
LIVE Database, Gaussianity
MotivationMethods
Results
Statistical SignificanceStatistical Significance
LIVE Database, Gaussianity
MotivationMethods
Results
Statistical SignificanceStatistical Significance
LIVE Database, Gaussianity
MotivationMethods
Results
J.B. Statistic = 1.5J.B. Statistic = 1.5Is Gaussian with greater than Is Gaussian with greater than 95% confidence95% confidence
ResultsResults
CSIQ (Categorical Subjective Image Quality) Database
Table top randomization approach Reference image always available Distorted images viewable at one
time Placement denotes linear quality Electronic table (four monitor
array)
MotivationMethods
Results
ResultsResults
CSIQ (Categorical Subjective Image Quality) Database
Table top randomization approach Reference image always available Distorted images viewable at one
time Placement denotes quality Electronic table (four monitor
array)
MotivationMethods
Results
Statistical SignificanceStatistical Significance
CSIQ Database, 99% Confidence
1 = better, 0 = same, -1 = worse
MotivationMethods
Results
ALL PSNR SSIM VSNR VIF MADPSNR 0 -1 0 -1 -1SSIM 1 0 1 0 -1VSNR 0 -1 0 -1 -1
VIF 1 0 1 0 -1MAD 1 1 1 1 0
Gaussian 0 1 1 0 0Conf. 0.500> 0.003 0.001 0.500> 0.425
JB Stat 0.414 17.619 24.255 0.652 1.560Skew -0.053 -0.303 -0.569 0.084 0.108Kurt 3.149 4.026 3.812 3.156 3.281
ResultsResults
CSIQ Overall Performance, no contrast
MotivationMethods
Results
PSNR SSIM VSNR VIF MAD
CC 0.9178 0.9313 0.9499 0.9263 0.9637
SROCC 0.9185 0.9364 0.9478 0.9294 0.9594
RMSE 166.55 152.80 131.10 158.11 111.94
Rout 0.344 0.316 0.240 0.312 0.236
RSOD 447.9 383.3 256.6 387.0 237.1