quantization - stanford university
TRANSCRIPT
Bernd Girod: EE398A Image and Video Compression Quantization no. 1
Quantization
Q q
Q -1
Sometimes, this
convention is used:
M represen-
tative levels
Q
Input-output characteristic of a scalar quantizer
M-1 decision thresholds
input signal x
Output
t q+1 t q+2 t q
x x̂
x̂2
ˆqx
1ˆ
qx
ˆqx
x
q x̂
Bernd Girod: EE398A Image and Video Compression Quantization no. 2
Example of a quantized waveform
Original and Quantized Signal
Quantization Error
Bernd Girod: EE398A Image and Video Compression Quantization no. 3
Lloyd-Max scalar quantizer
Problem : For a signal x with given PDF find a quantizer with
M representative levels such that
( )Xf x
Solution : Lloyd-Max quantizer
[Lloyd,1957] [Max,1960]
M-1 decision thresholds exactly
half-way between representative
levels.
M representative levels in the
centroid of the PDF between two
successive decision thresholds.
Necessary (but not sufficient)
conditions
1
1
1
1ˆ ˆ 1, 2, , -1
2
( )
ˆ 0,1, , -1
( )
q
q
q
q
q q q
t
X
t
q t
X
t
t x x q M
x f x dx
x q M
f x dx
2
ˆ min.d MSE E X X
Bernd Girod: EE398A Image and Video Compression Quantization no. 4
Iterative Lloyd-Max quantizer design
1. Guess initial set of representative levels
2. Calculate decision thresholds
3. Calculate new representative levels
4. Repeat 2. and 3. until no further distortion reduction
ˆ 0,1,2, , -1qx q M
1
1
( )
ˆ 0,1, , -1
( )
q
q
q
q
t
X
t
q t
X
t
x f x dx
x q M
f x dx
1
1ˆ ˆ 1,2, , -1
2q q qt x x q M
Bernd Girod: EE398A Image and Video Compression Quantization no. 5
Example of use of the Lloyd algorithm (I)
X zero-mean, unit-variance Gaussian r.v.
Design scalar quantizer with 4 quantization indices with
minimum expected distortion D*
Optimum quantizer, obtained with the Lloyd algorithm Decision thresholds -0.98, 0, 0.98
Representative levels –1.51, -0.45, 0.45, 1.51
D*=0.12=9.30 dB
Boundary
Reconstruction
Bernd Girod: EE398A Image and Video Compression Quantization no. 6
Example of use of the Lloyd algorithm (II)
Convergence
Initial quantizer A:
decision thresholds –3, 0 3
Initial quantizer B:
decision thresholds –½, 0, ½
After 6 iterations, in both cases, (D-D*)/D*<1%
5 10 15-6
-4
-2
0
2
4
6
Qu
an
tizati
on
Fu
ncti
on
Iteration Number
0 5 10 150
0.2
0.4
D
0 5 10 150
5
10
SN
RO
UT
[dB
]
Iteration Number
SNROUT
final
= 9.2978
5 10 15-6
-4
-2
0
2
4
6
Qu
an
tizati
on
Fu
ncti
on
Iteration Number
0 5 10 150.1
0.15
0.2
D
0 5 10 156
8
10
SN
RO
UT
[dB
]
Iteration Number
SNROUT
final
= 9.298
Bernd Girod: EE398A Image and Video Compression Quantization no. 7
Example of use of the Lloyd algorithm (III)
X zero-mean, unit-variance Laplacian r.v.
Design scalar quantizer with 4 quantization indices with
minimum expected distortion D*
Optimum quantizer, obtained with the Lloyd algorithm Decision thresholds -1.13, 0, 1.13
Representative levels -1.83, -0.42, 0.42, 1.83
D*=0.18=7.54 dB
Threshold
Representative
Bernd Girod: EE398A Image and Video Compression Quantization no. 8
Example of use of the Lloyd algorithm (IV)
Convergence
Initial quantizer A,
decision thresholds –3, 0 3
Initial quantizer B,
decision thresholds –½, 0, ½
After 6 iterations, in both cases, (D-D*)/D*<1%
0 5 10 150
0.2
0.4
D
0 5 10 154
6
8
SN
R [
dB
]
Iteration Number
SNRfinal
= 7.5415
2 4 6 8 10 12 14 16-10
-5
0
5
10
Qu
an
tizati
on
Fu
ncti
on
Iteration Number2 4 6 8 10 12 14 16
-10
-5
0
5
10
Qu
an
tizati
on
Fu
ncti
on
Iteration Number
0 5 10 150
0.2
0.4
D
0 5 10 154
6
8
SN
R [
dB
]
Iteration Number
SNRfinal
= 7.5415
Bernd Girod: EE398A Image and Video Compression Quantization no. 9
Lloyd algorithm with training data
1. Guess initial set of representative levels
2. Assign each sample xi in training set T to closest representative
3. Calculate new representative levels
4. Repeat 2. and 3. until no further distortion reduction
ˆ ; 0,1,2, , -1qx q M
x
1ˆ 0,1, , -1
q
q
Bq
x x q MB
ˆqx
B
q x T :Q x q q 0,1,2,, M -1
Bernd Girod: EE398A Image and Video Compression Quantization no. 10
Lloyd-Max quantizer properties
Zero-mean quantization error
Quantization error and reconstruction decorrelated
Variance subtraction property
ˆ 0E X X
ˆ ˆ 0E X X X
2
2 2
ˆˆ
XXE X X
Bernd Girod: EE398A Image and Video Compression Quantization no. 11
High rate approximation
Approximate solution of the "Max quantization problem,"
assuming high rate and smooth PDF [Panter, Dite, 1951]
Approximation for the quantization error variance:
3
1( )
( )X
x x constf x
Distance between two
successive quantizer
representative levels
Probability density
function of x
3
23
2
1ˆ ( )12
X
x
d E X X f x dxM
Number of representative levels
Bernd Girod: EE398A Image and Video Compression Quantization no. 12
High rate approximation (cont.)
High-rate distortion-rate function for scalar Lloyd-Max quantizer
Some example values for
2 2 2
3
2 2 3
2
1with ( )
12
R
X
X X
x
d R
f x dx
2
92
uniform 1
Laplacian 4.5
3Gaussian 2.721
2
Bernd Girod: EE398A Image and Video Compression Quantization no. 13
High rate approximation (cont.)
Partial distortion theorem: each interval makes an (approximately)
equal contribution to overall mean-squared error
2
1 1
2
1 1
ˆPr
ˆPr for all ,
i i i i
j j j j
t X t E X X t X t
t X t E X X t X t i j
[Panter, Dite, 1951], [Fejes Toth, 1959], [Gersho, 1979]
Bernd Girod: EE398A Image and Video Compression Quantization no. 14
Entropy-constrained scalar quantizer
Lloyd-Max quantizer optimum for fixed-rate encoding, how can we
do better for variable-length encoding of quantizer index?
Problem : For a signal x with given pdf find a quantizer with
rate
such that
Solution: Lagrangian cost function
2
ˆ min.d MSE E X X
( )Xf x
1
2
0
ˆ logM
q q
q
R H X p p
2
ˆ ˆ min.J d R E X X H X
Bernd Girod: EE398A Image and Video Compression Quantization no. 15
Iterative entropy-constrained scalar quantizer design
1. Guess initial set of representative levels
and corresponding probabilities pq
2. Calculate M-1 decision thresholds
3. Calculate M new representative levels and probabilities pq
4. Repeat 2. and 3. until no further reduction in Lagrangian cost
ˆ ; 0,1,2, , -1qx q M
1
1
( )
ˆ 0,1, , -1
( )
q
q
q
q
t
X
t
q t
X
t
xf x dx
x q M
f x dx
-1 2 1 2
-1
ˆ ˆ log log= 1,2, , -1
2 ˆ ˆ2
q q q q
q
q q
x x p pt q M
x x
Bernd Girod: EE398A Image and Video Compression Quantization no. 16
Lloyd algorithm for entropy-constrained
quantizer design based on training set
1. Guess initial set of representative levels
and corresponding probabilities pq
2. Assign each sample xi in training set T to representative minimizing Lagrangian cost
3. Calculate new representative levels and probabilities pq
4. Repeat 2. and 3. until no further reduction in overall Lagrangian cost
: 0,1,2, , -1qB x Q x q q M T
ˆ ; 0,1,2, , -1qx q M
x
1ˆ 0,1, , -1
q
q
Bq
x x q MB
ˆqx
2
2ˆ log
ix i q qJ q x x p
2
2logi i
i i
x i i q xx x
J J x Q x p
Bernd Girod: EE398A Image and Video Compression Quantization no. 17
Example of the EC Lloyd algorithm (I)
X zero-mean, unit-variance Gaussian r.v.
Design entropy-constrained scalar quantizer with rate R≈2 bits, and
minimum distortion D*
Optimum quantizer, obtained with the entropy-constrained Lloyd
algorithm 11 intervals (in [-6,6]), almost uniform
D*=0.09=10.53 dB, R=2.0035 bits (compare to fixed-length example)
Threshold
Representative level
Bernd Girod: EE398A Image and Video Compression Quantization no. 18
Example of the EC Lloyd algorithm (II)
Initial quantizer B, only 4 intervals
(<11) in [-6,6], with the same length
Same Lagrangian multiplier λ used in all experiments
Initial quantizer A, 15 intervals
(>11) in [-6,6], with the same length
5 10 15 20-6
-4
-2
0
2
4
6
Qu
an
tizati
on
Fu
ncti
on
Iteration Number
0 5 10 15 20
6
8
10
12
SN
R [
dB
]
SNRfinal
= 8.9452
0 5 10 15 201
1.5
2
2.5
R [
bit
/sym
bo
l]
Rfinal
= 1.7723
Iteration Number
0 5 10 15 20 25 30 35 40
6
8
10
12
SN
R [
dB
]
SNRfinal
= 10.5363
0 5 10 15 20 25 30 35 401
1.5
2
2.5
R [
bit
/sym
bo
l]
Rfinal
= 2.0044
Iteration Number
5 10 15 20 25 30 35 40-6
-4
-2
0
2
4
6
Qu
an
tizati
on
Fu
ncti
on
Iteration Number
Bernd Girod: EE398A Image and Video Compression Quantization no. 19
Example of the EC Lloyd algorithm (III)
X zero-mean, unit-variance Laplacian r.v.
Design entropy-constrained scalar quantizer with rate R≈2 bits
and minimum distortion D*
Optimum quantizer, obtained with the entropy-constrained Lloyd
algorithm 21 intervals (in [-10,10]), almost uniform
D*= 0.07=11.38 dB, R= 2.0023 bits (compare to fixed-length example)
Decision threshold
Representative level
Bernd Girod: EE398A Image and Video Compression Quantization no. 20
Example of the EC Lloyd algorithm (IV)
Initial quantizer B, only 4 intervals
(<21) in [-10,10], with the same length
Same Lagrangian multiplier λ used in all experiments
Initial quantizer A, 25 intervals
(>21 & odd) in [-10,10], with the
same length
Convergence in cost faster than convergence of decision thresholds
5 10 15 20-10
-5
0
5
10
Qu
an
tizati
on
Fu
ncti
on
Iteration Number
0 5 10 15 20
5
10
15
SN
R [
dB
]SNR
final = 7.4111
0 5 10 15 201
2
3
R [
bit
/sym
bo
l]
Rfinal
= 1.6186
Iteration Number
0 5 10 15 20 25 30 35 40
5
10
15
SN
R [
dB
]
SNRfinal
= 11.407
0 5 10 15 20 25 30 35 401
2
3
R [
bit
/sym
bo
l]
Rfinal
= 2.0063
Iteration Number
5 10 15 20 25 30 35 40-10
-5
0
5
10
Qu
an
tizati
on
Fu
ncti
on
Iteration Number
Bernd Girod: EE398A Image and Video Compression Quantization no. 21
High-rate results for EC scalar quantizers
For MSE distortion and high rates, uniform quantizers (followed by
entropy coding) are optimum [Gish, Pierce, 1968]
Distortion and entropy for smooth PDF and fine quantizer interval
Distortion-rate function
is factor or 1.53 dB from Shannon Lower Bound
222
2
12d d
2
ˆ logH X h X
2 212 2
12
h X Rd R
6
e
2 212 2
2
h X RD Re
Bernd Girod: EE398A Image and Video Compression Quantization no. 22
Comparison of high-rate performance of
scalar quantizers
High-rate distortion-rate function
Scaling factor
2 2 22 R
Xd R
2
2
Shannon LowBd Lloyd-Max Entropy-coded
6Uniform 0.703 1 1
e
9Laplacian 0.865 4.5 1.232
2 6
3Gaussian 1 2.721 1.423
2 6
e e
e
Bernd Girod: EE398A Image and Video Compression Quantization no. 23
Deadzone uniform quantizer
Quantizer
input
Quantizer
output ̂x
x
Bernd Girod: EE398A Image and Video Compression Quantization no. 24
Embedded quantizers
Motivation: “scalability” – decoding of compressed
bitstreams at different rates (with different qualities)
Nested quantization intervals
In general, only one quantizer can be optimum
(exception: uniform quantizers)
Q0
Q1
Q2
x
Bernd Girod: EE398A Image and Video Compression Quantization no. 25
Example: Lloyd-Max quantizers for Gaussian PDF
2-bit and 3-bit optimal
quantizers not embeddable
Performance loss for
embedded quantizers
0 1
0
0
0
1
1
0
1
1
0
0
0
0
0
1
0
1
0
0
1
1
1
0
0
1
0
1
1
1
0
1
1
1
-.98 .98
-1.75 -1.05 -.50 .50 1.05 1.75
Bernd Girod: EE398A Image and Video Compression Quantization no. 26
Information theoretic analysis
“Successive refinement” – Embedded coding at multiple
rates w/o loss relative to R-D function
“Successive refinement” with distortions and
can be achieved iff there exists a conditional distribution
Markov chain condition
1D 2 1D D
1 1
2 2
ˆ( , )
ˆ( , )
E d X X D
E d X X D
1 1
2 2
ˆ( ; ) ( )
ˆ( ; ) ( )
I X X R D
I X X R D
1 2 2 1 2ˆ ˆ ˆ1 2 2 1 2ˆ ˆ,
ˆ ˆ ˆ ˆ ˆ( , , ) ( , ) ( , )X X X X X X X
f x x x f x x f x x
2 1ˆ ˆX X X
[Equitz,Cover, 1991]
Bernd Girod: EE398A Image and Video Compression Quantization no. 27
Embedded deadzone uniform quantizers
Q0
Q1
Q2
x
8
4
2
x=0
4
2
Supported in JPEG-2000 with general for
quantization of wavelet coefficients.
Bernd Girod: EE398A Image and Video Compression Quantization no. 29
LBG algorithm
Lloyd algorithm generalized for VQ [Linde, Buzo, Gray, 1980]
Assumption: fixed code word length
Code book unstructured: full search
Best representative vectors
for given partitioning of training set
Best partitioning of training set
for given representative
vectors
Bernd Girod: EE398A Image and Video Compression Quantization no. 30
Design of entropy-coded vector quantizers
Extended LBG algorithm for entropy-coded VQ [Chou, Lookabaugh, Gray, 1989]
Lagrangian cost function: solve unconstrained problem rather than
constrained problem
Unstructured code book: full search for
2
ˆ ˆ min.J d R E X X H X
The most general coder structure:
Any source coder can be interpreted as VQ with VLC!
2
2ˆ log
ix i q qJ q x x p
Bernd Girod: EE398A Image and Video Compression Quantization no. 31
Lattice vector quantization
•
•
• • •• • • •
• • • • • • • •
• • • • • ••
• • • • • • •• •
• • • • • • • •
•• • • • • •
• • • • • • •
•
•
•
Amplitude 1
Am
plit
ude
2
cell
representative vector
Bernd Girod: EE398A Image and Video Compression Quantization no. 32
8D VQ of a Gauss-Markov source
r 0.95
SN
R [
dB
]
18
12
6
0
0 0.5 1.0
Rate [bit/sample]
scalar
E8-lattice
LBG variable CWL
LBG fixed CWL
Bernd Girod: EE398A Image and Video Compression Quantization no. 33
8D VQ of memoryless Laplacian source S
NR
[dB
]
Rate [bit/sample]
scalar
E8-lattice LBG variable CWL
LBG fixed CWL
9
6
3
0.5 1.0
0
0
Bernd Girod: EE398A Image and Video Compression Quantization no. 34
Reading
Taubman, Marcellin, Sections 3.2, 3.4
J. Max, “Quantizing for Minimum Distortion,” IEEE Trans.
Information Theory, vol. 6, no. 1, pp. 7-12, March 1960.
S. P. Lloyd, “Least Squares Quantization in PCM,” IEEE
Trans. Information Theory, vol. 28, no. 2, pp. 129-137,
March 1982.
P. A. Chou, T. Lookabaugh, R. M. Gray, “Entropy-
constrained vector quantization,” IEEE Trans. Signal
Processing, vol. 37, no. 1, pp. 31-42, January 1989.