quantization design - drexel engineering · • n-level lloyd-max quantizer : minimize the average...

Quantization Design

Jie Renjr843@drexel.edu

Adaptive Signal Processing and Information Theory GroupDepartment of Electrical and Computer Engineering

Drexel University, Philadelphia, PA 19104

July 28th and 30th, 2014

Jie Ren (Drexel ASPITRG) QD July 28th and 30th, 2014 1 / 35

References

S. P. Lloyd, “Least squares quantization in pcm,” IEEE Transactions on Acoustics, Speechand Signal Processing, vol. 28, no. 2, pp. 129–137, March 1982.

J. Max, “Quantizing for minimum distortion,” IEEE Transactions on Acoustics, Speechand Signal Processing, vol. 6, no. 1, pp. 7–12, March 1960.

N. Farvardin and J. W. Modestino, “Optimum quantizer performance for a class ofnon-gaussian memoryless sources,” IEEE Trans. Inform. Theory, vol. 30, no. 3, pp.485–497, May 1984.

D. K. Sharma, “Design of absolutely optimal quantizers for a wide class of distortionmeasures,” IEEE Trans. Inform. Theory, vol. 24, no. 6, pp. 693–702, November 1978.

P. A. Chou, T. Lookabaugh, and R. M. Gray, “Entropy-constrained vector quantization,”IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 37, no. 1, pp. 31–42,Juanuary 1989.

L. R. Varshney, “Unreliable and resource-constrained decoding,” Ph.D. dissertation,Massachusetts Institute of Technology, 2010.

Problem Setup

Outline

1 Problem Setup

2 Lloyd-max Quantizer DesignLocal Optimality ConditionsBy Alternating OptimizationBy Dynamic Programming

3 Variable Rate Optimum Quantizer DesignProblem SetupAnalysisGeneralized Lloyd-Max Algorithm

Problem Setup

Quantization

Map a Large Set Θ of Input Values to a Smaller set A

• Discrete Quantization : Countable Θ, Countable A

• Continuous Quantization : Uncountable Θ, Countable A

• Scalar QuantizationQ : Θ→ A (1)

• Vector Quantization

Q : Θ×Θ× · · · ×Θ→ A (2)

Problem Setup

N-level Continuous scalar Quantizer

• Source Θ with normalized support [0, 1] and pdf p(θ)

• N-level quantizer QN(·)• N Reconstruction levels A = {a1, . . . , aN}• Thresholds b1, . . . , bN−1 partitioning [0, 1] with b0 = 0 and bN = 1

Problem Setup

N-level Continuous scalar Quantizer

• Huffman encode/decode QN(Θ) with rate

R = −N∑

Pn log2 Pn (3)

Pn ,∫ bn

bn−1

p(θ)dθ (4)

• Distortion Metricd : [0, 1]×A → R+ (5)

withd(a, a) = 0 (6)

• Average Distortion

DN(b, a) =N∑

∫ bn

bn−1

d(θ, an)p(θ)dθ (7)

Problem Setup

Fixed Rate Quantizer Design and Variable Rate Quantizer Design

• N-level Lloyd-Max quantizer : minimize the average distortion for afixed number of levels N.[1][2]

• N-level Optimum Quantizer : minimize the average distortion for afixed number of levels N subject to an entropy constraint ofrate.[3]

Lloyd-max Quantizer Design

Outline

1 Problem Setup

Lloyd-max Quantizer Design Local Optimality Conditions

Outline

1 Problem Setup

Local Optimality Conditions

• Nearest Neighbor Condition : For fixed reconstruction levels {ak},Given any θ ∈ [ak , ak+1],

Q(θ) = d(θ, ak) ≤ d(θ, ak+1) ? ak : ak+1 (8)

• Centroid Condition : For fixed regions {Rk} with thresholds {bk},

ak = arg mina

∫ bk

bk−1

d(θ, a)p(θ)dθ (9)

• Zero Probability Boundary Condition, for all bk , k = 1, . . . ,K − 1

P(θ = bk) = 0 (10)

Necessary and Sufficient Conditions

TheoremThe nearest neighbor condition, the centroid condition, and the zeroprobability of boundary condition are necessary for a Lloyd-Max quantizerto be optimal.

TheoremIf the following conditions hold for a source Θ and distortion functiond(θ, a) :

1 p(θ) is positive and continuous in (0, 1)

2∫ 1

0 d(θ, a)p(θ)dθ is finite for all a

3 d(θ, a) is zero only for θ = a, is continuous in θ for all a, and iscontinuous and convex in a

then the nearest neighbor condition, centroid condition, and zeroprobability of boundary conditions are sufficient to guarantee localoptimality of a quantizer.

Lloyd-max Quantizer Design By Alternating Optimization

Outline

1 Problem Setup

Lloyd-Max algorithm

Alternating Minimization

• For fixed {ak}, minimize D w.r.t. {bk}• For fixed {bk}, minimize D w.r.t. {ak}• D monotone decreasing

Lloyd-Max algorithm

Algorithm 1: Lloyd-Max

Result: Minimize the average distortion for a N-level Lloyd-Max quantizerstep 1) Choose an arbitrary set of initial reconstruction levels {an}step 2) For each n = 1, . . . ,N set Rn = {θ|d(θ, an) ≤ d(θ, aj), j 6= n}step 3) For each n = 1, . . . ,N set an = arg mina E [d(Θ, a)|Θ ∈ Rn]step 4) Repeat step 2 and 3 until change in average distortion is negligiblestep 5) Revise {an} and {Rn} to satisfy the zero probability of boundarycondition

Analysis

• Local optimum guaranteed by Theorem. 2

• Monotonic Convergence in N

D∗N(b∗, a∗) =N∑

∫ b∗n

b∗n−1

d(θ, a∗n)p(θ)dθ (11)

D∗ = limN→∞

D∗N (12)

Monotonic Convergence in N

The Lloyd-Max N-level quantizer is the solution of the followingproblem:

D∗N = minN∑

∫ bn

bn−1

d(θ, an)p(θ)dθ

s.t. b0 = 0

bN = 1

bn−1 ≤ bn, n = 1, . . . ,N

an ≤ bn, n = 1, . . . ,N

Monotonic Convergence in N

• Degenerate the N-level Lloyd-Max quantizer to N − 1

• By adding the additional constraint bN−1 = 1 to (13) and forcingaN = 1, hence

D∗N−1 ≥ D∗N (14)

• D∗N bounded below by 0

• D∗N converges

Lloyd-max Quantizer Design By Dynamic Programming

Outline

1 Problem Setup

Lloyd-Max Quantizer Design Using DynamicProgramming

• For discrete Θ

• One construction level in the interval (β1, β2) ⊆ [0, 1]

T1(β1, β2) = mina

∑θ∈Θ∩(β1,β2)

d(θ, a)p(θ) (15)

• K construction levels in the interval (β1, β2)

TK (β1, β2) = mina,b:β1<b1<···<bK−1<β2

K∑k=1

∑θ∈Θ∩(bk−1,bk )

d(θ, a)p(θ) (16)

• Notice thatD∗N = TN(0, 1) (17)

TheoremLet b∗1, . . . , b

∗K−1 be the optimizing boundary points for

TK (b∗0 = 0, b∗K = 1), then b∗1, . . . , b∗K−2 must be the optimizing boundary

points for TK−1(b∗0, b∗K−1), and

TK (b∗0, b∗K ) = min

bK−1:b∗0<bK−1<b∗K[TK−1(b∗0, bK−1) + T1(bK−1, b

∗K )] (18)

• For any 1 < k ≤ K and any discrete β ∈ (b∗0, b∗K ]

Tk(b∗0, β) = minb:b∗0<b<β

[Tk−1(b∗0, b) + T1(b, β)] (19)

• Optimizing threshold

b∗k−1(b∗0, β) = arg minb:b∗0<b<β

[Tk−1(b∗0, b) + T1(b, β)] (20)

Algorithm 2: DP algorithm for Lloyd-Max Quantizer Design

Result: Minimize the average distortion for a N-level Lloyd-Max quantizerstep 1) Compute the values of T1(β1, β2) for all discrete β1 and β2 in [0, 1]step 2) For each n = 2, . . . ,N compute Tn(0, β) and b∗n−1(0, β) for all βin (0, 1] using (19) and (20)step 3) Let bN = 1, for each n = N, . . . , 2 set bn−1 = b∗n−1(0, bn)step 4) For each n = 1, . . . ,N, set

ak = arg mina E [d(Θ, a)|Θ ∈ (bk−1, bk)]

TheoremThe boundaries {bk}Kk=0 and reconstruction levels {ak}Kk=1 returned byAlgorithm 2 represent the optimal quantizer.[4]

Variable Rate Optimum Quantizer Design

Outline

1 Problem Setup

Variable Rate Optimum Quantizer Design Problem Setup

Outline

1 Problem Setup

Fixed Rate Quantizer Design and Variable Rate Quantizer Design

• N-level Lloyd-Max quantizer : minimize the average distortion for afixed number of levels N.[1][2]

• N-level Optimum Quantizer : minimize the average distortion for afixed number of levels N subject to an entropy constraint ofrate.[3]

Problem Setup

• Huffman encode/decode QN(Θ) with rate

R = −N∑

Pn log2 Pn (21)

Pn ,∫ bn

bn−1

p(θ)dθ (22)

• Distortion Metricd : [0, 1]×A → R+ (23)

withd(a, a) = 0 (24)

• Average Distortion

DN(b, a) =N∑

∫ bn

bn−1

d(θ, an)p(θ)dθ (25)

Problem Setup

Minimizing Distortion with a Rate Constraint

D∗N = minN∑

∫ bn

bn−1

d(θ, an)p(θ)dθ

s.t. R = −N∑

Pn log2 Pn ≤ H0

Form the Lagrangian

L = DN(b, a) + λ(R(b)− H0) (27)

Variable Rate Optimum Quantizer Design Analysis

Outline

1 Problem Setup

Analysis

Take Derivative w.r.t. an

∂anL =

∂anDN(b, a)

∫ bn

bn−1

∂and(θ, an)

]p(θ)dθ

Denote the optimum reconstruction levels by

a∗n , an(b) (29)

a∗n can be obtained by (28), i.e. for the mean-square distortion [2]

a∗n =

∫ bnbn−1

θp(θ)dθ∫ bnbn−1

p(θ)dθ(30)

Analysis

Take Derivative w.r.t. bn

∂bnL =

∂bnDN(b, a∗) + λ

∂bnR(b) (31)

which leads to

λ(ln(Pn+1/Pn)) = d(bn, a∗n+1)− d(bn, a

∗n) (32)

for all n ∈ {1, 2, . . . ,N − 1}.

Variable Rate Optimum Quantizer Design Generalized Lloyd-Max Algorithm

Outline

1 Problem Setup

Generalized Max Algorithm

Algorithm 3: Generalized Max Algorithm

Result: Minimize the average distortion for a N-level variable rateoptimum quantizer

step 1) Given N and fixed λ, set b0 = 0, choose an initial value for b1 andset n = 1step 2) For the present values of bn−1 and bn, use (28) and (32) to findbn+1. If n ≤ N − 1, replace n by n + 1 and go to step 2). Otherwisecontinuestep 3) If bN obtained in step 2) is equal to 1, the initial guess for b1 isgood and the resulting b and a satisfy the necessary conditions foroptimality. Otherwise go to step 1) and change b1

Generalized Lloyd Algorithm

Apply the mean square distortion to (32)

λ(ln(Pn+1/Pn)) = (a∗n+1 − a∗n)(a∗n+1 + a∗n − 2bn) (33)

which can be written as

b∗n =a∗n+1

a∗n− λ

2(a∗n+1 − a∗n)ln(Pn+1/Pn) (34)

Generalized Lloyd Algorithm

Algorithm 4: Generalized Lloyd Algorithm

Result: Minimize the average distortion for a N-level variable rateoptimum quantizer

step 1) Given N and fixed λ, set b0 = 0, bN = 1, and choose an initialvalue for b1, . . . , bN−1.step 2) Compute {a1, . . . , aN} by (28)step 3) Compute {b1, . . . , bN−1} by (34)step 4) Run step 2) and step 3) ` times. If D∗N converges, output b and a.Otherwise go to step 1) and change initial guess for b

quantization design - drexel engineering · • n-level lloyd-max quantizer : minimize the average...

Documents

the ductile iron newsductile.org/magazine/2005 issue...

joseph kiesecker ph.d. christina kennedy ph.d. · from...

smooth boundary point adaptive quantizer for on-chip image...

impact analysis of baseband quantizer on coding efﬁciency

a tutorial on quantizer design for networked control

minimize 2013 players pack

a 3-bit current mode quantizer for continous time delta...

scalar and vector...

wavelet-based image coder with entropy-constrained lattice...

minimize campfire impacts

manure handling and storage to minimize n loading of the...

analog vs. digital analog and digital...

wavelet transform. wavelet transform coding: multiresolution...

supervised learning of quantizer codebooks by information...

quantization -...

penrose quantizer assembly guide schematic and …€¦ ·...

84 ieee an algorithm for vector quantizer...

statistical analysis of the quantizer...

minpower | minimize your power

robert m. gray - columbia...