QMC Program: Trends and Advances in Monte Carlo Sampling Algorithms Workshop,...
presentation
1. Quantization Design
Jie Ren
jr843@drexel.edu
Adaptive Signal Processing and Information Theory Group
Department of Electrical and Computer Engineering
Drexel University, Philadelphia, PA 19104
July 28th and 30th, 2014
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 1 / 35
2. References
S. P. Lloyd, “Least squares quantization in pcm,” IEEE Transactions on Acoustics, Speech
and Signal Processing, vol. 28, no. 2, pp. 129–137, March 1982.
J. Max, “Quantizing for minimum distortion,” IEEE Transactions on Acoustics, Speech
and Signal Processing, vol. 6, no. 1, pp. 7–12, March 1960.
N. Farvardin and J. W. Modestino, “Optimum quantizer performance for a class of
non-gaussian memoryless sources,” IEEE Trans. Inform. Theory, vol. 30, no. 3, pp.
485–497, May 1984.
D. K. Sharma, “Design of absolutely optimal quantizers for a wide class of distortion
measures,” IEEE Trans. Inform. Theory, vol. 24, no. 6, pp. 693–702, November 1978.
P. A. Chou, T. Lookabaugh, and R. M. Gray, “Entropy-constrained vector quantization,”
IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 37, no. 1, pp. 31–42,
Juanuary 1989.
L. R. Varshney, “Unreliable and resource-constrained decoding,” Ph.D. dissertation,
Massachusetts Institute of Technology, 2010.
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 2 / 35
3. Problem Setup
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 3 / 35
4. Problem Setup
Quantization
Map a Large Set Θ of Input Values to a Smaller set A
• Discrete Quantization : Countable Θ, Countable A
• Continuous Quantization : Uncountable Θ, Countable A
• Scalar Quantization
Q : Θ → A (1)
• Vector Quantization
Q : Θ × Θ × · · · × Θ → A (2)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 4 / 35
5. Problem Setup
Problem Setup
N-level Continuous scalar Quantizer
• Source Θ with normalized support [0, 1] and pdf p(θ)
• N-level quantizer QN(·)
• N Reconstruction levels A = {a1, . . . , aN}
• Thresholds b1, . . . , bN−1 partitioning [0, 1] with b0 = 0 and bN = 1
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 5 / 35
6. Problem Setup
N-level Continuous scalar Quantizer
• Huffman encode/decode QN(Θ) with rate
R = −
N
n=1
Pn log2 Pn (3)
where
Pn
bn
bn−1
p(θ)dθ (4)
• Distortion Metric
d : [0, 1] × A → R+
(5)
with
d(a, a) = 0 (6)
• Average Distortion
DN(b, a) =
N
n=1
bn
bn−1
d(θ, an)p(θ)dθ (7)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 6 / 35
7. Problem Setup
Fixed Rate Quantizer Design and Variable Rate Quantizer Design
• N-level Lloyd-Max quantizer : minimize the average distortion for a
fixed number of levels N.[1][2]
• N-level Optimum Quantizer : minimize the average distortion for a
fixed number of levels N subject to an entropy constraint of
rate.[3]
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 7 / 35
8. Lloyd-max Quantizer Design
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 8 / 35
9. Lloyd-max Quantizer Design Local Optimality Conditions
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 9 / 35
10. Lloyd-max Quantizer Design Local Optimality Conditions
Local Optimality Conditions
• Nearest Neighbor Condition : For fixed reconstruction levels {ak},
Given any θ ∈ [ak, ak+1],
Q(θ) = d(θ, ak) ≤ d(θ, ak+1) ? ak : ak+1 (8)
• Centroid Condition : For fixed regions {Rk} with thresholds {bk},
ak = arg min
a
bk
bk−1
d(θ, a)p(θ)dθ (9)
• Zero Probability Boundary Condition, for all bk, k = 1, . . . , K − 1
P(θ = bk) = 0 (10)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 10 / 35
11. Lloyd-max Quantizer Design Local Optimality Conditions
Necessary and Sufficient Conditions
Theorem
The nearest neighbor condition, the centroid condition, and the zero
probability of boundary condition are necessary for a Lloyd-Max quantizer
to be optimal.
Theorem
If the following conditions hold for a source Θ and distortion function
d(θ, a) :
1 p(θ) is positive and continuous in (0, 1)
2
1
0 d(θ, a)p(θ)dθ is finite for all a
3 d(θ, a) is zero only for θ = a, is continuous in θ for all a, and is
continuous and convex in a
then the nearest neighbor condition, centroid condition, and zero
probability of bound- ary conditions are sufficient to guarantee local
optimality of a quantizer.
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 11 / 35
12. Lloyd-max Quantizer Design By Alternating Optimization
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 12 / 35
13. Lloyd-max Quantizer Design By Alternating Optimization
Lloyd-Max algorithm
Alternating Minimization
• For fixed {ak}, minimize D w.r.t. {bk}
• For fixed {bk}, minimize D w.r.t. {ak}
• D monotone decreasing
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 13 / 35
14. Lloyd-max Quantizer Design By Alternating Optimization
Lloyd-Max algorithm
Algorithm 1: Lloyd-Max
Result: Minimize the average distortion for a N-level Lloyd-Max quantizer
step 1) Choose an arbitrary set of initial reconstruction levels {an}
step 2) For each n = 1, . . . , N set Rn = {θ|d(θ, an) ≤ d(θ, aj ), j = n}
step 3) For each n = 1, . . . , N set an = arg mina E[d(Θ, a)|Θ ∈ Rn]
step 4) Repeat step 2 and 3 until change in average distortion is negligible
step 5) Revise {an} and {Rn} to satisfy the zero probability of boundary
condition
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 14 / 35
15. Lloyd-max Quantizer Design By Alternating Optimization
Analysis
• Local optimum guaranteed by Theorem. 2
• Monotonic Convergence in N
D∗
N(b∗
, a∗
) =
N
n=1
b∗
n
b∗
n−1
d(θ, a∗
n)p(θ)dθ (11)
D∗
= lim
N→∞
D∗
N (12)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 15 / 35
16. Lloyd-max Quantizer Design By Alternating Optimization
Monotonic Convergence in N
The Lloyd-Max N-level quantizer is the solution of the following
problem:
D∗
N = min
N
n=1
bn
bn−1
d(θ, an)p(θ)dθ
s.t. b0 = 0
bN = 1
bn−1 ≤ bn, n = 1, . . . , N
an ≤ bn, n = 1, . . . , N
(13)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 16 / 35
17. Lloyd-max Quantizer Design By Alternating Optimization
Monotonic Convergence in N
• Degenerate the N-level Lloyd-Max quantizer to N − 1
• By adding the additional constraint bN−1 = 1 to (13) and forcing
aN = 1, hence
D∗
N−1 ≥ D∗
N (14)
• D∗
N bounded below by 0
• D∗
N converges
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 17 / 35
18. Lloyd-max Quantizer Design By Dynamic Programming
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 18 / 35
19. Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using Dynamic
Programming
• For discrete Θ
• One construction level in the interval (β1, β2) ⊆ [0, 1]
T1(β1, β2) = min
a
θ∈Θ∩(β1,β2)
d(θ, a)p(θ) (15)
• K construction levels in the interval (β1, β2)
TK (β1, β2) = min
a,b:β1<b1<···<bK−1<β2
K
k=1 θ∈Θ∩(bk−1,bk )
d(θ, a)p(θ) (16)
• Notice that
D∗
N = TN(0, 1) (17)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 19 / 35
20. Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using Dynamic
Programming
Theorem
Let b∗
1, . . . , b∗
K−1 be the optimizing boundary points for
TK (b∗
0 = 0, b∗
K = 1), then b∗
1, . . . , b∗
K−2 must be the optimizing boundary
points for TK−1(b∗
0, b∗
K−1), and
TK (b∗
0, b∗
K ) = min
bK−1:b∗
0 <bK−1<b∗
K
[TK−1(b∗
0, bK−1) + T1(bK−1, b∗
K )] (18)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 20 / 35
21. Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using Dynamic
Programming
• For any 1 < k ≤ K and any discrete β ∈ (b∗
0, b∗
K ]
Tk(b∗
0, β) = min
b:b∗
0 <b<β
[Tk−1(b∗
0, b) + T1(b, β)] (19)
• Optimizing threshold
b∗
k−1(b∗
0, β) = arg min
b:b∗
0 <b<β
[Tk−1(b∗
0, b) + T1(b, β)] (20)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 21 / 35
22. Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using Dynamic
Programming
Algorithm 2: DP algorithm for Lloyd-Max Quantizer Design
Result: Minimize the average distortion for a N-level Lloyd-Max quantizer
step 1) Compute the values of T1(β1, β2) for all discrete β1 and β2 in [0, 1]
step 2) For each n = 2, . . . , N compute Tn(0, β) and b∗
n−1(0, β) for all β
in (0, 1] using (19) and (20)
step 3) Let bN = 1, for each n = N, . . . , 2 set bn−1 = b∗
n−1(0, bn)
step 4) For each n = 1, . . . , N, set
ak = arg mina E[d(Θ, a)|Θ ∈ (bk−1, bk)]
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 22 / 35
23. Lloyd-max Quantizer Design By Dynamic Programming
Lloyd-Max Quantizer Design Using Dynamic
Programming
Theorem
The boundaries {bk}K
k=0 and reconstruction levels {ak}K
k=1 returned by
Algorithm 2 represent the optimal quantizer.[4]
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 23 / 35
24. Variable Rate Optimum Quantizer Design
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 24 / 35
25. Variable Rate Optimum Quantizer Design Problem Setup
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 25 / 35
26. Variable Rate Optimum Quantizer Design Problem Setup
Fixed Rate Quantizer Design and Variable Rate Quantizer Design
• N-level Lloyd-Max quantizer : minimize the average distortion for a
fixed number of levels N.[1][2]
• N-level Optimum Quantizer : minimize the average distortion for a
fixed number of levels N subject to an entropy constraint of
rate.[3]
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 26 / 35
27. Variable Rate Optimum Quantizer Design Problem Setup
Problem Setup
• Huffman encode/decode QN(Θ) with rate
R = −
N
n=1
Pn log2 Pn (21)
where
Pn
bn
bn−1
p(θ)dθ (22)
• Distortion Metric
d : [0, 1] × A → R+
(23)
with
d(a, a) = 0 (24)
• Average Distortion
DN(b, a) =
N
n=1
bn
bn−1
d(θ, an)p(θ)dθ (25)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 27 / 35
28. Variable Rate Optimum Quantizer Design Problem Setup
Problem Setup
Minimizing Distortion with a Rate Constraint
D∗
N = min
N
n=1
bn
bn−1
d(θ, an)p(θ)dθ
s.t. R = −
N
n=1
Pn log2 Pn ≤ H0
(26)
Form the Lagrangian
L = DN(b, a) + λ(R(b) − H0) (27)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 28 / 35
29. Variable Rate Optimum Quantizer Design Analysis
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 29 / 35
30. Variable Rate Optimum Quantizer Design Analysis
Analysis
Take Derivative w.r.t. an
∂
∂an
L =
∂
∂an
DN(b, a)
=
bn
bn−1
∂
∂an
d(θ, an) p(θ)dθ
(28)
Denote the optimum reconstruction levels by
a∗
n an(b) (29)
a∗
n can be obtained by (28), i.e. for the mean-square distortion [2]
a∗
n =
bn
bn−1
θp(θ)dθ
bn
bn−1
p(θ)dθ
(30)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 30 / 35
31. Variable Rate Optimum Quantizer Design Analysis
Analysis
Take Derivative w.r.t. bn
∂
∂bn
L =
∂
∂bn
DN(b, a∗
) + λ
∂
∂bn
R(b) (31)
which leads to
λ(ln(Pn+1/Pn)) = d(bn, a∗
n+1) − d(bn, a∗
n) (32)
for all n ∈ {1, 2, . . . , N − 1}.
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 31 / 35
32. Variable Rate Optimum Quantizer Design Generalized Lloyd-Max Algorithm
Outline
1 Problem Setup
2 Lloyd-max Quantizer Design
Local Optimality Conditions
By Alternating Optimization
By Dynamic Programming
3 Variable Rate Optimum Quantizer Design
Problem Setup
Analysis
Generalized Lloyd-Max Algorithm
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 32 / 35
33. Variable Rate Optimum Quantizer Design Generalized Lloyd-Max Algorithm
Generalized Max Algorithm
Algorithm 3: Generalized Max Algorithm
Result: Minimize the average distortion for a N-level variable rate
optimum quantizer
step 1) Given N and fixed λ, set b0 = 0, choose an initial value for b1 and
set n = 1
step 2) For the present values of bn−1 and bn, use (28) and (32) to find
bn+1. If n ≤ N − 1, replace n by n + 1 and go to step 2). Otherwise
continue
step 3) If bN obtained in step 2) is equal to 1, the initial guess for b1 is
good and the resulting b and a satisfy the necessary conditions for
optimality. Otherwise go to step 1) and change b1
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 33 / 35
34. Variable Rate Optimum Quantizer Design Generalized Lloyd-Max Algorithm
Generalized Lloyd Algorithm
Apply the mean square distortion to (32)
λ(ln(Pn+1/Pn)) = (a∗
n+1 − a∗
n)(a∗
n+1 + a∗
n − 2bn) (33)
which can be written as
b∗
n =
a∗
n+1
a∗
n
−
λ
2(a∗
n+1 − a∗
n)
ln(Pn+1/Pn) (34)
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 34 / 35
35. Variable Rate Optimum Quantizer Design Generalized Lloyd-Max Algorithm
Generalized Lloyd Algorithm
Algorithm 4: Generalized Lloyd Algorithm
Result: Minimize the average distortion for a N-level variable rate
optimum quantizer
step 1) Given N and fixed λ, set b0 = 0, bN = 1, and choose an initial
value for b1, . . . , bN−1.
step 2) Compute {a1, . . . , aN} by (28)
step 3) Compute {b1, . . . , bN−1} by (34)
step 4) Run step 2) and step 3) times. If D∗
N converges, output b and a.
Otherwise go to step 1) and change initial guess for b
Jie Ren (Drexel ASPITRG) QD July 28th
and 30th
, 2014 35 / 35