1. Introduction TV Denoising L1 Regularization Split Bregman Method Results
A Review of the Split Bregman Method for L1
Regularized Problems
Pardis Noorzad1
1
Department of Computer Engineering and IT
Amirkabir University of Technology
Khordad 1389
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

2. Introduction TV Denoising L1 Regularization Split Bregman Method Results
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

3. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Image Restoration and Variational Models
• Fundamental problem in image restoration: denoising
• Denoising is an important step in machine vision tasks
• Concern is to preserve important image features
• edges, texture
while removing noise
• Variational models have been very successful
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

4. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Image Restoration and Variational Models
• Fundamental problem in image restoration: denoising
• Denoising is an important step in machine vision tasks
• Concern is to preserve important image features
• edges, texture
while removing noise
• Variational models have been very successful
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

5. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Image Restoration and Variational Models
• Fundamental problem in image restoration: denoising
• Denoising is an important step in machine vision tasks
• Concern is to preserve important image features
• edges, texture
while removing noise
• Variational models have been very successful
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

6. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Image Restoration and Variational Models
• Fundamental problem in image restoration: denoising
• Denoising is an important step in machine vision tasks
• Concern is to preserve important image features
• edges, texture
while removing noise
• Variational models have been very successful
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

7. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Image Restoration and Variational Models
• Fundamental problem in image restoration: denoising
• Denoising is an important step in machine vision tasks
• Concern is to preserve important image features
• edges, texture
while removing noise
• Variational models have been very successful
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

8. Introduction TV Denoising L1 Regularization Split Bregman Method Results
TV-based Image Restoration
• Total variation based image restoration models first introduced by
Rudin, Osher, and Fatemi [ROF92]
• An early example of PDE based edge preserving denoising
• Has been extended and solved in a variety of ways
• Here, the Split Bregman method is introduced
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

9. Introduction TV Denoising L1 Regularization Split Bregman Method Results
TV-based Image Restoration
• Total variation based image restoration models first introduced by
Rudin, Osher, and Fatemi [ROF92]
• An early example of PDE based edge preserving denoising
• Has been extended and solved in a variety of ways
• Here, the Split Bregman method is introduced
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

10. Introduction TV Denoising L1 Regularization Split Bregman Method Results
TV-based Image Restoration
• Total variation based image restoration models first introduced by
Rudin, Osher, and Fatemi [ROF92]
• An early example of PDE based edge preserving denoising
• Has been extended and solved in a variety of ways
• Here, the Split Bregman method is introduced
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

11. Introduction TV Denoising L1 Regularization Split Bregman Method Results
TV-based Image Restoration
• Total variation based image restoration models first introduced by
Rudin, Osher, and Fatemi [ROF92]
• An early example of PDE based edge preserving denoising
• Has been extended and solved in a variety of ways
• Here, the Split Bregman method is introduced
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

12. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Denoising
Decomposition
f=u+v
• f : Ω → R is the noisy image
• Ω is the bounded open subset of R2
• u is the true signal
• v ∼ N(0, σ2 ) is the white Gaussian noise
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

13. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Denoising
Decomposition
f=u+v
• f : Ω → R is the noisy image
• Ω is the bounded open subset of R2
• u is the true signal
• v ∼ N(0, σ2 ) is the white Gaussian noise
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

14. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Denoising
Decomposition
f=u+v
• f : Ω → R is the noisy image
• Ω is the bounded open subset of R2
• u is the true signal
• v ∼ N(0, σ2 ) is the white Gaussian noise
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

15. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Denoising
Decomposition
f=u+v
• f : Ω → R is the noisy image
• Ω is the bounded open subset of R2
• u is the true signal
• v ∼ N(0, σ2 ) is the white Gaussian noise
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

16. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Denoising
Decomposition
f=u+v
• f : Ω → R is the noisy image
• Ω is the bounded open subset of R2
• u is the true signal
• v ∼ N(0, σ2 ) is the white Gaussian noise
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

17. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Conventional Variational Model
Easy to solve — results are dissapointing
min (uxx + uyy )2 dxdy
Ω
such that
udxdy = fdxdy
Ω Ω
(white noise is of zero mean)
1
(u − f)2 dxdy = σ2 ,
0 2
(a priori information about v)
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

18. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Conventional Variational Model
Easy to solve — results are dissapointing
min (uxx + uyy )2 dxdy
Ω
such that
udxdy = fdxdy
Ω Ω
(white noise is of zero mean)
1
(u − f)2 dxdy = σ2 ,
0 2
(a priori information about v)
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

19. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Conventional Variational Model
Easy to solve — results are dissapointing
min (uxx + uyy )2 dxdy
Ω
such that
udxdy = fdxdy
Ω Ω
(white noise is of zero mean)
1
(u − f)2 dxdy = σ2 ,
0 2
(a priori information about v)
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

20. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Conventional Variational Model
Easy to solve — results are dissapointing
min (uxx + uyy )2 dxdy
Ω
such that
udxdy = fdxdy
Ω Ω
(white noise is of zero mean)
1
(u − f)2 dxdy = σ2 ,
0 2
(a priori information about v)
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

21. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
Outline
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

22. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
The ROF Model
Difficult to solve — successful for denoising
min { u BV + λ f − u 2}
2
u∈BV(Ω)
• λ > 0: scale parameter
• BV(Ω): space of functions with bounded variation on Ω
• . : BV seminorm or total variation given by,
u BV = | u|
Ω
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

23. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
The ROF Model
Difficult to solve — successful for denoising
min { u BV + λ f − u 2}
2
u∈BV(Ω)
• λ > 0: scale parameter
• BV(Ω): space of functions with bounded variation on Ω
• . : BV seminorm or total variation given by,
u BV = | u|
Ω
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

24. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
The ROF Model
Difficult to solve — successful for denoising
min { u BV + λ f − u 2}
2
u∈BV(Ω)
• λ > 0: scale parameter
• BV(Ω): space of functions with bounded variation on Ω
• . : BV seminorm or total variation given by,
u BV = | u|
Ω
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

25. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
The ROF Model
Difficult to solve — successful for denoising
min { u BV + λ f − u 2}
2
u∈BV(Ω)
• λ > 0: scale parameter
• BV(Ω): space of functions with bounded variation on Ω
• . : BV seminorm or total variation given by,
u BV = | u|
Ω
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

26. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
BV seminorm
• It’s use is essential — allows image recovery with edges
• What if first term were replaced by Ω | u|p ?
• Which is both differentiable and strictly convex
• No good! For p > 1, its derivative has smoothing effect in the
optimality condition
• For TV however, the operator is degenerate, and affects only level
lines of the image
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

27. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
BV seminorm
• It’s use is essential — allows image recovery with edges
• What if first term were replaced by Ω | u|p ?
• Which is both differentiable and strictly convex
• No good! For p > 1, its derivative has smoothing effect in the
optimality condition
• For TV however, the operator is degenerate, and affects only level
lines of the image
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

28. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
BV seminorm
• It’s use is essential — allows image recovery with edges
• What if first term were replaced by Ω | u|p ?
• Which is both differentiable and strictly convex
• No good! For p > 1, its derivative has smoothing effect in the
optimality condition
• For TV however, the operator is degenerate, and affects only level
lines of the image
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

29. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
BV seminorm
• It’s use is essential — allows image recovery with edges
• What if first term were replaced by Ω | u|p ?
• Which is both differentiable and strictly convex
• No good! For p > 1, its derivative has smoothing effect in the
optimality condition
• For TV however, the operator is degenerate, and affects only level
lines of the image
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

30. Introduction TV Denoising L1 Regularization Split Bregman Method Results
The ROF Model
BV seminorm
• It’s use is essential — allows image recovery with edges
• What if first term were replaced by Ω | u|p ?
• Which is both differentiable and strictly convex
• No good! For p > 1, its derivative has smoothing effect in the
optimality condition
• For TV however, the operator is degenerate, and affects only level
lines of the image
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

31. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Iterated Total Variation
Outline
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

32. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Iterated Total Variation
Iterative Regularization
Adding back the noise
• In the ROF model, u − f is treated as error and discarded
• In the decomposition of f into signal u and additive noise v
• There exists some signal in v
• And some smoothing of textures in u
• Osher et al. [OBG+ 05] propose an iterated procedure to add the
noise back
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

33. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Iterated Total Variation
Iterative Regularization
Adding back the noise
• In the ROF model, u − f is treated as error and discarded
• In the decomposition of f into signal u and additive noise v
• There exists some signal in v
• And some smoothing of textures in u
• Osher et al. [OBG+ 05] propose an iterated procedure to add the
noise back
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

34. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Iterated Total Variation
Iterative Regularization
Adding back the noise
• In the ROF model, u − f is treated as error and discarded
• In the decomposition of f into signal u and additive noise v
• There exists some signal in v
• And some smoothing of textures in u
• Osher et al. [OBG+ 05] propose an iterated procedure to add the
noise back
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

35. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Iterated Total Variation
Iterative Regularization
Adding back the noise
• In the ROF model, u − f is treated as error and discarded
• In the decomposition of f into signal u and additive noise v
• There exists some signal in v
• And some smoothing of textures in u
• Osher et al. [OBG+ 05] propose an iterated procedure to add the
noise back
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

36. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Iterated Total Variation
Iterative Regularization
Adding back the noise
• In the ROF model, u − f is treated as error and discarded
• In the decomposition of f into signal u and additive noise v
• There exists some signal in v
• And some smoothing of textures in u
• Osher et al. [OBG+ 05] propose an iterated procedure to add the
noise back
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

37. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Iterated Total Variation
Iterative Regularization
The iteration
Step 1: Solve the ROF model to obtain:
u1 = arg min | u| + λ (f − u)2
u∈BV(Ω)
Step 2: Perform a correction step:
u2 = arg min | u| + λ (f + v1 − u)2
u∈BV(Ω)
(v1 is the noise estimated by the first step, f = u1 + v1 )
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

38. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Iterated Total Variation
Iterative Regularization
The iteration
Step 1: Solve the ROF model to obtain:
u1 = arg min | u| + λ (f − u)2
u∈BV(Ω)
Step 2: Perform a correction step:
u2 = arg min | u| + λ (f + v1 − u)2
u∈BV(Ω)
(v1 is the noise estimated by the first step, f = u1 + v1 )
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

39. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Definition
L1 regularized optimization
min Φ(u) 1 + H(u)
u
• Many important problems in imaging science (and other problems in
engineering) can be posed as L1 regularized optimization problems
• . 1 : the L1 norm
• both Φ(u) 1 and H(u) are convex functions
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

40. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Definition
L1 regularized optimization
min Φ(u) 1 + H(u)
u
• Many important problems in imaging science (and other problems in
engineering) can be posed as L1 regularized optimization problems
• . 1 : the L1 norm
• both Φ(u) 1 and H(u) are convex functions
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

41. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Definition
L1 regularized optimization
min Φ(u) 1 + H(u)
u
• Many important problems in imaging science (and other problems in
engineering) can be posed as L1 regularized optimization problems
• . 1 : the L1 norm
• both Φ(u) 1 and H(u) are convex functions
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

42. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Definition
L1 regularized optimization
min Φ(u) 1 + H(u)
u
• Many important problems in imaging science (and other problems in
engineering) can be posed as L1 regularized optimization problems
• . 1 : the L1 norm
• both Φ(u) 1 and H(u) are convex functions
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

43. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Outline
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

44. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Easy Instances
2
arg min Au − f 2 differentiable
u
2
arg min u 1 + u−f 2 solvable by shrinkage
u
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

45. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Easy Instances
2
arg min Au − f 2 differentiable
u
2
arg min u 1 + u−f 2 solvable by shrinkage
u
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

46. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Shrinkage
or Soft Thresholding
Solves the L1 problem of the form (H(.) is convex and differentiable):
arg min µ u 1 + H(u)
u
Based on this iterative scheme
1 2
uk+1 → arg min µ u 1 + u − (uk − δk H(uk ))
u 2δk
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

47. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Shrinkage
or Soft Thresholding
Solves the L1 problem of the form (H(.) is convex and differentiable):
arg min µ u 1 + H(u)
u
Based on this iterative scheme
1 2
uk+1 → arg min µ u 1 + u − (uk − δk H(uk ))
u 2δk
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

48. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Shrinkage
Continued
Since unknown u is componentwise separable, each component can be
independently obtained:
uk+1 = shrink((uk − δk H(uk ))i , µδk ), i = 1, . . . , n,
i

y − α, y ∈ (α, ∞),

shrink(y, α) := sgn(y) max{|y| − α, 0} = 0, y ∈ [−α, α],


y + α, y ∈ (−∞, −α).
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

49. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Shrinkage
Continued
Since unknown u is componentwise separable, each component can be
independently obtained:
uk+1 = shrink((uk − δk H(uk ))i , µδk ), i = 1, . . . , n,
i

y − α, y ∈ (α, ∞),

shrink(y, α) := sgn(y) max{|y| − α, 0} = 0, y ∈ [−α, α],


y + α, y ∈ (−∞, −α).
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

50. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Shrinkage
Continued
Since unknown u is componentwise separable, each component can be
independently obtained:
uk+1 = shrink((uk − δk H(uk ))i , µδk ), i = 1, . . . , n,
i

y − α, y ∈ (α, ∞),

shrink(y, α) := sgn(y) max{|y| − α, 0} = 0, y ∈ [−α, α],


y + α, y ∈ (−∞, −α).
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

51. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Hard Instances
2
arg min Φ(u) 1 + u−f 2
u
2
arg min u 1 + Au − f 2
u
What makes these problems hard?
The coupling between the L1 and L2 terms.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

52. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Hard Instances
2
arg min Φ(u) 1 + u−f 2
u
2
arg min u 1 + Au − f 2
u
What makes these problems hard?
The coupling between the L1 and L2 terms.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

53. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Hard Instances
2
arg min Φ(u) 1 + u−f 2
u
2
arg min u 1 + Au − f 2
u
What makes these problems hard?
The coupling between the L1 and L2 terms.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

54. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Easy vs. Hard Problems
Hard Instances
2
arg min Φ(u) 1 + u−f 2
u
2
arg min u 1 + Au − f 2
u
What makes these problems hard?
The coupling between the L1 and L2 terms.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

55. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Outline
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

56. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Split the L1 and L2 components
To solve the general regularization problem:
arg min Φ(u) 1 + H(u)
u
Introduce d = Φ(u) and solve the constrained problem
arg min d 1 + H(u) such that d = Φ(u)
u,d
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

57. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Split the L1 and L2 components
To solve the general regularization problem:
arg min Φ(u) 1 + H(u)
u
Introduce d = Φ(u) and solve the constrained problem
arg min d 1 + H(u) such that d = Φ(u)
u,d
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

58. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Split the L1 and L2 components
To solve the general regularization problem:
arg min Φ(u) 1 + H(u)
u
Introduce d = Φ(u) and solve the constrained problem
arg min d 1 + H(u) such that d = Φ(u)
u,d
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

59. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Split the L1 and L2 components
To solve the general regularization problem:
arg min Φ(u) 1 + H(u)
u
Introduce d = Φ(u) and solve the constrained problem
arg min d 1 + H(u) such that d = Φ(u)
u,d
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

60. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Split the L1 and L2 components
Continued
Add an L2 penalty term to get an unconstrained problem
λ 2
arg min d 1 + H(u) + d − Φ(u)
u,d 2
• Obvious way is to use the penalty method to solve this
• However, as λk → ∞, the condition number of the Hessian
approaches infinity, making it impractical to use fast iterative
methods like Conjugate Gradient to approximate the inverse of the
Hessian.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

61. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Split the L1 and L2 components
Continued
Add an L2 penalty term to get an unconstrained problem
λ 2
arg min d 1 + H(u) + d − Φ(u)
u,d 2
• Obvious way is to use the penalty method to solve this
• However, as λk → ∞, the condition number of the Hessian
approaches infinity, making it impractical to use fast iterative
methods like Conjugate Gradient to approximate the inverse of the
Hessian.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

62. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Split the L1 and L2 components
Continued
Add an L2 penalty term to get an unconstrained problem
λ 2
arg min d 1 + H(u) + d − Φ(u)
u,d 2
• Obvious way is to use the penalty method to solve this
• However, as λk → ∞, the condition number of the Hessian
approaches infinity, making it impractical to use fast iterative
methods like Conjugate Gradient to approximate the inverse of the
Hessian.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

63. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Split Bregman Formulation
Split the L1 and L2 components
Continued
Add an L2 penalty term to get an unconstrained problem
λ 2
arg min d 1 + H(u) + d − Φ(u)
u,d 2
• Obvious way is to use the penalty method to solve this
• However, as λk → ∞, the condition number of the Hessian
approaches infinity, making it impractical to use fast iterative
methods like Conjugate Gradient to approximate the inverse of the
Hessian.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

64. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
Outline
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

65. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
Analog of adding the noise back
The optimization problem is solved by iterating
λ 2
(uk+1 , dk+1 ) = arg min d 1 + H(u) + d − Φ(u) − bk
u,d 2
bk+1 = bk + (Φ(u) − dk )
The iteration in the first line can be done separately for u and d.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

66. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
Analog of adding the noise back
The optimization problem is solved by iterating
λ 2
(uk+1 , dk+1 ) = arg min d 1 + H(u) + d − Φ(u) − bk
u,d 2
bk+1 = bk + (Φ(u) − dk )
The iteration in the first line can be done separately for u and d.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

67. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
Analog of adding the noise back
The optimization problem is solved by iterating
λ 2
(uk+1 , dk+1 ) = arg min d 1 + H(u) + d − Φ(u) − bk
u,d 2
bk+1 = bk + (Φ(u) − dk )
The iteration in the first line can be done separately for u and d.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

68. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
3-step Algorithm
λ 2
Step 1: uk+1 = arg minu H(u) + 2 dk − Φ(u) − bk 2
λ 2
Step 2: dk+1 = arg mind d 1 + 2 dk − Φ(u) − bk 2
Step 3: bk+1 = bk + Φ(uk+1 ) − dk+1
• Step 1 is now a differentiable optimization problem, we’ll solve with
Gauss Seidel
• Step 2 can be solved efficiently with shrinkage
• Step 3 is an explicit evaluation
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

69. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
3-step Algorithm
λ 2
Step 1: uk+1 = arg minu H(u) + 2 dk − Φ(u) − bk 2
λ 2
Step 2: dk+1 = arg mind d 1 + 2 dk − Φ(u) − bk 2
Step 3: bk+1 = bk + Φ(uk+1 ) − dk+1
• Step 1 is now a differentiable optimization problem, we’ll solve with
Gauss Seidel
• Step 2 can be solved efficiently with shrinkage
• Step 3 is an explicit evaluation
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

70. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
3-step Algorithm
λ 2
Step 1: uk+1 = arg minu H(u) + 2 dk − Φ(u) − bk 2
λ 2
Step 2: dk+1 = arg mind d 1 + 2 dk − Φ(u) − bk 2
Step 3: bk+1 = bk + Φ(uk+1 ) − dk+1
• Step 1 is now a differentiable optimization problem, we’ll solve with
Gauss Seidel
• Step 2 can be solved efficiently with shrinkage
• Step 3 is an explicit evaluation
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

71. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
3-step Algorithm
λ 2
Step 1: uk+1 = arg minu H(u) + 2 dk − Φ(u) − bk 2
λ 2
Step 2: dk+1 = arg mind d 1 + 2 dk − Φ(u) − bk 2
Step 3: bk+1 = bk + Φ(uk+1 ) − dk+1
• Step 1 is now a differentiable optimization problem, we’ll solve with
Gauss Seidel
• Step 2 can be solved efficiently with shrinkage
• Step 3 is an explicit evaluation
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

72. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
3-step Algorithm
λ 2
Step 1: uk+1 = arg minu H(u) + 2 dk − Φ(u) − bk 2
λ 2
Step 2: dk+1 = arg mind d 1 + 2 dk − Φ(u) − bk 2
Step 3: bk+1 = bk + Φ(uk+1 ) − dk+1
• Step 1 is now a differentiable optimization problem, we’ll solve with
Gauss Seidel
• Step 2 can be solved efficiently with shrinkage
• Step 3 is an explicit evaluation
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

73. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Bregman Iteration
3-step Algorithm
λ 2
Step 1: uk+1 = arg minu H(u) + 2 dk − Φ(u) − bk 2
λ 2
Step 2: dk+1 = arg mind d 1 + 2 dk − Φ(u) − bk 2
Step 3: bk+1 = bk + Φ(uk+1 ) − dk+1
• Step 1 is now a differentiable optimization problem, we’ll solve with
Gauss Seidel
• Step 2 can be solved efficiently with shrinkage
• Step 3 is an explicit evaluation
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

74. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Outline
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

75. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Anisotropic TV
µ
arg min | x u| +| y u| + u−f 2
2
u 2
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

76. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Anisotropic TV
The steps
Step 1: uk+1 = G(uk )
1
Step 2: dk+1 = shrink(
x xu
k+1
+ bk , λ )
x
1
Step 3: dk+1 = shrink(
y yu
k+1
+ bk , λ )
y
Step 4: bk+1 = bk + (
x x xu − x)
Step 5: bk+1 = bk + (
y y yu − y)
• G(uk ): result of one Gauss-Seidel sweep for the corresponding L2
optimization
• This algorithm is cheap — each step is a few operations per pixel
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

77. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Anisotropic TV
The steps
Step 1: uk+1 = G(uk )
1
Step 2: dk+1 = shrink(
x xu
k+1
+ bk , λ )
x
1
Step 3: dk+1 = shrink(
y yu
k+1
+ bk , λ )
y
Step 4: bk+1 = bk + (
x x xu − x)
Step 5: bk+1 = bk + (
y y yu − y)
• G(uk ): result of one Gauss-Seidel sweep for the corresponding L2
optimization
• This algorithm is cheap — each step is a few operations per pixel
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

78. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Anisotropic TV
The steps
Step 1: uk+1 = G(uk )
1
Step 2: dk+1 = shrink(
x xu
k+1
+ bk , λ )
x
1
Step 3: dk+1 = shrink(
y yu
k+1
+ bk , λ )
y
Step 4: bk+1 = bk + (
x x xu − x)
Step 5: bk+1 = bk + (
y y yu − y)
• G(uk ): result of one Gauss-Seidel sweep for the corresponding L2
optimization
• This algorithm is cheap — each step is a few operations per pixel
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

79. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Anisotropic TV
The steps
Step 1: uk+1 = G(uk )
1
Step 2: dk+1 = shrink(
x xu
k+1
+ bk , λ )
x
1
Step 3: dk+1 = shrink(
y yu
k+1
+ bk , λ )
y
Step 4: bk+1 = bk + (
x x xu − x)
Step 5: bk+1 = bk + (
y y yu − y)
• G(uk ): result of one Gauss-Seidel sweep for the corresponding L2
optimization
• This algorithm is cheap — each step is a few operations per pixel
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

80. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Anisotropic TV
The steps
Step 1: uk+1 = G(uk )
1
Step 2: dk+1 = shrink(
x xu
k+1
+ bk , λ )
x
1
Step 3: dk+1 = shrink(
y yu
k+1
+ bk , λ )
y
Step 4: bk+1 = bk + (
x x xu − x)
Step 5: bk+1 = bk + (
y y yu − y)
• G(uk ): result of one Gauss-Seidel sweep for the corresponding L2
optimization
• This algorithm is cheap — each step is a few operations per pixel
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

81. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Anisotropic TV
The steps
Step 1: uk+1 = G(uk )
1
Step 2: dk+1 = shrink(
x xu
k+1
+ bk , λ )
x
1
Step 3: dk+1 = shrink(
y yu
k+1
+ bk , λ )
y
Step 4: bk+1 = bk + (
x x xu − x)
Step 5: bk+1 = bk + (
y y yu − y)
• G(uk ): result of one Gauss-Seidel sweep for the corresponding L2
optimization
• This algorithm is cheap — each step is a few operations per pixel
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

82. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Anisotropic TV
The steps
Step 1: uk+1 = G(uk )
1
Step 2: dk+1 = shrink(
x xu
k+1
+ bk , λ )
x
1
Step 3: dk+1 = shrink(
y yu
k+1
+ bk , λ )
y
Step 4: bk+1 = bk + (
x x xu − x)
Step 5: bk+1 = bk + (
y y yu − y)
• G(uk ): result of one Gauss-Seidel sweep for the corresponding L2
optimization
• This algorithm is cheap — each step is a few operations per pixel
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

83. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Applying SB to TV Denoising
Isotropic TV
With similar steps
2 2 µ 2
arg min ( x u)i +( y u)i + u−f 2
u 2
i
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

84. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Fast Convergence
Outline
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

85. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Fast Convergence
Split Bregman is fast
Intel Core 2 Duo desktop (3 GHz), compiled with g++
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

86. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Fast Convergence
Split Bregman is fast
Compared to Graph Cuts
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

87. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Acceptable Intermediate Results
Outline
1 Introduction
2 TV Denoising
The ROF Model
Iterated Total Variation
3 L1 Regularization
Easy vs. Hard Problems
4 Split Bregman Method
Split Bregman Formulation
Bregman Iteration
Applying SB to TV Denoising
5 Results
Fast Convergence
Acceptable Intermediate Results
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

88. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Acceptable Intermediate Results
Intermediate images are smooth
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

89. Introduction TV Denoising L1 Regularization Split Bregman Method Results
Acceptable Intermediate Results
Intermediate images are smooth
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

90. References
Tom Goldstein and Stanley Osher, The split bregman method for
l1-regularized problems, SIAM Journal on Imaging Sciences 2
(2009), no. 2, 323–343.
Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and
Wotao Yin, An iterative regularization method for total
variation-based image restoration, Multiscale Modeling and
Simulation 4 (2005), 460–489.
Leonid I. Rudin, Stanley Osher, and Emad Fatemi, Nonlinear total
variation based noise removal algorithms, Physica D 60 (1992),
no. 1-4, 259–268.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

91. References
Tom Goldstein and Stanley Osher, The split bregman method for
l1-regularized problems, SIAM Journal on Imaging Sciences 2
(2009), no. 2, 323–343.
Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and
Wotao Yin, An iterative regularization method for total
variation-based image restoration, Multiscale Modeling and
Simulation 4 (2005), 460–489.
Leonid I. Rudin, Stanley Osher, and Emad Fatemi, Nonlinear total
variation based noise removal algorithms, Physica D 60 (1992),
no. 1-4, 259–268.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

92. References
Tom Goldstein and Stanley Osher, The split bregman method for
l1-regularized problems, SIAM Journal on Imaging Sciences 2
(2009), no. 2, 323–343.
Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and
Wotao Yin, An iterative regularization method for total
variation-based image restoration, Multiscale Modeling and
Simulation 4 (2005), 460–489.
Leonid I. Rudin, Stanley Osher, and Emad Fatemi, Nonlinear total
variation based noise removal algorithms, Physica D 60 (1992),
no. 1-4, 259–268.
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad

93. Thank you for your attention. Any questions?
A Review of the Split Bregman Method for L1 Regularized Problems Pardis Noorzad