SlideShare a Scribd company logo
1 of 123
Download to read offline
Statistical Image Models


          Eero Simoncelli

   Howard Hughes Medical Institute,
     Center for Neural Science, and
Courant Institute of Mathematical Sciences
          New York University
Photographic Images
Diverse specialized structures:
ā€¢ edges/lines/contours
ā€¢ shadows/highlights
ā€¢ smooth regions
ā€¢ textured regions
Photographic Images
 Diverse specialized structures:
 ā€¢ edges/lines/contours
 ā€¢ shadows/highlights
 ā€¢ smooth regions
 ā€¢ textured regions


Occupy a small region of the full space
spa
                                        ce o
                                            f all
                                                    ima
                                                       ges


                    typical images




One could describe this set as a
deterministic manifold....
ā€¢ Step edges are rare (lighting, junctions, texture, noise)
ā€¢ Step edges are rare (lighting, junctions, texture, noise)
ā€¢ One scaleā€™s texture is another scaleā€™s edge
ā€¢ Step edges are rare (lighting, junctions, texture, noise)
ā€¢ One scaleā€™s texture is another scaleā€™s edge
ā€¢ Need seamless transitions from isolated features to
  dense textures
spa
                                        ce o
                                            f all
                                                    ima
                                                       ges


                    typical images




One could describe this set as a
deterministic manifold....
spa
                                        ce o
                                            f all
                                                    ima
                                                       ges


                    typical images




One could describe this set as a
deterministic manifold....
But seems more natural to use probability
spa
                                        ce o
                                            f all
                                                    ima
                                                       ges


                    typical images




One could describe this set as a
                                                          P(x)
deterministic manifold....
But seems more natural to use probability
ā€œApplicationsā€
ā€¢ Engineering: compression, denoising, restoration,
  enhancement/modiļ¬cation, synthesis, manipulation




                                           [Hubel ā€˜95]
ā€œApplicationsā€
ā€¢ Engineering: compression, denoising, restoration,
  enhancement/modiļ¬cation, synthesis, manipulation

ā€¢ Science: optimality principles for neurobiology (evolution,
  development, learning, adaptation)




                                            [Hubel ā€˜95]
Density models


nonparametric                    parametric/
                                 constrained
Density models


nonparametric                    parametric/
                                 constrained


build a histogram
from lots of
observations...
Density models


nonparametric                        parametric/
                                     constrained


build a histogram        use ā€œnatural constraintsā€
from lots of               (geometry/photometry
observations...               of image formation,
                           computation, maxEnt)
Density models


nonparametric                                 parametric/
                       historical trend       constrained
                    (technology driven)

build a histogram                 use ā€œnatural constraintsā€
from lots of                        (geometry/photometry
observations...                        of image formation,
                                    computation, maxEnt)
histogram


Original image




Range: [0, 237]
Dims: [256, 256]


                   0   50   100   150   200   250
histogram


Original image




 Range: [0, 237]
 Dims: [256, 256]


                     0   50   100   150   200   250

                              histogram


Equalized image




Range: [1.99, 238]
 Dims: [256, 256]


                     0   50   100   150   200   250
histogram


Original image




 Range: [0, 237]
 Dims: [256, 256]


                     0   50   100   150   200   250

                              histogram


Equalized image




Range: [1.99, 238]
 Dims: [256, 256]


                     0   50   100   150   200   250
General methodology


Observe ā€œinterestingā€       Transform to
   Joint Statistics     Optimal Representation
General methodology


Observe ā€œinterestingā€       Transform to
   Joint Statistics     Optimal Representation
General methodology


Observe ā€œinterestingā€       Transform to
   Joint Statistics     Optimal Representation




ā€œOnion peelingā€
Evolution of image models
I. (1950ā€™s): Fourier + Gaussian

II. (mid 80ā€™s - late 90ā€™s): Wavelets + kurtotic marginals

III. (mid 90ā€™s - present): Wavelets + local context
  ā€¢ local amplitude (contrast)
  ā€¢ local orientation
IV. (last 5 years): Hierarchical models
a.
                    Pixel correlation                                        b.
                                                                        1




                                                               Correlation
                       I(x+2,y)




                                           I(x+4,y)
I(x+1,y)




           I(x,y)                 I(x,y)              I(x,y)            0
                                                                                    10
                                                                                  Spatia
a.
                                        Pixel correlation                                                                  b.
                                                                                                                      1




                                                                                                             Correlation
                                                           I(x+2,y)




                                                                                         I(x+4,y)
           I(x+1,y)




                               I(x,y)                                   I(x,y)                      I(x,y)            0
                                                                                                                                  10
                                                      b.                                                                        Spatia
                                                 1
                                        Correlation
I(x+4,y)




                      I(x,y)                     0
                                                               10      20       30       40
                                                             Spatial separation (pixels)
Translation invariance

Assuming translation invariance,
Translation invariance

Assuming translation invariance,

 => covariance matrix is Toeplitz (convolutional)
Translation invariance

Assuming translation invariance,

 => covariance matrix is Toeplitz (convolutional)

   => eigenvectors are sinusoids
Translation invariance

Assuming translation invariance,

 => covariance matrix is Toeplitz (convolutional)

   => eigenvectors are sinusoids

      => can diagonalize (decorrelate) with F.T.
Translation invariance

Assuming translation invariance,

  => covariance matrix is Toeplitz (convolutional)

    => eigenvectors are sinusoids

      => can diagonalize (decorrelate) with F.T.


Power spectrum captures full covariance structure
Spectral power
Structural:

Assume scale-invariance:
     F (sĻ‰) = s F (Ļ‰)
                   p



 then:
                 1
         F (Ļ‰) āˆ p
                Ļ‰


              [Ritterman 52; DeRiugin 56; Field 87; Tolhurst 92; Ruderman/Bialek 94; ...]
Spectral power
Structural:                                             Empirical:
                                                        6




Assume scale-invariance:                                5




     F (sĻ‰) = s F (Ļ‰)
                   p                                    4




                                            Log power
                                                        3




                                                  10
 then:                                                  2

                 1
         F (Ļ‰) āˆ p
                Ļ‰                                       1




                                                        0
                                                            0           1                  2          3

                                                                Log spatialfrequency (cycles/image)
                                                                   10



              [Ritterman 52; DeRiugin 56; Field 87; Tolhurst 92; Ruderman/Bialek 94; ...]
Principal Components Analysis
           (PCA) + whitening
 a.               b.           c.
20               20           4




-20              -20          -4
  -20       20     -20   20    -4    4
PCA basis for image blocks
PCA basis for image blocks




       PCA is not unique
Maximum entropy (maxEnt)
  The density with maximal entropy satisfying
           E (f (x)) = c
  is of the form
       pME (x) āˆ exp (āˆ’Ī»f (x))

  where Ī» depends on c


Examples: f (x) = x  2
                           f (x) = |x|
Model I (Fourier/Gaussian)
    Coefficient
     density:         Basis set:   Image:


                  :



                  :



                  :



                  :
Gaussian model is weak

         Ļ‰ āˆ’2              āˆ’1
                1/f2
                       F   F -1


P(x)   P(c)
Gaussian model is weak

            Ļ‰ āˆ’2               āˆ’1
                    1/f2
                           F   F -1


P(x)   P(c)

       a.                             b.




 F              2              āˆ’1
            Ļ‰              F
Gaussian model is weak

                             Ļ‰ āˆ’2                   āˆ’1
                                     1/f2
                                                F   F -1


        P(x)            P(c)

                        a.                                     b.




         F                       2                  āˆ’1
                             Ļ‰                  F
 a.              b.                        c.
20              20                        4




-20             -20                       -4
  -20      20     -20                20    -4              4
Bandpass Filter Responses
                0
               10
                                         Response histogram
                                         Gaussian density
 Probability




                    -2
               10




                    -4
               10
                500               0                           500
                          Filter Response


                         [Burt&Adelson 82; Field 87; Mallat 89; Daugman 89, ...]
ā€œIndependentā€ Components Analysis
                       (ICA)
  a.                         b.             c.                      d.
20                         20              4                       4




-20                        -20            -4                       -4
   -20                20         -20   20   -4                 4     -4               4




     For Linearly Transformed Factorial (LTF) sources:
     guaranteed independence
     (with some minor caveats)

                                         [Comon 94; Cardoso 96; Bell/Sejnowski 97; ...]
ICA on image blocks




           [Olshausen/Field ā€™96; Bell/Sejnowski ā€™97]
        [example obtained with FastICA, Hyvarinen]
Marginal densities
 log(Probability)




                                                      log(Probability)




                                                                                                         log(Probability)




                                                                                                                                                        log(Probability)
                           p = 0.46                                       p = 0.58                                           p = 0.48
                           !H/H = 0.0031                                  !H/H = 0.0011                                      !H/H = 0.0014

                         Wavelet coefficient value                       Wavelet coefficient value                          Wavelet coefficient value


                    Fig. 4. Log histograms of a single wavelet subband of four example images (see Fig. 1 for image
                    histogram, tails are truncated so as to show 99.8% of the distribution. Also shown (dashed lines) are
                    corresponding to equation (3). Text indicates the maximum-likelihood value of p used for the ļ¬tte
                      Well-ļ¬t by a generalized Gaussian:
                    the relative entropy (Kullback-Leibler divergence) of the model and histogram, as a fraction of th
                    histogram.

                                                     P (x) āˆ exp āˆ’|x/s|                              p
non-Gaussian than others. By the mid 1990s, a number
of authors had developed methods of optimizing a ba-
sis of ļ¬lters in order to to maximize the non-Gaussianity
of the responses [e.g., 36, 4]. Often these methods oper-
                                   [Mallat 89; Simoncelli&Adelson 96; Moulin&Liu 99; ...]
ate by optimizing a higher-order statistic such as kurto-
Kurtosis vs. bandwidth
                                    16

                                    14

                                    12
                  Sample Kurtosis




                                    10

                                     8

                                     6

                                     4

                                         0   0.5        1     1.5       2     2.5   3
                                                   Filter Bandwidth (octaves)


Note: Bandwidth matters much more than orientation
[see Bethge 06]
                                                                                        [after Field 87]
Octave-bandwidth representations


  Spatial
Frequency
Selectivity:




    Filter:
Model II (LTF)
Coefficient
 density:         Basis set:   Image:


              :



              :



              :
LTF also a weak model...

  Sample            Gaussianized




 Sample        ICA-transformed
               and Gaussianized
Trouble in paradise
Trouble in paradise
ā€¢ Biology: Visual system uses a cascade
  -   Whereā€™s the retina? The LGN?
  -   What happens after V1? Why donā€™t responses get
      sparser? [Baddeley etal 97; Chechik etal 06]
Trouble in paradise
ā€¢ Biology: Visual system uses a cascade
  -   Whereā€™s the retina? The LGN?
  -   What happens after V1? Why donā€™t responses get
      sparser? [Baddeley etal 97; Chechik etal 06]

ā€¢ Statistics: Images donā€™t obey ICA source model
  -   Any bandpass ļ¬lter gives sparse marginals [Baddeley 96]
      => Shallow optimum [Bethge 06; Lyu & Simoncelli 08]
  -   The responses of ICA ļ¬lters are highly dependent
      [Wegmann & Zetzsche 90, Simoncelli 97]
Conditional densities
                   1                                         1




                   0.6                                       0.6




                   0.2                                       0.2



                         -40   0               40   50             -40        0   40




                                   40




                                    0




                                   -40

                                         -40             0               40




Linear responses are not independent, even for optimized ļ¬lters!
                                                             [Simoncelli 97; Schwartz&Simoncelli 01]
CSH-02
[Schwartz&Simoncelli 01]
ā€¢ Large-magnitude subband coefļ¬cients are found at
  neighboring positions, orientations, and scales.
Modeling heteroscedasticity
                     (i.e., variable variance)

   Method 1: Conditional Gaussian




P (xn |{xk }) āˆ¼ N   0;       wnk |xk | + Ļƒ
                                     2       2

                         k




                                   [Simoncelli 97; Buccigrossi&Simoncelli 99;
                                     see also ARCH models in econometrics!]
Joint densities
        adjacent                       near                       far                other scale               other ori
 150                      150                        150                       150                      150

 100                      100                        100                       100                      100

  50                       50                         50                        50                       50

   0                        0                          0                         0                        0

!50                       !50                       !50                       !50                       !50

!100                     !100                       !100                      !100                     !100

!150                     !150                       !150                      !150                     !150

       !100   0    100          !100     0    100          !100     0   100          !500   0   500           !100   0     100


 150                      150                        150                       150                      150

 100                      100                        100                       100                      100

  50                       50                         50                        50                       50

   0                        0                          0                         0                        0

!50                       !50                       !50                       !50                       !50

!100                     !100                       !100                      !100                     !100

!150                     !150                       !150                      !150                     !150

       !100   0    100          !100     0    100          !100     0   100          !500   0   500           !100   0     100




       ā€¢ Nearby: densities are approximately circular/elliptical
Fig. 8. Empirical joint distributions of wavelet coefļ¬cients associated with different pairs of basis functions, for a single
image of a New York City street scene (see Fig. 1 for image description). The top row shows joint distributions as contour
plots, with lines drawn at equal intervals of log probability. The three leftmost examples correspond to pairs of basis func-


       ā€¢ Distant: densities are approximately factorial
tions at the same scale and orientation, but separated by different spatial offsets. The next corresponds to a pair at adjacent
scales (but the same orientation, and nearly the same position), and the rightmost corresponds to a pair at orthogonal orien-
tations (but the same scale and nearly the same position). The bottom row shows corresponding conditional distributions:
brightness corresponds to frequency of occurance, except that each column has been independently rescaled to ļ¬ll the full
range of intensities.                                          [Simoncelli, ā€˜97; Wainwright&Simoncelli, ā€˜99]
ICA-transformed joint densities
                        d=2                               d=16                            d=32




           12                               12                              12


           10                               10                              10
kurtosis




            8                                8                               8


            6                                6                               6


            4                                4                               4
             0    !/4   !/2      3!/4   !     0     !/4    !/2   3!/4   !     0     !/4    !/2   3!/4   !
                   orientation
                  data (ICAā€™d):                   sphericalized:                  factorialized:
ICA-transformed joint densities
                        d=2                             d=16                          d=32




           12                               12                            12


           10                               10                            10
kurtosis




            ā€¢ Local densities are elliptical (but non-Gaussian)
            8                                8                             8


            6                                6                             6

            ā€¢ Distant densities are factorial
            4                                4                             4
             0    !/4   !/2      3!/4   !     0   !/4    !/2   3!/4   !     0   !/4    !/2   3!/4   !
                   orientation
                  data (ICAā€™d): [Wegmann&Zetzsche ā€˜90; Simoncelli ā€™97; + many recent models]
                                        sphericalized:              factorialized:
Spherical vs LTF
  0.2
                                       blk                                                     blk               0.4                               blk
            blk size = 3x3                               0.2
                                                                    blk size = 7x7                                         blk size = 11x11
                                       spherical                                               spherical                                           spherical
                                       factorial                                               factorial        0.35                               factorial
 0.15
                                                                                                                 0.3
                                                        0.15
                                                                                                                0.25
  0.1
                                                                                                                 0.2
                                                         0.1
                                                                                                                0.15
 0.05                                                                                                            0.1
                                                        0.05
                                                                                                                0.05
   0                                                      0                                                       0
        3    6      9        12   15       18      20           3    6      9        12   15       18      20          3    6      9     12   15       18      20
                  kurtosis                                                kurtosis                                               kurtosis


             3x3                                                     7x7                                                 15x15
            data (ICAā€™d):                                      sphericalized:                                     factorialized:

ā€¢ Histograms, kurtosis of projections of image blocks onto random
unit-norm basis functions.
ā€¢ These imply data are closer to spherical than factorial
                                                                                                                  [Lyu & Simoncelli 08]
non-Gaussian elliptical observations
and models of natural images:

             - Zetzsche & Krieger, 1999;
             - Huang & Mumford, 1999;
             - Wainwright & Simoncelli, 2000;
             - HyvƤrinen and Hoyer, 2000;
             - Parra et al., 2001;
             - Srivastava et al., 2002;
             - Sendur & Selesnick, 2002;
             - Teh et al., 2003;
             - Gehler and Welling, 2006
             - Lyu & Simoncelli, 2008
             - etc.
Modeling heteroscedasticity
Method 2: Hidden scaling variable for each patch
Gaussian scale mixture (GSM)
[Andrews & Mallows 74]:
                               āˆš
                          x=       zu

ā€¢ u is Gaussian, z > 0
ā€¢ z and u are independent
ā€¢ x is elliptically symmetric, with covariance āˆ Cu
ā€¢ marginals of x are leptokurtotic
                                        [Wainwright&Simoncelli 99]
GSM - prior on z
ā€¢ Empirically, z is approximately lognormal
  [Portilla etal, icip-01]


                        exp (āˆ’(log z āˆ’ Āµl )2 /(2Ļƒl ))
                                                 2
               pz (z) =               2 )1/2
                               z(2Ļ€Ļƒl

ā€¢ Alternatively, can use Jeffreyā€™s noninformative prior
  [Figueiredo&Nowak, ā€˜01; Portilla etal, ā€˜03]


                             pz (z) āˆ 1/z
GSM simulation
      Image data           GSM simulation
 !                      !
#"                    #"




 "                      "
#"                    #"
     !!"    "   !"          !!"    "     !"




                     [Wainwright & Simoncelli, NIPS*99]
Model III (GSM)
Coefficient density:       Basis set:   Image:


          X            X



          X            X


sqrt(z)
          X            X


                 u
āˆš
           Original coefļ¬cients                              Normalized by                                    z
                                !2
                                                                                    !4

                                !4                                                  !5

marginal


              Log probability




                                                                 Log probability
                                                                                    !6
                                !6
                                                                                    !7
                                                                                                                   [Ruderman&Bialek 94]
                                !8                                                  !8

                                                                                    !9
                            !10
                                      !500   0   500                               !10
                                                                                     !5           0       5


            100                                              8


             50                                              6


   joint      0                                              4                                                    [Schwartz&Simoncelli 01]
           !50                                               2


           !100                                              0
             !100                    !50     0   50    100       0                        2   4       6   8




subband
6
Model Encoding Cost (bits/coeff)




                                                                                          Model Encoding cost (bits/coeff)
                                   5.5

                                                                                                                             5
                                    5


                                   4.5                                                                                       4

                                    4
                                                                                                                             3
                                   3.5


                                    3                                                                                        2
                                                     Gaussian Model                                                                                First Order Ideal
                                   2.5               Generalized Laplacian                                                                         Conditional Model

                                                                                                                             1
                                         3              4             5                                                       1   2         3          4         5     6
                                             Empirical First Order Entropy (bits/coeff)                                               Empirical Conditional Entropy




                                                                                                                                      [Buccigrossi & Simoncelli 99]
Bayesian denoising
ā€¢ Additive Gaussian noise:
            y =x+w
                                 2      2
      P (y|x) āˆ exp[āˆ’(y āˆ’ x)         /2Ļƒw ]


ā€¢ Bayesā€™ least squares solution is conditional mean:
        x(y) = IE(x|y)
        Ė†
              =     dxP(y|x)P(x)x/P(y)
I. Classical

If signal is Gaussian, BLS estimator is linear:




                          denoised (Ė†)
                                    x
             2
            Ļƒx
x(y) =
Ė†         2    2
                    Ā·y
         Ļƒx + Ļƒn


=> suppress ļ¬ne scales,                  noisy (y)
   retain coarse scales
Non-Gaussian coefļ¬cients
                 "
                #"
                                               -*./01.*,6'.)07+48
                                               94:..'41,;*1.')5,,
  2+0343'(')5



                 !%
                #"




                 !$
                #"
                 !!""                   "                           !""
                                &'()*+,-*./01.*



                        [Burt&Adelson ā€˜81; Field ā€˜87; Mallat ā€˜89; Daugman ā€˜89; etc]
II. BLS for non-Gaussian prior
ā€¢ Assume marginal distribution   [Mallat ā€˜89]:


             P (x) āˆ exp āˆ’|x/s|   p



ā€¢ Then Bayes estimator is generally nonlinear:



   p = 2.0            p = 1.0                 p = 0.5
                                      [Simoncelli & Adelson, ā€˜96]
MAP shrinkage




p=2.0        p=1.0      p=0.5




                        [Simoncelli 99]
Denoising: Joint
             IE(x|y) = dz P(z|y) IE(x|y, z)
                                                            āˆ’1y ļ£»
                                            ļ£®                    ļ£¹

                       = dz P(z|y)          ļ£°
                                             zCu(zCu + Cw )
                                                                 ctr


where
         P(y|z) P(z)                         exp(āˆ’y T (zCu + Cw )āˆ’1y/2)
P(z|y) =             ,              P(y|z) =
             Py                                   (2Ļ€)N |zCu + Cw |

Numerical computation of solution is reasonably efļ¬cient if
one jointly diagonalizes Cu and Cw ...
[Portilla, Strela, Wainwright, Simoncelli, ā€™03]


IPAM, 9/04                                                             20
ESTIMATED COEFF.     Example estimators

                                                       !"




                                    +'1&2/1+3)*%+,,-
                                                        "
                           !w

                                                       !!"
                                                        #"
                                                                                          !"
                                                             "
                                                                                "
                    NOISY COEFF.                                 !#"
                                   $%&'()./0+$1                        !!"     $%&'()*%+,,-




Estimators for the scalar and single-neighbor cases
                                                                             [Portilla etal 03]
Comparison to other methods
                    "'&                                         "'&
                     "                                           "
                                                                           ,456748+91:;<=
/456,(74-.)/-0123




                !"'&                                           !"'&        :>6965#8*>?6<
                    !!                                          !!
                !!'&                                           !!'&
                    !#                                          !#
                !#'&                                           !#'&
                                        )89!:1:;<=>?
                    !$                  .=9@A?-9?=@BC8D         !$
                !$'&                                           !$'&
                          !"    #"     $"      %"         &"          !"        #"     $"      %"           &"
                               ()*+,-*.)/-0123                                 ()*+,-*.)/-0123


                               Results averaged over 3 images
                                                                                       [Portilla etal 03]
Noisy
Original
           (22.1 dB)




Matlabā€™s
           BLS-GSM
wiener2
           (30.5 dB)
(28 dB)
Noisy
 Original
             (8.1 dB)




UndWvlt
            BLS-GSM
  Thresh
            (21.2 dB)
(19.0 dB)
Real sensor noise




400 ISO       denoised
GSM summary
ā€¢ GSM captures local variance
ā€¢ Underlying Gaussian leads to simple computation
ā€¢ Excellent denoising results
ā€¢ Whatā€™s missing?
  ā€¢ Global model of z variables [Wainwright etal 99;
    Romberg etal ā€˜99; Hyvarinen/Hoyer ā€˜02; Karklin/
    Lewicki ā€˜02; Lyu/Simoncelli 08]

  ā€¢ Explicit geometry: phase and orientation
Global models for z
ā€¢ Non-overlapping neighborhoods, tree-structured z
     [Wainwright etal 99; Romberg etal ā€™99]



                          z


             u
                                              Coarse scale

                          Fine scale




ā€¢ Field of GSMs: z is an exponentiated GMRF, u is
  a GMRF, subband is the product
  [Lyu&Simoncelli 08]
D MACHINE INTELLIGENCE, VOL. X, NO. X, XX 200X                                                                9

                             State-of-the-art denoising
                                  Lena                                            Boats
                "                                               "


                #                                               #


               !"                                              !"
      āˆ†()*+,




                                                      āˆ†()*+,
               !$                                              !$


               !'                                              !'


               !&                                              !&

"##                 ! "#"!   $!     !#    %!    "##                 ! "#"!   $!      !#      %!      "##
                                     Ļƒ                                                Ļƒ

                                  FoGSM               BM3D                          kSVD
thods for three diļ¬€erent images. Plotted are diļ¬€erences in PSNR for diļ¬€erent input noise levels (Ļƒ) between
   BLS-GSM [17], kSVD [39] and FoE [27]). The PSNR values for these methods were taken from
                                                      GSM                           FoE
                                                                                  [Lyu&Simoncelli, PAMI 08]
Measuring
 Orientation




2-band steerable pyramid: Image decomposition in
terms of multi-scale gradient measurements
                    [Simoncelli et.al., 1992; Simoncelli & Freeman 1995]
Multi-scale gradient basis
Multi-scale gradient basis
ā€¢ Multi-scale bases: efļ¬cient representation
Multi-scale gradient basis
ā€¢ Multi-scale bases: efļ¬cient representation
ā€¢ Derivatives: good for analysis
  ā€¢ Local Taylor expansion of image structures
  ā€¢ Explicit geometry (orientation)
Multi-scale gradient basis
ā€¢ Multi-scale bases: efļ¬cient representation
ā€¢ Derivatives: good for analysis
  ā€¢ Local Taylor expansion of image structures
  ā€¢ Explicit geometry (orientation)
ā€¢ Combination:
  ā€¢ Explicit incorporation of geometry in basis
  ā€¢ Bridge between PDE / harmonic analysis
     approaches
orientation




 magnitude

 orientation


[Hammond&Simoncelli 06; cf. Oppenheim and Lim 81]
Importance of local orientation
 Randomized orientation   Randomized magnitude




                                [Hammond&Simoncelli 05]
Reconstruction from orientation
          Original                Quantized to 2 bits




 ā€¢ Reconstruction by projections onto convex sets
 ā€¢ Resilient to quantization
                                        [Hammond&Simoncelli 06]
Image patches related by rotation




                               two-band steerable
[Hammond&Simoncelli 06]        pyramid coefficients
raw     rotated
patches   patches
                    PCA of normalized
                    gradient patches



                         --- Raw Patches
                             Rotated Patches




                         [Hammond&Simoncelli 06]
Orientation-Adaptive GSM model

Model a vectorized patch of wavelet coefficients as:




                 patch rotation operator

                 hidden magnitude/orientation variables




                                           [Hammond&Simoncelli 06]
Orientation-Adaptive GSM model

Model a vectorized patch of wavelet coefficients as:




                 patch rotation operator

                 hidden magnitude/orientation variables

Conditioned on       ;    is zero mean gaussian with covariance



                                           [Hammond&Simoncelli 06]
Estimation of C(Īø) from noisy data
           noisy patch


    unknown, approximate by           measured from noisy data.
Assuming                 independent and noise rotationally invariant




      (assuming w.l.o.g. E[z] =1 )

                                               [Hammond&Simoncelli 06]
Bayesian MMSE Estimator




                   [Hammond&Simoncelli 06]
Bayesian MMSE Estimator
           condition on and integrate
           over hidden variables




                       [Hammond&Simoncelli 06]
Bayesian MMSE Estimator
           condition on and integrate
           over hidden variables




                       [Hammond&Simoncelli 06]
Bayesian MMSE Estimator
           condition on and integrate
           over hidden variables




           Wiener estimate




                       [Hammond&Simoncelli 06]
Bayesian MMSE Estimator
                      condition on and integrate
                      over hidden variables




                      Wiener estimate


  has covariance


separable prior for
 hidden variables

                                  [Hammond&Simoncelli 06]
Bayesian MMSE Estimator
                      condition on and integrate
                      over hidden variables




                      Wiener estimate


  has covariance


separable prior for
 hidden variables

                                  [Hammond&Simoncelli 06]
Ļƒ = 40

          noisy
          2.81 dB




  gsm2    oagsm
12.4 dB   13.1 dB
Locally adaptive covariance
 ā€¢ Karklin & Lewicki 08: Each patch is Gaussian,
   with covariance constructed from a weighted outer-
   product of ļ¬xed vectors:
 p(x) = G (x; C(y))          log C(y) =         yn Bn
                                            n
                                                      T
p(y) =       exp(āˆ’|yn |)      Bn =        wnk bk bk
         n                            k

 ā€¢ Guerrero-Colon, Simoncelli & Portilla 08: Each
   patch is a mixture of GSMs (MGSMs):

     p(x) =        Pk      p(zk ) G(x; zk Ck ) dzk
               k
MGSMs generative model
                           āˆš         āˆš   āˆš
Patch x chosen from { z1 u1 , z2 u2 , ... zK uK }

     with probabilities {P1 , P2 , ..., PK }
Parameters:
             ā€¢ Covariances      Ck
          ā€¢ Scale densities     pk (zk )
  ā€¢ Component probabilities     Pk
   ā€¢ Number of components       K

Parameters can be ļ¬t to data of one or more images
by maximizing likelihood (EM-like)
                               [Guerrero-Colon, Simoncelli, Portilla 08]
MGSM ā€œsegmentationā€
       image    1         2                 4




    First six
eigenvectors
    of GSM
  covariance
    matrices
                    [Guerrero-Colon, Simoncelli, Portilla 08]
MGSM
ā€œsegmentationā€
Eigenvectors of GSM
components represent
 invariant subspaces:
ā€œgeneralized complex
                cellsā€
Potential of local homogeneous
               models?
Consider an implicit model:
 maxEnt subject to constraints on subband coefļ¬cients:

      ā€¢ marginal statistics [var,skew,kurtosis]
      ā€¢ local raw correlations
      ā€¢ local variance correlations
      ā€¢ local phase correlations
                                     [Portilla & Simoncelli 00;
                                      cf. Zhu, Wu & Mumford 97]
Visual texture
Visual texture




Homogeneous, with repeated structures
Visual texture




Homogeneous, with repeated structures
     ā€œYou know it when you see itā€
All Images


    Texture Images




             Equivalence class (visually indistinguishable)
Iterative synthesis algorithm
           Analysis


Example        Transform     Measure
 Texture                     Statistics




           Synthesis        Measure
                            Statistics



Random         Transform                           Inverse     Synthesized
                                          Adjust
   Seed                                            Transform   Texture




                           [Portilla&Simoncelli 00; cf. Heeger&Bergen ā€˜95]
Examples: Artiļ¬cial
Photographic, quasi-periodic
Photographic, aperiodic
Photographic, structured
Photographic, color
Non-textures?
Texture mixtures
Texture mixtures




Convex combinations in parameter space
Texture mixtures




Convex combinations in parameter space
=> Parameter space includes non-textures
Summary
ā€¢ Fusion of empirical data with structural principles
ā€¢ Statistical models have led to state-of-the-art image
  processing, and are relevant for biological vision

ā€¢ Local adaptation to {variance, orientation,
  phase, ...} gives improvement, but makes learning
  harder

ā€¢ Cascaded representations emerge naturally
ā€¢ Thereā€™s still much room for improvement!
Cast
ā€¢ Local GSM model: Martin Wainwright, Javier Portilla
ā€¢ GSM Denoising: Javier Portilla, Martin Wainwright,
  Vasily Strela

ā€¢ Variance-adaptive compression: Robert Buccigrossi
ā€¢ Local orientation and OAGSM: David Hammond
ā€¢ Field of GSMs: Siwei Lyu
ā€¢ Mixture of GSMs: Jose-Antonio Guerrero-ColĆ³n,
  Javier Portilla

ā€¢ Texture representation/synthesis: Javier Portilla

More Related Content

What's hot

Modern features-part-1-detectors
Modern features-part-1-detectorsModern features-part-1-detectors
Modern features-part-1-detectorszukun
Ā 
Lesson 12: Linear Approximation
Lesson 12: Linear ApproximationLesson 12: Linear Approximation
Lesson 12: Linear ApproximationMatthew Leingang
Ā 
CVPR2010: Advanced ITinCVPR in a Nutshell: part 4: additional slides
CVPR2010: Advanced ITinCVPR in a Nutshell: part 4: additional slidesCVPR2010: Advanced ITinCVPR in a Nutshell: part 4: additional slides
CVPR2010: Advanced ITinCVPR in a Nutshell: part 4: additional slideszukun
Ā 
Note on Coupled Line Cameras for Rectangle Reconstruction (ACDDE 2012)
Note on Coupled Line Cameras for Rectangle Reconstruction (ACDDE 2012)Note on Coupled Line Cameras for Rectangle Reconstruction (ACDDE 2012)
Note on Coupled Line Cameras for Rectangle Reconstruction (ACDDE 2012)Joo-Haeng Lee
Ā 
02 probabilistic inference in graphical models
02 probabilistic inference in graphical models02 probabilistic inference in graphical models
02 probabilistic inference in graphical modelszukun
Ā 
05 history of cv a machine learning (theory) perspective on computer vision
05  history of cv a machine learning (theory) perspective on computer vision05  history of cv a machine learning (theory) perspective on computer vision
05 history of cv a machine learning (theory) perspective on computer visionzukun
Ā 
Lesson 16: Inverse Trigonometric Functions
Lesson 16: Inverse Trigonometric FunctionsLesson 16: Inverse Trigonometric Functions
Lesson 16: Inverse Trigonometric FunctionsMatthew Leingang
Ā 
FABIA: Large Data Biclustering in Drug Design
FABIA: Large Data Biclustering in Drug DesignFABIA: Large Data Biclustering in Drug Design
FABIA: Large Data Biclustering in Drug DesignMartin Heusel
Ā 
05 Random Variables
05 Random Variables05 Random Variables
05 Random VariablesHadley Wickham
Ā 
Kccsi 2012 a real-time robust object tracking-v2
Kccsi 2012   a real-time robust object tracking-v2Kccsi 2012   a real-time robust object tracking-v2
Kccsi 2012 a real-time robust object tracking-v2Prarinya Siritanawan
Ā 
IGARSS_AMASM_woo_20110727.pdf
IGARSS_AMASM_woo_20110727.pdfIGARSS_AMASM_woo_20110727.pdf
IGARSS_AMASM_woo_20110727.pdfgrssieee
Ā 
Lesson 3: Limit Laws
Lesson 3: Limit LawsLesson 3: Limit Laws
Lesson 3: Limit LawsMatthew Leingang
Ā 
IGARSS2011.pdf
IGARSS2011.pdfIGARSS2011.pdf
IGARSS2011.pdfgrssieee
Ā 
pres_IGARSS_2011_LFF_poltom.pdf
pres_IGARSS_2011_LFF_poltom.pdfpres_IGARSS_2011_LFF_poltom.pdf
pres_IGARSS_2011_LFF_poltom.pdfgrssieee
Ā 

What's hot (15)

Modern features-part-1-detectors
Modern features-part-1-detectorsModern features-part-1-detectors
Modern features-part-1-detectors
Ā 
Lesson 12: Linear Approximation
Lesson 12: Linear ApproximationLesson 12: Linear Approximation
Lesson 12: Linear Approximation
Ā 
CVPR2010: Advanced ITinCVPR in a Nutshell: part 4: additional slides
CVPR2010: Advanced ITinCVPR in a Nutshell: part 4: additional slidesCVPR2010: Advanced ITinCVPR in a Nutshell: part 4: additional slides
CVPR2010: Advanced ITinCVPR in a Nutshell: part 4: additional slides
Ā 
Note on Coupled Line Cameras for Rectangle Reconstruction (ACDDE 2012)
Note on Coupled Line Cameras for Rectangle Reconstruction (ACDDE 2012)Note on Coupled Line Cameras for Rectangle Reconstruction (ACDDE 2012)
Note on Coupled Line Cameras for Rectangle Reconstruction (ACDDE 2012)
Ā 
02 probabilistic inference in graphical models
02 probabilistic inference in graphical models02 probabilistic inference in graphical models
02 probabilistic inference in graphical models
Ā 
Lecture9
Lecture9Lecture9
Lecture9
Ā 
05 history of cv a machine learning (theory) perspective on computer vision
05  history of cv a machine learning (theory) perspective on computer vision05  history of cv a machine learning (theory) perspective on computer vision
05 history of cv a machine learning (theory) perspective on computer vision
Ā 
Lesson 16: Inverse Trigonometric Functions
Lesson 16: Inverse Trigonometric FunctionsLesson 16: Inverse Trigonometric Functions
Lesson 16: Inverse Trigonometric Functions
Ā 
FABIA: Large Data Biclustering in Drug Design
FABIA: Large Data Biclustering in Drug DesignFABIA: Large Data Biclustering in Drug Design
FABIA: Large Data Biclustering in Drug Design
Ā 
05 Random Variables
05 Random Variables05 Random Variables
05 Random Variables
Ā 
Kccsi 2012 a real-time robust object tracking-v2
Kccsi 2012   a real-time robust object tracking-v2Kccsi 2012   a real-time robust object tracking-v2
Kccsi 2012 a real-time robust object tracking-v2
Ā 
IGARSS_AMASM_woo_20110727.pdf
IGARSS_AMASM_woo_20110727.pdfIGARSS_AMASM_woo_20110727.pdf
IGARSS_AMASM_woo_20110727.pdf
Ā 
Lesson 3: Limit Laws
Lesson 3: Limit LawsLesson 3: Limit Laws
Lesson 3: Limit Laws
Ā 
IGARSS2011.pdf
IGARSS2011.pdfIGARSS2011.pdf
IGARSS2011.pdf
Ā 
pres_IGARSS_2011_LFF_poltom.pdf
pres_IGARSS_2011_LFF_poltom.pdfpres_IGARSS_2011_LFF_poltom.pdf
pres_IGARSS_2011_LFF_poltom.pdf
Ā 

Viewers also liked

CVPR2010: Learnings from founding a computer vision startup: Chapter 8: Softw...
CVPR2010: Learnings from founding a computer vision startup: Chapter 8: Softw...CVPR2010: Learnings from founding a computer vision startup: Chapter 8: Softw...
CVPR2010: Learnings from founding a computer vision startup: Chapter 8: Softw...zukun
Ā 
Tuto cvpr part1
Tuto cvpr part1Tuto cvpr part1
Tuto cvpr part1zukun
Ā 
CVPR2010: Learnings from founding a computer vision startup: Chapter 1, 2: Wh...
CVPR2010: Learnings from founding a computer vision startup: Chapter 1, 2: Wh...CVPR2010: Learnings from founding a computer vision startup: Chapter 1, 2: Wh...
CVPR2010: Learnings from founding a computer vision startup: Chapter 1, 2: Wh...zukun
Ā 
Iccv2009 recognition and learning object categories p2 c01 - recognizing a ...
Iccv2009 recognition and learning object categories   p2 c01 - recognizing a ...Iccv2009 recognition and learning object categories   p2 c01 - recognizing a ...
Iccv2009 recognition and learning object categories p2 c01 - recognizing a ...zukun
Ā 
NIPS2009: Understand Visual Scenes - Part 2
NIPS2009: Understand Visual Scenes - Part 2NIPS2009: Understand Visual Scenes - Part 2
NIPS2009: Understand Visual Scenes - Part 2zukun
Ā 
ECCV2010: distance function and metric learning part 2
ECCV2010: distance function and metric learning part 2ECCV2010: distance function and metric learning part 2
ECCV2010: distance function and metric learning part 2zukun
Ā 

Viewers also liked (6)

CVPR2010: Learnings from founding a computer vision startup: Chapter 8: Softw...
CVPR2010: Learnings from founding a computer vision startup: Chapter 8: Softw...CVPR2010: Learnings from founding a computer vision startup: Chapter 8: Softw...
CVPR2010: Learnings from founding a computer vision startup: Chapter 8: Softw...
Ā 
Tuto cvpr part1
Tuto cvpr part1Tuto cvpr part1
Tuto cvpr part1
Ā 
CVPR2010: Learnings from founding a computer vision startup: Chapter 1, 2: Wh...
CVPR2010: Learnings from founding a computer vision startup: Chapter 1, 2: Wh...CVPR2010: Learnings from founding a computer vision startup: Chapter 1, 2: Wh...
CVPR2010: Learnings from founding a computer vision startup: Chapter 1, 2: Wh...
Ā 
Iccv2009 recognition and learning object categories p2 c01 - recognizing a ...
Iccv2009 recognition and learning object categories   p2 c01 - recognizing a ...Iccv2009 recognition and learning object categories   p2 c01 - recognizing a ...
Iccv2009 recognition and learning object categories p2 c01 - recognizing a ...
Ā 
NIPS2009: Understand Visual Scenes - Part 2
NIPS2009: Understand Visual Scenes - Part 2NIPS2009: Understand Visual Scenes - Part 2
NIPS2009: Understand Visual Scenes - Part 2
Ā 
ECCV2010: distance function and metric learning part 2
ECCV2010: distance function and metric learning part 2ECCV2010: distance function and metric learning part 2
ECCV2010: distance function and metric learning part 2
Ā 

Similar to NIPS2008: tutorial: statistical models of visual images

Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learningkkkc
Ā 
20100822 computervision veksler
20100822 computervision veksler20100822 computervision veksler
20100822 computervision vekslerComputer Science Club
Ā 
Online Signal Processing Assignment Help
Online Signal Processing Assignment HelpOnline Signal Processing Assignment Help
Online Signal Processing Assignment HelpMatlab Assignment Experts
Ā 
Robust Super-Resolution by minimizing a Gaussian-weighted L2 error norm
Robust Super-Resolution by minimizing a Gaussian-weighted L2 error normRobust Super-Resolution by minimizing a Gaussian-weighted L2 error norm
Robust Super-Resolution by minimizing a Gaussian-weighted L2 error normTuan Q. Pham
Ā 
Johan Suykens: "Models from Data: a Unifying Picture"
Johan Suykens: "Models from Data: a Unifying Picture" Johan Suykens: "Models from Data: a Unifying Picture"
Johan Suykens: "Models from Data: a Unifying Picture" ieee_cis_cyprus
Ā 
Sparse codes for natural images
Sparse codes for natural imagesSparse codes for natural images
Sparse codes for natural imagesJorge Leandro, Ph.D.
Ā 
von Mises lecture, Berlin
von Mises lecture, Berlinvon Mises lecture, Berlin
von Mises lecture, BerlinChristian Robert
Ā 
CVPR2009 tutorial: Kernel Methods in Computer Vision: part I: Introduction to...
CVPR2009 tutorial: Kernel Methods in Computer Vision: part I: Introduction to...CVPR2009 tutorial: Kernel Methods in Computer Vision: part I: Introduction to...
CVPR2009 tutorial: Kernel Methods in Computer Vision: part I: Introduction to...zukun
Ā 
Fishermatrix extended ctics_reloaded
Fishermatrix extended ctics_reloadedFishermatrix extended ctics_reloaded
Fishermatrix extended ctics_reloadedcapetownreloaded
Ā 
16 17 bag_words
16 17 bag_words16 17 bag_words
16 17 bag_wordskhawarbashir
Ā 
Biao Hou--SAR IMAGE DESPECKLING BASED ON IMPROVED DIRECTIONLET DOMAIN GAUSSIA...
Biao Hou--SAR IMAGE DESPECKLING BASED ON IMPROVED DIRECTIONLET DOMAIN GAUSSIA...Biao Hou--SAR IMAGE DESPECKLING BASED ON IMPROVED DIRECTIONLET DOMAIN GAUSSIA...
Biao Hou--SAR IMAGE DESPECKLING BASED ON IMPROVED DIRECTIONLET DOMAIN GAUSSIA...grssieee
Ā 
Introduction to Artificial Intelligence
Introduction to Artificial IntelligenceIntroduction to Artificial Intelligence
Introduction to Artificial IntelligenceManoj Harsule
Ā 
MLIP - Chapter 6 - Generation, Super-Resolution, Style transfer
MLIP - Chapter 6 - Generation, Super-Resolution, Style transferMLIP - Chapter 6 - Generation, Super-Resolution, Style transfer
MLIP - Chapter 6 - Generation, Super-Resolution, Style transferCharles Deledalle
Ā 
Biased normalized cuts
Biased normalized cutsBiased normalized cuts
Biased normalized cutsirisshicat
Ā 
Rethinking of Generalization
Rethinking of GeneralizationRethinking of Generalization
Rethinking of GeneralizationHikaru Ibayashi
Ā 
Digital Image Processing - Image Restoration
Digital Image Processing - Image RestorationDigital Image Processing - Image Restoration
Digital Image Processing - Image RestorationMathankumar S
Ā 

Similar to NIPS2008: tutorial: statistical models of visual images (20)

Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
Ā 
Berlin
BerlinBerlin
Berlin
Ā 
20100822 computervision veksler
20100822 computervision veksler20100822 computervision veksler
20100822 computervision veksler
Ā 
Online Signal Processing Assignment Help
Online Signal Processing Assignment HelpOnline Signal Processing Assignment Help
Online Signal Processing Assignment Help
Ā 
Robust Super-Resolution by minimizing a Gaussian-weighted L2 error norm
Robust Super-Resolution by minimizing a Gaussian-weighted L2 error normRobust Super-Resolution by minimizing a Gaussian-weighted L2 error norm
Robust Super-Resolution by minimizing a Gaussian-weighted L2 error norm
Ā 
Johan Suykens: "Models from Data: a Unifying Picture"
Johan Suykens: "Models from Data: a Unifying Picture" Johan Suykens: "Models from Data: a Unifying Picture"
Johan Suykens: "Models from Data: a Unifying Picture"
Ā 
Sparse codes for natural images
Sparse codes for natural imagesSparse codes for natural images
Sparse codes for natural images
Ā 
von Mises lecture, Berlin
von Mises lecture, Berlinvon Mises lecture, Berlin
von Mises lecture, Berlin
Ā 
CVPR2009 tutorial: Kernel Methods in Computer Vision: part I: Introduction to...
CVPR2009 tutorial: Kernel Methods in Computer Vision: part I: Introduction to...CVPR2009 tutorial: Kernel Methods in Computer Vision: part I: Introduction to...
CVPR2009 tutorial: Kernel Methods in Computer Vision: part I: Introduction to...
Ā 
Fishermatrix extended ctics_reloaded
Fishermatrix extended ctics_reloadedFishermatrix extended ctics_reloaded
Fishermatrix extended ctics_reloaded
Ā 
16 17 bag_words
16 17 bag_words16 17 bag_words
16 17 bag_words
Ā 
Biao Hou--SAR IMAGE DESPECKLING BASED ON IMPROVED DIRECTIONLET DOMAIN GAUSSIA...
Biao Hou--SAR IMAGE DESPECKLING BASED ON IMPROVED DIRECTIONLET DOMAIN GAUSSIA...Biao Hou--SAR IMAGE DESPECKLING BASED ON IMPROVED DIRECTIONLET DOMAIN GAUSSIA...
Biao Hou--SAR IMAGE DESPECKLING BASED ON IMPROVED DIRECTIONLET DOMAIN GAUSSIA...
Ā 
Introduction to Artificial Intelligence
Introduction to Artificial IntelligenceIntroduction to Artificial Intelligence
Introduction to Artificial Intelligence
Ā 
Mathematical tools in dip
Mathematical tools in dipMathematical tools in dip
Mathematical tools in dip
Ā 
MLIP - Chapter 6 - Generation, Super-Resolution, Style transfer
MLIP - Chapter 6 - Generation, Super-Resolution, Style transferMLIP - Chapter 6 - Generation, Super-Resolution, Style transfer
MLIP - Chapter 6 - Generation, Super-Resolution, Style transfer
Ā 
Biased normalized cuts
Biased normalized cutsBiased normalized cuts
Biased normalized cuts
Ā 
Rethinking of Generalization
Rethinking of GeneralizationRethinking of Generalization
Rethinking of Generalization
Ā 
Digital Image Processing - Image Restoration
Digital Image Processing - Image RestorationDigital Image Processing - Image Restoration
Digital Image Processing - Image Restoration
Ā 
Edges and lines
Edges and linesEdges and lines
Edges and lines
Ā 
Lecture 5
Lecture 5Lecture 5
Lecture 5
Ā 

More from zukun

My lyn tutorial 2009
My lyn tutorial 2009My lyn tutorial 2009
My lyn tutorial 2009zukun
Ā 
ETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVzukun
Ā 
ETHZ CV2012: Information
ETHZ CV2012: InformationETHZ CV2012: Information
ETHZ CV2012: Informationzukun
Ā 
Siwei lyu: natural image statistics
Siwei lyu: natural image statisticsSiwei lyu: natural image statistics
Siwei lyu: natural image statisticszukun
Ā 
Lecture9 camera calibration
Lecture9 camera calibrationLecture9 camera calibration
Lecture9 camera calibrationzukun
Ā 
Brunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionBrunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionzukun
Ā 
Modern features-part-4-evaluation
Modern features-part-4-evaluationModern features-part-4-evaluation
Modern features-part-4-evaluationzukun
Ā 
Modern features-part-2-descriptors
Modern features-part-2-descriptorsModern features-part-2-descriptors
Modern features-part-2-descriptorszukun
Ā 
Modern features-part-0-intro
Modern features-part-0-introModern features-part-0-intro
Modern features-part-0-introzukun
Ā 
Lecture 02 internet video search
Lecture 02 internet video searchLecture 02 internet video search
Lecture 02 internet video searchzukun
Ā 
Lecture 01 internet video search
Lecture 01 internet video searchLecture 01 internet video search
Lecture 01 internet video searchzukun
Ā 
Lecture 03 internet video search
Lecture 03 internet video searchLecture 03 internet video search
Lecture 03 internet video searchzukun
Ā 
Icml2012 tutorial representation_learning
Icml2012 tutorial representation_learningIcml2012 tutorial representation_learning
Icml2012 tutorial representation_learningzukun
Ā 
Advances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionAdvances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionzukun
Ā 
Gephi tutorial: quick start
Gephi tutorial: quick startGephi tutorial: quick start
Gephi tutorial: quick startzukun
Ā 
EM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysisEM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysiszukun
Ā 
Object recognition with pictorial structures
Object recognition with pictorial structuresObject recognition with pictorial structures
Object recognition with pictorial structureszukun
Ā 
Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities zukun
Ā 
Icml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featuresIcml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featureszukun
Ā 
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...zukun
Ā 

More from zukun (20)

My lyn tutorial 2009
My lyn tutorial 2009My lyn tutorial 2009
My lyn tutorial 2009
Ā 
ETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCV
Ā 
ETHZ CV2012: Information
ETHZ CV2012: InformationETHZ CV2012: Information
ETHZ CV2012: Information
Ā 
Siwei lyu: natural image statistics
Siwei lyu: natural image statisticsSiwei lyu: natural image statistics
Siwei lyu: natural image statistics
Ā 
Lecture9 camera calibration
Lecture9 camera calibrationLecture9 camera calibration
Lecture9 camera calibration
Ā 
Brunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionBrunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer vision
Ā 
Modern features-part-4-evaluation
Modern features-part-4-evaluationModern features-part-4-evaluation
Modern features-part-4-evaluation
Ā 
Modern features-part-2-descriptors
Modern features-part-2-descriptorsModern features-part-2-descriptors
Modern features-part-2-descriptors
Ā 
Modern features-part-0-intro
Modern features-part-0-introModern features-part-0-intro
Modern features-part-0-intro
Ā 
Lecture 02 internet video search
Lecture 02 internet video searchLecture 02 internet video search
Lecture 02 internet video search
Ā 
Lecture 01 internet video search
Lecture 01 internet video searchLecture 01 internet video search
Lecture 01 internet video search
Ā 
Lecture 03 internet video search
Lecture 03 internet video searchLecture 03 internet video search
Lecture 03 internet video search
Ā 
Icml2012 tutorial representation_learning
Icml2012 tutorial representation_learningIcml2012 tutorial representation_learning
Icml2012 tutorial representation_learning
Ā 
Advances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionAdvances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer vision
Ā 
Gephi tutorial: quick start
Gephi tutorial: quick startGephi tutorial: quick start
Gephi tutorial: quick start
Ā 
EM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysisEM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysis
Ā 
Object recognition with pictorial structures
Object recognition with pictorial structuresObject recognition with pictorial structures
Object recognition with pictorial structures
Ā 
Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities
Ā 
Icml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featuresIcml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant features
Ā 
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
ECCV2010: Modeling Temporal Structure of Decomposable Motion Segments for Act...
Ā 

Recently uploaded

Introduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher EducationIntroduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher Educationpboyjonauth
Ā 
Micromeritics - Fundamental and Derived Properties of Powders
Micromeritics - Fundamental and Derived Properties of PowdersMicromeritics - Fundamental and Derived Properties of Powders
Micromeritics - Fundamental and Derived Properties of PowdersChitralekhaTherkar
Ā 
PSYCHIATRIC History collection FORMAT.pptx
PSYCHIATRIC   History collection FORMAT.pptxPSYCHIATRIC   History collection FORMAT.pptx
PSYCHIATRIC History collection FORMAT.pptxPoojaSen20
Ā 
Sanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfSanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfsanyamsingh5019
Ā 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeThiyagu K
Ā 
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Krashi Coaching
Ā 
Accessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactAccessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactdawncurless
Ā 
Mastering the Unannounced Regulatory Inspection
Mastering the Unannounced Regulatory InspectionMastering the Unannounced Regulatory Inspection
Mastering the Unannounced Regulatory InspectionSafetyChain Software
Ā 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxheathfieldcps1
Ā 
Q4-W6-Restating Informational Text Grade 3
Q4-W6-Restating Informational Text Grade 3Q4-W6-Restating Informational Text Grade 3
Q4-W6-Restating Informational Text Grade 3JemimahLaneBuaron
Ā 
A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformChameera Dedduwage
Ā 
URLs and Routing in the Odoo 17 Website App
URLs and Routing in the Odoo 17 Website AppURLs and Routing in the Odoo 17 Website App
URLs and Routing in the Odoo 17 Website AppCeline George
Ā 
Alper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentAlper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentInMediaRes1
Ā 
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Sapana Sha
Ā 
Industrial Policy - 1948, 1956, 1973, 1977, 1980, 1991
Industrial Policy - 1948, 1956, 1973, 1977, 1980, 1991Industrial Policy - 1948, 1956, 1973, 1977, 1980, 1991
Industrial Policy - 1948, 1956, 1973, 1977, 1980, 1991RKavithamani
Ā 
18-04-UA_REPORT_MEDIALITERAŠ”Y_INDEX-DM_23-1-final-eng.pdf
18-04-UA_REPORT_MEDIALITERAŠ”Y_INDEX-DM_23-1-final-eng.pdf18-04-UA_REPORT_MEDIALITERAŠ”Y_INDEX-DM_23-1-final-eng.pdf
18-04-UA_REPORT_MEDIALITERAŠ”Y_INDEX-DM_23-1-final-eng.pdfssuser54595a
Ā 
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxSOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxiammrhaywood
Ā 

Recently uploaded (20)

CĆ³digo Creativo y Arte de Software | Unidad 1
CĆ³digo Creativo y Arte de Software | Unidad 1CĆ³digo Creativo y Arte de Software | Unidad 1
CĆ³digo Creativo y Arte de Software | Unidad 1
Ā 
Introduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher EducationIntroduction to ArtificiaI Intelligence in Higher Education
Introduction to ArtificiaI Intelligence in Higher Education
Ā 
Micromeritics - Fundamental and Derived Properties of Powders
Micromeritics - Fundamental and Derived Properties of PowdersMicromeritics - Fundamental and Derived Properties of Powders
Micromeritics - Fundamental and Derived Properties of Powders
Ā 
PSYCHIATRIC History collection FORMAT.pptx
PSYCHIATRIC   History collection FORMAT.pptxPSYCHIATRIC   History collection FORMAT.pptx
PSYCHIATRIC History collection FORMAT.pptx
Ā 
Sanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfSanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdf
Ā 
Model Call Girl in Tilak Nagar Delhi reach out to us at šŸ”9953056974šŸ”
Model Call Girl in Tilak Nagar Delhi reach out to us at šŸ”9953056974šŸ”Model Call Girl in Tilak Nagar Delhi reach out to us at šŸ”9953056974šŸ”
Model Call Girl in Tilak Nagar Delhi reach out to us at šŸ”9953056974šŸ”
Ā 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and Mode
Ā 
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Ā 
Accessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactAccessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impact
Ā 
Mastering the Unannounced Regulatory Inspection
Mastering the Unannounced Regulatory InspectionMastering the Unannounced Regulatory Inspection
Mastering the Unannounced Regulatory Inspection
Ā 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptx
Ā 
Q4-W6-Restating Informational Text Grade 3
Q4-W6-Restating Informational Text Grade 3Q4-W6-Restating Informational Text Grade 3
Q4-W6-Restating Informational Text Grade 3
Ā 
A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy Reform
Ā 
URLs and Routing in the Odoo 17 Website App
URLs and Routing in the Odoo 17 Website AppURLs and Routing in the Odoo 17 Website App
URLs and Routing in the Odoo 17 Website App
Ā 
Alper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentAlper Gobel In Media Res Media Component
Alper Gobel In Media Res Media Component
Ā 
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Ā 
Model Call Girl in Bikash Puri Delhi reach out to us at šŸ”9953056974šŸ”
Model Call Girl in Bikash Puri  Delhi reach out to us at šŸ”9953056974šŸ”Model Call Girl in Bikash Puri  Delhi reach out to us at šŸ”9953056974šŸ”
Model Call Girl in Bikash Puri Delhi reach out to us at šŸ”9953056974šŸ”
Ā 
Industrial Policy - 1948, 1956, 1973, 1977, 1980, 1991
Industrial Policy - 1948, 1956, 1973, 1977, 1980, 1991Industrial Policy - 1948, 1956, 1973, 1977, 1980, 1991
Industrial Policy - 1948, 1956, 1973, 1977, 1980, 1991
Ā 
18-04-UA_REPORT_MEDIALITERAŠ”Y_INDEX-DM_23-1-final-eng.pdf
18-04-UA_REPORT_MEDIALITERAŠ”Y_INDEX-DM_23-1-final-eng.pdf18-04-UA_REPORT_MEDIALITERAŠ”Y_INDEX-DM_23-1-final-eng.pdf
18-04-UA_REPORT_MEDIALITERAŠ”Y_INDEX-DM_23-1-final-eng.pdf
Ā 
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxSOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
Ā 

NIPS2008: tutorial: statistical models of visual images

  • 1. Statistical Image Models Eero Simoncelli Howard Hughes Medical Institute, Center for Neural Science, and Courant Institute of Mathematical Sciences New York University
  • 2. Photographic Images Diverse specialized structures: ā€¢ edges/lines/contours ā€¢ shadows/highlights ā€¢ smooth regions ā€¢ textured regions
  • 3. Photographic Images Diverse specialized structures: ā€¢ edges/lines/contours ā€¢ shadows/highlights ā€¢ smooth regions ā€¢ textured regions Occupy a small region of the full space
  • 4. spa ce o f all ima ges typical images One could describe this set as a deterministic manifold....
  • 5.
  • 6.
  • 7. ā€¢ Step edges are rare (lighting, junctions, texture, noise)
  • 8. ā€¢ Step edges are rare (lighting, junctions, texture, noise) ā€¢ One scaleā€™s texture is another scaleā€™s edge
  • 9. ā€¢ Step edges are rare (lighting, junctions, texture, noise) ā€¢ One scaleā€™s texture is another scaleā€™s edge ā€¢ Need seamless transitions from isolated features to dense textures
  • 10. spa ce o f all ima ges typical images One could describe this set as a deterministic manifold....
  • 11. spa ce o f all ima ges typical images One could describe this set as a deterministic manifold.... But seems more natural to use probability
  • 12. spa ce o f all ima ges typical images One could describe this set as a P(x) deterministic manifold.... But seems more natural to use probability
  • 13. ā€œApplicationsā€ ā€¢ Engineering: compression, denoising, restoration, enhancement/modiļ¬cation, synthesis, manipulation [Hubel ā€˜95]
  • 14. ā€œApplicationsā€ ā€¢ Engineering: compression, denoising, restoration, enhancement/modiļ¬cation, synthesis, manipulation ā€¢ Science: optimality principles for neurobiology (evolution, development, learning, adaptation) [Hubel ā€˜95]
  • 15. Density models nonparametric parametric/ constrained
  • 16. Density models nonparametric parametric/ constrained build a histogram from lots of observations...
  • 17. Density models nonparametric parametric/ constrained build a histogram use ā€œnatural constraintsā€ from lots of (geometry/photometry observations... of image formation, computation, maxEnt)
  • 18. Density models nonparametric parametric/ historical trend constrained (technology driven) build a histogram use ā€œnatural constraintsā€ from lots of (geometry/photometry observations... of image formation, computation, maxEnt)
  • 19. histogram Original image Range: [0, 237] Dims: [256, 256] 0 50 100 150 200 250
  • 20. histogram Original image Range: [0, 237] Dims: [256, 256] 0 50 100 150 200 250 histogram Equalized image Range: [1.99, 238] Dims: [256, 256] 0 50 100 150 200 250
  • 21. histogram Original image Range: [0, 237] Dims: [256, 256] 0 50 100 150 200 250 histogram Equalized image Range: [1.99, 238] Dims: [256, 256] 0 50 100 150 200 250
  • 22. General methodology Observe ā€œinterestingā€ Transform to Joint Statistics Optimal Representation
  • 23. General methodology Observe ā€œinterestingā€ Transform to Joint Statistics Optimal Representation
  • 24. General methodology Observe ā€œinterestingā€ Transform to Joint Statistics Optimal Representation ā€œOnion peelingā€
  • 25. Evolution of image models I. (1950ā€™s): Fourier + Gaussian II. (mid 80ā€™s - late 90ā€™s): Wavelets + kurtotic marginals III. (mid 90ā€™s - present): Wavelets + local context ā€¢ local amplitude (contrast) ā€¢ local orientation IV. (last 5 years): Hierarchical models
  • 26. a. Pixel correlation b. 1 Correlation I(x+2,y) I(x+4,y) I(x+1,y) I(x,y) I(x,y) I(x,y) 0 10 Spatia
  • 27. a. Pixel correlation b. 1 Correlation I(x+2,y) I(x+4,y) I(x+1,y) I(x,y) I(x,y) I(x,y) 0 10 b. Spatia 1 Correlation I(x+4,y) I(x,y) 0 10 20 30 40 Spatial separation (pixels)
  • 29. Translation invariance Assuming translation invariance, => covariance matrix is Toeplitz (convolutional)
  • 30. Translation invariance Assuming translation invariance, => covariance matrix is Toeplitz (convolutional) => eigenvectors are sinusoids
  • 31. Translation invariance Assuming translation invariance, => covariance matrix is Toeplitz (convolutional) => eigenvectors are sinusoids => can diagonalize (decorrelate) with F.T.
  • 32. Translation invariance Assuming translation invariance, => covariance matrix is Toeplitz (convolutional) => eigenvectors are sinusoids => can diagonalize (decorrelate) with F.T. Power spectrum captures full covariance structure
  • 33. Spectral power Structural: Assume scale-invariance: F (sĻ‰) = s F (Ļ‰) p then: 1 F (Ļ‰) āˆ p Ļ‰ [Ritterman 52; DeRiugin 56; Field 87; Tolhurst 92; Ruderman/Bialek 94; ...]
  • 34. Spectral power Structural: Empirical: 6 Assume scale-invariance: 5 F (sĻ‰) = s F (Ļ‰) p 4 Log power 3 10 then: 2 1 F (Ļ‰) āˆ p Ļ‰ 1 0 0 1 2 3 Log spatialfrequency (cycles/image) 10 [Ritterman 52; DeRiugin 56; Field 87; Tolhurst 92; Ruderman/Bialek 94; ...]
  • 35. Principal Components Analysis (PCA) + whitening a. b. c. 20 20 4 -20 -20 -4 -20 20 -20 20 -4 4
  • 36. PCA basis for image blocks
  • 37. PCA basis for image blocks PCA is not unique
  • 38. Maximum entropy (maxEnt) The density with maximal entropy satisfying E (f (x)) = c is of the form pME (x) āˆ exp (āˆ’Ī»f (x)) where Ī» depends on c Examples: f (x) = x 2 f (x) = |x|
  • 39. Model I (Fourier/Gaussian) Coefficient density: Basis set: Image: : : : :
  • 40. Gaussian model is weak Ļ‰ āˆ’2 āˆ’1 1/f2 F F -1 P(x) P(c)
  • 41. Gaussian model is weak Ļ‰ āˆ’2 āˆ’1 1/f2 F F -1 P(x) P(c) a. b. F 2 āˆ’1 Ļ‰ F
  • 42. Gaussian model is weak Ļ‰ āˆ’2 āˆ’1 1/f2 F F -1 P(x) P(c) a. b. F 2 āˆ’1 Ļ‰ F a. b. c. 20 20 4 -20 -20 -4 -20 20 -20 20 -4 4
  • 43. Bandpass Filter Responses 0 10 Response histogram Gaussian density Probability -2 10 -4 10 500 0 500 Filter Response [Burt&Adelson 82; Field 87; Mallat 89; Daugman 89, ...]
  • 44. ā€œIndependentā€ Components Analysis (ICA) a. b. c. d. 20 20 4 4 -20 -20 -4 -4 -20 20 -20 20 -4 4 -4 4 For Linearly Transformed Factorial (LTF) sources: guaranteed independence (with some minor caveats) [Comon 94; Cardoso 96; Bell/Sejnowski 97; ...]
  • 45. ICA on image blocks [Olshausen/Field ā€™96; Bell/Sejnowski ā€™97] [example obtained with FastICA, Hyvarinen]
  • 46. Marginal densities log(Probability) log(Probability) log(Probability) log(Probability) p = 0.46 p = 0.58 p = 0.48 !H/H = 0.0031 !H/H = 0.0011 !H/H = 0.0014 Wavelet coefficient value Wavelet coefficient value Wavelet coefficient value Fig. 4. Log histograms of a single wavelet subband of four example images (see Fig. 1 for image histogram, tails are truncated so as to show 99.8% of the distribution. Also shown (dashed lines) are corresponding to equation (3). Text indicates the maximum-likelihood value of p used for the ļ¬tte Well-ļ¬t by a generalized Gaussian: the relative entropy (Kullback-Leibler divergence) of the model and histogram, as a fraction of th histogram. P (x) āˆ exp āˆ’|x/s| p non-Gaussian than others. By the mid 1990s, a number of authors had developed methods of optimizing a ba- sis of ļ¬lters in order to to maximize the non-Gaussianity of the responses [e.g., 36, 4]. Often these methods oper- [Mallat 89; Simoncelli&Adelson 96; Moulin&Liu 99; ...] ate by optimizing a higher-order statistic such as kurto-
  • 47. Kurtosis vs. bandwidth 16 14 12 Sample Kurtosis 10 8 6 4 0 0.5 1 1.5 2 2.5 3 Filter Bandwidth (octaves) Note: Bandwidth matters much more than orientation [see Bethge 06] [after Field 87]
  • 48. Octave-bandwidth representations Spatial Frequency Selectivity: Filter:
  • 49. Model II (LTF) Coefficient density: Basis set: Image: : : :
  • 50. LTF also a weak model... Sample Gaussianized Sample ICA-transformed and Gaussianized
  • 52. Trouble in paradise ā€¢ Biology: Visual system uses a cascade - Whereā€™s the retina? The LGN? - What happens after V1? Why donā€™t responses get sparser? [Baddeley etal 97; Chechik etal 06]
  • 53. Trouble in paradise ā€¢ Biology: Visual system uses a cascade - Whereā€™s the retina? The LGN? - What happens after V1? Why donā€™t responses get sparser? [Baddeley etal 97; Chechik etal 06] ā€¢ Statistics: Images donā€™t obey ICA source model - Any bandpass ļ¬lter gives sparse marginals [Baddeley 96] => Shallow optimum [Bethge 06; Lyu & Simoncelli 08] - The responses of ICA ļ¬lters are highly dependent [Wegmann & Zetzsche 90, Simoncelli 97]
  • 54. Conditional densities 1 1 0.6 0.6 0.2 0.2 -40 0 40 50 -40 0 40 40 0 -40 -40 0 40 Linear responses are not independent, even for optimized ļ¬lters! [Simoncelli 97; Schwartz&Simoncelli 01] CSH-02
  • 56. ā€¢ Large-magnitude subband coefļ¬cients are found at neighboring positions, orientations, and scales.
  • 57. Modeling heteroscedasticity (i.e., variable variance) Method 1: Conditional Gaussian P (xn |{xk }) āˆ¼ N 0; wnk |xk | + Ļƒ 2 2 k [Simoncelli 97; Buccigrossi&Simoncelli 99; see also ARCH models in econometrics!]
  • 58. Joint densities adjacent near far other scale other ori 150 150 150 150 150 100 100 100 100 100 50 50 50 50 50 0 0 0 0 0 !50 !50 !50 !50 !50 !100 !100 !100 !100 !100 !150 !150 !150 !150 !150 !100 0 100 !100 0 100 !100 0 100 !500 0 500 !100 0 100 150 150 150 150 150 100 100 100 100 100 50 50 50 50 50 0 0 0 0 0 !50 !50 !50 !50 !50 !100 !100 !100 !100 !100 !150 !150 !150 !150 !150 !100 0 100 !100 0 100 !100 0 100 !500 0 500 !100 0 100 ā€¢ Nearby: densities are approximately circular/elliptical Fig. 8. Empirical joint distributions of wavelet coefļ¬cients associated with different pairs of basis functions, for a single image of a New York City street scene (see Fig. 1 for image description). The top row shows joint distributions as contour plots, with lines drawn at equal intervals of log probability. The three leftmost examples correspond to pairs of basis func- ā€¢ Distant: densities are approximately factorial tions at the same scale and orientation, but separated by different spatial offsets. The next corresponds to a pair at adjacent scales (but the same orientation, and nearly the same position), and the rightmost corresponds to a pair at orthogonal orien- tations (but the same scale and nearly the same position). The bottom row shows corresponding conditional distributions: brightness corresponds to frequency of occurance, except that each column has been independently rescaled to ļ¬ll the full range of intensities. [Simoncelli, ā€˜97; Wainwright&Simoncelli, ā€˜99]
  • 59. ICA-transformed joint densities d=2 d=16 d=32 12 12 12 10 10 10 kurtosis 8 8 8 6 6 6 4 4 4 0 !/4 !/2 3!/4 ! 0 !/4 !/2 3!/4 ! 0 !/4 !/2 3!/4 ! orientation data (ICAā€™d): sphericalized: factorialized:
  • 60. ICA-transformed joint densities d=2 d=16 d=32 12 12 12 10 10 10 kurtosis ā€¢ Local densities are elliptical (but non-Gaussian) 8 8 8 6 6 6 ā€¢ Distant densities are factorial 4 4 4 0 !/4 !/2 3!/4 ! 0 !/4 !/2 3!/4 ! 0 !/4 !/2 3!/4 ! orientation data (ICAā€™d): [Wegmann&Zetzsche ā€˜90; Simoncelli ā€™97; + many recent models] sphericalized: factorialized:
  • 61. Spherical vs LTF 0.2 blk blk 0.4 blk blk size = 3x3 0.2 blk size = 7x7 blk size = 11x11 spherical spherical spherical factorial factorial 0.35 factorial 0.15 0.3 0.15 0.25 0.1 0.2 0.1 0.15 0.05 0.1 0.05 0.05 0 0 0 3 6 9 12 15 18 20 3 6 9 12 15 18 20 3 6 9 12 15 18 20 kurtosis kurtosis kurtosis 3x3 7x7 15x15 data (ICAā€™d): sphericalized: factorialized: ā€¢ Histograms, kurtosis of projections of image blocks onto random unit-norm basis functions. ā€¢ These imply data are closer to spherical than factorial [Lyu & Simoncelli 08]
  • 62. non-Gaussian elliptical observations and models of natural images: - Zetzsche & Krieger, 1999; - Huang & Mumford, 1999; - Wainwright & Simoncelli, 2000; - HyvƤrinen and Hoyer, 2000; - Parra et al., 2001; - Srivastava et al., 2002; - Sendur & Selesnick, 2002; - Teh et al., 2003; - Gehler and Welling, 2006 - Lyu & Simoncelli, 2008 - etc.
  • 63. Modeling heteroscedasticity Method 2: Hidden scaling variable for each patch Gaussian scale mixture (GSM) [Andrews & Mallows 74]: āˆš x= zu ā€¢ u is Gaussian, z > 0 ā€¢ z and u are independent ā€¢ x is elliptically symmetric, with covariance āˆ Cu ā€¢ marginals of x are leptokurtotic [Wainwright&Simoncelli 99]
  • 64. GSM - prior on z ā€¢ Empirically, z is approximately lognormal [Portilla etal, icip-01] exp (āˆ’(log z āˆ’ Āµl )2 /(2Ļƒl )) 2 pz (z) = 2 )1/2 z(2Ļ€Ļƒl ā€¢ Alternatively, can use Jeffreyā€™s noninformative prior [Figueiredo&Nowak, ā€˜01; Portilla etal, ā€˜03] pz (z) āˆ 1/z
  • 65. GSM simulation Image data GSM simulation ! ! #" #" " " #" #" !!" " !" !!" " !" [Wainwright & Simoncelli, NIPS*99]
  • 66. Model III (GSM) Coefficient density: Basis set: Image: X X X X sqrt(z) X X u
  • 67. āˆš Original coefļ¬cients Normalized by z !2 !4 !4 !5 marginal Log probability Log probability !6 !6 !7 [Ruderman&Bialek 94] !8 !8 !9 !10 !500 0 500 !10 !5 0 5 100 8 50 6 joint 0 4 [Schwartz&Simoncelli 01] !50 2 !100 0 !100 !50 0 50 100 0 2 4 6 8 subband
  • 68. 6 Model Encoding Cost (bits/coeff) Model Encoding cost (bits/coeff) 5.5 5 5 4.5 4 4 3 3.5 3 2 Gaussian Model First Order Ideal 2.5 Generalized Laplacian Conditional Model 1 3 4 5 1 2 3 4 5 6 Empirical First Order Entropy (bits/coeff) Empirical Conditional Entropy [Buccigrossi & Simoncelli 99]
  • 69. Bayesian denoising ā€¢ Additive Gaussian noise: y =x+w 2 2 P (y|x) āˆ exp[āˆ’(y āˆ’ x) /2Ļƒw ] ā€¢ Bayesā€™ least squares solution is conditional mean: x(y) = IE(x|y) Ė† = dxP(y|x)P(x)x/P(y)
  • 70. I. Classical If signal is Gaussian, BLS estimator is linear: denoised (Ė†) x 2 Ļƒx x(y) = Ė† 2 2 Ā·y Ļƒx + Ļƒn => suppress ļ¬ne scales, noisy (y) retain coarse scales
  • 71. Non-Gaussian coefļ¬cients " #" -*./01.*,6'.)07+48 94:..'41,;*1.')5,, 2+0343'(')5 !% #" !$ #" !!"" " !"" &'()*+,-*./01.* [Burt&Adelson ā€˜81; Field ā€˜87; Mallat ā€˜89; Daugman ā€˜89; etc]
  • 72. II. BLS for non-Gaussian prior ā€¢ Assume marginal distribution [Mallat ā€˜89]: P (x) āˆ exp āˆ’|x/s| p ā€¢ Then Bayes estimator is generally nonlinear: p = 2.0 p = 1.0 p = 0.5 [Simoncelli & Adelson, ā€˜96]
  • 73. MAP shrinkage p=2.0 p=1.0 p=0.5 [Simoncelli 99]
  • 74. Denoising: Joint IE(x|y) = dz P(z|y) IE(x|y, z) āˆ’1y ļ£» ļ£® ļ£¹ = dz P(z|y) ļ£° zCu(zCu + Cw ) ctr where P(y|z) P(z) exp(āˆ’y T (zCu + Cw )āˆ’1y/2) P(z|y) = , P(y|z) = Py (2Ļ€)N |zCu + Cw | Numerical computation of solution is reasonably efļ¬cient if one jointly diagonalizes Cu and Cw ... [Portilla, Strela, Wainwright, Simoncelli, ā€™03] IPAM, 9/04 20
  • 75. ESTIMATED COEFF. Example estimators !" +'1&2/1+3)*%+,,- " !w !!" #" !" " " NOISY COEFF. !#" $%&'()./0+$1 !!" $%&'()*%+,,- Estimators for the scalar and single-neighbor cases [Portilla etal 03]
  • 76. Comparison to other methods "'& "'& " " ,456748+91:;<= /456,(74-.)/-0123 !"'& !"'& :>6965#8*>?6< !! !! !!'& !!'& !# !# !#'& !#'& )89!:1:;<=>? !$ .=9@A?-9?=@BC8D !$ !$'& !$'& !" #" $" %" &" !" #" $" %" &" ()*+,-*.)/-0123 ()*+,-*.)/-0123 Results averaged over 3 images [Portilla etal 03]
  • 77. Noisy Original (22.1 dB) Matlabā€™s BLS-GSM wiener2 (30.5 dB) (28 dB)
  • 78. Noisy Original (8.1 dB) UndWvlt BLS-GSM Thresh (21.2 dB) (19.0 dB)
  • 79. Real sensor noise 400 ISO denoised
  • 80. GSM summary ā€¢ GSM captures local variance ā€¢ Underlying Gaussian leads to simple computation ā€¢ Excellent denoising results ā€¢ Whatā€™s missing? ā€¢ Global model of z variables [Wainwright etal 99; Romberg etal ā€˜99; Hyvarinen/Hoyer ā€˜02; Karklin/ Lewicki ā€˜02; Lyu/Simoncelli 08] ā€¢ Explicit geometry: phase and orientation
  • 81. Global models for z ā€¢ Non-overlapping neighborhoods, tree-structured z [Wainwright etal 99; Romberg etal ā€™99] z u Coarse scale Fine scale ā€¢ Field of GSMs: z is an exponentiated GMRF, u is a GMRF, subband is the product [Lyu&Simoncelli 08]
  • 82. D MACHINE INTELLIGENCE, VOL. X, NO. X, XX 200X 9 State-of-the-art denoising Lena Boats " " # # !" !" āˆ†()*+, āˆ†()*+, !$ !$ !' !' !& !& "## ! "#"! $! !# %! "## ! "#"! $! !# %! "## Ļƒ Ļƒ FoGSM BM3D kSVD thods for three diļ¬€erent images. Plotted are diļ¬€erences in PSNR for diļ¬€erent input noise levels (Ļƒ) between BLS-GSM [17], kSVD [39] and FoE [27]). The PSNR values for these methods were taken from GSM FoE [Lyu&Simoncelli, PAMI 08]
  • 83. Measuring Orientation 2-band steerable pyramid: Image decomposition in terms of multi-scale gradient measurements [Simoncelli et.al., 1992; Simoncelli & Freeman 1995]
  • 85. Multi-scale gradient basis ā€¢ Multi-scale bases: efļ¬cient representation
  • 86. Multi-scale gradient basis ā€¢ Multi-scale bases: efļ¬cient representation ā€¢ Derivatives: good for analysis ā€¢ Local Taylor expansion of image structures ā€¢ Explicit geometry (orientation)
  • 87. Multi-scale gradient basis ā€¢ Multi-scale bases: efļ¬cient representation ā€¢ Derivatives: good for analysis ā€¢ Local Taylor expansion of image structures ā€¢ Explicit geometry (orientation) ā€¢ Combination: ā€¢ Explicit incorporation of geometry in basis ā€¢ Bridge between PDE / harmonic analysis approaches
  • 89. Importance of local orientation Randomized orientation Randomized magnitude [Hammond&Simoncelli 05]
  • 90. Reconstruction from orientation Original Quantized to 2 bits ā€¢ Reconstruction by projections onto convex sets ā€¢ Resilient to quantization [Hammond&Simoncelli 06]
  • 91. Image patches related by rotation two-band steerable [Hammond&Simoncelli 06] pyramid coefficients
  • 92. raw rotated patches patches PCA of normalized gradient patches --- Raw Patches Rotated Patches [Hammond&Simoncelli 06]
  • 93. Orientation-Adaptive GSM model Model a vectorized patch of wavelet coefficients as: patch rotation operator hidden magnitude/orientation variables [Hammond&Simoncelli 06]
  • 94. Orientation-Adaptive GSM model Model a vectorized patch of wavelet coefficients as: patch rotation operator hidden magnitude/orientation variables Conditioned on ; is zero mean gaussian with covariance [Hammond&Simoncelli 06]
  • 95. Estimation of C(Īø) from noisy data noisy patch unknown, approximate by measured from noisy data. Assuming independent and noise rotationally invariant (assuming w.l.o.g. E[z] =1 ) [Hammond&Simoncelli 06]
  • 96. Bayesian MMSE Estimator [Hammond&Simoncelli 06]
  • 97. Bayesian MMSE Estimator condition on and integrate over hidden variables [Hammond&Simoncelli 06]
  • 98. Bayesian MMSE Estimator condition on and integrate over hidden variables [Hammond&Simoncelli 06]
  • 99. Bayesian MMSE Estimator condition on and integrate over hidden variables Wiener estimate [Hammond&Simoncelli 06]
  • 100. Bayesian MMSE Estimator condition on and integrate over hidden variables Wiener estimate has covariance separable prior for hidden variables [Hammond&Simoncelli 06]
  • 101. Bayesian MMSE Estimator condition on and integrate over hidden variables Wiener estimate has covariance separable prior for hidden variables [Hammond&Simoncelli 06]
  • 102. Ļƒ = 40 noisy 2.81 dB gsm2 oagsm 12.4 dB 13.1 dB
  • 103. Locally adaptive covariance ā€¢ Karklin & Lewicki 08: Each patch is Gaussian, with covariance constructed from a weighted outer- product of ļ¬xed vectors: p(x) = G (x; C(y)) log C(y) = yn Bn n T p(y) = exp(āˆ’|yn |) Bn = wnk bk bk n k ā€¢ Guerrero-Colon, Simoncelli & Portilla 08: Each patch is a mixture of GSMs (MGSMs): p(x) = Pk p(zk ) G(x; zk Ck ) dzk k
  • 104. MGSMs generative model āˆš āˆš āˆš Patch x chosen from { z1 u1 , z2 u2 , ... zK uK } with probabilities {P1 , P2 , ..., PK } Parameters: ā€¢ Covariances Ck ā€¢ Scale densities pk (zk ) ā€¢ Component probabilities Pk ā€¢ Number of components K Parameters can be ļ¬t to data of one or more images by maximizing likelihood (EM-like) [Guerrero-Colon, Simoncelli, Portilla 08]
  • 105. MGSM ā€œsegmentationā€ image 1 2 4 First six eigenvectors of GSM covariance matrices [Guerrero-Colon, Simoncelli, Portilla 08]
  • 106. MGSM ā€œsegmentationā€ Eigenvectors of GSM components represent invariant subspaces: ā€œgeneralized complex cellsā€
  • 107. Potential of local homogeneous models? Consider an implicit model: maxEnt subject to constraints on subband coefļ¬cients: ā€¢ marginal statistics [var,skew,kurtosis] ā€¢ local raw correlations ā€¢ local variance correlations ā€¢ local phase correlations [Portilla & Simoncelli 00; cf. Zhu, Wu & Mumford 97]
  • 109. Visual texture Homogeneous, with repeated structures
  • 110. Visual texture Homogeneous, with repeated structures ā€œYou know it when you see itā€
  • 111. All Images Texture Images Equivalence class (visually indistinguishable)
  • 112. Iterative synthesis algorithm Analysis Example Transform Measure Texture Statistics Synthesis Measure Statistics Random Transform Inverse Synthesized Adjust Seed Transform Texture [Portilla&Simoncelli 00; cf. Heeger&Bergen ā€˜95]
  • 121. Texture mixtures Convex combinations in parameter space => Parameter space includes non-textures
  • 122. Summary ā€¢ Fusion of empirical data with structural principles ā€¢ Statistical models have led to state-of-the-art image processing, and are relevant for biological vision ā€¢ Local adaptation to {variance, orientation, phase, ...} gives improvement, but makes learning harder ā€¢ Cascaded representations emerge naturally ā€¢ Thereā€™s still much room for improvement!
  • 123. Cast ā€¢ Local GSM model: Martin Wainwright, Javier Portilla ā€¢ GSM Denoising: Javier Portilla, Martin Wainwright, Vasily Strela ā€¢ Variance-adaptive compression: Robert Buccigrossi ā€¢ Local orientation and OAGSM: David Hammond ā€¢ Field of GSMs: Siwei Lyu ā€¢ Mixture of GSMs: Jose-Antonio Guerrero-ColĆ³n, Javier Portilla ā€¢ Texture representation/synthesis: Javier Portilla