Archive for September, 2008

Thermodynamics of Natural Images

September 25, 2008

This is a brief extraction from Mora et.al. paper.

The idea originates from an observation by Field that the spatial spectrum of images of natural scenes follow a 1/f^2 behavior. Now if this behavior was observed in a statistical system, the first conclusion was that the system is in its critical point.

Here the authors to show the same result assume that the probability of an image \sigma is related to an energy function E(\sigma) through Boltzmann relation

P_T(\sigma)= {1 \over Z} e^{-E(\sigma)/T} where Z = \sum_{\sigma} P(\sigma)

An L\times L pixel black and white image has 2^{L^2} possible configurations. In order to make this number they studied the system for L=2 , 3 and 4 using coarse graining ideas and block averaged the natural images.

Using their set of images they

1) assumed T=1 in natural images and found P(\sigma)

2) the probability at any other temperature can be found by

P_T(\sigma) = {P(\sigma)^{1/T} \over \sum_{\sigma} P(\sigma)^{1/T} }

3) the entropy at any other temperature can be derived by

S(T) = - \sum_{\sigma} P_T(\sigma) \log P_T(\sigma)

4) calculated the specific heat

C(T)=T {\partial S(T) \over \partial T}

5) repeated all these steps for L=2,3 and 4.

They observed that there is a peak in normalized specific heat C(T,L)/L^2 and as L increase the peak becomes sharper and approaches T=1. This is a clear indication for critical behavior in natural scenes (divergence of the specific heat at T=1).

In the macrocanonical ensemble, we study the statistical system under the fixed energy constraint. Under this conditions we can define the partition function as an integral over the density of states with constant energy \rho (E),

Z(T) = \int dE \rho(E) e^{-E/T}

Entropy will be defined as logarithm of density of states i.e. \rho(E)=e^{S(E)} ,

since both entropy and energy are extensive it is more suitable to work with \epsilon = E/N and s(\epsilon)=S(\epsilon)/N. In the large N limit,

C={N \over T} \left[ - {d^2 s(\epsilon) \over d\epsilon^2} \right].

When the S/N vs E/N curve for a system is become linear we should expect critical behavior. For natural images this can be done by looking at the probability of occurrence of each configuration, P(\sigma), then calculating P=\exp (- E /T) of each configuration up to a constant. The density of states \rho (E) is the histogram of the configurations’ energy. Finally, the entropy is logarithm of density of states.

This calculation has been done for 8 to 50 pixel sample and they all shown a linear S , E behavior at low energy limit, where the sampling methods are more reliable (more occurrence).

They also prove that in a system which follows generalized Zipf’s law entropy has a linear dependence on energy.

Note: Generalized Zipf’s law i.e. p_r \propto 1/r^{\alpha}, where it  relates the ranking of a state r to its probability.

Energy Landscape:

Finally they studied the energy landscape of the natural images by looking at the local minima of the energy. They examined all the 4 \times 4 pixel images. The local minima are defined as configurations that flipping any pixel will increase the energy of the configuration. They found approximately 100 of these minima. The local minima are important in answering the question that if the natural images are at the critical point of T=1 what will happen to system at T=0?, and what order parameter can be defined for this phase transition?

Most of the local minima are representing an edge between black and white regions or a strip. This is consistent with the visual system that the response triggered averages of neurons in response to natural scenes has a prominent edge detection component.

Posted in Information Theory | Leave a Comment »