By now, most of the people following the financial crisis closely have seen the following graph (left).

It shows the job recovery during all the financial recessions in US after WWII. As you can see the unemployment curve is symmetric. Looking at the current crisis data, if the historic analysis is valid, we can conclude we are not even half way through. An interesting trend in the graph is that the time to recovery has been increasing. This should be due to the more effective govermental policies to control the crisis and its damage to job market. These policies effectively has flatten and elongated the unemployment curve. To see this trend we plotted the number of months that it took the job market to recover vs the year that recession started (right). The linear fit is not a good fit but if it is of any clue, in the current crisis the job market will take around 36 months to recover.

In a small system fluctuations can not be ignored, experiments can not be repeated. However, it is still desirable to have some understanding over the non-equilibrium behavior of the system.

Control Parameters vs Fluctuation Variables: For small systems the equation of state and the spectrum of fluctuations are fully determined by so-called control parameters.

The total change in the energy of the system has two components

The first term can be interpretes as internal energy, while the second term can be thought as work . For a specific experiment the value of and are not reproducible due to the fluctuations in the system, although we can think of their probability distributions.

Probability Distributions: The work and heat probability distributions and characterize the work and heat collected over an infinite number of experiments.

Fluctuation Theorems: The rate at which the system exchange heat with the bath is called the “entropy production“. It is convenient to define where is the interval of time over which the system exchange the heat . Associated with the entropy production is a time-dependent probability distribution . Under very general condition the following relation holds (Gallavotti and Cohen) for systems in steady states

Although the relation holds for infinite limit, it is considered as a good approximation for finite time. Note that the ratio for a system in equilibrium with the thermal bath is equal to $1$. This shows that steady state systems are more likely to deliver heat to the bath than it is to absorb heat form bath (Time reversal microscopic laws give birth to non time reversal macroscopic phenomenon, an answer to Loschmidt’s paradox).

The Jarzynski Equality: In a system in contact with thermal bath at temperature T, the difference in free energy between equilibrium state and with and control parameter.

,

the average being taken over infinite number of non-equilibrium processes.

The Crooks Fluctuation Theorem:

Non-Gaussian in the heat pobability distribution:

Generalized JE to arbitrary transition between non-equilibrium steady states:

for more information Ref to “The Nonequilibrium Thermodynamics of Small Systems”, by C. Bustamante, J. Liphardt and F. Ritort

It is interesting to have a statistical description of foam. In doing so few simple characteristics may come to mind,

Network Properties:

A network can be associated to a foam structure by replacing every cell(bubble) with a node and attach these nodes to their physical neighbors. The statistical properties of this network can be interesting.

Bubble size Distribution:

Various empirical formulae have been proposed by fitting data, including

Here is the radius of the bubble assuming they have spherical shapes and is the relative size of bubbles to average radius of the population .

Dynamical Aging:

1) Average bubble sizes increase.

2) The probability distribution broadens.

3) Total volume decreases

4) Average life time of a bubble with radius , we may need to know all the neighbors radii too.

In the case of a single hemispherical bubble at the surface of liquid and gas the bubble burst because of drainage and thinning effect at the top of the buuble. The life time in this case inversely depends on

5) Transition probability of the network after a bubble burst.

Taken from: “An Information Maximization Approach to Blind Separation and Blind Deconvolution” by A.J. Bell and T.J. Sejnowski

The idea is to maximize the mutual information between the output of a neural network and its input . Mutual information is defined as .

There are divergences due to the continuous variables. As a result it is better to work with gradient of mutual information with respect to some parameter, . They considered the systems that have additive noise, ,  where . Therefore to maximize the mutual information it is sufficient to maximize entropy of the output. Assuming that the input probability distribution is then probability distribution of the output is

the entropy in the output can be written as

A stochastic gradient descent learning rule can be implemented as

In the case of logistic transfer function or tanh function . The learning rule has two component an Anti-Hebbian, , and an Anti-Decay component, .

Multi-component network:

This result can be generalized to multi-component network, where . The result will be

Casual Filters:

Furthermore it can be generalized to casual filters where , or equivalently . Now is lower triangular matrix.

The probability distribution function of and are

The gradient descend learning rule is

The delay weights keep shrinking and try to de-correlate the past signal from future. It is very interesting to see what kind of will be created depending on the input statistics.

Time Delay:

following the same set of steps we find

Generalizing the transfer function:

Blind Separation: A set of source signals, is mixed together linearly, with a matrix . We ought to find a square matrix that is inverse of up to a permutation. The problem reduces to minimize the mutual information between the inputs(Independent Component Analysis). Obviously, if the matrix is singular one can not solve the problem.

Blind De-convolution: A signal is corrupted with a linear filter, ,  . We have to find a filter to recover from . The problem reduces to remove statistical dependency across time (Whitening of ).

The fundamental question is how can we reduce mutual information between two outputs by information maximization. The joint entropy of two output signals is

The algorithm discussed above maximize and this is mostly done by minimizing . When is zero the probability distribution of ’s is separable. Can we introduce a constraint and keep the total conserved?

It was briefed up to section 5.

Taken from: “The Effect of Correlated Variability on the Accuracy of a Population Code” by L.F. Abbott and P. Dayan

They answer to the following questions for three different class of models: “(1) Does correlation increase or decrease the accuracy with which the value of an encoded quantity can be extracted from a population of neurons? (2) Does this accuracy approach a fixed limit as increases?”

The average firing rate is noted as . The correlation matrix is used to calculate Fisher information .  The Fisher information can be calculated assuming Gaussian character of correlation

The models being studied are:

Additive Noise Model:

A nice example of collective quantities and and the dependence of their respective variance, and , is presented.

where and . It worths mentioning that when the Fisher information fails to grow linearly. This will put a constraint on the individual neurons tuning curves among the population

Multiplicative Noise Model:

Limited-Range Correlation Model:

is the number of encoded variables. “The main conclusion of this paper is that neurons should have different selectivity to the quantities they are encoding. In particular their tuning curves should not be additively or multiplicatively separable”.

The configuration space of any system with conserved energy (time-independent Hamiltonian) can be broken into subspaces surrounding potential energy minima, which the minima will be reached by the steepest descent path of the potential. This hyper-volume around a potential minimum is called basin. The structure based on these neighboring basins is called as the “Inherent Structure”. The statistical behavior of a complex system can be studied by analyzing the characteristics of this structure. The partition function of a system of particles can be written as

We characterize each basin by its minimum potential . The total volume is partitioned by these basins and so does the above integral.

Defining an average free energy for basins

Now if we know the probability distribution of energy minima the partition function can be written easily as,

General Properties of PEL’s:

The number of total minima is exponentially depends on the number of underlying degrees of freedom (e.g. number of atoms),

The energy distribution of minima follows Guassian distribution. This can be understood through central limit theorem for a large system with independent subsystems.

Also on average an exponential relation between the minimas’ energy and their corresponding volume was observed

The probability distributions, , of the volume, , of these basins for various dynamical systems (Binary Lennard-Jones, Dzugutov Liquids and Amorphous Silicon) show power law behavior

There is a similarity between this phenomenon and “Apollonian Packing” of an arbitrary volume, where in the limit of large dimensionality of the target space obeys the same power law. Moreover, the network consist of connecting neighboring minima is a scale free network, which means

There is a strong correlation (a power law) between the volume of the basin and its number of neighbors, ,

This relation subsequently dictates a relation between the minimum energy and number of neighbors.

As a reference look through the papers by J. P. K. Doye and C. P. Massen.

Taken from: “Fluctuations in Network Dynamics”, by M. Argollo de Menezes and A.-L. Barabasi

The flow through a node in a network is time dependent. This time dependence can be partly be described by mean flow, and the fluctuation around this mean .
In most natural networks, random, scale-free or small-world there is a functional dependence between mean and fluctuation of the flow through the nodes. In this paper it is claimed that the relation between and in different networks (only scale-free and random network) fall into two different categories and characterized by their -exponent

In all the networks they studied is equal to or 1. In the networks with the internal noise is responsible for flow fluctuations, while it was claimed that in networks with the external noise is responsible for the fluctuations. Two different models are proposed.

Model 1: At any time step, number of walkers are placed randomly on the network nodes, the preform step walks.

Model 2: At any time step random pairs of nodes are selected and they are connected through the shortest path between them (degeneracy problem is not discussed).

In both cases we observe if is fixed and if has large fluctuations. In general the fluctuations on a given nodes can be decompose into internal and external components

where , represent the external driving force in the noise magnitude. Moreover, by increasing a transition form to behavior can be seen (some issues regarding the fitting process should be considered.).

In conclusion: “The captures an endogenous behavior determined by the system’s internal fluctuations” while “The exponent describes driven systems, in which the fluctuations of individual nodes are dominated by the time dependent changes in the external driving forces.”

This post is taken from the following paper,

“Optimal sampling of Natural Images: A design Principle for the Visual Systems?” by W. Bialek, D.L. Ruderman and A. Zee

If the natural scene images can be characterized by and the cells are indexed by integers and positioned respectively at . The output of each cell, , is a linear filter (receptive field) with an additional noisy component

The goal is to find a kernel that maximze the information content of output, , about the input, . The information content of assuming that is Gaussian is

here is the power spectrum of the signal. We can approximate in large noise regime as

We need to put extra constraints to get solutions that are physically realistic

1) We should fix the filters gain,

2) There should be a cost for “long-range interactions in spatial space” or “sharp fluctuations in momentum space”,

Using variational methods they found that filter should satisfy the schrodinger like equation in -space

the playes the role of the mass and of potential in the schrodinger equation.

In the case that they are interested the power spectrum of natural images has a power law and as a result an accidental symmetry. This accidental symmetry allows them to recombine various angular momentum eigenstates and create orientation selective eigenstates (filters).

What interests me the most is that for any linear filter this formalism holds and, the power spectrum of the input signal appears as the potential of the schrodinger equation.

Taken from the paper by G.J. Stephens and W. Bialek.

There are possible four letter words, the total entropy for such a sample is bits. Out of this large number of possibilities only a small fraction of is in use in English language. By looking at large database of American literature, words and their frequency of occurrence is found (the database had words all the four letter words which they have appeared more than times are considered.). The entropy of this set is approximately . The paper is trying to regenerate legal words using statistical properties of their building blocks (the letters).

Independent Letters Approximation:

Not all the letters appear with the same frequencies. Considering this point the probability of having a four letter words is . The entropy of a such an ensemble is bits.

Pairwise Interaction Approximation:

We assume that can be written as a Boltzmann factor of pairwise interacting between different letters

are the pairwise potential and have free parameters. They can be determined by solving coupled equations (pairwise marginal distribution).

The ensemble built in this construction has an entropy of bits. The pairwise potentials define an energy landscape. There are 136 local minima in this landscape (changing any letter increases the energy) out of them 118 are real English words, that capture of the probability distribution.

The induced probability for each real word using pairwise interactions reproduce the observed probability of that word especially for frequent words (The plot of vs is almost diagonal).

The independent letter approximation reduced the number of possibilites by a factor of , a further reduction factor of , the higher order interaction only contribute for a factor of .

Zipf’s Law:

Plotting the probability vs the rank of different words shows an approximate power law.The four letter words have a cut off tail. The pairwise model removes some weight from the bulk of the distribution and reassigns it to the tail.

This is a brief extraction from Mora et.al. paper.

The idea originates from an observation by Field that the spatial spectrum of images of natural scenes follow a behavior. Now if this behavior was observed in a statistical system, the first conclusion was that the system is in its critical point.

Here the authors to show the same result assume that the probability of an image is related to an energy function through Boltzmann relation

where

An pixel black and white image has possible configurations. In order to make this number they studied the system for and using coarse graining ideas and block averaged the natural images.

Using their set of images they

1) assumed in natural images and found

2) the probability at any other temperature can be found by

3) the entropy at any other temperature can be derived by

4) calculated the specific heat

5) repeated all these steps for and .

They observed that there is a peak in normalized specific heat and as increase the peak becomes sharper and approaches . This is a clear indication for critical behavior in natural scenes (divergence of the specific heat at ).

In the macrocanonical ensemble, we study the statistical system under the fixed energy constraint. Under this conditions we can define the partition function as an integral over the density of states with constant energy ,

Entropy will be defined as logarithm of density of states i.e. ,

since both entropy and energy are extensive it is more suitable to work with and . In the large N limit,

.

When the vs curve for a system is become linear we should expect critical behavior. For natural images this can be done by looking at the probability of occurrence of each configuration, , then calculating of each configuration up to a constant. The density of states is the histogram of the configurations’ energy. Finally, the entropy is logarithm of density of states.

This calculation has been done for 8 to 50 pixel sample and they all shown a linear behavior at low energy limit, where the sampling methods are more reliable (more occurrence).

They also prove that in a system which follows generalized Zipf’s law entropy has a linear dependence on energy.

Note: Generalized Zipf’s law i.e. , where it  relates the ranking of a state to its probability.

Energy Landscape:

Finally they studied the energy landscape of the natural images by looking at the local minima of the energy. They examined all the pixel images. The local minima are defined as configurations that flipping any pixel will increase the energy of the configuration. They found approximately 100 of these minima. The local minima are important in answering the question that if the natural images are at the critical point of what will happen to system at ?, and what order parameter can be defined for this phase transition?

Most of the local minima are representing an edge between black and white regions or a strip. This is consistent with the visual system that the response triggered averages of neurons in response to natural scenes has a prominent edge detection component.