# April 2013

You are currently browsing the monthly archive for April 2013.

## “Resurgence in Neural Networks”

T Jake Luciani wrote a nice, easy to read blog post on the recent developments in neural networks.

## “Autoencoders, MDL, and Helmholtz Free Energy”

In “Autoencoders, MDL, and Helmholtz Free Energy“, Hinton and Zemel (2001) use Minimum Description Length as an objective function for formulating generative and recognition weights for an autoencoding neural net.  They develop a stochastic Vector Quantization method very similar to mixture of Gaussians where each input vector is encoded with

$$E_i = – \log \pi_i – k \log t + {k\over2} \log 2 \pi \sigma^2 + {{d^2} \over{2\sigma^2}}$$

nats (1 nat = 1/log(2) bits = 1.44 bits) where $t$ is the quantization width, $d$ is the Mahalanobis distance to the mean of the Gaussian, $k$ is the dimension of the input space, $\pi_i$ is the weight of the $i$th Gaussian.  They call this the “energy” of the code.  Encoding only using this scheme wastes bits because, for example, there may be vectors that are equally distant from two Gaussian. The amount wasted is

$$H = -\sum p_i \log p_i$$

where $p_i$ the probability that the code will be assigned to the $i$th Gaussian. So the “true” expected description length is

$$F = \sum_i p_i E_i – H$$

which “has exactly the form of the Helmholtz free energy.”  This free energy is minimized by setting

$$p_i = {{e^{-E_i}}\over{\sum_j e^{-E_j}}}.$$

In order to make computation practical, they recommend using a suboptimal distributions “as a Lyapunov function for learning” (see Neal and Hinton 1993). They apply their method to learn factorial codes.

## A New Dalton Minimum ? (Off topic)

Seemingly weak Solar Cycle #24

A long time ago I was a meteorologist at the Joint Typhoon Warning Center in Guam.  One of my forecasters had a friend that used solar images to forecast wheat futures.  I wonder what those people are predicting these days.

The Dalton minimum occurred between 1790 to 1830.  If the sun does not perk up, solar cycle #24 will resemble the Dalton minimum solar cycles.  (See also the Spörer minimum and the Maunder minimum.)  For more information check out NOAA’s Solar Cycle Progression page.