Gaussian distributions are the most “natural” distributions. They show up everywhere. Here is a list of the properties that make me think that Gaussians are the most natural distributions:

• The sum of several random variables (like dice) tends to be Gaussian. (Central Limit Theorem).
• There are two natural ideas that appear in Statistics, the standard deviation and the maximum entropy principle. If you ask the question, “Among all distributions with standard deviation 1 and mean 0, what is the distribution with maximum entropy?” The answer is the Gaussian.
• Randomly select a point inside a high dimensional hypersphere. The distribution of any particular coordinate is approximately Gaussian. The same is true for a random point on the surface of the hypersphere.
• Take several samples from a Gaussian Distribution. Compute the Discrete Fourier Transform of the samples. The results have a Gaussian Distribution. I am pretty sure that the Gaussian is the only distribution with this property.
• The eigenfunctions of the Fourier Transforms are products of polynomials and Gaussians.
• The solution to the differential equation y’ = -x y is a Gaussian. This fact makes computations with Gaussians easier. (Higher derivatives involve Hermite polynomials.)
• I think Gaussians are the only distributions closed under multiplication, convolution, and linear transformations.
• Maximum likelihood estimators to problems involving Gaussians tend to also be the least squares solutions.
• I think all solutions to stochastic differential equations involve Gaussians. (This is mainly a consequence of the Central Limit Theorem.
• “The normal distribution is the only absolutely continuous distribution all of whose cumulants beyond the first two (i.e. other than the mean and variance) are zero.” – Wikipedia.
• For even n, the nth moment of the Gaussian is simply an integer multiplied by the standard deviation to the nth power.
• Many of the other standard distributions are strongly related to the Gaussian (i.e. binomial, Poisson, chi-squared, Student t, Rayleigh, Logistic, Log-Normal, Hypergeometric …)
• “If X1 and X2 are independent and their sum X1 + X2 is distributed normally, then both X1 and X2 must also be normal.” — From the Wikipedia.
• “The conjugate prior of the mean of a normal distribution is another normal distribution.” — From the Wikipedia.
• When using Gaussians, the math is easier.
• The Erdős–Kac theorem implies that the distribution of the prime factors of a “random” integer is Gaussian.
• The velocities of random molecules in a gas are distributed as a Gaussian. (With standard deviation = $z*\sqrt{ k\, T / m} $ where $z$ is a “nice” constant, $m$ is the mass of the particle, and $k$ is Boltzmann’s constant.)
• “A Gaussian function is the wave function of the ground state of the quantum harmonic oscillator.” — From Wikipedia
• Kalman Filters.
• The Gauss–Markov theorem.