Nuit Blanche‘s article “The Summer of the Deeper Kernels” references the two page paper “Deep Support Vector Machines for Regression Problems” by Schutten, Meijster, and Schomaker (2013).
The deep SMV is a pretty cool idea. A normal support vector machine (SVM) classifier, finds $\alpha_i$ such that
$f(x) = \sum_i \alpha_i K(x_i, x)$ is positive for one class of $x_i$ and negative for the other class (sometimes allowing exceptions). ($K(x,y)$ is called the kernel function which is in the simplest case just the dot product of $x$ and $y$.) SVM’s are great because they are fast and the solution is sparse (i.e. most of the $\alpha_i$ are zero).
Schutten, Meijster, and Schomaker apply the ideas of deep neural nets to SVMs.
They construct $d$ SVMs of the form
$f_a(x) = \sum_i \alpha_i(a) K(x_i, x)+b_a$
and then compute a more complex two layered SVM
$g(x) = \sum_i \alpha_i K(f(x_i), f(x))+b$
where $f(x) = (f_1(x), f_2(x), \ldots, f_d(x))$. They use a simple gradient descent algorithm to optimize the alphas and obtain numerical results on ten different data sets comparing the mean squared error to a standard SVM.