“A probabilistic programming language is a high-level language that makes it easy for a developer to define probability models and then “solve” these models automatically. These languages incorporate random events as primitives and their runtime environment handles inference. Now, it is a matter of programming that enables a clean separation between modeling and inference.”

writes Beau Cronin in this post about Probabilistic Programming (PP) (see e.g. [1] and [2]). He goes on to informally describe a probabilistic graphical model (PGM) and how PP languages or extensions to existing languages like BLOG, BUGS, Church (Lisp), FACTORIE, Figaro (Scala), HANSEI (Ocaml), Hierarchical Bayesian Compiler, Infer.NET, ProbLog, and Stan make it much easier to set up and solve PGMs. He provides links to a tutorial, a great easy-to-understand video, the NIPS workshop on PP, and several ongoing PP projects.

I also enjoyed the more detailed post “Why Probabilistic Programming Matters” by Rob Zinkov. Rob shows how to represent the following machine learning techniques in a PP language:

- Bayesian Linear Regression
- Logistic Regression
- Naive Bayes
- K-Means Clustering
- Latent Dirichlet Allocation (LDA)
- Correlated Topic Models (CTM)
- Autoregressive Integrated Moving Average (ARIMA)
- Hidden Markov Models (HMM)
- Matrix Factorization
- Sparsity and Sparse Bayes
- Conditional Random Fields (CRF)