• Lots of bugfixes.
• Various small improvements in speed.
• A unified CRF training interface to make things easier for those not training on images
• Binaries are now provided for Linux as well as OS X.
• The code for inference and learning using TRW is now multithreaded, using openmp.
• Switched to using a newer version of Eigen

There is also far more detailed examples, including a full tutorial of how to train a CRF to do “semantic segmentation” on the Stanford Backgrounds dataset. Just using simple color, position, and Histogram of Gradient features, the error rates are 23%, which appear to be state of the art (and better than previous CRF based approaches.) It takes about 90 minutes to train on my 8-core machine, and processes new frames in a little over a second each.

For fun, I also ran this model on a video of someone driving from Alexandria into Georgetown. You can see that the results are far from perfect but are reasonably good. (Notice it successfully distinguishes trees and grass at 0:12)

I’m keen to have others use the code (what with the hundreds of hours spent writing it), so please send email if you have any issues.

The first thing I’d like to discuss is “surrogate likelihood” training.  (So far as I know, Martin Wainwright was the first person to give a name to this method.)

Background

Suppose we want to fit a Markov random field (MRF).  I’m writing this as a generative model with an MRF for simplicity– pretty much the same story holds with a Conditional Random Field in the discriminative setting.

Here, the first product is over all cliques/factors in the graph, and the second is over all single variables.  Now, it is convenient to note that MRFs can be seen as members of the exponential family

,

where

is a function consisting of indicator functions for each possible configuration of each clique and variable, and the log-partition function

.

ensures normalization.

Now, the log-partition function has the very important (and easy to show) property that the gradient is the expected value of .

With a graphical model, what does this mean?  Well, notice that the expected value of, say, will be exactly . Thus, the expected value of will be a vector containing all univariate and clique-wise marginals.  If we write this as , then we have

.

The usual story

Suppose we want to do maximum likelihood learning.  This means we want to set to maximize

If we want to use gradient ascent, we would just take a small step along the gradient.  This has a very intuitive form: it is the difference of the expected value of under the model to the expected value of under the current distribution.

.

.

Note the lovely property of moment matching here. If we have found a solution, then and so the expected value of under the current distribution will be exactly equal to that under the data.

Unfortunately, in a high-treewidth setting, we can’t compute the marginals.  That’s too bad.  However, we have all these lovely approximate inference algorithms (loopy belief propagation, tree-reweighted belief propagation, mean field, etc.).  Suppose we write the resulting approximate marginals as .  Then, instead of taking the above gradient step, why not instead just use

?

That’s all fine!  However, I often see people say/imply/write some or all of the following:

1. This is not guaranteed to converge.
2. There is no longer any well-defined objective function being maximized.
3. We can’t use line searches.
4. We have to use (possibly stochastic) gradient ascent.
5. This whole procedure is frightening and shouldn’t be mentioned in polite company.

I agree that we should view this procedure with some suspicion, but it gets far more than it deserves! The first four points, in my view, are simply wrong.

What’s missing

The critical thing that is missing from the above story is this: Approximate marginals come together with an approximate partition function!

That is, if you are computing approximate marginals using loopy belief propagation, mean-field, or tree-reweighted belief propagation, there is a well-defined approximate log-partition function such that

.

What this means is that you should think, not of approximating the likelihood gradient, but of approximating the likelihood itself. Specifically, what the above is really doing is optimizing the “surrogate likelihood”

What’s the gradient of this? It is

or exactly the gradient that was being used above. The advantage of doing things this way is that it is a normal optimization.  There is a well-defined objective. It can be plugged into a standard optimization routine, such as BFGS, which will probably be faster than gradient ascent.  Line searches guarantee convergence. is perfectly tractable to compute. In fact, if you have already computed approximate marginals, has almost no cost. Life is good.

The only counterargument I can think of is that mean-field and loopy BP can have different local optima, which might mean that a no-line-search-refuse-to-look-at-the-objective-function-just-follow-the-gradient-and-pray style optimization could be more robust, though I’d like to see that argument made…

I’m not sure of the history, but I think part of the reason this procedure has such a bad reputation (even from people that use it!) might be that it predates the “modern” understanding of inference procedures as producing approximate partition functions as well as approximate marginals.

make_energy_fun = @(x,f) @(y,w) energy(y,x,f,w);

My thoughts:

• This is great!  I’m creating an anonymous function which takes two parameters and returns an anonymous function taking two parameters which evaluates the energy with the original two parameters baked in. Then the (original) anonymous function is assigned to the variable make_energy_fun.
• But… in a civilized programming language wouldn’t this be, literally, an easy problem on a freshman problem set?
• …Why is it that we don’t use civilized programming languages, again?

Then, the usual rule for the quotient is that

This qualifies as non-memorizable. On the other hand, take the reciprocal of

.

This is simple enough (“the complex conjugate divided by the squared norm”), and we recover the rule for the quotient easily enough by multiplying with .

The same thing holds true for derivatives. I’ve never been able to remember that quotient rule from high-school; Namely that if , then

Ick. Instead, better to note that if then

along with the standard rule for differentiating products, so that if , then

.

Another case would be the “matrix quotient” . Of course, everyone already thinks of the matrix multiplication and inverse as separate operations– to do otherwise would be horrible– but I think that just proves the point…

(Although, I assume that computing as a single operation would be more numerically stable than first taking an explicit inverse. This might mean something to people who feel that mathematical notation ought to suggest an obvious stable implementation in IEEE floating point (if any).)

This is basically the simplest possible vision problem. We will train a conditional random field (CRF) to take some noisy image as input:

and produce marginals that well predict a binary image as output:

The noisy image is produced by setting where is random on and is a noise level as described in this paper. For now, we set which is a pretty challenging amount of noise, as you see above.

The main file, which does the learning described here can be found in the toolbox in examples/train_binary_denoisers.m.

First off, we will train a model using perturbation, with the univariate likelihood loss function, based on TRW inference, with a convergence threshold of 1e-5. We do this by typing:

>> train_binary_denoisers('pert_ul_trw_1e-5')
                                                        First-order
 Iteration  Func-count       f(x)        Step-size       optimality
     0           1         0.692759                         0.096
     1           3         0.686739       0.432616         0.0225
     2           5         0.682054             10         0.0182
     3           6         0.670785              1         0.0317
     4          10         0.542285        48.8796          0.932
     5          12         0.515509            0.1          0.965
     6          13         0.439039              1          0.355
     7          14         0.302082              1          0.279
     8          15         0.228832              1          0.471
     9          17         0.223659       0.344464         0.0159
    10          18         0.223422              1        0.00417
    11          19         0.223231              1        0.00269
    12          20         0.223227              1        0.00122
    13          22         0.223221        4.33583       0.000857
    14          23         0.223201              1        0.00121
    15          24         0.223138              1        0.00306
    16          25         0.223035              1        0.00509
    17          26         0.222903              1        0.00564
    18          27         0.222824              1         0.0035
    19          28         0.222806              1       0.000945
    20          29         0.222803              1       0.000798
    21          30         0.222802              1        0.00079
    22          31         0.222798              1        0.00111
    23          32         0.222788              1        0.00168
    24          33         0.222763              1        0.00255
    25          34         0.222707              1        0.00364
    26          35         0.222603              1        0.00435
    27          36         0.222479              1        0.00339
    28          37         0.222408              1        0.00117
    29          38         0.222394              1       9.64e-05
    30          39         0.222393              1       2.05e-05
    31          40         0.222393              1       4.06e-06
    32          41         0.222393              1       2.86e-07  

The final model trained has an error rate of 0.096. We can visualize the marginals produced by making an image where each pixel has an intensity proportional to the predicted probability that that pixel takes label “1″.

On the other hand, we might train a model using truncated fitting, with the univariate likelihood, and 5 iterations of TRW.

>> train_binary_denoisers('trunc_ul_trw_5')

Sparing you the details of the optimization, this yields a total error rate of .0984 and the marginals:

Thus, restricting to only 5 iterations pays only a small accuracy penalty compared to a right convergence threshold.

Or, we could fit using the surrogate conditional likelihood. (Here called E.M., though we don’t happen to have any hidden variables.)

>> train_binary_denoisers('em_trw_1e-5')

This yields an error rate of .1016, and the marginals:

There are many permutations of loss functions, inference algorithms, etc. (Some of which have not yet been published.) Rather than exhaust all the possibilities, I’ll just list a bunch of examples:

'pert_ul_trw_1e-5' (Perturbation + univariate likelihood + TRW with 1e-5 threshold)

'trunc_cl_trw_5' (Truncated Fitting + clique likelihood + TRW with 5 iterations)

'trunc_cl_mnf_5' (Truncated Fitting + clique likelihood + Mean Field with 5 iterations)

'trunc_em_trw_5' (Truncated EM, with TRW used to compute both upper and lower bounds on partition function + 5 iterations)

'em_trw_1e-5' (Regular EM, with TRW used to compute both upper and lower bounds on partition function + 1e-5 threshold)

'em_trw/mnf_1e-5' (Regular EM, with TRW used for upper bound and meanfield used for lower bound + 1e-5 threshold)

'pseudo' (Pseudolikelihood)

About the pseudolikelihood, let’s try it.

>> train_binary_denoisers(‘pseudo’)

This yields an near-change error rate of .44, and the marginals (produced by TRW)

Which is why you probably shouldn’t use the pseudolikelihood…

There are, at present, a bunch of limitations:

• All the inference algorithms are for marginal inference. No MAP inference, at all.
• The code handles pairwise graphs only
• All variables must have the same number of possible values.
• For tree-reweighted belief propagation, a single edge appearance probability must be used for all edges

For vision, these are usually no big deal. (Except if you are a MAP inference person. But that is not advisable.) In other domains, though, these might be showstoppers.

The code can be used in a bunch of different ways, depending on if you are looking for a specific tool to use, or a large framework.

• Just use the low-level [Inference] algorithms, namely A) Tree-Reweighted Belief propagation + variants (Loopy BP, TRW-S) or B) Mean-field. Take care of everything else yourself.
• Use the [Differentiation] methods (back-TRW or implicit differentiation) to calculate parameter gradients by providing your own loss functions. Do everything else on your own.
• Use the [Loss] methods (em, implicit_loss) to calculate parameter gradients by providing a true vector x and a loss name (univariate likelihood, clique likelihood, etc.) Unlike the above usages, these methods explicitly consider the conditional learning setting where one has an input and an output.
• Use the [CRF] methods to calculate calculate almost everything (deal with parameter ties for a specific type of model, etc.) These methods consider specific classes of CRFs and given and input, output, loss function, inference method, etc. give the parameter gradient. Employing this gradient in a learning framework is quite straightforward.

In making the notes, I used a couple innovations, which I am still debating the wisdom of. The first is the rule for calculating derivatives of scalar-valued functions of a matrix input . Traditionally, this is written like so:

if then .

I initially found the presence of the trace here baffling. However, there is the simple rule that

where is the matrix inner product. This puts the rule in the much more intuitive (to me!) form:

if then .

This seems more straightforward, but it comes at a cost. When working with the rule in the trace form, one often needs to do quite a bit of shuffling around of matrices. This is easy to do using the standard trace identities like

.

If we are to work with inner-products, we will require a similar set of rules. It isn’t too hard to show that there are “dual” identities like

which allow a similar shuffling with dot products. Still, these are certainly less easy to remember.

There are also a set of other rules that seem to be needed in practice, but aren’t included in standard texts. For example, if is a function that is applied elementwise to a matrix or vector (e.g. ), then

where is the elementwise product. This then requires other (very simple) identities for getting rid of the elementwise product, such as

.

Another issue with using dot products everywhere is the need to constantly convert between transposes and inner-products. (This issue comes up because I prefer a “all vectors are column vectors” convention) The never-ending debate of if we should write

or

seems to have particular importance here, and I’m not sure of the best choice.

At present, automatic differentiation involves a degree of pain. Typically, the user needs to write C or C++ or Fortran code and augment the code with “taping” commands to get it to work. There is also usually a significant computational overhead.

In a just world, there would be no pain, and there would be no computational overhead. In a just world, reverse-mod autodiff would work like this:

STEP 1 The user writes the function they want to optimize in a convenient high level language like Python.

This is easy, but slow, and provides no derivatives.

Consider a function that sums the elements in a matrix product. The user (apparently unaware that numpy provides matrix multiplication) writes:

    def fun(A,B):
        rez = 0
        for i in range(A.shape[0]):
            for j in range(B.shape[1]):
                rez += dot(A[i,:],B[:,j])
        return rez

STEP 2 The user feeds their high level function into a library. This library uses operator overloading magic to build an expression graph representation of the function. This expression graph is then used to generate efficient machine ode for both the function, and its derivatives.

The user merely should have to do something like

    A = numpy.random.rand(3,5)
    B = numpy.random.rand(5,4)
    dfun = compile_fun(fun,A,B)

They could then compute the function and derivatives (all in compiled code) using

    F,[dFdA,dFdB] = dfun(A,B)

Thus, the user gets everything, with out having to compromise: maximum performance, and maximum convenience.

Another simple example, mixing matrices and scalars:

def fun(A,B,a):
    return sum(sum((A*a-B)**2))
a = 2.
A = random.rand(15,22)
B = random.rand(15,22)
import etree
dfun = etree.compile_fun(fun,A,B,a)
F,[dFdA,dFdB,dFda] = dfun(A,B,a)

Consider the above matrix multiplication example, with 60×60 matrices. The library can generate the expression graph, transform that into a simple bytecode, than transform that bytecode into C code acceptably quickly– 11.97 seconds on my machine.  The code then runs very fast:  .0086 seconds as opposed to .0254 seconds for just running the original function or .0059 seconds for calling numpy’s matrix multiplication routine dot(A,B). (Which, of course, does not provide derivatives!)

There is, inevitably, one large problem:  horrific compilation times on large functions.  To take the C code and transform it into machine code, gcc takes 1116 seconds.  Why, you ask?  The reason is because gcc is trying to compile a single 36.5 MB function:

#include "math.h"
void rock(double* A, double* D){
A[12800]=A[0] * A[6400];
A[12801]=A[1] * A[6480];
A[12802]=A[12800] + A[12801];
A[12803]=A[2] * A[6560];
A[12804]=A[12802] + A[12803];
A[12805]=A[3] * A[6640];
// thousands of pages of same
D[12801] = D[12802];
D[1] += D[12801] * A[6480];
D[6480] += D[12801] * A[1];
D[0] += D[12800] * A[6400];
D[6400] += D[12800] * A[0];
}

Though this is very simple code, it seems that gcc still uses some super-linear algorithms, so it can’t handle large functions.

(To be clear, for small or medium sized problems, the code compiles very quickly.)

Now, clearly, I am doing something wrong. Frankly I am writing this largely in the hope that someone who actually knows something about compilers and programming languages can enlighten me.

1) Use a better compiler.  I’ve tried this.  I can turn off gcc’s optimization, which decreases compilation times, though to a still very slow level.  Alternatively, I could use a fast compiler.  tcc flies right through the code.  Unfortunately, the machine code that tcc generates is very slow– I think due to poor register allocation.

2) Don’t generate intermediate C code– directly generate machine code or assembly.  This would certainly work, but there is no way I am up to it. Maybe it would make sense to target the JVM?

3) Use one of the newfangled just in time virtual machine things, like LLVM or libJIT.  This seems promising, but after quite a bit of research I’m still really unsure about how well this is likely to work, or how to proceed.

4) Write a fast interpreter, and just pass the bytecode to this.  I’ve tried this as well.  This is basically the strategy used by some autodiff packages, e.g. cppad. This can get reasonable performance, but will never compete with native code. The reason is– I think– that the interpreter is basically a giant switch statement, and all the branching screws up the pipeline on the CPU.

Anyway, the code is available as etree.py, and a tiny file mycode.h. (Let’s say it’s available under the MIT license, and the condition that you not laugh too much at my programming skills.) To try it out, just start python and do

import etree
etree.runtests()

At this point, the library is useful (I use it!) but only for relatively small problems, containing no more than, say, a few thousand variables.

If anyone has ideas for better compiler flags, or knows about the feasibility of targeting LLVM or libJIT instead of C code, please get in touch.

Disclaimer: There are a bunch of minor issues with the library. These are all fixable, but it isn’t really worth the effort unless the compilation times can be dealt with.

• The function cannot branch. (No if statements).
• If you want to multiply a matrix by a scalar, the scalar must come after the matrix.
• I know it works on 32 bit ubuntu with everything installed via synaptic package manager, and on 64 bit OS X, simply using Sage (with this patch to make scipy/weave work). If you want to play around with this, but aren’t a python programmer, I would suggest using Sage.

We have the symbol ,with the interpretation

for some number ,

but there doesn’t appear to exist a symbol (Here, I use a boxed question mark: to denote the symbol I claim doesn’t exist) with the interpretation

for some number .

This pains me.  People sometimes have to resort to writing something like

“

where the constants are (in general) different on lines (1) and (2).”

Even worse (or maybe not?), sometimes people seem to leave exponents lying around when they otherwise wouldn’t, e.g. write

.

Exhibit B:

We have no symbol meaning “normalized sum”.  How many thousands of times have you seen some variant of

“

where “?

Why do we need to define ?  Can’t we just use another mystery symbol to write

?

In some situations you could use , but that doesn’t always really work and is rarely done.

The basic idea is to fit a FoE (Field of Experts) image prior such that when one takes a very few gradient descent steps on a denoising posterior, the results are accurate. From the abstract:

We argue that through appropriate training, a MRF/CRF model can be trained to perform very well on a suboptimal inference algorithm. The model is trained together with a fast inference algorithm through an optimization of a loss function [...] We apply the proposed method to an image denoising application [...] with a 1-4 iteration gradient descent inference algorithm. [...] the proposed training approach obtains an improved benchmark performance as well as a 1000-3000 times speedup compared to the Fields of Experts MRF.

The implausible-sounding 1000-fold speedup comes simply from using only 4 iterations of gradient descent rather than several thousand. (Incidentally, L-BFGS requires far fewer iterations for this problem.) The results are a bit better than the generative FoE model– that takes much more work for training and inference.

I have every confidence that this does work well, and similar strategies could probably be used to create fast inference models/algorithms for many different problems. My thesis was largely an attempt to do the same thing for marginal, rather than MAP inference.

The disturbing/embarrassing question, for me, is does this really have anything to do with probabilistic modeling any more? Formally speaking, a probability density is being fit, but I doubt it would transfer to, say, inpainting, or that samples from the density would look like natural images. The best interpretation of what is happening might be that one is simply fitting a big, nonlinear, black box function approximation.

It seems that the more effort we expend to wring the best performance out of a probabilistic model, the less “probabilistic” it is.

P.S. Some of my friends have invited me to never mention autodiff again, ever, but this is one of the many papers where I think the learning optimization would be made much easier/faster by using it.