CRF Toolbox Updated

I updated the code for my Graphical Models / Conditional Random Fields toolbox This is a Matlab toolbox, though almost all the real work is done in compiled C++ for efficiency. The main improvements are:

  • Lots of bugfixes.
  • Various small improvements in speed.
  • A unified CRF training interface to make things easier for those not training on images
  • Binaries are now provided for Linux as well as OS X.
  • The code for inference and learning using TRW is now multithreaded, using openmp.
  • Switched to using a newer version of Eigen

There is also far more detailed examples, including a full tutorial of how to train a CRF to do “semantic segmentation” on the Stanford Backgrounds dataset. Just using simple color, position, and Histogram of Gradient features, the error rates are 23%, which appear to be state of the art (and better than previous CRF based approaches.) It takes about 90 minutes to train on my 8-core machine, and processes new frames in a little over a second each.

For fun, I also ran this model on a video of someone driving from Alexandria into Georgetown. You can see that the results are far from perfect but are reasonably good. (Notice it successfully distinguishes trees and grass at 0:12)

I’m keen to have others use the code (what with the hundreds of hours spent writing it), so please send email if you have any issues.

69 thoughts on “CRF Toolbox Updated

  1. I’m eager to support Octave. I hadn’t done so thus far as I was under the impression that Octave couldn’t use mex files. I see now that isn’t true, but I have relied on matlab’s ability to pass arrays to C++ code by pointer. Since Octave makes copies beforehand, this means (I think!) it won’t work out of the box. The other reason I haven’t focused on Octave the lack of parfor. With multithreaded TRW, though, this is less of an issue.

    Thanks for the pointer to I’ll add it there.

  2. Hi Justin, thanks a lot for sharing the code mate, it is a great piece of software. I would like to ask you, how can you train the CRF assuming your “image” (for not saying data) contains more than 3 dimensions? In one of the examples you provide (example_backgrounds.m), you perform segmentation using RGB images i.e., data with 3 dimensions or features, but what if you have images (well,… data) with more than 3 dimensions or features? How can you train the CRF model with your software?, how do you generate the “efeats” matrix.

    Thanks a lot for the help!


  3. Hi Karin. Glad to hear you find the code useful. You can certainly apply it to multispectral images, if that is what you are asking. In terms of the backgrounds dataset example, as long as you create the matrices feats{:} and efeats{:} for each example image, everything should work fine. This might require some minor modification to “featurize_im.m” and ” edgeify_im.m” to handle multispectal images. (e.g. rgb2gray won’t work for multispectral.) I think, however, that you could still get away with using simple edge features based on pixel intensity differences, i.e. the same

    edge_params = {{‘const’},{‘diffthresh’},{‘pairtypes’}};
    efeats{n} = edgeify_im(ims{n},edge_params,models{n}.pairs,models{n}.pairtype);

    used in the backgrounds example.

    The best thing to do, though, would be to decide what edge features would be best for your application, and write your own simple function to replace edgify_im.m.

  4. Hi Justin, thanks a lot for the reply.

    I am working with several datasets and in these datasets I know the connections between every variable and feature, the problem that I have is that I don’t know how to create the efeats matrix, I checked the code, but I couldn’t get how to build that matrix from scratch (without the edgeify_im function),

    Hope you can help me with this: suppose you have a dataset with “N” examples where each example contains 6 labels “yi”. And you know that the connections between them are: y3-y2-y4-y5-y1-y6. Each of these labels is binary (so let’s pretend each example corresponds to a 1-by-6 binary image). Every “pixel” here is connected or described by a set of 72 features “xj”. All of the “yi’s” share the same features. So each “yi” is connected to every “xj” (x1, x2,…,x72). And there are no connections among the “xj’s”.

    It is basically a linear chain where each label yi in the chain shares the same features (x1 to x72). How can you model this?

    Thanks in advance,

  5. Seems like no problem to me– just create a standard “chain” CRF and use the same features for all the “yi”, but don’t use parameter ties, so that the different “yi” can have their own relationship with the “xi”.

    With the existing code base, this could be done by modeling a “1×6” image (gridmodel). Since the code is set up to use feature ties, you could hack this by using 72×6 features for each “yi”, where only one block is nonzero. (The first block for “y1”, the second for “y2”,etc.).

  6. Hi Justin, I have been running your code on Linux, but every time I run it I receive different outputs (different test error numbers, different classes, etc.) even when of course, the training and test datasets are the same, always!

    I am using Belief Prop, and the weird(est) thing is that the outputs are the same when I run it on windows! (but who wants to use windows?),

    Any idea about what’s going on?


  7. Hi Liam,

    Do you use a random function anywhere in your code? (maybe when you generate the train/test splits?)

    Do you initialize the seeds before calling any random function? (you have to use the rgn function)


  8. Hello there,

    yeah I use random functions twice in the code. But before I call such functions I use rng(1); like you mention.

    I don’t know why the program works properly in Windows but not in linux. In linux even if I remove the random functions and leave the splits for training and testing fixed, I get different results.


  9. … with respect to my previous comment:

    I already checked the training/testing splits’ indexes in linux, they are always the same, which means that the rng(1); functions are working properly. So maybe there is another random function somewhere else in the code, I just can’t find it…


  10. Hi Liam. There “shouldn’t” be any other random numbers used, since parameters are just initialized to zero. Are you using the multithreaded trw code? It is possible that non-determinism is leaking in there somehow, so try switching to the non-multithreaded version (change “trwpll” to “trw”) and see if the problem goes away. If not, could you create the simplest example you can that reproduces the problem?

    Also, if you’ve compiled binaries for windows, I’d appreciate it if you could send them to me so I can include them in the package.

  11. Hi Thank you for your code. I am using this code on windows. I tried to run the code for ‘Background Dataset’. But I am getting an error ‘Undefined function or method ‘reflectim_helper’ for input arguments of type ‘double” should I recompile your code into windows? How can I get rid of this error?

  12. Hi Siyam,

    I think you just need to compile the code. (I don’t have a windows machine and so don’t provide binaries for windows.) Essentially, this would be done just by typing “compile” from the main toolbox directory.

  13. Hi Justin,
    I am using your code on octave on linux. On trying to run the code for ‘Background Dataset’, I’m getting an error “Undefined function or method ‘reflectim_helper”. How can I solve this error ? Please reply asap!
    Thanks in advance.

  14. Hello Justin
    I am trying to make some changes in your CRF toolbox, so I will be able to use it in my project.
    Could you please tell me, in which function inside your toolbox the CRF conditional distribution is implemented?
    I want to add some temporal information to this formula

    Thank you in advance

  15. Hi Monika, Sorry for the slow reply. The problem you are experiencing is that reflectim_helper is implemented in C++, not in matlab/octave. Thus, to make it available it needs to be compiled. In matlab, this would be done by something like “mex reflectim_helper.cpp”. (There is a script compile.m that compiles all the needed files.) Unfortunately, I don’t know how to do this for octave, but please do get in touch if you figure it out, as I’ve heard from several people who would like octave support…

  16. Hi Jeiran,

    Roughly speaking, the conditional distribution is implemented in the files in the CRF/ subdirectory. For example, crf_linear_linear.m maps the features for each point/edge to MRF parameters using a linear mapping for both univariate and edge features. Thus, I think you would likely want to proceed by implementing an alternative crf_something.m file, and then slightly modifying train_crf.m to take your new crf_type variable and call the appropriate crf_something function. (Search for “if strcmp(crf_type,’linear_linear’)” addn then add your new type.) Good luck!

  17. Hi Justin,
    Can I use your toolbox in a C/C++ program? I’m a doing a C++ project and I’ve been having great trouble in finding some CRF toolbox for images. I’ve found some c++ toolboxes, but i think they are all for natural language processing problems. I’m very new to this, so I don’t know how to use a toolbox intended for NLP in my computer vision project. So can your toolbox be called from a c++ program? Thanks a lot!!!

  18. Hi Diana. Unfortunately, probably not– only the slowest parts of the code are in C++, while the majority is in Matlab.

  19. Hi Justin,

    I am new to CRF implementations, so i tried the examples in your toolbox before I edit it to suit my needs. I am repeatedly getting Undefined function ‘reflectim_helper’ for input arguments of type ‘double’. I have recompiled the binaries, includes ll the files and folders. How do I solve this issue. I am using windows 64 bit.

    Thank a lot !!

  20. Hi Janani,

    That error means that matlab is not able to find the mex file corresponding to Vision/reflectim_helper.cpp. Can you verify that Vision/reflectim_helper.mexw64 exists? If not, that means compilation hasn’t worked correctly. If so, the only other issue is that your path might not be set correctly. This can be done by typing “addpath(genpath(‘.’))’ in the main JGMT directory. Finally, if you’ve really compiled all the mex files, it would be great if you could email them to me so I can include them for others. My email is (myfirstname).(mylastname) Thanks!

  21. Hi Justin

    I’m trying to run the demo code does not work. Displays the following error: undefined function ‘kfold_sets’ for input arguments of type ‘double’. I do not know where to locate this function code.

  22. Sorry, I found the code kfolds_sets function but I get an error with frunción “feats” as it says it is undefined.

    I’m using Linux to run the sample.

    thank you very much

  23. Hello Justin

    I am new to CRF implementation and understanding, I found your code is very help for the beginners to learn and understand about CRF. I tried to run the code usinf MATLAB 2011 on windows PC. when i am trying to use train_crf function, it gives me error. i tired best to my knowledge but i couldn’t solve.. the error is e:\\justinsgraphicalmodelstoolboxpublic\differentiation\eigenstuff.cpp(8) : fatal error C1083: Cannot open include file: ‘Eigen/Eigen’: No such file or directory
    E:\PROGRAM FILES\MATLAB\R2011A\BIN\MEX.PL: Error: Compile of trw_bprop_scheduled_helper1.cpp’ failed.

    please could help me the solve this error..

  24. Hi Adnan,

    Are you trying to compile the code? You should be able to use the included binaries, which is typically the easiest way to avoid compilation problems…

  25. Hello Justin, Thanks i solved that error.

    i want to know more about this code, could you please tell me that these results are published in any paper?

  26. Hello Justin,

    I want to check your CRF toolbox on my idea. So i want to discuss some related to that on email, so is it comfortable for you?

  27. I want to use this code for activity recognition using depth image, is it possible to use? what would be labels as a input in that case?

  28. How to you encode an “activity”? If you just have a finite set, it isn’t clear to me why you would want to use a CRF at all (as opposed to just a regular classifier).

  29. Actually i’m interested to use CRF as classifier .. i have depth sequence of a person jumping, 500 frames. i made a feature vector and i am don’t want to use edge feature in this case i think (correct me if i am wrong) . So what would i use as input label?

  30. Hey Justin

    Thanks a lot for making this available online. I was trying to figure out how to use your tool on an activity dataset I found online ( which usually stores data in following format:
    activity label, start_time, sensor_id, sensor_status
    I am not quite sure the data format for test and training sets the function accept? Could you please suggest something as to how I can use this dataset with your toolkit for activity recognition?


  31. I get an error in the second example with frunción “feats” as it says it is undefined.

    “Undefined function or variable “feats”.
    Error in example_backgrounds (line 72)
    feats_train = feats(who_train);”

    Thanks in advance

  32. the last error is resolved by correcting the imsdir and the labdir
    but I have this error :

    Error using hog (line 50)
    I must be a double 2 or 3 dim array.

    Error in featurize_im (line 195)
    H = hog(single(reflectim(im,rez)),rez,9,10);

    Error in example_backgrounds>(parfor body) (line 30)
    feats{n} = featurize_im(ims{n},feat_params);

    Error in example_backgrounds (line 22)
    parfor n=1:N

    Caused by:
    Error using hog1
    I must be a double 2 or 3 dim array.

    thanks in advance

  33. I think the error with hog is due to a change in the format used by the external toolbox. I will look into this, but it looks like it can be dealt with by removing the cast to single precision.

  34. Hi Justin,

    Thanks for sharing the toolbox. I am looking for a CRF implementation where one minimizes a custom loss function. In my case I would like to learn a CRF while optimizing area under ROC curve. Is this possible with your toolbox?


  35. HI Vlad,

    It won’t work off the shelf, but the modifications needed should be fairly minimal. You’d need to modify the code to accept “aoc” in the current places where the losses “ul” and “cl” are used. The only substantial change needed is computing the loss and the gradient of the loss with respect to marginals.

    The easiest way to implement this is probably to start training a model as if “aoc” was a valid loss function and modify the code in each of the places an error crops up. (There will only be a few.)

  36. Hi,

    Having the same issues as Work York. Is there any third party code on which this toolbox depends? E.g., the functions fspecial or feats are undefined.

    Would be grateful if you could help me get started, thanks in advance!

  37. This does depend on third party code, but you will automatically be prompted to download it if it can’t be found. I think the issue you are experiencing is due to a change in some of this third-party code. Work York’s first problem is resolved as in the following message, by making sure imsdir and labdir are pointing to the correct place. As for the second problem, try changing this line:




    I think that the external toolbox that is being used has changed in a recent version to use doubles rather than singles.

  38. Hi dear justin
    Thanks for sharing your code.
    I cant get your result with pseudo loss on background dataset even with independent initialization.
    could you please help me?

  39. I can certainly try, but I’ll need more information. How do your results vary? Are you using the example_background.m file, and just changing the loss_spec value to the pseudolikelihood? Often a problem that appears here is that at test time you should switch to a different loss (something that requires inference) since the pseudolikelihood loss doesn’t need to do inference itself. (‘trunc_cl_trw_5’ or whatever)

  40. yes, I use example_background. You mean the model should learn the params with ‘pseudo’ loss and infer with another?

  41. Well, I think the pseudolikelihood is a bad loss function, generally speaking, but if you want to use it, then yes: you’ll need to use a different loss function for inference. The reason is that you don’t need to do inference to compute the pseudolikelihood (that’s the point of it, after all) and so it doesn’t provide any marginals. Alternatively, you can just use a “pure” inference routine without computing a loss function at all.

  42. Hi Justin,

    first of all thanks a lot for sharing your. It was really easy to use and the paper associated explain very well many things. I am not a machine learning expert, I am working on image segmentation for the first time since a month, on a Curb Detection problem.

    I am using your example code for binary denoising because for now I am just trying to identify the position of the curb with respect to the street/sidewalk/buildings around. I am not using RGB-D images, but using my own code I extracted features from a point cloud generated with the Velodyne sensor. For now I am using just one feature for each point, each feature represents the score for this specific point to be part of a curb or not.
    So, to finish to explain my problem, I am currently using as input only the very same feature I have for every point in the cloud, running the learning step without any edge features.

    The first thing that I need to do is adding other unary features, again extracted with my own code, so I will need to import my new features from my code.
    – How would you suggested to do it, which part of the code I need to modify? (I saw the example_background.m code, in which you create the feature manually (unary and edge) with two functions.
    The second thing I need to integrate an edge feature that take into account connection with 8 neighbors. As far as I understood you use pairwise 4 connected grid to build the model.
    – How can I modify it? This new feature will be calculated inside the tool, so I need to create it starting with the first one that I am using now computed in my code. How and where can I add it inside the toolbox?

    I know that I am asking a lot, but it would be really helpful if you could help me. the deadline is so close now… 🙂

    Thanks a lot in advance.

  43. Hello Sir,
    Please help me I am a B.Tech second year student and want to learn about working on conditional random field using matlab or r tool or any other way but i have no idea how to start please provide guidance.

  44. Hello Sir,
    Please help me I am MSc student in AI and i want change graph model. i want ask you can i do that?
    my graph is a tree
    thanks a lot

  45. Sorry, I’m afraid that without a more detailed description of what you are trying to do, I can’t give a meaningful answer to your question.

  46. My work is assigning social role to different characters in video. in each video we have reference role that is most important role in video and every other roles have interaction with it. we have unary and pairwise features. so surly we must have a graph for our problem but we can’t use full graph because we can’t inference tractably and calculate necessary parameters so we should simplify graph. for this purpose we just consider interaction of different characters (roles) with reference role(most important role of video ) so we have a tree that his root is reference role and leafs are other characters (roles). now as we can see instead of full graph or complicated graph we have a tree that we can do tractable inference on it and calculate parameters.
    I looked at your gridmodel code and i thunk that i should change this code and implement my graph but i don’t know there is any other place in your package that i should change ?
    please help me
    thank you so much

  47. Hi Sayyed, yes that sounds about right. To use the toolbox with a different graph, you just need to create the right graph structure. You can base this on gridmodel.m, and use some of the support functions– essentially you just need to create a list of edges. Then, everything should work in roughly the same way.

  48. Hi Dear Justin
    Thanks for your great toolbox.
    I have a question on the function train_crf.m line 232.
    Is it right L = L + reg*sum(params).^2; in your code?
    Isn’t it to be L = L + reg*sum(params.^2);


  49. Hi sir
    I changed your gridmodel to Mygridmodel and execute program but i have a following error :
    error(‘theta_i should have size [model.nvals^2 size(model.pairs,1)]’);
    Please help me, thank you so much

  50. Christine,

    Thanks for writing this toolbox, it looks like one of the more useful ones out there.

    I’m trying to modify the gridmodel.m function to be able to handle graphs with any clique groupings, i.e.. not just horizontal and vertical and also with any number of vertices in a clique. However, the code seems to be limited to having one pair of vertices per clique, since gridmodel.m sets tree2clique_hor to the model.pairs index, “ind_cnt”, and because trw_bprop_scheduled.m uses the values in tree2clique to index model.pairs.

    Modifying this code has proven to be difficult, since it isn’t clear how the variables in the “model” structure should be defined. Can you provide a detailed definition of the variables in the “model” structure: treenum, tree2clique, treeschedule, N1, and N2, ncliques, with hints on how to modify them to make this a CRF with a general structure?

    The model I’m interested in uses a CRF to model a subset of pixels in an image, which is represented as a 1-by-N vector, not a n-by-m image (N<=nxm). So, the grid model can't be used here. The clique neighborhood for a pixel can be any K pixels from the subset of pixels, not just the adjacent neighbors.

    I have adapted the gridmodel.m script to what I think it should be, but I feel like I don't understand the "model" variables well enough to trust any model that is learned. Any feedback will be greatly appreciated.

    Thank you.

  51. Hi Eran,

    Unfortunately, generalizing the toolbox beyond pairs would be *extremely* complicated. All the inference algorithms and such are also pairwise specialized, so you’d almost be starting over. If you want factors of triplets or more, consider using MARBL:

    It can take arbitrary factors, and the documentation is clearer. (You just input the factors in a simple text file format.)

  52. Thanks for the feedback. Does MARBL allow the training examples to be different sizes and for the user to supply the edge features?

  53. Yes, it allows both of those things. The idea is, for conditional random fields, you specify a “factor type” for each factor. All factors of the same type then have the same mapping from input features to output potentials. So, the graph structure can vary arbitrarily between different examples.

  54. Dear Justin,
    I have a question about the used formula of CRF in your toolbox. I would be really grateful if you could help me. Thank you so very much.

  55. Dear Sir,

    Thanks so much for sharing the code…
    I’m new to Matlab and I have been trying my best to get the code work… However, I cant get away from the following errors:

    – Error using gradientMag (line 50)
    I has incorrect type.

    – Error in hog (line 71)
    [M,O] = gradientMag( I );

    – Error in featurize_im (line 195)
    H = hog(double(reflectim(im,rez)),rez,9,10);

    – Error in example_backgrounds (line 22)
    parfor n=1:N

    May I know how I can fix it? Your help will be much appreciated…
    Thank You!

    Best Regards,

  56. Hi Justin,

    Thank you for the efforts and for the great work. I was wondering if you have any documentation of your code available on hand? I would really appreciate it.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s