CRF Toolbox Updated

I updated the code for my Graphical Models / Conditional Random Fields toolbox This is a Matlab toolbox, though almost all the real work is done in compiled C++ for efficiency. The main improvements are:

  • Lots of bugfixes.
  • Various small improvements in speed.
  • A unified CRF training interface to make things easier for those not training on images
  • Binaries are now provided for Linux as well as OS X.
  • The code for inference and learning using TRW is now multithreaded, using openmp.
  • Switched to using a newer version of Eigen

There is also far more detailed examples, including a full tutorial of how to train a CRF to do “semantic segmentation” on the Stanford Backgrounds dataset. Just using simple color, position, and Histogram of Gradient features, the error rates are 23%, which appear to be state of the art (and better than previous CRF based approaches.) It takes about 90 minutes to train on my 8-core machine, and processes new frames in a little over a second each.

For fun, I also ran this model on a video of someone driving from Alexandria into Georgetown. You can see that the results are far from perfect but are reasonably good. (Notice it successfully distinguishes trees and grass at 0:12)

I’m keen to have others use the code (what with the hundreds of hours spent writing it), so please send email if you have any issues.

This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink.

69 Responses to CRF Toolbox Updated

  1. Hi Justin, thank you for sharing your code.

    Does it also work with Octave?

    If you want your package to have more attention, add it to, there are already a few other CRF implementations:

  2. justindomke says:

    I’m eager to support Octave. I hadn’t done so thus far as I was under the impression that Octave couldn’t use mex files. I see now that isn’t true, but I have relied on matlab’s ability to pass arrays to C++ code by pointer. Since Octave makes copies beforehand, this means (I think!) it won’t work out of the box. The other reason I haven’t focused on Octave the lack of parfor. With multithreaded TRW, though, this is less of an issue.

    Thanks for the pointer to I’ll add it there.

  3. karin zachinelly says:

    Hi Justin, thanks a lot for sharing the code mate, it is a great piece of software. I would like to ask you, how can you train the CRF assuming your “image” (for not saying data) contains more than 3 dimensions? In one of the examples you provide (example_backgrounds.m), you perform segmentation using RGB images i.e., data with 3 dimensions or features, but what if you have images (well,… data) with more than 3 dimensions or features? How can you train the CRF model with your software?, how do you generate the “efeats” matrix.

    Thanks a lot for the help!


  4. justindomke says:

    Hi Karin. Glad to hear you find the code useful. You can certainly apply it to multispectral images, if that is what you are asking. In terms of the backgrounds dataset example, as long as you create the matrices feats{:} and efeats{:} for each example image, everything should work fine. This might require some minor modification to “featurize_im.m” and ” edgeify_im.m” to handle multispectal images. (e.g. rgb2gray won’t work for multispectral.) I think, however, that you could still get away with using simple edge features based on pixel intensity differences, i.e. the same

    edge_params = {{‘const’},{‘diffthresh’},{‘pairtypes’}};
    efeats{n} = edgeify_im(ims{n},edge_params,models{n}.pairs,models{n}.pairtype);

    used in the backgrounds example.

    The best thing to do, though, would be to decide what edge features would be best for your application, and write your own simple function to replace edgify_im.m.

  5. karin zachinelly says:

    Hi Justin, thanks a lot for the reply.

    I am working with several datasets and in these datasets I know the connections between every variable and feature, the problem that I have is that I don’t know how to create the efeats matrix, I checked the code, but I couldn’t get how to build that matrix from scratch (without the edgeify_im function),

    Hope you can help me with this: suppose you have a dataset with “N” examples where each example contains 6 labels “yi”. And you know that the connections between them are: y3-y2-y4-y5-y1-y6. Each of these labels is binary (so let’s pretend each example corresponds to a 1-by-6 binary image). Every “pixel” here is connected or described by a set of 72 features “xj”. All of the “yi’s” share the same features. So each “yi” is connected to every “xj” (x1, x2,…,x72). And there are no connections among the “xj’s”.

    It is basically a linear chain where each label yi in the chain shares the same features (x1 to x72). How can you model this?

    Thanks in advance,

  6. justindomke says:

    Seems like no problem to me– just create a standard “chain” CRF and use the same features for all the “yi”, but don’t use parameter ties, so that the different “yi” can have their own relationship with the “xi”.

    With the existing code base, this could be done by modeling a “1×6” image (gridmodel). Since the code is set up to use feature ties, you could hack this by using 72×6 features for each “yi”, where only one block is nonzero. (The first block for “y1”, the second for “y2”,etc.).

  7. karin zachinelly says:

    All right, I am going to try then 🙂

    Cheers and thanks a lot for all of the help

  8. Liam Parker says:

    Hi Justin, I have been running your code on Linux, but every time I run it I receive different outputs (different test error numbers, different classes, etc.) even when of course, the training and test datasets are the same, always!

    I am using Belief Prop, and the weird(est) thing is that the outputs are the same when I run it on windows! (but who wants to use windows?),

    Any idea about what’s going on?


  9. karin zachinelly says:

    Hi Liam,

    Do you use a random function anywhere in your code? (maybe when you generate the train/test splits?)

    Do you initialize the seeds before calling any random function? (you have to use the rgn function)


  10. Liam Parker says:

    Hello there,

    yeah I use random functions twice in the code. But before I call such functions I use rng(1); like you mention.

    I don’t know why the program works properly in Windows but not in linux. In linux even if I remove the random functions and leave the splits for training and testing fixed, I get different results.


  11. Liam Parker says:

    … with respect to my previous comment:

    I already checked the training/testing splits’ indexes in linux, they are always the same, which means that the rng(1); functions are working properly. So maybe there is another random function somewhere else in the code, I just can’t find it…


  12. justindomke says:

    Hi Liam. There “shouldn’t” be any other random numbers used, since parameters are just initialized to zero. Are you using the multithreaded trw code? It is possible that non-determinism is leaking in there somehow, so try switching to the non-multithreaded version (change “trwpll” to “trw”) and see if the problem goes away. If not, could you create the simplest example you can that reproduces the problem?

    Also, if you’ve compiled binaries for windows, I’d appreciate it if you could send them to me so I can include them in the package.

  13. siyam says:

    Hi Thank you for your code. I am using this code on windows. I tried to run the code for ‘Background Dataset’. But I am getting an error ‘Undefined function or method ‘reflectim_helper’ for input arguments of type ‘double” should I recompile your code into windows? How can I get rid of this error?

  14. justindomke says:

    Hi Siyam,

    I think you just need to compile the code. (I don’t have a windows machine and so don’t provide binaries for windows.) Essentially, this would be done just by typing “compile” from the main toolbox directory.

  15. Monika says:

    Hi Justin,
    I am using your code on octave on linux. On trying to run the code for ‘Background Dataset’, I’m getting an error “Undefined function or method ‘reflectim_helper”. How can I solve this error ? Please reply asap!
    Thanks in advance.

  16. Jeiran Choupan says:

    Hello Justin
    I am trying to make some changes in your CRF toolbox, so I will be able to use it in my project.
    Could you please tell me, in which function inside your toolbox the CRF conditional distribution is implemented?
    I want to add some temporal information to this formula

    Thank you in advance

  17. justindomke says:

    Hi Monika, Sorry for the slow reply. The problem you are experiencing is that reflectim_helper is implemented in C++, not in matlab/octave. Thus, to make it available it needs to be compiled. In matlab, this would be done by something like “mex reflectim_helper.cpp”. (There is a script compile.m that compiles all the needed files.) Unfortunately, I don’t know how to do this for octave, but please do get in touch if you figure it out, as I’ve heard from several people who would like octave support…

  18. justindomke says:

    Hi Jeiran,

    Roughly speaking, the conditional distribution is implemented in the files in the CRF/ subdirectory. For example, crf_linear_linear.m maps the features for each point/edge to MRF parameters using a linear mapping for both univariate and edge features. Thus, I think you would likely want to proceed by implementing an alternative crf_something.m file, and then slightly modifying train_crf.m to take your new crf_type variable and call the appropriate crf_something function. (Search for “if strcmp(crf_type,’linear_linear’)” addn then add your new type.) Good luck!

  19. Diana says:

    Hi Justin,
    Can I use your toolbox in a C/C++ program? I’m a doing a C++ project and I’ve been having great trouble in finding some CRF toolbox for images. I’ve found some c++ toolboxes, but i think they are all for natural language processing problems. I’m very new to this, so I don’t know how to use a toolbox intended for NLP in my computer vision project. So can your toolbox be called from a c++ program? Thanks a lot!!!

  20. justindomke says:

    Hi Diana. Unfortunately, probably not– only the slowest parts of the code are in C++, while the majority is in Matlab.

  21. Janani says:

    Hi Justin,

    I am new to CRF implementations, so i tried the examples in your toolbox before I edit it to suit my needs. I am repeatedly getting Undefined function ‘reflectim_helper’ for input arguments of type ‘double’. I have recompiled the binaries, includes ll the files and folders. How do I solve this issue. I am using windows 64 bit.

    Thank a lot !!

  22. justindomke says:

    Hi Janani,

    That error means that matlab is not able to find the mex file corresponding to Vision/reflectim_helper.cpp. Can you verify that Vision/reflectim_helper.mexw64 exists? If not, that means compilation hasn’t worked correctly. If so, the only other issue is that your path might not be set correctly. This can be done by typing “addpath(genpath(‘.’))’ in the main JGMT directory. Finally, if you’ve really compiled all the mex files, it would be great if you could email them to me so I can include them for others. My email is (myfirstname).(mylastname) Thanks!

  23. Mario says:

    Hi Justin

    I’m trying to run the demo code does not work. Displays the following error: undefined function ‘kfold_sets’ for input arguments of type ‘double’. I do not know where to locate this function code.

  24. Mario says:

    Sorry, I found the code kfolds_sets function but I get an error with frunción “feats” as it says it is undefined.

    I’m using Linux to run the sample.

    thank you very much

  25. Adnan says:

    Hello Justin

    I am new to CRF implementation and understanding, I found your code is very help for the beginners to learn and understand about CRF. I tried to run the code usinf MATLAB 2011 on windows PC. when i am trying to use train_crf function, it gives me error. i tired best to my knowledge but i couldn’t solve.. the error is e:\\justinsgraphicalmodelstoolboxpublic\differentiation\eigenstuff.cpp(8) : fatal error C1083: Cannot open include file: ‘Eigen/Eigen’: No such file or directory
    E:\PROGRAM FILES\MATLAB\R2011A\BIN\MEX.PL: Error: Compile of trw_bprop_scheduled_helper1.cpp’ failed.

    please could help me the solve this error..

  26. justindomke says:

    Hi Adnan,

    Are you trying to compile the code? You should be able to use the included binaries, which is typically the easiest way to avoid compilation problems…

  27. Adnan says:

    Hello Justin, Thanks i solved that error.

    i want to know more about this code, could you please tell me that these results are published in any paper?

  28. justindomke says:

    HI Adnan,

    Glad to hear it. Yes, the results are published in this paper:

  29. Adnan says:

    Hello Justin,

    I want to check your CRF toolbox on my idea. So i want to discuss some related to that on email, so is it comfortable for you?

  30. saleem says:

    I want to use this code for activity recognition using depth image, is it possible to use? what would be labels as a input in that case?

  31. justindomke says:

    How to you encode an “activity”? If you just have a finite set, it isn’t clear to me why you would want to use a CRF at all (as opposed to just a regular classifier).

  32. saleem says:

    Actually i’m interested to use CRF as classifier .. i have depth sequence of a person jumping, 500 frames. i made a feature vector and i am don’t want to use edge feature in this case i think (correct me if i am wrong) . So what would i use as input label?

  33. Megha says:

    Hey Justin

    Thanks a lot for making this available online. I was trying to figure out how to use your tool on an activity dataset I found online ( which usually stores data in following format:
    activity label, start_time, sensor_id, sensor_status
    I am not quite sure the data format for test and training sets the function accept? Could you please suggest something as to how I can use this dataset with your toolkit for activity recognition?


  34. Adnan says:

    please reply to the quires. it would be helpful for everyone.

  35. Work York says:

    I get an error in the second example with frunción “feats” as it says it is undefined.

    “Undefined function or variable “feats”.
    Error in example_backgrounds (line 72)
    feats_train = feats(who_train);”

    Thanks in advance

  36. Work York says:

    the last error is resolved by correcting the imsdir and the labdir
    but I have this error :

    Error using hog (line 50)
    I must be a double 2 or 3 dim array.

    Error in featurize_im (line 195)
    H = hog(single(reflectim(im,rez)),rez,9,10);

    Error in example_backgrounds>(parfor body) (line 30)
    feats{n} = featurize_im(ims{n},feat_params);

    Error in example_backgrounds (line 22)
    parfor n=1:N

    Caused by:
    Error using hog1
    I must be a double 2 or 3 dim array.

    thanks in advance

  37. justindomke says:

    I think the error with hog is due to a change in the format used by the external toolbox. I will look into this, but it looks like it can be dealt with by removing the cast to single precision.

  38. vvladd says:

    Hi Justin,

    Thanks for sharing the toolbox. I am looking for a CRF implementation where one minimizes a custom loss function. In my case I would like to learn a CRF while optimizing area under ROC curve. Is this possible with your toolbox?


  39. justindomke says:

    HI Vlad,

    It won’t work off the shelf, but the modifications needed should be fairly minimal. You’d need to modify the code to accept “aoc” in the current places where the losses “ul” and “cl” are used. The only substantial change needed is computing the loss and the gradient of the loss with respect to marginals.

    The easiest way to implement this is probably to start training a model as if “aoc” was a valid loss function and modify the code in each of the places an error crops up. (There will only be a few.)

  40. cliffcurtish says:


    Having the same issues as Work York. Is there any third party code on which this toolbox depends? E.g., the functions fspecial or feats are undefined.

    Would be grateful if you could help me get started, thanks in advance!

  41. justindomke says:

    This does depend on third party code, but you will automatically be prompted to download it if it can’t be found. I think the issue you are experiencing is due to a change in some of this third-party code. Work York’s first problem is resolved as in the following message, by making sure imsdir and labdir are pointing to the correct place. As for the second problem, try changing this line:




    I think that the external toolbox that is being used has changed in a recent version to use doubles rather than singles.

  42. cgshen says:

    Thanks for sharing the toolbox. However, it seems the webpage has expired. Are there any new hosting location?

  43. cgshen says:

    Sorry I just found the new version at . Thanks again.

  44. Ehsan says:

    Hi dear justin
    Thanks for sharing your code.
    I cant get your result with pseudo loss on background dataset even with independent initialization.
    could you please help me?

  45. justindomke says:

    I can certainly try, but I’ll need more information. How do your results vary? Are you using the example_background.m file, and just changing the loss_spec value to the pseudolikelihood? Often a problem that appears here is that at test time you should switch to a different loss (something that requires inference) since the pseudolikelihood loss doesn’t need to do inference itself. (‘trunc_cl_trw_5’ or whatever)

  46. Ehsan says:

    yes, I use example_background. You mean the model should learn the params with ‘pseudo’ loss and infer with another?

  47. justindomke says:

    Well, I think the pseudolikelihood is a bad loss function, generally speaking, but if you want to use it, then yes: you’ll need to use a different loss function for inference. The reason is that you don’t need to do inference to compute the pseudolikelihood (that’s the point of it, after all) and so it doesn’t provide any marginals. Alternatively, you can just use a “pure” inference routine without computing a loss function at all.

  48. Ehsan says:

    Thanks a lot, I just use pseudolikelihood loss for the comparison purpose.

  49. Martina says:

    Hi Justin,

    first of all thanks a lot for sharing your. It was really easy to use and the paper associated explain very well many things. I am not a machine learning expert, I am working on image segmentation for the first time since a month, on a Curb Detection problem.

    I am using your example code for binary denoising because for now I am just trying to identify the position of the curb with respect to the street/sidewalk/buildings around. I am not using RGB-D images, but using my own code I extracted features from a point cloud generated with the Velodyne sensor. For now I am using just one feature for each point, each feature represents the score for this specific point to be part of a curb or not.
    So, to finish to explain my problem, I am currently using as input only the very same feature I have for every point in the cloud, running the learning step without any edge features.

    The first thing that I need to do is adding other unary features, again extracted with my own code, so I will need to import my new features from my code.
    – How would you suggested to do it, which part of the code I need to modify? (I saw the example_background.m code, in which you create the feature manually (unary and edge) with two functions.
    The second thing I need to integrate an edge feature that take into account connection with 8 neighbors. As far as I understood you use pairwise 4 connected grid to build the model.
    – How can I modify it? This new feature will be calculated inside the tool, so I need to create it starting with the first one that I am using now computed in my code. How and where can I add it inside the toolbox?

    I know that I am asking a lot, but it would be really helpful if you could help me. the deadline is so close now… 🙂

    Thanks a lot in advance.

  50. Megha Sharma says:

    Hello Sir,
    Please help me I am a B.Tech second year student and want to learn about working on conditional random field using matlab or r tool or any other way but i have no idea how to start please provide guidance.

  51. Anonymous says:

    Hello Sir,
    Please help me I am MSc student in AI and i want change graph model. i want ask you can i do that?
    my graph is a tree
    thanks a lot

  52. justindomke says:

    Sorry, I’m afraid that without a more detailed description of what you are trying to do, I can’t give a meaningful answer to your question.

  53. Sayyed_Mehrdad says:

    My work is assigning social role to different characters in video. in each video we have reference role that is most important role in video and every other roles have interaction with it. we have unary and pairwise features. so surly we must have a graph for our problem but we can’t use full graph because we can’t inference tractably and calculate necessary parameters so we should simplify graph. for this purpose we just consider interaction of different characters (roles) with reference role(most important role of video ) so we have a tree that his root is reference role and leafs are other characters (roles). now as we can see instead of full graph or complicated graph we have a tree that we can do tractable inference on it and calculate parameters.
    I looked at your gridmodel code and i thunk that i should change this code and implement my graph but i don’t know there is any other place in your package that i should change ?
    please help me
    thank you so much

  54. justindomke says:

    Hi Sayyed, yes that sounds about right. To use the toolbox with a different graph, you just need to create the right graph structure. You can base this on gridmodel.m, and use some of the support functions– essentially you just need to create a list of edges. Then, everything should work in roughly the same way.

  55. Sayyed_Mehrdad says:

    Thank you so much, you are great
    i hope you help me in future

  56. Ehsan says:

    Hi Dear Justin
    Thanks for your great toolbox.
    I have a question on the function train_crf.m line 232.
    Is it right L = L + reg*sum(params).^2; in your code?
    Isn’t it to be L = L + reg*sum(params.^2);


  57. Sayyed_mehrdad says:

    Hi sir
    I changed your gridmodel to Mygridmodel and execute program but i have a following error :
    error(‘theta_i should have size [model.nvals^2 size(model.pairs,1)]’);
    Please help me, thank you so much

  58. Christine says:

    The link does not work anymore, where will i find a copy of your code?

  59. justindomke says:

    Hi Christine,

    Thanks for pointing that out. I’ve updated the link.

  60. Eran says:


    Thanks for writing this toolbox, it looks like one of the more useful ones out there.

    I’m trying to modify the gridmodel.m function to be able to handle graphs with any clique groupings, i.e.. not just horizontal and vertical and also with any number of vertices in a clique. However, the code seems to be limited to having one pair of vertices per clique, since gridmodel.m sets tree2clique_hor to the model.pairs index, “ind_cnt”, and because trw_bprop_scheduled.m uses the values in tree2clique to index model.pairs.

    Modifying this code has proven to be difficult, since it isn’t clear how the variables in the “model” structure should be defined. Can you provide a detailed definition of the variables in the “model” structure: treenum, tree2clique, treeschedule, N1, and N2, ncliques, with hints on how to modify them to make this a CRF with a general structure?

    The model I’m interested in uses a CRF to model a subset of pixels in an image, which is represented as a 1-by-N vector, not a n-by-m image (N<=nxm). So, the grid model can't be used here. The clique neighborhood for a pixel can be any K pixels from the subset of pixels, not just the adjacent neighbors.

    I have adapted the gridmodel.m script to what I think it should be, but I feel like I don't understand the "model" variables well enough to trust any model that is learned. Any feedback will be greatly appreciated.

    Thank you.

  61. Eran says:

    Sorry Justin, my last message should have been to you, not Christine.

  62. justindomke says:

    Hi Eran,

    Unfortunately, generalizing the toolbox beyond pairs would be *extremely* complicated. All the inference algorithms and such are also pairwise specialized, so you’d almost be starting over. If you want factors of triplets or more, consider using MARBL:

    It can take arbitrary factors, and the documentation is clearer. (You just input the factors in a simple text file format.)

  63. Eran says:

    Thanks for the feedback. Does MARBL allow the training examples to be different sizes and for the user to supply the edge features?

  64. justindomke says:

    Yes, it allows both of those things. The idea is, for conditional random fields, you specify a “factor type” for each factor. All factors of the same type then have the same mapping from input features to output potentials. So, the graph structure can vary arbitrarily between different examples.

  65. justindomke says:

    Thanks, I’ve updated it.

  66. Lale says:

    Dear Justin,
    I have a question about the used formula of CRF in your toolbox. I would be really grateful if you could help me. Thank you so very much.

  67. Alfred says:

    Dear Sir,

    Thanks so much for sharing the code…
    I’m new to Matlab and I have been trying my best to get the code work… However, I cant get away from the following errors:

    – Error using gradientMag (line 50)
    I has incorrect type.

    – Error in hog (line 71)
    [M,O] = gradientMag( I );

    – Error in featurize_im (line 195)
    H = hog(double(reflectim(im,rez)),rez,9,10);

    – Error in example_backgrounds (line 22)
    parfor n=1:N

    May I know how I can fix it? Your help will be much appreciated…
    Thank You!

    Best Regards,

  68. Reem says:

    Hi Justin,

    Thank you for the efforts and for the great work. I was wondering if you have any documentation of your code available on hand? I would really appreciate it.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s