CRF Toolbox Updated

I updated the code for my Graphical Models / Conditional Random Fields toolbox This is a Matlab toolbox, though almost all the real work is done in compiled C++ for efficiency. The main improvements are:

  • Lots of bugfixes.
  • Various small improvements in speed.
  • A unified CRF training interface to make things easier for those not training on images
  • Binaries are now provided for Linux as well as OS X.
  • The code for inference and learning using TRW is now multithreaded, using openmp.
  • Switched to using a newer version of Eigen

There is also far more detailed examples, including a full tutorial of how to train a CRF to do “semantic segmentation” on the Stanford Backgrounds dataset. Just using simple color, position, and Histogram of Gradient features, the error rates are 23%, which appear to be state of the art (and better than previous CRF based approaches.) It takes about 90 minutes to train on my 8-core machine, and processes new frames in a little over a second each.

For fun, I also ran this model on a video of someone driving from Alexandria into Georgetown. You can see that the results are far from perfect but are reasonably good. (Notice it successfully distinguishes trees and grass at 0:12)

I’m keen to have others use the code (what with the hundreds of hours spent writing it), so please send email if you have any issues.

About these ads
This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink.

37 Responses to CRF Toolbox Updated

  1. Hi Justin, thank you for sharing your code.

    Does it also work with Octave?

    If you want your package to have more attention, add it to, there are already a few other CRF implementations:

  2. justindomke says:

    I’m eager to support Octave. I hadn’t done so thus far as I was under the impression that Octave couldn’t use mex files. I see now that isn’t true, but I have relied on matlab’s ability to pass arrays to C++ code by pointer. Since Octave makes copies beforehand, this means (I think!) it won’t work out of the box. The other reason I haven’t focused on Octave the lack of parfor. With multithreaded TRW, though, this is less of an issue.

    Thanks for the pointer to I’ll add it there.

  3. karin zachinelly says:

    Hi Justin, thanks a lot for sharing the code mate, it is a great piece of software. I would like to ask you, how can you train the CRF assuming your “image” (for not saying data) contains more than 3 dimensions? In one of the examples you provide (example_backgrounds.m), you perform segmentation using RGB images i.e., data with 3 dimensions or features, but what if you have images (well,… data) with more than 3 dimensions or features? How can you train the CRF model with your software?, how do you generate the “efeats” matrix.

    Thanks a lot for the help!


  4. justindomke says:

    Hi Karin. Glad to hear you find the code useful. You can certainly apply it to multispectral images, if that is what you are asking. In terms of the backgrounds dataset example, as long as you create the matrices feats{:} and efeats{:} for each example image, everything should work fine. This might require some minor modification to “featurize_im.m” and ” edgeify_im.m” to handle multispectal images. (e.g. rgb2gray won’t work for multispectral.) I think, however, that you could still get away with using simple edge features based on pixel intensity differences, i.e. the same

    edge_params = {{‘const’},{‘diffthresh’},{‘pairtypes’}};
    efeats{n} = edgeify_im(ims{n},edge_params,models{n}.pairs,models{n}.pairtype);

    used in the backgrounds example.

    The best thing to do, though, would be to decide what edge features would be best for your application, and write your own simple function to replace edgify_im.m.

  5. karin zachinelly says:

    Hi Justin, thanks a lot for the reply.

    I am working with several datasets and in these datasets I know the connections between every variable and feature, the problem that I have is that I don’t know how to create the efeats matrix, I checked the code, but I couldn’t get how to build that matrix from scratch (without the edgeify_im function),

    Hope you can help me with this: suppose you have a dataset with “N” examples where each example contains 6 labels “yi”. And you know that the connections between them are: y3-y2-y4-y5-y1-y6. Each of these labels is binary (so let’s pretend each example corresponds to a 1-by-6 binary image). Every “pixel” here is connected or described by a set of 72 features “xj”. All of the “yi’s” share the same features. So each “yi” is connected to every “xj” (x1, x2,…,x72). And there are no connections among the “xj’s”.

    It is basically a linear chain where each label yi in the chain shares the same features (x1 to x72). How can you model this?

    Thanks in advance,

  6. justindomke says:

    Seems like no problem to me– just create a standard “chain” CRF and use the same features for all the “yi”, but don’t use parameter ties, so that the different “yi” can have their own relationship with the “xi”.

    With the existing code base, this could be done by modeling a “1×6″ image (gridmodel). Since the code is set up to use feature ties, you could hack this by using 72×6 features for each “yi”, where only one block is nonzero. (The first block for “y1″, the second for “y2″,etc.).

  7. karin zachinelly says:

    All right, I am going to try then :)

    Cheers and thanks a lot for all of the help

  8. Liam Parker says:

    Hi Justin, I have been running your code on Linux, but every time I run it I receive different outputs (different test error numbers, different classes, etc.) even when of course, the training and test datasets are the same, always!

    I am using Belief Prop, and the weird(est) thing is that the outputs are the same when I run it on windows! (but who wants to use windows?),

    Any idea about what’s going on?


  9. karin zachinelly says:

    Hi Liam,

    Do you use a random function anywhere in your code? (maybe when you generate the train/test splits?)

    Do you initialize the seeds before calling any random function? (you have to use the rgn function)


  10. Liam Parker says:

    Hello there,

    yeah I use random functions twice in the code. But before I call such functions I use rng(1); like you mention.

    I don’t know why the program works properly in Windows but not in linux. In linux even if I remove the random functions and leave the splits for training and testing fixed, I get different results.


  11. Liam Parker says:

    … with respect to my previous comment:

    I already checked the training/testing splits’ indexes in linux, they are always the same, which means that the rng(1); functions are working properly. So maybe there is another random function somewhere else in the code, I just can’t find it…


  12. justindomke says:

    Hi Liam. There “shouldn’t” be any other random numbers used, since parameters are just initialized to zero. Are you using the multithreaded trw code? It is possible that non-determinism is leaking in there somehow, so try switching to the non-multithreaded version (change “trwpll” to “trw”) and see if the problem goes away. If not, could you create the simplest example you can that reproduces the problem?

    Also, if you’ve compiled binaries for windows, I’d appreciate it if you could send them to me so I can include them in the package.

  13. siyam says:

    Hi Thank you for your code. I am using this code on windows. I tried to run the code for ‘Background Dataset’. But I am getting an error ‘Undefined function or method ‘reflectim_helper’ for input arguments of type ‘double” should I recompile your code into windows? How can I get rid of this error?

  14. justindomke says:

    Hi Siyam,

    I think you just need to compile the code. (I don’t have a windows machine and so don’t provide binaries for windows.) Essentially, this would be done just by typing “compile” from the main toolbox directory.

  15. Monika says:

    Hi Justin,
    I am using your code on octave on linux. On trying to run the code for ‘Background Dataset’, I’m getting an error “Undefined function or method ‘reflectim_helper”. How can I solve this error ? Please reply asap!
    Thanks in advance.

  16. Jeiran Choupan says:

    Hello Justin
    I am trying to make some changes in your CRF toolbox, so I will be able to use it in my project.
    Could you please tell me, in which function inside your toolbox the CRF conditional distribution is implemented?
    I want to add some temporal information to this formula

    Thank you in advance

  17. justindomke says:

    Hi Monika, Sorry for the slow reply. The problem you are experiencing is that reflectim_helper is implemented in C++, not in matlab/octave. Thus, to make it available it needs to be compiled. In matlab, this would be done by something like “mex reflectim_helper.cpp”. (There is a script compile.m that compiles all the needed files.) Unfortunately, I don’t know how to do this for octave, but please do get in touch if you figure it out, as I’ve heard from several people who would like octave support…

  18. justindomke says:

    Hi Jeiran,

    Roughly speaking, the conditional distribution is implemented in the files in the CRF/ subdirectory. For example, crf_linear_linear.m maps the features for each point/edge to MRF parameters using a linear mapping for both univariate and edge features. Thus, I think you would likely want to proceed by implementing an alternative crf_something.m file, and then slightly modifying train_crf.m to take your new crf_type variable and call the appropriate crf_something function. (Search for “if strcmp(crf_type,’linear_linear’)” addn then add your new type.) Good luck!

  19. Diana says:

    Hi Justin,
    Can I use your toolbox in a C/C++ program? I’m a doing a C++ project and I’ve been having great trouble in finding some CRF toolbox for images. I’ve found some c++ toolboxes, but i think they are all for natural language processing problems. I’m very new to this, so I don’t know how to use a toolbox intended for NLP in my computer vision project. So can your toolbox be called from a c++ program? Thanks a lot!!!

  20. justindomke says:

    Hi Diana. Unfortunately, probably not– only the slowest parts of the code are in C++, while the majority is in Matlab.

  21. Janani says:

    Hi Justin,

    I am new to CRF implementations, so i tried the examples in your toolbox before I edit it to suit my needs. I am repeatedly getting Undefined function ‘reflectim_helper’ for input arguments of type ‘double’. I have recompiled the binaries, includes ll the files and folders. How do I solve this issue. I am using windows 64 bit.

    Thank a lot !!

  22. justindomke says:

    Hi Janani,

    That error means that matlab is not able to find the mex file corresponding to Vision/reflectim_helper.cpp. Can you verify that Vision/reflectim_helper.mexw64 exists? If not, that means compilation hasn’t worked correctly. If so, the only other issue is that your path might not be set correctly. This can be done by typing “addpath(genpath(‘.’))’ in the main JGMT directory. Finally, if you’ve really compiled all the mex files, it would be great if you could email them to me so I can include them for others. My email is (myfirstname).(mylastname) Thanks!

  23. Mario says:

    Hi Justin

    I’m trying to run the demo code does not work. Displays the following error: undefined function ‘kfold_sets’ for input arguments of type ‘double’. I do not know where to locate this function code.

  24. Mario says:

    Sorry, I found the code kfolds_sets function but I get an error with frunción “feats” as it says it is undefined.

    I’m using Linux to run the sample.

    thank you very much

  25. Adnan says:

    Hello Justin

    I am new to CRF implementation and understanding, I found your code is very help for the beginners to learn and understand about CRF. I tried to run the code usinf MATLAB 2011 on windows PC. when i am trying to use train_crf function, it gives me error. i tired best to my knowledge but i couldn’t solve.. the error is e:\\justinsgraphicalmodelstoolboxpublic\differentiation\eigenstuff.cpp(8) : fatal error C1083: Cannot open include file: ‘Eigen/Eigen': No such file or directory
    E:\PROGRAM FILES\MATLAB\R2011A\BIN\MEX.PL: Error: Compile of trw_bprop_scheduled_helper1.cpp’ failed.

    please could help me the solve this error..

  26. justindomke says:

    Hi Adnan,

    Are you trying to compile the code? You should be able to use the included binaries, which is typically the easiest way to avoid compilation problems…

  27. Adnan says:

    Hello Justin, Thanks i solved that error.

    i want to know more about this code, could you please tell me that these results are published in any paper?

  28. justindomke says:

    HI Adnan,

    Glad to hear it. Yes, the results are published in this paper:

  29. Adnan says:

    Hello Justin,

    I want to check your CRF toolbox on my idea. So i want to discuss some related to that on email, so is it comfortable for you?

  30. saleem says:

    I want to use this code for activity recognition using depth image, is it possible to use? what would be labels as a input in that case?

  31. justindomke says:

    How to you encode an “activity”? If you just have a finite set, it isn’t clear to me why you would want to use a CRF at all (as opposed to just a regular classifier).

  32. saleem says:

    Actually i’m interested to use CRF as classifier .. i have depth sequence of a person jumping, 500 frames. i made a feature vector and i am don’t want to use edge feature in this case i think (correct me if i am wrong) . So what would i use as input label?

  33. Megha says:

    Hey Justin

    Thanks a lot for making this available online. I was trying to figure out how to use your tool on an activity dataset I found online ( which usually stores data in following format:
    activity label, start_time, sensor_id, sensor_status
    I am not quite sure the data format for test and training sets the function accept? Could you please suggest something as to how I can use this dataset with your toolkit for activity recognition?


  34. Adnan says:

    please reply to the quires. it would be helpful for everyone.

  35. Work York says:

    I get an error in the second example with frunción “feats” as it says it is undefined.

    “Undefined function or variable “feats”.
    Error in example_backgrounds (line 72)
    feats_train = feats(who_train);”

    Thanks in advance

  36. Work York says:

    the last error is resolved by correcting the imsdir and the labdir
    but I have this error :

    Error using hog (line 50)
    I must be a double 2 or 3 dim array.

    Error in featurize_im (line 195)
    H = hog(single(reflectim(im,rez)),rez,9,10);

    Error in example_backgrounds>(parfor body) (line 30)
    feats{n} = featurize_im(ims{n},feat_params);

    Error in example_backgrounds (line 22)
    parfor n=1:N

    Caused by:
    Error using hog1
    I must be a double 2 or 3 dim array.

    thanks in advance

  37. justindomke says:

    I think the error with hog is due to a change in the format used by the external toolbox. I will look into this, but it looks like it can be dealt with by removing the cast to single precision.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s