I updated the code for my Graphical Models / Conditional Random Fields toolbox This is a Matlab toolbox, though almost all the real work is done in compiled C++ for efficiency. The main improvements are:
- Lots of bugfixes.
- Various small improvements in speed.
- A unified CRF training interface to make things easier for those not training on images
- Binaries are now provided for Linux as well as OS X.
- The code for inference and learning using TRW is now multithreaded, using openmp.
- Switched to using a newer version of Eigen
There is also far more detailed examples, including a full tutorial of how to train a CRF to do “semantic segmentation” on the Stanford Backgrounds dataset. Just using simple color, position, and Histogram of Gradient features, the error rates are 23%, which appear to be state of the art (and better than previous CRF based approaches.) It takes about 90 minutes to train on my 8-core machine, and processes new frames in a little over a second each.
For fun, I also ran this model on a video of someone driving from Alexandria into Georgetown. You can see that the results are far from perfect but are reasonably good. (Notice it successfully distinguishes trees and grass at 0:12)
I’m keen to have others use the code (what with the hundreds of hours spent writing it), so please send email if you have any issues.
Hi Justin, thank you for sharing your code.
Does it also work with Octave?
If you want your package to have more attention, add it to mloss.org, there are already a few other CRF implementations: http://mloss.org/software/search/?searchterm=CRF
I’m eager to support Octave. I hadn’t done so thus far as I was under the impression that Octave couldn’t use mex files. I see now that isn’t true, but I have relied on matlab’s ability to pass arrays to C++ code by pointer. Since Octave makes copies beforehand, this means (I think!) it won’t work out of the box. The other reason I haven’t focused on Octave the lack of parfor. With multithreaded TRW, though, this is less of an issue.
Thanks for the pointer to mloss.org. I’ll add it there.
Hi Justin, thanks a lot for sharing the code mate, it is a great piece of software. I would like to ask you, how can you train the CRF assuming your “image” (for not saying data) contains more than 3 dimensions? In one of the examples you provide (example_backgrounds.m), you perform segmentation using RGB images i.e., data with 3 dimensions or features, but what if you have images (well,… data) with more than 3 dimensions or features? How can you train the CRF model with your software?, how do you generate the “efeats” matrix.
Thanks a lot for the help!
Cheers
Hi Karin. Glad to hear you find the code useful. You can certainly apply it to multispectral images, if that is what you are asking. In terms of the backgrounds dataset example, as long as you create the matrices feats{:} and efeats{:} for each example image, everything should work fine. This might require some minor modification to “featurize_im.m” and ” edgeify_im.m” to handle multispectal images. (e.g. rgb2gray won’t work for multispectral.) I think, however, that you could still get away with using simple edge features based on pixel intensity differences, i.e. the same
edge_params = {{‘const’},{‘diffthresh’},{‘pairtypes’}};
efeats{n} = edgeify_im(ims{n},edge_params,models{n}.pairs,models{n}.pairtype);
used in the backgrounds example.
The best thing to do, though, would be to decide what edge features would be best for your application, and write your own simple function to replace edgify_im.m.
Hi Justin, thanks a lot for the reply.
I am working with several datasets and in these datasets I know the connections between every variable and feature, the problem that I have is that I don’t know how to create the efeats matrix, I checked the code, but I couldn’t get how to build that matrix from scratch (without the edgeify_im function),
Hope you can help me with this: suppose you have a dataset with “N” examples where each example contains 6 labels “yi”. And you know that the connections between them are: y3-y2-y4-y5-y1-y6. Each of these labels is binary (so let’s pretend each example corresponds to a 1-by-6 binary image). Every “pixel” here is connected or described by a set of 72 features “xj”. All of the “yi’s” share the same features. So each “yi” is connected to every “xj” (x1, x2,…,x72). And there are no connections among the “xj’s”.
It is basically a linear chain where each label yi in the chain shares the same features (x1 to x72). How can you model this?
Thanks in advance,
Cheers
Seems like no problem to me– just create a standard “chain” CRF and use the same features for all the “yi”, but don’t use parameter ties, so that the different “yi” can have their own relationship with the “xi”.
With the existing code base, this could be done by modeling a “1×6″ image (gridmodel). Since the code is set up to use feature ties, you could hack this by using 72×6 features for each “yi”, where only one block is nonzero. (The first block for “y1″, the second for “y2″,etc.).
All right, I am going to try then
Cheers and thanks a lot for all of the help
Hi Justin, I have been running your code on Linux, but every time I run it I receive different outputs (different test error numbers, different classes, etc.) even when of course, the training and test datasets are the same, always!
I am using Belief Prop, and the weird(est) thing is that the outputs are the same when I run it on windows! (but who wants to use windows?),
Any idea about what’s going on?
Regards,
Liam
Hi Liam,
Do you use a random function anywhere in your code? (maybe when you generate the train/test splits?)
Do you initialize the seeds before calling any random function? (you have to use the rgn function)
Cheers
Hello there,
yeah I use random functions twice in the code. But before I call such functions I use rng(1); like you mention.
I don’t know why the program works properly in Windows but not in linux. In linux even if I remove the random functions and leave the splits for training and testing fixed, I get different results.
Regards,
Liam
… with respect to my previous comment:
I already checked the training/testing splits’ indexes in linux, they are always the same, which means that the rng(1); functions are working properly. So maybe there is another random function somewhere else in the code, I just can’t find it…
Regards,
Liam
Hi Liam. There “shouldn’t” be any other random numbers used, since parameters are just initialized to zero. Are you using the multithreaded trw code? It is possible that non-determinism is leaking in there somehow, so try switching to the non-multithreaded version (change “trwpll” to “trw”) and see if the problem goes away. If not, could you create the simplest example you can that reproduces the problem?
Also, if you’ve compiled binaries for windows, I’d appreciate it if you could send them to me so I can include them in the package.
Hi Thank you for your code. I am using this code on windows. I tried to run the code for ‘Background Dataset’. But I am getting an error ‘Undefined function or method ‘reflectim_helper’ for input arguments of type ‘double” should I recompile your code into windows? How can I get rid of this error?
Hi Siyam,
I think you just need to compile the code. (I don’t have a windows machine and so don’t provide binaries for windows.) Essentially, this would be done just by typing “compile” from the main toolbox directory.
Hi Justin,
I am using your code on octave on linux. On trying to run the code for ‘Background Dataset’, I’m getting an error “Undefined function or method ‘reflectim_helper”. How can I solve this error ? Please reply asap!
Thanks in advance.
Hello Justin
I am trying to make some changes in your CRF toolbox, so I will be able to use it in my project.
Could you please tell me, in which function inside your toolbox the CRF conditional distribution is implemented?
I want to add some temporal information to this formula
Thank you in advance
Hi Monika, Sorry for the slow reply. The problem you are experiencing is that reflectim_helper is implemented in C++, not in matlab/octave. Thus, to make it available it needs to be compiled. In matlab, this would be done by something like “mex reflectim_helper.cpp”. (There is a script compile.m that compiles all the needed files.) Unfortunately, I don’t know how to do this for octave, but please do get in touch if you figure it out, as I’ve heard from several people who would like octave support…
Hi Jeiran,
Roughly speaking, the conditional distribution is implemented in the files in the CRF/ subdirectory. For example, crf_linear_linear.m maps the features for each point/edge to MRF parameters using a linear mapping for both univariate and edge features. Thus, I think you would likely want to proceed by implementing an alternative crf_something.m file, and then slightly modifying train_crf.m to take your new crf_type variable and call the appropriate crf_something function. (Search for “if strcmp(crf_type,’linear_linear’)” addn then add your new type.) Good luck!
Hi Justin,
Can I use your toolbox in a C/C++ program? I’m a doing a C++ project and I’ve been having great trouble in finding some CRF toolbox for images. I’ve found some c++ toolboxes, but i think they are all for natural language processing problems. I’m very new to this, so I don’t know how to use a toolbox intended for NLP in my computer vision project. So can your toolbox be called from a c++ program? Thanks a lot!!!
Hi Diana. Unfortunately, probably not– only the slowest parts of the code are in C++, while the majority is in Matlab.