Tag Archives: optimization

Truncated Bi-Level Optimization

In 2012, I wrote a paper that I probably should have called “truncated bi-level optimization”.  I vaguely remembered telling the reviewers I would release some code, so I’m finally getting around to it. The idea of bilevel optimization is quite … Continue reading

Posted in Uncategorized | Tagged , , , , | 5 Comments

Fitting an inference algorithm instead of a model

One recent trend seems to be the realization that one can get better performance by tuning a CRF (Conditional Random Field) to a particular inference algorithm. Basically, forget about the distribution that the CRF represents, and instead only care how … Continue reading

Posted in Uncategorized | Tagged , , | 1 Comment

What Gauss-Seidel is Really Doing

I’ve been reading Alan Sokal’s lecture notes “Monte Carlo Methods in Statistical Mechanics: Foundations and New Algorithms” today. Once I learned to take the word “Hamiltonian” and mentally substitute “function to be minimized”, they are very clearly written. Anyway, the … Continue reading

Posted in Uncategorized | Tagged , , | 1 Comment

Hessian-Vector products

You have some function .  You have figured out how to compute it’s gradient, .  Now, however, you find that you are implementing some algorithm (like, say, Stochastic Meta Descent), and you need to compute the product of the Hessian … Continue reading

Posted in Uncategorized | Tagged , , | 5 Comments

Why does regularization work?

When fitting statistical models, we usually need to “regularize” the model. The simplest example is probably linear regression. Take some training data, . Given a vector of weights , the total squared distance is So to fit the model, we … Continue reading

Posted in Uncategorized | Tagged , , | 2 Comments