Regular single-variate implicit differentiation is taught in calc 101.
Consider a relationship between and implicitly defined by
Now, consider some (infinitesimally) small perturbation to , inducing a corresponding peturbation in .
(Here I am using the convention that if arguments to are unspecified, it is to be evaluated at ) Recall that , then solve to get
We again have an implicit relationship, but now everything is a vector.
Consider perturbations and .
Use in optimization
Suppose some variables are implicitly defined by a minimization of some function also of .
Suppose furthermore, that there is some criterion that we would like to minimize.
So we want to set so that the resulting gives the best value for .
By setting the gradient of the above to zero, we obtain (We assume here that is convex over for all .)
Making the substitution in the last equation from the previous section, we get
Now, by putting the pieces together, we can see how a small change in affects the objective function.