Banana-shaped sloppy valley. Cost contours for a sloppy model of exponential decay. The horizontal and vertical axes were chosen in the sloppiest and (second) stiffest directions. Notice that the horizontal axis is squashed by a factor of a thousand - our figure should really be stretched horizontally to roughly half the size of a sports arena. The aspect ratio is roughly that of a human hair. |
James Sethna > Sloppy Models > What are Sloppy Models? We claim that many if not most models with several fitting parameters (say, more than five) are sloppy. Sloppy models are called 'poorly constrained' and 'ill-conditioned' because it is difficult to use experimental data to figure out what their parameters are. Suppose the data y_{i}=y(t_{i}) is being fit by a model y(θ,t) with parameters θ=&theta_{1}, &theta_{2}, ... &theta_{17}, ... We can measure the quality of the fit to the data by defining a cost C, the sum of the squares of the differences between theory and experiment (sometimes called χ^{2}):
The figure at left shows a contour plot of this cost for one of our
sloppy models (fitting exponentials),
zoomed in near the best fit at the center. Crossing
contours corresponds to increases in cost; the system behavior predicted
by the model no longer agrees as well with the data. The horizontal
and vertical axes move in certain carefully chosen stiff and sloppy
directions in parameter space (along eigenparameters that we
describe below). Notice first that the behavior changes much less as we
move the parameters in the sloppy, side-to-side direction, than if we change
them along the vertical stiff direction. Notice second that we have squashed
the plot by a factor of a thousand; the sloppy direction is really enormously
less constrained than the stiff direction.
Stiff and sloppy directions in parameter space. Moving up and right in parameter space changes the system behavior very little, so few contours are crossed - a sloppy direction. Moving up and left in parameter space changes the system behavior rapidly - a stiff direction. The length of the axes of the ellipses are given by the square roots of the eigenvalues. For sloppy multiparameter models, there is an enormous range of eigenvalues - only a few eigenparameters are well determined by the data. |
The banana-shaped contour plot above is a two-dimensional cross-section of the cost in a large-dimensional parameter space. How can we describe the behavior as we change the parameters in a general direction?
Near enough to the best fit, we almost always will find that the cost contour surfaces form ellipsoids - stretched long along sloppy directions, and pinched in along the stiff directions. These ellipsoids can be described by the eigenvalues and eigenvectors of the Hessian H_{αβ} = ∂^{2}C/∂θ_{α}∂θ_{β} of the cost near the best fit. The eigenvectors point along the different axes of the ellipse; we call these parameter combinations eigenparameters, as opposed to the bare parameters that we originally chose to write our model in terms of. The eigenvalues λ_{n} are the squares of the lengths of the ellipsoids along the different eigenparameters.
We can see from the schematic figure at left that the behavior does depend
sensitively on the value of the bare parameter θ_{1};
we cross contours as we move horizontally almost as rapidly as if we
moved along the stiff eigenparameter up and to the left. But the error bar
P_{1} for bare parameter θ_{1} is
big because of the sloppy direction. One can't use the system behavior
to determine θ_{1} because it can be exchanged for
θ_{2} with little change in cost, moving along the
sloppy direction.
Sloppy systems biology eigenvalue spectra. The eigenvalues of all of these models are spread rather uniformly over more than six decades (corresponding to an aspect ratio of over a thousand between stiffest and sloppiest directions). The systems analyzed here cover everything from fruit fly embryo formation, to the metabolism of nucleotides for DNA, to biological clocks, to hormonal signal transduction. |
We first encountered sloppiness in a systems biology model of the response of a certain cell to hormones. The eigenvalues for that model are shown in the figure at right in column (i). We were struck by the fact that the 48 eigenvalues in that model were roughly equally spread over six decades. (A few eigenvalues near zero might be something weird with the model, but such nicely spread-out eigenvalues seemed like it might be there for a reason.) Ryan Gutenkunst then took sixteen other systems biology models from the literature, covering everything from fruit fly embryo formation, to the metabolism of nucleotides for DNA, to biological clocks. All of these models had many parameters, and when we studied the change in experimental behavior as we varied the parameters we found that they too shared the same sloppy features (right, a-q). That is, they all had an enormous range of eigenvalues, with a few stiff combinations of parameters that were well constrained by the data and many sloppy combinations, which can vary by vast amounts without significant change in the system behavior.
Sloppy (and non-sloppy) eigenvalues from many fields. (a) our systems biology model; (b) quantum Monte Carlo variational wavefunction; (c) radioactive decay for a mixture of twelve common nuclides; (d) fit of 48 exponential decays; (e) random 48x48 matrix (GOE, not sloppy); (f) product of five 48x48 random matrices (not sloppy, but exponentially spread), (g) multiple linear regression fit of a plane to data in 48 dimensions (not sloppy), and (h) a polynomial fit to data (sloppy). |
Is sloppiness special to biological systems? Apparently not! We've found sloppiness in multiparameter models spanning many fields, from models of insect flight, to interatomic potentials, to accelerator design - every multiparameter model that we have studied so far appears to be sloppy. At left you can see a few examples. Quantum Monte Carlo, for example, is our most precise tool for solving for the energies and quantum behavior of atoms and small molecules; Cyrus Umrigar's wonderfully accurate variational wave-functions underlying this method, however, are remarkably sloppy (column b). Precise predictions can be made from sloppy models. In mathematics, fitting sums of exponential decays to radioactivity data (columns c and d) can give excellent fits with decay constants that are wildly different from the true ones - a classic example of an ill-posed fitting problem. Finally, fitting polynomials to data is sloppy, at least insofar as one is interested in the coefficients of the polynomials (column h).
To gain more intuition about how predictions are possible without good
parameters, see Fitting Exponentials:
Prediction without parameters.
James P. Sethna, sethna@lassp.cornell.edu; This work supported by the Division of Materials Research of the U.S. National Science Foundation, through grant DMR-070167.
Statistical Mechanics: Entropy, Order Parameters, and Complexity, now available at Oxford University Press (USA, Europe).