Ridge regression is called ridge because its coefficient estimates, plotted as the penalty strength changes, form a ridge-like path called the ridge trace—and because the method was designed to stabilize estimates that otherwise wander along a long, flat “ridge” in the least-squares solution space. The name is historical and geometric, not a reference to ridges in the raw data. It was popularized by Arthur Hoerl and Robert Kennard in their influential 1970 papers on biased estimation for multicollinear data, where they introduced the ridge trace as a practical tool for choosing the tuning parameter.
The Short Answer: A Name Born From a Trace
At its core, ridge regression modifies ordinary least squares by adding a penalty on the size of the coefficients. Instead of minimizing only the residual sum of squares, it minimizes:
RSS + λ × Σβⱼ²
Here, λ controls how much shrinkage is applied. Day to day, when λ is zero, ridge regression becomes ordinary least squares. As λ increases, the coefficients are pulled toward zero.
The word ridge enters through the ridge trace: a plot of the estimated coefficients against different values of λ. When you plot the ridge estimates across a range of λ values, the coefficients often move sharply at first and then settle onto a smoother, more stable path. Small changes in the data can produce large swings in the coefficients. Now, in datasets with multicollinearity—where predictors are highly correlated—the ordinary least squares estimates can be extremely unstable. Hoerl and Kennard described this path as a ridge and used the trace to select a λ where the estimates become stable.
So the name is not saying that the data contain a ridge. It is saying that the estimation path looks like one.
Why “Ridge” and Not “Shrinkage” or “Penalty”?
Ridge regression does involve shrinkage and penalty terms, and in modern machine learning it is