Linear regression fits a straight-line (or, with more inputs, flat-plane) formula to past data by minimizing the sum of the squared gaps between the observed values and the values the formula predicts. It is one of the simplest prediction models in machine learning, and the software library scikit-learn documents it as ordinary least squares. A fitted line can look convincing on summary numbers alone, so plotting the data and checking residuals matters as much as the fit itself.
Key takeaways
- Ordinary least squares chooses the coefficients that minimize the squared difference between observed and predicted targets.
- "Linear" refers to the parameters, so a model with a squared or logarithmic input term can still be a linear regression.
- One or two outliers can seriously skew a least squares fit, according to the National Institute of Standards and Technology (NIST).
- Anscombe's 1973 quartet shows four very different datasets that produce the same fitted line and the same correlation.
- Strongly correlated inputs make coefficient estimates unstable, even when predictions stay reasonable.
What the model actually computes
A linear regression predicts a number, such as a price, a temperature, or a wait time, as a weighted sum of inputs plus a constant. Training means finding the weights. The scikit-learn documentation states that its LinearRegression class fits coefficients "to minimize the residual sum of squares between the observed targets in the dataset, and the targets predicted by the linear approximation." A residual is the observed value minus the predicted one, and squaring it means large misses count much more than small ones.
The NIST Engineering Statistics Handbook defines the model class more broadly: each explanatory variable is multiplied by an unknown parameter and the terms are summed, which makes the model "linear in the parameters." A quadratic curve therefore qualifies, while a model where the slope is a product of two unknown parameters does not. Because the method is so well understood, NIST lists as advantages that it makes efficient use of data, so small datasets can still give good results, and that it supports easily interpreted statistical intervals for predictions.
Why a good fit can still be wrong
Frank Anscombe published four small datasets in The American Statistician in 1973, each with 11 points. The R statistical software ships them as the built-in anscombe dataset and describes them as having the same traditional statistical properties yet being quite different. Computed from that published data, all four sets share a mean of y of 7.50, a fitted line of y = 3.00 + 0.50x, and a correlation of about 0.82. The table summarizes what a scatter plot reveals that the summary numbers hide. The numbers are those of the published dataset, and the descriptions of the shapes are an illustrative reading of the plots.
| Dataset | Shape when plotted | What the line misses |
|---|---|---|
| 1 | Scattered around a roughly straight trend | Nothing serious; the line is a fair summary |
| 2 | A smooth curve that rises and then falls | The relationship is not linear at all |
| 3 | A tight line with one point far above it | One outlier tilts the fitted line away from the other 10 points |
| 4 | Ten points at the same x value and one far to the right | A single point alone determines the slope |
This is the reason NIST stresses that model validation is critical: the fitting step never reports that the model form is wrong.
Limits stated in the documentation
NIST names three practical weaknesses. Linear models with nonlinear predictor terms curve slowly, so they fit inherently nonlinear processes poorly as the data range widens. They may be ineffective for extrapolating beyond the range where data could be collected, although NIST adds that extrapolation is risky for any model type. And one or two outliers can sometimes seriously skew the results.
The scikit-learn documentation adds a different warning. Coefficient estimates rely on the independence of the features. When columns of the input matrix are nearly linearly dependent, a condition called multicollinearity, the least squares estimate becomes highly sensitive to random errors in the target and its variance grows large. The documentation notes this can arise when data are collected without an experimental design. As an illustrative example, two overlapping inputs, such as floor area in square feet and number of rooms in a housing dataset, can swap weight between them from one sample to the next, so individual coefficients should not be read as stable effects.
Where it sits next to other AI methods
Linear regression is a supervised method: it learns from examples where the right answer is known. NIST observes that it underlies many other approaches, including nonlinear, weighted, and LOESS regression. Whatever the model, the same questions apply: does the training data cover the situations where predictions will be used, and are the errors checked on data the model never saw?
A short checklist before trusting a fitted line
- Plot the target against each input and look for curves, clusters, and isolated points.
- Plot the residuals against the predictions; a visible pattern means the model form is missing something.
- Refit without the most extreme points and compare the coefficients.
- Check whether inputs overlap heavily before interpreting individual coefficients.
- Keep predictions inside the range of the training data, or state clearly that they are extrapolations.
- Measure error on held-out data, not on the data used for fitting.
FAQ
Is linear regression a form of artificial intelligence?
It is a statistical method that machine learning libraries include as a basic model. Whether it counts as AI depends on the definition used, but it is widely taught as the starting point of supervised learning.
Does a straight line mean the data must be linear?
No. Because the term refers to linearity in the parameters, a model can include squared or logarithmic versions of an input and remain a linear regression, as the NIST handbook explains.
What does a high correlation prove?
Very little by itself. Anscombe's four datasets all have a correlation near 0.82, yet only one of them looks like a simple straight-line relationship.
When does the least squares method perform badly?
The documented trouble spots are outliers, nonlinear processes over wide ranges, extrapolation beyond the data, and strongly correlated inputs.
Related reading
- Predictive analytics and data-driven decisions
- How AI programs sort through large datasets
- AI in data science work
- Artificial intelligence compared with data science
More in AI Basics.
Last verified: October 2026
Sources
- Linear Models (Ordinary Least Squares), scikit-learn documentation, scikit-learn developers, 2026
- Engineering Statistics Handbook, section 4.1.4.1 on linear least squares regression, National Institute of Standards and Technology
- Anscombe FJ, Graphs in Statistical Analysis, The American Statistician 27(1), 1973
- The anscombe dataset, R datasets package documentation, R Core Team