What Is Overfitting in Machine Learning? A Tale of Two Curves

Blog › Machine Learning Explained

· by Julia Berens · 4 min read

To answer “what is overfitting in machine learning,” draw two fits on the same scatterplot. Make one pass through every training point and let the other miss a few. Now add points that neither model got to see during training. If the perfect fit does worse on the new points because it learned quirks of the training data, you are looking at overfitting. The difference is between fitting the examples you already have and making predictions you can trust on examples you have not seen.

What is overfitting in machine learning on a scatterplot?

Overfitting happens when a model learns details of its training data that do not carry over to new data. To see it without a large dataset, use a made-up measuring experiment. The underlying relationship is y = 2x, but one training measurement reads one unit too high. A model receives only the measured pairs, not the rule that produced them or a warning about the high reading.

Training input xMeasured output y
00
13
24

Put x on the horizontal axis and y on the vertical axis. Plot the three training points as solid dots. The middle dot sits above the line you could draw through the two outer dots. Looking only at those three measurements, you cannot tell whether that middle dot is a measurement error or evidence that the relationship really bends. That uncertainty is what makes the example useful: the model has to make predictions without knowing the explanation you were given.

Two fits on the same three dots

Draw a straight line with the equation y = 2x + 1/3. This is the least-squares line for these points: among straight lines, it gives the smallest sum of squared vertical misses. At x = 0, 1, and 2, its predictions are 1/3, 2 and 1/3, and 4 and 1/3. So it passes a little above each outer dot and below the middle one. It does not ignore the high reading. The line shifts upward to account for it without changing slope.

On the same axes, draw the parabola y = 4x − x2. It goes through all three dots. It climbs more steeply than the line on the left side of the plot, then flattens as it approaches the rightmost dot. The extra term gives the parabola a way to change its slope across the plot, which a straight line cannot do. On the training data, the parabola wins: its squared misses add up to zero.

Trace both fits with your finger from left to right. At the three training inputs, you can judge each prediction against a dot. In between, there are no dots yet. The curves still give answers there, but those answers follow from their shapes, not from measurements at those exact inputs. A model can be wrong in those gaps even if the new inputs are inside the range it trained on.

New points change which fit wins

Now suppose two new measurements land exactly on the underlying relationship. Add them as open circles at (0.5, 1) and (1.5, 3). Keep both fits where they are. The point of this step is to score predictions from models that did not have access to the open circles when they were fitted.

New input xNew measurementLine predictionParabola prediction
0.511 and 1/31.75
1.533 and 1/33.75

At each new point, the line is high by 1/3 and the parabola is high by 3/4. On your sketch, look at the vertical gap from each open circle to each fit. The parabola has the larger gap both times. Its bend helped it reach the high training dot, but that bend also pulled its predictions farther above the new dots. The line was affected by the high reading too, just less so at these inputs.

This example is small enough to check by hand. Two new points would not be enough to rank models confidently on a real project. Here, we know the rule that generated the example and can see exactly why the bend is misleading. In a real dataset, the underlying rule is usually unknown, and new measurements can be noisy too.

A perfect fit is a clue, not a verdict

A curve is not automatically worse than a line. If the underlying relationship really were y = 4x − x2 and the measurements were exact, the parabola would predict correctly at both new inputs. A straight line would miss genuine curvature. When a model is too simple to capture a pattern that matters for prediction, the problem is called underfitting.

Instead of judging a model by how complicated it looks, compare its errors on training data and on examples held out from training. If adding flexibility reduces training error but repeatedly increases error on held-out data, that is evidence of overfitting. A common setup uses a validation set to choose between models and a separate test set to check the chosen model. If you keep choosing models based on how they score on the test set, it stops serving as an independent final check. Scikit-learn's guide to evaluating models on held-out data explains how cross-validation can help when you have limited data.

Move one dot and watch what changes

On a fresh copy of the scatterplot, move the middle training dot from (1, 3) to (1, 2). Leave the open circles alone. The three training dots now lie on y = 2x, so the best straight line and the polynomial of degree at most two through the dots become the same line. Neither fit needs to miss a training point to predict the two open circles correctly.

Try moving the middle dot down one more unit and sketch where each fit would go before calculating anything. Mark which fit changes its slope across the plot and which only shifts vertically. For more practice with model behavior, see IMLC's training and practice problems and machine learning learning resources.

Frequently asked questions

Does zero training error always mean overfitting?
No. A model can fit every training point and still predict new data well. You need evidence from data the model did not train on.

Can a straight line overfit too?
Yes. Even a straight line can pick up a misleading slope if its training sample is small or noisy.

Why are the new points inside the training range?
They show that overfitting can hurt predictions between known inputs, not just beyond the ends of the scatterplot.

About the International Machine Learning Competition: an online machine learning competition for high school students from around the world. Qualification Round 2026: submissions are open until Sunday, 13 December 2026. How the rounds work

Written by Julia Berens, Social Media Assistant of the IMLC team.

More articles

All articles