Ideas worth a closer look
VIVA PARADOX.Independent stories, clearly told.
AI Guides

What Is Overfitting in Machine Learning? A Practical Guide

Conceptual editorial artwork: What Is Overfitting in Machine Learning? A Practical Guide

Overfitting in machine learning happens when a model fits the training data so closely that it fails to generalize well to new examples. The model may appear strong during development, but its performance drops on validation data, test data, or real-world inputs it has not seen before.

For students, analysts, software engineers, and product teams, the key point is practical: do not judge a model by training performance alone. A model that memorizes quirks in historical data can look accurate in a notebook and still make unreliable predictions after deployment.

Reproduce a small overfitting example

This locally executed synthetic polynomial calculation is separate from the house-price illustration in this article. It is not a product benchmark or a real housing dataset.

The six training inputs are −1, −0.6, −0.2, 0.2, 0.6, 1. Targets are x² plus fixed noise: 0.08, −0.12, 0.10, −0.10, 0.12, −0.08. Fit degree-2 and degree-5 polynomials by least squares. Validate at −0.9, −0.7, …, 0.9 against x². MSE is the mean squared prediction error.

DegreeTraining MSEValidation MSE
20.010080.000132
5< 1e-200.0223906162

The degree-5 polynomial fits all six noisy training points almost exactly but has higher validation error. The degree-2 polynomial has training MSE 0.01008 and validation MSE 0.000132. The degree-5 validation MSE is approximately 0.0223906162. Lower training error did not select the better generalizing model in this fixture.

The underlying function and noise are invented, and validation targets are deliberately noiseless to isolate the effect. Real labels need not be noiseless. This is one small example, not evidence that degree 2 or simpler models always win. Choosing a model using these validation values does not produce an independent final test estimate.

Download Python source (no dependencies) · Download reference calculations (JSON)

What overfitting means

A machine learning model is trained to learn patterns from examples. Overfitting occurs when the model learns details that are too specific to the training set, including noise, accidental correlations, or unusual cases that do not represent the broader problem.

Google's Machine Learning Crash Course explains overfitting as a situation where a model fits the training data well but does not generalize effectively to new examples. Google also emphasizes the training-versus-validation behavior: if training performance keeps improving while validation performance lags behind or worsens, the model may be overfitting.

That distinction matters. High training accuracy by itself does not prove overfitting. A model can perform well on training data for legitimate reasons. The warning sign is the gap between how the model behaves on data it learned from and how it behaves on data it did not learn from.

Why overfitting matters

Overfitting turns a model into a poor decision-maker outside its development environment. A fraud model may learn quirks of last quarter's transactions. A recommendation model may learn temporary user behavior. A house-price model may memorize a few unusual listings instead of learning broader market relationships.

The consequence is false confidence. During training, the metrics may look impressive. In production, the model may make worse predictions because new users, records, documents, listings, or images do not match the narrow details it memorized.

Microsoft's Azure Machine Learning guidance discusses overfitting as a practical machine learning pitfall and connects prevention to evaluation practices, not just model definitions. In particular, Microsoft notes the importance of validation and testing practices because good training performance alone is not enough evidence that a model will perform well on unseen data.

Training data vs. validation data vs. test data

To recognize overfitting, you need to understand the role of each dataset.

  • Training data is the data the model uses to learn patterns. The model's parameters are adjusted based on this data.
  • Validation data is used during development to compare models, tune settings, and decide whether performance is improving on data the model did not train on.
  • Test data is usually reserved for a final evaluation after model choices are made. It provides a less biased estimate of how the finished model may perform on unseen data.

The exact split depends on the project, dataset size, and modeling approach. The principle is more important than the percentage: keep some data separate from training so you can evaluate generalization.

How to recognize overfitting

A common overfitting pattern looks like this:

  1. Training error keeps decreasing, or training accuracy keeps rising.
  2. Validation error stops improving, improves only slightly, or starts getting worse.
  3. The gap between training performance and validation performance widens.

This pattern suggests that the model is becoming better at fitting the training examples but not better at handling new examples. Google's overfitting guidance highlights this type of training-versus-validation gap as a practical signal.

There is no universal threshold that says a specific gap always means overfitting. A tolerable gap depends on the metric, domain, dataset size, model type, and cost of mistakes. A medical screening model, a retail forecasting model, and a movie recommendation model may all require different standards.

Worked example: house-price prediction

The following example is hypothetical. The inputs and numbers are invented for explanation; they are not measured results from an executed experiment.

Imagine you are building a model to predict house prices from features such as:

  • Square footage
  • Number of bedrooms
  • ZIP code
  • Lot size
  • Year built
  • Recent renovation status

You train two models on a small dataset of past listings.

Model Training behavior Validation behavior Interpretation
Model A: simple model Makes some mistakes on training listings Makes similar mistakes on validation listings May be learning broad patterns, though it might still be too simple
Model B: very complex model Nearly matches the training listings Performs worse on validation listings May be memorizing quirks in the training set

A useful inspection step is to ask what Model B may have learned. Suppose several expensive homes in the training data happened to share a rare combination: a specific ZIP code, unusually large lot size, and a recent renovation. A complex model might treat that combination as a near-certain sign of a high price, even if the pattern is accidental or based on too few examples.

Reference answer: If Model B's training error keeps falling while its validation error rises, investigate overfitting before making the model more complex or training it longer.

Reason: The model is improving on examples it already learned from but worsening on examples held out from training. That is the practical pattern associated with overfitting.

Boundary case: If the validation set is too small or not representative of the homes the model will see later, the conclusion is less reliable. A misleading validation set can exaggerate overfitting, hide it, or point you toward the wrong fix.

Common causes of overfitting

Overfitting usually comes from a mismatch between model flexibility, data quality, and evaluation design.

A model is too complex for the available data

A highly flexible model can capture subtle relationships, but it can also learn noise. If the training dataset is small or narrow, extra complexity may help the model memorize rather than generalize.

The training data is not representative

A model trained on a narrow slice of reality may not perform well elsewhere. For example, a model trained only on one region, season, customer segment, or time period may struggle when conditions change.

Features contain noise or accidental correlations

Some input features may correlate with the target in the training data for reasons that do not hold later. The model may treat those accidental relationships as meaningful.

Data leakage distorts the evaluation

Data leakage happens when information unavailable at prediction time finds its way into training or evaluation. Leakage can make model performance look unrealistically good. It can also coexist with overfitting or obscure it, because the evaluation no longer reflects the real prediction problem.

The model is tuned too aggressively to the validation set

Validation data helps select models, but repeatedly optimizing every choice around the same validation set can make the validation score less trustworthy. In that case, the model development process may start fitting the validation set indirectly.

How to reduce overfitting

No remedy guarantees better real-world performance, but several practices can reduce the risk.

Use a simpler model when appropriate

If validation performance gets worse as the model becomes more complex, try a simpler model. Simpler models have fewer opportunities to memorize accidental details. The tradeoff is that a model can also become too simple and underfit, so compare validation performance rather than assuming simpler is always better.

Apply regularization

Regularization adds a constraint or penalty that discourages overly complex patterns. In practice, it can help a model focus on broader relationships rather than extreme fits to the training data. The right regularization choice depends on the model and task.

Stop training earlier

For models trained over many iterations, early stopping can help. If validation performance stops improving while training performance continues to improve, stopping earlier may preserve better generalization.

Use cross-validation when it fits the project

Cross-validation evaluates the model across multiple train-validation splits. This can provide a more stable view than a single split, especially when the dataset is limited. It does not fix bad data, but it can reveal whether performance depends heavily on one lucky or unlucky split.

Review features for leakage and unstable signals

Inspect whether features would truly be available when the model makes predictions. Also review whether a feature represents a stable relationship or a temporary artifact of the dataset.

Improve the data

More representative data can help a model learn patterns that generalize. This may mean collecting examples from more user groups, time periods, regions, product categories, or operating conditions. More data is not automatically better if it is noisy, biased, duplicated, or from the wrong distribution.

Keep a final test set for the end

A separate test set helps evaluate the finished model after major choices are made. This reduces the risk that repeated development decisions have overfit the validation set.

A practical decision rule

Use this rule during model development:

If validation performance worsens while training performance keeps improving, stop increasing model complexity and investigate overfitting, data leakage, and validation-data quality.

That rule does not prove overfitting in every case. It tells you what to inspect next. Ask:

  • Is the validation set representative of future data?
  • Is there possible leakage from the target or future information?
  • Did performance degrade after adding complexity, features, or training time?
  • Are the errors concentrated in a specific segment?
  • Would a simpler or more regularized model perform better on validation data?

The goal is not to maximize the training score. The goal is to build a model that performs reliably on the data it will actually encounter.

Realistic failure case: when validation misleads you

A validation set is only useful if it reflects the real prediction problem. Suppose a customer churn model is trained on historical users and validated on users from the same time period. If the business later changes pricing, onboarding, or product features, future users may behave differently. The validation result may have been honest for the old data but less useful for the new environment.

The opposite can also happen. A validation set may be unusually difficult, unusually small, or drawn from a different population than the intended deployment setting. In that case, it may make a reasonable model look overfit.

This is why overfitting analysis should not stop at a metric gap. The gap is a signal. The next step is to inspect the data split, feature design, modeling choices, and intended use.

Overfitting vs. underfitting

Overfitting and underfitting are opposite failure modes.

Problem Training performance Validation performance What it usually means
Overfitting Strong Weaker The model learned training-specific details that do not generalize
Underfitting Weak Weak The model is too simple, poorly specified, or missing important information

An underfit model has not learned enough from the training data. An overfit model has learned too much of the wrong detail from the training data. In both cases, the solution depends on validation behavior and error analysis, not on a single metric in isolation.

Bottom line

Overfitting in machine learning is a generalization problem: the model fits training data well but performs worse on new or validation data. The clearest warning sign is a widening gap where training performance improves while validation performance stalls or declines.

When that happens, do not simply train longer or make the model more complex. Check the data split, look for leakage, review features, consider regularization or a simpler model, and verify that validation data represents the real-world problem. The best model is not the one that memorizes the past most precisely; it is the one that handles new examples reliably.