Overfitting and Underfitting in Machine Learning Explained

Quick answer: Learn the difference between overfitting and underfitting, how to detect each with a bias-variance perspective, and the standard techniques to fix them.

What is Overfitting?

Overfitting occurs when a model learns the training data too specifically, including its noise and random quirks, rather than the genuine underlying pattern. An overfit model performs very well on training data but poorly on new, unseen data.

What is Underfitting?

Underfitting occurs when a model is too simple to capture the genuine underlying pattern in the data at all, performing poorly on both training data and new data.

The Bias-Variance Perspective

Bias is the error from overly simplistic assumptions. Variance is the error from excessive sensitivity to the specific training data. Good model building looks for the sweet spot balancing both.

How to Detect Overfitting and Underfitting

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)

print(f"Training score: {train_score:.3f}, Test score: {test_score:.3f}")

A large gap where training score is much higher than test score signals overfitting. Both scores being low together signals underfitting.

How to Fix Overfitting

• Gather more training data

• Simplify the model, using fewer features or a less complex algorithm

• Apply regularisation (L1 or L2 penalty) to constrain model complexity

• Use cross validation to get a more reliable performance estimate

• For tree-based models, limit tree depth or the number of leaf nodes

• Use dropout for neural networks

How to Fix Underfitting

• Use a more complex model

• Add more relevant features through feature engineering

• Reduce regularisation strength if it is currently too aggressive

• Train for longer, or with more iterations, in the case of iterative algorithms

Using Learning Curves to Diagnose

Plotting training and validation error against training set size reveals the pattern clearly. If both curves converge to a high error, that indicates underfitting. If there is a persistent large gap between a low training error and a high validation error, that indicates overfitting.

Why Cross Validation Helps

A single train-test split can give a misleadingly optimistic or pessimistic result by chance. K-Fold Cross Validation trains and tests across multiple different splits, giving a more reliable estimate of how the model actually generalises.

Common Interview Questions

How would you tell overfitting apart from underfitting just from the numbers?

Overfitting shows a large gap between a high training score and a much lower test score. Underfitting shows both scores being low, close to each other.

What is regularisation and how does it help with overfitting?

Regularisation adds a penalty term that discourages overly complex models, such as very large coefficients in a linear model, encouraging the model to find a simpler pattern that generalises better.