Overfitting and Underfitting
The central tension in ML: a model too simple misses patterns, too complex memorises noise.
Every model balances two failure modes:
- Underfitting — too simple; poor on both training and test data (high bias).
- Overfitting — too complex; great on training data, poor on test data (high variance). It memorised noise.
How to spot and fix it
Compare training vs test scores. A big gap means overfitting. Fixes: more data, simpler model, regularisation, or fewer features.
Example
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
import numpy as np
rng = np.random.RandomState(0)
X = rng.rand(200, 5); y = (X[:,0] > 0.5).astype(int)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
deep = DecisionTreeClassifier(max_depth=None).fit(Xtr, ytr)
print('train', deep.score(Xtr, ytr)) # ~1.00 (memorised)
print('test ', deep.score(Xte, yte)) # lower -> overfitWhen to use it
- An ML engineer detects overfitting by plotting training and validation loss per epoch and seeing validation loss rise while training loss keeps falling.
- A researcher applies L2 (Ridge) regularisation to reduce a logistic regression model's tendency to memorise noise in a small training set.
- A data scientist uses early stopping during neural network training to halt updates once validation accuracy stops improving, preventing memorisation.
More examples
Detect overfitting with learning curves
Computes training and validation scores across different training-set sizes; a large gap between the two at full data size signals overfitting.
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import learning_curve
from sklearn.datasets import load_digits
import numpy as np
X, y = load_digits(return_X_y=True)
tree = DecisionTreeClassifier(max_depth=None) # no limit = overfits
train_sizes, train_scores, val_scores = learning_curve(tree, X, y, cv=5)
print('Train score at max data:', train_scores[-1].mean().round(3))
print('Val score at max data: ', val_scores[-1].mean().round(3))Regularisation reduces overfitting
Sweeps the regularisation strength C (lower = stronger regularisation) and compares cross-validated accuracy to find the optimal balance between bias and variance.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
for C in [0.01, 0.1, 1.0, 10.0, 100.0]:
model = LogisticRegression(C=C, max_iter=5000)
cv = cross_val_score(model, X, y, cv=5).mean()
print(f'C={C}: CV accuracy={cv:.4f}')max_depth controls decision-tree bias/variance
Shows how increasing max_depth raises training accuracy but eventually causes test accuracy to fall, directly illustrating the overfitting phenomenon.
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_digits
X, y = load_digits(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
for depth in [1, 3, 5, 10, None]:
tree = DecisionTreeClassifier(max_depth=depth).fit(X_tr, y_tr)
print(f'depth={depth}: train={tree.score(X_tr,y_tr):.3f} test={tree.score(X_te,y_te):.3f}')
Discussion