Skip to content

Chapter 10: Model Evaluation and Cross-Validation

Estimating honestly how well a model will do on data it has never seen

Learning outcomes

After this chapter you should be able to:

  • explain why a model must be judged on data not used to build it, and what the test set is for;
  • use training, validation and test sets correctly;
  • estimate performance with k-fold cross-validation and report its mean and spread;
  • tune hyperparameters with a grid search inside cross-validation;
  • recognise and prevent data leakage, including leakage through preprocessing, time and groups;
  • read a ROC curve and its AUC, and a precision–recall curve;
  • compare models fairly and report results with their uncertainty.

A model is useful only if it works on cases it has not seen: next month's customers, tomorrow's transactions, the patient who walks in today. Its performance on the data it was trained on says little about that, because a flexible model can memorise its training data (Chapters 8 and 9). This chapter is about estimating generalisation performance honestly. It is the stage of the workflow where most of the self-deception in data science happens, usually by accident, and the habits it teaches matter more than any single algorithm.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier

We reuse the churn data of Chapter 9, generated by the same code, wrapped in a function so that it can be called again:

def make_churn(n=1000, seed=10):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charge = rng.uniform(200, 1200, n).round(0)
    calls = rng.poisson(1.5, n)
    contract = rng.choice(["monthly", "yearly"], n, p=[0.55, 0.45])
    logit = (-2.5 - 0.05 * tenure + 0.0025 * charge + 0.45 * calls
             + np.where(contract == "monthly", 1.0, -0.8))
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    X = pd.DataFrame({"tenure": tenure, "charge": charge, "calls": calls,
                      "monthly": (contract == "monthly").astype(int)})
    return X, pd.Series(churn, name="churn")

X, y = make_churn()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1, stratify=y)
print(X_train.shape, X_test.shape, round(y.mean(), 3))
(800, 4) (200, 4) 0.244

10.1 Three roles for data

Data in a modelling project plays three distinct roles.

  • The training set is used to fit the model's parameters (coefficients, tree splits).
  • The validation set is used to make choices: which model, which features, which hyperparameters (\(k\) in k-NN, depth of a tree, \(\alpha\) in ridge).
  • The test set is used once, at the end, to estimate how the final chosen model will perform on new data.

Why not make the choices on the test set? Because every choice made by looking at a dataset adapts the model to that dataset. Try twenty models and keep the one with the best test score, and that score is the maximum of twenty noisy numbers, which is optimistic. The test set must be kept out of every decision so that its score remains an honest estimate. Think of it as an exam paper: if you have seen the questions while studying, your mark no longer measures what you know.

Key idea

Any data used to make a decision about the model, even just to choose between two options, is no longer a fair test. Hold the test set back until all decisions are made, and look at it once.

A single validation set is wasteful when data is limited, and its score is noisy: it depends on which examples happened to land in it. Cross-validation solves both problems.

10.2 k-fold cross-validation

In k-fold cross-validation the training data is divided into \(k\) equal parts, called folds. The model is trained \(k\) times, each time on \(k - 1\) folds, and scored on the remaining fold. Every example is used for validation exactly once, and the \(k\) scores are averaged (Figure 10.1). Five or ten folds are the usual choices.

The data is first split into a training part (80%) and a test part (20%) that is set aside. The training part is divided into five folds. In each of five rounds a different fold is used for validation and the other four for training, giving five scores, s1 to s5, whose mean is the cross-validated score.
Figure 10.1 A test set and five-fold cross-validation. The data is first split into a training part (80%) and a test part (20%) that is set aside. The training part is divided into five folds. In each of five rounds a different fold is used for validation and the other four for training, giving five scores, s1 to s5, whose mean is the cross-validated score.
logreg = make_pipeline(StandardScaler(), LogisticRegression())
scores = cross_val_score(logreg, X_train, y_train, cv=5, scoring="accuracy")
print(scores.round(3))
print(f"mean {scores.mean():.3f}, sd {scores.std():.3f}")
[0.819 0.856 0.825 0.819 0.831]
mean 0.830, sd 0.014

The five scores differ by several percentage points, which shows how much a single split could have misled us. Report both the mean and the spread. For classification, scikit-learn's cv=5 uses stratified folds, each with the same proportion of each class as the whole, which keeps the folds comparable when one class is rare.

The scoring argument chooses the metric: "accuracy", "f1", "precision", "recall", "roc_auc" for classifiers; "r2", "neg_mean_absolute_error", "neg_root_mean_squared_error" for regressors (negated so that larger is always better). cross_validate computes several at once:

from sklearn.model_selection import cross_validate

res = cross_validate(logreg, X_train, y_train, cv=5, scoring=["accuracy", "f1", "roc_auc"])
print(pd.DataFrame(res).drop(columns=["fit_time", "score_time"]).agg(["mean", "std"]).round(3))
      test_accuracy  test_f1  test_roc_auc
mean          0.830    0.600         0.859
std           0.016    0.055         0.019

Comparing models

Cross-validation puts several candidate models on the same footing. Using the same folds for each (by passing the same cv object) makes the comparison fairer still.

from sklearn.model_selection import StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.dummy import DummyClassifier

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
candidates = {
    "baseline": DummyClassifier(strategy="most_frequent"),
    "logistic": make_pipeline(StandardScaler(), LogisticRegression()),
    "k-NN (k=25)": make_pipeline(StandardScaler(), KNeighborsClassifier(25)),
    "tree (depth 3)": DecisionTreeClassifier(max_depth=3, random_state=0),
    "forest": RandomForestClassifier(n_estimators=200, min_samples_leaf=5, random_state=0),
}
for name, model in candidates.items():
    s = cross_val_score(model, X_train, y_train, cv=cv, scoring="roc_auc")
    print(f"{name:>15}: AUC {s.mean():.3f} +/- {s.std():.3f}")
       baseline: AUC 0.500 +/- 0.000
       logistic: AUC 0.854 +/- 0.004
    k-NN (k=25): AUC 0.839 +/- 0.007
 tree (depth 3): AUC 0.763 +/- 0.019
         forest: AUC 0.828 +/- 0.019

The AUC (Section 10.5) of the baseline is exactly 0.5: it ranks every customer the same. Logistic regression comes out on top here, which is no surprise since the data was generated by a logistic rule; on real data the ranking could be different, which is exactly why we compare. The gaps between the top three are only one to three hundredths of AUC, comparable to the fold-to-fold spread of the forest, so we should not read too much into their exact order. The shallow tree is clearly behind.

10.3 Tuning hyperparameters

Hyperparameters are settings chosen before training: \(k\) for k-NN, the depth of a tree, the penalty \(C\) of logistic regression. The standard way to choose them is to try a grid of values and keep the one with the best cross-validated score. GridSearchCV does exactly this, and then refits the best setting on the whole training set.

from sklearn.model_selection import GridSearchCV

knn = make_pipeline(StandardScaler(), KNeighborsClassifier())
grid = GridSearchCV(knn, param_grid={"kneighborsclassifier__n_neighbors": [1, 3, 5, 9, 15, 25, 41, 61, 91]},
                    cv=cv, scoring="roc_auc")
grid.fit(X_train, y_train)
print("best k:", grid.best_params_, "CV AUC", round(grid.best_score_, 3))
results = pd.DataFrame(grid.cv_results_)[["param_kneighborsclassifier__n_neighbors", "mean_test_score", "std_test_score"]]
print(results.round(3).to_string(index=False))
best k: {'kneighborsclassifier__n_neighbors': 25} CV AUC 0.839
 param_kneighborsclassifier__n_neighbors  mean_test_score  std_test_score
                                       1            0.669           0.039
                                       3            0.751           0.020
                                       5            0.793           0.013
                                       9            0.810           0.021
                                      15            0.823           0.011
                                      25            0.839           0.007
                                      41            0.836           0.016
                                      61            0.832           0.019
                                      91            0.834           0.024

Parameters of a step inside a pipeline are named step__parameter, with two underscores; make_pipeline names each step after its class in lower case.

A plot of the score against the hyperparameter, a validation curve, shows the trade-off between underfitting and overfitting. Here it is for the depth of a decision tree, with the training score for comparison:

from sklearn.model_selection import validation_curve

depths = np.arange(1, 16)
train_sc, val_sc = validation_curve(DecisionTreeClassifier(random_state=0), X_train, y_train,
                                    param_name="max_depth", param_range=depths, cv=cv, scoring="roc_auc")
fig, ax = plt.subplots(figsize=(6.5, 3.5))
ax.plot(depths, train_sc.mean(axis=1), marker="o", label="training score")
ax.plot(depths, val_sc.mean(axis=1), marker="o", label="cross-validated score")
ax.set_xlabel("maximum depth of the tree")
ax.set_ylabel("ROC AUC")
ax.set_title("Validation curve for a decision tree")
ax.legend()
fig.savefig("validation-curve.svg")
print("best depth:", depths[val_sc.mean(axis=1).argmax()])
best depth: 5
A validation curve for a decision tree. The training score rises steadily with depth towards 1. The cross-validated score rises at first, peaks at a small depth, and then falls as the tree grows deeper, while the gap between the two curves widens.
Figure 10.2 Underfitting on the left, overfitting on the right. A validation curve for a decision tree. The training score rises steadily with depth towards 1. The cross-validated score rises at first, peaks at a small depth, and then falls as the tree grows deeper, while the gap between the two curves widens.

On the left both scores are low: shallow trees underfit. On the right the training score keeps climbing while the validation score falls: deep trees overfit. The best depth lies where the validation score peaks.

Note: the best CV score is itself optimistic

The winning setting of a grid search was selected because its cross-validated score was highest, so that score is slightly optimistic, for the same reason as choosing on the test set. With a held-out test set this does not matter: the final test score is unbiased. When the data is too small for a test set, nested cross-validation (an outer cross-validation loop around the whole grid search) gives an honest estimate.

10.4 Data leakage

Data leakage means that information which would not be available at prediction time finds its way into training or evaluation. It produces models that look excellent in evaluation and fail in use. It takes several forms.

Leakage through preprocessing

Any step that learns from data (a scaler's means, an imputer's medians, a choice of features) must learn from the training folds only. Done on the whole dataset before splitting, it lets the validation data influence training. With a mild step like scaling the effect is small; with feature selection it can be dramatic. Here is a stark demonstration on pure noise: 100 examples, 5,000 random features, and random labels, so that no model can truly do better than 50%.

from sklearn.feature_selection import SelectKBest, f_classif

rng = np.random.default_rng(0)
X_noise = rng.normal(size=(100, 5000))
y_noise = rng.integers(0, 2, 100)

# WRONG: choose the 20 "best" features using all the data, then cross-validate
selected = SelectKBest(f_classif, k=20).fit(X_noise, y_noise).get_support()
wrong = cross_val_score(LogisticRegression(), X_noise[:, selected], y_noise, cv=5)

# RIGHT: feature selection inside the pipeline, refitted within each training fold
right = cross_val_score(make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression()),
                        X_noise, y_noise, cv=5)
print(f"selection outside CV: accuracy {wrong.mean():.2f}")
print(f"selection inside CV:  accuracy {right.mean():.2f}")
selection outside CV: accuracy 0.85
selection inside CV:  accuracy 0.48

Selecting features on all the data finds the 20 noise features that happen to line up with the labels including the validation examples, so cross-validation then reports an accuracy far above chance on data with no signal at all. Inside the pipeline, the selection is made on each training fold alone and the accuracy falls back to about 50%, the truth.

Caution: put every learned step inside the pipeline

Scaling, imputing missing values, encoding categories, selecting features, reducing dimensions: if a step learns anything from the data, put it in a Pipeline with the model and cross-validate the whole pipeline. Then each fold repeats the full procedure on its own training data, exactly as will happen when the model is used.

Target leakage

A feature can leak the answer. In a churn model, a column "cancellation_reason" is filled in only for customers who have already left; in a model to predict hospital readmission, "number of days in hospital this year" may include the readmission itself. Such features predict superbly in historical data and are useless, or impossible to obtain, at prediction time. The defence is to ask of every feature: would I know this value at the moment I need to make the prediction?

Leakage through time and groups

When data has an order in time, random splits let the model train on the future and predict the past. A forecast must be evaluated on periods that come after its training data; scikit-learn's TimeSeriesSplit makes folds that respect order (Chapter 12).

When several rows belong to the same unit (several visits by one patient, several photos of one person, several readings from one machine), a random split puts rows of the same unit in both training and validation, and the model is rewarded for recognising the unit rather than for generalising to new ones. GroupKFold keeps all rows of a group in the same fold.

from sklearn.model_selection import GroupKFold

patients = np.repeat(np.arange(10), 3)      # 10 patients, 3 visits each
gkf = GroupKFold(n_splits=5)
for i, (tr, va) in enumerate(gkf.split(np.zeros(30), groups=patients)):
    print(f"fold {i}: validation patients {sorted(set(patients[va].tolist()))}")
fold 0: validation patients [4, 9]
fold 1: validation patients [3, 8]
fold 2: validation patients [2, 7]
fold 3: validation patients [1, 6]
fold 4: validation patients [0, 5]

10.5 ROC curves, AUC and precision–recall curves

Chapter 9 showed that moving the classification threshold trades precision against recall. Two curves show the whole trade-off at once, for every threshold.

The ROC curve (receiver operating characteristic) plots the true positive rate (recall, \(TP/(TP+FN)\)) against the false positive rate (\(FP/(FP+TN)\)) as the threshold moves from 1 down to 0. A model that ranks perfectly goes straight up to the top-left corner; a model that guesses at random follows the diagonal. The area under the curve (AUC) summarises it in one number: 1 is perfect, 0.5 is random. AUC has a useful interpretation: it is the probability that the model gives a randomly chosen positive example a higher score than a randomly chosen negative one.

The precision–recall curve plots precision against recall over all thresholds. It is more informative than the ROC curve when the positive class is rare, because the false positive rate can look tiny even when false positives far outnumber true positives.

from sklearn.metrics import roc_curve, roc_auc_score, precision_recall_curve, average_precision_score

final = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
scores_test = final.predict_proba(X_test)[:, 1]
fpr, tpr, _ = roc_curve(y_test, scores_test)
prec, rec, _ = precision_recall_curve(y_test, scores_test)

fig, axes = plt.subplots(1, 2, figsize=(10, 3.8))
axes[0].plot(fpr, tpr, label=f"logistic (AUC = {roc_auc_score(y_test, scores_test):.3f})")
axes[0].plot([0, 1], [0, 1], linestyle="--", color="gray", label="random guessing")
axes[0].set_xlabel("false positive rate")
axes[0].set_ylabel("true positive rate (recall)")
axes[0].set_title("ROC curve")
axes[0].legend(loc="lower right")
axes[1].plot(rec, prec)
axes[1].axhline(y_test.mean(), linestyle="--", color="gray", label="baseline (share of positives)")
axes[1].set_xlabel("recall")
axes[1].set_ylabel("precision")
axes[1].set_title(f"Precision-recall (AP = {average_precision_score(y_test, scores_test):.3f})")
axes[1].legend(loc="upper right")
fig.tight_layout()
fig.savefig("roc-pr.svg")
print("test AUC:", round(roc_auc_score(y_test, scores_test), 3))
test AUC: 0.877
Left: a ROC curve that rises steeply from the origin and bends towards the top-left corner, well above the dashed diagonal of random guessing. Right: a precision-recall curve that is jagged at low recall, where only a few customers are flagged, and then falls steadily as recall increases, staying above a dashed horizontal baseline at the share of churners, about 0.24.
Figure 10.3 ROC and precision-recall curves on the test set. Left: a ROC curve that rises steeply from the origin and bends towards the top-left corner, well above the dashed diagonal of random guessing. Right: a precision-recall curve that is jagged at low recall, where only a few customers are flagged, and then falls steadily as recall increases, staying above a dashed horizontal baseline at the share of churners, about 0.24.

The test AUC (0.877) is close to the cross-validated estimate of Section 10.2 (0.854); a gap of this size is normal sampling variation for a test set of 200, and agreement of this kind is what to expect when the test set has been kept clean. The average precision (AP) summarises the precision–recall curve; its baseline is the share of positives, not 0.5.

10.6 Reporting results

A good evaluation report says:

  1. What was predicted, and for whom: the target, the population, the period.
  2. How the data was split: random, by time, by group; the sizes; whether the test set was used only once.
  3. The metric and why it suits the problem: accuracy, F1, AUC, MAE, and the costs of the different errors.
  4. The baseline's score, so the reader can judge the gain.
  5. Cross-validated scores with their spread, and the final test score.
  6. Limitations: groups the model does worse for, conditions under which it was not tested, assumptions about the future resembling the past.

Here is the final step for the churn model, carried out once:

from sklearn.metrics import accuracy_score, f1_score

base = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
pred = final.predict(X_test)
print(f"test accuracy: model {accuracy_score(y_test, pred):.3f}, baseline {accuracy_score(y_test, base.predict(X_test)):.3f}")
print(f"test F1 (churn class): {f1_score(y_test, pred):.3f}")
print(f"test AUC: {roc_auc_score(y_test, scores_test):.3f}")
test accuracy: model 0.840, baseline 0.755
test F1 (churn class): 0.610
test AUC: 0.877

Caution: the test score is an estimate too

With 200 test examples, an accuracy of 0.84 has a standard error of about \(\sqrt{0.84 \times 0.16 / 200} \approx 0.026\), so differences of a couple of percentage points between models are within the noise. Report results with sensible precision (two or three digits), not as if the fourth decimal meant something.

10.7 Summary

Summary

  • Judge models on data not used to build them. Training data fits parameters, validation data (or cross-validation) makes choices, and the test set is used once at the end.
  • k-fold cross-validation trains \(k\) times, each time validating on a different fold; report the mean and spread of the scores. Classification uses stratified folds.
  • Compare candidate models with the same folds, and always include a baseline.
  • Tune hyperparameters with GridSearchCV; a validation curve shows underfitting and overfitting. The best CV score is slightly optimistic; the test set or nested CV corrects that.
  • Leakage makes evaluations falsely optimistic: fit every learned preprocessing step inside a pipeline; drop features unavailable at prediction time; split by time for forecasts and by group when rows share a unit.
  • The ROC curve plots true against false positive rate over all thresholds; AUC is the probability that a random positive outranks a random negative. Precision–recall curves suit rare positive classes.
  • Report the target, the split, the metric, the baseline, CV spread, the test score and the limitations.

10.8 Exercises

Exercise 1

A student tunes \(k\) for k-NN by trying 30 values and picking the one with the best score on the test set, then reports that score. What is wrong, and how should it be done?

Answer

The test set has been used to make a choice, so the reported score is the best of 30 noisy estimates and is optimistic. The value of \(k\) should be chosen by cross-validation (or a separate validation set) within the training data, for example with GridSearchCV; the chosen model is then scored once on the untouched test set.

Exercise 2

With 1,000 training examples and 10-fold cross-validation, how many examples is each model trained on, and how many models are fitted in total? How many are fitted by a grid search over 6 values with 5-fold CV (including the final refit)?

Answer

Each of the 10 models is trained on 900 examples and validated on 100. The grid search fits \(6 \times 5 = 30\) models in cross-validation, plus one final refit of the best setting on all the training data: 31 fits.

Exercise 3

Cross-validate a decision tree of depth 4 and a random forest on the churn training data with the folds cv, using F1 as the metric. Which is better, and is the difference larger than the fold-to-fold spread?

Answer
for name, m in [("tree", DecisionTreeClassifier(max_depth=4, random_state=0)),
                ("forest", RandomForestClassifier(n_estimators=200, min_samples_leaf=5, random_state=0))]:
    s = cross_val_score(m, X_train, y_train, cv=cv, scoring="f1")
    print(f"{name}: F1 {s.mean():.3f} +/- {s.std():.3f}")
tree: F1 0.567 +/- 0.049
forest: F1 0.536 +/- 0.077

Compare the means with the standard deviations: when the gap between the means is smaller than the spread across folds, the evidence that one is better is weak.

Exercise 4

A model to predict whether a loan will default includes the feature "number of missed payments on this loan". The model's AUC is 0.99. Why should you be suspicious?

Answer

Missed payments on the loan are only known after the loan has been running for a while, and they are part of what default means; at the moment of deciding whether to grant the loan, the value would be zero or unknown. This is target leakage: the feature carries information about the outcome that is not available at prediction time, and it explains the implausibly high AUC.

Exercise 5

A classifier's AUC is 0.80. Explain what this means in terms of pairs of examples.

Answer

If you pick one positive example and one negative example at random, the model gives the positive one a higher predicted score 80% of the time. (Ties count as half.)

Exercise 6

A dataset has 5 X-ray images from each of 200 patients. Why is ordinary k-fold cross-validation inappropriate, and what should be used instead?

Answer

Ordinary k-fold puts images of the same patient in both training and validation folds, so the model can score well by recognising patient-specific features rather than the condition, overestimating its performance on new patients. Use GroupKFold with the patient as the group, so that all images of a patient fall in the same fold.

10.9 Quiz

Check your understanding of Chapter 10

Exercise 1

The test set should be used

  1. to choose hyperparameters
  2. to choose between models
  3. once at the end to estimate performance
  4. as extra training data
Answer

once at the end to estimate performance

Exercise 2

In 5-fold cross-validation, each example is used for validation

  1. never
  2. exactly once
  3. five times
  4. four times
Answer

exactly once

Exercise 3

What AUC does a model that guesses at random have?

  1. 0
  2. 0.25
  3. 0.5
  4. 1
Answer

0.5

Exercise 4

Selecting features on the whole dataset before cross-validation is an example of

  1. regularisation
  2. data leakage
  3. stratification
  4. underfitting
Answer

data leakage

Exercise 5

Which splitter keeps all rows of the same patient in one fold?

  1. KFold
  2. StratifiedKFold
  3. GroupKFold
  4. TimeSeriesSplit
Answer

GroupKFold

Exercise 6

On a validation curve, a large gap between a high training score and a low validation score indicates

  1. underfitting
  2. overfitting
  3. leakage
  4. a good model
Answer

overfitting

Exercise 7

Which scikit-learn class searches a grid of hyperparameters with cross-validation?

Answer

GridSearchCV