Lesson 3 of 6

Honest validation

Hold-out, overfitting curves, repeated K-fold. The score you can trust.

Left: what & whyRight: the code and its real output

03 · Validation: an honest score we can compute ourselves

In 02 the model scored ~1.00 on training data but much less on the leaderboard. Here we:

hide part of our own training data and grade the model on it (hold-out validation),
watch overfitting happen in a picture,
see how noisy one split is on a small dataset,
upgrade to repeated K-fold cross-validation (CV), the score we'll trust for the rest of the project.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split, StratifiedKFold
from utils import load_data, basic_prep, cv_score, TARGET, TIER_TO_NUM, SEED
from score import score

train, test = load_data()
tr, te = basic_prep(train, test)
FEATURES = [c for c in tr.columns if c not in ("id", TARGET)]
X, y = tr[FEATURES], tr[TARGET].map(TIER_TO_NUM)

def rf():
    return RandomForestClassifier(n_estimators=300, random_state=SEED, n_jobs=-1)

1. Hold-out validation

Split train into 80% "study material" and 20% "practice exam". The model never sees the practice exam while learning.

stratify=y keeps the same tier mix (⅓ each) in both parts.

X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.2, random_state=SEED, stratify=y)
print(X_tr.shape, X_va.shape)

m = rf().fit(X_tr, y_tr)
print(f"train accuracy:      {accuracy_score(y_tr, m.predict(X_tr)):.4f}")
print(f"validation accuracy: {accuracy_score(y_va, m.predict(X_va)):.4f}")
print("leaderboard (from 02):", score("submissions/02_random_forest.csv"))

(326, 12) (82, 12)

train accuracy:      1.0000
validation accuracy: 0.6707
leaderboard (from 02): {'public': 0.629}

The validation score is far below the training score and much closer to the leaderboard. Now we have a leaderboard-like score without submitting anything.

2. Overfitting, in a picture

A single decision tree asks yes/no questions ("area > 6000?"). max_depth = how many questions in a row it may ask. Shallow tree = too simple (underfitting). Very deep tree = memorises every row (overfitting).

depths = range(1, 16)
train_scores, val_scores = [], []
for d in depths:
    tree = DecisionTreeClassifier(max_depth=d, random_state=SEED).fit(X_tr, y_tr)
    train_scores.append(accuracy_score(y_tr, tree.predict(X_tr)))
    val_scores.append(accuracy_score(y_va, tree.predict(X_va)))

plt.figure(figsize=(8, 4))
plt.plot(depths, train_scores, label="train")
plt.plot(depths, val_scores, label="validation")
plt.xlabel("max_depth"); plt.ylabel("accuracy"); plt.legend(); plt.title("Train vs validation accuracy");

Left side: both scores are low → the tree is too simple.
Right side: train shoots up towards 1.0 while validation stays flat or drops → it is memorising noise.

The gap between train and validation is how you spot overfitting.

3. How noisy is one split? (very, on small data)

Same model, 10 different random 80/20 splits:

split_scores = []
for seed in range(10):
    a, b, c, d = train_test_split(X, y, test_size=0.2, random_state=seed, stratify=y)
    split_scores.append(accuracy_score(d, rf().fit(a, c).predict(b)))
    print(f"split {seed}: {split_scores[-1]:.4f}")
print(f"range: {min(split_scores):.3f} – {max(split_scores):.3f}")

split 0: 0.6829

split 1: 0.7439

split 2: 0.7317

split 3: 0.7683

split 4: 0.7805

split 5: 0.6585

split 6: 0.7073

split 7: 0.7317

split 8: 0.7439

split 9: 0.7195
range: 0.659 – 0.780

The validation part has only ~80 houses, so each house is worth more than 1% of the score. Just which houses land in the 20% moves the result by several percent. If a new idea "improves" the score by 2%, one split can't tell us if it's real or luck.

4. K-fold cross-validation

Cut the training data into 5 equal folds. Train on 4, score on the 5th, and repeat so every fold gets a turn as the exam:

fold 0: [VAL][ tr ][ tr ][ tr ][ tr ]
fold 1: [ tr ][VAL][ tr ][ tr ][ tr ]
...
fold 4: [ tr ][ tr ][ tr ][ tr ][VAL]

CV score = the average of the 5. Every row is used for validation exactly once.

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=SEED)
scores = []
for fold, (tr_idx, va_idx) in enumerate(skf.split(X, y)):
    m = rf().fit(X.iloc[tr_idx], y.iloc[tr_idx])
    scores.append(accuracy_score(y.iloc[va_idx], m.predict(X.iloc[va_idx])))
    print(f"fold {fold}: {scores[-1]:.4f}")
print(f"CV accuracy: {np.mean(scores):.4f} ± {np.std(scores):.4f}")

fold 0: 0.6951

fold 1: 0.7195

fold 2: 0.7561

fold 3: 0.6914

fold 4: 0.6543
CV accuracy: 0.7033 ± 0.0336

Repeated K-fold: for small data

Because our data is tiny, we go one step further: run the 5-fold 3 times with different shuffles, and average all 15 scores. This is cv_score() in utils.py, and every later notebook calls it in one line:

mean, std, oof = cv_score(rf, X, y)

  repeat 0 fold 0: 0.6951

  repeat 0 fold 1: 0.7195

  repeat 0 fold 2: 0.7561

  repeat 0 fold 3: 0.6914

  repeat 0 fold 4: 0.6543

  repeat 1 fold 0: 0.6951

  repeat 1 fold 1: 0.6829

  repeat 1 fold 2: 0.6829

  repeat 1 fold 3: 0.6420

  repeat 1 fold 4: 0.7160

  repeat 2 fold 0: 0.6829

  repeat 2 fold 1: 0.6585

  repeat 2 fold 2: 0.7195

  repeat 2 fold 3: 0.7037

  repeat 2 fold 4: 0.7407
CV accuracy: 0.6961 ± 0.0303

(oof = "out-of-fold" predictions: every training row predicted by models that never saw it. We'll use these in 06.)

The rule for the rest of the project

Trust your CV score, not the public leaderboard.

Here the public leaderboard is only ~60 houses, so one house = ~1.6%. Our CV uses all ~400 training houses, 3 times over. If CV improves but public drops a little, believe CV. Teams that chase the public leaderboard often fall when the private leaderboard is revealed (a "shake-up").

Next: 04_feature_engineering, to push the CV score up.