House Price Tiers an ML project, end to end

Lesson 6 of 6

Blend & final reveal

Averaging models, then the private leaderboard is revealed.

Download the notebook

Left: what & whyRight: the code and its real output

06 · Ensembling and the final submission

Different models make different mistakes. If we average their predicted probabilities, the mistakes partly cancel out. This is ensembling (or blending), and nearly every winning Kaggle solution does it.

Plan:

  1. get out-of-fold (OOF) probabilities from 5 tuned models (same CV folds for all),
  2. check how often they disagree,
  3. blend and measure the blend with CV,
  4. train on everything, predict test, submit,
  5. 🎉 the competition "ends": reveal the private leaderboard for every submission we made.
import warnings
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from lightgbm import LGBMClassifier
from xgboost import XGBClassifier
from catboost import CatBoostClassifier
from utils import make_xy, cv_score, save_submission, SEED
from score import score

warnings.filterwarnings("ignore")
X, y, X_test, test = make_xy()

The five models, with the best settings found in 05:

models = {
    "logreg": lambda: make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=3000)),
    "rf":     lambda: RandomForestClassifier(n_estimators=500, min_samples_leaf=5, random_state=SEED, n_jobs=-1),
    "lgb":    lambda: LGBMClassifier(n_estimators=300, learning_rate=0.03, num_leaves=4, min_child_samples=10, subsample=0.8,
                                     subsample_freq=1, colsample_bytree=0.7, random_state=SEED, verbose=-1),
    "xgb":    lambda: XGBClassifier(n_estimators=300, learning_rate=0.03, max_depth=2, subsample=0.8, colsample_bytree=0.7,
                                    random_state=SEED),
    "cat":    lambda: CatBoostClassifier(iterations=500, learning_rate=0.03, depth=4, random_seed=SEED, verbose=0),
}
oof, cv = {}, {}
for name, make in models.items():
    cv[name], std, oof[name] = cv_score(make, X, y, verbose=False)
    print(f"{name:7s} CV {cv[name]:.4f} ± {std:.4f}")

logreg  CV 0.7190 ± 0.0481

rf      CV 0.7214 ± 0.0429

lgb     CV 0.7296 ± 0.0397

xgb     CV 0.7296 ± 0.0324

cat     CV 0.7345 ± 0.0308

Do the models disagree?

If two models always predicted the same thing, averaging them would change nothing. Fraction of houses where each pair disagrees (using the OOF predictions):

preds = pd.DataFrame({k: v.argmax(1) for k, v in oof.items()})
pd.DataFrame({a: {b: round((preds[a] != preds[b]).mean(), 3) for b in preds} for a in preds})
logreg rf lgb xgb cat
logreg 0.000 0.189 0.169 0.179 0.159
rf 0.189 0.000 0.081 0.074 0.069
lgb 0.169 0.081 0.000 0.022 0.051
xgb 0.179 0.074 0.022 0.000 0.054
cat 0.159 0.069 0.051 0.054 0.000

Logistic Regression thinks completely differently from the tree models (straight-line rule vs yes/no questions), so it disagrees with them the most. The three boosting models are close cousins and mostly agree with each other. Different kinds of model bring the most diversity.

Blending

Average the probabilities, then pick the most likely tier. Measured on the OOF predictions, so it's still an honest CV score:

def blend_acc(names):
    proba = np.mean([oof[n] for n in names], axis=0)
    return accuracy_score(y, proba.argmax(1))

combos = [["cat"], ["logreg", "cat"], ["lgb", "xgb", "cat"], ["logreg", "lgb", "xgb", "cat"], ["logreg", "rf", "lgb", "xgb", "cat"]]
blend_df = pd.DataFrame({"blend": [" + ".join(c) for c in combos], "oof_acc": [round(blend_acc(c), 4) for c in combos]})
blend_df
blend oof_acc
0 cat 0.7377
1 logreg + cat 0.7328
2 lgb + xgb + cat 0.7377
3 logreg + lgb + xgb + cat 0.7377
4 logreg + rf + lgb + xgb + cat 0.7377

⚠️ Two honesty notes:

  • These numbers are accuracy on the OOF predictions averaged over 3 repeats. That's slightly kinder than the per-fold numbers above, so compare blends with each other, not with the single-model CV.
  • Picking the best of many blends is itself a mild form of fitting to validation data. With 5 candidates it's fine. With hundreds you'd start overfitting your CV.

We keep it simple and robust: equal weights over all 5 models. Even when it isn't the very top row, an all-model average is the least likely to be a lucky pick.

Final submission

Train each model on all training data, average their test probabilities, submit.

test_proba = {}
for name, make in models.items():
    test_proba[name] = make().fit(X, y).predict_proba(X_test)
    save_submission(test, test_proba[name].argmax(1), f"06_{name}")

final_proba = np.mean([test_proba[n] for n in models], axis=0)
path = save_submission(test, final_proba.argmax(1), "06_final_blend")
print("final blend - public leaderboard:", score(path))

final blend - public leaderboard: {'public': 0.6935}

🎉 The competition ends: private leaderboard revealed

Until now we only saw the public score (~60 houses). The final ranking uses the private part (~75 houses), which we've never seen. Here is every submission we made, next to its CV score (repeated 5-fold):

cv_known = {"02_majority": None, "02_random_forest": 0.6961,
            "04_leaky_target_encoding": 0.7998, "04_features_te": 0.7001, "05_best_single": 0.7345}
for name in models:
    cv_known[f"06_{name}"] = round(cv[name], 4)
cv_known["06_final_blend"] = round(blend_acc(list(models)), 4)

rows = []
for name, c in cv_known.items():
    s = score(f"submissions/{name}.csv", show_private=True)
    rows.append({"submission": name, "cv": c, "public": s["public"], "private": s["private"]})
board = pd.DataFrame(rows)
board
submission cv public private
0 02_majority NaN 0.4032 0.2933
1 02_random_forest 0.6961 0.6290 0.6533
2 04_leaky_target_encoding 0.7998 0.5323 0.4533
3 04_features_te 0.7001 0.7258 0.6000
4 05_best_single 0.7345 0.6935 0.7067
5 06_logreg 0.7190 0.7097 0.7333
6 06_rf 0.7214 0.6935 0.7333
7 06_lgb 0.7296 0.7097 0.7333
8 06_xgb 0.7296 0.6935 0.7333
9 06_cat 0.7345 0.6935 0.7067
10 06_final_blend 0.7377 0.6935 0.7467

How to read this table

  • The leaky submission has by far the highest CV, and the worst score of every real model on both leaderboards. Leakage makes CV lie.
  • Public vs private for the same submission can differ by 5+ points. With only ~60 and ~75 houses, a handful of houses decides the score. On a small test set the leaderboard is mostly luck, and your CV is the only steady signal.
  • CV is noisy too (±0.8%), so models within ~1–2% of each other are effectively tied. Don't over-read small gaps anywhere.
  • The final blend is what we'd pick in a real competition: built on honest CV, combining different kinds of model, with no tuning to the leaderboard.

Full journey

majority guess → Random Forest → honest (repeated) CV → feature ideas tested (and rejected) → leak caught → regularised models → blend.

That's the whole workflow. For the real competition, it is the same steps with a different CSV. See GUIDE.md.

← Models & tuningFull guide →