{ "cells": [ { "cell_type": "markdown", "id": "a0cee112", "metadata": {}, "source": [ "# 06 · Ensembling and the final submission\n", "\n", "Different models make **different mistakes**. If we average their predicted probabilities, the mistakes partly cancel out.\n", "This is **ensembling** (or **blending**), and nearly every winning Kaggle solution does it.\n", "\n", "Plan:\n", "1. get out-of-fold (OOF) probabilities from 5 tuned models (same CV folds for all),\n", "2. check how often they disagree,\n", "3. blend and measure the blend with CV,\n", "4. train on everything, predict test, submit,\n", "5. 🎉 the competition \"ends\": reveal the **private leaderboard** for every submission we made." ] }, { "cell_type": "code", "execution_count": 1, "id": "b1779226", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T06:16:05.099367Z", "iopub.status.busy": "2026-10-23T06:16:05.099367Z", "iopub.status.idle": "2026-10-23T06:16:10.771043Z", "shell.execute_reply": "2026-10-23T06:16:10.771043Z" } }, "outputs": [], "source": [ "import warnings\n", "import numpy as np\n", "import pandas as pd\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.ensemble import RandomForestClassifier\n", "from sklearn.metrics import accuracy_score\n", "from lightgbm import LGBMClassifier\n", "from xgboost import XGBClassifier\n", "from catboost import CatBoostClassifier\n", "from utils import make_xy, cv_score, save_submission, SEED\n", "from score import score\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "X, y, X_test, test = make_xy()" ] }, { "cell_type": "markdown", "id": "b7b224c2", "metadata": {}, "source": [ "The five models, with the best settings found in 05:" ] }, { "cell_type": "code", "execution_count": 2, "id": "ecaa9dbd", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T06:16:10.771043Z", "iopub.status.busy": "2026-10-23T06:16:10.771043Z", "iopub.status.idle": "2026-10-23T06:16:40.304230Z", "shell.execute_reply": "2026-10-23T06:16:40.304230Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "logreg CV 0.7190 ± 0.0481\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "rf CV 0.7214 ± 0.0429\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "lgb CV 0.7296 ± 0.0397\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "xgb CV 0.7296 ± 0.0324\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "cat CV 0.7345 ± 0.0308\n" ] } ], "source": [ "models = {\n", " \"logreg\": lambda: make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=3000)),\n", " \"rf\": lambda: RandomForestClassifier(n_estimators=500, min_samples_leaf=5, random_state=SEED, n_jobs=-1),\n", " \"lgb\": lambda: LGBMClassifier(n_estimators=300, learning_rate=0.03, num_leaves=4, min_child_samples=10, subsample=0.8,\n", " subsample_freq=1, colsample_bytree=0.7, random_state=SEED, verbose=-1),\n", " \"xgb\": lambda: XGBClassifier(n_estimators=300, learning_rate=0.03, max_depth=2, subsample=0.8, colsample_bytree=0.7,\n", " random_state=SEED),\n", " \"cat\": lambda: CatBoostClassifier(iterations=500, learning_rate=0.03, depth=4, random_seed=SEED, verbose=0),\n", "}\n", "oof, cv = {}, {}\n", "for name, make in models.items():\n", " cv[name], std, oof[name] = cv_score(make, X, y, verbose=False)\n", " print(f\"{name:7s} CV {cv[name]:.4f} ± {std:.4f}\")" ] }, { "cell_type": "markdown", "id": "e58788d7", "metadata": {}, "source": [ "## Do the models disagree?\n", "If two models always predicted the same thing, averaging them would change nothing. Fraction of houses where each pair **disagrees** (using the OOF predictions):" ] }, { "cell_type": "code", "execution_count": 3, "id": "40d731d6", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T06:16:40.306092Z", "iopub.status.busy": "2026-10-23T06:16:40.306092Z", "iopub.status.idle": "2026-10-23T06:16:40.333099Z", "shell.execute_reply": "2026-10-23T06:16:40.333099Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
logregrflgbxgbcat
logreg0.0000.1890.1690.1790.159
rf0.1890.0000.0810.0740.069
lgb0.1690.0810.0000.0220.051
xgb0.1790.0740.0220.0000.054
cat0.1590.0690.0510.0540.000
\n", "
" ], "text/plain": [ " logreg rf lgb xgb cat\n", "logreg 0.000 0.189 0.169 0.179 0.159\n", "rf 0.189 0.000 0.081 0.074 0.069\n", "lgb 0.169 0.081 0.000 0.022 0.051\n", "xgb 0.179 0.074 0.022 0.000 0.054\n", "cat 0.159 0.069 0.051 0.054 0.000" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "preds = pd.DataFrame({k: v.argmax(1) for k, v in oof.items()})\n", "pd.DataFrame({a: {b: round((preds[a] != preds[b]).mean(), 3) for b in preds} for a in preds})" ] }, { "cell_type": "markdown", "id": "acb66234", "metadata": {}, "source": [ "Logistic Regression thinks completely differently from the tree models (straight-line rule vs yes/no questions), so it disagrees with them the most.\n", "The three boosting models are close cousins and mostly agree with each other. Different *kinds* of model bring the most diversity." ] }, { "cell_type": "markdown", "id": "df0c5bc1", "metadata": {}, "source": [ "## Blending\n", "Average the probabilities, then pick the most likely tier. Measured on the OOF predictions, so it's still an honest CV score:" ] }, { "cell_type": "code", "execution_count": 4, "id": "08297833", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T06:16:40.337451Z", "iopub.status.busy": "2026-10-23T06:16:40.337451Z", "iopub.status.idle": "2026-10-23T06:16:40.353373Z", "shell.execute_reply": "2026-10-23T06:16:40.353373Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
blendoof_acc
0cat0.7377
1logreg + cat0.7328
2lgb + xgb + cat0.7377
3logreg + lgb + xgb + cat0.7377
4logreg + rf + lgb + xgb + cat0.7377
\n", "
" ], "text/plain": [ " blend oof_acc\n", "0 cat 0.7377\n", "1 logreg + cat 0.7328\n", "2 lgb + xgb + cat 0.7377\n", "3 logreg + lgb + xgb + cat 0.7377\n", "4 logreg + rf + lgb + xgb + cat 0.7377" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def blend_acc(names):\n", " proba = np.mean([oof[n] for n in names], axis=0)\n", " return accuracy_score(y, proba.argmax(1))\n", "\n", "combos = [[\"cat\"], [\"logreg\", \"cat\"], [\"lgb\", \"xgb\", \"cat\"], [\"logreg\", \"lgb\", \"xgb\", \"cat\"], [\"logreg\", \"rf\", \"lgb\", \"xgb\", \"cat\"]]\n", "blend_df = pd.DataFrame({\"blend\": [\" + \".join(c) for c in combos], \"oof_acc\": [round(blend_acc(c), 4) for c in combos]})\n", "blend_df" ] }, { "cell_type": "markdown", "id": "0eccdaad", "metadata": {}, "source": [ "⚠️ Two honesty notes:\n", "- These numbers are accuracy on the OOF predictions *averaged over 3 repeats*. That's slightly kinder than the per-fold numbers above, so compare blends **with each other**, not with the single-model CV.\n", "- Picking the best of many blends is itself a mild form of fitting to validation data. With 5 candidates it's fine. With hundreds you'd start overfitting your CV.\n", "\n", "We keep it simple and robust: **equal weights over all 5 models.** Even when it isn't the very top row, an all-model average is the least likely to be a lucky pick." ] }, { "cell_type": "markdown", "id": "9a4b5d5a", "metadata": {}, "source": [ "## Final submission\n", "Train each model on **all** training data, average their test probabilities, submit." ] }, { "cell_type": "code", "execution_count": 5, "id": "1b2064cd", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T06:16:40.356757Z", "iopub.status.busy": "2026-10-23T06:16:40.356757Z", "iopub.status.idle": "2026-10-23T06:16:42.347366Z", "shell.execute_reply": "2026-10-23T06:16:42.347366Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "final blend - public leaderboard: {'public': 0.6935}\n" ] } ], "source": [ "test_proba = {}\n", "for name, make in models.items():\n", " test_proba[name] = make().fit(X, y).predict_proba(X_test)\n", " save_submission(test, test_proba[name].argmax(1), f\"06_{name}\")\n", "\n", "final_proba = np.mean([test_proba[n] for n in models], axis=0)\n", "path = save_submission(test, final_proba.argmax(1), \"06_final_blend\")\n", "print(\"final blend - public leaderboard:\", score(path))" ] }, { "cell_type": "markdown", "id": "03fc4770", "metadata": {}, "source": [ "## 🎉 The competition ends: private leaderboard revealed\n", "Until now we only saw the **public** score (~60 houses). The final ranking uses the **private** part (~75 houses), which we've never seen.\n", "Here is every submission we made, next to its CV score (repeated 5-fold):" ] }, { "cell_type": "code", "execution_count": 6, "id": "1fd6bf36", "metadata": { "execution": { "iopub.execute_input": "2026-10-23T06:16:42.349024Z", "iopub.status.busy": "2026-10-23T06:16:42.349024Z", "iopub.status.idle": "2026-10-23T06:16:42.412693Z", "shell.execute_reply": "2026-10-23T06:16:42.412187Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
submissioncvpublicprivate
002_majorityNaN0.40320.2933
102_random_forest0.69610.62900.6533
204_leaky_target_encoding0.79980.53230.4533
304_features_te0.70010.72580.6000
405_best_single0.73450.69350.7067
506_logreg0.71900.70970.7333
606_rf0.72140.69350.7333
706_lgb0.72960.70970.7333
806_xgb0.72960.69350.7333
906_cat0.73450.69350.7067
1006_final_blend0.73770.69350.7467
\n", "
" ], "text/plain": [ " submission cv public private\n", "0 02_majority NaN 0.4032 0.2933\n", "1 02_random_forest 0.6961 0.6290 0.6533\n", "2 04_leaky_target_encoding 0.7998 0.5323 0.4533\n", "3 04_features_te 0.7001 0.7258 0.6000\n", "4 05_best_single 0.7345 0.6935 0.7067\n", "5 06_logreg 0.7190 0.7097 0.7333\n", "6 06_rf 0.7214 0.6935 0.7333\n", "7 06_lgb 0.7296 0.7097 0.7333\n", "8 06_xgb 0.7296 0.6935 0.7333\n", "9 06_cat 0.7345 0.6935 0.7067\n", "10 06_final_blend 0.7377 0.6935 0.7467" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cv_known = {\"02_majority\": None, \"02_random_forest\": 0.6961,\n", " \"04_leaky_target_encoding\": 0.7998, \"04_features_te\": 0.7001, \"05_best_single\": 0.7345}\n", "for name in models:\n", " cv_known[f\"06_{name}\"] = round(cv[name], 4)\n", "cv_known[\"06_final_blend\"] = round(blend_acc(list(models)), 4)\n", "\n", "rows = []\n", "for name, c in cv_known.items():\n", " s = score(f\"submissions/{name}.csv\", show_private=True)\n", " rows.append({\"submission\": name, \"cv\": c, \"public\": s[\"public\"], \"private\": s[\"private\"]})\n", "board = pd.DataFrame(rows)\n", "board" ] }, { "cell_type": "markdown", "id": "e1ee2d41", "metadata": {}, "source": [ "## How to read this table\n", "- **The leaky submission** has by far the highest CV, and the worst score of every real model on both leaderboards. Leakage makes CV lie.\n", "- **Public vs private** for the same submission can differ by 5+ points. With only ~60 and ~75 houses, a handful of houses decides the score. On a small test set **the leaderboard is mostly luck**, and your CV is the only steady signal.\n", "- **CV is noisy too** (±0.8%), so models within ~1–2% of each other are effectively tied. Don't over-read small gaps anywhere.\n", "- The **final blend** is what we'd pick in a real competition: built on honest CV, combining different kinds of model, with no tuning to the leaderboard.\n", "\n", "## Full journey\n", "majority guess → Random Forest → honest (repeated) CV → feature ideas tested (and rejected) → leak caught → regularised models → blend.\n", "\n", "That's the whole workflow. For the real competition, it is the same steps with a different CSV. See **GUIDE.md**." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.10" } }, "nbformat": 4, "nbformat_minor": 5 }