Lesson 5 of 6

Models & tuning

Five model families, early stopping, regularisation, the ordinal trick.

Left: what & whyRight: the code and its real output

05 · Models and tuning

Features are fixed now (from 04: the 12 raw columns). Here we change the model:

try several model families with default settings, same CV,
understand why small data changes which models win,
learn early stopping (how many trees to build),
tune each model a little, then compare fairly,
try the ordinal trick: the tiers have an order, so treat the problem as regression.

import warnings
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, RepeatedStratifiedKFold
from sklearn.metrics import accuracy_score
from lightgbm import LGBMClassifier, LGBMRegressor, early_stopping, log_evaluation
from xgboost import XGBClassifier
from catboost import CatBoostClassifier
from utils import make_xy, cv_score, save_submission, load_data, basic_prep, add_features, SEED
from score import score

warnings.filterwarnings("ignore")
X, y, X_test, test = make_xy()
print(X.shape, X_test.shape)

(408, 12) (137, 12)

1. Model shoot-out (default settings)

model	idea in one line
Logistic Regression	one weighted sum of the columns per tier; simple, needs scaled inputs
Random Forest	many deep trees on random samples, then a vote
LightGBM / XGBoost / CatBoost	gradient boosting: trees built one after another, each fixing the previous ones' mistakes

StandardScaler rescales every column to mean 0 / std 1. Logistic Regression needs it, because otherwise area (thousands) would drown out bathrooms (1–4). Trees don't care.

defaults = {
    "LogisticRegression": lambda: make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000)),
    "RandomForest":       lambda: RandomForestClassifier(n_estimators=300, random_state=SEED, n_jobs=-1),
    "LightGBM":           lambda: LGBMClassifier(random_state=SEED, verbose=-1),
    "XGBoost":            lambda: XGBClassifier(random_state=SEED),
    "CatBoost":           lambda: CatBoostClassifier(random_seed=SEED, verbose=0),
}
results = []
for name, make in defaults.items():
    mean, std, _ = cv_score(make, X, y, verbose=False)
    results.append({"model": name, "cv_acc": round(mean, 4), "std": round(std, 4)})
    print(f"{name:20s} {mean:.4f} ± {std:.4f}")
pd.DataFrame(results).sort_values("cv_acc", ascending=False)

LogisticRegression   0.7190 ± 0.0481

RandomForest         0.6961 ± 0.0303

LightGBM             0.6953 ± 0.0371

XGBoost              0.6879 ± 0.0401

CatBoost             0.7132 ± 0.0340

	model	cv_acc	std
0	LogisticRegression	0.7190	0.0481
4	CatBoost	0.7132	0.0340
1	RandomForest	0.6961	0.0303
2	LightGBM	0.6953	0.0371
3	XGBoost	0.6879	0.0401

The simplest model wins. Logistic Regression is at the top, and default LightGBM/XGBoost are at the bottom.

Why? Small data.

With 400 rows, flexible models (deep trees, hundreds of boosting rounds) have plenty of room to memorise noise. A simple model can't memorise much, and here the true pattern is fairly simple anyway: bigger + more bathrooms + AC + good area = pricier. The defaults of LightGBM/XGBoost were designed for datasets with thousands to millions of rows.

Fix: restrict (regularise) the flexible models: smaller trees, fewer rounds, bigger leaves.

2. Early stopping: how many trees?

A boosting model adds trees one by one. Early stopping watches a validation set and stops once the validation error hasn't improved for N rounds.

X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.2, random_state=SEED, stratify=y)
m = LGBMClassifier(n_estimators=2000, learning_rate=0.03, random_state=SEED, verbose=-1)
m.fit(X_tr, y_tr, eval_set=[(X_tr, y_tr), (X_va, y_va)],
      callbacks=[early_stopping(100, verbose=False), log_evaluation(0)])
print("best number of trees:", m.best_iteration_)

curves = m.evals_result_
plt.figure(figsize=(8, 4))
plt.plot(curves["training"]["multi_logloss"], label="train")
plt.plot(curves["valid_1"]["multi_logloss"], label="validation")
plt.axvline(m.best_iteration_, ls="--", c="gray")
plt.xlabel("number of trees"); plt.ylabel("log-loss (lower = better)"); plt.legend();

best number of trees: 167

Same picture as the depth plot in 03: training error keeps falling, while validation error bottoms out early and then rises. Past the dashed line, extra trees only make validation worse. The default LightGBM never checks this: it always builds 100 large trees (31 leaves each) at a 3× faster learning rate, far too much model for 400 houses.

3. Tuning each model (small, manual grids)

For each model we try a few settings that go from "flexible" to "restricted", scoring each with CV:

Logistic Regression C: smaller = more regularised (simpler).
Random Forest min_samples_leaf: every leaf must contain at least this many houses.
LightGBM num_leaves / min_child_samples, XGBoost max_depth: tree size. Plus fewer, slower rounds and random row/column sampling.

(Tools like Optuna search these automatically. For learning, a small grid is clearer.)

grids = {
    "LogisticRegression": [
        lambda C=C: make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=3000)) for C in [0.1, 0.3, 1.0, 3.0]
    ],
    "RandomForest": [
        lambda leaf=leaf: RandomForestClassifier(n_estimators=500, min_samples_leaf=leaf, random_state=SEED, n_jobs=-1) for leaf in [1, 3, 5, 10]
    ],
    "LightGBM": [
        lambda nl=nl: LGBMClassifier(n_estimators=300, learning_rate=0.03, num_leaves=nl, min_child_samples=10, subsample=0.8,
                                     subsample_freq=1, colsample_bytree=0.7, random_state=SEED, verbose=-1) for nl in [4, 7, 15, 31]
    ],
    "XGBoost": [
        lambda d=d: XGBClassifier(n_estimators=300, learning_rate=0.03, max_depth=d, subsample=0.8, colsample_bytree=0.7,
                                  random_state=SEED) for d in [2, 3, 4, 6]
    ],
    "CatBoost": [
        lambda d=d: CatBoostClassifier(iterations=500, learning_rate=0.03, depth=d, random_seed=SEED, verbose=0) for d in [2, 4, 6]
    ],
}
labels = {"LogisticRegression": "C", "RandomForest": "min_samples_leaf", "LightGBM": "num_leaves", "XGBoost": "max_depth", "CatBoost": "depth"}
values = {"LogisticRegression": [0.1, 0.3, 1.0, 3.0], "RandomForest": [1, 3, 5, 10], "LightGBM": [4, 7, 15, 31], "XGBoost": [2, 3, 4, 6], "CatBoost": [2, 4, 6]}
tuned = []
for name, makers in grids.items():
    for v, make in zip(values[name], makers):
        mean, std, _ = cv_score(make, X, y, verbose=False)
        tuned.append({"model": name, "setting": f"{labels[name]}={v}", "cv_acc": round(mean, 4)})
tuned_df = pd.DataFrame(tuned)
tuned_df

	model	setting	cv_acc
0	LogisticRegression	C=0.1	0.7092
1	LogisticRegression	C=0.3	0.7149
2	LogisticRegression	C=1.0	0.7190
3	LogisticRegression	C=3.0	0.7174
4	RandomForest	min_samples_leaf=1	0.6928
5	RandomForest	min_samples_leaf=3	0.7174
6	RandomForest	min_samples_leaf=5	0.7214
7	RandomForest	min_samples_leaf=10	0.7100
8	LightGBM	num_leaves=4	0.7296
9	LightGBM	num_leaves=7	0.7173
10	LightGBM	num_leaves=15	0.7017
11	LightGBM	num_leaves=31	0.6985
12	XGBoost	max_depth=2	0.7296
13	XGBoost	max_depth=3	0.7279
14	XGBoost	max_depth=4	0.7213
15	XGBoost	max_depth=6	0.7181
16	CatBoost	depth=2	0.7280
17	CatBoost	depth=4	0.7345
18	CatBoost	depth=6	0.7263

best = tuned_df.loc[tuned_df.groupby("model").cv_acc.idxmax()].sort_values("cv_acc", ascending=False)
best

	model	setting	cv_acc
17	CatBoost	depth=4	0.7345
8	LightGBM	num_leaves=4	0.7296
12	XGBoost	max_depth=2	0.7296
6	RandomForest	min_samples_leaf=5	0.7214
2	LogisticRegression	C=1.0	0.7190

Every tree model improves as it gets more restricted (smaller trees / bigger leaves), up to a point.
After tuning, the models are all close together (within ~1–2%). Remember the noise level is ~±0.8%.
The tuned settings above are copied into notebook 06.

4. Re-testing a dropped idea

In 04, log_area did nothing for the Random Forest, as expected for trees. Logistic Regression is not a tree, so a log can matter for it:

_tr, _ = basic_prep(*load_data())
log_area = np.log(_tr.area).rename("log_area")
lr = lambda: make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000))
print(f"LogReg raw:        {cv_score(lr, X, y, verbose=False)[0]:.4f}")
print(f"LogReg + log_area: {cv_score(lr, X.join(log_area), y, verbose=False)[0]:.4f}")

LogReg raw:        0.7190
LogReg + log_area: 0.7206

The difference is within noise, so it stays dropped. Still, re-testing old ideas when you switch model type is a good habit.

5. The ordinal trick

Classification treats "budget" and "premium" as unrelated labels, so calling a premium house "budget" costs the same as calling it "mid". But the tiers have an order. Alternative: predict a number (0–2) with a regressor, then round to the nearest tier.

rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=SEED)
accs = []
for tr_idx, va_idx in rskf.split(X, y):
    reg = LGBMRegressor(n_estimators=300, learning_rate=0.03, num_leaves=7, min_child_samples=10, subsample=0.8,
                        subsample_freq=1, colsample_bytree=0.7, random_state=SEED, verbose=-1)
    reg.fit(X.iloc[tr_idx], y.iloc[tr_idx])
    pred = np.clip(np.round(reg.predict(X.iloc[va_idx])), 0, 2)
    accs.append(accuracy_score(y.iloc[va_idx], pred))
print(f"ordinal LightGBM (regression + rounding): {np.mean(accs):.4f}")
print(f"best LightGBM classifier:                 {best.set_index('model').cv_acc['LightGBM']:.4f}")

ordinal LightGBM (regression + rounding): 0.7239
best LightGBM classifier:                 0.7296

Compare the two numbers. If they're within noise, the trick doesn't pay off here, but it's always worth one try when the target is ordered.

Submit the best single model

Train on all training data (no fold held out), then predict test:

top_name = best.iloc[0].model
top_setting = best.iloc[0].setting
top_make = grids[top_name][[f"{labels[top_name]}={v}" for v in values[top_name]].index(top_setting)]
print("best single model by CV:", top_name, top_setting)
final = top_make().fit(X, y)
print("leaderboard:", score(save_submission(test, final.predict(X_test), "05_best_single")))

best single model by CV: CatBoost depth=4

leaderboard: {'public': 0.6935}

Next: 06. Instead of picking one model, combine them.