04 · Feature engineering

Feature engineering = creating new columns that make the pattern easier for the model to see. Ideas come from the EDA hypotheses and from common sense about houses.

We work like scientists: change one thing → re-run CV → keep it only if CV improves. Every experiment goes into a log. The model stays fixed (the same Random Forest), so only the features change.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from utils import (load_data, basic_prep, add_features, add_target_encoding, cv_score,
                   save_submission, TARGET, TIER_TO_NUM, LAYOUT_KEYS, SEED)
from score import score

train, test = load_data()
tr, te = basic_prep(train, test)
full, te_full = add_features(tr), add_features(te)
y = tr[TARGET].map(TIER_TO_NUM)
BASE = [c for c in tr.columns if c not in ("id", TARGET)]

def rf():
    return RandomForestClassifier(n_estimators=300, random_state=SEED, n_jobs=-1)

log = []  # our experiment log
def experiment(name, X, **kw):
    mean, std, _ = cv_score(rf, X, y, verbose=False, **kw)
    log.append({"experiment": name, "n_features": X.shape[1], "cv_acc": round(mean, 4), "std": round(std, 4)})
    print(f"{name:38s} CV = {mean:.4f} ± {std:.4f}")

experiment("0. baseline (raw columns)", tr[BASE])

0. baseline (raw columns)              CV = 0.6961 ± 0.0303

The new columns (all in add_features() in utils.py):

• amenities: how many of the 6 yes/no boxes are "yes" (one "how fancy is it" score),
• total_rooms = bedrooms + bathrooms,
• area_per_room: how spacious the rooms are,
• bath_per_bed: 1 bathroom for 4 bedrooms feels cheaper than 2 for 2,
• log_area: area on a log scale.

full[["amenities", "total_rooms", "area_per_room", "bath_per_bed", "log_area"]].describe().T.round(2)

Idea 1: amenities count

experiment("1. + amenities", full[BASE + ["amenities"]])

1. + amenities                         CV = 0.7001 ± 0.0309

Idea 2: room ratios

experiment("2. + total_rooms, area_per_room, bath_per_bed", full[BASE + ["total_rooms", "area_per_room", "bath_per_bed"]])

2. + total_rooms, area_per_room, bath_per_bed CV = 0.6887 ± 0.0385

Idea 3: log of area

Trees only care about the order of values ("is area bigger than X?"), and taking the log doesn't change the order. So this should do nothing for a Random Forest. Let's confirm. (It can help linear models like Logistic Regression; we re-test it in 05.)

experiment("3. + log_area", full[BASE + ["log_area"]])

3. + log_area                          CV = 0.6961 ± 0.0331

Exactly the same score as the baseline: confirmed.

Now read ideas 1–2 carefully. The changes are small (under 1%), and on ~400 rows CV moves by about ±0.8% from noise alone (the std divided by √15, the number of folds). Neither difference is clearly bigger than the noise. That's a real and common outcome: the raw columns already describe the houses well.

Idea 4: "same description" price (target encoding), and the leakage trap ⚠️

Idea: houses with the same description (same bedrooms, bathrooms, stories, parking, furnishing, AC and preferred area) probably sell in a similar range. So for each house, compute the average tier of houses with the same description.

This uses the target to build a feature, which is dangerous. Let's do it the wrong way first.

Wrong way: compute the averages on all of train, then run CV

keys = list(LAYOUT_KEYS)
layout = tr[keys].astype(str).agg("_".join, axis=1)
print("different descriptions:", layout.nunique(), "| houses per description (median):", layout.value_counts().median())
leaky = tr[BASE].assign(layout_price=y.groupby(layout).transform("mean"))
experiment("4a. + layout_price (LEAKY)", leaky)

different descriptions: 210 | houses per description (median): 1.0

4a. + layout_price (LEAKY)             CV = 0.7998 ± 0.0307

A huge jump, about 10 points! Too good to be true? Check the leaderboard. For test houses we can only use train's averages, which is exactly the situation the model will face for real:

layout_means = y.groupby(layout).mean()
te_layout = te[keys].astype(str).agg("_".join, axis=1)
te_leaky = te[BASE].assign(layout_price=te_layout.map(layout_means).fillna(y.mean()))
m = rf().fit(leaky, y)
print("leaderboard:", score(save_submission(te, m.predict(te_leaky), "04_leaky_target_encoding")))

leaderboard: {'public': 0.5323}

CV says ~0.80, but the leaderboard collapses to worse than our first Random Forest. Why? Most descriptions occur only once in train (median = 1 house). Each house's layout_price was computed including its own answer, so for most houses layout_price is the answer. The model learned "just copy layout_price". On test houses that shortcut doesn't exist (most of their descriptions were never seen), so the model is lost.

This is data leakage: information about the answer sneaking into the features. It is the #1 way people fool themselves, and it's why "too good to be true" CV jumps deserve suspicion.

Right way: learn the averages inside each fold, from the training part only

add_target_encoding() in utils.py does it properly:

• validation/test rows get averages computed only from the training fold,
• training rows get a leave-one-out average (their own answer removed),
• small groups are pulled towards the overall average (smoothing), so a description seen twice doesn't get an extreme value.

cv_score(..., fold_fn=add_target_encoding) runs it separately inside every fold. (The log shows one feature fewer because layout_price is added inside each fold.)

experiment("4b. + layout_price (correct, in-fold)", tr[BASE], fold_fn=add_target_encoding)

4b. + layout_price (correct, in-fold)  CV = 0.7001 ± 0.0330

X_fit, X_te = add_target_encoding(tr[BASE], y, te[BASE])
m = rf().fit(X_fit, y)
print("leaderboard:", score(save_submission(te, m.predict(X_te), "04_features_te")))

leaderboard: {'public': 0.7258}

Done honestly, the "magic" disappears: CV goes back to roughly the baseline. The feature carried no real extra information, only the leaked answer.

Experiment log

log_df = pd.DataFrame(log)
log_df["vs_baseline"] = (log_df.cv_acc - log_df.cv_acc[0]).round(4)
log_df

What does the model rely on?

A Random Forest can report how much each feature helped its splits (feature importance):

m = rf().fit(tr[BASE], y)
pd.Series(m.feature_importances_, index=BASE).sort_values().plot.barh(figsize=(7, 5), title="Feature importance");