Housing Price-Tier Prediction: an ML project from start to finish

This is a practice run for the Kaggle Workstation Rental Price Tier Prediction competition. We use your Housing.csv (545 houses), turn the price into 3 tiers (budget / mid / premium), and solve it the same way you'd solve the real competition.

Every number in this guide comes from actually running the notebooks.

0. What's in the folder

housing-ml-tutorial/
├── raw/Housing.csv            ← your original dataset
├── prepare_data.py            ← plays "Kaggle": makes train/test files (run once, then ignore)
├── data/
│   ├── train.csv              ← 408 houses WITH the answer (price_tier)
│   ├── test.csv               ← 137 houses WITHOUT the answer
│   └── sample_submission.csv  ← the format to submit in
├── _hidden/test_labels.csv    ← the answers for test. Don't look! Only score.py reads it
├── score.py                   ← our fake Kaggle leaderboard (public + private)
├── utils.py                   ← code written once in a notebook, reused later
├── 01_eda.ipynb … 06_ensemble_and_final.ipynb   ← the project, in order
├── submissions/               ← every CSV we "submitted"
└── extra_california/          ← bonus: the same 6 steps on a 16,500-row dataset

How our fake Kaggle works. prepare_data.py hides 137 houses as the test set. score.py scores your submission against their real tiers. Like on Kaggle, the test set is split into a public part (62 houses, the score you see during the competition) and a private part (75 houses, revealed only at the end and used for the final ranking).

Running it

Install once (from this folder):

uv venv .venv --python 3.12

uv pip install --python .venv/Scripts/python.exe -r requirements.txt

Then open the notebooks in VS Code or Jupyter, choose the .venv Python as the kernel, and run them in order, 01 → 06. (On Kaggle itself everything is pre-installed; just upload the notebook.)

The workflow (memorise this)

understand the data → baseline → honest validation → improve ONE thing → check CV → repeat → blend → submit
        01               02              03              04 / 05            (every step)      06

Every notebook is one step of that loop. The rest of this guide walks through them.

01 · EDA: look before you model

Question: what is in this data, and what seems to drive the price tier?

What we found:

Concept: hypotheses. EDA ends with a list of guesses ("bigger + AC + good area = pricier"). Later steps test these guesses.

02 · Baseline: get on the leaderboard fast

Two submissions:

1. Dumb baseline: predict "budget" for every house. Public: 0.403. This is the floor every real model must beat.
2. First real model: a Random Forest with minimal cleaning:
   • missing area → train median, missing furnishing → most common value (always learned from train, then applied to test),
   • yes/no → 1/0,
   • furnishing → 0/1/2 in its natural order (ordinal encoding),
   • target → 0/1/2.

Concept: overfitting. A 100% training score means the model memorised the rows. That's like grading a student on the exact questions they studied. The training score is useless as a measure, so we need an honest one → 03.

03 · Validation: an honest score without submitting

1. Hold-out: hide 20% of train and score on it → 0.671, close to the leaderboard. Now we have a leaderboard we control.
2. Overfitting picture: a decision tree's train accuracy climbs to 1.0 as it gets deeper, while validation flattens and drops. The gap between the two curves is overfitting.
3. One split is noisy: the same model on 10 random 80/20 splits scored anywhere from 0.659 to 0.780. With ~80 houses in validation, each house is worth >1%.
4. K-fold CV: 5 folds, each takes a turn as the exam → 0.703 ± 0.034.
5. Repeated K-fold: 5-fold done 3 times with different shuffles (15 scores averaged) → 0.696 ± 0.030. This is cv_score() in utils.py, used for everything after this.

Rule: trust your CV, not the public leaderboard. The public leaderboard here is 62 houses (one house ≈ 1.6%). CV uses all 408 houses, three times over.

Noise level to remember: std 0.03 / √15 ≈ ±0.8%. Differences smaller than that are probably luck.

04 · Feature engineering: make new columns, keep only what CV likes

Method: change one thing → re-run CV with the same model → keep it only if CV clearly improves. Log everything.

The leakage lesson (the most important thing in this project)

Idea 4: for each house, take the average tier of houses with the same description (same bedrooms, bathrooms, stories, parking, furnishing, AC, area). Done the wrong way (averages computed on all of train, including the house itself), CV jumped to 0.80, our "best idea".

Most descriptions appear only once, so the feature was the answer. The model learned to copy it, and on test houses the shortcut doesn't exist. Data leakage = information about the answer sneaking into the features. Fix: compute anything that uses the target inside each CV fold, from the training part only (add_target_encoding() + cv_score(..., fold_fn=...)).

Decision: nothing beat the noise, so we keep the 12 raw columns. Simpler wins when the evidence is unclear. (In extra_california/, which has 40× more data, distance-to-city features gave a real +3%. Feature engineering matters more with more data, and when domain knowledge adds something the raw columns don't show.)

05 · Models and tuning

Defaults shoot-out (same features, same CV)

The simplest model wins with default settings. With 400 rows, flexible models memorise noise. LightGBM and XGBoost defaults are built for thousands to millions of rows. Fix: regularise, i.e. force the models to stay simpler.

Early stopping

LightGBM watching a validation set found the best point at 167 trees (learning rate 0.03). After that, validation error rises again. It's the same overfitting curve as in 03.

Tuning (small manual grids, scored by CV)

Every tree model got better when it was restricted: RF went from 0.693 to 0.721 just by requiring 5 houses per leaf. After tuning, all five are within ~1.5% of each other, which is close to the noise level.

Other things tried

• log_area for Logistic Regression (a non-tree model, where a log can matter): 0.719 → 0.721, within noise, so it stays dropped.
• Ordinal trick (predict 0–2 as a number, then round): 0.724 vs 0.730 for the classifier, within noise. It's always worth one try when the target is ordered.

06 · Ensemble and the final reveal

Why blend: models make different mistakes, and averaging their probabilities cancels some of them. How often the models disagree: Logistic Regression vs the tree models 16–19% of houses. LightGBM vs XGBoost only 2%, because they're near-identical cousins. Diversity comes from different kinds of model.

Final submission = equal-weight average of all 5 tuned models (OOF accuracy 0.738).

🎉 Private leaderboard revealed

What this table teaches:

1. Leakage: highest CV, worst private score of any real model.
2. Shake-up: "04 honest TE" had the best public score (0.726) and dropped to 0.600 on private. Anyone choosing by the public leaderboard would have picked it.
3. Public is mostly luck on small test sets. Every tuned model scored 0.69–0.71 on public; private reorders them.
4. The blend, chosen only by honest CV, ended up best on private. Following the process paid off.

Overall progress: 0.29 (dumb) → 0.65 (first model) → 0.75 (final), with no peeking at the private set.

Glossary

Mapping this to the real competition

The workstation rental competition should be the same shape: tabular rows, a price tier target, and some categorical columns.

Differences to watch for: more rows (→ boosting usually wins, like in extra_california/), more or high-cardinality categoricals (→ CatBoost or in-fold target encoding), and a different metric (→ change accuracy_score in cv_score).

How to teach it

1. One notebook per session. Before running a cell, ask her: "what do you expect this to print?" Then run it.
2. Let her break things. Good exercises:
   • 02: submit the training-set accuracy as if it were real, then compare.
   • 03: change n_repeats to 1 and watch the CV numbers jump around.
   • 04: invent one feature of her own and run it through experiment().
   • 05: set min_samples_leaf=1 vs 20 and explain the CV change.
   • 06: blend only LightGBM + XGBoost and see that it barely helps (they agree 98% of the time).
3. Make her keep a log (idea → CV → kept?) from day one. It's the habit that matters most in a real competition.
4. Then compare with extra_california/. Same steps on 16,500 rows: there feature engineering does help, tree models beat Logistic Regression by 15 points, and fold scores vary by ±0.8% instead of ±3%. Same process, different conclusions. That's why we measure instead of assuming.