Lesson 1 of 6

Explore the data

Look before you model: size, missing values, the target, what drives price.

Left: what & whyRight: the code and its real output

01 · Exploring the data (EDA)

Goal: understand the data before building any model. By the end we should be able to answer:

How big is the data, and what does each column mean?
Is anything broken or missing?
What is the target and how is it distributed?
Which columns seem to drive the price tier? (our hypotheses)

No models in this notebook, just looking.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from utils import load_data, TIERS, TIER_TO_NUM, YES_NO

pd.set_option("display.width", 140)
train, test = load_data()
train.head()

	id	area	bedrooms	bathrooms	stories	mainroad	guestroom	basement	hotwaterheating	airconditioning	parking	prefarea	furnishingstatus	price_tier
0	0	7420.0	4	2	3	yes	no	no	no	yes	2	yes	furnished	premium
1	1	8960.0	4	4	4	yes	no	no	no	yes	3	no	furnished	premium
2	2	9960.0	3	2	2	yes	no	yes	no	no	2	yes	semi-furnished	premium
3	5	7500.0	3	3	1	yes	no	yes	no	yes	2	yes	semi-furnished	premium
4	9	5750.0	3	2	4	yes	yes	no	no	yes	1	yes	unfurnished	premium

Each row is one house for sale.

column	meaning
`area`	size of the house/plot in square feet
`bedrooms`, `bathrooms`, `stories`	counts
`mainroad`	is it on a main road? (yes/no)
`guestroom`, `basement`	has one? (yes/no)
`hotwaterheating`, `airconditioning`	has it? (yes/no)
`parking`	number of parking spots
`prefarea`	in a preferred (desirable) neighbourhood? (yes/no)
`furnishingstatus`	furnished / semi-furnished / unfurnished
`price_tier`	the target: budget / mid / premium

print("train:", train.shape, " test:", test.shape)
print("columns only in train:", set(train.columns) - set(test.columns))
train.info()

train: (408, 14)  test: (137, 13)
columns only in train: {'price_tier'}
<class 'pandas.DataFrame'>
RangeIndex: 408 entries, 0 to 407
Data columns (total 14 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   id                408 non-null    int64  
 1   area              398 non-null    float64
 2   bedrooms          408 non-null    int64  
 3   bathrooms         408 non-null    int64  
 4   stories           408 non-null    int64  
 5   mainroad          408 non-null    str    
 6   guestroom         408 non-null    str    
 7   basement          408 non-null    str    
 8   hotwaterheating   408 non-null    str    
 9   airconditioning   408 non-null    str    
 10  parking           408 non-null    int64  
 11  prefarea          408 non-null    str    
 12  furnishingstatus  393 non-null    str    
 13  price_tier        408 non-null    str    
dtypes: float64(1), int64(5), str(8)
memory usage: 44.8 KB

train has the answer column price_tier; test doesn't. That's what we must predict.
It's a small dataset: about 400 training rows. Keep that in mind; it will matter a lot in 03.
area and furnishingstatus have fewer non-null values than the rest → missing values.
Many columns are text (yes/no, furnishing). Models need numbers, so we'll convert them in 02.

train.isna().sum()

id                   0
area                10
bedrooms             0
bathrooms            0
stories              0
mainroad             0
guestroom            0
basement             0
hotwaterheating      0
airconditioning      0
parking              0
prefarea             0
furnishingstatus    15
price_tier           0
dtype: int64

train.describe().T.round(2)

	count	mean	std	min	25%	50%	75%	max
id	408.0	272.63	155.88	0.0	141.75	272.5	408.25	544.0
area	398.0	5109.51	2141.57	1650.0	3520.00	4505.0	6405.00	15600.0
bedrooms	408.0	2.95	0.71	1.0	2.00	3.0	3.00	5.0
bathrooms	408.0	1.28	0.49	1.0	1.00	1.0	2.00	4.0
stories	408.0	1.81	0.89	1.0	1.00	2.0	2.00	4.0
parking	408.0	0.69	0.87	0.0	0.00	0.0	1.00	3.0

area goes from ~1,650 to 16,200 sq ft, and most houses have 2–4 bedrooms, 1–2 bathrooms and 1–2 stories. A few big houses stretch the range, but nothing looks broken.

print(train.price_tier.value_counts())
train.price_tier.value_counts().reindex(TIERS).plot.bar(title="Target distribution");

price_tier
budget     139
premium    136
mid        133
Name: count, dtype: int64

The 3 tiers are balanced (~⅓ each). So:

guessing randomly gets ~33% accuracy,
accuracy is a fair metric here.

Numeric columns vs price tier

Boxplots: one box per tier. If the boxes move up as the tier rises, that column is useful.

fig, axes = plt.subplots(1, 5, figsize=(18, 4))
for ax, col in zip(axes, ["area", "bedrooms", "bathrooms", "stories", "parking"]):
    sns.boxplot(data=train, x="price_tier", y=col, order=TIERS, ax=ax)
    ax.set_title(col)
plt.tight_layout()

Yes/no columns vs price tier

For each yes/no column: what share of each tier has a "yes"?

yes_share = pd.DataFrame({col: train.groupby("price_tier")[col].apply(lambda s: (s == "yes").mean()) for col in YES_NO}).T[TIERS]
yes_share.round(2)

price_tier	budget	mid	premium
mainroad	0.75	0.86	0.97
guestroom	0.04	0.14	0.33
basement	0.19	0.41	0.43
hotwaterheating	0.04	0.03	0.07
airconditioning	0.10	0.29	0.57
prefarea	0.08	0.20	0.45

yes_share.plot.bar(figsize=(10, 4), title='Share of "yes" in each tier');

pd.crosstab(train.furnishingstatus, train.price_tier, normalize="index")[TIERS].round(2)

price_tier	budget	mid	premium
furnishingstatus
furnished	0.22	0.29	0.49
semi-furnished	0.21	0.46	0.33
unfurnished	0.62	0.15	0.23

Correlation with the tier (budget=0, mid=1, premium=2), after turning yes/no into 1/0:

num = train.drop(columns=["id", "furnishingstatus"]).copy()
for col in YES_NO:
    num[col] = (num[col] == "yes").astype(int)
num["tier_num"] = num.pop("price_tier").map(TIER_TO_NUM)
num.corr()["tier_num"].drop("tier_num").sort_values()

hotwaterheating    0.057344
basement           0.213995
mainroad           0.263354
parking            0.300412
guestroom          0.322620
prefarea           0.354464
stories            0.412867
airconditioning    0.415406
bedrooms           0.430120
bathrooms          0.492073
area               0.551234
Name: tier_num, dtype: float64

What we learned → our hypotheses

area is the strongest signal: bigger houses sit in higher tiers.
Bathrooms, stories, parking and air conditioning all climb with the tier.
prefarea (good neighbourhood) and mainroad matter. Premium houses are almost always on a main road.
Furnished houses lean premium, and unfurnished ones lean budget.
Rare features (hotwaterheating, only a few % say yes) can't help much on their own.
Data issues to handle: missing area / furnishingstatus, and text columns.

Next: 02_baseline: get a first submission on the leaderboard.