Filling Missing Data (fillna, ffill, interpolate)

Dropping rows throws data away. Often it is better to fill the holes with a sensible value. In the IPL notebook, Munna fills the almost-empty method column with a label (df.fillna({"method": "fare_match"})) so that the column means something — "normal match" vs "D/L" — instead of deleting 97 % of the rows. The art is choosing the right fill value for each column.

Definition

Imputation means replacing missing values with substitute values. In pandas, fillna() fills missing cells with a constant, a per-column dict or a Series; ffill() / bfill() copy the previous / next valid value; and interpolate() estimates values between known neighbours (straight-line by default).

• Substitute value — a value you choose to stand in for the unknown one. It is a guess, so choose it based on the column's meaning.
• Per-column dict — {"runs": 0, "team": "Unknown"} fills each column with its own value.
• Previous / next valid value — useful for ordered data (dates, ball-by-ball) where the last known value is still true.
• Between known neighbours — for numbers that change smoothly (a running total, a temperature).

Syntax

df.fillna(value, axis=None, inplace=False, limit=None)
df.ffill(axis=None, inplace=False, limit=None)      # forward fill
df.bfill(axis=None, inplace=False, limit=None)      # backward fill
df.interpolate(method="linear", axis=0, limit=None, limit_direction=None)

Note (pandas 3): fillna(method="ffill") and fillna(method="bfill") — still seen in old tutorials and in Munna's older notebook — were removed. They now raise TypeError: ... unexpected keyword argument 'method'. Use df.ffill() and df.bfill().

Example — constant and per-column dict

stats = pd.DataFrame({
    "player":  ["Virat", "Rohit", "Dhoni", "Gill", "Bumrah"],
    "team":    ["RCB", "MI", None, "GT", "MI"],
    "runs":    [741, np.nan, 161, 890, np.nan],
    "catches": [5, np.nan, 7, np.nan, 2],
})
print(stats.fillna(0))
print()
print(stats.fillna({"team": "Unknown", "runs": 0, "catches": 0}))

   player team   runs  catches
0   Virat  RCB  741.0      5.0
1   Rohit   MI    0.0      0.0
2   Dhoni    0  161.0      7.0
3    Gill   GT  890.0      0.0
4  Bumrah   MI    0.0      2.0

   player     team   runs  catches
0   Virat      RCB  741.0      5.0
1   Rohit       MI    0.0      0.0
2   Dhoni  Unknown  161.0      7.0
3    Gill       GT  890.0      0.0
4  Bumrah       MI    0.0      2.0

How it works: fillna(0) fills every column with 0 — even the text column team, which is rarely what you want. The dict version gives each column a meaningful value and leaves other columns alone.

Example — mean, median and mode

print("mean runs  :", stats["runs"].mean())
print("median runs:", stats["runs"].median())
print("mode team  :", stats["team"].mode()[0])

filled = stats.fillna({
    "runs": stats["runs"].median(),
    "team": stats["team"].mode()[0],
})
print(filled)

mean runs  : 597.3333333333334
median runs: 741.0
mode team  : MI
   player team   runs  catches
0   Virat  RCB  741.0      5.0
1   Rohit   MI  741.0      NaN
2   Dhoni   MI  161.0      7.0
3    Gill   GT  890.0      NaN
4  Bumrah   MI  741.0      2.0

How it works: .mean(), .median() and .mode() skip NaN automatically. .mode() returns a Series (there can be ties), so [0] takes the first most-frequent value.

Example — group-wise fill (smarter mean)

A bowler's missing runs should not be filled with a batter's average. Fill with the mean of the player's own team using groupby().transform():

squad = pd.DataFrame({
    "team": ["MI", "MI", "MI", "CSK", "CSK", "CSK"],
    "runs": [400, np.nan, 200, 500, 300, np.nan],
})
team_mean = squad.groupby("team")["runs"].transform("mean")
squad["runs_filled"] = squad["runs"].fillna(team_mean)
print(squad)

  team   runs  runs_filled
0   MI  400.0        400.0
1   MI    NaN        300.0
2   MI  200.0        200.0
3  CSK  500.0        500.0
4  CSK  300.0        300.0
5  CSK    NaN        400.0

How it works: transform("mean") returns a Series of the same length as squad, where each row holds its team's mean (MI 300, CSK 400). fillna with a Series fills each hole with the value at the same index.

Example — forward fill, backward fill, interpolate

A season points table recorded only on some days:

points = pd.Series([2, np.nan, np.nan, 6, np.nan, 10],
                   index=["M1", "M2", "M3", "M4", "M5", "M6"])
print(pd.DataFrame({
    "raw":    points,
    "ffill":  points.ffill(),
    "bfill":  points.bfill(),
    "interp": points.interpolate(),
    "ffill1": points.ffill(limit=1),
}))

     raw  ffill  bfill     interp  ffill1
M1   2.0    2.0    2.0   2.000000     2.0
M2   NaN    2.0    6.0   3.333333     2.0
M3   NaN    2.0    6.0   4.666667     NaN
M4   6.0    6.0    6.0   6.000000     6.0
M5   NaN    6.0   10.0   8.000000     6.0
M6  10.0   10.0   10.0  10.000000    10.0

How it works:

• ffill() copies the last known value forward (M2, M3 become 2).
• bfill() copies the next known value backward (M2, M3 become 6).
• interpolate() draws a straight line between known points: 2 → 6 over 3 steps gives 3.33, 4.67.
• limit=1 fills at most one consecutive gap, so M3 stays NaN.

  ffill:   2 → 2 → 2 → 6        bfill:   2 ← 6 ← 6 ← 6
  interp:  2 → 3.33 → 4.67 → 6 (straight line)

Example — flag before you fill

Filling hides the fact that data was missing. If that fact matters, keep a flag column first:

df = pd.DataFrame({"player": ["A", "B", "C"], "runs": [50, np.nan, 30]})
df["runs_missing"] = df["runs"].isna()
df["runs"] = df["runs"].fillna(df["runs"].median())
print(df)

  player  runs  runs_missing
0      A  50.0         False
1      B  40.0          True
2      C  30.0         False

Common mistakes

• ❌ df["runs"].fillna(0, inplace=True) — in pandas 3 (Copy-on-Write) this never changes df and raises a ChainedAssignmentError warning. ✅ df["runs"] = df["runs"].fillna(0) or df = df.fillna({"runs": 0}).
• ❌ fillna(method="ffill") → TypeError in pandas 3. ✅ df.ffill().
• ❌ df.fillna(df.mode()) — mode() returns a DataFrame. ✅ df["team"].mode()[0] for one column, or df.fillna(df.mode().iloc[0]) for all.
• ❌ Filling with the mean when the column has big outliers. ✅ Use the median.
• ❌ Forward-filling data that is not sorted. ✅ Sort by date/order first (sort_values), then ffill().

Comparison — fill strategies

Interview tip

Q: Mean or median for filling a salary column? A: Median. Salaries are skewed by a few very large values; the mean gets pulled up by them, the median does not.