Dropping Missing Data (dropna)

Once you know where the holes are, the simplest fix is to remove the rows (or columns) that contain them. Munna's IPL notebook shows why you must do it carefully: df.dropna() on the full matches data returned only 19 of 816 rows, because the method column is empty in 97 % of matches — so almost every row had some missing value. Dropping is powerful, but one careless call can throw away nearly all your data.

Definition

dropna() returns a new DataFrame with the rows (or columns) that contain missing values removed, according to a rule you choose: drop if any value is missing, only if all are missing, if fewer than thresh values are present, or only when specific subset columns are missing.

• New — the original DataFrame is untouched unless you assign the result back (df = df.dropna()).
• Rows or columns — controlled by axis: 0 (default) drops rows, 1 drops columns.
• Rule — how, thresh and subset decide which rows count as "bad".

Syntax

df.dropna(axis=0, how="any", thresh=None, subset=None,
          inplace=False, ignore_index=False)
series.dropna()

Example — the default: drop any row with a hole

scores = pd.DataFrame({
    "player": ["Virat", "Rohit", "Dhoni", "Gill", "Bumrah"],
    "team":   ["RCB", "MI", None, "GT", None],
    "runs":   [741, np.nan, 161, 890, np.nan],
    "sr":     [154.7, np.nan, np.nan, 157.8, np.nan],
})
print(scores)
print()
print(scores.dropna())

   player team   runs     sr
0   Virat  RCB  741.0  154.7
1   Rohit   MI    NaN    NaN
2   Dhoni  NaN  161.0    NaN
3    Gill   GT  890.0  157.8
4  Bumrah  NaN    NaN    NaN

  player team   runs     sr
0  Virat  RCB  741.0  154.7
3   Gill   GT  890.0  157.8

How it works: with how="any", a single NaN anywhere in the row is enough to remove it. Only Virat and Gill have complete rows. Notice the index keeps the old labels 0 and 3 — add ignore_index=True (or .reset_index(drop=True)) to renumber.

Example — how="all" and thresh

print(scores.dropna(how="all", subset=["team", "runs", "sr"]))
print()
print(scores.dropna(thresh=3))

  player team   runs     sr
0  Virat  RCB  741.0  154.7
1  Rohit   MI    NaN    NaN
2  Dhoni  NaN  161.0    NaN
3   Gill   GT  890.0  157.8

  player team   runs     sr
0  Virat  RCB  741.0  154.7
3   Gill   GT  890.0  157.8

How it works:

• how="all" with subset drops a row only if team, runs and sr are all missing. Bumrah's team is None and his runs and sr are NaN, so all three are missing → dropped. Rohit still has team="MI", so he stays even though his numbers are missing.
• thresh=3 keeps rows with at least 3 non-missing values out of 4. Rohit has 2 (player, team) → dropped; Dhoni has 2 (player, runs) → dropped; Bumrah has 1 → dropped.

Example — subset: only care about key columns

Usually only some columns matter. A match without a winner is useless for "who won most" analysis, but a missing city is not a problem for that question.

print(scores.dropna(subset=["runs"]))
print()
print(scores.dropna(subset=["runs"], ignore_index=True))

  player team   runs     sr
0  Virat  RCB  741.0  154.7
2  Dhoni  NaN  161.0    NaN
3   Gill   GT  890.0  157.8

  player team   runs     sr
0  Virat  RCB  741.0  154.7
1  Dhoni  NaN  161.0    NaN
2   Gill   GT  890.0  157.8

How it works: only the runs column is checked; Dhoni's missing team and sr are ignored, so he stays.

Example — dropping columns (axis=1)

When a column is mostly empty (like method in the IPL data), drop the column, not the rows:

print(scores.dropna(axis=1))
print()
print(scores.dropna(axis=1, thresh=4))

   player
0   Virat
1   Rohit
2   Dhoni
3    Gill
4  Bumrah

   player
0   Virat
1   Rohit
2   Dhoni
3    Gill
4  Bumrah

How it works: axis=1 applies the same rules to columns. With the default how="any", every column except player has a hole. With thresh=4, a column must have at least 4 non-missing values out of 5 — only player (5) qualifies; team and runs have 3, sr has 2.

Dropping a column by its missing percentage

A common real-world rule: "drop columns that are more than 50 % empty".

pct = scores.isna().mean() * 100
print(pct)
keep = pct[pct <= 50].index
print(scores[keep])

player     0.0
team      40.0
runs      40.0
sr        60.0
dtype: float64
   player team   runs
0   Virat  RCB  741.0
1   Rohit   MI    NaN
2   Dhoni  NaN  161.0
3    Gill   GT  890.0
4  Bumrah  NaN    NaN

Common mistakes

• ❌ df.dropna() on data with one almost-empty column → you lose almost every row (the IPL method trap: 816 → 19). ✅ Check isna().mean() first; drop or fill that column, then drop rows.
• ❌ Calling df.dropna() and expecting df to change. ✅ df = df.dropna() — it returns a new DataFrame.
• ❌ df.dropna(how="all", thresh=2) → TypeError: You cannot set both the how and thresh arguments at the same time. ✅ Use one or the other.
• ❌ Forgetting the index has gaps after dropping, then using df.loc[2]. ✅ Use ignore_index=True or reset_index(drop=True).

Comparison — which rule to use?

When to drop vs when to fill

Drop when the missing rows are few (a few %), random, and the field is essential (a match with no winner cannot be counted). Fill (next topic) when dropping would lose a lot of data or bias the result.

Interview tip

Q: What does thresh do in dropna? A: It keeps only rows (or columns with axis=1) that have at least thresh non-missing values. It counts present values, not missing ones — a common trick question.