Detecting Missing Values

Real data is never complete. In the IPL matches dataset (2008–2020, 816 matches) that we analyse in this course, 13 matches have no city, 4 have no winner (washed-out games) and the method column is empty for 797 of 816 rows — it is only filled when a match was decided by D/L. Before you calculate a single average, you must know where the holes are and how big they are. Every cleaning job starts here.

Definition

A missing value is a cell that has no data recorded. pandas marks it with a special sentinel value — NaN for numbers and text, NaT for dates, or pd.NA for nullable dtypes — and detects all of them the same way with isna().

• No data recorded — the value was never collected, was lost, or does not apply (no D/L method in a normal match). It is not the same as 0 or an empty string "".
• Sentinel value — a special placeholder that means "nothing here". pandas has a few, listed below.
• Detects all of them the same way — you never need to check NaN, None and NaT separately; isna() treats every one of them as missing.

Note (pandas 3): text columns now have the dedicated str dtype (older tutorials show object). A missing value in a str column is displayed as NaN, and isna() detects it exactly as before.

Syntax

df.isna()                 # same-shape DataFrame of True/False (True = missing)
df.isnull()               # exact alias of isna()
df.notna()                # opposite: True = value present  (alias: notnull)
df.isna().sum()           # missing count per column
df.isna().mean() * 100    # missing percentage per column
df.isna().any(axis=1)     # True for rows with at least one missing value
df.info()                 # non-null count + dtype per column
pd.isna(value)            # works on a single scalar too

Example — spot the holes

players = pd.DataFrame({
    "player":  ["Virat", "Rohit", "Dhoni", "Gill", "Bumrah", None],
    "team":    ["RCB", "MI", None, "GT", "MI", "CSK"],
    "runs":    [741, np.nan, 161, 890, np.nan, 267],
    "wickets": [0, 0, np.nan, 0, 20, 8],
    "debut":   pd.to_datetime(["2008-04-18", None, "2008-04-19",
                               "2018-04-14", "2013-04-04", "2012-04-04"]),
})
print(players)
print(players.dtypes)

   player team   runs  wickets      debut
0   Virat  RCB  741.0      0.0 2008-04-18
1   Rohit   MI    NaN      0.0        NaT
2   Dhoni  NaN  161.0      NaN 2008-04-19
3    Gill   GT  890.0      0.0 2018-04-14
4  Bumrah   MI    NaN     20.0 2013-04-04
5     NaN  CSK  267.0      8.0 2012-04-04
player                str
team                  str
runs              float64
wickets           float64
debut      datetime64[us]
dtype: object

How it works: None in the player and team lists became NaN (they are str columns), np.nan stays NaN in the float columns, and None in the date list became NaT. Notice wickets is float64, not int — a plain NumPy integer column cannot hold NaN, so pandas upgrades it to float.

Example — the True/False mask and the count

print(players.isna())

   player   team   runs  wickets  debut
0   False  False  False    False  False
1   False  False   True    False   True
2   False   True  False     True  False
3   False  False  False    False  False
4   False  False   True    False  False
5    True  False  False    False  False

isna() returns a DataFrame of the same shape with True where data is missing. On a big dataset you never read this whole grid — you summarise it:

print(players.isna().sum())
print("Total missing cells:", players.isna().sum().sum())

player     1
team       1
runs       2
wickets    1
debut      1
dtype: int64
Total missing cells: 6

How it works: in Python True == 1 and False == 0, so .sum() on each column counts the True values = the number of missing cells. A second .sum() adds up all columns.

Example — percentage missing (the decision number)

The percentage tells you what to do next. Munna's IPL notebook uses (df.isna().sum() / len(df)) * 100; isna().mean() * 100 gives the same thing in one step.

pct = players.isna().mean() * 100
print(pct.round(1))

player     16.7
team       16.7
runs       33.3
wickets    16.7
debut      16.7
dtype: float64

Example — which rows have problems?

mask = players.isna().any(axis=1)
print(players[mask])
print(players["runs"].notna().sum(), "players have a runs value")

   player team   runs  wickets      debut
1   Rohit   MI    NaN      0.0        NaT
2   Dhoni  NaN  161.0      NaN 2008-04-19
4  Bumrah   MI    NaN     20.0 2013-04-04
5     NaN  CSK  267.0      8.0 2012-04-04
4 players have a runs value

How it works: any(axis=1) looks across each row and returns True if at least one cell is missing; using that as a boolean filter shows only the incomplete rows.

info() — the quick overview

players.info()

<class 'pandas.DataFrame'>
RangeIndex: 6 entries, 0 to 5
Data columns (total 5 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   player   5 non-null      str
 1   team     5 non-null      str
 2   runs     4 non-null      float64
 3   wickets  5 non-null      float64
 4   debut    5 non-null      datetime64[us]
dtypes: datetime64[us](1), float64(2), str(2)
memory usage: 372.0 bytes

Compare Non-Null Count with the total entries (6). Anything less than 6 has missing data. info() also shows the dtype, which is the second thing you check when cleaning.

Why == np.nan never works

print(np.nan == np.nan)
print((players["runs"] == np.nan).sum())
print(players["runs"].isna().sum())

False
0
2

By the IEEE floating-point rule, NaN is not equal to anything — not even itself. So == np.nan is always False and finds nothing. Always use isna().

Common mistakes

• ❌ df[df["runs"] == np.nan] → always empty. ✅ df[df["runs"].isna()].
• ❌ Thinking 0, "" or "NA" text is missing. ✅ They are real values to pandas; convert them first (e.g. replace("", np.nan) or read_csv(..., na_values=["NA", "-"])).
• ❌ Looking only at counts. ✅ Look at percentages — 13 missing out of 816 (1.6 %) is very different from 797 out of 816 (97.7 %).
• ❌ Expecting an int column to stay int after a NaN appears. ✅ It becomes float64; use the nullable "Int64" dtype if you need integers with missing values.

Comparison — isna vs isnull vs notna

isnull exists only for historical reasons (it mirrors R / SQL naming). Pick one — most modern code uses isna.

Interview tip

Q: How do you find the percentage of missing values in each column? A: df.isna().mean() * 100 (or df.isna().sum() / len(df) * 100). isna() builds a boolean mask, and the mean of booleans is the fraction of True values.