Fixing Data Types & Outliers

A column that looks numeric is often stored as text: "45", "N/A", " 7 ", "DNB" (did not bat). You cannot take the mean of text, and sorting text puts "100" before "45". In Munna's IPL notebook, info() showed the date column as text (object), so it had to be converted with pd.to_datetime. The second problem is outliers — values far away from the rest, such as a typo 9999 runs. This topic fixes both.

Definition

A dtype (data type) is the kind of data a column stores — int64, float64, bool, str, datetime64, category, … — and it decides which operations work on the column. astype() converts a column to another dtype and fails if any value cannot be converted; pd.to_numeric(errors="coerce") converts what it can and turns the rest into NaN.

An outlier is a value that lies unusually far from the rest of the data. The IQR rule flags values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR, where IQR = Q3 − Q1.

• Fails — astype(int) on "DNB" raises ValueError. That is safe, but stops your script.
• Coerce — "force": bad values become NaN, which you then handle with the missing-value tools.
• Q1 / Q3 — the 25th and 75th percentiles; the middle 50 % of the data lies between them.

Syntax

df["col"].astype(dtype, errors="raise")
pd.to_numeric(arg, errors="raise", downcast=None)
pd.to_datetime(arg, errors="raise", format=None)     # see Dates & Times
df["col"].quantile(q)                                # q = 0.25, 0.75 …
df["col"].clip(lower=None, upper=None)
df["col"].between(left, right)

Example — text that should be numbers

raw = pd.DataFrame({
    "player": ["Virat", "Rohit", "Dhoni", "Gill", "Bumrah"],
    "runs":   ["741", "417", "DNB", " 890 ", "N/A"],
    "sixes":  ["38", "28", "13", "33", "0"],
})
print(raw.dtypes)
print("Sorted as text:", sorted(raw["sixes"]))

player    str
runs      str
sixes     str
dtype: object
Sorted as text: ['0', '13', '28', '33', '38']

runs and sixes are str columns. Sorting text compares character by character, so "13" comes before "28" only by luck — "100" would sort before "38".

raw["runs"].astype(int)

ValueError: invalid literal for int() with base 10: 'DNB'

Example — astype for clean columns, to_numeric for dirty ones

raw["sixes"] = raw["sixes"].astype(int)
raw["runs"] = pd.to_numeric(raw["runs"], errors="coerce")
print(raw)
print(raw.dtypes)
print("Mean runs:", raw["runs"].mean())

   player   runs  sixes
0   Virat  741.0     38
1   Rohit  417.0     28
2   Dhoni    NaN     13
3    Gill  890.0     33
4  Bumrah    NaN      0
player        str
runs      float64
sixes       int64
dtype: object
Mean runs: 682.6666666666666

How it works: sixes is clean, so astype(int) works. runs has "DNB" and "N/A", so to_numeric(errors="coerce") turns them into NaN (note " 890 " with spaces is parsed fine). The column becomes float64 because it now holds NaN. Now .mean() works.

Example — nullable integers and categories

raw["runs_int"] = raw["runs"].astype("Int64")
raw["team"] = pd.Series(["RCB", "MI", "CSK", "GT", "MI"]).astype("category")
print(raw[["player", "runs_int", "team"]])
print(raw[["runs_int", "team"]].dtypes)

   player  runs_int team
0   Virat       741  RCB
1   Rohit       417   MI
2   Dhoni      <NA>  CSK
3    Gill       890   GT
4  Bumrah      <NA>   MI
runs_int       Int64
team        category
dtype: object

Example — finding outliers with the IQR rule

runs = pd.Series([34, 45, 52, 38, 61, 47, 9999, 55, 41, 3],
                 index=list("ABCDEFGHIJ"))
q1, q3 = runs.quantile(0.25), runs.quantile(0.75)
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print(f"Q1={q1}, Q3={q3}, IQR={iqr}")
print(f"fences: {low} to {high}")
print(runs[(runs < low) | (runs > high)])

Q1=38.75, Q3=54.25, IQR=15.5
fences: 15.5 to 77.5
G    9999
J       3
dtype: int64

How it works: the middle 50 % of the scores lies between Q1 = 38.75 and Q3 = 54.25, so IQR = 15.5 and 1.5 × IQR = 23.25. Anything more than 1.5 IQRs outside that box is suspicious. 9999 is clearly a typo, and 3 is also flagged as unusually low.

      low fence      Q1     median    Q3       high fence
  ──────────|──────────[=======|=======]──────────|────────
          15.5       38.75     46     54.25        77.5
    3 ●  (outlier)                                          9999 ● (outlier)

Example — handling outliers: remove, cap or flag

print("remove:", runs[runs.between(low, high)].tolist())
print("cap   :", runs.clip(lower=low, upper=high).tolist())
flagged = pd.DataFrame({"runs": runs, "outlier": ~runs.between(low, high)})
print(flagged.T)

remove: [34, 45, 52, 38, 61, 47, 55, 41]
cap   : [34.0, 45.0, 52.0, 38.0, 61.0, 47.0, 77.5, 55.0, 41.0, 15.5]
             A      B      C      D      E      F     G      H      I     J
runs        34     45     52     38     61     47  9999     55     41     3
outlier  False  False  False  False  False  False  True  False  False  True

Tip: an outlier is not always an error. A 175* by Chris Gayle is a real outlier. Use domain knowledge — remove only impossible values, and flag or cap the rest.

Common mistakes

• ❌ astype(int) on a column with "N/A" or NaN → ValueError / IntCastingNaNError. ✅ pd.to_numeric(..., errors="coerce"), then fill, then astype (or use "Int64").
• ❌ Calculating with numbers stored as text — "10" + "5" gives "105". ✅ Check df.dtypes (or info()) first.
• ❌ Deleting every IQR outlier automatically. ✅ Investigate; cap or flag when the value could be real.
• ❌ Using mean ± 3 std on tiny or very skewed data. ✅ IQR is more robust because quartiles are not pulled by extremes.

Interview tip

Q: Why does pd.to_numeric(s, errors="coerce") often produce a float column? A: Bad values become NaN, and NaN is a float. A NumPy integer column cannot hold it, so the result is float64. Use .astype("Int64") afterwards for a nullable integer column.