Case & Whitespace Cleaning

To a human, "Mumbai", "mumbai", " MUMBAI " and "Mumbai " are the same city. To pandas they are four different values. value_counts(), groupby(), drop_duplicates() and merge() all treat them separately, so your "matches per city" table is silently wrong. Case and whitespace normalisation is the first thing to do with any text column — Munna uses ipl['venue'].str.lower() in the IPL notebook for exactly this check.

Definition

Case normalisation converts every string to one consistent letter case (e.g. Title Case), and whitespace trimming removes spaces, tabs and newlines from the start and end of each string, so that values which mean the same thing become exactly equal.

• Consistent letter case — pick one style per column: lower for codes/emails, title for names and cities, upper for short codes like MI.
• Start and end — strip() does not touch spaces inside the string ("Eden Gardens" keeps both spaces). Use a regex replace for those.
• Exactly equal — the goal is that equal meaning ⇒ equal string.

Syntax

s.str.lower()       s.str.upper()       s.str.title()
s.str.capitalize()  s.str.swapcase()    s.str.casefold()
s.str.strip(to_strip=None)              # both ends
s.str.lstrip(to_strip=None)             # left end only
s.str.rstrip(to_strip=None)             # right end only
s.str.replace(r"\s+", " ", regex=True)  # collapse inner spaces

Example — case changes

names = pd.Series(["virat KOHLI", "ROHIT sharma", "ms dhoni", "AB de Villiers"])
print(pd.DataFrame({
    "lower": names.str.lower(),
    "upper": names.str.upper(),
    "title": names.str.title(),
    "capitalize": names.str.capitalize(),
}))

            lower           upper           title      capitalize
0     virat kohli     VIRAT KOHLI     Virat Kohli     Virat kohli
1    rohit sharma    ROHIT SHARMA    Rohit Sharma    Rohit sharma
2        ms dhoni        MS DHONI        Ms Dhoni        Ms dhoni
3  ab de villiers  AB DE VILLIERS  Ab De Villiers  Ab de villiers

How it works: title() capitalises the first letter of every word and lowercases the rest — notice "AB de Villiers" becomes "Ab De Villiers" and "ms dhoni" becomes "Ms Dhoni". Title case is great for cities but can damage names with special capitalisation; fix those few with replace afterwards.

Example — the counting problem, before and after

cities = pd.Series(["Mumbai", "mumbai", " MUMBAI ", "Mumbai  ", "Delhi", "delhi "])
print(cities.value_counts())
print()
clean = cities.str.strip().str.title()
print(clean.value_counts())

Mumbai      1
mumbai      1
 MUMBAI     1
Mumbai      1
Delhi       1
delhi       1
Name: count, dtype: int64

Mumbai    4
Delhi     2
Name: count, dtype: int64

How it works: before cleaning, pandas sees 6 different values. After strip() (remove edge spaces) and title() (one case style) there are just two cities with the right counts.

Example — strip, lstrip, rstrip and to_strip

raw = pd.Series(["  Eden Gardens  ", "**Wankhede**", "...Chepauk", "Kotla\n"])
print(raw.str.strip().tolist())
print(raw.str.lstrip().tolist())
print(raw.str.rstrip().tolist())
print(raw.str.strip("*.").tolist())

['Eden Gardens', '**Wankhede**', '...Chepauk', 'Kotla']
['Eden Gardens  ', '**Wankhede**', '...Chepauk', 'Kotla\n']
['  Eden Gardens', '**Wankhede**', '...Chepauk', 'Kotla']
['  Eden Gardens  ', 'Wankhede', 'Chepauk', 'Kotla\n']

How it works:

• strip() with no argument removes whitespace (spaces, tabs \t, newlines \n) from both ends.
• lstrip() / rstrip() only clean the left / right end.
• strip("*.") removes any * or . characters from the ends — the argument is a set of characters, not a whole word.

Example — spaces inside the string

strip() never touches inner spaces. Use a regular expression: \s+ means "one or more whitespace characters".

venues = pd.Series(["Eden    Gardens", " M  Chinnaswamy   Stadium ", "Wankhede\tStadium"])
fixed = venues.str.replace(r"\s+", " ", regex=True).str.strip()
print(fixed.tolist())

['Eden Gardens', 'M Chinnaswamy Stadium', 'Wankhede Stadium']

Example — case-insensitive comparison without changing the data

Sometimes you want to compare ignoring case but keep the original text:

teams = pd.Series(["Mumbai Indians", "MUMBAI INDIANS", "Chennai Super Kings"])
print(teams[teams.str.lower() == "mumbai indians"])
print(teams.str.lower().nunique(), "unique teams ignoring case")

0    Mumbai Indians
1    MUMBAI INDIANS
dtype: str
2 unique teams ignoring case

This is the check from Munna's notebook: ipl3['venue'].str.lower().nunique() — if it is smaller than nunique() on the raw column, you have case duplicates.

Common mistakes

• ❌ Expecting strip() to remove spaces between words. ✅ str.replace(r"\s+", " ", regex=True).
• ❌ strip("Mr. ") to remove a prefix — it removes the characters M, r, ., space from both ends, which can eat real letters. ✅ Use str.removeprefix("Mr. ") or a regex replace with ^.
• ❌ Cleaning only for display, then grouping on the dirty column. ✅ Assign it back: df["city"] = df["city"].str.strip().str.title().
• ❌ Blindly title-casing names like "AB de Villiers" or "MS Dhoni". ✅ Title-case, then fix exceptions with replace.

Comparison — which case for which column?

Interview tip

Q: value_counts() shows Mumbai and mumbai separately. How do you fix it? A: Normalise first — df["city"] = df["city"].str.strip().str.title() — then count again.