Performance & Memory Tips

The IPL file in notebook 06 has 816 rows — tiny. But ball-by-ball IPL data has over 200,000 rows, and real company files have millions. At that size two things hurt: memory (the laptop runs out of RAM) and speed (a cell runs for minutes). This topic collects the habits that fix both.

Definition

Pandas performance tuning means reducing the memory a DataFrame occupies (by choosing smaller or smarter data types and loading only what you need) and reducing run time (by using vectorised, compiled operations instead of Python loops).

• Memory depends on rows × columns × bytes per value — and the dtype decides the bytes per value. Text is the most expensive.
• Run time depends mostly on whether the work happens in compiled code (vectorised) or in a Python loop.

Syntax — the toolkit

df.memory_usage(deep=True)                   # bytes per column (deep=True counts text properly)
df.info(memory_usage="deep")
s.astype("category")                         # repeated text → small integer codes
pd.to_numeric(s, downcast="integer")         # int64 → int8/int16/int32 if values fit
s.astype("float32")                          # half the memory of float64
pd.read_csv(path, usecols=[...], dtype={...})    # load only what you need, with small types
for chunk in pd.read_csv(path, chunksize=100_000): ...   # stream a big file

Example — measure, then shrink with category and downcasting

import pandas as pd
import numpy as np

rng = np.random.default_rng(42)
n = 100_000
teams = np.array(["MI", "CSK", "RCB", "KKR", "DC", "SRH", "RR", "PBKS", "GT", "LSG"])
df = pd.DataFrame({
    "team": rng.choice(teams, n),
    "runs": rng.integers(0, 150, n),
    "balls": rng.integers(1, 100, n),
    "strike_rate": rng.uniform(50, 250, n),
})
print(df.memory_usage(deep=True))
print(f"total: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB")

Index              132
team           5170222
runs            800000
balls           800000
strike_rate     800000
dtype: int64
total: 7.22 MB

The text column team alone uses 5.2 MB — more than the three numeric columns together (8 bytes × 100,000 = 800,000 bytes each).

df["team"] = df["team"].astype("category")
for col in ["runs", "balls"]:
    df[col] = pd.to_numeric(df[col], downcast="integer")
df["strike_rate"] = df["strike_rate"].astype("float32")
print(df.dtypes)
print(f"total after: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB")

team           category
runs              int16
balls              int8
strike_rate     float32
dtype: object
total after: 0.76 MB

How it works:

• category — only 10 distinct teams, so pandas stores the 10 names once and keeps a tiny integer code (1 byte) per row: 5,049 KB → 98 KB.
• downcast — balls (1–99) fits in int8 (−128…127) and runs (0–149) needs int16: 8 bytes → 1–2 bytes per value.
• float32 — 4 bytes instead of 8; plenty of precision for a strike rate.

Total: 7.22 MB → 0.76 MB, almost 10× smaller. (Exact byte counts can vary slightly between machines and pandas builds.)

Tip: category helps when a column has few unique values compared with its length (teams, cities, fuel type). For a column of unique player names it can make memory worse. Check s.nunique() / len(s) first.

Example — load only what you need: usecols + dtype

small = pd.read_csv(path, usecols=["season", "team", "runs"],
                    dtype={"team": "category", "season": "int16", "runs": "int16"})
print(small.dtypes)

season       int16
team      category
runs         int16
dtype: object

On a 1,000-row, 7-column test file (simulated with io.StringIO), loading everything took 244,092 bytes, while usecols + dtype took 5,949 bytes. Columns you never use (umpires, venue IDs…) never enter memory, and the types are small from the start.

Example — stream a big file with chunksize

totals = None
for chunk in pd.read_csv(path, usecols=["team", "runs"], chunksize=250):
    part = chunk.groupby("team")["runs"].sum()
    totals = part if totals is None else totals.add(part, fill_value=0)
print(totals.astype(int).sort_values(ascending=False).head(3))

team
PBKS    10400
DC      10300
CSK     10200
Name: runs, dtype: int64

How it works: with chunksize, read_csv returns a TextFileReader iterator; each loop gives a DataFrame of up to 250 rows. You compute a partial result per chunk and combine them with .add(..., fill_value=0). Only one chunk is in memory at a time, so you can process files bigger than your RAM. The answer matches a full groupby on the whole file.

Speed habits

Common mistakes

• ❌ df.memory_usage() without deep=True — text columns look tiny. ✅ Always deep=True for real numbers.
• ❌ Converting high-cardinality columns (unique IDs, names) to category. ✅ Only columns with few repeated values.
• ❌ Downcasting a column and later adding values that overflow (int8 max is 127). ✅ Leave headroom, or downcast only final analysis tables.
• ❌ Loading a 2 GB CSV fully to use 3 columns. ✅ usecols + dtype, or chunksize.
• ❌ Optimising before measuring. ✅ Measure memory and time first, fix the biggest problem.

Interview tip

Q: A 5 GB CSV doesn't fit in memory. What do you do? A: Load only the needed columns (usecols) with compact types (dtype with category/int16/float32). If it still doesn't fit, process it in pieces with chunksize, computing partial aggregates per chunk and combining them. Also consider a columnar format like Parquet for repeated use.