read_csv Parameters in Depth

A real IPL matches file can have 20+ columns and thousands of rows, some cells marked - or DNB (did not bat), dates stored as text, and IDs that should never be treated as numbers. Cleaning all of that after loading is possible — but read_csv can do most of it while loading, with a few parameters. Loading only what you need is also faster and uses less memory.

All examples in this topic use this small match-results file:

import pandas as pd
import io

matches_csv = """match_id,date,season,team1,team2,winner,venue,margin
1001,2024-03-22,2024,CSK,RCB,CSK,Chennai,6
1002,2024-03-23,2024,PBKS,DC,PBKS,Mohali,4
1003,2024-03-23,2024,KKR,SRH,KKR,Kolkata,4
1004,2024-03-24,2024,RR,LSG,RR,Jaipur,20
1005,2024-03-24,2024,GT,MI,GT,Ahmedabad,6
1006,2024-03-25,2024,RCB,PBKS,RCB,Bengaluru,4
"""

Definition

read_csv parameters are keyword arguments that tell pandas which rows and columns to load, what type each column should be, and which text should count as missing, so the DataFrame comes out clean in a single step.

• Which rows — nrows (only the first n), skiprows (skip some).
• Which columns — usecols (keep only some), index_col (turn one into the row labels).
• What type — dtype (force a type), parse_dates (turn text into real dates), thousands (strip separators from numbers).
• What is missing — na_values (extra markers that mean "no value"), keep_default_na.
• How the bytes are decoded — encoding.

Syntax

pd.read_csv(path, index_col=None, usecols=None, dtype=None, parse_dates=None,
            na_values=None, keep_default_na=True, nrows=None, skiprows=None,
            encoding="utf-8", thousands=None)

Example — index_col and usecols

df = pd.read_csv(io.StringIO(matches_csv), index_col="match_id")
print(df.head(3))

df = pd.read_csv(io.StringIO(matches_csv), usecols=["match_id", "team1", "team2", "winner"])
print(df)

                date  season team1 team2 winner    venue  margin
match_id
1001      2024-03-22    2024   CSK   RCB    CSK  Chennai       6
1002      2024-03-23    2024  PBKS    DC   PBKS   Mohali       4
1003      2024-03-23    2024   KKR   SRH    KKR  Kolkata       4
   match_id team1 team2 winner
0      1001   CSK   RCB    CSK
1      1002  PBKS    DC   PBKS
2      1003   KKR   SRH    KKR
3      1004    RR   LSG     RR
4      1005    GT    MI     GT
5      1006   RCB  PBKS    RCB

How it works: index_col="match_id" turns that column into the row labels — notice match_id printed on its own line above the index, and it is no longer a normal column. Now df.loc[1004] finds match 1004 directly. usecols loads only the four listed columns; the others are never even stored in memory. usecols also accepts positions, e.g. usecols=[0, 5].

Example — dtype and parse_dates

df = pd.read_csv(io.StringIO(matches_csv))
print(df.dtypes)

df = pd.read_csv(io.StringIO(matches_csv),
                 dtype={"match_id": "str", "season": "category", "margin": "float64"},
                 parse_dates=["date"])
print(df.dtypes)
print(df["date"].dt.day_name().tolist())

match_id    int64
date          str
season      int64
team1         str
team2         str
winner        str
venue         str
margin      int64
dtype: object
match_id               str
date        datetime64[us]
season            category
team1                  str
team2                  str
winner                 str
venue                  str
margin             float64
dtype: object
['Friday', 'Saturday', 'Saturday', 'Sunday', 'Sunday', 'Monday']

How it works: without help, date is just text (str) — you cannot ask it for a weekday. parse_dates=["date"] converts it to a real datetime column, unlocking the .dt accessor. The dtype dict forces types: match_id as text (IDs are labels, not quantities — you never add two match IDs), season as category (few repeated values → less memory), and margin as float.

Note (pandas 3): parsed dates show as datetime64[us] (microsecond resolution). Older pandas always used datetime64[ns]. For everyday work they behave the same.

Example — na_values: custom "missing" markers

Scorecards often write - or DNB instead of leaving a cell empty:

scores = """player,team,runs,wickets
Virat Kohli,RCB,741,-
Jasprit Bumrah,MI,DNB,20
MS Dhoni,CSK,161,NA
Harshal Patel,PBKS,,24
"""
df = pd.read_csv(io.StringIO(scores))
print(df)
print(df.dtypes)

df = pd.read_csv(io.StringIO(scores), na_values=["-", "DNB"])
print(df)
print(df.dtypes)

           player  team runs wickets
0     Virat Kohli   RCB  741       -
1  Jasprit Bumrah    MI  DNB      20
2        MS Dhoni   CSK  161     NaN
3   Harshal Patel  PBKS  NaN      24
player     str
team       str
runs       str
wickets    str
dtype: object
           player  team   runs  wickets
0     Virat Kohli   RCB  741.0      NaN
1  Jasprit Bumrah    MI    NaN     20.0
2        MS Dhoni   CSK  161.0      NaN
3   Harshal Patel  PBKS    NaN     24.0
player         str
team           str
runs       float64
wickets    float64
dtype: object

How it works: by default, empty cells and markers like NA are already read as NaN. But - and DNB are unknown text, so the whole column became str — you can't even sum the runs. Adding them to na_values turns them into NaN, and the columns become numeric. They are float64 (not int64) because NaN is a float. You can also target columns: na_values={"runs": ["DNB"], "wickets": ["-"]}.

The reverse problem exists too: a team code that is literally "NA" would be read as missing. keep_default_na=False switches the built-in markers off.

Example — nrows and skiprows (a list)

print(pd.read_csv(io.StringIO(matches_csv), nrows=2))
print(pd.read_csv(io.StringIO(matches_csv), skiprows=[1, 2]))

   match_id        date  season team1 team2 winner    venue  margin
0      1001  2024-03-22    2024   CSK   RCB    CSK  Chennai       6
1      1002  2024-03-23    2024  PBKS    DC   PBKS   Mohali       4
   match_id        date  season team1 team2 winner      venue  margin
0      1003  2024-03-23    2024   KKR   SRH    KKR    Kolkata       4
1      1004  2024-03-24    2024    RR   LSG     RR     Jaipur      20
2      1005  2024-03-24    2024    GT    MI     GT  Ahmedabad       6
3      1006  2024-03-25    2024   RCB  PBKS    RCB  Bengaluru       4

How it works: nrows=2 stops after two data rows — great for peeking at a 2 GB file without loading it all. skiprows=[1, 2] skips file lines 1 and 2 (line 0 is the header, so the header survives and the first two matches are dropped).

Example — encoding and thousands

A file saved by old Windows/Excel software may not be UTF-8. Reading it with the default encoding fails:

raw = "player,team\nAnrich Nortjé,DC\nKagiso Rabada,PBKS\n".encode("latin-1")
try:
    pd.read_csv(io.BytesIO(raw))
except UnicodeDecodeError as e:
    print("UnicodeDecodeError:", e)
print(pd.read_csv(io.BytesIO(raw), encoding="latin-1"))

prize = 'team,prize_money\nKKR,"20,00,00,000"\nSRH,"12,50,00,000"\n'
print(pd.read_csv(io.StringIO(prize), thousands=","))

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 24: invalid continuation byte
          player  team
0  Anrich Nortjé    DC
1  Kagiso Rabada  PBKS
  team  prize_money
0  KKR    200000000
1  SRH    125000000

How it works: the byte 0xe9 is é in Latin-1 but is not valid UTF-8, so pandas stops. Telling it encoding="latin-1" (or "cp1252" for Windows files) decodes it correctly. thousands="," removes the separators inside "20,00,00,000", so prize_money becomes int64 instead of text. (Indian-style grouping works too — pandas just deletes every comma inside the number.)

Quick reference — which parameter fixes which problem?

Common mistakes

• ❌ A numeric column shows dtype str, so .sum() fails. ✅ Some cell contains a non-number marker — find it and add it to na_values.
• ❌ Expecting int64 for a column with missing values. ✅ NaN is a float, so the column becomes float64 (or use dtype={"runs": "Int64"} — capital I — for a nullable integer).
• ❌ skiprows=2 when you meant "skip the 2nd data row". ✅ An int skips that many lines from the top; use a list like skiprows=[2] to skip a specific line.
• ❌ Doing date math on a text column. ✅ Use parse_dates (or pd.to_datetime later).

Interview tip

Q: How would you load only 3 columns and the first 10,000 rows of a 5 GB CSV? A: pd.read_csv(path, usecols=["a", "b", "c"], nrows=10_000). Both filters are applied while reading, so pandas never builds the full table in memory. For processing the whole file in pieces, chunksize=100_000 returns an iterator of smaller DataFrames.