The EDA Workflow
A repeatable six-step recipe for exploring any new dataset: ask questions, load, inspect, clean, analyse and summarise — worked end to end on a messy IPL players file.
You know dozens of pandas methods now. The hard part of a real project is not which method — it is what to do first, second and third when someone hands you an unknown CSV. The car-pricing (notebooks 04/05) and IPL (notebook 06) projects in this course all follow the same rhythm. This topic turns that rhythm into a checklist you can reuse on any dataset.
Definition
Exploratory Data Analysis (EDA) is the process of examining a new dataset to understand its structure, quality and main patterns — using summary statistics, counts, group comparisons and charts — before drawing conclusions or building models.
- Structure — how many rows and columns, what each column means, what data types they have.
- Quality — missing values, duplicates, wrong types, inconsistent spellings, impossible values.
- Patterns — which groups are bigger/better, what changes over time, which columns move together.
- Before conclusions — EDA is how you avoid answering a question with dirty data.
The six steps
text
1. QUESTIONS What do I want to know? ("Which team scores the most?")
↓
2. LOAD pd.read_csv / read_json / DataFrame(...)
↓
3. INSPECT shape · head · info · describe · isna().sum() · duplicated() · nunique()
↓
4. CLEAN drop_duplicates · str.strip/title · fillna/dropna · astype · to_datetime · replace
↓
5. ANALYSE value_counts · groupby/agg · pivot_table · sort/nlargest · corr · charts
↓
6. SUMMARISE 3–5 plain-English findings with numbersSteps 3–5 are a loop: analysis often reveals new problems ("why are there two spellings of Bengaluru?"), so you go back and clean again.
Syntax — the inspection toolkit
python
df.shape # (rows, columns)
df.head(n) / df.tail(n) / df.sample(n)
df.info() # dtypes + non-null counts + memory
df.describe() # count, mean, std, min, quartiles, max (numeric)
df.describe(include="str") # count, unique, top, freq (text columns)
df.isna().sum() # missing values per column
df.duplicated().sum() # number of fully duplicated rows
df.nunique() # distinct values per column
df["col"].value_counts() # frequency of each value| Method | Answers the question… |
|---|---|
shape | How big is it? |
info() | Which types? Which columns have gaps? |
describe() | What are typical / extreme values? Any impossible numbers? |
isna().sum() | How much is missing, and where? |
duplicated().sum() | Are any rows repeated? |
nunique() / value_counts() | Is a column categorical? Any spelling variants? |
Example — steps 1–3: questions, load, inspect
Questions: Which team has the most runs and wickets? Who is the best run-scorer per match? How do roles compare?
python
import pandas as pd
import io
csv = io.StringIO("""player,team,role,matches,runs,wickets,debut
Virat Kohli,RCB,Batter,15,741,0,2008-04-18
rohit sharma,MI,Batter,14,417,0,2008-05-04
MS Dhoni,CSK,Wicketkeeper,14,161,,2008-04-19
Shubman Gill,GT,Batter,12,426,0,2018-04-15
Jasprit Bumrah,MI,Bowler,13,12,20,2013-04-04
Ravindra Jadeja,CSK,All-rounder,14,267,8,2008-04-19
Virat Kohli,RCB,Batter,15,741,0,2008-04-18
Rashid Khan ,GT,Bowler,12,,10,2017-04-05
Andre Russell,KKR,All-rounder,14,222,19,2012-04-28
""")
df = pd.read_csv(csv)
print(df)
df.info()
print(df.isna().sum())
print("duplicates:", df.duplicated().sum())text
player team role matches runs wickets debut
0 Virat Kohli RCB Batter 15 741.0 0.0 2008-04-18
1 rohit sharma MI Batter 14 417.0 0.0 2008-05-04
2 MS Dhoni CSK Wicketkeeper 14 161.0 NaN 2008-04-19
3 Shubman Gill GT Batter 12 426.0 0.0 2018-04-15
4 Jasprit Bumrah MI Bowler 13 12.0 20.0 2013-04-04
5 Ravindra Jadeja CSK All-rounder 14 267.0 8.0 2008-04-19
6 Virat Kohli RCB Batter 15 741.0 0.0 2008-04-18
7 Rashid Khan GT Bowler 12 NaN 10.0 2017-04-05
8 Andre Russell KKR All-rounder 14 222.0 19.0 2012-04-28
<class 'pandas.DataFrame'>
RangeIndex: 9 entries, 0 to 8
Data columns (total 7 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 player 9 non-null str
1 team 9 non-null str
2 role 9 non-null str
3 matches 9 non-null int64
4 runs 8 non-null float64
5 wickets 8 non-null float64
6 debut 9 non-null str
dtypes: float64(2), int64(1), str(4)
memory usage: 636.0 bytes
player 0
team 0
role 0
matches 0
runs 1
wickets 1
debut 0
dtype: int64
duplicates: 1What inspection told us — write these down, they become the cleaning plan:
| Finding | Evidence | Fix |
|---|---|---|
| Virat appears twice | duplicated().sum() = 1 (row 6) | drop_duplicates() |
| inconsistent names | rohit sharma lower case, " Rashid Khan " has spaces | .str.strip().str.title() |
runs, wickets are float | one NaN each forces float64 | fillna(0).astype(int) |
debut is text (str) | info() shows str | pd.to_datetime |
Example — step 4: clean in one chain
python
clean = (
df
.drop_duplicates()
.assign(
player=lambda d: d["player"].str.strip().str.title(),
runs=lambda d: d["runs"].fillna(0).astype(int),
wickets=lambda d: d["wickets"].fillna(0).astype(int),
debut=lambda d: pd.to_datetime(d["debut"]),
)
.reset_index(drop=True)
)
print(clean)
print(clean.dtypes)text
player team role matches runs wickets debut
0 Virat Kohli RCB Batter 15 741 0 2008-04-18
1 Rohit Sharma MI Batter 14 417 0 2008-05-04
2 Ms Dhoni CSK Wicketkeeper 14 161 0 2008-04-19
3 Shubman Gill GT Batter 12 426 0 2018-04-15
4 Jasprit Bumrah MI Bowler 13 12 20 2013-04-04
5 Ravindra Jadeja CSK All-rounder 14 267 8 2008-04-19
6 Rashid Khan GT Bowler 12 0 10 2017-04-05
7 Andre Russell KKR All-rounder 14 222 19 2012-04-28
player str
team str
role str
matches int64
runs int64
wickets int64
debut datetime64[us]
dtype: objectHow it works: one readable pipeline fixes every problem from the plan. But look closely at row 2: .str.title() turned MS Dhoni into Ms Dhoni. Cleaning can create errors — always re-inspect the result. (A fix: .replace({"Ms Dhoni": "MS Dhoni"}) after title().) Also note a judgement call: we filled Rashid's missing runs with 0. For a bowler that is reasonable; for a batter you might drop the row instead. Write such decisions down.
Example — steps 5–6: analyse and summarise
python
print(clean.groupby("team")[["runs", "wickets"]].sum().sort_values("runs", ascending=False))
clean = clean.assign(runs_per_match=lambda d: (d["runs"] / d["matches"]).round(1))
print(clean.nlargest(3, "runs_per_match")[["player", "team", "runs_per_match"]])
print(clean.pivot_table(index="role", values=["runs", "wickets"], aggfunc="mean").round(1))
top = clean.loc[clean["runs"].idxmax()]
print(f"Top run-scorer: {top['player']} ({top['team']}) with {top['runs']} runs")text
runs wickets
team
RCB 741 0
MI 429 20
CSK 428 8
GT 426 10
KKR 222 19
player team runs_per_match
0 Virat Kohli RCB 49.4
3 Shubman Gill GT 35.5
1 Rohit Sharma MI 29.8
runs wickets
role
All-rounder 244.5 13.5
Batter 528.0 0.0
Bowler 6.0 15.0
Wicketkeeper 161.0 0.0
Top run-scorer: Virat Kohli (RCB) with 741 runsSummary (step 6) — plain English, with numbers:
- RCB lead team runs (741) but took no wickets; MI have the best balance (429 runs, 20 wickets).
- Virat Kohli is the most productive batter at 49.4 runs per match.
- All-rounders average 244.5 runs and 13.5 wickets — the most complete role.
Common mistakes
- ❌ Jumping straight to
groupbywithout inspecting. ✅ Always runinfo(),isna().sum()andduplicated().sum()first — duplicates silently double totals. - ❌
dropna()on the whole DataFrame by reflex. ✅ Check where data is missing; dropping can delete most rows (in notebook 06,dropna()left only 19 of 816 matches because one column was 97% empty). - ❌ Cleaning in place over many cells and losing track. ✅ Keep
dfraw and buildcleanwith a chain — you can always re-run it. - ❌ Ending with a pile of tables. ✅ Finish with written findings that answer your step-1 questions.
Interview tip
Q: You receive a new CSV. What are the first five things you run?
A: df.shape, df.head(), df.info() (types and nulls), df.describe() (ranges, impossible values) and df.isna().sum() plus df.duplicated().sum(). Together they reveal size, types, missing data, duplicates and outliers — the cleaning plan comes from these.
Key points to remember
- EDA = understanding a dataset's structure, quality and patterns before drawing conclusions.
- Six steps: questions → load → inspect → clean → analyse → summarise; inspect/clean/analyse is a loop.
- Inspection toolkit: shape, head, info, describe, isna().sum(), duplicated().sum(), nunique(), value_counts().
- Turn inspection findings into a written cleaning plan (problem, evidence, fix).
- Clean with a method chain from the raw DataFrame so the steps are repeatable.
- Re-inspect after cleaning — cleaning can introduce errors (title() turned 'MS Dhoni' into 'Ms Dhoni').
- End with 3–5 plain-English findings backed by numbers that answer your original questions.
Try it yourself
Inspect a messy file
Clean, analyse and summarise
Practice: run the checklist yourself
Revision Flashcards
Tap a card to flip it and see the answer.