Correlation & describe() by Group
Measure how two numeric columns move together with corr() (Pearson/Spearman), read a correlation table, and summarise each group with describe() — notebook 08, video 08.
Watch the lesson
1 video on CodeWithMunnaX
Do batters who face more balls score more runs? Do bowlers who bowl more overs have a better economy? These are questions about the relationship between two numeric columns, and the standard number for it is the correlation coefficient. In notebook 08 Munna built Study_hours vs Marks and called data.corr(); here we do the same with IPL batting data.
python
bat = pd.DataFrame({
"player": ["Kohli", "Rohit", "Gill", "Faf", "Surya", "Pant", "Rahul", "Samson"],
"team": ["RCB", "MI", "GT", "RCB", "MI", "DC", "LSG", "RR"],
"balls": [550, 300, 610, 520, 360, 290, 380, 400],
"runs": [741, 417, 890, 730, 605, 446, 520, 531],
"sixes": [38, 23, 33, 31, 32, 28, 19, 26],
"dot_balls": [150, 110, 160, 140, 90, 85, 140, 120],
})Definition
Correlation is a number between −1 and +1 that measures how strongly two numeric variables move together in a straight-line (linear) way.
DataFrame.corr()computes it for every pair of numeric columns and returns a square correlation matrix.
- +1 — perfect positive relationship: when one goes up, the other always goes up proportionally.
- 0 — no linear relationship.
- −1 — perfect negative relationship: when one goes up, the other goes down.
- Pearson (default) — measures linear relationship using actual values. Spearman — measures monotonic relationship using ranks (robust to outliers and curved-but-increasing patterns). Kendall — another rank method (needs SciPy installed).
Reading the strength
| |r| (absolute value) | Usual reading | |---|---| | 0.0 – 0.2 | very weak / none | | 0.2 – 0.4 | weak | | 0.4 – 0.6 | moderate | | 0.6 – 0.8 | strong | | 0.8 – 1.0 | very strong |
The sign gives the direction; the size gives the strength. These cut-offs are rules of thumb, not laws.
Syntax
python
DataFrame.corr(method="pearson", min_periods=1, numeric_only=False)
Series.corr(other, method="pearson")
DataFrame.corrwith(other, method="pearson")
DataFrame.describe(percentiles=None, include=None, exclude=None)
df.groupby(key)[col].describe()| Parameter | Meaning | Default |
|---|---|---|
method | "pearson", "spearman" or "kendall" (or a function) | "pearson" |
min_periods | minimum number of paired non-NaN rows needed to compute a value | 1 |
numeric_only | use only numeric columns; with text columns present and False, corr() raises ValueError | False |
other | the Series to correlate with | required |
percentiles (describe) | which percentiles to show, e.g. [0.1, 0.9] | [.25, .5, .75] |
include (describe) | "all" to also describe text columns | None (numeric only) |
Example — perfect correlation (notebook 08)
python
data = pd.DataFrame({"Study_hours": range(1, 11), "Marks": range(10, 101, 10)})
print(data.corr())text
Study_hours Marks
Study_hours 1.0 1.0
Marks 1.0 1.0Marks are exactly 10 × hours, a perfect straight line, so r = 1.0. The diagonal is always 1 (every column perfectly correlates with itself).
Example — the correlation matrix
python
print(bat.corr(numeric_only=True).round(2))text
balls runs sixes dot_balls
balls 1.00 0.96 0.65 0.87
runs 0.96 1.00 0.72 0.74
sixes 0.65 0.72 1.00 0.24
dot_balls 0.87 0.74 0.24 1.00How to read it: find a row, then a column. balls–runs = 0.96: very strong positive — more balls faced, more runs. runs–sixes = 0.72: strong. sixes–dot_balls = 0.24: weak. The matrix is symmetric (upper and lower triangles mirror each other), so you only need to read one half.
Note (pandas 3):
bat.corr()withoutnumeric_only=TrueraisesValueError: could not convert string to float: 'Kohli'because of theplayer/teamtext columns. Passnumeric_only=Trueor select number columns first.
Example — one pair, one target, and Spearman
python
print(bat["balls"].corr(bat["runs"]))
print(bat.corr(numeric_only=True)["runs"].drop("runs").sort_values(ascending=False).round(2))
print(bat.corr(method="spearman", numeric_only=True).round(2))text
0.9648788079637409
balls 0.96
dot_balls 0.74
sixes 0.72
Name: runs, dtype: float64
balls runs sixes dot_balls
balls 1.00 0.90 0.60 0.93
runs 0.90 1.00 0.83 0.75
sixes 0.60 0.83 1.00 0.41
dot_balls 0.93 0.75 0.41 1.00How it works: Series.corr(other) gives a single number. Selecting the runs column of the matrix and sorting ranks which features relate most to runs (a common step before building a model). corrwith(bat["runs"]) does the same in one call. Spearman works on ranks, so its values differ from Pearson's.
Negative and "hidden" relationships
python
x = pd.DataFrame({"overs_bowled": [4, 4, 3, 2, 4, 1], "economy": [6.0, 6.5, 8.0, 9.5, 7.0, 11.0]})
print(x.corr().round(2))
y = pd.DataFrame({"x": [-3, -2, -1, 0, 1, 2, 3]})
y["x_squared"] = y["x"] ** 2
print(y.corr().round(2))text
overs_bowled economy
overs_bowled 1.00 -0.99
economy -0.99 1.00
x x_squared
x 1.0 0.0
x_squared 0.0 1.0The first shows a strong negative correlation (captains give more overs to bowlers with low economy). The second is a warning: x_squared depends completely on x, yet r = 0 — because the relationship is a U-shape, not a straight line. r = 0 means "no linear relationship", not "no relationship".
Remember: correlation is not causation. Balls faced and runs are correlated because both depend on how long a batter stays in — not because facing balls by itself creates runs.
In notebook 08 the matrix is visualised with sns.heatmap(df.corr(numeric_only=True), annot=True) — a coloured grid of the same numbers (plotting is not available in the browser playground).
describe() and describe() by group
describe() gives the standard summary (count, mean, std, min, quartiles, max) of every numeric column; include="all" adds unique, top (most frequent) and freq for text columns. On a GroupBy it produces one summary row per group:
python
s = pd.DataFrame({
"team": ["MI", "MI", "MI", "CSK", "CSK", "CSK"],
"runs": [45, 80, 12, 60, 20, 95],
})
print(s.groupby("team")["runs"].describe())text
count mean std min 25% 50% 75% max
team
CSK 3.0 58.333333 37.527767 20.0 40.0 60.0 77.5 95.0
MI 3.0 45.666667 34.004902 12.0 28.5 45.0 62.5 80.0How it works: each team's runs is described separately — a quick way to compare centre and spread across groups. Correlation can be grouped too: df.groupby("team")[["balls", "runs"]].corr() returns one small matrix per team (stacked with a MultiIndex).
Common mistakes
- ❌
df.corr()on a frame with text columns →ValueError. ✅df.corr(numeric_only=True). - ❌ Reading r = 0 as "unrelated". ✅ It only rules out a linear relationship — plot the data.
- ❌ Treating high correlation as proof of cause. ✅ Correlation is not causation.
- ❌ Trusting correlation from 5 rows. ✅ Small samples give unstable r values; use
min_periodsand more data. - ❌ Using
method="kendall"without SciPy. ✅ Pearson and Spearman work in pure pandas; Kendall needs SciPy.
Interview tip
Q: Pearson vs Spearman correlation — when to use which? A: Pearson measures linear relationships on raw values and is sensitive to outliers. Spearman correlates the ranks, so it captures any monotonic (always increasing/decreasing) relationship and is robust to outliers.
Key points to remember
- Correlation is a number from −1 to +1 measuring how strongly two numeric variables move together linearly.
- df.corr() returns a symmetric correlation matrix with 1.0 on the diagonal; Series.corr(other) returns one number.
- In pandas 3, corr() with text columns raises ValueError — use numeric_only=True.
- method='pearson' (default) uses values; method='spearman' uses ranks and is robust to outliers; 'kendall' needs SciPy.
- r = 0 means no linear relationship, not no relationship — and correlation is not causation.
- describe() summarises count, mean, std, min, quartiles and max; groupby(key)[col].describe() does it per group.
- corr()['target'].sort_values() or corrwith() ranks which columns relate most to a target column.
Try it yourself
Correlation matrix of batting stats
Negative, zero and per-group summaries
Practice: what drives a bowler's wickets?
Revision Flashcards
Tap a card to flip it and see the answer.