Opening notes

Loading…

Notes NumPyStatistics & Aggregation

Statistics & AggregationTopic 36 of 775 min read

Describe Your Data

Put every statistic together into a mini describe() function — count, mean, std, min, quartiles, max — and print a per-player summary table using only NumPy.

Before any serious analysis, a data analyst first asks: what does this data look like? Pandas answers that with df.describe(). In this topic you build the same summary yourself with the NumPy tools from this chapter — which also shows you exactly what describe() is doing under the hood.

Definition

A descriptive summary (what Pandas calls describe()) is a small table of statistics that describes a dataset's size, centre, spread and shape: count, mean, std, min, 25%, 50% (median), 75% and max.

Statistic	Tells you	NumPy
count	how many real values	`np.count_nonzero(~np.isnan(a))`
mean	centre (average)	`np.nanmean(a)`
std	spread (consistency)	`np.nanstd(a, ddof=1)` — Pandas uses sample std
min	lowest	`np.nanmin(a)`
25% / 50% / 75%	quartiles (50% = median)	`np.nanpercentile(a, [25, 50, 75])`
max	highest	`np.nanmax(a)`

Using the nan- versions makes the summary safe for missing data, exactly like Pandas.

Syntax — our helper function

python

def describe(a, ddof=1):
    a = np.asarray(a, dtype=float)
    q1, q2, q3 = np.nanpercentile(a, [25, 50, 75])
    return {
        "count": np.count_nonzero(~np.isnan(a)),
        "mean": np.nanmean(a), "std": np.nanstd(a, ddof=ddof),
        "min": np.nanmin(a), "25%": q1, "50%": q2, "75%": q3,
        "max": np.nanmax(a),
    }

Parameter	Meaning	Default
`a`	the data (list or array, NaN allowed)	— required
`ddof`	1 = sample std (like Pandas), 0 = population std (NumPy's default)	`1`

Example — describe one player

python

import numpy as np

def describe(a, ddof=1):
    a = np.asarray(a, dtype=float)
    q1, q2, q3 = np.nanpercentile(a, [25, 50, 75])
    return {
        "count": int(np.count_nonzero(~np.isnan(a))),
        "mean": float(np.nanmean(a)), "std": float(np.nanstd(a, ddof=ddof)),
        "min": float(np.nanmin(a)), "25%": float(q1), "50%": float(q2),
        "75%": float(q3), "max": float(np.nanmax(a)),
    }

virat = [72, 45, 88, 31, 64, 0, 103, 52, 19, 77]
for name, value in describe(virat).items():
    print(f"{name:<6} {value:8.2f}")

text

count     10.00
mean      55.10
std       32.06
min        0.00
25%       34.50
50%       58.00
75%       75.75
max      103.00

How it works: every line is one NumPy call from this chapter. Converting with np.asarray(a, dtype=float) lets the function accept plain lists and arrays containing NaN. Wrapping results in float()/int() turns NumPy scalars into plain Python numbers for clean printing.

Example — a summary table for the whole squad (axis=1)

The same idea works on a 2D table: compute each statistic with axis=1 and you get one row per player — all without a loop over the data.

python

import numpy as np

players = ["Virat", "Rohit", "Hardik", "Shubman"]
scores = np.array([
    [72, 45, 88, 31, 64, 0],
    [56,  0, 74, 88, 32, 45],
    [45, 32, np.nan, 67, 18, 54],
    [33, 61, 45,  0, 77, 83],
])

count = np.count_nonzero(~np.isnan(scores), axis=1)
mean = np.nanmean(scores, axis=1)
std = np.nanstd(scores, axis=1, ddof=1)
q1, med, q3 = np.nanpercentile(scores, [25, 50, 75], axis=1)
low, high = np.nanmin(scores, axis=1), np.nanmax(scores, axis=1)

print(f"{'player':<8}{'count':>6}{'mean':>7}{'std':>7}{'min':>6}{'25%':>7}{'50%':>7}{'75%':>7}{'max':>6}")
for i, p in enumerate(players):
    print(f"{p:<8}{count[i]:>6}{mean[i]:>7.1f}{std[i]:>7.1f}{low[i]:>6.0f}"
          f"{q1[i]:>7.1f}{med[i]:>7.1f}{q3[i]:>7.1f}{high[i]:>6.0f}")

text

player   count   mean    std   min    25%    50%    75%   max
Virat        6   50.0   31.7     0   34.5   54.5   70.0    88
Rohit        6   49.2   31.3     0   35.2   50.5   69.5    88
Hardik       5   43.2   19.0    18   32.0   45.0   54.0    67
Shubman      6   49.8   30.8     0   36.0   53.0   73.0    83

How it works: np.nanpercentile(..., [25, 50, 75], axis=1) returns a (3, 4) array — one row per percentile — which we unpack into q1, med, q3. Hardik's washed-out match is ignored (count 5).

Reading a summary — what to look for

Pattern in the summary	What it suggests
mean much bigger than 50% (median)	a few very big values (right-skew), e.g. a couple of centuries
std large compared to mean	inconsistent player
min = 0	at least one duck
count smaller than the number of matches	missing data (NaN) — check why
75% − 25% (IQR) small	middle innings are tightly grouped

Example — one-line extras: range, coefficient of variation, ducks and fifties

python

import numpy as np

virat = np.array([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])
print("range           :", np.ptp(virat))
print("coeff. of var % :", round(float(virat.std() / virat.mean() * 100), 1))
print("ducks           :", np.count_nonzero(virat == 0))
print("50+ scores      :", np.count_nonzero(virat >= 50))
print("100+ scores     :", np.count_nonzero(virat >= 100))

text

range           : 103
coeff. of var % : 55.2
ducks           : 1
50+ scores      : 6
100+ scores     : 1

How it works: the coefficient of variation (std ÷ mean × 100) lets you compare consistency between players with different averages. Comparisons like virat >= 50 give booleans; counting the Trues is a quick way to count milestones.

Common mistakes

❌ Comparing your NumPy std with Pandas' describe() and finding a mismatch. ✅ describe() uses ddof=1; pass the same.
❌ Using plain mean/min in the helper → one NaN ruins every statistic. ✅ Use the nan-versions.
❌ Printing NumPy 2 scalars inside a list or dict and getting np.float64(55.1). ✅ Print them one by one, or convert with float() / .tolist().
❌ Looking only at the mean. ✅ Read the whole summary — centre, spread and extremes together.

Interview tip

Q: Which statistics does Pandas' describe() show for a numeric column, and how would you reproduce it in NumPy? A: count, mean, std, min, 25%, 50%, 75%, max. In NumPy: np.count_nonzero(~np.isnan(a)), np.nanmean, np.nanstd(a, ddof=1), np.nanmin, np.nanpercentile(a, [25, 50, 75]) and np.nanmax.

Key points to remember

A descriptive summary shows count, mean, std, min, 25%, 50%, 75% and max — the same as Pandas describe().
Use nan-functions so the summary ignores missing values; count = number of non-NaN values.
Pandas describe() uses the sample std (ddof=1); NumPy's default is ddof=0.
On a 2D table, pass axis=1 to every statistic to get one summary row per player without loops.
np.nanpercentile(a, [25, 50, 75], axis=1) returns one row per requested percentile.
Mean far above the median signals big outliers; std large relative to mean signals inconsistency.
Counting conditions (np.count_nonzero(a >= 50)) is a quick way to add milestones to a summary.

Try it yourself

A mini describe() function

import numpy as np

def describe(a, ddof=1):
    a = np.asarray(a, dtype=float)
    q1, q2, q3 = np.nanpercentile(a, [25, 50, 75])
    stats = {
        "count": np.count_nonzero(~np.isnan(a)),
        "mean": np.nanmean(a), "std": np.nanstd(a, ddof=ddof),
        "min": np.nanmin(a), "25%": q1, "50%": q2, "75%": q3,
        "max": np.nanmax(a),
    }
    for k, v in stats.items():
        print(f"{k:<6} {float(v):8.2f}")

describe([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])

import numpy as np

def describe(a, ddof=1):
    a = np.asarray(a, dtype=float)
    q1, q2, q3 = np.nanpercentile(a, [25, 50, 75])
    stats = {
        "count": np.count_nonzero(~np.isnan(a)),
        "mean": np.nanmean(a), "std": np.nanstd(a, ddof=ddof),
        "min": np.nanmin(a), "25%": q1, "50%": q2, "75%": q3,
        "max": np.nanmax(a),
    }
    for k, v in stats.items():
        print(f"{k:<6} {float(v):8.2f}")

describe([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])

Squad summary table

import numpy as np

players = ["Virat", "Rohit", "Hardik"]
scores = np.array([
    [72, 45, 88, 31, 64, 0],
    [56,  0, 74, 88, 32, 45],
    [45, 32, np.nan, 67, 18, 54],
])

mean = np.nanmean(scores, axis=1)
std = np.nanstd(scores, axis=1, ddof=1)
med = np.nanmedian(scores, axis=1)
for p, m, s, md in zip(players, mean, std, med):
    print(f"{p:<7} mean {m:5.1f}  median {md:5.1f}  std {s:5.1f}")

import numpy as np

players = ["Virat", "Rohit", "Hardik"]
scores = np.array([
    [72, 45, 88, 31, 64, 0],
    [56,  0, 74, 88, 32, 45],
    [45, 32, np.nan, 67, 18, 54],
])

mean = np.nanmean(scores, axis=1)
std = np.nanstd(scores, axis=1, ddof=1)
med = np.nanmedian(scores, axis=1)
for p, m, s, md in zip(players, mean, std, med):
    print(f"{p:<7} mean {m:5.1f}  median {md:5.1f}  std {s:5.1f}")

Milestones and consistency

import numpy as np

virat = np.array([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])
rohit = np.array([56, 0, 74, 88, 32, 45, 61, 17, 90, 38])

for name, s in [("Virat", virat), ("Rohit", rohit)]:
    cv = s.std() / s.mean() * 100
    print(name, "fifties:", np.count_nonzero(s >= 50), " ducks:", np.count_nonzero(s == 0), " CV%:", round(float(cv), 1))

import numpy as np

virat = np.array([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])
rohit = np.array([56, 0, 74, 88, 32, 45, 61, 17, 90, 38])

for name, s in [("Virat", virat), ("Rohit", rohit)]:
    cv = s.std() / s.mean() * 100
    print(name, "fifties:", np.count_nonzero(s >= 50), " ducks:", np.count_nonzero(s == 0), " CV%:", round(float(cv), 1))

Practice: describe your own player

import numpy as np

# 🏏 Your turn: replace these scores with your favourite player's last 8 innings
# (use np.nan for a match he missed) and Run to see his summary.
my_player = np.array([12, 54, np.nan, 77, 3, 41, 98, 26])

q1, med, q3 = np.nanpercentile(my_player, [25, 50, 75])
print("count :", np.count_nonzero(~np.isnan(my_player)))
print("mean  :", round(float(np.nanmean(my_player)), 2))
print("std   :", round(float(np.nanstd(my_player, ddof=1)), 2))
print("min   :", np.nanmin(my_player))
print("25%   :", q1)
print("median:", med)
print("75%   :", q3)
print("max   :", np.nanmax(my_player))

import numpy as np

# 🏏 Your turn: replace these scores with your favourite player's last 8 innings
# (use np.nan for a match he missed) and Run to see his summary.
my_player = np.array([12, 54, np.nan, 77, 3, 41, 98, 26])

q1, med, q3 = np.nanpercentile(my_player, [25, 50, 75])
print("count :", np.count_nonzero(~np.isnan(my_player)))
print("mean  :", round(float(np.nanmean(my_player)), 2))
print("std   :", round(float(np.nanstd(my_player, ddof=1)), 2))
print("min   :", np.nanmin(my_player))
print("25%   :", q1)
print("median:", med)
print("75%   :", q3)
print("max   :", np.nanmax(my_player))

Revision Flashcards

1 / 5

Tap a card to flip it and see the answer.

Opening notes

Notes NumPyStatistics & Aggregation

Statistics & AggregationTopic 36 of 775 min read

Describe Your Data

Put every statistic together into a mini describe() function — count, mean, std, min, quartiles, max — and print a per-player summary table using only NumPy.

Definition

A descriptive summary (what Pandas calls describe()) is a small table of statistics that describes a dataset's size, centre, spread and shape: count, mean, std, min, 25%, 50% (median), 75% and max.

Statistic	Tells you	NumPy
count	how many real values	`np.count_nonzero(~np.isnan(a))`
mean	centre (average)	`np.nanmean(a)`
std	spread (consistency)	`np.nanstd(a, ddof=1)` — Pandas uses sample std
min	lowest	`np.nanmin(a)`
25% / 50% / 75%	quartiles (50% = median)	`np.nanpercentile(a, [25, 50, 75])`
max	highest	`np.nanmax(a)`

Using the nan- versions makes the summary safe for missing data, exactly like Pandas.

Syntax — our helper function

python

def describe(a, ddof=1):
    a = np.asarray(a, dtype=float)
    q1, q2, q3 = np.nanpercentile(a, [25, 50, 75])
    return {
        "count": np.count_nonzero(~np.isnan(a)),
        "mean": np.nanmean(a), "std": np.nanstd(a, ddof=ddof),
        "min": np.nanmin(a), "25%": q1, "50%": q2, "75%": q3,
        "max": np.nanmax(a),
    }

Parameter	Meaning	Default
`a`	the data (list or array, NaN allowed)	— required
`ddof`	1 = sample std (like Pandas), 0 = population std (NumPy's default)	`1`

Example — describe one player

python

import numpy as np

def describe(a, ddof=1):
    a = np.asarray(a, dtype=float)
    q1, q2, q3 = np.nanpercentile(a, [25, 50, 75])
    return {
        "count": int(np.count_nonzero(~np.isnan(a))),
        "mean": float(np.nanmean(a)), "std": float(np.nanstd(a, ddof=ddof)),
        "min": float(np.nanmin(a)), "25%": float(q1), "50%": float(q2),
        "75%": float(q3), "max": float(np.nanmax(a)),
    }

virat = [72, 45, 88, 31, 64, 0, 103, 52, 19, 77]
for name, value in describe(virat).items():
    print(f"{name:<6} {value:8.2f}")

text

count     10.00
mean      55.10
std       32.06
min        0.00
25%       34.50
50%       58.00
75%       75.75
max      103.00

Example — a summary table for the whole squad (axis=1)

The same idea works on a 2D table: compute each statistic with axis=1 and you get one row per player — all without a loop over the data.

python

import numpy as np

players = ["Virat", "Rohit", "Hardik", "Shubman"]
scores = np.array([
    [72, 45, 88, 31, 64, 0],
    [56,  0, 74, 88, 32, 45],
    [45, 32, np.nan, 67, 18, 54],
    [33, 61, 45,  0, 77, 83],
])

count = np.count_nonzero(~np.isnan(scores), axis=1)
mean = np.nanmean(scores, axis=1)
std = np.nanstd(scores, axis=1, ddof=1)
q1, med, q3 = np.nanpercentile(scores, [25, 50, 75], axis=1)
low, high = np.nanmin(scores, axis=1), np.nanmax(scores, axis=1)

print(f"{'player':<8}{'count':>6}{'mean':>7}{'std':>7}{'min':>6}{'25%':>7}{'50%':>7}{'75%':>7}{'max':>6}")
for i, p in enumerate(players):
    print(f"{p:<8}{count[i]:>6}{mean[i]:>7.1f}{std[i]:>7.1f}{low[i]:>6.0f}"
          f"{q1[i]:>7.1f}{med[i]:>7.1f}{q3[i]:>7.1f}{high[i]:>6.0f}")

text

player   count   mean    std   min    25%    50%    75%   max
Virat        6   50.0   31.7     0   34.5   54.5   70.0    88
Rohit        6   49.2   31.3     0   35.2   50.5   69.5    88
Hardik       5   43.2   19.0    18   32.0   45.0   54.0    67
Shubman      6   49.8   30.8     0   36.0   53.0   73.0    83

Reading a summary — what to look for

Pattern in the summary	What it suggests
mean much bigger than 50% (median)	a few very big values (right-skew), e.g. a couple of centuries
std large compared to mean	inconsistent player
min = 0	at least one duck
count smaller than the number of matches	missing data (NaN) — check why
75% − 25% (IQR) small	middle innings are tightly grouped

Example — one-line extras: range, coefficient of variation, ducks and fifties

python

import numpy as np

virat = np.array([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])
print("range           :", np.ptp(virat))
print("coeff. of var % :", round(float(virat.std() / virat.mean() * 100), 1))
print("ducks           :", np.count_nonzero(virat == 0))
print("50+ scores      :", np.count_nonzero(virat >= 50))
print("100+ scores     :", np.count_nonzero(virat >= 100))

text

range           : 103
coeff. of var % : 55.2
ducks           : 1
50+ scores      : 6
100+ scores     : 1

Common mistakes

❌ Comparing your NumPy std with Pandas' describe() and finding a mismatch. ✅ describe() uses ddof=1; pass the same.
❌ Using plain mean/min in the helper → one NaN ruins every statistic. ✅ Use the nan-versions.
❌ Printing NumPy 2 scalars inside a list or dict and getting np.float64(55.1). ✅ Print them one by one, or convert with float() / .tolist().
❌ Looking only at the mean. ✅ Read the whole summary — centre, spread and extremes together.

Interview tip

Key points to remember

A descriptive summary shows count, mean, std, min, 25%, 50%, 75% and max — the same as Pandas describe().
Use nan-functions so the summary ignores missing values; count = number of non-NaN values.
Pandas describe() uses the sample std (ddof=1); NumPy's default is ddof=0.
On a 2D table, pass axis=1 to every statistic to get one summary row per player without loops.
np.nanpercentile(a, [25, 50, 75], axis=1) returns one row per requested percentile.
Mean far above the median signals big outliers; std large relative to mean signals inconsistency.
Counting conditions (np.count_nonzero(a >= 50)) is a quick way to add milestones to a summary.

Try it yourself

A mini describe() function

import numpy as np

def describe(a, ddof=1):
    a = np.asarray(a, dtype=float)
    q1, q2, q3 = np.nanpercentile(a, [25, 50, 75])
    stats = {
        "count": np.count_nonzero(~np.isnan(a)),
        "mean": np.nanmean(a), "std": np.nanstd(a, ddof=ddof),
        "min": np.nanmin(a), "25%": q1, "50%": q2, "75%": q3,
        "max": np.nanmax(a),
    }
    for k, v in stats.items():
        print(f"{k:<6} {float(v):8.2f}")

describe([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])

import numpy as np

def describe(a, ddof=1):
    a = np.asarray(a, dtype=float)
    q1, q2, q3 = np.nanpercentile(a, [25, 50, 75])
    stats = {
        "count": np.count_nonzero(~np.isnan(a)),
        "mean": np.nanmean(a), "std": np.nanstd(a, ddof=ddof),
        "min": np.nanmin(a), "25%": q1, "50%": q2, "75%": q3,
        "max": np.nanmax(a),
    }
    for k, v in stats.items():
        print(f"{k:<6} {float(v):8.2f}")

describe([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])

Squad summary table

import numpy as np

players = ["Virat", "Rohit", "Hardik"]
scores = np.array([
    [72, 45, 88, 31, 64, 0],
    [56,  0, 74, 88, 32, 45],
    [45, 32, np.nan, 67, 18, 54],
])

mean = np.nanmean(scores, axis=1)
std = np.nanstd(scores, axis=1, ddof=1)
med = np.nanmedian(scores, axis=1)
for p, m, s, md in zip(players, mean, std, med):
    print(f"{p:<7} mean {m:5.1f}  median {md:5.1f}  std {s:5.1f}")

import numpy as np

players = ["Virat", "Rohit", "Hardik"]
scores = np.array([
    [72, 45, 88, 31, 64, 0],
    [56,  0, 74, 88, 32, 45],
    [45, 32, np.nan, 67, 18, 54],
])

mean = np.nanmean(scores, axis=1)
std = np.nanstd(scores, axis=1, ddof=1)
med = np.nanmedian(scores, axis=1)
for p, m, s, md in zip(players, mean, std, med):
    print(f"{p:<7} mean {m:5.1f}  median {md:5.1f}  std {s:5.1f}")

Milestones and consistency

import numpy as np

virat = np.array([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])
rohit = np.array([56, 0, 74, 88, 32, 45, 61, 17, 90, 38])

for name, s in [("Virat", virat), ("Rohit", rohit)]:
    cv = s.std() / s.mean() * 100
    print(name, "fifties:", np.count_nonzero(s >= 50), " ducks:", np.count_nonzero(s == 0), " CV%:", round(float(cv), 1))

import numpy as np

virat = np.array([72, 45, 88, 31, 64, 0, 103, 52, 19, 77])
rohit = np.array([56, 0, 74, 88, 32, 45, 61, 17, 90, 38])

for name, s in [("Virat", virat), ("Rohit", rohit)]:
    cv = s.std() / s.mean() * 100
    print(name, "fifties:", np.count_nonzero(s >= 50), " ducks:", np.count_nonzero(s == 0), " CV%:", round(float(cv), 1))

Practice: describe your own player

import numpy as np

# 🏏 Your turn: replace these scores with your favourite player's last 8 innings
# (use np.nan for a match he missed) and Run to see his summary.
my_player = np.array([12, 54, np.nan, 77, 3, 41, 98, 26])

q1, med, q3 = np.nanpercentile(my_player, [25, 50, 75])
print("count :", np.count_nonzero(~np.isnan(my_player)))
print("mean  :", round(float(np.nanmean(my_player)), 2))
print("std   :", round(float(np.nanstd(my_player, ddof=1)), 2))
print("min   :", np.nanmin(my_player))
print("25%   :", q1)
print("median:", med)
print("75%   :", q3)
print("max   :", np.nanmax(my_player))

import numpy as np

# 🏏 Your turn: replace these scores with your favourite player's last 8 innings
# (use np.nan for a match he missed) and Run to see his summary.
my_player = np.array([12, 54, np.nan, 77, 3, 41, 98, 26])

q1, med, q3 = np.nanpercentile(my_player, [25, 50, 75])
print("count :", np.count_nonzero(~np.isnan(my_player)))
print("mean  :", round(float(np.nanmean(my_player)), 2))
print("std   :", round(float(np.nanstd(my_player, ddof=1)), 2))
print("min   :", np.nanmin(my_player))
print("25%   :", q1)
print("median:", med)
print("75%   :", q3)
print("max   :", np.nanmax(my_player))

Revision Flashcards

1 / 5

Tap a card to flip it and see the answer.