transform() & filter()

agg() shrinks each group to one row. But many questions need the group value next to every player: "What share of his team's runs did Kohli score?", "Is Rohit above or below MI's average?". For that you need the team total repeated on each row — transform(). Other questions are about whole groups: "Show only teams that scored more than 1000 runs." — that is filter().

Definition

transform(func) applies a function to each group and returns a result with the same index and length as the original data, so each row receives the value computed for its own group.

filter(func) calls a function on each group's sub-DataFrame; the function must return True or False, and the result contains all rows of the groups that returned True (with their original index), dropping the other groups entirely.

• Same length as the original — 8 players in, 8 values out. That is why the result can be stored as a new column with df["new"] = ....
• Its own group's value — Kohli and Faf (both RCB) both receive RCB's total, 1471.
• Whole groups — filter never keeps half a team: either all of MI's rows stay or none do.

agg vs transform vs filter

                  agg("sum")              transform("sum")          filter(sum > 1000)
 team runs       team  runs              runs                      team runs
 RCB  741   -->  CSK    441              1471   (RCB)              RCB  741
 MI   417        GT     890              1037   (MI)               MI   417
 CSK  161        MI    1037               441   (CSK)              MI    15
 GT   890        RCB   1471               890   (GT)               RCB  730
 ...             (1 row per group)        ... (1 value per ROW)    MI   605
                                                                   (rows of passing groups)

Syntax

df.groupby("team")["runs"].transform("sum")             # string name
df.groupby("team")["runs"].transform(lambda s: s - s.mean())
df.groupby("team").filter(lambda g: g["runs"].sum() > 1000)

Example — team total and share on every row

df = pd.DataFrame({
    "player":  ["Kohli", "Rohit", "Dhoni", "Gill", "Jadeja", "Bumrah", "Faf", "Surya"],
    "team":    ["RCB", "MI", "CSK", "GT", "CSK", "MI", "RCB", "MI"],
    "runs":    [741, 417, 161, 890, 280, 15, 730, 605],
})
df["team_total"] = df.groupby("team")["runs"].transform("sum")
df["share_%"] = (df["runs"] / df["team_total"] * 100).round(1)
print(df)

   player team  runs  team_total  share_%
0   Kohli  RCB   741        1471     50.4
1   Rohit   MI   417        1037     40.2
2   Dhoni  CSK   161         441     36.5
3    Gill   GT   890         890    100.0
4  Jadeja  CSK   280         441     63.5
5  Bumrah   MI    15        1037      1.4
6     Faf  RCB   730        1471     49.6
7   Surya   MI   605        1037     58.3

How it works: transform("sum") computes the 4 team totals, then broadcasts each total back to the rows of that team, keeping the original order (0–7). Because the result lines up with df's index, plain column arithmetic works.

Example — compare each player with his team average

d2 = df[["player", "team", "runs"]].copy()
d2["team_avg"] = d2.groupby("team")["runs"].transform("mean").round(1)
d2["vs_avg"] = (d2["runs"] - d2["team_avg"]).round(1)
print(d2)

   player team  runs  team_avg  vs_avg
0   Kohli  RCB   741     735.5     5.5
1   Rohit   MI   417     345.7    71.3
2   Dhoni  CSK   161     220.5   -59.5
3    Gill   GT   890     890.0     0.0
4  Jadeja  CSK   280     220.5    59.5
5  Bumrah   MI    15     345.7  -330.7
6     Faf  RCB   730     735.5    -5.5
7   Surya   MI   605     345.7   259.3

The same vs_avg column can be made in one step with a lambda: df.groupby("team")["runs"].transform(lambda s: s - s.mean()) — here the function returns a whole Series (one value per row), which transform also accepts.

Example — rank inside each team and fill missing values per group

d3 = df[["player", "team", "runs"]].copy()
d3["rank_in_team"] = d3.groupby("team")["runs"].rank(ascending=False).astype(int)
print(d3.sort_values(["team", "rank_in_team"]))

   player team  runs  rank_in_team
4  Jadeja  CSK   280             1
2   Dhoni  CSK   161             2
3    Gill   GT   890             1
7   Surya   MI   605             1
1   Rohit   MI   417             2
5  Bumrah   MI    15             3
0   Kohli  RCB   741             1
6     Faf  RCB   730             2

A very common cleaning trick (video 12 data cleaning) is to fill a missing score with its own team's mean rather than the overall mean:

import numpy as np
m = pd.DataFrame({"team": ["MI", "MI", "MI", "CSK", "CSK"], "runs": [40, np.nan, 60, 90, np.nan]})
m["runs_filled"] = m["runs"].fillna(m.groupby("team")["runs"].transform("mean"))
print(m)

  team  runs  runs_filled
0   MI  40.0         40.0
1   MI   NaN         50.0
2   MI  60.0         60.0
3  CSK  90.0         90.0
4  CSK   NaN         90.0

Running totals per group work the same way: df.groupby("player")["runs"].cumsum() gives each player's career runs match by match.

Example — filter(): keep whole groups

d = df[["player", "team", "runs"]]
print(d.groupby("team").filter(lambda g: g["runs"].sum() > 1000))

   player team  runs
0   Kohli  RCB   741
1   Rohit   MI   417
5  Bumrah   MI    15
6     Faf  RCB   730
7   Surya   MI   605

How it works: the lambda receives each team's sub-DataFrame g. RCB (1471) and MI (1037) return True, so all their rows are kept — even Bumrah's 15. CSK (441) and GT (890) return False and disappear. The original index is preserved. Another typical condition is group size: filter(lambda g: len(g) >= 2) drops teams with only one player.

Compare with a normal row filter d[d["runs"] > 600], which tests each row on its own and keeps Gill but drops Bumrah. The same group-level filter can also be written with a transformed mask: d[d.groupby("team")["runs"].transform("sum") > 1000] — identical result.

Common mistakes

• ❌ df["team_total"] = df.groupby("team")["runs"].sum() → mostly NaN (index is team names, not 0–7). ✅ Use .transform("sum").
• ❌ A filter function that returns a Series of booleans. ✅ It must return a single True/False per group — use g["runs"].sum() > 1000, not g["runs"] > 100.
• ❌ Expecting filter to pick rows inside a group. ✅ It keeps or drops whole groups; for rows use boolean indexing.
• ❌ Filling missing values with the overall mean when groups differ a lot. ✅ fillna(groupby(...).transform("mean")) fills per group.

Interview tip

Q: What is the difference between agg and transform on a GroupBy? A: agg returns one row per group (a reduced summary indexed by the keys). transform returns an object the same length as the original, with each row holding its group's value, so it can be assigned directly as a new column.