Inspecting Data
The first five commands on any new dataset: head, tail, sample, info and describe (including include='all'), plus memory_usage.
Whenever you load a new dataset — an IPL ball-by-ball file, a car-price CSV, anything — you should inspect it before doing any analysis. Data scientists run the same handful of commands every single time: look at a few rows, check the columns and types, count missing values, and see the basic statistics. Skipping this step is how wrong conclusions happen.
Definition
Inspecting a DataFrame means examining its structure and contents without changing it.
head/tail/sampleshow rows;infoshows the structure (columns, non-null counts, dtypes, memory);describeshows summary statistics;memory_usageshows bytes per column.
- Without changing it — none of these methods modify
df. - Structure vs statistics —
infoanswers "what is in this table?",describeanswers "what do the numbers look like?".
Syntax
python
df.head(n=5) df.tail(n=5) df.sample(n=None, frac=None, random_state=None)
df.info(verbose=None, memory_usage=None, show_counts=None)
df.describe(percentiles=None, include=None, exclude=None)
df.memory_usage(index=True, deep=False)| Method / parameter | Meaning | Default |
|---|---|---|
head(n) / tail(n) | first / last n rows (head(-2) = all rows except the last 2) | 5 |
sample(n) / sample(frac=...) | random rows / random fraction of rows | 1 row |
random_state | seed for repeatable sampling | None |
info(memory_usage="deep") | measure the real memory of text columns | estimate |
describe(include=...) | "all", "number", "str", "category", or a list of dtypes | numeric columns only |
describe(exclude=...) | dtypes to leave out | None |
describe(percentiles=[...]) | which percentiles to show | [.25, .5, .75] |
memory_usage(deep=True) | include the real size of text values | False |
Example — head, tail, sample
python
print(df.head(3))
print(df.tail(2))
print(df.sample(3, random_state=7))text
player team matches runs strike_rate
0 Shubman Gill GT 17 890 157.8
1 Faf du Plessis RCB 14 730 153.7
2 Devon Conway CSK 16 672 139.7
player team matches runs strike_rate
6 Ruturaj Gaikwad CSK 16 590 147.5
7 Rinku Singh KKR 14 474 149.5
player team matches runs strike_rate
2 Devon Conway CSK 16 672 139.7
5 Suryakumar Yadav MI 16 605 181.1
0 Shubman Gill GT 17 890 157.8How it works: head shows the top of the file (are the column names right? did the header load?), tail the bottom (is there a "Total" row or junk at the end?), and sample random rows from the middle — files are often sorted, so the first rows can be misleading.
Example — info()
python
df.info()text
<class 'pandas.DataFrame'>
RangeIndex: 8 entries, 0 to 7
Data columns (total 5 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 player 8 non-null str
1 team 8 non-null str
2 matches 8 non-null int64
3 runs 8 non-null int64
4 strike_rate 8 non-null float64
dtypes: float64(1), int64(2), str(2)
memory usage: 452.0 bytes| Line | Tells you |
|---|---|
RangeIndex: 8 entries, 0 to 7 | number of rows and the index type |
Data columns (total 5 columns) | number of columns |
Non-Null Count | how many values are present in each column → compare with the row count to spot missing data |
Dtype | the type of each column — numbers stored as str show up here |
dtypes: ... | how many columns of each type |
memory usage | approximate memory used |
info() prints its report and returns None. Here is the same check on a table with missing values:
text
player team matches runs strike_rate
0 Shubman Gill GT 17 890 157.8
1 Faf du Plessis RCB 14 730 NaN
2 Devon Conway CSK 16 672 139.7
3 Virat Kohli NaN 14 639 139.8text
<class 'pandas.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 5 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 player 4 non-null str
1 team 3 non-null str
2 matches 4 non-null int64
3 runs 4 non-null int64
4 strike_rate 3 non-null float64
dtypes: float64(1), int64(2), str(2)
memory usage: 292.0 bytesteam and strike_rate show 3 non-null out of 4 entries → each has one missing value.
Example — describe()
python
print(df.describe())text
matches runs strike_rate
count 8.000000 8.000000 8.000000
mean 15.125000 653.125000 154.087500
std 1.246423 120.565853 13.678287
min 14.000000 474.000000 139.700000
25% 14.000000 601.250000 145.575000
50% 15.000000 632.000000 151.600000
75% 16.000000 686.500000 159.250000
max 17.000000 890.000000 181.100000How it works: by default describe summarises only numeric columns (player and team are skipped). For each: count (non-missing), mean, std (spread), min, the 25th/50th(median)/75th percentiles, and max. Reading it: the median run tally (632) is below the mean (653) because one big value (Gill's 890) pulls the mean up. Add .round(1) for a cleaner view, or .T to flip it when there are many columns.
Example — describe(include="all")
python
print(df.describe(include="all"))text
player team matches runs strike_rate
count 8 8 8.000000 8.000000 8.000000
unique 8 6 NaN NaN NaN
top Shubman Gill RCB NaN NaN NaN
freq 1 2 NaN NaN NaN
mean NaN NaN 15.125000 653.125000 154.087500
std NaN NaN 1.246423 120.565853 13.678287
min NaN NaN 14.000000 474.000000 139.700000
25% NaN NaN 14.000000 601.250000 145.575000
50% NaN NaN 15.000000 632.000000 151.600000
75% NaN NaN 16.000000 686.500000 159.250000
max NaN NaN 17.000000 890.000000 181.100000How it works: include="all" adds the text columns. Text columns get count, unique (distinct values), top (most frequent) and freq (its count); numeric columns get the usual stats. Statistics that do not apply show NaN. To see only the text columns use df.describe(include="str") (or exclude="number").
Note (pandas 3): old tutorials write
describe(include="object")for text columns. In pandas 3 text isstr, so useinclude="str"— the"object"form still works for now but raises a deprecation warning. Also, custompercentiles=[0.1, 0.9]now shows exactly those percentiles (the 50% row is no longer added automatically).
Example — memory_usage
python
print(df.memory_usage(deep=True))text
Index 132
player 499
team 413
matches 64
runs 64
strike_rate 64
dtype: int64How it works: each numeric column uses 8 rows × 8 bytes = 64 bytes. Text columns are larger and their true size is only measured with deep=True (without it, pandas shows a rough estimate). df.memory_usage(deep=True).sum() gives the total in bytes — useful for spotting columns to convert to category on big datasets.
The inspection checklist
text
1. df.shape how big is it?
2. df.head() / df.sample() does it look right?
3. df.info() column types + missing values
4. df.describe() ranges, outliers, suspicious min/max
5. df.isna().sum() missing values per column (data-cleaning chapter)
6. df["col"].value_counts() categories in a text columnCommon mistakes
- ❌
print(df.info())→ prints the report and thenNone. ✅ Just calldf.info(). - ❌ Assuming
describe()covers all columns. ✅ It is numeric-only by default; useinclude="all". - ❌ Judging a sorted dataset by
head()alone. ✅ Also checktail()andsample(). - ❌ Ignoring a numeric-looking column with dtype
strininfo(). ✅ Convert it (see "Data Types & astype").
Interview tip
Q: You receive a new CSV. What are the first things you run?
A: df.shape, df.head() and df.sample() to see the data, df.info() for column types and non-null counts, df.describe(include="all") for ranges and categories, and df.isna().sum() to quantify missing values — before any cleaning or analysis.
Key points to remember
- head(n)/tail(n) show the first/last rows; sample(n, random_state=…) shows random rows.
- info() prints row count, column names, non-null counts, dtypes and memory, and returns None.
- Compare each column's non-null count with the row count to spot missing values.
- describe() summarises numeric columns: count, mean, std, min, 25%, 50%, 75%, max.
- describe(include='all') adds text columns with count, unique, top and freq.
- In pandas 3 use include='str' (not 'object') to describe text columns.
- memory_usage(deep=True) shows the real bytes used by each column.
Try it yourself
First look: head, tail, sample, info
describe: numbers only vs include='all'
Spot missing values and memory
Practice: inspect a bowling table
Revision Flashcards
Tap a card to flip it and see the answer.
