Handling Missing Data
Real data has gaps; pandas marks them NaN and gives tools to find and fill them.
Syntax
df.fillna(df['col'].mean())Missing values show up as NaN (Not a Number). Ignoring them breaks models, so every project handles them.
Detect, drop or fill
df.isna().sum()counts missing values per column.df.dropna()removes rows (or columns) with gaps.df.fillna(value)replaces gaps — often with the column mean or median.
Example
import pandas as pd
import numpy as np
df = pd.DataFrame({'age': [34, np.nan, 41, np.nan]})
print(df.isna().sum()) # age 2
# Fill missing ages with the column mean
df['age'] = df['age'].fillna(df['age'].mean())
print(df['age'].tolist()) # [34.0, 37.5, 41.0, 37.5]When to use it
- A data scientist calls df.isnull().sum() to audit every column for missing values as the first step in data quality assessment.
- An ML engineer fills missing numeric features with each column's median using fillna to avoid the mean being skewed by outliers.
- A data pipeline drops rows where the target label is NaN before training, since imputing the label itself would corrupt the model.
More examples
Detect missing values
Uses isnull().sum() to count NaN values per column and converts to a percentage to assess how severe the missingness is.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'age': [25, np.nan, 47, np.nan],
'salary': [50000, 62000, np.nan, 48000]
})
print(df.isnull().sum())
print(df.isnull().mean() * 100) # % missing per columnFill missing values
Imputes numeric NaN with the column median, categorical NaN with a placeholder, and time-series gaps with forward fill.
import pandas as pd
df = pd.read_csv('patients.csv')
df['age'] = df['age'].fillna(df['age'].median())
df['gender'] = df['gender'].fillna('Unknown')
df['score'] = df['score'].ffill() # forward fill time-series
print(df.isnull().sum())Drop rows or columns with too many NaN
Removes columns with too high a missing rate and drops any row whose label is NaN, leaving only rows with valid training targets.
import pandas as pd
df = pd.read_csv('survey.csv')
# Drop columns missing more than 40% of values
threshold = 0.4
df = df.dropna(axis=1, thresh=int(len(df) * (1 - threshold)))
# Drop rows where the target label is missing
df = df.dropna(subset=['label'])
print(df.shape)
Discussion