Cleaning Data
Real data is messy; cleaning fixes types, duplicates and inconsistent values.
Models are only as good as their data. Cleaning is usually the biggest part of a project.
Common cleaning steps
- Fix wrong types (numbers stored as text).
- Remove duplicate rows with
drop_duplicates. - Standardise text: trim spaces, fix casing.
- Handle missing values (see the pandas chapter).
Example
import pandas as pd
df = pd.DataFrame({
'name': [' Ann ', 'Bob', 'Bob'],
'price': ['10', '20', '20'],
})
df = df.drop_duplicates()
df['name'] = df['name'].str.strip()
df['price'] = df['price'].astype(float)
print(df.dtypes) # price is now float64
print(df)When to use it
- A data engineer removes 3,000 duplicate order rows from a transaction DataFrame before aggregating daily totals.
- An ML pipeline corrects mixed data types in an age column (strings mixed with integers) by casting to numeric and coercing errors to NaN.
- A data scientist strips leading/trailing whitespace and lowercases a city column before joining it with a reference lookup table.
More examples
Drop duplicates
Detects and removes duplicate rows using drop_duplicates, reducing the DataFrame from 4 to 3 rows.
import pandas as pd
df = pd.DataFrame({'id': [1,2,2,3], 'val': ['a','b','b','c']})
print('Before:', len(df))
df = df.drop_duplicates()
print('After:', len(df))Fix dtypes and coerce bad values
Converts an age column containing string integers and garbage values to float, coercing unrecognisable values to NaN.
import pandas as pd
df = pd.DataFrame({'age': ['25', '34', 'unknown', '45', '?']})
df['age'] = pd.to_numeric(df['age'], errors='coerce')
print(df)
print(df['age'].dtype)Normalise text and strip whitespace
Strips surrounding whitespace and applies title casing to a city column, making values consistent for reliable joins and groupby operations.
import pandas as pd
df = pd.DataFrame({'city': [' New York', 'chicago ', 'LOS ANGELES', ' london']})
df['city'] = df['city'].str.strip().str.title()
print(df)
Discussion