Pandas: Handling Missing Values
Last updated: 2026-08-26
Missing values (NaN) are the number-one enemy of data cleaning — real-world data is never perfect. A sensor failure leaves temperature readings missing, a user skips an optional field and age goes missing, a system merge wipes out IDs. Pandas provides a complete toolchain for detecting, dropping, filling, and interpolating missing values. This section walks you through each one and explains how to operate safely under Copy-on-Write.
1. What You'll Learn
- ❶ The nature and semantics of NaN
- ❷ Detection with isnull / notnull
- ❸ Dropping missing rows/columns with dropna
- ❹ Filling missing values with fillna
- ❺ Interpolation methods with interpolate
2. The Gap in Carol's Workout Data
(1) The Pain Point: 30% of the Heart-Rate Data Is Missing
Carol's fitness tracker recorded 30 days of heart-rate data, but the signal dropped out (NaN) on 9 of those days during her workouts:
import pandas as pd
import numpy as np
hr = pd.DataFrame({
'date': pd.date_range('2024-01-01', periods=30),
'heart_rate': [72, 75, np.nan, 78, np.nan, 80, 82, np.nan,
85, 88, np.nan, 90, 87, np.nan, 83, 80,
np.nan, 78, 76, np.nan, 74, 72, np.nan, 70,
68, np.nan, 65, 63, np.nan, 60]
})
print(f"Missing: {hr['heart_rate'].isnull().sum()} / {len(hr)}")
# Missing: 9 / 30
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(2) The Solution: Four Strategies, Each with Its Own Use Case
| Strategy | Method | When to Use |
|---|---|---|
| Detect | isnull / notnull | Understand the missing-value distribution |
| Drop | dropna | Few missing values, rows aren't important |
| Fill | fillna | A reasonable default value exists |
| Interpolate | interpolate | Time-series data with a clear trend |
3. The Nature of NaN
(1) NaN Is Not Equal to NaN
▶ Example: The Special Behavior of NaN (Difficulty ⭐)
import numpy as np
# NaN is NOT equal to itself — this is IEEE 754 standard
print(np.nan == np.nan) # False!
print(np.nan != np.nan) # True!
# Use np.isnan() to detect NaN
print(np.isnan(np.nan)) # True
# Pandas handles this internally — isnull() works correctly
import pandas as pd
s = pd.Series([1, np.nan, 3, None, 5])
print(s.isnull())
# 0 False
# 1 True
# 2 False
# 3 True
# 4 False
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(2) Missing-Value Types in Pandas
| Type | Missing Representation | Example |
|---|---|---|
| float64 | np.nan | Default floating-point missing value |
| int64 | Cannot represent NaN natively → auto-converts to float64 | [1, None, 3] → float64 |
| object | None / np.nan | Mixed-type columns |
| string | pd.NA | Pandas 1.x+ nullable string |
| Int64 | pd.NA | nullable integer (capital I) |
| datetime64 | NaT | Missing time value |
Int64, boolean, and string) use pd.NA instead of np.nan, solving the problem that integer columns can't represent missing values. Convert with: df['col'].astype('Int64').
4. Detection with isnull / notnull
(1) Basic Detection
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Missing-Value Detection (Difficulty ⭐)
import pandas as pd
import numpy as np
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David'],
'age': [28, np.nan, 25, 30, np.nan],
'salary': [75000, 92000, np.nan, 88000, 105000],
'department': ['Sales', 'Engineering', np.nan, 'Sales', 'Management']
})
# Per-element detection
print(df.isnull())
# name age salary department
# 0 False False False False
# 1 False True False False
# 2 False False True True
# 3 False False False False
# 4 False True False False
# Per-column count
print(df.isnull().sum())
# name 0
# age 2
# salary 1
# department 1
# Total missing
print(f"Total missing: {df.isnull().sum().sum()}") # 4
# Not-null (inverse)
print(df.notnull().sum()) # count of non-missing per column
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(2) Visualizing and Analyzing Missing Values
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Missing-Pattern Analysis (Difficulty ⭐⭐)
import pandas as pd
import numpy as np
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David', 'Eve'],
'age': [28, np.nan, 25, 30, np.nan, 27],
'salary': [75000, 92000, np.nan, 88000, 105000, np.nan],
'department': ['Sales', 'Engineering', np.nan, 'Sales', 'Management', 'HR']
})
# Which rows have ANY missing?
rows_with_missing = df[df.isnull().any(axis=1)]
print(f"Rows with missing values: {len(rows_with_missing)}")
# 3 rows (Bob, Charlie, David)
# Which columns have missing?
cols_with_missing = df.columns[df.isnull().any()].tolist()
print(f"Columns with missing: {cols_with_missing}")
# ['age', 'salary', 'department']
# Missing percentage per column
missing_pct = (df.isnull().sum() / len(df) * 100).round(1)
print(missing_pct[missing_pct > 0])
# age 33.3%
# salary 33.3%
# department 16.7%
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
5. Dropping Missing Values with dropna
(1) Basic Dropping
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Dropping with dropna (Difficulty ⭐)
import pandas as pd
import numpy as np
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David'],
'age': [28, np.nan, 25, 30, np.nan],
'salary': [75000, 92000, np.nan, 88000, 105000]
})
# Drop rows with ANY missing (default)
print(df.dropna())
# name age salary
# 0 Alice 28.0 75000.0
# 3 Carol 30.0 88000.0
# Drop rows where ALL values are missing
print(df.dropna(how='all'))
# Drop columns with any missing
print(df.dropna(axis=1))
# name
# 0 Alice
# 1 Bob
# ...
# Drop rows with threshold (keep rows with >= N non-missing)
print(df.dropna(thresh=2)) # at least 2 non-NaN values
# name age salary
# 0 Alice 28.0 75000.0
# 1 Bob NaN 92000.0
# 3 Carol 30.0 88000.0
# 4 David NaN 105000.0
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(2) dropna Parameters in Detail
| Parameter | Default | Description |
|---|---|---|
| axis | 0 | 0 = drop rows, 1 = drop columns |
| how | 'any' | 'any' = drop if any NaN, 'all' = drop only if all NaN |
| thresh | None | Keep rows with at least N non-missing values |
| subset | None | Check only the specified columns |
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Column-Specific Checking with dropna (Difficulty ⭐⭐)
import pandas as pd
import numpy as np
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David'],
'age': [28, np.nan, 25, 30, np.nan],
'salary': [75000, 92000, np.nan, 88000, 105000],
'department': ['Sales', 'Engineering', 'Marketing', 'Sales', 'Management']
})
# Only drop rows where 'salary' is missing
# (keep rows with missing 'age' — we can impute age)
print(df.dropna(subset=['salary']))
# name age salary department
# 0 Alice 28.0 75000.0 Sales
# 1 Bob NaN 92000.0 Engineering
# 3 Carol 30.0 88000.0 Sales
# 4 David NaN 105000.0 Management
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
6. Filling Missing Values with fillna
(1) Filling with a Constant
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Constant and Statistical Filling with fillna (Difficulty ⭐⭐)
import pandas as pd
import numpy as np
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David'],
'age': [28, np.nan, 25, 30, np.nan],
'salary': [75000, 92000, np.nan, 88000, 105000]
})
# Fill with constant
df_const = df.fillna({'age': 0, 'salary': 0})
print(df_const)
# Fill with mean (most common for numeric)
df_mean = df.copy()
df_mean['age'] = df_mean['age'].fillna(df_mean['age'].mean())
df_mean['salary'] = df_mean['salary'].fillna(df_mean['salary'].median())
print(df_mean)
# age: filled with 27.67, salary: filled with 88000.0
# Fill with forward fill (use previous value)
df_ffill = df.fillna(method='ffill')
print(df_ffill)
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
fillna(method=...) is deprecated — use df.ffill() / df.bfill() instead.
(2) Forward / Backward Fill
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: ffill and bfill (Difficulty ⭐)
import pandas as pd
import numpy as np
# Time-series data — ffill makes sense here
ts = pd.DataFrame({
'date': pd.date_range('2024-01-01', periods=8),
'temperature': [22.5, np.nan, np.nan, 23.1, np.nan, 24.0, np.nan, 24.5]
})
# Forward fill (propagate last valid value)
print(ts.assign(temperature_ffill=ts['temperature'].ffill()))
# NaN → use previous non-NaN value
# Backward fill (propagate next valid value)
print(ts.assign(temperature_bfill=ts['temperature'].bfill()))
# NaN → use next non-NaN value
# Limit: only fill N consecutive NaN
print(ts.assign(temperature_limited=ts['temperature'].ffill(limit=1)))
# Only fill 1 consecutive NaN, leave the rest
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
7. Interpolation with interpolate
(1) Linear Interpolation
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Linear Interpolation with interpolate (Difficulty ⭐⭐)
import pandas as pd
import numpy as np
# Carol's heart rate data — linear interpolation is natural
hr = pd.DataFrame({
'time_min': list(range(10)),
'heart_rate': [72, 75, np.nan, 78, np.nan, 80, 82, np.nan, 85, 88]
})
# Linear interpolation (default)
hr['hr_linear'] = hr['heart_rate'].interpolate()
print(hr[['time_min', 'heart_rate', 'hr_linear']])
# time 2: (75+78)/2 = 76.5
# time 4: (78+80)/2 = 79.0
# time 7: (82+85)/2 = 83.5
# Index-based vs time-based interpolation
hr_ts = pd.DataFrame({
'heart_rate': [72, 75, np.nan, 78, np.nan, 80]
}, index=pd.to_timedelta([0, 5, 15, 20, 40, 45], unit='min'))
# Time-weighted interpolation
hr_ts['hr_time'] = hr_ts['heart_rate'].interpolate(method='time')
print(hr_ts)
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(2) Comparing Interpolation Methods
| Method | method | When to Use |
|---|---|---|
| Linear | 'linear' | Evenly sampled data, smooth trends |
| Time-weighted | 'time' | Uneven time intervals |
| Index-weighted | 'index' | Uneven index spacing |
| Quadratic | 'quadratic' | Curved trends |
| Cubic | 'cubic' | Smooth curves |
| Nearest | 'nearest' | Discrete / step data |
| Spline | 'spline' | Smooth interpolation (requires the order parameter) |
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Comparing Interpolation Methods (Difficulty ⭐⭐)
import pandas as pd
import numpy as np
s = pd.Series([0.0, np.nan, np.nan, np.nan, np.nan, 5.0, 10.0])
print("linear: ", s.interpolate(method='linear').tolist())
# [0.0, ~1.67, ~3.33, 5.0, ~6.67, 5.0, 10.0]
# But this includes NaN boundaries.
print("linear (trim):", s.dropna().interpolate(method='linear').tolist())
# Demonstrates linear interpolation only.
# quadratic needs 1 non-NaN value on each side of the boundary; here we use 7 data points
print("quadratic:", s.interpolate(method='quadratic').tolist())
# Fits a quadratic curve; requires at least 3 non-NaN points
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
8. Copy-on-Write and Missing-Value Operations
(1) Writing Safely Under CoW
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Safe Operations Under CoW (Difficulty ⭐⭐)
import pandas as pd
import numpy as np
pd.options.mode.copy_on_write = True # Pandas 3.x default
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'age': [28, np.nan, 25],
'salary': [75000, 92000, np.nan]
})
# ✅ Safe: assign result back
df['age'] = df['age'].fillna(df['age'].mean())
# ✅ Safe: use inplace (works with CoW)
df.fillna({'salary': 0}, inplace=True)
# ❌ Dangerous (pre-CoW): chained assignment
# subset = df[df['age'].notnull()]
# subset['salary'] = 100 # May not modify df!
# ✅ Safe with CoW: loc
df.loc[df['age'].notnull(), 'salary'] = df.loc[df['age'].notnull(), 'salary'] * 1.1
print(df)
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
9. Complete Example: The Full Missing-Value Workflow
(5) ▶ Missing-Value Handling Decision Tree
graph TB
A[Detect missing values] --> B{Missing rate?}
B -->|<5%| C[Drop with dropna]
B -->|5-30%| D{Data type?}
D -->|Numeric| E[fillna with mean/median]
D -->|Categorical| F[fillna with mode]
D -->|Time-series| G[interpolate]
B -->|>50%| H[Consider dropping the column]
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
: Full Missing-Value Cleaning Workflow (Difficulty ⭐⭐⭐)
import pandas as pd
import numpy as np
# ============================================
# Comprehensive example: Full missing-value
# handling pipeline
# ============================================
# 1. Load data with missing values
np.random.seed(42)
df = pd.DataFrame({
'customer_id': range(1, 21),
'name': [f'Customer_{i}' for i in range(1, 21)],
'age': [25, np.nan, 35, 40, np.nan, 28, np.nan, 50, 33, np.nan,
29, np.nan, 45, 38, np.nan, 27, np.nan, 42, 31, np.nan],
'purchase_amount': [120, 85, np.nan, 200, 150, np.nan, 90, 310,
np.nan, 60, 175, np.nan, 250, 140, np.nan,
95, 180, np.nan, 110, 75],
'category': ['Electronics', np.nan, 'Home', 'Electronics', 'Clothing',
'Home', np.nan, 'Electronics', 'Clothing', np.nan',
'Home', 'Electronics', 'Clothing', np.nan, 'Home',
'Electronics', np.nan, 'Home', 'Clothing', 'Electronics']
})
# 2. Analyze missing pattern
print("=== Missing Analysis ===")
missing_report = pd.DataFrame({
'count': df.isnull().sum(),
'pct': (df.isnull().sum() / len(df) * 100).round(1)
})
print(missing_report[missing_report['count'] > 0])
# 3. Drop rows where critical fields are missing
df = df.dropna(subset=['customer_id', 'name'])
# 4. Fill numeric columns
df['age'] = df['age'].fillna(df['age'].median())
df['purchase_amount'] = df['purchase_amount'].interpolate(method='linear')
# 5. Fill categorical column
df['category'] = df['category'].fillna(df['category'].mode()[0])
# 6. Verify: no more missing
print(f"\n=== After Cleaning ===")
print(f"Remaining missing: {df.isnull().sum().sum()}")
print(df.head(10))
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
❓ FAQ
df = df.fillna(...), which is clearer semantically. CoW guarantees that both approaches produce the same result.df['col'].isnull().any() returns True/False. To count: df['col'].isnull().sum() returns the number of missing values. For the percentage: df['col'].isnull().mean() * 100. To check the whole table: df.isnull().any().any() tells you whether the DataFrame has any NaN at all.df['col'].astype('Int64') (capital I), which represents missing values with pd.NA while preserving integer semantics.📖 Summary
- NaN is not equal to itself (IEEE 754), so you must detect it with isnull() — you can't use ==
- isnull / notnull detect missing values; .sum() counts them, and .any() gives a quick check
- dropna drops missing rows/columns: how='any'/'all', the thresh threshold, and subset for column-specific checking
- fillna fills values: constant / mean / median / mode; in Pandas 2.x use ffill()/bfill() instead of method
- interpolate fills by interpolation: linear is the default, time weights by time, and quadratic/cubic fit curves
- Safe operations under Copy-on-Write: assign back to a variable or use inplace
- The workflow: detect and analyze → drop what can't be repaired → fill/interpolate → verify zero missing values
📝 Exercises
- Basic (Difficulty ⭐): Create a DataFrame containing NaN. Use isnull().sum() to count the missing values in each column, drop the missing rows with dropna(), and compare the change in row count.
- Intermediate (Difficulty ⭐⭐): Create a time-series DataFrame (10 rows with 3 NaN values). Fill it with ffill() / bfill() / interpolate(method='linear') respectively, and compare the differences among the three results.
- Challenge (Difficulty ⭐⭐⭐): Simulate 50 rows of customer data (age/salary/category each 15% missing) and complete the full cleaning workflow: missing analysis → dropna(subset=[key columns]) → fill age with the mean → interpolate salary → fill category with the mode → verify zero missing values.