Pandas: Handling Missing Values

Last updated: 2026-08-26

Missing values (NaN) are the number-one enemy of data cleaning — real-world data is never perfect. A sensor failure leaves temperature readings missing, a user skips an optional field and age goes missing, a system merge wipes out IDs. Pandas provides a complete toolchain for detecting, dropping, filling, and interpolating missing values. This section walks you through each one and explains how to operate safely under Copy-on-Write.

⚠️ Note: The code below must be run in a local Python environment.

1. What You'll Learn


2. The Gap in Carol's Workout Data

(1) The Pain Point: 30% of the Heart-Rate Data Is Missing

Carol's fitness tracker recorded 30 days of heart-rate data, but the signal dropped out (NaN) on 9 of those days during her workouts:

PYTHON
import pandas as pd
import numpy as np

hr = pd.DataFrame({
    'date': pd.date_range('2024-01-01', periods=30),
    'heart_rate': [72, 75, np.nan, 78, np.nan, 80, 82, np.nan,
                   85, 88, np.nan, 90, 87, np.nan, 83, 80,
                   np.nan, 78, 76, np.nan, 74, 72, np.nan, 70,
                   68, np.nan, 65, 63, np.nan, 60]
})
print(f"Missing: {hr['heart_rate'].isnull().sum()} / {len(hr)}")
# Missing: 9 / 30
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(2) The Solution: Four Strategies, Each with Its Own Use Case

Strategy Method When to Use
Detect isnull / notnull Understand the missing-value distribution
Drop dropna Few missing values, rows aren't important
Fill fillna A reasonable default value exists
Interpolate interpolate Time-series data with a clear trend

3. The Nature of NaN

(1) NaN Is Not Equal to NaN

▶ Example: The Special Behavior of NaN (Difficulty ⭐)

PYTHON
import numpy as np

# NaN is NOT equal to itself — this is IEEE 754 standard
print(np.nan == np.nan)   # False!
print(np.nan != np.nan)   # True!

# Use np.isnan() to detect NaN
print(np.isnan(np.nan))   # True

# Pandas handles this internally — isnull() works correctly
import pandas as pd
s = pd.Series([1, np.nan, 3, None, 5])
print(s.isnull())
# 0    False
# 1     True
# 2    False
# 3     True
# 4    False
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(2) Missing-Value Types in Pandas

Type Missing Representation Example
float64 np.nan Default floating-point missing value
int64 Cannot represent NaN natively → auto-converts to float64 [1, None, 3] → float64
object None / np.nan Mixed-type columns
string pd.NA Pandas 1.x+ nullable string
Int64 pd.NA nullable integer (capital I)
datetime64 NaT Missing time value
💡 Tip: The Nullable dtypes in Pandas 2.x (capitalized, such as Int64, boolean, and string) use pd.NA instead of np.nan, solving the problem that integer columns can't represent missing values. Convert with: df['col'].astype('Int64').


4. Detection with isnull / notnull

(1) Basic Detection

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Missing-Value Detection (Difficulty ⭐)

PYTHON
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David'],
    'age': [28, np.nan, 25, 30, np.nan],
    'salary': [75000, 92000, np.nan, 88000, 105000],
    'department': ['Sales', 'Engineering', np.nan, 'Sales', 'Management']
})

# Per-element detection
print(df.isnull())
#     name    age  salary  department
# 0  False  False   False       False
# 1  False   True   False       False
# 2  False  False    True        True
# 3  False  False   False       False
# 4  False   True   False       False

# Per-column count
print(df.isnull().sum())
# name          0
# age           2
# salary        1
# department    1

# Total missing
print(f"Total missing: {df.isnull().sum().sum()}")  # 4

# Not-null (inverse)
print(df.notnull().sum())  # count of non-missing per column
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(2) Visualizing and Analyzing Missing Values

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Missing-Pattern Analysis (Difficulty ⭐⭐)

PYTHON
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David', 'Eve'],
    'age': [28, np.nan, 25, 30, np.nan, 27],
    'salary': [75000, 92000, np.nan, 88000, 105000, np.nan],
    'department': ['Sales', 'Engineering', np.nan, 'Sales', 'Management', 'HR']
})

# Which rows have ANY missing?
rows_with_missing = df[df.isnull().any(axis=1)]
print(f"Rows with missing values: {len(rows_with_missing)}")
# 3 rows (Bob, Charlie, David)

# Which columns have missing?
cols_with_missing = df.columns[df.isnull().any()].tolist()
print(f"Columns with missing: {cols_with_missing}")
# ['age', 'salary', 'department']

# Missing percentage per column
missing_pct = (df.isnull().sum() / len(df) * 100).round(1)
print(missing_pct[missing_pct > 0])
# age          33.3%
# salary       33.3%
# department   16.7%
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

5. Dropping Missing Values with dropna

(1) Basic Dropping

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Dropping with dropna (Difficulty ⭐)

PYTHON
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David'],
    'age': [28, np.nan, 25, 30, np.nan],
    'salary': [75000, 92000, np.nan, 88000, 105000]
})

# Drop rows with ANY missing (default)
print(df.dropna())
#     name   age    salary
# 0  Alice  28.0   75000.0
# 3  Carol  30.0   88000.0

# Drop rows where ALL values are missing
print(df.dropna(how='all'))

# Drop columns with any missing
print(df.dropna(axis=1))
#      name
# 0   Alice
# 1     Bob
# ...

# Drop rows with threshold (keep rows with >= N non-missing)
print(df.dropna(thresh=2))  # at least 2 non-NaN values
#      name   age    salary
# 0   Alice  28.0   75000.0
# 1     Bob   NaN   92000.0
# 3   Carol  30.0   88000.0
# 4   David   NaN  105000.0
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(2) dropna Parameters in Detail

Parameter Default Description
axis 0 0 = drop rows, 1 = drop columns
how 'any' 'any' = drop if any NaN, 'all' = drop only if all NaN
thresh None Keep rows with at least N non-missing values
subset None Check only the specified columns

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Column-Specific Checking with dropna (Difficulty ⭐⭐)

PYTHON
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David'],
    'age': [28, np.nan, 25, 30, np.nan],
    'salary': [75000, 92000, np.nan, 88000, 105000],
    'department': ['Sales', 'Engineering', 'Marketing', 'Sales', 'Management']
})

# Only drop rows where 'salary' is missing
# (keep rows with missing 'age' — we can impute age)
print(df.dropna(subset=['salary']))
#      name   age    salary  department
# 0   Alice  28.0   75000.0       Sales
# 1     Bob   NaN   92000.0  Engineering
# 3   Carol  30.0   88000.0       Sales
# 4   David   NaN  105000.0  Management
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

6. Filling Missing Values with fillna

(1) Filling with a Constant

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Constant and Statistical Filling with fillna (Difficulty ⭐⭐)

PYTHON
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Carol', 'David'],
    'age': [28, np.nan, 25, 30, np.nan],
    'salary': [75000, 92000, np.nan, 88000, 105000]
})

# Fill with constant
df_const = df.fillna({'age': 0, 'salary': 0})
print(df_const)

# Fill with mean (most common for numeric)
df_mean = df.copy()
df_mean['age'] = df_mean['age'].fillna(df_mean['age'].mean())
df_mean['salary'] = df_mean['salary'].fillna(df_mean['salary'].median())
print(df_mean)
# age: filled with 27.67, salary: filled with 88000.0

# Fill with forward fill (use previous value)
df_ffill = df.fillna(method='ffill')
print(df_ffill)
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
⚠️ Note: In Pandas 2.x, fillna(method=...) is deprecated — use df.ffill() / df.bfill() instead.

(2) Forward / Backward Fill

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: ffill and bfill (Difficulty ⭐)

PYTHON
import pandas as pd
import numpy as np

# Time-series data — ffill makes sense here
ts = pd.DataFrame({
    'date': pd.date_range('2024-01-01', periods=8),
    'temperature': [22.5, np.nan, np.nan, 23.1, np.nan, 24.0, np.nan, 24.5]
})

# Forward fill (propagate last valid value)
print(ts.assign(temperature_ffill=ts['temperature'].ffill()))
# NaN → use previous non-NaN value

# Backward fill (propagate next valid value)
print(ts.assign(temperature_bfill=ts['temperature'].bfill()))
# NaN → use next non-NaN value

# Limit: only fill N consecutive NaN
print(ts.assign(temperature_limited=ts['temperature'].ffill(limit=1)))
# Only fill 1 consecutive NaN, leave the rest
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

7. Interpolation with interpolate

(1) Linear Interpolation

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Linear Interpolation with interpolate (Difficulty ⭐⭐)

PYTHON
import pandas as pd
import numpy as np

# Carol's heart rate data — linear interpolation is natural
hr = pd.DataFrame({
    'time_min': list(range(10)),
    'heart_rate': [72, 75, np.nan, 78, np.nan, 80, 82, np.nan, 85, 88]
})

# Linear interpolation (default)
hr['hr_linear'] = hr['heart_rate'].interpolate()
print(hr[['time_min', 'heart_rate', 'hr_linear']])
# time 2: (75+78)/2 = 76.5
# time 4: (78+80)/2 = 79.0
# time 7: (82+85)/2 = 83.5

# Index-based vs time-based interpolation
hr_ts = pd.DataFrame({
    'heart_rate': [72, 75, np.nan, 78, np.nan, 80]
}, index=pd.to_timedelta([0, 5, 15, 20, 40, 45], unit='min'))

# Time-weighted interpolation
hr_ts['hr_time'] = hr_ts['heart_rate'].interpolate(method='time')
print(hr_ts)
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(2) Comparing Interpolation Methods

Method method When to Use
Linear 'linear' Evenly sampled data, smooth trends
Time-weighted 'time' Uneven time intervals
Index-weighted 'index' Uneven index spacing
Quadratic 'quadratic' Curved trends
Cubic 'cubic' Smooth curves
Nearest 'nearest' Discrete / step data
Spline 'spline' Smooth interpolation (requires the order parameter)

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Comparing Interpolation Methods (Difficulty ⭐⭐)

PYTHON
import pandas as pd
import numpy as np

s = pd.Series([0.0, np.nan, np.nan, np.nan, np.nan, 5.0, 10.0])

print("linear:    ", s.interpolate(method='linear').tolist())
# [0.0, ~1.67, ~3.33, 5.0, ~6.67, 5.0, 10.0]
# But this includes NaN boundaries.

print("linear (trim):", s.dropna().interpolate(method='linear').tolist())
# Demonstrates linear interpolation only.

# quadratic needs 1 non-NaN value on each side of the boundary; here we use 7 data points
print("quadratic:", s.interpolate(method='quadratic').tolist())
# Fits a quadratic curve; requires at least 3 non-NaN points
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

8. Copy-on-Write and Missing-Value Operations

(1) Writing Safely Under CoW

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Safe Operations Under CoW (Difficulty ⭐⭐)

PYTHON
import pandas as pd
import numpy as np

pd.options.mode.copy_on_write = True  # Pandas 3.x default

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [28, np.nan, 25],
    'salary': [75000, 92000, np.nan]
})

# ✅ Safe: assign result back
df['age'] = df['age'].fillna(df['age'].mean())

# ✅ Safe: use inplace (works with CoW)
df.fillna({'salary': 0}, inplace=True)

# ❌ Dangerous (pre-CoW): chained assignment
# subset = df[df['age'].notnull()]
# subset['salary'] = 100  # May not modify df!

# ✅ Safe with CoW: loc
df.loc[df['age'].notnull(), 'salary'] = df.loc[df['age'].notnull(), 'salary'] * 1.1
print(df)
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

9. Complete Example: The Full Missing-Value Workflow

(5) ▶ Missing-Value Handling Decision Tree

100%
graph TB
    A[Detect missing values] --> B{Missing rate?}
    B -->|<5%| C[Drop with dropna]
    B -->|5-30%| D{Data type?}
    D -->|Numeric| E[fillna with mean/median]
    D -->|Categorical| F[fillna with mode]
    D -->|Time-series| G[interpolate]
    B -->|>50%| H[Consider dropping the column]
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

▶ Example

TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

: Full Missing-Value Cleaning Workflow (Difficulty ⭐⭐⭐)

PYTHON
import pandas as pd
import numpy as np

# ============================================
# Comprehensive example: Full missing-value
# handling pipeline
# ============================================

# 1. Load data with missing values
np.random.seed(42)
df = pd.DataFrame({
    'customer_id': range(1, 21),
    'name': [f'Customer_{i}' for i in range(1, 21)],
    'age': [25, np.nan, 35, 40, np.nan, 28, np.nan, 50, 33, np.nan,
            29, np.nan, 45, 38, np.nan, 27, np.nan, 42, 31, np.nan],
    'purchase_amount': [120, 85, np.nan, 200, 150, np.nan, 90, 310,
                        np.nan, 60, 175, np.nan, 250, 140, np.nan,
                        95, 180, np.nan, 110, 75],
    'category': ['Electronics', np.nan, 'Home', 'Electronics', 'Clothing',
                 'Home', np.nan, 'Electronics', 'Clothing', np.nan',
                 'Home', 'Electronics', 'Clothing', np.nan, 'Home',
                 'Electronics', np.nan, 'Home', 'Clothing', 'Electronics']
})

# 2. Analyze missing pattern
print("=== Missing Analysis ===")
missing_report = pd.DataFrame({
    'count': df.isnull().sum(),
    'pct': (df.isnull().sum() / len(df) * 100).round(1)
})
print(missing_report[missing_report['count'] > 0])

# 3. Drop rows where critical fields are missing
df = df.dropna(subset=['customer_id', 'name'])

# 4. Fill numeric columns
df['age'] = df['age'].fillna(df['age'].median())
df['purchase_amount'] = df['purchase_amount'].interpolate(method='linear')

# 5. Fill categorical column
df['category'] = df['category'].fillna(df['category'].mode()[0])

# 6. Verify: no more missing
print(f"\n=== After Cleaning ===")
print(f"Remaining missing: {df.isnull().sum().sum()}")
print(df.head(10))
TEXT 📖 Display only
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

❓ FAQ

Q What's the difference between NaN and None?
A NaN (np.nan) is the IEEE 754 floating-point missing value, and NaN ≠ NaN. None is Python's object-level missing value, and None == None. In Pandas, both are detected by isnull(). When an integer column contains NaN, it's automatically converted to float64; using the nullable Int64 type (capital I) preserves the integer type, with missing values represented by pd.NA.
Q How do I choose between dropna and fillna?
A If less than 5% is missing and unimportant → drop with dropna. If 5-30% is missing and a reasonable default exists → fill with fillna (mean/median/mode). For time-series data → interpolate. If more than 50% is missing → consider dropping the column. The key principle: dropping loses information, filling introduces bias, and interpolation assumes a trend.
Q Why does fillna(method='ffill') throw an error?
A Pandas 2.x deprecated the method parameter of fillna. Use df.ffill() (forward fill) and df.bfill() (backward fill) instead. ffill suits time-series data (using the last valid value), but not randomly ordered data (it can propagate wrong values very far).
Q Which interpolation method is best?
A The default linear is the most versatile. For time series, use method='time' (weighted by time distance). For clear trends, use quadratic/cubic. For step-change data, use nearest. spline is good for smoothing but requires tuning the order parameter. Start with linear, and switch only if the results aren't good enough.
Q Is fillna(inplace=True) still safe?
A Under the Copy-on-Write mode in Pandas 3.x, inplace is still safe — it modifies the original DataFrame directly. But the more recommended style is to assign back to a variable with df = df.fillna(...), which is clearer semantically. CoW guarantees that both approaches produce the same result.
Q How do I check whether a column contains NaN?
A The fastest way: df['col'].isnull().any() returns True/False. To count: df['col'].isnull().sum() returns the number of missing values. For the percentage: df['col'].isnull().mean() * 100. To check the whole table: df.isnull().any().any() tells you whether the DataFrame has any NaN at all.
Q Why does an integer column turn into float once NaN appears?
A NumPy's int64 can't represent NaN (every bit is a valid digit). Pandas has no choice but to convert the whole column to float64 to accommodate np.nan. The fix: use the nullable type df['col'].astype('Int64') (capital I), which represents missing values with pd.NA while preserving integer semantics.

📖 Summary


📝 Exercises

  1. Basic (Difficulty ⭐): Create a DataFrame containing NaN. Use isnull().sum() to count the missing values in each column, drop the missing rows with dropna(), and compare the change in row count.
  2. Intermediate (Difficulty ⭐⭐): Create a time-series DataFrame (10 rows with 3 NaN values). Fill it with ffill() / bfill() / interpolate(method='linear') respectively, and compare the differences among the three results.
  3. Challenge (Difficulty ⭐⭐⭐): Simulate 50 rows of customer data (age/salary/category each 15% missing) and complete the full cleaning workflow: missing analysis → dropna(subset=[key columns]) → fill age with the mean → interpolate salary → fill category with the mode → verify zero missing values.

← Previous: Data I/O · Next: Handling Duplicate Data →

Web-Tutorial.com

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

100%

🙏 帮我们做得更好

我们是刚上线的编程教程站,几个人的小团队,精力有限。页面虽经检查,难免还有疏漏——链接失效、排版错乱、内容有误、语言生硬……

如果您发现了,麻烦告诉我们,我们会在收到反馈后第一时间进行修复,再次感谢您的光临 🙏