NumPy: Project — Data Cleaning
Last updated: 2026-08-26
1. Project: Data Cleaning with NumPy
(1) Scenario
You're a data analyst. You receive a CSV file with 10,000 rows of sensor data — but it's messy: missing values, outliers, inconsistent scales, and duplicate rows.
(2) Tasks
1. Load and Inspect
PYTHON
import numpy as np
# Load data (simulated)
rng = np.random.default_rng(42)
n = 10000
# Generate clean data
temperature = rng.normal(25, 5, n) # mean 25°C, std 5°C
humidity = rng.uniform(30, 80, n) # 30-80%
pressure = rng.normal(1013, 10, n) # mean 1013 hPa
# Add some NaN values
temperature[0:50] = np.nan
humidity[200:250] = np.nan
# Add outliers
temperature[1000:1010] = 100 # impossible temperature
pressure[2000:2005] = 0 # impossible pressure
# Stack into structured array
data = np.column_stack([temperature, humidity, pressure])
print(f"Shape: {data.shape}")
print(f"NaN count: {np.sum(np.isnan(data), axis=0)}")
2. Handle Missing Values
PYTHON
# Replace NaN with column mean
col_mean = np.nanmean(data, axis=0)
data_clean = np.where(np.isnan(data), col_mean, data)
3. Remove Outliers (IQR method)
PYTHON
# IQR-based outlier removal
Q1 = np.percentile(data_clean, 25, axis=0)
Q3 = np.percentile(data_clean, 75, axis=0)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
# Filter rows that are within bounds for all columns
mask = np.all((data_clean >= lower) & (data_clean <= upper), axis=1)
data_filtered = data_clean[mask]
print(f"Rows before: {len(data_clean)}, after: {len(data_filtered)}")
4. Normalize Columns (Z-score)
PYTHON
mean = data_filtered.mean(axis=0)
std = data_filtered.std(axis=0)
data_normalized = (data_filtered - mean) / std
print(f"Normalized mean: {data_normalized.mean(axis=0).round(6)}")
print(f"Normalized std: {data_normalized.std(axis=0).round(6)}")
5. Export Clean Data
PYTHON
# Save cleaned data
np.save('sensor_data_clean.npy', data_filtered)
np.savetxt('sensor_data_clean.csv', data_filtered, delimiter=',',
header='temperature,humidity,pressure', comments='')
print("Clean data exported.")
▶ Example: Loading and inspecting data (Difficulty ⭐)
PYTHON
import numpy as np
rng = np.random.default_rng(42)
data = rng.normal(25, 5, size=(1000, 3))
print("Shape:", data.shape)
print("First 5 rows:\n", data[:5])
print("NaN count:", np.sum(np.isnan(data)))
print("Mean:", data.mean(axis=0))
print("Std:", data.std(axis=0))
Output:
TEXT 📖 Display onlyShape: (1000, 3) First 5 rows: [[28.698 20.884 28.847] [24.899 25.891 29.574] [27.198 28.793 25.009] [24.694 25.771 25.929] [26.676 25.174 23.938]] NaN count: 0 Mean: [25.045 24.908 25.039] Std: [5.019 5.045 5.030]
▶ Example: Handling missing values with np.where (Difficulty ⭐⭐)
PYTHON
import numpy as np
data = np.array([1.0, 2.0, np.nan, 4.0, np.nan, 6.0, 7.0])
# Replace NaN with column mean
col_mean = np.nanmean(data)
clean = np.where(np.isnan(data), col_mean, data)
print("Original:", data)
print("Cleaned:", clean)
print("Mean after:", clean.mean())
Output:
TEXT 📖 Display onlyOriginal: [ 1. 2. nan 4. nan 6. 7.] Cleaned: [1. 2. 4. 4. 4. 6. 7.] Mean after: 4.0
▶ Example: Outlier removal with IQR (Difficulty ⭐⭐)
PYTHON
import numpy as np
rng = np.random.default_rng(42)
data = np.concatenate([rng.normal(50, 10, 95), [200, -50, 300, -80, 150]])
Q1 = np.percentile(data, 25)
Q3 = np.percentile(data, 75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
filtered = data[(data >= lower) & (data <= upper)]
print(f"Total: {len(data)}, Outliers: {len(data) - len(filtered)}")
print(f"Before: mean={data.mean():.1f}, After: mean={filtered.mean():.1f}")
Output:
TEXT 📖 Display onlyTotal: 100, Outliers: 5 Before: mean=56.8, After: mean=49.1
❓ FAQ
Q What if my data has string columns?
A NumPy's loadtxt can't handle mixed types. Use
np.genfromtxt with dtype=None for mixed data, or use Pandas for complex CSV files.Q How do I know which IQR multiplier to use?
A 1.5 is the standard for "mild" outliers, 3.0 for "extreme" outliers. These come from the standard normal distribution where 1.5×IQR ≈ ±2.7σ.
Q Should I normalize before or after removing outliers?
A After. Outliers skew the mean and std, making normalization less effective. Always clean data first, then normalize.
📖 Summary
In this project you applied:
np.nanmean/np.wherefor missing value handlingnp.percentilefor IQR-based outlier detection- Boolean masking for filtering
- Vectorized normalization (broadcasting)
np.save/np.savetxtfor export
📝 Exercises
-
Beginner (Difficulty ⭐): Modify the IQR multiplier to 3.0. How many fewer outliers are detected?
-
Intermediate (Difficulty ⭐⭐): Add a column for 'timestamp' and filter out data outside business hours (9 AM - 5 PM).
-
Advanced (Difficulty ⭐⭐⭐): Implement a moving window for outlier detection — flag a point as an outlier only if it's outside the IQR of its local neighborhood (100 surrounding points).
