Data Fundamentals
Data is the fuel for AI—without data, even the best algorithms cannot train effective models. This chapter will help you understand data types, feature engineering, dataset splitting, and the ironclad rule of “Garbage In, Garbage Out.”
1. What You'll Learn
- The Central Role of Data in AI
- Structured vs. Unstructured Data
- Features and Labels
- Dataset split: training / validation / test
- How Does Data Quality Affect AI Performance?
2. A True Story of a Machine Learning Project
(1) Pain Point: After one month of training, the accuracy is 60%
Bob's team spent a month training an image classification model, but its accuracy was only 60%. Upon investigation, they discovered that 70% of the training data consisted of images taken on sunny days, whereas real-world scenarios included a variety of weather conditions—rain, fog, and nighttime. Since the model had "never seen" these conditions, it was naturally unable to recognize them.
(2) Data-Driven Solutions
Charlie said, "That's 'Garbage In, Garbage Out'—AI can only learn what you feed it. Instead of tweaking the algorithm parameters, you should fix the data distribution first."
▶ Example: Checking Data Distribution—Identifying Skewness (Difficulty: ⭐⭐)
PYTHON
# Check data distribution — first step before any AI project
import pandas as pd
df = pd.DataFrame({
'weather': ['sunny'] * 70 + ['rainy'] * 15 + ['foggy'] * 10 + ['night'] * 5,
'label': ['car'] * 50 + ['car'] * 5 + ['car'] * 3 + ['car'] * 2 +
['truck'] * 20 + ['truck'] * 10 + ['truck'] * 7 + ['truck'] * 3
})
print("Weather distribution:")
print(df['weather'].value_counts())
print(f"\nSunny images: {70/100*100:.0f}% — That's a problem!")
💻 Output:
TEXT
Weather distribution:
sunny 70
rainy 15
foggy 10
night 5
Name: weather, dtype: int64
Sunny images: 70% — That's a problem!
(3) Benefits: A Data-First Mindset
After Bob added images for each weather condition, the accuracy rate rose from 60% to 85%. He concluded that there is one ironclad rule: "Tuning the model is not as effective as tuning the data."
3. Data Types
(1) Classification by Structure
| Type | Description | Example | AI Processing Method |
|---|---|---|---|
| Structured | Has a fixed format (rows and columns) | CSV, SQL tables, Excel | Directly input into traditional ML models |
| Semi-structured | Partially structured but flexible in format | JSON, XML, HTML, logs | Input into the model after extracting fields |
| Unstructured | No fixed format | Images, audio, video, natural language | Deep learning (CNN/RNN/Transformer) |
(2) Classification by Numeric Type
| Type | Description | Example | Model Processing |
|---|---|---|---|
| Numeric | Continuous or discrete numbers | Price, area, age | Used directly |
| Categorical | Finite number of discrete values | Gender, City, Occupation | Requires encoding (One-Hot) |
| Text Type | Natural language strings | Comments, email bodies | Requires embedding |
| Time-based | Date and Time | Order Time, Log Time | Features to Extract (Year/Month/Hour) |
4. Features and Labels
Features are the inputs to the model, and labels are the targets the model predicts.
graph LR
A[Feature 1<br/>Area 120 sqm] --> C[AI Model]
B[Feature 2<br/>Rooms 4] --> C
D[Feature 3<br/>Location Score 8] --> C
C --> E[Label<br/>Price $350,000]
(1) What Is a Feature?
A feature is an attribute in the data that is "helpful for making predictions":
| Dataset | Possible Features | Label |
|---|---|---|
| Housing Price Data | Size, Number of Rooms, Location Rating, Floor Level | Property Price |
| Email Data | Sender, Keyword Frequency, Number of Attachments | Is it Spam? |
| User Data | Age, Number of Page Views, Purchase History | Likelihood of Purchase |
| Image Data | Pixel Values (Features Automatically Extracted by Deep Learning) | Image Category |
(2) Feature Engineering—Manual vs. Automatic Extraction
▶ Example: Manual Feature Engineering—Extracting Features from Text (Difficulty: ⭐⭐)
PYTHON
# Example: Manual feature engineering
import pandas as pd
df = pd.DataFrame({
'text': ['Free iPhone winner!', 'Meeting at 3pm', 'URGENT: Claim now'],
'label': ['spam', 'ham', 'spam']
})
# Manual feature extraction: count specific words
df['has_free'] = df['text'].str.lower().str.contains('free').astype(int)
df['has_urgent'] = df['text'].str.lower().str.contains('urgent').astype(int)
df['exclamation_count'] = df['text'].str.count('!')
df['text_length'] = df['text'].str.len()
print(df)
💻 Output:
TEXT
text label has_free has_urgent exclamation_count text_length
0 Free iPhone winner! spam 1 0 1 19
1 Meeting at 3pm ham 0 0 0 14
2 URGENT: Claim now spam 0 1 0 17
💡 Tip: Traditional ML requires manual feature design (feature engineering), whereas deep learning can automatically learn features from raw data. This is one of the core advantages of the deep learning revolution.
5. Dataset Split
When training an AI model, the data must be divided into three mutually exclusive sets:
graph TB
A[Full Dataset<br/>1000 samples] --> B[Training Set<br/>800 samples<br/>80%]
A --> C[Validation Set<br/>100 samples<br/>10%]
A --> D[Test Set<br/>100 samples<br/>10%]
B --> E[Train the model]
C --> F[Tune hyperparameters]
D --> G[Final evaluation<br/>never used during training]
| Set | Operation | Ratio | Key Rules |
|---|---|---|---|
| Training Set | Train the model (learn parameters) | 60–80% | The model learns directly from this set |
| Validation Set | Tuning hyperparameters (selecting the optimal configuration) | 10–20% | Indirectly affects the model, but is not directly learned |
| Test Set | Final Evaluation (Reported Performance) | 10–20% | Must not be used during training |
(1) Splitting the Dataset Using Python
▶ Example: Splitting the Data into Training, Validation, and Test Sets (Difficulty: ⭐⭐)
PYTHON
# Split dataset into train/validation/test sets
from sklearn.model_selection import train_test_split
import pandas as pd
# Sample dataset
df = pd.DataFrame({
'area_sqm': [50, 80, 120, 65, 200, 90, 55, 110, 75, 150,
60, 85, 130, 95, 180, 70, 100, 45, 140, 105],
'price_usd': [150000, 280000, 350000, 220000, 500000, 260000,
165000, 320000, 240000, 420000, 175000, 270000,
380000, 290000, 460000, 210000, 300000, 135000,
400000, 310000]
})
# First split: 80% train+val, 20% test
train_val, test = train_test_split(df, test_size=0.2, random_state=42)
# Second split: 75% of train_val = 60% total train, 25% = 20% total val
train, val = train_test_split(train_val, test_size=0.25, random_state=42)
print(f"Full dataset: {len(df)} samples")
print(f"Training set: {len(train)} samples ({len(train)/len(df)*100:.0f}%)")
print(f"Validation set:{len(val)} samples ({len(val)/len(df)*100:.0f}%)")
print(f"Test set: {len(test)} samples ({len(test)/len(df)*100:.0f}%)")
💻 Output:
TEXT
Full dataset: 20 samples
Training set: 12 samples (60%)
Validation set:4 samples (20%)
Test set: 4 samples (20%)
⚠️ Note: The test set must never be used during training. If the model has “seen” the data in the test set, the evaluation results will not be objective—just as if you looked at the answers before taking an exam, your score would not reflect your true ability.
6. Data Quality — Garbage In, Garbage Out
Data quality directly determines the upper limit of AI performance. No matter how good the algorithm is, it can’t make up for poor data:
| Quality Issue | Description | Impact | Testing Method |
|---|---|---|---|
| Missing Values | Certain fields are empty | The model cannot process missing inputs | df.isnull().sum() |
| Duplicate Data | The same record appears multiple times | The model overfits to duplicate samples | df.duplicated().sum() |
| Class Imbalance | One class has far more samples than the others | The model favors the majority class | df[label].value_counts() |
| Noise/Errors | Data values are clearly unreasonable | The model has learned the wrong pattern | Statistical outlier detection |
| Labeling Error | Incorrect labeling | Model learned an incorrect mapping | Manual spot-check |
(1) Identifying Data Quality Issues
▶ Example: Data Quality Inspection Report (Difficulty: ⭐⭐)
PYTHON
# Data quality check checklist
import pandas as pd
import numpy as np
df = pd.DataFrame({
'age': [25, 30, np.nan, 45, 200, 28, 30, 30, 35, np.nan],
'income': [50000, 60000, 45000, np.nan, 80000, 55000, 60000, 62000, 70000, 48000],
'label': ['buy', 'no', 'buy', 'no', 'buy', 'no', 'no', 'no', 'buy', 'no']
})
print("=== Data Quality Report ===")
print(f"Total rows: {len(df)}")
print(f"\nMissing values per column:")
print(df.isnull().sum())
print(f"\nDuplicate rows: {df.duplicated().sum()}")
print(f"\nLabel distribution:")
print(df['label'].value_counts())
print(f"\nSuspicious values (age > 120):")
print(df[df['age'] > 120])
💻 Output:
TEXT
=== Data Quality Report ===
Total rows: 10
Missing values per column:
age 2
income 1
label 0
dtype: int64
Duplicate rows: 0
Label distribution:
no 6
buy 4
Name: label, dtype: int64
Suspicious values (age > 120):
age income label
4 200 80000 buy
💡 Tip: age=200 is clearly incorrect data. If outliers like this aren’t cleaned up, the model will “learn” an absurd rule such as “people aged 200 will make a purchase.”
7. Commonly Used Public Datasets
Conducting AI experiments doesn't necessarily require collecting your own data; many high-quality public datasets are available for immediate use:
| Dataset | Task | Number of Samples | Features |
|---|---|---|---|
| Iris | Category | 150 | The simplest introductory ML dataset |
| Titanic | Category | 891 | Classic Kaggle Introductory Challenge |
| MNIST | Image Classification | 70,000 | Handwritten digits, Introduction to Computer Vision |
| CIFAR-10 | Image classification | 60,000 | 10 classes of color images; more challenging than MNIST |
| California Housing | Back | 20,640 | California Housing Price Forecast |
| IMDb Reviews | Sentiment Analysis | 50,000 | Classification of Movie Reviews as Positive or Negative |
8. Complete Example: Titanic Data Preparation Pipeline
On the Titanic dataset, complete the following steps: data overview → missing value check → category distribution statistics → splitting into training and test sets, to form a complete "data preparation pipeline":
▶ Example: Complete Data Preparation Pipeline for the Titanic Dataset (Difficulty: ⭐⭐⭐)
PYTHON
# ============================================
# Complete Data Preparation Pipeline
# Dataset: Titanic (simulated)
# ============================================
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
# Simulated Titanic-like dataset
np.random.seed(42)
df = pd.DataFrame({
'pclass': np.random.choice([1, 2, 3], 100, p=[0.2, 0.3, 0.5]),
'sex': np.random.choice(['male', 'female'], 100, p=[0.6, 0.4]),
'age': np.where(np.random.rand(100) > 0.15,
np.random.randint(1, 80, 100), np.nan),
'fare': np.round(np.random.uniform(10, 500, 100), 2),
'survived': np.random.choice([0, 1], 100, p=[0.6, 0.4])
})
# Step 1: Overview
print("=== Step 1: Data Overview ===")
print(f"Shape: {df.shape}")
print(df.head(5))
# Step 2: Missing values
print("\n=== Step 2: Missing Values ===")
missing = df.isnull().sum()
missing_pct = (df.isnull().sum() / len(df) * 100).round(1)
print(pd.DataFrame({'count': missing, 'percent': missing_pct}))
# Step 3: Label distribution
print("\n=== Step 3: Label Distribution ===")
print(df['survived'].value_counts())
print(f"Survival rate: {df['survived'].mean()*100:.1f}%")
# Step 4: Train/test split
train, test = train_test_split(df, test_size=0.2, random_state=42,
stratify=df['survived'])
print(f"\n=== Step 4: Data Split ===")
print(f"Train: {len(train)} samples, survival rate: {train['survived'].mean()*100:.1f}%")
print(f"Test: {len(test)} samples, survival rate: {test['survived'].mean()*100:.1f}%")
print(f"\nStratified split ensures same survival rate in both sets!")
💻 Output:
TEXT
=== Step 1: Data Overview ===
Shape: (100, 5)
pclass sex age fare survived
0 3 female 47.0 339.89 1
1 3 female 63.0 361.38 0
2 1 female 44.0 309.18 0
3 3 male 28.0 385.59 0
4 1 male 8.0 477.95 0
=== Step 2: Missing Values ===
count percent
pclass 0 0.0
sex 0 0.0
age 14 14.0
fare 0 0.0
survived 0 0.0
=== Step 3: Label Distribution ===
0 58
1 42
Name: survived, dtype: int64
Survival rate: 42.0%
=== Step 4: Data Split ===
Train: 80 samples, survival rate: 42.5%
Test: 20 samples, survival rate: 40.0%
Stratified split ensures same survival rate in both sets!
❓ FAQ
Q Does AI become more accurate as the amount of data increases?
A Not necessarily. Data quality is more important than quantity. 10,000 clean records may outperform 1 million noisy records when training a better model. Key factors: data volume, quality, diversity, and representativeness—all are essential.
Q Why can’t the training set and test set overlap?
A Because the test set is used to evaluate the model’s “true capability.” If the model has “seen” data from the test set during training, it will memorize the answers rather than learn the method—this is called data leakage, and it will lead to inflated evaluation results.
Q What is data skew? How can it be addressed?
A Data skew refers to a significant disparity in the number of samples across different classes (e.g., 99% normal vs. 1% abnormal). Solutions include: ① Collecting more data from the minority class; ② Oversampling the minority class (SMOTE); ③ Undersampling the majority class; ④ Adjusting class weights; ⑤ Using F1 score rather than accuracy for evaluation.
Q What is feature engineering? Why is it done?
A Feature engineering is the process of extracting or constructing new features from raw data that are useful for prediction. For example, features such as “whether it’s the weekend” or “whether it’s nighttime” can be derived from “order time.” Good feature engineering can enable simple models to achieve results comparable to complex models, making it crucial in traditional machine learning. Deep learning can automatically perform feature extraction, reducing the need for manual feature engineering.
Q Where can I find free datasets?
A Kaggle (kaggle.com/datasets), the UCI Machine Learning Repository, Google Dataset Search, Hugging Face Datasets, and government open data platforms. This tutorial uses the datasets included with sklearn, so no additional downloads are required.
Q What is a stratified split?
A A standard random split may result in an imbalance in class proportions between the training and testing sets. A stratified split ensures that the class proportions in each set match those in the original data. For skewed datasets (such as 90% vs. 10%), a stratified split is virtually essential.
📖 Summary
- Data is the fuel for AI; without high-quality data, there can be no good models—Garbage In, Garbage Out
- Data is divided into three categories: structured (tables), semi-structured (JSON/XML), and unstructured (images/text/audio)
- Features are the model's inputs, and labels are the targets the model predicts; feature engineering is a core skill in traditional machine learning.
- The dataset must be divided into a training set, a validation set, and a test set; the test set must never be used during training.
- Data quality issues (missing values, duplicates, skew, noise, and incorrect labels) must be resolved in the upstream stages; training a model cannot fix bad data.
- Public datasets (Iris, Titanic, MNIST) are the best place to start when learning about AI
📝 Exercises
- Basic Exercise (Difficulty ⭐): Load a CSV dataset (or a dataset included with sklearn) using Python, and use
df.info()anddf.describe()to summarize the basic information. - Advanced Problem (Difficulty ⭐⭐): Check for missing values and examine the distribution of categories in the data, then plot a histogram of a specific column (Hint:
df[column].hist()). - Challenge Problem (Difficulty ⭐⭐⭐): Use
train_test_splitto split the data according to the 80/20 rule, and compare whether the category proportions are the same after a standard split and a stratified split.
