Pandas: Getting Started with Pandas
NumPy solved the performance problem for numerical computing in Python, but real-world data is far more complex than "pure numeric arrays" — it has column names, mixed types, missing values, and a variety of file formats. Pandas was built for exactly this: it gives Python data manipulation power rivaling SQL while keeping code concise and intuitive. This section starts from NumPy's limitations and reveals why Pandas is the "Swiss Army knife" of data analysis.
1. What You'll Learn
- ❶ The pain points of doing data analysis with NumPy
- ❷ Pandas core advantages (labeled indexing / mixed types / missing values / IO)
- ❸ Key differences between DataFrame and ndarray / Excel / SQL
- ❹ An overview of the Pandas ecosystem
- ❺ Installing Pandas and running your first program
2. The Evolution from NumPy to Pandas
(1) Pain Point: Alice's Mixed-Type Dilemma
Alice is a data analyst who needs to work with customer data: names are strings, ages are integers, salaries are floats, and hire dates are datetime values. She tries NumPy first:
▶ Example: NumPy's Struggle with Mixed-Type Data (Difficulty ⭐)
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
import numpy as np
# Customer data: name(str), age(int), salary(float)
names = np.array(['Alice', 'Bob', 'Charlie'])
ages = np.array([28, 34, 25])
salaries = np.array([75000.0, 92000.0, 68000.0])
# Problem 1: mixing types forces upcast to object
mixed = np.array(['Alice', 28, 75000.0])
print(mixed.dtype) # object — loses numeric operations!
# Problem 2: cannot compute mean on object array
try:
print(np.mean(mixed)) # Error or meaningless result
except TypeError:
print("Cannot compute mean on object array")
# Problem 3: missing values need special handling
ages_with_nan = np.array([28, 34, np.nan, 25])
print(np.mean(ages_with_nan)) # nan — need np.nanmean()
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
NumPy's core limitation is homogeneous typing — a single ndarray can only store one data type. When you put strings and numbers into the same array, NumPy upcasts everything to object type, losing all vectorized operation capabilities. On top of that, NumPy has no concept of "column names" and no built-in missing value system (you can only rely on np.nan, which only works with floats).
(2) Solution: Bob's Pandas DataFrame
Bob handles the same data with a Pandas DataFrame — in one smooth move:
▶ Example: Handling Mixed-Type Data with Pandas (Difficulty ⭐⭐)
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
import pandas as pd
# One DataFrame handles mixed types naturally
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'age': [28, 34, 25],
'salary': [75000.0, 92000.0, 68000.0],
'hire_date': pd.to_datetime(['2022-03-15', '2019-08-01', '2023-01-10'])
})
print(df.dtypes)
# name object
# age int64
# salary float64
# hire_date datetime64[ns]
print(df['salary'].mean()) # 78333.33 — works naturally
print(df[df['age'] > 27]) # filter by condition
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
Each column keeps its own type, statistical operations work automatically, and conditional filtering is as intuitive as SQL — that's the beauty of Pandas.
(3) Benefit: 4 Core Capabilities
Pandas adds 4 key capabilities on top of NumPy:
| Capability | NumPy | Pandas |
|---|---|---|
| Labeled indexing | Access by integer position only | Access by column name / row label |
| Mixed types | One type per array | Independent type per column |
| Missing values | np.nan (floats only) |
NaN / NaT / pd.NA (full coverage) |
| Data IO | np.loadtxt / np.savetxt |
CSV / Excel / JSON / SQL / Parquet / HDF5 |
3. Pandas Core Advantages in Detail
(1) Labeled Indexing: Self-Describing Data
NumPy accesses data with arr[0], arr[1, 2] — you have to remember "column 0 is name, column 2 is salary." Pandas lets the data speak for itself:
▶ Example: Labeled Indexing vs Positional Indexing (Difficulty ⭐)
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
import numpy as np
import pandas as pd
# NumPy: must remember column order
arr = np.array([['Alice', 28, 75000.0],
['Bob', 34, 92000.0]])
# What is column 2? Have to check documentation
print(arr[0, 2]) # 75000.0 — but what does this mean?
# Pandas: column names are self-documenting
df = pd.DataFrame(arr, columns=['name', 'age', 'salary'])
df['age'] = df['age'].astype(int)
df['salary'] = df['salary'].astype(float)
print(df['salary']) # column name says it all
# 0 75000.0
# 1 92000.0
print(df.loc[0, 'name']) # Alice — label-based access
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(2) Mixed Types: Independent dtype per Column
Each column in a DataFrame (a Series) has its own dtype, just like how each column in an Excel spreadsheet can have a different format.
| Feature | NumPy ndarray | Pandas DataFrame |
|---|---|---|
| Type constraint | Single dtype for the entire array | Independent dtype per column |
| String columns | Entire array downgraded to object | Single column as object / StringDtype |
| Numeric columns | Must be separated from strings | Coexist with string columns |
| Operations | Global vectorization | Independent per-column operations |
(3) Missing Value Handling: Native Support
Pandas provides a complete missing value system with NaN (float missing), NaT (datetime missing), and pd.NA (universal missing), whereas NumPy's np.nan only works with float types.
▶ Example: Missing Value Handling Comparison (Difficulty ⭐)
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
import numpy as np
import pandas as pd
# NumPy: nan in integer array forces float conversion
arr = np.array([1, 2, np.nan, 4])
print(arr.dtype) # float64 — integers lost!
print(np.mean(arr)) # nan — need np.nanmean()
# Pandas: handles missing values per column
s = pd.Series([1, 2, pd.NA, 4], dtype='Int64') # nullable integer
print(s) # keeps integer type with <NA>
print(s.mean()) # 2.333 — auto-skips missing
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(4) Rich IO Tools: A One-Stop Data Hub
▶ Example: Reading and Writing Data with Pandas (Difficulty ⭐)
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
import pandas as pd
from io import StringIO
# Read from CSV string
csv_data = """name,age,salary
Alice,28,75000
Bob,34,92000
Charlie,25,68000"""
df = pd.read_csv(StringIO(csv_data))
print(df)
# Write to different formats
# df.to_csv('output.csv', index=False)
# df.to_excel('output.xlsx', sheet_name='Employees')
# df.to_json('output.json', orient='records')
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
4. DataFrame vs Excel / SQL: An Analogy
(1) Side-by-Side Comparison
| Concept | Excel | SQL Table | Pandas DataFrame |
|---|---|---|---|
| Data structure | Worksheet (Sheet) | Table | DataFrame |
| Rows | Row numbers | Rows | Index (row labels) |
| Columns | Column letters (A, B, C) | Column names | Columns (column names) |
| Filtering | AutoFilter | WHERE | df[condition] |
| Sorting | Sort | ORDER BY | df.sort_values() |
| Grouping | PivotTable | GROUP BY | df.groupby() |
| Merging | VLOOKUP | JOIN | pd.merge() |
| Adding columns | Formula columns | SELECT expr | df['new'] = expr |
| Deduplication | Remove Duplicates | DISTINCT | df.drop_duplicates() |
(2) Key Differences
While all three share similar concepts, DataFrame has capabilities that Excel and SQL lack:
- Programmable: Automate all operations with Python code, fully reproducible
- Extensible: Seamless integration with NumPy / Matplotlib / scikit-learn
- Handles large data: Easily process millions of rows (Excel caps at ~1 million rows)
- Version control: Code files can be managed with Git; Excel binary files cannot
5. Pandas Ecosystem Overview
(1) Where Pandas Fits in Data Science
mindmap
root((Pandas Ecosystem))
Data Input
CSV Excel JSON
SQL Database
Parquet HDF5
Web Scraping
Data Processing
Cleaning
Transforming
Merging
Reshaping
Data Analysis
Groupby Aggregate
Pivot Table
Time Series
Statistical Summary
Data Output
Visualization Matplotlib
ML scikit-learn
Reports HTML Excel
Foundation
NumPy ndarray
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(2) Pandas' Role in the Stack
Pandas sits at the core of the Python data science stack, bridging low-level computation and high-level analysis:
| Layer | Tool | Role |
|---|---|---|
| Low-level computation | NumPy | High-performance numerical computing engine |
| Data processing | Pandas | Loading / cleaning / transforming / analyzing tabular data |
| Visualization | Matplotlib / Seaborn | Charts and plots |
| Machine learning | scikit-learn | Model training and prediction |
| Deep learning | PyTorch / TensorFlow | Neural networks |
6. Complete Example: NumPy vs Pandas Full Workflow Comparison
▶ Example: Student Grade Analysis — Full Workflow (Difficulty ⭐⭐⭐)
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
import numpy as np
import pandas as pd
# ============================================
# Comprehensive comparison: NumPy vs Pandas
# for student grade analysis workflow
# ============================================
# --- Raw data ---
students = ['Alice', 'Bob', 'Charlie', 'Carol', 'David']
math = [85, 92, 78, 95, 88]
english = [90, 85, 82, 88, 91]
science = [88, 90, 75, 93, 86]
# --- NumPy approach ---
scores_np = np.array([math, english, science]) # shape: (3, 5)
# Which row is which subject? Must remember: row 0=math, 1=english, 2=science
# Which column is which student? Must remember order
avg_per_student = scores_np.mean(axis=0)
print("NumPy - Average per student (by position):")
for i, avg in enumerate(avg_per_student):
print(f" Student {i}: {avg:.1f}") # Who is Student 0?
# --- Pandas approach ---
df = pd.DataFrame({
'student': students,
'math': math,
'english': english,
'science': science
})
df['average'] = df[['math', 'english', 'science']].mean(axis=1)
df['grade'] = df['average'].map(
lambda x: 'A' if x >= 90 else 'B' if x >= 85 else 'C'
)
print("\nPandas - Student grades:")
print(df[['student', 'average', 'grade']])
# Top performers
top = df[df['grade'] == 'A']
print(f"\nA-grade students: {top['student'].tolist()}")
# Subject statistics
print("\nSubject statistics:")
print(df[['math', 'english', 'science']].describe())
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
Expected output (excerpt):
NumPy - Average per student (by position):
Student 0: 87.7
Student 1: 89.0
Student 2: 78.3
Pandas - Student grades:
student average grade
0 Alice 87.7 B
1 Bob 89.0 B
2 Charlie 78.3 C
3 Carol 92.0 A
4 David 88.3 B
A-grade students: ['Carol']
Subject statistics:
math english science
count 5.00 5.00 5.00
mean 87.60 87.20 86.40
std 6.19 3.70 7.60
min 78.00 82.00 75.00
max 95.00 91.00 93.00
NumPy can do the computation, but the results lack "self-descriptiveness" — you have to separately remember which number corresponds to which student. Pandas keeps data and labels bound together at all times, making results immediately clear.
7. Installation and Verification
(1) Installation Methods
| Method | Command | Use Case |
|---|---|---|
| pip | pip install pandas |
Most universal |
| conda | conda install pandas |
Anaconda environments |
| From source | pip install . |
Developers / contributors |
▶ Example: Verifying Your Installation (Difficulty ⭐)
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
import pandas as pd
import numpy as np
print(f"Pandas version: {pd.__version__}")
print(f"NumPy version: {np.__version__}")
# Quick functionality check
df = pd.DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
print(df)
print(f"\nMean of x: {df['x'].mean()}")
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
(2) Version Notes
| Version | Status | Notes |
|---|---|---|
| Pandas 3.x | Latest | Released in 2026, Copy-on-Write enabled by default |
| Pandas 2.x | Widely used | Released in 2023, major performance improvements |
| Pandas 1.x | Legacy | Some syntax has been deprecated |
append) was removed in 2.0 and is no longer used here.
❓ FAQ
append and DataFrame.ix have been removed; (3) PyArrow is used under the hood for better performance. This tutorial uses 2.x/3.x syntax — 1.x users should upgrade.pip install pandas.📖 Summary
- Pandas adds 4 core capabilities on top of NumPy: labeled indexing, mixed types, missing value handling, and rich IO tools
- DataFrame ≈ a programmable Excel sheet ≈ an in-memory SQL table, but with automation / extensibility / version control advantages
- Pandas sits at the core of the Python data science stack: NumPy → Pandas → Matplotlib/scikit-learn
- 95% of data analysis work is done with Pandas; NumPy is the underlying computation engine
- Pandas 2.x/3.x are the current mainstream versions; some 1.x syntax has been deprecated
- For data that exceeds single-machine capacity, scale up to Dask/Polars/PySpark
📝 Exercises
- Basic (Difficulty ⭐): Install Pandas and run the verification code to confirm
pd.__version__outputs correctly. Create a DataFrame with 3 columns (name / age / city) and print it. - Intermediate (Difficulty ⭐⭐): Process the same student grade data (5 students / 3 subjects) with both NumPy and Pandas. Calculate each student's average score and the highest score per subject. Compare the code volume and readability of both approaches.
- Challenge (Difficulty ⭐⭐⭐): Load data containing missing values from a CSV string, then use Pandas to: view info → fill missing values → group by category and compute statistics → output describe. Use only Pandas throughout — no loops allowed.