Pandas: Getting Started with Pandas

NumPy solved the performance problem for numerical computing in Python, but real-world data is far more complex than "pure numeric arrays" — it has column names, mixed types, missing values, and a variety of file formats. Pandas was built for exactly this: it gives Python data manipulation power rivaling SQL while keeping code concise and intuitive. This section starts from NumPy's limitations and reveals why Pandas is the "Swiss Army knife" of data analysis.

⚠️ Note: The code below requires a local Python environment to run.

1. What You'll Learn



2. The Evolution from NumPy to Pandas

(1) Pain Point: Alice's Mixed-Type Dilemma

Alice is a data analyst who needs to work with customer data: names are strings, ages are integers, salaries are floats, and hire dates are datetime values. She tries NumPy first:

▶ Example: NumPy's Struggle with Mixed-Type Data (Difficulty ⭐)

TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
PYTHON
import numpy as np

# Customer data: name(str), age(int), salary(float)
names = np.array(['Alice', 'Bob', 'Charlie'])
ages = np.array([28, 34, 25])
salaries = np.array([75000.0, 92000.0, 68000.0])

# Problem 1: mixing types forces upcast to object
mixed = np.array(['Alice', 28, 75000.0])
print(mixed.dtype)  # object — loses numeric operations!

# Problem 2: cannot compute mean on object array
try:
    print(np.mean(mixed))  # Error or meaningless result
except TypeError:
    print("Cannot compute mean on object array")

# Problem 3: missing values need special handling
ages_with_nan = np.array([28, 34, np.nan, 25])
print(np.mean(ages_with_nan))  # nan — need np.nanmean()
TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

NumPy's core limitation is homogeneous typing — a single ndarray can only store one data type. When you put strings and numbers into the same array, NumPy upcasts everything to object type, losing all vectorized operation capabilities. On top of that, NumPy has no concept of "column names" and no built-in missing value system (you can only rely on np.nan, which only works with floats).

(2) Solution: Bob's Pandas DataFrame

Bob handles the same data with a Pandas DataFrame — in one smooth move:

▶ Example: Handling Mixed-Type Data with Pandas (Difficulty ⭐⭐)

TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
PYTHON
import pandas as pd

# One DataFrame handles mixed types naturally
df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [28, 34, 25],
    'salary': [75000.0, 92000.0, 68000.0],
    'hire_date': pd.to_datetime(['2022-03-15', '2019-08-01', '2023-01-10'])
})

print(df.dtypes)
# name                 object
# age                   int64
# salary              float64
# hire_date    datetime64[ns]

print(df['salary'].mean())  # 78333.33 — works naturally
print(df[df['age'] > 27])   # filter by condition
TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

Each column keeps its own type, statistical operations work automatically, and conditional filtering is as intuitive as SQL — that's the beauty of Pandas.

(3) Benefit: 4 Core Capabilities

Pandas adds 4 key capabilities on top of NumPy:

Capability NumPy Pandas
Labeled indexing Access by integer position only Access by column name / row label
Mixed types One type per array Independent type per column
Missing values np.nan (floats only) NaN / NaT / pd.NA (full coverage)
Data IO np.loadtxt / np.savetxt CSV / Excel / JSON / SQL / Parquet / HDF5


3. Pandas Core Advantages in Detail

(1) Labeled Indexing: Self-Describing Data

NumPy accesses data with arr[0], arr[1, 2] — you have to remember "column 0 is name, column 2 is salary." Pandas lets the data speak for itself:

▶ Example: Labeled Indexing vs Positional Indexing (Difficulty ⭐)

TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
PYTHON
import numpy as np
import pandas as pd

# NumPy: must remember column order
arr = np.array([['Alice', 28, 75000.0],
                ['Bob', 34, 92000.0]])
# What is column 2? Have to check documentation
print(arr[0, 2])  # 75000.0 — but what does this mean?

# Pandas: column names are self-documenting
df = pd.DataFrame(arr, columns=['name', 'age', 'salary'])
df['age'] = df['age'].astype(int)
df['salary'] = df['salary'].astype(float)
print(df['salary'])  # column name says it all
# 0    75000.0
# 1    92000.0
print(df.loc[0, 'name'])  # Alice — label-based access
TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(2) Mixed Types: Independent dtype per Column

Each column in a DataFrame (a Series) has its own dtype, just like how each column in an Excel spreadsheet can have a different format.

Feature NumPy ndarray Pandas DataFrame
Type constraint Single dtype for the entire array Independent dtype per column
String columns Entire array downgraded to object Single column as object / StringDtype
Numeric columns Must be separated from strings Coexist with string columns
Operations Global vectorization Independent per-column operations

(3) Missing Value Handling: Native Support

Pandas provides a complete missing value system with NaN (float missing), NaT (datetime missing), and pd.NA (universal missing), whereas NumPy's np.nan only works with float types.

▶ Example: Missing Value Handling Comparison (Difficulty ⭐)

TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
PYTHON
import numpy as np
import pandas as pd

# NumPy: nan in integer array forces float conversion
arr = np.array([1, 2, np.nan, 4])
print(arr.dtype)  # float64 — integers lost!
print(np.mean(arr))  # nan — need np.nanmean()

# Pandas: handles missing values per column
s = pd.Series([1, 2, pd.NA, 4], dtype='Int64')  # nullable integer
print(s)  # keeps integer type with <NA>
print(s.mean())  # 2.333 — auto-skips missing
TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(4) Rich IO Tools: A One-Stop Data Hub

▶ Example: Reading and Writing Data with Pandas (Difficulty ⭐)

TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
PYTHON
import pandas as pd
from io import StringIO

# Read from CSV string
csv_data = """name,age,salary
Alice,28,75000
Bob,34,92000
Charlie,25,68000"""
df = pd.read_csv(StringIO(csv_data))
print(df)

# Write to different formats
# df.to_csv('output.csv', index=False)
# df.to_excel('output.xlsx', sheet_name='Employees')
# df.to_json('output.json', orient='records')
TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.


4. DataFrame vs Excel / SQL: An Analogy

(1) Side-by-Side Comparison

Concept Excel SQL Table Pandas DataFrame
Data structure Worksheet (Sheet) Table DataFrame
Rows Row numbers Rows Index (row labels)
Columns Column letters (A, B, C) Column names Columns (column names)
Filtering AutoFilter WHERE df[condition]
Sorting Sort ORDER BY df.sort_values()
Grouping PivotTable GROUP BY df.groupby()
Merging VLOOKUP JOIN pd.merge()
Adding columns Formula columns SELECT expr df['new'] = expr
Deduplication Remove Duplicates DISTINCT df.drop_duplicates()

(2) Key Differences

While all three share similar concepts, DataFrame has capabilities that Excel and SQL lack:



5. Pandas Ecosystem Overview

(1) Where Pandas Fits in Data Science

100%
mindmap
  root((Pandas Ecosystem))
    Data Input
      CSV Excel JSON
      SQL Database
      Parquet HDF5
      Web Scraping
    Data Processing
      Cleaning
      Transforming
      Merging
      Reshaping
    Data Analysis
      Groupby Aggregate
      Pivot Table
      Time Series
      Statistical Summary
    Data Output
      Visualization Matplotlib
      ML scikit-learn
      Reports HTML Excel
    Foundation
      NumPy ndarray
TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(2) Pandas' Role in the Stack

Pandas sits at the core of the Python data science stack, bridging low-level computation and high-level analysis:

Layer Tool Role
Low-level computation NumPy High-performance numerical computing engine
Data processing Pandas Loading / cleaning / transforming / analyzing tabular data
Visualization Matplotlib / Seaborn Charts and plots
Machine learning scikit-learn Model training and prediction
Deep learning PyTorch / TensorFlow Neural networks
💡 Tip: Under the hood, a Pandas DataFrame is backed by NumPy ndarrays — when you perform operations on numeric columns, you're actually calling NumPy's vectorized functions. Pandas is the layer that "makes NumPy work with tabular data."



6. Complete Example: NumPy vs Pandas Full Workflow Comparison

▶ Example: Student Grade Analysis — Full Workflow (Difficulty ⭐⭐⭐)

TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
PYTHON
import numpy as np
import pandas as pd

# ============================================
# Comprehensive comparison: NumPy vs Pandas
# for student grade analysis workflow
# ============================================

# --- Raw data ---
students = ['Alice', 'Bob', 'Charlie', 'Carol', 'David']
math = [85, 92, 78, 95, 88]
english = [90, 85, 82, 88, 91]
science = [88, 90, 75, 93, 86]

# --- NumPy approach ---
scores_np = np.array([math, english, science])  # shape: (3, 5)
# Which row is which subject? Must remember: row 0=math, 1=english, 2=science
# Which column is which student? Must remember order
avg_per_student = scores_np.mean(axis=0)
print("NumPy - Average per student (by position):")
for i, avg in enumerate(avg_per_student):
    print(f"  Student {i}: {avg:.1f}")  # Who is Student 0?

# --- Pandas approach ---
df = pd.DataFrame({
    'student': students,
    'math': math,
    'english': english,
    'science': science
})
df['average'] = df[['math', 'english', 'science']].mean(axis=1)
df['grade'] = df['average'].map(
    lambda x: 'A' if x >= 90 else 'B' if x >= 85 else 'C'
)
print("\nPandas - Student grades:")
print(df[['student', 'average', 'grade']])

# Top performers
top = df[df['grade'] == 'A']
print(f"\nA-grade students: {top['student'].tolist()}")

# Subject statistics
print("\nSubject statistics:")
print(df[['math', 'english', 'science']].describe())
TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

Expected output (excerpt):

TEXT
NumPy - Average per student (by position):
  Student 0: 87.7
  Student 1: 89.0
  Student 2: 78.3

Pandas - Student grades:
   student  average grade
0    Alice     87.7     B
1      Bob     89.0     B
2  Charlie     78.3     C
3    Carol     92.0     A
4    David     88.3     B

A-grade students: ['Carol']

Subject statistics:
        math  english  science
count   5.00     5.00     5.00
mean   87.60    87.20    86.40
std     6.19     3.70     7.60
min    78.00    82.00    75.00
max    95.00    91.00    93.00

NumPy can do the computation, but the results lack "self-descriptiveness" — you have to separately remember which number corresponds to which student. Pandas keeps data and labels bound together at all times, making results immediately clear.



7. Installation and Verification

(1) Installation Methods

Method Command Use Case
pip pip install pandas Most universal
conda conda install pandas Anaconda environments
From source pip install . Developers / contributors
💡 Tip: Pandas automatically installs NumPy as a dependency. If you already have NumPy installed, Pandas will reuse the existing version.

▶ Example: Verifying Your Installation (Difficulty ⭐)

TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.
PYTHON
import pandas as pd
import numpy as np

print(f"Pandas version: {pd.__version__}")
print(f"NumPy version: {np.__version__}")

# Quick functionality check
df = pd.DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
print(df)
print(f"\nMean of x: {df['x'].mean()}")
TEXT
> Output: Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) to follow along. Actual values may vary slightly depending on your pandas version.

(2) Version Notes

Version Status Notes
Pandas 3.x Latest Released in 2026, Copy-on-Write enabled by default
Pandas 2.x Widely used Released in 2023, major performance improvements
Pandas 1.x Legacy Some syntax has been deprecated
⚠️ Note: This tutorial is written for Pandas 2.x / 3.x. Some 1.x syntax (such as append) was removed in 2.0 and is no longer used here.


❓ FAQ

Q What's the relationship between Pandas and NumPy?
A Pandas is built on top of NumPy. The numeric columns of a DataFrame are ndarrays under the hood — when you perform operations on numeric columns, you're actually calling NumPy functions. Pandas adds labeled indexing, mixed types, missing value handling, and IO tools on top of NumPy.
Q Why learn Pandas after NumPy?
A NumPy is a computation engine; Pandas is a data analysis tool. Real-world data has column names, mixed types, missing values, and multiple file formats — handling these with NumPy requires a lot of manual work, while Pandas supports them natively. 95% of data analysis work is done with Pandas; you only drop down to NumPy when you need high-performance numerical computing.
Q What's the difference between a DataFrame and Excel?
A Conceptually they're similar (both are 2D tables), but a DataFrame is programmable: automate operations with Python code, handle millions of rows, integrate with ML libraries, and use Git version control. Excel is great for manual interaction and report presentation; DataFrames are built for automated data processing pipelines.
Q Can Pandas replace SQL?
A Not entirely. Pandas excels at analytical work (exploration / cleaning / transformation), while SQL excels at query work (multi-table joins / transactions / concurrency). A common combination: use SQL to pull data from a database, then use Pandas for analysis. For datasets under a million rows, Pandas can replace most SQL analytical functionality.
Q What are the main differences between Pandas 2.x and 1.x?
A Three major changes: (1) Copy-on-Write is enabled by default, preventing accidental modification of views; (2) deprecated APIs like append and DataFrame.ix have been removed; (3) PyArrow is used under the hood for better performance. This tutorial uses 2.x/3.x syntax — 1.x users should upgrade.
Q How does Pandas perform with large data?
A Pandas is well-suited for single-machine analysis at the million-row scale. For larger data (billion-scale), consider: (1) Dask — a distributed version of the Pandas API; (2) Polars — a faster alternative library (implemented in Rust); (3) PySpark — a distributed computing framework. Pandas remains the go-to choice for learning and prototyping.
Q Why does the code say "requires a local Python environment"?
A Our online editor doesn't support running Pandas (due to memory and security constraints). Please install Python + Pandas locally to run the example code. Installation takes just one command: pip install pandas.

📖 Summary


📝 Exercises

  1. Basic (Difficulty ⭐): Install Pandas and run the verification code to confirm pd.__version__ outputs correctly. Create a DataFrame with 3 columns (name / age / city) and print it.
  2. Intermediate (Difficulty ⭐⭐): Process the same student grade data (5 students / 3 subjects) with both NumPy and Pandas. Calculate each student's average score and the highest score per subject. Compare the code volume and readability of both approaches.
  3. Challenge (Difficulty ⭐⭐⭐): Load data containing missing values from a CSV string, then use Pandas to: view info → fill missing values → group by category and compute statistics → output describe. Use only Pandas throughout — no loops allowed.

Next: Getting Started with Series →

Web-Tutorial.com

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

100%

🙏 帮我们做得更好

我们是刚上线的编程教程站,几个人的小团队,精力有限。页面虽经检查,难免还有疏漏——链接失效、排版错乱、内容有误、语言生硬……

如果您发现了,麻烦告诉我们,我们会在收到反馈后第一时间进行修复,再次感谢您的光临 🙏