Pandas: Advanced Time Series

Last updated: 2026-08-26

In Lesson 16, you learned about datetime types. This lesson dives into practical time series analysis — resample turns daily data into monthly summaries, shift/diff builds lag features, and the three window functions (rolling/expanding/ewm) make trends and volatility crystal clear. This is the core foundation for quantitative analysis and ML feature engineering.

⚠️ Note: The code below requires a local Python environment to run.

1. What You Will Learn

❶ resample — resampling
❷ shift / diff / pct_change
❸ rolling — moving window
❹ expanding — cumulative window
❺ ewm — exponential weighting

2. Alice's Coffee Shop Monthly Report

(1) The Problem: Daily Data Is Too Granular — Monthly Aggregation Needed

Alice has 90 days of daily sales data, and her boss wants a monthly report:

PYTHON

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'sales': np.random.poisson(50, 90) * 10,
    'customers': np.random.poisson(30, 90)
}, index=pd.date_range('2024-01-01', periods=90))
# Daily data → need monthly summary

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

(2) The Solution: One-Line Aggregation with resample

▶ Example

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: resample monthly aggregation (Difficulty ⭐)

PYTHON

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'sales': np.random.poisson(50, 90) * 10,
    'customers': np.random.poisson(30, 90)
}, index=pd.date_range('2024-01-01', periods=90))

# Resample daily → monthly
monthly = df.resample('M').agg({
    'sales': 'sum',
    'customers': 'mean'
})
print(monthly)
#              sales  customers
# 2024-01-31   14680  29.55
# 2024-02-29   13520  31.10
# 2024-03-31   15430  30.45

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

3. resample — Resampling

(1) Downsampling and Upsampling

▶ Example

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: resample downsampling/upsampling (Difficulty ⭐⭐)

PYTHON

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'price': np.random.uniform(100, 200, 60).round(2)
}, index=pd.date_range('2024-01-01', periods=60))

# Downsample: daily → weekly
weekly = df.resample('W').agg({
    'price': ['first', 'max', 'min', 'last']  # OHLC
})
print("Weekly OHLC:")
print(weekly.head())

# Downsample: daily → quarterly
quarterly = df.resample('Q').mean()

# Upsample: daily → hourly (need fill method)
hourly = df.resample('h').ffill()  # forward fill
# Warning: creates many rows!

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

(2) resample vs groupby

Feature	resample	groupby
Grouping basis	Time frequency	Any column
Works with	DatetimeIndex	Any DataFrame
Upsampling	✅ Fills missing values	❌
Frequency aliases	D/W/M/Q/H	—
Essence	groupby along the time dimension	General-purpose grouping

4. shift / diff / pct_change

(1) Lag and Difference

▶ Example

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: shift/diff/pct_change (Difficulty ⭐⭐)

PYTHON

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'sales': np.random.poisson(50, 10) * 10
}, index=pd.date_range('2024-01-01', periods=10))

# shift: move data forward/backward
df['sales_prev'] = df['sales'].shift(1)    # yesterday's sales
df['sales_next'] = df['sales'].shift(-1)   # tomorrow's sales

# diff: difference with previous value
df['sales_diff'] = df['sales'].diff()       # sales - prev_sales

# pct_change: percentage change
df['sales_pct'] = df['sales'].pct_change() * 100  # % change

print(df.round(1))

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

💡 Tip: shift is the cornerstone of ML time series feature engineering — df['lag_1'] = df['value'].shift(1) uses yesterday's value to predict today. diff extracts trend changes, and pct_change extracts growth rates.

5. rolling — Moving Window

(1) Moving Average

▶ Example

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: rolling moving average (Difficulty ⭐⭐)

PYTHON

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'temperature': np.round(np.random.normal(20, 5, 30), 1)
}, index=pd.date_range('2024-01-01', periods=30))

# 7-day moving average
df['ma_7'] = df['temperature'].rolling(window=7).mean()

# With min_periods (allow partial windows)
df['ma_7_flex'] = df['temperature'].rolling(window=7, min_periods=3).mean()

# Multiple aggregations
# Pandas 2.x: rolling.agg() returns a multi-column DataFrame and cannot be assigned to a single column directly; use join instead
rolling_agg = df['temperature'].rolling(7).agg(['mean', 'std', 'min', 'max'])
rolling_agg.columns = [f'temp_{stat}' for stat in ['mean', 'std', 'min', 'max']]
df = df.join(rolling_agg)

# Centered rolling (include future values — for smoothing, not forecasting)
df['ma_7_center'] = df['temperature'].rolling(7, center=True).mean()

print(df[['temperature', 'ma_7', 'ma_7_center']].head(10))

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

(2) Time-Based Windows

▶ Example

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: time-based rolling window (Difficulty ⭐⭐)

PYTHON

import pandas as pd
import numpy as np

# Irregular time series — row-based window is wrong
df = pd.DataFrame({
    'value': [10, 20, 15, 30, 25, 40, 35, 50, 45, 60]
}, index=pd.to_datetime([
    '2024-01-01', '2024-01-02', '2024-01-05',  # gap on 03-04
    '2024-01-06', '2024-01-07', '2024-01-15',  # gap on 08-14
    '2024-01-16', '2024-01-20', '2024-01-21', '2024-01-25'
]))

# Row-based window (wrong for irregular data)
print("Row-based 3:")
print(df['value'].rolling(3).mean())

# Time-based window (correct — 7 calendar days)
print("\nTime-based 7D:")
print(df['value'].rolling('7D').mean())

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

6. expanding — Cumulative Window

▶ Example

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: expanding cumulative statistics (Difficulty ⭐)

PYTHON

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'sales': np.random.poisson(50, 10) * 10
})

# Expanding: all values up to current position
df['cumsum'] = df['sales'].expanding().sum()
df['cummean'] = df['sales'].expanding().mean()
df['cummax'] = df['sales'].expanding().max()

# With min_periods
df['cummean_3'] = df['sales'].expanding(min_periods=3).mean()

print(df)

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

(1) rolling vs expanding vs ewm Comparison

Feature	rolling	expanding	ewm
Window	Fixed size	Ever-growing	Infinite (weights decay)
Weights	Equal	Equal	Recent values weighted more
Use case	Local trends	Cumulative statistics	Smoothed trends
Parameters	window	min_periods	span/com/halflife

7. ewm — Exponential Weighting

▶ Example

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: ewm exponential smoothing (Difficulty ⭐⭐)

PYTHON

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'price': np.cumsum(np.random.randn(30)) + 100
}, index=pd.date_range('2024-01-01', periods=30))

# Exponentially weighted moving average
df['ewm_5'] = df['price'].ewm(span=5).mean()
df['ewm_10'] = df['price'].ewm(span=10).mean()
df['ewm_20'] = df['price'].ewm(span=20).mean()

# Compare with simple moving average
df['sma_5'] = df['price'].rolling(5).mean()

print(df[['price', 'sma_5', 'ewm_5']].head(10).round(2))

# ewm parameters (only specify one):
# span=5     → similar to 5-period SMA but recent-heavy
# com=4      → center of mass (com = span-1)
# halflife=3 → weight halves every 3 periods
# alpha=0.3  → smoothing factor (0<α≤1)

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

8. Full Example: Daily Stock Candlestick Analysis

(5) ▶ rolling Window Sliding Along the Time Axis

graph TB
    A[Time Series Data] --> B[rolling Window]
    B --> C[Window slides along the time axis]
    C --> D[Window 1: t1 to tN]
    C --> E[Window 2: t2 to t(N+1)]
    C --> F[Window 3: t3 to t(N+2)]
    D --> G[mean / std / max / min]
    E --> G
    F --> G
    G --> H[Smoothed trend / Volatility]

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

▶ Example

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: full time series analysis pipeline (Difficulty ⭐⭐⭐)

PYTHON

import pandas as pd
import numpy as np

# ============================================
# Comprehensive example: Stock daily data
# resample → rolling → diff → ewm
# ============================================

# 1. Generate daily stock data (60 trading days)
np.random.seed(42)
dates = pd.bdate_range('2024-01-01', periods=60)
price = 100 + np.cumsum(np.random.randn(60) * 2)
df = pd.DataFrame({
    'open': price + np.random.randn(60) * 0.5,
    'high': price + np.abs(np.random.randn(60)) * 1.5,
    'low': price - np.abs(np.random.randn(60)) * 1.5,
    'close': price + np.random.randn(60) * 0.5,
    'volume': np.random.randint(100000, 500000, 60)
}, index=dates)

# 2. Resample: daily → weekly
weekly = df.resample('W-FRI').agg({
    'open': 'first', 'high': 'max', 'low': 'min',
    'close': 'last', 'volume': 'sum'
})
print("=== Weekly OHLCV ===")
print(weekly.head(4).round(2))

# 3. Rolling: 20-day moving average
df['ma_20'] = df['close'].rolling(20).mean()
df['vol_20'] = df['close'].rolling(20).std()

# 4. Daily returns
df['returns'] = df['close'].pct_change()
df['log_returns'] = np.log(df['close'] / df['close'].shift(1))

# 5. EWM: exponential smoothing
df['ewm_12'] = df['close'].ewm(span=12).mean()
df['ewm_26'] = df['close'].ewm(span=26).mean()
df['macd'] = df['ewm_12'] - df['ewm_26']

# 6. Summary
print(f"\n=== Statistics ===")
print(f"Period: {df.index[0].date()} to {df.index[-1].date()}")
print(f"Total return: {((df['close'].iloc[-1] / df['close'].iloc[0] - 1) * 100):.1f}%")
print(f"Volatility (20d): {df['vol_20'].iloc[-1]:.2f}")
print(f"Max drawdown: {((df['close'] / df['close'].cummax() - 1).min() * 100):.1f}%")

TEXT 📖 Display only

> **Output:** Run in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed. Please install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

❓ FAQ

Q What is the difference between resample and groupby?

A resample is essentially groupby along the time dimension — it groups by frequency (daily/weekly/monthly/quarterly) and only works with DatetimeIndex/PeriodIndex. groupby groups by any column and is not limited to time. resample uniquely supports upsampling (low frequency → high frequency requires filling), which groupby cannot do. Rule of thumb: use resample for time-based aggregation, groupby for everything else.

Q How do I choose the rolling window size?

A It depends on the scenario. For short-term trends, use a window of 5-10; for medium-term, 20-30 (monthly); for long-term, 50-200 (quarterly/yearly). In finance, 20/50/200-day moving averages are standard. Too small a window → too much noise; too large → excessive lag. Recommendation: look at your data frequency first, and set the window to cover 1-3 "natural cycles."

Q What is the difference between shift and diff?

A shift moves data without computing anything; diff calculates the difference. s.diff() = s - s.shift(1). shift is used to build lag features (ML), while diff extracts the amount of change (trend analysis). pct_change calculates the rate of change: s.pct_change() = (s - s.shift(1)) / s.shift(1).

Q What is the difference between expanding and cumsum?

A expanding is a generalized version of cumsum — it supports any aggregation function (mean/std/max/min), while cumsum only computes the cumulative sum. s.expanding().sum() is equivalent to s.cumsum(). expanding is more flexible but slightly slower; for everyday use, cumsum is more concise.

Q What is ewm?

A EWM (Exponentially Weighted Moving) is an exponentially weighted moving average — recent data gets higher weight, and older data decays exponentially. span=5 behaves similarly to a 5-period SMA but gives more weight to recent values. It responds to changes faster than SMA and is the foundation of technical analysis (MACD) and online learning.

Q How do I fill values after upsampling with resample?

A Upsampling (low frequency → high frequency) produces many NaN values. Fill options: ffill() (forward fill, uses the previous value), bfill() (backward fill), interpolate() (interpolation), asfreq() (keeps NaN). ffill is recommended (simple and conservative); for time series analysis, use interpolate (smoother results).

Q Can center=True be used for forecasting?

A Absolutely not! center=True makes the window "look into the future" (includes subsequent values), so it is only suitable for smoothing historical data — never for forecasting. In real-time or forecasting scenarios, you must use center=False (the default), which only looks at past values. This mistake causes severe data leakage in quantitative analysis.

📖 Summary

resample aggregates by time frequency — it is "groupby along the time dimension"
shift moves data to build lag features; diff/pct_change compute differences and rates of change
rolling computes over a fixed-size moving window (moving average/standard deviation/extremes)
Time-based rolling('7D') is ideal for irregular time series
expanding is a cumulative window (from the start to the current position); ewm applies exponential weighting (recent values weighted more)
Choosing: local trends → rolling, cumulative statistics → expanding, smoothing → ewm

📝 Exercises

Basic (Difficulty ⭐): Create a 30-day daily sales DataFrame. Use resample('W') to aggregate into weekly data, and rolling(7) to compute a 7-day moving average.
Intermediate (Difficulty ⭐⭐): Create 60 days of stock closing prices. Compute: shift(1) lag → diff() daily difference → pct_change() returns → rolling(20) volatility → ewm(span=12) smoothing.
Challenge (Difficulty ⭐⭐⭐): Simulate 90 days of daily sales. Complete: resample monthly/quarterly summaries → rolling 7/14/30-day moving average comparison → expanding cumulative mean → ewm smoothing → maximum drawdown calculation.

← Previous Lesson: Date and Time · Next Lesson: MultiIndex →

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

About Us

Pandas: Advanced Time Series

1. What You Will Learn

2. Alice's Coffee Shop Monthly Report

(1) The Problem: Daily Data Is Too Granular — Monthly Aggregation Needed

(2) The Solution: One-Line Aggregation with resample

▶ Example

3. resample — Resampling

(1) Downsampling and Upsampling

▶ Example

(2) resample vs groupby

4. shift / diff / pct_change

(1) Lag and Difference

▶ Example

5. rolling — Moving Window

(1) Moving Average

▶ Example

(2) Time-Based Windows

▶ Example

6. expanding — Cumulative Window

▶ Example

(1) rolling vs expanding vs ewm Comparison

7. ewm — Exponential Weighting

▶ Example

8. Full Example: Daily Stock Candlestick Analysis

(5) ▶ rolling Window Sliding Along the Time Axis

▶ Example

❓ FAQ

📖 Summary

📝 Exercises

🙏 帮我们做得更好