Pandas: String Operations

Q: How do I extract digits from a phone number?

Use s.str.replace(r'[^0-9]', '', regex=True) to strip all non-digit characters. Alternatively, use extract: s.str.extract(r'(\d{3})[-.)\s]*(\d{3})[-.)\s]*(\d{4})') to capture the area code, prefix, and line number separately. Regular expressions are the core tool for text extraction.

Last updated: 2026-08-26

Text in real-world data is never clean — extra whitespace, inconsistent casing, special characters, mixed formats. The Pandas .str accessor brings vectorization to string operations: clean an entire column of text in a single line of code, up to 100x faster than a Python loop. This section covers the most commonly used string methods and shows you how to build chained cleaning pipelines.

⚠️ Note: The code below requires a local Python environment to run.

1. What You Will Learn

❶ The .str accessor
❷ Common methods (strip/lower/upper/title/replace)
❸ Regular expressions
❹ split / extract
❺ contains / startswith / match

2. Carol's User Input Cleanup

(1) The Problem: User Input Comes in All Shapes

The user data Carol collected is a mess:

PYTHON

import pandas as pd

df = pd.DataFrame({
    'name': ['  alice  ', 'BOB', 'Charlie!', '  carol  ', 'DaVid123'],
    'email': ['alice@EXAMPLE.com', 'bob@company.ORG', 'charlie@uni.edu',
              'CAROL@example.COM', 'david@test.net'],
    'phone': ['555-0100', '(555) 0200', '555.0300', '+1-555-0400', '5550500']
})

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

(2) The Solution: Chained .str Cleaning

▶ Example: Chained Text Cleaning (Difficulty ⭐)

PYTHON

import pandas as pd

df = pd.DataFrame({
    'name': ['  alice  ', 'BOB', 'Charlie!', '  carol  ', 'DaVid123'],
    'email': ['alice@EXAMPLE.com', 'bob@company.ORG', 'charlie@uni.edu',
              'CAROL@example.COM', 'david@test.net']
})

# Chain .str methods: strip → replace → title
df['name_clean'] = (df['name']
    .str.strip()
    .str.replace(r'[^a-zA-Z ]', '', regex=True)
    .str.title()
)
print(df[['name', 'name_clean']])
#          name name_clean
# 0    alice        Alice
# 1         BOB        Bob
# 2   Charlie!    Charlie
# 3     carol        Carol
# 4   DaVid123       David

# Lowercase email domain
df['email_clean'] = df['email'].str.lower()
print(df['email_clean'])

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

3. The .str Accessor

(1) .str vs Python str

Feature	Python str	Pandas .str
Operates on	A single string	An entire column (vectorized)
NaN handling	Raises an error	Skips automatically (returns NaN)
Chaining	Not supported	✅ Consecutive calls
Performance	Loop → slow	Vectorized → fast

▶ Example

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: Basic .str Methods (Difficulty ⭐)

PYTHON

import pandas as pd

s = pd.Series(['Hello World', 'pandas', 'DATA SCIENCE', None, 'python'])

# Case conversion
print(s.str.lower())    # hello world, pandas, data science, NaN, python
print(s.str.upper())    # HELLO WORLD, PANDAS, DATA SCIENCE, NaN, PYTHON
print(s.str.title())    # Hello World, Pandas, Data Science, NaN, Python
print(s.str.capitalize())  # Hello world, Pandas, Data science, NaN, Python

# Whitespace
s2 = pd.Series(['  hello  ', '\tworld\n', '  pandas  '])
print(s2.str.strip())    # hello, world, pandas
print(s2.str.lstrip())   # hello  , world\n, pandas  
print(s2.str.rstrip())   #   hello, \tworld,   pandas

# Length
print(s.str.len())       # 11, 6, 12, NaN, 6

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

4. replace / contains / match

(1) replace

▶ Example

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: String Replacement with replace (Difficulty ⭐⭐)

PYTHON

import pandas as pd

s = pd.Series(['price: $100', 'price: $200', 'cost: €50'])

# Simple replacement
print(s.str.replace('$', 'USD'))
# price: USD100, price: USD200, cost: €50

# Regex replacement
print(s.str.replace(r'[\$€]', '', regex=True))
# price: 100, price: 200, cost: 50

# Replace multiple patterns
phones = pd.Series(['555-0100', '(555) 0200', '555.0300'])
clean = phones.str.replace(r'[^0-9]', '', regex=True)
print(clean)  # 5550100, 5550200, 5550300

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

(2) contains / startswith / endswith

▶ Example

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: Filtering Strings by Condition (Difficulty ⭐⭐)

PYTHON

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice Johnson', 'Bob Smith', 'Charlie Brown', 'Carol White'],
    'email': ['alice@example.com', 'bob@company.org', 'charlie@uni.edu', 'carol@test.net']
})

# Contains substring
has_com = df[df['email'].str.contains('.com')]
print(has_com['name'])  # Alice Johnson

# Startswith / endswith
starts_with_a = df[df['name'].str.startswith('A')]
print(starts_with_a)

edu_email = df[df['email'].str.endswith('.edu')]
print(edu_email['name'])  # Charlie Brown

# Regex contains — find org or edu domains
org_or_edu = df[df['email'].str.contains(r'\.(org|edu)$', regex=True)]
print(org_or_edu[['name', 'email']])

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

(3) contains vs match

Method	Match Position	Regex
contains	Anywhere in the string	✅
match	From the beginning	✅
startswith	From the beginning	❌ (plain string only)
endswith	From the end	❌ (plain string only)

5. split / extract

(1) split

▶ Example

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: Splitting Strings with split (Difficulty ⭐⭐)

PYTHON

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice Johnson', 'Bob Smith', 'Charlie Brown'],
    'email': ['alice@example.com', 'bob@company.org', 'charlie@uni.edu']
})

# Split into list
print(df['name'].str.split(' '))
# 0    [Alice, Johnson]
# 1    [Bob, Smith]
# 2    [Charlie, Brown]

# Split into separate columns
name_split = df['name'].str.split(' ', expand=True)
print(name_split)
#         0        1
# 0   Alice  Johnson
# 1     Bob    Smith
# 2 Charlie    Brown

# Split and keep only first part
df['first_name'] = df['name'].str.split(' ').str[0]
print(df['first_name'])

# Split email to get domain
df['domain'] = df['email'].str.split('@').str[1]
print(df['domain'])

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

(2) extract with Regex

▶ Example

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: Extracting Patterns with extract (Difficulty ⭐⭐⭐)

PYTHON

import pandas as pd

df = pd.DataFrame({
    'contact': [
        'Alice <alice@example.com>',
        'Bob <bob@company.org>',
        'Charlie (555-0100)',
        'Carol <carol@test.net>'
    ]
})

# Extract email with regex group
df['email'] = df['contact'].str.extract(r'<(.+?)>')
print(df[['contact', 'email']])
#                   contact             email
# 0  Alice <alice@example.com>  alice@example.com
# 1      Bob <bob@company.org>    bob@company.org
# 2     Charlie (555-0100)                NaN
# 3    Carol <carol@test.net>    carol@test.net

# Extract multiple groups
df2 = pd.DataFrame({
    'email': ['alice@example.com', 'bob@company.org']
})
parts = df2['email'].str.extract(r'(.+?)@(.+)')
print(parts)
#        0             1
# 0  alice  example.com
# 1    bob   company.org

# Extract all matches
colors = pd.Series(['red,blue,green', 'yellow,pink'])
print(colors.str.extractall(r'(\w+)'))

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

6. Other Common Methods

(1) Quick Reference

Method	Purpose	Example
len()	String length	`s.str.len()`
count()	Number of substring occurrences	`s.str.count('a')`
find()	Position of substring	`s.str.find('bc')`
isdigit()	All digits?	`s.str.isdigit()`
isalpha()	All letters?	`s.str.isalpha()`
join()	Join with separator	`s.str.join(',')`
pad()	Pad to fixed width	`s.str.pad(10, fillchar='0')`
repeat()	Repeat string	`s.str.repeat(3)`
slice()	Slice	`s.str.slice(0, 5)`

▶ Example

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: pad/slice/zfill (Difficulty ⭐)

PYTHON

import pandas as pd

ids = pd.Series(['1', '23', '456', '7890'])

# Pad with zeros to 4 digits
print(ids.str.zfill(4))
# 0    0001
# 1    0023
# 2    0456
# 3    7890

# Slice first 2 characters
codes = pd.Series(['US-001', 'UK-002', 'JP-003'])
print(codes.str.slice(0, 2))  # US, UK, JP
print(codes.str[:2])           # same, Python slice syntax

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

7. Full Example: User Data Text Cleaning Pipeline

(5) ▶ Text Cleaning Workflow

graph TB
    A[Raw Text] --> B[strip: remove whitespace]
    B --> C[lower / upper: normalize case]
    C --> D[replace: remove special characters]
    D --> E[split / extract: extract information]
    E --> F[title: format output]
    F --> G[Clean Text]

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

▶ Example

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

: End-to-End Text Cleaning (Difficulty ⭐⭐⭐)

PYTHON

import pandas as pd

# ============================================
# Comprehensive example: Text cleaning
# pipeline for user data
# ============================================

# 1. Raw messy user data
df = pd.DataFrame({
    'raw_name': ['  ALICE johnson  ', 'bob   SMITH', '  CHARLIE! brown  ', 'CAROL white123'],
    'raw_email': ['ALICE@Example.COM', 'bob@company.ORG', 'charlie@uni.EDU', 'CAROL@TEST.NET'],
    'raw_phone': ['555-0100', '(555) 0200', '555.0300 ext 42', '+1-555-0400'],
    'raw_zip': ['0123', '12345', '  90210  ', '10001-2345']
})

# 2. Name cleaning: strip → remove special chars → title case
df['name'] = (df['raw_name']
    .str.strip()
    .str.replace(r'[^a-zA-Z\s]', '', regex=True)
    .str.replace(r'\s+', ' ', regex=True)  # collapse multiple spaces
    .str.title()
)

# 3. Email cleaning: lowercase
df['email'] = df['raw_email'].str.lower()

# 4. Phone cleaning: keep only digits
df['phone_digits'] = df['raw_phone'].str.replace(r'[^0-9]', '', regex=True)
df['phone_formatted'] = df['phone_digits'].str.slice(0, 3) + '-' + df['phone_digits'].str.slice(3, 7)

# 5. Zip code: extract 5-digit code
df['zip5'] = df['raw_zip'].str.strip().str.extract(r'(\d{5})')

# 6. Email domain
df['domain'] = df['email'].str.split('@').str[1]

# 7. Show cleaned results
print("=== Cleaned Data ===")
print(df[['name', 'email', 'phone_formatted', 'zip5', 'domain']])

TEXT 📖 Display only

> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas pre-installed — install it locally (`pip install pandas`) and follow along. Actual values may vary slightly depending on your pandas version.

❓ FAQ

Q What is the difference between .str and Python str?

A Python str methods operate on a single string, while the .str accessor operates on an entire column (vectorized). The .str accessor automatically skips NaN values (returning NaN), whereas Python str raises an error on NaN. The .str accessor supports method chaining (s.str.strip().str.lower()), which Python str does not. In terms of performance, .str is 10-100x faster than looping.

Q How are NaN values handled?

A The .str methods automatically skip NaN — if the input is NaN, the output is NaN. There is no need to manually check for missing values. This is one of the reasons .str is more convenient than using apply + Python str. Note that contains/match return False (not NaN) for NaN values, so keep this in mind when filtering.

Q What does extract return?

A extract returns a DataFrame — one column per regex group. A single group yields a 1-column DataFrame; multiple groups yield multiple columns. extractall returns all matches (with a MultiIndex). Note: rows with no match return NaN. If you need a Series, use .str.extract(...)[0].

Q What is the difference between contains and match?

A contains matches anywhere in the string (e.g., 'abc' contains 'b' → True). match only matches from the beginning of the string (e.g., 'abc' match 'b' → False). startswith/endswith are plain string comparisons (no regex support). Use contains/match when you need regex, and startswith/endswith for simple text checks.

Q What are the benefits of StringDtype?

A By default, string columns use the object dtype (Python objects), where each value is a separate str instance. StringDtype (df['col'].astype('string')) is a dedicated Pandas string type that uses pd.NA for missing values, is more memory-efficient, and provides more consistent .str behavior. Pandas 2.x recommends StringDtype over object for string data.

Q What is the difference between str.replace and df.replace?

A str.replace operates only on string columns, supports regex (regex=True), and replaces element by element. df.replace operates on the entire DataFrame, can replace values of any type, and also supports regex. For string replacement, prefer str.replace (clearer semantics); for cross-column replacement, use df.replace.

Q How do I extract digits from a phone number?

A Use s.str.replace(r'[^0-9]', '', regex=True) to strip all non-digit characters. Alternatively, use extract: s.str.extract(r'(\d{3})[-.)\s]*(\d{3})[-.)\s]*(\d{4})') to capture the area code, prefix, and line number separately. Regular expressions are the core tool for text extraction.

📖 Summary

The .str accessor provides vectorized string operations and handles NaN automatically
strip/lstrip/rstrip remove whitespace; lower/upper/title/capitalize convert case
replace performs substitution (with regex support); contains filters by condition (with regex support)
split breaks strings apart (expand=True creates separate columns); extract pulls out regex matches
match checks from the beginning; contains checks anywhere; startswith/endswith are plain-text only
Chain methods to build cleaning pipelines: strip → replace → lower/title
StringDtype replaces object for more efficient string storage

📝 Exercises

Basic (Difficulty ⭐): Create a Series of names with extra whitespace and inconsistent casing. Clean it with .str.strip().str.title(), then count how many cleaned names start with 'A'.
Intermediate (Difficulty ⭐⭐): Create a Series of phone numbers in mixed formats. Use replace + regex to extract only the digits, then format them to 10 characters with zfill.
Challenge (Difficulty ⭐⭐⭐): Simulate user registration data (name/email/phone/zip all with formatting issues). Build a 5-step chained cleaning pipeline — one .str method per step — and output the final clean table.

← Previous: Reshaping and Pivoting · Next: Date and Time →

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

About Us

Pandas: String Operations

1. What You Will Learn

2. Carol's User Input Cleanup

(1) The Problem: User Input Comes in All Shapes

(2) The Solution: Chained .str Cleaning

▶ Example: Chained Text Cleaning (Difficulty ⭐)

3. The .str Accessor

(1) .str vs Python str

▶ Example

4. replace / contains / match

(1) replace

▶ Example

(2) contains / startswith / endswith

▶ Example

(3) contains vs match

5. split / extract

(1) split

▶ Example

(2) extract with Regex

▶ Example

6. Other Common Methods

(1) Quick Reference

▶ Example

7. Full Example: User Data Text Cleaning Pipeline

(5) ▶ Text Cleaning Workflow

▶ Example

❓ FAQ

📖 Summary

📝 Exercises

🙏 帮我们做得更好