Pandas: Data Selection
Last updated: 2026-08-26
Data selection is the most frequent operation in data analysis—"show me the data that meets these conditions." Pandas offers a rich set of selection methods: by label, by position, by condition, by expression, and even SQL-like query strings. This section walks you through every selection method and helps you avoid the most common "chained indexing" pitfall.
1. What You Will Learn
- ❶ loc label selection
- ❷ iloc position selection
- ❸ Conditional filtering (boolean indexing)
- ❹ The query method
- ❺ isin filtering and the chained indexing pitfall
2. Charlie's Product Filtering Needs
(1) The Pain Point: Combining Multiple Conditions Is Too Cumbersome
Charlie needs to filter "Electronics products with price > 50 and rating > 4.0" out of 1000 products. He first wrote a long string of conditions using boolean indexing:
import pandas as pd
df = pd.DataFrame({
'name': ['Laptop', 'Phone', 'Tablet', 'Monitor', 'Keyboard',
'Mouse', 'Headset', 'Speaker', 'Camera', 'Charger'],
'category': ['Electronics', 'Electronics', 'Electronics', 'Electronics',
'Accessories', 'Accessories', 'Accessories', 'Accessories',
'Electronics', 'Accessories'],
'price': [999.99, 699.99, 349.99, 449.99, 79.99,
29.99, 149.99, 89.99, 599.99, 19.99],
'stock': [50, 120, 80, 35, 200, 300, 150, 180, 25, 500],
'rating': [4.7, 4.5, 4.3, 4.6, 4.2, 4.0, 4.4, 4.1, 4.8, 3.9]
})
# Boolean indexing — works but verbose
result = df[(df['price'] > 50) & (df['rating'] > 4.0) & (df['category'] == 'Electronics')]
print(result[['name', 'price', 'rating']])
# name price rating
# 0 Laptop 999.99 4.7
# 1 Phone 699.99 4.5
# 2 Tablet 349.99 4.3
# 3 Monitor 449.99 4.6
# 8 Camera 599.99 4.8
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
(2) The Solution: The query Method Is More Readable
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: Readable queries with query (Difficulty ⭐⭐)
# Same result, much more readable
result = df.query('price > 50 and rating > 4.0 and category == "Electronics"')
print(result[['name', 'price', 'rating']])
# With variable
min_price = 50
min_rating = 4.0
result2 = df.query('price > @min_price and rating > @min_rating')
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
(3) The Benefit: Four Selection Methods, Each with Its Strengths
| Method | Syntax | Best For |
|---|---|---|
| loc | df.loc[row, col] |
Precise label selection |
| iloc | df.iloc[r, c] |
Position-based selection |
| Boolean indexing | df[condition] |
Simple conditions |
| query | df.query('expr') |
Complex multi-condition filters |
3. loc: Label Selection
(1) loc Syntax
df.loc[row_label, col_label] — selects by label; slices include the endpoint.
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: loc label selection (Difficulty ⭐)
import pandas as pd
df = pd.DataFrame({
'name': ['Laptop', 'Phone', 'Tablet', 'Monitor', 'Keyboard'],
'price': [999.99, 699.99, 349.99, 449.99, 79.99],
'stock': [50, 120, 80, 35, 200]
}, index=['A001', 'A002', 'A003', 'A004', 'A005'])
# Single row by label
print(df.loc['A001'])
# name Laptop
# price 999.99
# stock 50
# Multiple rows
print(df.loc[['A001', 'A003']])
# Row slice + column slice (labels, INCLUDES endpoint)
print(df.loc['A001':'A003', 'name':'price'])
# name price
# A001 Laptop 999.99
# A002 Phone 699.99
# A003 Tablet 349.99
# Specific rows + specific columns
print(df.loc[['A001', 'A004'], ['name', 'stock']])
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
(2) Conditional Selection with loc
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: loc + condition (Difficulty ⭐⭐)
import pandas as pd
df = pd.DataFrame({
'name': ['Laptop', 'Phone', 'Tablet', 'Monitor', 'Keyboard'],
'price': [999.99, 699.99, 349.99, 449.99, 79.99],
'stock': [50, 120, 80, 35, 200]
})
# Filter rows + select columns in one operation
expensive = df.loc[df['price'] > 400, ['name', 'price']]
print(expensive)
# name price
# 0 Laptop 999.99
# 1 Phone 699.99
# 3 Monitor 449.99
# With assignment (safe — no chained indexing)
df.loc[df['stock'] < 50, 'stock'] = 50 # restock low items
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
4. iloc: Position Selection
(1) iloc Syntax
df.iloc[row_pos, col_pos] — selects by integer position; slices exclude the endpoint (Python style).
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: iloc position selection (Difficulty ⭐)
import pandas as pd
df = pd.DataFrame({
'name': ['Laptop', 'Phone', 'Tablet', 'Monitor', 'Keyboard'],
'price': [999.99, 699.99, 349.99, 449.99, 79.99],
'stock': [50, 120, 80, 35, 200]
})
# Single row by position
print(df.iloc[0]) # first row (Laptop)
# Multiple rows by position list
print(df.iloc[[0, 2, 4]])
# Row slice + column slice (positions, EXCLUDES endpoint)
print(df.iloc[0:3, 0:2])
# name price
# 0 Laptop 999.99
# 1 Phone 699.99
# 2 Tablet 349.99
# Last 2 rows
print(df.iloc[-2:])
# Every other row
print(df.iloc[::2])
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
(2) Key Differences Between loc and iloc
| Feature | loc | iloc |
|---|---|---|
| Index basis | Label | Integer position |
| Slice endpoint | Includes endpoint | Excludes endpoint |
| String labels | ✅ | ❌ |
| Negative indices | ❌ | ✅ |
| Conditional filtering | ✅ | ❌ |
| Recommended scenario | When you know the name | When you know the position |
df.loc[0:3] includes rows 0,1,2,3 (4 rows), while df.iloc[0:3] includes rows 0,1,2 (3 rows). This is the most common point of confusion between loc and iloc.
5. Conditional Filtering (Boolean Indexing)
(1) Single and Multiple Conditions
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: Conditional filtering (Difficulty ⭐⭐)
import pandas as pd
df = pd.DataFrame({
'name': ['Laptop', 'Phone', 'Tablet', 'Monitor', 'Keyboard'],
'category': ['Electronics', 'Electronics', 'Electronics', 'Electronics', 'Accessories'],
'price': [999.99, 699.99, 349.99, 449.99, 79.99],
'stock': [50, 120, 80, 35, 200],
'rating': [4.7, 4.5, 4.3, 4.6, 4.2]
})
# Single condition
print(df[df['price'] > 400])
# Multiple conditions: & (AND), | (OR), ~ (NOT)
# MUST use parentheses around each condition!
high_price_good_rating = df[(df['price'] > 400) & (df['rating'] >= 4.5)]
print(high_price_good_rating[['name', 'price', 'rating']])
# OR condition
hot_items = df[(df['price'] > 500) | (df['stock'] > 100)]
print(hot_items[['name', 'price', 'stock']])
# NOT condition
not_electronics = df[~(df['category'] == 'Electronics')]
print(not_electronics)
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
& / | / ~, not and / or / not. Each condition must be wrapped in parentheses, otherwise operator precedence will cause an error.
(2) String Conditions
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: String condition filtering (Difficulty ⭐)
import pandas as pd
df = pd.DataFrame({
'name': ['Wireless Mouse', 'USB Keyboard', 'Bluetooth Headset',
'Wireless Speaker', 'USB Camera'],
'price': [29.99, 79.99, 149.99, 89.99, 599.99]
})
# Contains substring
wireless = df[df['name'].str.contains('Wireless')]
print(wireless)
# Starts with
usb = df[df['name'].str.startswith('USB')]
print(usb)
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
6. The query Method
(1) Basic Syntax
df.query('expression') — filters rows using a string expression, offering readability closer to SQL.
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: query expression queries (Difficulty ⭐⭐)
import pandas as pd
df = pd.DataFrame({
'name': ['Laptop', 'Phone', 'Tablet', 'Monitor', 'Keyboard'],
'category': ['Electronics', 'Electronics', 'Electronics', 'Electronics', 'Accessories'],
'price': [999.99, 699.99, 349.99, 449.99, 79.99],
'stock': [50, 120, 80, 35, 200],
'rating': [4.7, 4.5, 4.3, 4.6, 4.2]
})
# Simple condition
print(df.query('price > 400'))
# Multiple conditions (and/or instead of &/|)
print(df.query('price > 400 and rating >= 4.5'))
# String comparison
print(df.query('category == "Electronics"'))
# Use @ to reference Python variables
threshold = 100
print(df.query('stock > @threshold'))
# Column names with spaces (backtick)
# df.query('`unit price` > 100')
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
(2) query vs. Boolean Indexing
| Feature | Boolean Indexing | query |
|---|---|---|
| Syntax | df[(df.a>1) & (df.b<2)] |
df.query('a>1 and b<2') |
| Readability | Nested parentheses | Close to SQL |
| Variable reference | Used directly | Requires the @ prefix |
| Column names with spaces | No problem | Requires backticks |
| Performance | Slightly slower (large DF) | Slightly faster (engine optimized) |
| Flexibility | Highest | Limited by expression syntax |
7. isin Filtering
(1) Multi-Value Matching
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: isin multi-value filtering (Difficulty ⭐)
import pandas as pd
df = pd.DataFrame({
'name': ['Laptop', 'Phone', 'Tablet', 'Monitor', 'Keyboard'],
'category': ['Electronics', 'Electronics', 'Electronics', 'Electronics', 'Accessories'],
'price': [999.99, 699.99, 349.99, 449.99, 79.99]
})
# Filter by multiple category values
target_cats = ['Electronics', 'Audio']
# Note: Audio not in data → no match, no error
result = df[df['category'].isin(target_cats)]
print(result[['name', 'category']])
# Negation: NOT in list
other = df[~df['category'].isin(['Electronics'])]
print(other)
# Only Accessories
# isin with numeric values
target_prices = [999.99, 79.99]
print(df[df['price'].isin(target_prices)])
# Laptop and Keyboard
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
(2) isin vs. == for Multiple Values
| Method | Syntax | Use Case |
|---|---|---|
| isin | df[col].isin([v1,v2,v3]) |
Multi-value matching |
| == | df[col] == v1 |
Single-value matching |
| | combination | (df[col]==v1) | (df[col]==v2) |
A few multiple values |
| query | df.query('col in ["v1","v2"]') |
Readable multi-value matching |
8. The Chained Indexing Pitfall
(1) What Is Chained Indexing
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: The hidden danger of chained indexing (Difficulty ⭐⭐)
import pandas as pd
df = pd.DataFrame({
'name': ['Laptop', 'Phone', 'Tablet'],
'price': [999.99, 699.99, 349.99],
'category': ['Electronics', 'Electronics', 'Electronics']
})
# ❌ Chained indexing — may trigger SettingWithCopyWarning
# df[df['price'] > 400]['price'] = 0 # Unpredictable!
# ✅ Single operation with loc
df.loc[df['price'] > 400, 'price'] = 0 # Always works
# ❌ Another chained form
# df[df['category'] == 'Electronics']['stock'] = 100
# ✅ Correct approach
df.loc[df['category'] == 'Electronics', 'stock'] = 100
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
df[condition][col] = value may not modify the original DataFrame (the intermediate result is a copy). The correct way to write it is: df.loc[condition, col] = value—a single operation that specifies both rows and columns at once.
(2) Safe Selection Decision Tree
graph TB
Q["Need to select data?"] -->|"Read only"| LOC["loc / iloc / boolean"]
Q -->|"Read + Assign"| SAFE["loc[row_cond, col] = value"]
Q -->|"Complex filter"| QUERY["query()"]
Q -->|"Multi-value match"| ISIN["isin()"]
SAFE --> WARN["❌ Never: df[cond][col] = val"]
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
9. Complete Example: Multi-Dimension Product Filtering for E-Commerce
▶ Example
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
: The full product filtering workflow (Difficulty ⭐⭐⭐)
import pandas as pd
import numpy as np
# ============================================
# Comprehensive example: E-commerce product
# multi-dimension selection workflow
# ============================================
# 1. Create product catalog
np.random.seed(42)
df = pd.DataFrame({
'name': [f'Product_{i:03d}' for i in range(1, 51)],
'category': np.random.choice(['Electronics', 'Clothing', 'Home', 'Sports'], 50),
'price': np.round(np.random.uniform(10, 500, 50), 2),
'stock': np.random.randint(0, 300, 50),
'rating': np.round(np.random.uniform(3.0, 5.0, 50), 1),
'brand': np.random.choice(['Alpha', 'Beta', 'Gamma', 'Delta'], 50)
})
# 2. loc: precise selection
print("=== Electronics with price > 200 ===")
elec_expensive = df.loc[
(df['category'] == 'Electronics') & (df['price'] > 200),
['name', 'price', 'rating']
]
print(elec_expensive.head())
# 3. iloc: position-based selection
print("\n=== First 3 rows, first 4 columns ===")
print(df.iloc[:3, :4])
# 4. Boolean: multi-condition
print("\n=== High rating, in stock, under $100 ===")
bargains = df[(df['rating'] >= 4.5) & (df['stock'] > 0) & (df['price'] < 100)]
print(bargains[['name', 'price', 'rating', 'stock']])
# 5. query: readable complex filter
print("\n=== Alpha brand Electronics or Home, rating >= 4.0 ===")
query_result = df.query(
'brand == "Alpha" and category in ["Electronics", "Home"] and rating >= 4.0'
)
print(query_result[['name', 'category', 'rating']])
# 6. isin: multi-value filter
print("\n=== Selected brands ===")
top_brands = df[df['brand'].isin(['Alpha', 'Beta'])]
print(f"Alpha + Beta products: {len(top_brands)}")
# 7. Safe assignment
df.loc[df['stock'] == 0, 'status'] = 'Out of Stock'
df.loc[df['stock'] > 0, 'status'] = 'In Stock'
print(f"\n=== Stock Status ===")
print(df['status'].value_counts())
> **Output:** Run this in a local Python environment (pandas 2.x). The Piston server does not have pandas preinstalled, so please install it on your machine (`pip install pandas`) and follow along. Actual values may vary slightly depending on the pandas version.
❓ FAQ
df['a'] > 1 & df['b'] < 2 is parsed as df['a'] > (1 & df['b']) < 2, which raises an error. You must write (df['a'] > 1) & (df['b'] < 2). You cannot use and/or/not—they do not support element-wise operations at the array level.df[condition][col] = value is two separate operations: the first may return either a view or a copy, and assigning to a copy in the second step will not affect the original DataFrame. This leads to the subtle bug of "I changed it but it didn't take effect." The correct way to write it is: df.loc[condition, col] = value, a single operation that specifies both rows and columns at once.df.query('price > @min_price'). For column names with spaces, use backticks: df.query('unit price > 100'). Inside query, keywords like and/or/not/in are supported, making it closer to natural language.df[col].isin([v1,v2,v3]) is equivalent to (df[col]==v1) | (df[col]==v2) | (df[col]==v3), but is more concise and efficient. Use ~df[col].isin(...) when you need negation.df[df.a > 0] = 1 sets the entire row to 1 (not just column a). The correct form is: df.loc[df.a > 0, 'a'] = 1. Always use loc to specify both the row condition and the target column at the same time.📖 Summary
- loc selects by label, and slices include the endpoint; iloc selects by position, and slices exclude the endpoint
- Use boolean indexing for conditional filtering, and combine multiple conditions with & / | / ~ (parentheses are required)
- The query method uses string expressions, offering readability closer to SQL, and supports @variable references
- isin is used for multi-value matching, and is more concise and efficient than combining multiple == with |
- Chained indexing
df[cond][col] = valis a subtle bug; you must usedf.loc[cond, col] = val - Use .str.contains() / .str.startswith() and similar methods for string condition filtering
- Selection decisions: for reading data → loc/iloc/query/isin; for assignment → loc[row_condition, column_name]
📝 Exercises
- Basic (Difficulty ⭐): Create a student DataFrame (name / math / english / science). Use loc to select the name and math columns of students with math > 80, and use iloc to select the first 3 rows.
- Intermediate (Difficulty ⭐⭐): Filter the same data in 3 ways (boolean indexing / query / isin): price between 50 and 200, and category is Electronics or Home. Compare the amount of code and readability.
- Challenge (Difficulty ⭐⭐⭐): Create a 50-row product DataFrame and complete 5 selection exercises: loc single row / iloc slice / multi-condition boolean / query with variable / isin multi-value. After each selection, safely assign a new marker column using loc, and finally count the number of each marker.