Redis Pipelining
Pipelining allows batch-sending commands, reducing network round trips and dramatically improving performance.
What is Pipelining?
Traditional Approach
Each command requires one network round trip:
Client Redis Server
| |
|----- SET key1 -------->|
|<----- OK --------------|
| |
|----- SET key2 -------->|
|<----- OK --------------|
| |
|----- SET key3 -------->|
|<----- OK --------------|
3 commands = 6 network transfers (3 requests + 3 responses)
Pipelining Approach
Multiple commands sent at once:
Client Redis Server
| |
|----- SET key1 -------->|
|----- SET key2 -------->|
|----- SET key3 -------->|
| |
|<----- OK --------------|
|<----- OK --------------|
|<----- OK --------------|
3 commands = 2 network transfers (1 batch request + 1 batch response)
💡 Performance gain: Pipelining reduces network round trips, dramatically improving performance — especially when network latency is high.
How Pipelining Works
Effect of Network Latency
Assuming a 1ms network round trip:
Traditional approach:
- 100 commands = 100 round trips = 100ms network latency
- Plus Redis processing time (assume 0.1ms/command) = 10ms
- Total time = 100ms + 10ms = 110ms
Pipelining:
- 100 commands = 1 round trip = 1ms network latency
- Plus Redis processing time = 10ms
- Total time = 1ms + 10ms = 11ms
Performance improvement: 110ms → 11ms, a 10x boost!
Pipeline Characteristics
- Batch sending: multiple commands sent at once
- Sequential execution: Redis executes commands in order
- Batch response: all results returned at once
- Non-atomic: not a transaction — commands can be interrupted by other clients
⚠️ Note: Pipelining is not a transaction and does not guarantee atomicity. Use MULTI/EXEC for atomicity.
Using Pipelining
redis-cli Pipeline Mode
BASH
# Use redis-cli pipeline mode
echo -e "SET key1 value1\nSET key2 value2\nSET key3 value3" | redis-cli
# Or read from a file
cat commands.txt | redis-cli
Python Pipelining
PYTHON
import redis
r = redis.Redis(host='localhost', port=6379)
# Create pipeline
pipe = r.pipeline()
# Add commands to pipeline
pipe.set('key1', 'value1')
pipe.set('key2', 'value2')
pipe.set('key3', 'value3')
# Execute pipeline
results = pipe.execute()
print(results) # [True, True, True]
Java Pipelining (Jedis)
JAVA
Jedis jedis = new Jedis("localhost", 6379);
Pipeline pipeline = jedis.pipelined();
pipeline.set("key1", "value1");
pipeline.set("key2", "value2");
pipeline.set("key3", "value3");
List<Object> results = pipeline.syncAndReturnAll();
Node.js Pipelining (ioredis)
JAVASCRIPT
const Redis = require('ioredis');
const redis = new Redis();
const pipeline = redis.pipeline();
pipeline.set('key1', 'value1');
pipeline.set('key2', 'value2');
pipeline.set('key3', 'value3');
const results = await pipeline.exec();
Pipeline vs Transactions
Pipeline + Transactions
Pipelining can be combined with transactions:
PYTHON
import redis
r = redis.Redis()
# Create pipeline with transaction enabled
pipe = r.pipeline(transaction=True)
pipe.set('key1', 'value1')
pipe.set('key2', 'value2')
pipe.incr('counter')
# Execute transaction
results = pipe.execute()
Pipeline vs Transaction
| Aspect | Pipeline | Transaction |
|---|---|---|
| Atomicity | ❌ No | ✅ Yes |
| Isolation | ❌ No | ✅ Yes |
| Performance gain | ✅ Significant | ✅ Significant |
| Batch sending | ✅ Yes | ✅ Yes |
| Use case | Batch operations, performance | Batch operations needing atomicity |
💡 Choice: Use pipeline when atomicity is not needed. Use transactions or pipeline+transaction when atomicity is required.
Pipeline Use Cases
Use Case 1: Batch Setting
PYTHON
import redis
r = redis.Redis()
pipe = r.pipeline()
# Batch set 1000 keys
for i in range(1000):
pipe.set(f'key:{i}', f'value:{i}')
results = pipe.execute()
print(f'Set {len(results)} keys')
Use Case 2: Batch Getting
PYTHON
import redis
r = redis.Redis()
pipe = r.pipeline()
# Batch get 1000 keys
keys = [f'key:{i}' for i in range(1000)]
for key in keys:
pipe.get(key)
values = pipe.execute()
print(f'Got {len(values)} values')
Use Case 3: Batch Deletion
PYTHON
import redis
r = redis.Redis()
pipe = r.pipeline()
# Batch delete
keys = ['key1', 'key2', 'key3', 'key4', 'key5']
for key in keys:
pipe.delete(key)
results = pipe.execute()
print(f'Deleted {sum(results)} keys')
Use Case 4: Data Import
PYTHON
import redis
import json
r = redis.Redis()
pipe = r.pipeline()
# Import data from JSON file
with open('data.json', 'r') as f:
data = json.load(f)
for item in data:
key = f"user:{item['id']}"
value = json.dumps(item)
pipe.set(key, value)
pipe.execute()
print('Import complete')
Use Case 5: Batch Counter Updates
PYTHON
import redis
r = redis.Redis()
pipe = r.pipeline()
# Batch update counters
counters = {
'article:1:views': 10,
'article:2:views': 20,
'article:3:views': 30,
}
for key, increment in counters.items():
pipe.incrby(key, increment)
pipe.execute()
print('Counters updated')
Pipeline Performance Comparison
Test Code
PYTHON
import redis
import time
r = redis.Redis()
# Test data
n = 10000
# Method 1: regular approach
start = time.time()
for i in range(n):
r.set(f'key:{i}', f'value:{i}')
end = time.time()
print(f'Regular: {end - start:.2f} seconds')
# Method 2: pipelining
start = time.time()
pipe = r.pipeline()
for i in range(n):
pipe.set(f'key:{i}', f'value:{i}')
pipe.execute()
end = time.time()
print(f'Pipeline: {end - start:.2f} seconds')
Sample Results
Regular: 5.23 seconds
Pipeline: 0.51 seconds
Performance improvement: about 10x
Pipeline Best Practices
1. Set a Reasonable Batch Size
PYTHON
# ❌ Sending too many commands at once
pipe = r.pipeline()
for i in range(100000):
pipe.set(f'key:{i}', f'value:{i}')
pipe.execute() # May use a lot of memory
# ✅ Execute in batches
batch_size = 1000
for batch in range(0, 100000, batch_size):
pipe = r.pipeline()
for i in range(batch, batch + batch_size):
pipe.set(f'key:{i}', f'value:{i}')
pipe.execute()
2. Error Handling
PYTHON
import redis
r = redis.Redis()
pipe = r.pipeline()
pipe.set('key1', 'value1')
pipe.incr('key1') # This will fail (key1 is not a number)
pipe.set('key2', 'value2')
try:
results = pipe.execute()
for i, result in enumerate(results):
if isinstance(result, Exception):
print(f'Command {i} failed: {result}')
except Exception as e:
print(f'Pipeline execution failed: {e}')
3. Use Transactions for Atomicity
PYTHON
import redis
r = redis.Redis()
# Use transaction when atomicity is needed
pipe = r.pipeline(transaction=True)
pipe.set('account:a', '100')
pipe.set('account:b', '50')
try:
results = pipe.execute()
print('Transaction executed successfully')
except Exception as e:
print(f'Transaction failed: {e}')
4. Monitor Pipeline Performance
PYTHON
import redis
import time
r = redis.Redis()
start = time.time()
pipe = r.pipeline()
for i in range(10000):
pipe.set(f'key:{i}', f'value:{i}')
results = pipe.execute()
end = time.time()
print(f'Execution time: {end - start:.2f} seconds')
print(f'QPS: {10000 / (end - start):.0f}')
Pipeline Limitations
1. Memory Usage
PYTHON
# Pipeline caches all commands and results
# Too large a batch uses a lot of memory
# Solution: execute in batches
2. Non-atomic
PYTHON
# Commands in a pipeline can be interrupted by other clients
# Use transactions when atomicity is needed
3. Cannot Use Intermediate Results
PYTHON
# Cannot use the result of a previous command in the pipeline
# ❌ Incorrect example
pipe = r.pipeline()
pipe.incr('counter')
# Cannot get incr result for subsequent commands
pipe.set('result', ???) # Cannot use incr's result
pipe.execute()
# ✅ Solution: use Lua scripts
❓ FAQ
Q What's the difference between pipeline and transaction?
A Pipeline reduces network round trips but doesn't guarantee atomicity. Transactions guarantee atomicity and also reduce network round trips.
Q How much performance improvement can pipelining provide?
A It depends on network latency. Higher latency means greater improvement. Typically 5-10x.
Q What batch size should I use?
A Recommended 100-1000 commands per batch. Too large uses too much memory; too small gives minimal improvement.
Q What happens if a command in the pipeline fails?
A A single command failure does not affect other commands. Check the returned results for errors.
Q When should I use pipelining?
A Batch operations, data import/export, and any scenario where performance improvement is needed.
📖 Summary
- Pipelining sends commands in batches, reducing network round trips
- Significant performance improvement, especially with high network latency
- Pipelining does not guarantee atomicity — use transactions when needed
- Use cases: batch set, batch get, data import
- Best practices: reasonable batch size, error handling, batch execution
- Pipeline limitations: memory usage, non-atomic, cannot use intermediate results
📝 Exercises
- Basic pipeline: Use pipelining to batch-set 100 keys, compare performance with the regular approach
- Batch operations: Implement batch get and batch delete with pipelining
- Performance comparison: Test performance with different batch sizes (10, 100, 1000)
- Pipeline + transaction: Implement atomic batch operations using pipeline with transactions
Next Lesson
In the next lesson, we will learn Python with Redis, covering Python Redis operations.