Linux: Pipes and Filters — The Essence of Unix Philosophy

The core of Unix philosophy is "do one thing well." The pipe | chains these small programs together — the output of one becomes the input of the next.

📋 Prerequisites: You should first complete

• Lesson 12: Standard Streams and Redirection

1. What You Will Learn

• How pipes | work
• sort for sorting and uniq for deduplication
• wc for counting lines/words/characters
• tee for dual output
• The art of combining multi-stage pipelines

2. A 10GB Log File Story

(1) The Pain: Logs Too Large to Browse Manually

Alex needed to find the top 10 IPs by access count from a 10GB Nginx access log. Opening it in an editor was impossible (10GB won't open), and writing a Python script was overkill.

(2) The Power of a One-Liner Pipeline

Bob taught him to combine commands with pipes:

cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

ℹ️ Note: cat file | command is common, but cat here only passes the file content to the pipe — using command < file or awk '{...}' file directly is more efficient, saving a process. This is called "UUOC" (Useless Use of Cat); while it works fine, it can be optimized for performance.

Output:

 45231 192.168.1.100
 38902 10.0.0.45
 21045 192.168.1.200

(3) The Benefit: Complex Tasks in One Command

Alex discovered that pipeline combinations can handle almost any data extraction task — counting, filtering, sorting, ranking, deduplication. A single command is a mini data-processing pipeline.

3. Knowledge Points

(1) How Pipes Work

command1 | command2 | command3
    │           │           │
    │   output   │   output   │
    └──────────→│──────────→│──────────→ Final output
                │           │
           command1's stdout connects to command2's stdin

(2) Common Filters

(3) ▶ Example: Count File Information

# Count number of files
ls -la | wc -l
# Note: wc -l includes the "total" line, so actual file count is one less

# Count total lines in all .md files
cat *.md | wc -l

# Show disk usage of current directory
ls -lh | tail -n +2 | awk '{print $5, $9}'

Output:

TEXT

📋 Copy

15
342
4.0K config.yml
12M data.tar.gz

(4) ▶ Example: Sorting and Deduplication

# Sort IP addresses (alphabetical vs numeric)
cat << EOF | sort
192.168.1.100
10.0.0.45
192.168.1.200
10.0.0.2
EOF

# sort -n for numeric sort (by IP segment)
cat << EOF | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n
192.168.1.100
10.0.0.45
192.168.1.200
10.0.0.2
EOF

# Deduplicate and count
cat << EOF | sort | uniq -c | sort -rn
apple
banana
apple
orange
banana
apple
EOF
#       3 apple
#       2 banana
#       1 orange

Output:

TEXT

📋 Copy

10.0.0.2
10.0.0.45
192.168.1.100
192.168.1.200
      3 apple
      2 banana
      1 orange

(5) ▶ Example: head and tail

# View first 5 lines
ls -la /etc | head -5

# View last 5 lines
ls -la /etc | tail -5

# Show lines 10-20
cat longfile.txt | head -20 | tail -11

# Follow a log in real-time (press Ctrl+C to exit)
tail -f /var/log/syslog

# View the last 10 log entries
tail -f -n 10 /var/log/syslog

Output:

TEXT

📋 Copy

total 72
drwxr-xr-x   2 root root 4096 Jul  7 10:00 bin
drwxr-xr-x   3 root root 4096 Jul  7 10:00 lib
drwxrwxrwt  15 root root 4096 Jul  7 10:30 tmp

# See results on screen and save to file simultaneously
ls -la | tee output.txt

# Append mode
echo "New content" | tee -a output.txt

# Save intermediate pipeline results
ps aux | tee processes.txt | grep nginx
# First save all processes with tee, then filter for nginx

Output:

TEXT

📋 Copy

total 72
drwxr-xr-x   2 root root 4096 Jul  7 10:00 bin
...
New content
root      1234  0.0  0.1  nginx: master

(6) ▶ Example: Practical Pipeline Combinations

# 5 largest files
ls -lhS | head -6

# Number of running shells
ps aux | grep bash | wc -l

# Partitions with disk usage over 80%
df -h | grep -v "tmpfs" | awk '{print $5, $6}' | sort -rn

# Find log files in /var/log modified within 7 days
find /var/log -name "*.log" -mtime -7 | sort

# Get external IP
curl -s ifconfig.me | tee my-ip.txt

Output:

TEXT

📋 Copy

-r--r--r-- 1 root root  12K config.pkt
-rw-r--r-- 1 root root 8.9M data.tar.gz
3
203.0.113.42

(7) ▶ Comprehensive Example: Log Analysis Pipeline Chain

# Simulate an Nginx access log
cat << 'EOF' > /tmp/access.log
192.168.1.100 - - [07/Jul/2026:10:15:30 +0000] "GET /index.html HTTP/1.1" 200 1234
10.0.0.45 - - [07/Jul/2026:10:16:30 +0000] "GET /api/users HTTP/1.1" 200 5678
192.168.1.100 - - [07/Jul/2026:10:17:30 +0000] "POST /api/login HTTP/1.1" 401 234
10.0.0.45 - - [07/Jul/2026:10:18:30 +0000] "GET /api/users HTTP/1.1" 200 5678
192.168.1.200 - - [07/Jul/2026:10:19:30 +0000] "GET /index.html HTTP/1.1" 404 123
10.0.0.45 - - [07/Jul/2026:10:20:30 +0000] "GET /api/users HTTP/1.1" 200 5678
EOF

echo "=== Total Requests ==="
wc -l /tmp/access.log

echo ""
echo "=== Access Count per IP (Top 5) ==="
cat /tmp/access.log | awk '{print $1}' | sort | uniq -c | sort -rn

echo ""
echo "=== Count per HTTP Status Code ==="
cat /tmp/access.log | awk '{print $9}' | sort | uniq -c | sort -rn

echo ""
echo "=== Most Accessed URLs ==="
cat /tmp/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -5

echo ""
echo "=== 404 Error Requests ==="
grep " 404 " /tmp/access.log | awk '{print $1, $7}'

❓ FAQ

Q: Are pipelines processed serially or in parallel? A: \1

Q: Can pipes and redirection be used together? A: Yes. For example, command > file 2>&1 (redirection) and command | grep "error" (pipe) can be combined: command 2>&1 | grep "error".

Q: Does an error mid-pipeline affect the final result? A: Yes. If a command in the pipeline fails, downstream commands may receive incomplete data. Use set -o pipefail (in scripts) to make any pipeline command's failure cause the entire pipeline to fail.

Q: How do I exit tail -f? A: Press Ctrl + C. tail -f continuously monitors the file for new content until you manually interrupt it.

Q: Will pipeline processing of large files run out of memory? A: \1

📖 Summary

• Pipe | connects commands: previous command's stdout → next command's stdin
• sort sorts, uniq -c deduplicates and counts, wc -l counts lines
• head -n shows first N lines, tail -f follows in real-time, tee splits output
• Combination example: cat log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
• Pipelines are parallel and streaming — processing large files won't run out of memory

📝 Exercises

1. Basic (Difficulty ⭐): Use ls -la | wc -l to count files, then use /var/log/syslog or create a test file to extract lines containing "error" and count them
2. Intermediate (Difficulty ⭐⭐): Use df -h | grep "^/" | sort -k5 -rn to find the partition with the highest disk usage, and use tee to save ps aux output to a file while displaying it in the terminal
3. Advanced (Difficulty ⭐⭐⭐): Combine 3+ pipeline commands for a data-processing task — find the top 5 memory-consuming processes on the system, sorted by memory usage in descending order, formatted as "PID Command Memory%"

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

About Us