English العربية Português 日本語

Linux: Phase 3 Mini Project — Server Log Analysis

The Phase 3 capstone project. You'll combine redirection, pipes, grep, awk, sed, sort, uniq, and all your data processing skills to perform a full-dimensional analysis of Nginx access logs and generate a report.

📋 Prerequisites: You should already know

1. What You'll Learn

2. Xiaoming's Website Slowed Down

(1) The Pain Point: Boss Wants Data, but the 10GB Log Is Impossible to Navigate

Xiaoming's website had been slow lately, and the boss demanded "give me data." Nginx's access.log generates hundreds of MB daily — opening it is impossible, and searching for keywords is like finding a needle in a haystack.

(2) Pipeline Chain Analysis

Xiaoming used the skills from Phase 3, extracting key metrics line by line with pipe commands:

BASH 
# Top 10 IPs by access count
cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

# Top 10 most-requested URLs
cat access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -10

> 💡 Tip: The log analysis pipeline pattern is fixed: `awk extract fields → sort → uniq -c deduplicate and count → sort -rn descending by count → head take top N`. Remember this template — any log Top N analysis can use it.

# Requests per hour
cat access.log | awk -F: '{print $2":00"}' | sort | uniq -c | sort -rn

Output:

TEXT 
    210 192.168.1.100
    198 10.0.0.45
    195 192.168.1.200
    190 10.0.0.88
    180 172.16.0.50
    150 /index.html
    120 /api/users
    110 /api/login
    100 /about
     90 /contact
     85 10:00
     80 14:00
     75 09:00
     70 22:00
     65 18:00

(3) The Benefit: Report in 5 Minutes

Xiaoming compiled the analysis results into a report — access trends, top 10 slow requests, pages with the most 404 errors. The boss said: "Good, let's add more servers."

3. Knowledge Points

(1) Standard Nginx Log Format (combined)

TEXT 

192.168.1.100 - - [07/Jul/2026:10:15:30 +0000] "GET /index.html HTTP/1.1" 200 1234 "-" "Mozilla/5.0"
│              │ │ │                           │                           │    │    │
│              │ │ └─ User-Agent               │                           │    │    └─ Response bytes
│              │ └───────────────────────────── Referer                    │    └────── Status code
│              └─────────────────────────────────────────────────────────── Request line (method/URL/protocol)
└─── Client IP                                   Timestamp

ℹ️ Note: In Nginx's combined log format, fields are space-separated, but some fields contain spaces themselves (e.g., the request line and timestamp are wrapped in brackets and quotes). When awk splits by spaces, $1 is the IP, $7 is the URL, $9 is the status code, $10 is the bytes — these field positions are stable and can be extracted directly with awk.

Field positions (space-separated):

Field Meaning
$1 Client IP
$4 Timestamp [ part
$7 Requested URL
$9 HTTP status code
$10 Response bytes

(2) ▶ Example: Generate Sample Logs

First, generate a sample log file for analysis:

TEXT 
#!/bin/bash
# generate-sample-log.sh

IPS=("192.168.1.100" "10.0.0.45" "192.168.1.200" "10.0.0.88" "172.16.0.50")
URLS=("/index.html" "/api/users" "/api/login" "/about" "/contact" "/images/logo.png" "/api/orders" "/products" "/blog")
STATUSES=(200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 201 301 302 400 401 403 404 500 502 503)

rm -f /tmp/access.log

for i in {1..1000}; do
    IP=${IPS[$RANDOM % ${#IPS[@]}]}
    URL=${URLS[$RANDOM % ${#URLS[@]}]}
    STATUS=${STATUSES[$RANDOM % ${#STATUSES[@]}]}
    SIZE=$((RANDOM % 10000 + 100))
    HOUR=$((RANDOM % 24))
    MIN=$((RANDOM % 60))
    SEC=$((RANDOM % 60))
    echo "$IP - - [07/Jul/2026:$HOUR:$MIN:$SEC +0000] \"GET $URL HTTP/1.1\" $STATUS $SIZE \"-\" \"Mozilla/5.0\"" >> /tmp/access.log
done

echo "Generated 1000 sample log entries to /tmp/access.log"
BASH 
chmod +x generate-sample-log.sh
./generate-sample-log.sh

Output:

TEXT 
Generated 1000 sample log entries to /tmp/access.log

(3) ▶ Example: Request Volume Statistics

BASH 
echo "=== Total Requests ==="
wc -l /tmp/access.log

echo ""
echo "=== Requests per IP ==="
cat /tmp/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

echo ""
echo "=== Request Method Distribution ==="
cat /tmp/access.log | awk '{print $6}' | tr -d '"' | sort | uniq -c | sort -rn

Output:

TEXT 
=== Total Requests ===
1000 /tmp/access.log

=== Requests per IP ===
    210 192.168.1.100
    198 10.0.0.45
    195 192.168.1.200
    190 10.0.0.88
    180 172.16.0.50

=== Request Method Distribution ===
   1000 GET

(4) ▶ Example: Status Code Analysis

BASH 
echo "=== Status Code Distribution ==="
cat /tmp/access.log | awk '{print $9}' | sort | uniq -c | sort -rn

echo ""
echo "=== 4xx Errors (Client Errors) ==="
cat /tmp/access.log | awk '$9 ~ /^4/ {print $9, $7, $1}' | sort | uniq -c | sort -rn | head -10

echo ""
echo "=== 5xx Errors (Server Errors) ==="
cat /tmp/access.log | awk '$9 ~ /^5/ {print $9, $7, $1}' | sort | uniq -c | sort -rn | head -10

echo ""
echo "=== Error Rate ==="
TOTAL=$(wc -l < /tmp/access.log)
ERRORS=$(awk '$9 ~ /^[45]/ {count++} END {print count}' /tmp/access.log)
echo "Total requests: $TOTAL"
echo "Errors: $ERRORS"
echo "Error rate: $(echo "scale=2; $ERRORS * 100 / $TOTAL" | bc)%"

Output:

TEXT 
=== Status Code Distribution ===
    960 200
      5 201
      3 301
      2 302
      8 400
      2 401
      3 403
     10 404
      4 500
      2 502
      1 503

=== 4xx Errors (Client Errors) ===
   3 404 /api/login 192.168.1.100
   2 404 /about 10.0.0.45
   2 403 /api/admin 192.168.1.200

=== 5xx Errors (Server Errors) ===
   2 500 /api/data 10.0.0.88
   2 502 /api/users 172.16.0.50
   1 503 /api/health 192.168.1.100

=== Error Rate ===
Total requests: 1000
Errors: 32
Error rate: 3.20%

(5) ▶ Example: URL Hotspot Analysis

BASH 
echo "=== Most Popular URLs ==="
cat /tmp/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -10

echo ""
echo "=== URLs with Most 404s ==="
cat /tmp/access.log | awk '$9 == 404 {print $7}' | sort | uniq -c | sort -rn | head -10

echo ""
echo "=== URLs with Largest Response Size ==="
cat /tmp/access.log | awk '{print $10, $7}' | sort -rn | head -10

Output:

TEXT 
=== Most Popular URLs ===
    150 /index.html
    120 /api/users
    110 /api/login
    100 /about
     90 /contact

=== URLs with Most 404s === 5 /api/login 3 /old-page 2 /favicon.ico

=== URLs with Largest Response Size === 10098 /images/logo.png 9876 /api/users 8765 /api/orders 7654 /products 6543 /blog

(6) ▶ Example: Time Dimension Analysis

BASH 
echo "=== Requests per Hour ==="
cat /tmp/access.log | awk -F: '{print $2":00"}' | sort | uniq -c | sort -rn

echo ""
echo "=== Peak Traffic Hours ==="
cat /tmp/access.log | awk -F: '{print $2}' | sort | uniq -c | sort -rn | head -5

echo ""
echo "=== Requests per Minute (Top 10) ==="
cat /tmp/access.log | awk -F: '{print $2":"$3}' | sort | uniq -c | sort -rn | head -10

Output:

TEXT 
=== Requests per Hour ===
     85 10:00
     80 14:00
     75 09:00
     70 22:00
     65 18:00

=== Peak Traffic Hours ===
     85 10
     80 14
     75 09
     70 22
     65 18

=== Requests per Minute (Top 10) ===
      5 10:15
      4 14:30
      4 09:45
      3 22:10
      3 18:05

(7) ▶ Example: Performance Analysis

BASH 
echo "=== Response Size Analysis (using $10 as simulated response size) ==="
echo "Largest response: $(awk '{print $10}' /tmp/access.log | sort -rn | head -1) bytes"
echo "Average response: $(awk '{sum += $10; count++} END {printf "%.0f", sum/count}' /tmp/access.log) bytes"
echo ""

echo "=== Large Responses (> 5KB) ==="
cat /tmp/access.log | awk '$10 > 5000 {print $10, $7}' | sort -rn | head -10

Output:

TEXT 
=== Response Size Analysis (using $10 as simulated response size) ===
Largest response: 10098 bytes
Average response: 5142 bytes

=== Large Responses (> 5KB) ===
10098 /images/logo.png
 9876 /api/users
 8765 /api/orders
 7654 /products
 6543 /blog
 5678 /api/data

(8) ▶ Comprehensive Example: Complete Log Analysis Report Script

BASH 
#!/bin/bash
# log-report.sh - Generate Nginx log analysis report

LOG_FILE="${1:-/tmp/access.log}"

if [ ! -f "$LOG_FILE" ]; then
    echo "Error: Log file $LOG_FILE does not exist"
    exit 1
fi

TOTAL=$(wc -l < "$LOG_FILE")

echo "========================================="
echo "     Nginx Access Log Analysis Report"
echo "     File: $LOG_FILE"
echo "     Total requests: $TOTAL"
echo "     Generated at: $(date)"
echo "========================================="
echo ""

# 1. Status code overview
echo "1. HTTP Status Code Distribution"
echo "--------------------"
cat "$LOG_FILE" | awk '{print $9}' | sort | uniq -c | sort -rn | \
    awk '{printf "  %-5s %s\n", $2, $1}'
echo ""

# 2. TOP 10 IPs
echo "2. Top 10 IPs by Access Count"
echo "---------------------"
cat "$LOG_FILE" | awk '{print $1}' | sort | uniq -c | sort -rn | head -10 | \
    awk '{printf "  %-20s %s\n", $2, $1}'
echo ""

# 3. TOP 10 URLs
echo "3. Top 10 URLs by Request Count"
echo "-----------------------"
cat "$LOG_FILE" | awk '{print $7}' | sort | uniq -c | sort -rn | head -10 | \
    awk '{printf "  %-30s %s\n", $1, $2}'
echo ""

# 4. 4xx/5xx errors
echo "4. Error Requests"
echo "-------------"
FOURXX=$(awk '$9 ~ /^4/ {count++} END {print count}' "$LOG_FILE")
FIVEXX=$(awk '$9 ~ /^5/ {count++} END {print count}' "$LOG_FILE")
echo "  4xx errors: $FOURXX ($(echo "scale=1; $FOURXX*100/$TOTAL" | bc)%)"
echo "  5xx errors: $FIVEXX ($(echo "scale=1; $FIVEXX*100/$TOTAL" | bc)%)"
echo ""

# 5. Requests per hour
echo "5. Requests per Hour"
echo "--------------------"
cat "$LOG_FILE" | awk -F: '{print $2":00"}' | sort | uniq -c | sort -rn | head -10 | \
    awk '{printf "  %s: %s\n", $2, $1}'
echo ""

# 6. Largest responses
echo "6. Largest Responses (Top 5)"
echo "----------------------"
cat "$LOG_FILE" | awk '{print $10, $7}' | sort -rn | head -5 | \
    awk '{printf "  %s bytes: %s\n", $1, $2}'
echo ""

echo "========================================="
echo "End of Report"
BASH 
# Run the report script
chmod +x log-report.sh
./log-report.sh /tmp/access.log

❓ FAQ

Q: What do the fields in Nginx log format mean? A: Standard combined format: IP - user [time] "method URL protocol" status_code bytes "Referer" "User-Agent". You can customize it via Nginx's log_format directive.

Q: Will pipelines choke on huge (GB-level) log files? A: Not too badly. Pipeline streaming means data is processed as it's read. However, sort will block because it needs to read all data first. For very large files, use sort -S 1G to limit memory, or use awk arrays for statistics instead.

Q: How to count real PV (page views) and UV (unique visitors)? A: PV = wc -l (excluding non-page requests like images/CSS/API). UV = awk '{print $1}' access.log | sort -u | wc -l.

Q: How to continuously monitor log changes? A: tail -f tracks new lines in real time. Combine with grep: tail -f access.log | grep " 500 " — shows all 500 errors in real time.

Q: How to calculate P99 response time for API endpoints? A: Nginx needs to be configured with $upstream_response_time or $request_time. Then: awk '{print $NF}' access.log | sort -n | awk '{all[NR]=$1} END{p99=int(NR*0.99); print all[p99]}'.

📖 Summary

📝 Exercises

  1. Basic (⭐): Run the sample log generation script to create 1000 test entries, then write a pipeline chain to find the top 5 IPs by access count and their request counts
  2. Intermediate (⭐⭐): Count the number and percentage of 404 and 500 status codes, then use awk '{print $7}' to extract URLs and find the top 5 most popular pages
  3. Challenge (⭐⭐⭐): Create a log analysis script that outputs a formatted Markdown report including: total requests, Top 10 IPs, status code distribution, hourly request trends, Top 5 accessed URLs
Previous: The Three Musketeers of Text Processing — awk / sed / cut Next: Process Management — ps / top / kill and System Monitoring
Web-Tutorial.com

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

100%

🙏 帮我们做得更好

我们是刚上线的编程教程站，几个人的小团队，精力有限。页面虽经检查，难免还有疏漏——链接失效、排版错乱、内容有误、语言生硬……

如果您发现了，麻烦告诉我们，我们会在收到反馈后第一时间进行修复，再次感谢您的光临 🙏