Write awk one-liners to analyze NGINX access logs: find top 10 IPs, calculate request rate per second, and identify slow requests.
Quick Answer
Use awk to parse space-delimited NGINX log format, associative arrays for aggregation, and END block for reporting. Combine with sort and head for top-N queries.
Detailed Answer
NGINX Default Log Format
$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time
Fields: IP(-) user [timestamp] "method path proto" status bytes "referer" "ua" response_time
Why awk over other tools
- Processes files line-by-line with minimal memory - Built-in field splitting (perfect for log formats) - Associative arrays for aggregation - No external dependencies - Faster than Python/Ruby for simple text processing
Performance tip: For very large files (10GB+), use mawk instead of gawk — it's 5-10x faster for simple processing. Or use LC_ALL=C to skip locale processing.
Code Example
# Top 10 IPs by request count
awk '{ip[$1]++} END {for (i in ip) print ip[i], i}' access.log | sort -rn | head -10
# Request rate per second
awk '{split($4, a, "["); split(a[2], b, ":"); sec=b[2]":"b[3]":"b[4]; rate[sec]++}
END {for (s in rate) print s, rate[s], "req/s"}' access.log | sort -t: -k1,3 | tail -20
# Slow requests (>1s response time, last field)
awk '{if ($NF > 1.0) print $NF"s", $7, $1}' access.log | sort -rn | head -20
# HTTP status code distribution
awk '{status[$9]++} END {for (s in status) printf "%s: %d (%.1f%%)\n", s, status[s], status[s]/NR*100}' access.log | sort
# Bandwidth per endpoint (MB)
awk '{endpoint=$7; bytes[endpoint]+=$10}
END {for (e in bytes) printf "%.2f MB %s\n", bytes[e]/1048576, e}' access.log | sort -rn | head -10
# Error rate per minute
awk '$9 >= 500 {split($4, a, "["); split(a[2], t, ":"); min=t[2]":"t[3]; errors[min]++}
{split($4, a, "["); split(a[2], t, ":"); min=t[2]":"t[3]; total[min]++}
END {for (m in total) printf "%s %.2f%% errors (%d/%d)\n", m, errors[m]/total[m]*100, errors[m]+0, total[m]}' access.log | sortInterview Tip
Awk one-liners for log analysis are a must-know for SRE interviews, especially at Google. Practice these until they're second nature. The key patterns: associative arrays for counting, END block for output, sort | head for top-N.