Write a production-safe Bash script that processes log files in parallel, extracts error counts per service, and handles signals gracefully.
Quick Answer
Use xargs -P for parallelism, awk for field extraction, trap for signal handling (SIGTERM/SIGINT), and temporary files with cleanup on exit.
Detailed Answer
Requirements
1. Process multiple log files concurrently 2. Extract error count per service name 3. Handle SIGTERM/SIGINT for graceful shutdown 4. Clean up temporary files on exit (success or failure) 5. Provide progress reporting
Key Techniques
- xargs -P N for bounded parallelism - awk for efficient text processing - trap for signal handling and cleanup - mktemp -d for safe temporary directory - flock for safe concurrent file writes - Process substitution <() for avoiding subshell variable issues
Code Example
#!/usr/bin/env bash
set -euo pipefail
MAX_PARALLEL=4
TMPDIR=$(mktemp -d)
RESULT_FILE="$TMPDIR/results.txt"
PID_FILE="$TMPDIR/pids.txt"
cleanup() {
# Kill any background jobs
if [[ -f "$PID_FILE" ]]; then
while read -r pid; do
kill "$pid" 2>/dev/null || true
done < "$PID_FILE"
fi
rm -rf "$TMPDIR"
echo "Cleanup complete" >&2
}
trap cleanup EXIT
trap 'echo "Interrupted" >&2; exit 130' INT TERM
process_log() {
local file=$1
local out="$TMPDIR/$(basename "$file").partial"
awk '/ERROR/ {
# Extract service name from log format: timestamp SERVICE ERROR message
service = $2
errors[service]++
}
END {
for (s in errors) print s, errors[s]
}' "$file" > "$out"
echo "Processed: $file ($(wc -l < "$out") services)" >&2
}
export -f process_log
export TMPDIR
# Find log files and process in parallel
find /var/log/app/ -name '*.log' -mtime -1 -print0 | \
xargs -0 -P "$MAX_PARALLEL" -I {} bash -c 'process_log "$@"' _ {}
# Aggregate partial results
awk '{
errors[$1] += $2
}
END {
for (s in errors) printf "%-30s %d\n", s, errors[s]
}' "$TMPDIR"/*.partial | sort -t' ' -k2 -rn > "$RESULT_FILE"
echo "=== Error Count by Service ==="
cat "$RESULT_FILE"Interview Tip
Show production-safe patterns: trap for cleanup, mktemp for temp files, bounded parallelism with xargs -P, and graceful signal handling. This demonstrates you write scripts that won't leave orphan processes or temp files.