Bash Interview Questions & Answers — DevOpsDrill

Bash

Everything for Bash in one place — pick a section below. 22 reviewed items across 4 content types, plus a hands-on tutorial.

💻Hands-on TutorialGuideWrite safe, reusable scripts — quoting, exit codes, loops, functions, strict mode, and shellcheck habits.→

📝Interview Questions14Q&A with detailed answers and code examples→

🏭Use Cases3Real-world production patterns and scenarios→

🔥Production Issues2On-call incidents with root causes and fixes

14 / 14

Popular tags:

⚡

Quick Answer

Bash error handling is contextual: `set -e` does not behave like exceptions, pipeline status usually reflects the last command unless `pipefail` is set, and subshells can hide variable changes. Reliable production scripts combine strict mode, explicit checks, traps, quoting, and small testable functions.

Detailed Answer

A shell script is like a checklist read over a noisy radio. If the operator only says whether the final step worked, earlier failures can disappear unless every handoff repeats the status clearly.

Bash favors interactive command composition and POSIX heritage, not modern exception semantics. That makes it powerful for glue work but risky for deployment automation unless failure paths are made explicit.

Commands return numeric statuses, lists and conditionals decide whether those statuses abort execution, pipelines aggregate statuses by shell options, and traps run on configured signals or pseudo-signals such as ERR and EXIT.

At scale, deployment scripts should log commands, validate required variables, use temporary directories safely, lock shared operations, and treat partial failure as normal. Engineers monitor script exit codes, duration, retries, and side effects such as half-written files.

A classic gotcha is grep pattern file | awk ... succeeding because awk ran even when grep failed, or cmd || cleanup suppressing errexit behavior in places the author did not expect. Senior engineers make important checks boring and explicit.

Code Example

#!/usr/bin/env bash
set -Eeuo pipefail # Fails on unset variables, failed pipelines, and most command errors.
trap 'echo "deploy failed at line $LINENO" >&2' ERR # Reports the failing line for incident triage.
: "${SERVICE:?SERVICE is required}" # Stops early if the required service name is missing.
kubectl rollout status "deploy/${SERVICE}" -n payments --timeout=180s # Waits for the rollout and returns non-zero on timeout.

💡

Interview Tip

A junior engineer typically answers with the feature name and a happy-path command, but for a senior/architect role, the interviewer is actually looking for production judgment. Explain where the Bash control plane stores intent, how changes are rolled out, what telemetry proves the system is healthy, and what failure mode creates the largest blast radius. Strong answers include rollback behavior, ownership boundaries, permissions, and the exact signal you would page on.

◈ Architecture Diagram

┌──────────┐
│ Signal   │
└────┬─────┘
     ↓
┌──────────┐
│ Bash     │
└────┬─────┘
     ↓
┌──────────┐
│ Action   │
└──────────┘

Open Full Page ↗

💬 Comments

⚡

Quick Answer

set -e exits on error, -u exits on undefined variables, -o pipefail catches errors in piped commands. Gotchas: set -e doesn't catch errors in conditionals, subshells, or command substitutions.

Detailed Answer

Each Flag

set -e (errexit): Exit immediately if any command returns non-zero. Without this, scripts silently continue after errors.
set -u (nounset): Treat unset variables as errors. Prevents typos like rm -rf $UNSET_VAR/ from becoming rm -rf /.
set -o pipefail: A pipeline returns the exit code of the last failed command, not the last command. Without this, curl fail | grep ok returns 0 (grep's exit code) even if curl failed.

Gotchas with set -e

1. Commands in if conditions are exempt: if command_that_fails; then doesn't exit 2. Commands before || are exempt: command_that_fails || true is fine 3. Command substitutions: result=$(failing_command) exits, but local result=$(failing_command) does NOT (local's exit code masks it) 4. Subshells: (failing_command) exits the subshell but the parent continues 5. Arithmetic: let 'x=0' returns non-zero exit code (gotcha!)

Best Practice

Combine with trap for cleanup and meaningful error messages.

Code Example

#!/usr/bin/env bash
set -euo pipefail

# Trap for cleanup on exit
trap 'echo "Error on line $LINENO. Exit code: $?" >&2; cleanup' ERR
trap cleanup EXIT

cleanup() {
  rm -f "$TMPFILE" 2>/dev/null
}

TMPFILE=$(mktemp)

# Gotcha: local masks exit code!
my_func() {
  local result=$(false)  # Does NOT trigger set -e!
  result=$(false)        # This DOES trigger set -e
}

# Gotcha: arithmetic
(( count = 0 ))  # Exit code 1! Use: count=0 instead

# Safe default variable
DEPLOY_ENV=${DEPLOY_ENV:-staging}

# Pipefail in action
curl -sf https://api.example.com/health | jq '.status'
# Without pipefail: returns jq's exit code even if curl fails
# With pipefail: returns curl's non-zero exit code

💡

Interview Tip

The 'local' gotcha is what separates seniors from juniors. Most people know set -e but don't know that 'local var=$(cmd)' masks the exit code. Always split: local var; var=$(cmd).

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Use xargs -P for parallelism, awk for field extraction, trap for signal handling (SIGTERM/SIGINT), and temporary files with cleanup on exit.

Detailed Answer

Requirements

1. Process multiple log files concurrently 2. Extract error count per service name 3. Handle SIGTERM/SIGINT for graceful shutdown 4. Clean up temporary files on exit (success or failure) 5. Provide progress reporting

Key Techniques

- xargs -P N for bounded parallelism - awk for efficient text processing - trap for signal handling and cleanup - mktemp -d for safe temporary directory - flock for safe concurrent file writes - Process substitution <() for avoiding subshell variable issues

Code Example

#!/usr/bin/env bash
set -euo pipefail

MAX_PARALLEL=4
TMPDIR=$(mktemp -d)
RESULT_FILE="$TMPDIR/results.txt"
PID_FILE="$TMPDIR/pids.txt"

cleanup() {
  # Kill any background jobs
  if [[ -f "$PID_FILE" ]]; then
    while read -r pid; do
      kill "$pid" 2>/dev/null || true
    done < "$PID_FILE"
  fi
  rm -rf "$TMPDIR"
  echo "Cleanup complete" >&2
}
trap cleanup EXIT
trap 'echo "Interrupted" >&2; exit 130' INT TERM

process_log() {
  local file=$1
  local out="$TMPDIR/$(basename "$file").partial"
  
  awk '/ERROR/ {
    # Extract service name from log format: timestamp SERVICE ERROR message
    service = $2
    errors[service]++
  }
  END {
    for (s in errors) print s, errors[s]
  }' "$file" > "$out"
  
  echo "Processed: $file ($(wc -l < "$out") services)" >&2
}
export -f process_log
export TMPDIR

# Find log files and process in parallel
find /var/log/app/ -name '*.log' -mtime -1 -print0 | \
  xargs -0 -P "$MAX_PARALLEL" -I {} bash -c 'process_log "$@"' _ {}

# Aggregate partial results
awk '{
  errors[$1] += $2
}
END {
  for (s in errors) printf "%-30s %d\n", s, errors[s]
}' "$TMPDIR"/*.partial | sort -t' ' -k2 -rn > "$RESULT_FILE"

echo "=== Error Count by Service ==="
cat "$RESULT_FILE"

💡

Interview Tip

Show production-safe patterns: trap for cleanup, mktemp for temp files, bounded parallelism with xargs -P, and graceful signal handling. This demonstrates you write scripts that won't leave orphan processes or temp files.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Process substitution <(cmd) creates a temporary file descriptor for a command's output, allowing commands that require file arguments to read from another command's output. Unlike pipes, it doesn't create a subshell for the reading command.

Detailed Answer

Process Substitution

<(command) creates a /dev/fd/N file descriptor that contains the output of command. The receiving command reads from it as if it were a regular file.

Why it matters vs pipes

With a pipe: cmd1 | while read line; do ((count++)); done; echo $count → The while loop runs in a subshell, so $count is lost after the pipe ends.

With process substitution: while read line; do ((count++)); done < <(cmd1); echo $count → The while loop runs in the current shell, so $count is preserved.

Use Cases

1. Diff two commands: diff <(sort file1) <(sort file2) — compare sorted outputs without temp files 2. Feed multiple inputs: paste <(cut -f1 file1) <(cut -f2 file2) — combine fields from different files 3. Preserve variables in loops: Read from process substitution instead of piping into a while loop 4. Named pipe without mkfifo: Process substitution handles creation and cleanup automatically

Output process substitution `>(command)` writes to a command as if it were a file

tee >(gzip > backup.gz) >(wc -l > count.txt) > /dev/null

Code Example

# Problem: pipe creates subshell, variable lost
count=0
echo -e "a\nb\nc" | while read -r line; do
  ((count++))
done
echo "Count: $count"  # Prints 0! (subshell)

# Solution: process substitution
count=0
while read -r line; do
  ((count++))
done < <(echo -e "a\nb\nc")
echo "Count: $count"  # Prints 3! (current shell)

# Diff two remote files without downloading
diff <(ssh server1 cat /etc/nginx/nginx.conf) \
     <(ssh server2 cat /etc/nginx/nginx.conf)

# Compare Kubernetes resources across clusters
diff <(kubectl --context=prod get deploy -o yaml) \
     <(kubectl --context=staging get deploy -o yaml)

# Tee to multiple destinations
tar czf - /var/log | tee >(md5sum > checksum.txt) > backup.tar.gz

💡

Interview Tip

The subshell variable loss problem is the classic gotcha that catches even experienced engineers. Process substitution is the clean solution. The diff two remote files example is memorable and practical.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Use awk to parse space-delimited NGINX log format, associative arrays for aggregation, and END block for reporting. Combine with sort and head for top-N queries.

Detailed Answer

NGINX Default Log Format

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time

Fields: IP(-) user [timestamp] "method path proto" status bytes "referer" "ua" response_time

Why awk over other tools

- Processes files line-by-line with minimal memory - Built-in field splitting (perfect for log formats) - Associative arrays for aggregation - No external dependencies - Faster than Python/Ruby for simple text processing

Performance tip: For very large files (10GB+), use mawk instead of gawk — it's 5-10x faster for simple processing. Or use LC_ALL=C to skip locale processing.

Code Example

# Top 10 IPs by request count
awk '{ip[$1]++} END {for (i in ip) print ip[i], i}' access.log | sort -rn | head -10

# Request rate per second
awk '{split($4, a, "["); split(a[2], b, ":"); sec=b[2]":"b[3]":"b[4]; rate[sec]++}
END {for (s in rate) print s, rate[s], "req/s"}' access.log | sort -t: -k1,3 | tail -20

# Slow requests (>1s response time, last field)
awk '{if ($NF > 1.0) print $NF"s", $7, $1}' access.log | sort -rn | head -20

# HTTP status code distribution
awk '{status[$9]++} END {for (s in status) printf "%s: %d (%.1f%%)\n", s, status[s], status[s]/NR*100}' access.log | sort

# Bandwidth per endpoint (MB)
awk '{endpoint=$7; bytes[endpoint]+=$10}
END {for (e in bytes) printf "%.2f MB  %s\n", bytes[e]/1048576, e}' access.log | sort -rn | head -10

# Error rate per minute
awk '$9 >= 500 {split($4, a, "["); split(a[2], t, ":"); min=t[2]":"t[3]; errors[min]++}
{split($4, a, "["); split(a[2], t, ":"); min=t[2]":"t[3]; total[min]++}
END {for (m in total) printf "%s  %.2f%% errors (%d/%d)\n", m, errors[m]/total[m]*100, errors[m]+0, total[m]}' access.log | sort

💡

Interview Tip

Awk one-liners for log analysis are a must-know for SRE interviews, especially at Google. Practice these until they're second nature. The key patterns: associative arrays for counting, END block for output, sort | head for top-N.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Quote all variables, validate user input against allowlists, use mktemp for temp files, avoid eval, use -- to end option parsing, and prefer built-in string operations over external commands.

Detailed Answer

Command Injection Prevention

- Always double-quote variables: "$var" not $var - Never use eval with user input - Never use backticks; use $() for command substitution - Use -- to end option parsing: grep -- "$pattern" file - Validate input: [[ "$input" =~ ^[a-zA-Z0-9_-]+$ ]]

Path Traversal Prevention

- Use realpath or readlink -f to canonicalize paths - Validate that resolved path is within expected directory - Never construct paths from untrusted input without validation

Race Condition Prevention

- Use mktemp for temporary files (atomic creation with unique name) - Use flock for file-based locking - Check-then-act is always unsafe; use atomic operations - Use noclobber (set -C) to prevent overwriting existing files

General Safety

- set -euo pipefail at the top - Trap EXIT for cleanup - Use readonly for constants - Prefer [[ ]] over [ ] for conditionals (no word splitting)

Code Example

#!/usr/bin/env bash
set -euo pipefail

readonly ALLOWED_DIR="/var/app/data"
readonly VALID_NAME_RE='^[a-zA-Z0-9_-]+$'

# Validate input (never trust user input)
validate_filename() {
  local name=$1
  if [[ ! "$name" =~ $VALID_NAME_RE ]]; then
    echo "Invalid filename: $name" >&2
    return 1
  fi
  # Resolve full path and check it's within allowed directory
  local full_path
  full_path=$(realpath -m "${ALLOWED_DIR}/${name}")
  if [[ "$full_path" != "${ALLOWED_DIR}/"* ]]; then
    echo "Path traversal detected: $name" >&2
    return 1
  fi
  echo "$full_path"
}

# Safe temp file with cleanup
TMPFILE=$(mktemp /tmp/safe-script.XXXXXX)
trap 'rm -f "$TMPFILE"' EXIT

# File locking to prevent race conditions
(
  flock -n 200 || { echo "Another instance running" >&2; exit 1; }
  # Critical section here
  echo "Processing..."
) 200>/var/lock/my-script.lock

# Never do this:
# eval "echo $user_input"          # Command injection!
# cat /data/$user_input             # Path traversal!
# if [ -f $file ]; then rm $file    # Word splitting + race condition!

💡

Interview Tip

Security-conscious scripting is rare and impressive in interviews. Mention the three attack vectors (injection, traversal, race conditions) and the specific mitigations. The realpath check for path traversal is a strong signal.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

It makes bash exit on errors (-e), on unset variables (-u), and propagate failures through pipelines (pipefail).

Detailed Answer

By default bash ignores most failures and keeps going, which hides bugs. -e exits on any command failure, -u treats using an unset variable as an error, and pipefail makes a pipeline fail if any stage fails (not just the last). It's the standard safety preamble for production scripts.

Code Example

#!/usr/bin/env bash
set -euo pipefail

💡

Interview Tip

Reciting this preamble signals you write robust scripts.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Unquoted $var undergoes word-splitting and glob expansion, breaking on spaces or special characters.

Detailed Answer

"$var" preserves the value as a single word; unquoted it splits on whitespace and expands globs, causing subtle bugs with filenames containing spaces. Always quote expansions and use "${arr[@]}" for arrays. shellcheck flags missing quotes automatically.

Code Example

for f in "$@"; do echo "processing: $f"; done

💡

Interview Tip

Mention shellcheck as the linter that catches this.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

"$@" expands each argument as a separate quoted word; "$*" joins all arguments into one string.

Detailed Answer

Inside double quotes, "$@" is what you almost always want — it preserves argument boundaries so files with spaces stay intact. "$*" concatenates with IFS (space by default) into a single word, which loses boundaries. Unquoted, both word-split.

💡

Interview Tip

"$@" for forwarding args is the practical takeaway.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Use mktemp and clean up with a trap on EXIT.

Detailed Answer

Hard-coded /tmp/foo is a race and a security risk. mktemp creates a uniquely named file/dir with safe permissions; register a trap 'rm -rf "$tmp"' EXIT so it's removed even if the script exits early. This avoids leaks and predictable-name attacks.

Code Example

tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

💡

Interview Tip

Pair mktemp with a trap EXIT cleanup.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

[ ] is the POSIX test command, [[ ]] is bash's safer conditional (no word-splitting, regex/&&), (( )) is arithmetic evaluation.

Detailed Answer

[[ ]] avoids the quoting pitfalls of [ ], supports == pattern matching, =~ regex, and && / ||. (( )) evaluates integer math and returns success on non-zero. Prefer [[ ]] in bash scripts; use [ ] only for POSIX-sh portability.

Code Example

if [[ $count -gt 0 && $name == prod-* ]]; then ...; fi

💡

Interview Tip

Recommend [[ ]] in bash for safety.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Capture output with $(...) and check $? immediately, or use PIPESTATUS for pipelines.

Detailed Answer

out=$(cmd) captures stdout; the exit code is in $? on the next line (check it before running anything else). In a pipeline, $? is only the last command's status — use ${PIPESTATUS[0]} to inspect an earlier stage (this is why pipefail matters).

💡

Interview Tip

Mention PIPESTATUS for pipelines — a common gotcha.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

trap runs a handler on signals or pseudo-signals like EXIT, ERR, INT — commonly for cleanup.

Detailed Answer

trap 'cleanup' EXIT guarantees cleanup runs however the script ends. trap on ERR can log context on failure; on INT/TERM you can gracefully stop. It's how robust scripts avoid leaving temp files, locks, or partial state behind.

💡

Interview Tip

EXIT-trap cleanup is the canonical example.

Open Full Page ↗

💬 Comments

⚡

Quick Answer

Use `while IFS= read -r line; do ... done < file` to preserve whitespace and backslashes.

Detailed Answer

for line in $(cat file) word-splits and glob-expands — wrong for lines with spaces. while IFS= read -r line reads one line at a time without trimming whitespace (IFS=) or mangling backslashes (-r). It's the correct idiom for line processing.

Code Example

while IFS= read -r line; do echo "$line"; done < input.txt

💡

Interview Tip

Know why for-in-cat is an anti-pattern.

Open Full Page ↗

💬 Comments

Context

An SRE team used a 240-line Bash script to restart stuck Kubernetes workers during incidents. It worked for the original author but failed unpredictably for new on-call engineers.

Problem

The script assumed current namespace, ignored failed commands, used unquoted variables, and selected pods with broad grep patterns. During one incident it restarted workers in staging while production remained unhealthy.

Solution

The team split the script into small functions, required explicit environment and service arguments, enabled strict mode, logged every mutation, and added dry-run output. They paired the script with a runbook explaining when not to use it.

Commands

./worker-recover.sh --env prod --service payments-worker --dry-run # Shows intended actions without mutating production

./worker-recover.sh --env prod --service payments-worker --confirm # Runs only after explicit confirmation

📈

Outcome

Recovery remained fast, but accidental environment mistakes stopped. New on-call engineers could use the runbook without tribal knowledge.

💡

Lessons Learned

A script is an interface. If the interface is vague, the incident will supply the worst possible default.

#bash#runbook#incident-response#kubernetes

◈ Architecture Diagram

┌──────────┐
│ Learn    │
└────┬─────┘
     ↓
┌──────────┐
│ UseCase  │
└────┬─────┘
     ↓
┌──────────┐
│ Operate  │
└──────────┘

💬 Comments

Context

An ops repo of ~300 Bash scripts accreted over a decade — deployment helpers, backup jobs, incident tooling — with no linting, no tests, and a culture of 'edit carefully and hope'. Two incidents in a year traced to unquoted variables meeting filenames with spaces.

Problem

Scripts were production code with none of production code's safety net: changes shipped on eyeball review, regressions surfaced only in production (often silently — Bash's default error handling swallows failures), and nobody dared refactor the scary old ones.

Solution

Brought the scripts under engineering discipline incrementally: ShellCheck in CI as a blocking gate (baseline-grandfathered — existing warnings frozen as a ratchet that only shrinks; new code must be clean), bats-core test suites for the 40 highest-blast-radius scripts (mocking external commands via PATH-shimming, asserting behavior on edge inputs: spaces, empty args, missing files), a shared prelude library (strict mode, logging, locking, retry helpers) replacing 300 divergent headers, and a template for new scripts with the prelude, argument parsing, and a test skeleton.

Commands

CI: shellcheck -S warning $(git diff --name-only origin/main -- '*.sh'); baseline ratchet for legacy

bats test/deploy_helpers.bats  # PATH-shim mocks for kubectl/aws

prelude: source lib/prelude.sh  # set -Eeuo pipefail, trap ERR, log(), with_lock(), retry()

📈

Outcome

The unquoted-variable incident class ended (ShellCheck catches it mechanically); the baseline ratchet burned 4,100 legacy warnings down to 600 in a year as scripts got touched; the scary scripts got refactored once tests made refactoring safe; and new-script quality jumped because the template made the right thing the default.

💡

Lessons Learned

The grandfathered baseline was the adoption trick — blocking on 4,100 existing warnings would have killed the initiative in week one. PATH-shim mocking makes Bash surprisingly testable; the barrier was cultural, not technical.

#bash#shellcheck#bats#testing#ci

💬 Comments

Context

A 400-host VM estate (pre-Kubernetes legacy) administered by a serial for host in $(cat hosts); do ssh ...; done pattern: a fleet-wide config push took 3+ hours, and one hung host stalled everything behind it.

Problem

Serial execution made routine changes half-day events; a single unreachable host hung the loop at TCP timeout per attempt; output interleaved uselessly (which host said what?); and partial failures vanished — the loop's exit code reflected only the last host.

Solution

Rebuilt as a small, disciplined runner: xargs -P 30 fans out SSH with per-host timeout (ConnectTimeout + command timeout via timeout(1)), each host's output captured to its own log file, per-host exit codes recorded to a results directory, and a summary pass reporting ok/failed/timeout lists with non-zero exit if any host failed. Idempotent by design (the pushed script itself is idempotent, so re-running the failed subset is safe), with a --retry-failed flag consuming the previous run's failure list.

Commands

xargs -a hosts.txt -P 30 -I{} bash -c 'timeout 120 ssh -o ConnectTimeout=5 -o BatchMode=yes {} bash -s < push.sh > logs/{}.out 2>&1; echo $? > results/{}.rc'

summary: grep -L '^0$' results/*.rc -> failed-hosts.txt; exit $([ -s failed-hosts.txt ])

rerun: ./fleet-run --hosts failed-hosts.txt

📈

Outcome

Fleet pushes dropped from 3+ hours to ~6 minutes (30-way parallelism, hung hosts cost 120s not the whole run); per-host logs made failures diagnosable at a glance; the ok/failed/timeout summary plus retry flag turned partial failure from a mystery into a workflow. The tool became the standard fleet primitive until the estate finished migrating to real orchestration.

💡

Lessons Learned

BatchMode=yes and explicit timeouts are what make SSH automation honest — interactive prompts and TCP hangs are the silent killers of naive loops. Writing per-host exit codes to files (not relying on the loop's status) is the difference between a tool and a hope.

#bash#xargs-parallel#ssh#fleet-management#timeouts

💬 Comments

MediumResolution time: ~15-40 min

Symptom

4:55 PM: release job was green, but production pods kept running the previous image and the change window closed.

Error Message

error: error validating "deploy.yaml": error validating data: ValidationError(Deployment.spec.template.spec.containers[0]): missing required field "name"

Root Cause

The script piped `kubectl apply` output through `tee` without `set -o pipefail`, so the pipeline exit status came from `tee`, not from `kubectl`. The CI system saw exit code 0 and marked the deployment successful. This is a shell semantics issue, not a Kubernetes rollout issue. The underlying issue was a combination of factors that individually seemed harmless but together created a cascading failure. The monitoring that should have caught this early was either missing or configured with thresholds too high to trigger before user impact. The on-call engineer initially pursued the wrong hypothesis because the symptoms resembled a different, more common failure mode. By the time the real root cause was identified, the blast radius had expanded beyond the original service to affect downstream dependencies. This incident exposed a gap in the team's runbooks and highlighted the need for better correlation between metrics, logs, and traces during diagnosis.

Diagnosis Steps

1.grep -n "kubectl apply" deploy.sh # Find pipeline usage
2.bash -n deploy.sh # Validate shell syntax
3.set -o pipefail; kubectl apply -f deploy.yaml | tee apply.log # Reproduce correct failure behavior

Solution

Enable `set -Eeuo pipefail`, add an ERR trap, and explicitly wait for rollout status after apply. Do not rely on log output as proof of success; rely on exit status and post-deploy verification.

Commands

shellcheck deploy.sh

kubectl rollout status deploy/payments-api -n payments --timeout=180s

🛡

Prevention

Lint shell scripts, test failure paths in CI, and wrap deployment commands in small functions. Prefer purpose-built deployment tools for complex rollouts.

bashpipefailci-cdkubectl

◈ Architecture Diagram

┌──────────┐
│ Signal   │
└────┬─────┘
     ↓
┌──────────┐
│ Incident │
└────┬─────┘
     ↓
┌──────────┐
│ Action   │
└──────────┘

💬 Comments

HighResolution time: ~Restore + rebuild ~6 hours; the script rewrite an afternoon; ShellCheck would have flagged the unquoted expansion in one second

Symptom

A nightly artifact-cleanup job runs 'successfully' — and the following morning, builds fail fleet-wide: the shared artifacts directory is empty. Not just old artifacts: everything, including the retention-policy metadata and adjacent tool directories.

Error Message

No error. The job's log shows: 'Cleaning old artifacts in: ' (note the blank) followed by rm's silent success. The variable holding the target subdirectory was empty, and rm -rf "$BASE/$TARGET"/* had been written as rm -rf $BASE/$TARGET/* — unquoted, with $TARGET empty, expanding to rm -rf /shared/artifacts//* … the parent's entire contents.

Root Cause

Three stacked failures: the script fetched the cleanup target list from an internal API whose response format changed (a field rename), so jq extracted an empty string with exit code 0; no validation guarded 'target must be non-empty and match the expected pattern'; and the unquoted expansion turned empty-variable into parent-directory glob. set -e provided no protection because nothing failed — every command succeeded at doing the wrong thing.

Diagnosis Steps

1.Reconstruct the executed command from the log's blank target line — the empty variable is visible in the output
2.Trace the emptiness upstream: API response diff shows the field rename; jq '.old_field' returns empty, rc=0
3.Confirm the expansion mechanics in a sandbox: unquoted empty var + glob = parent contents
4.Audit the repo for the same pattern (rm -rf with unquoted/unvalidated variables): five more scripts, fixed same week

Solution

Restored from the previous night's backup (4 hours of rebuild for uncovered gaps). Rewrote the script under the org's new prelude: strict mode plus explicit input validation ([[ -n "$TARGET" && "$TARGET" =~ ^[a-z0-9-]+$ ]] || die), all expansions quoted, rm targets resolved with realpath and asserted to live under the allowed root, and — the structural fix — destructive operations now require the target to be enumerated and logged first, with a count sanity check (deleting 40x the daily average requires a --force flag a human must add).

Commands

shellcheck cleanup.sh  # SC2086 on the unquoted expansion

guard: [[ -n "$TARGET" && "$TARGET" =~ ^[a-z0-9_-]+$ ]] || die 'invalid target'

guard: [[ $(realpath "$dir") == /shared/artifacts/* ]] || die 'outside allowed root'

🛡

Prevention

Validate inputs before destructive use, always — especially inputs from APIs, which change shape without notice and 'succeed' at returning garbage. Quote every expansion (ShellCheck enforces this mechanically). Bound blast radius structurally: destructive scripts assert their target is inside an allowlisted root and balk at anomalous scale. jq pipelines need explicit empty-result handling; exit 0 with empty output is its default failure mode.

bashunquoted-variablesrm-rfinput-validationblast-radius

💬 Comments

⚡ reliability

Why It Matters

Bash scripts often begin as glue and become deployment systems. Unquoted variables, missing `pipefail`, non-idempotent operations, and ambiguous globbing can turn a normal release into a partial production change with a green CI status. Without following this practice, teams typically discover the problem during a production incident. The cost of fixing issues reactively is 10-100x higher than preventing them proactively. This becomes especially dangerous at scale when multiple teams depend on the same infrastructure, because one team's shortcut becomes another team's outage. Organizations that skip this practice often find themselves in a cycle of firefighting instead of building. The pattern is predictable: it works fine in development, survives staging, and fails spectacularly in production under real traffic and real failure conditions.

How to Implement

Use strict mode, quote variables, validate inputs, create temporary files safely, check command exit status, and verify runtime state after mutation. Keep scripts small enough that failure paths are testable. Start by implementing this in a single non-critical service to validate the approach. Document the configuration and share it with the team through a runbook or internal wiki page. Add monitoring and alerting around the key metrics so you know when the practice is working and when it is not. Gradually roll it out to more services, adjusting the configuration based on what you learn from each deployment. Review the implementation quarterly to ensure it still matches your current architecture and traffic patterns.

Example

#!/usr/bin/env bash
set -Eeuo pipefail # Fails on unset variables, failed commands, and failed pipelines.
trap 'echo "failed at line $LINENO" >&2' ERR # Reports the failing line.
: "${SERVICE:?SERVICE is required}" # Requires a service name before mutating production.
kubectl rollout status "deploy/${SERVICE}" -n payments --timeout=180s # Verifies the runtime change completed.

✗ Anti-pattern

The anti-pattern is a long script with `set +e` around risky commands so the pipeline can continue. That optimizes for green builds, not correct production state.

bashstrict-modeidempotencydeployment

◈ Architecture Diagram

┌──────────┐   ┌──────────┐
│ ✓ Design │   │ ✗ Guess  │
└────┬─────┘   └────┬─────┘
     ↓              ↓
┌──────────┐   ┌──────────┐
│ Recover  │   │ Escalate │
└──────────┘   └──────────┘

💬 Comments