Python DevOps Automation

This is like a recipe that works perfectly in your own kitchen with your own knives, oven, and pantry, but fails when a caterer tries to run it in an unfamiliar commercial kitchen with different equipment and a stricter time limit — the recipe itself didn't change, but every assumption about what's available and how much time you have quietly shifted.

Python DevOps automation scripts are deceptively easy to write correctly for a developer's own machine, because a laptop has a stable, long-lived environment: cached AWS credentials, a persistent Python virtualenv, unlimited wall-clock time, and reliable network access. Production execution environments, Lambda, CI runners, Kubernetes CronJobs, break every one of those assumptions in different ways, which is exactly why "works on my machine" is such a common and specific failure category for this kind of script, distinct from a plain logic bug.

Credential resolution is the most frequent culprit: boto3's default credential chain checks environment variables, then shared credential files, then an EC2, ECS, or Lambda execution role, in a specific order — a script that works locally because it silently picked up your personal AWS profile will behave completely differently in Lambda, where it must instead pick up the function's IAM execution role, and any hardcoded profile name or region assumption breaks immediately. Connection behavior is the second most common issue: HTTP clients and boto3 clients that create fresh connections per invocation, typical in Lambda since the runtime environment may be reused or may be entirely fresh per invocation unpredictably, behave very differently from a long-running local process that reuses one connection pool for its entire lifetime, and default timeouts tuned for a patient human at a laptop are often too generous for a Lambda function with a hard execution time limit.

In production, the fix is to make environment assumptions explicit rather than implicit: log the resolved credential source, since calling get_credentials().method on a boto3 session reveals whether credentials came from an environment variable, a profile, or an IAM role, pin dependency versions exactly, since an unpinned requirements file can resolve differently between a developer's cached pip environment and a fresh CI or Lambda build, and set explicit, conservative timeouts rather than relying on SDK defaults that assume an interactive session. Monitoring should track cold-start-specific failures separately from warm-execution failures, since a script that only fails on the first invocation after a deployment often points at initialization-time issues, such as module-level client construction or credential resolution, rather than logic bugs.

The non-obvious gotcha is that Lambda execution environments are reused across invocations for performance, meaning any client object created outside the handler function, at module import time, persists across multiple invocations, including its cached connections and any state. This is usually a good optimization, but it means a client configured with credentials that were valid at cold start can silently hold onto now-stale session tokens if a script assumes credentials never need refreshing during a long-lived Lambda container's life, causing intermittent authentication failures that only appear after the container has been warm for a while, which is very hard to reproduce locally since a laptop script never experiences this specific reuse pattern.

This is like a form that says "click submit to place your order" with no confirmation of whether the order already went through — if the page seems to hang and you click submit again just to be safe, you might end up with two identical orders showing up at your door, because the system had no way to recognize "this is the same order being resubmitted" versus "this is a genuinely new order."

True idempotency, an operation that, no matter how many times you perform it, results in the same final state, requires deliberate design; it does not happen automatically just because a script "usually doesn't error." Automation scripts are retried constantly in production — a Lambda timing out mid-execution, a CI job getting killed by a runner restart, a network blip during an API call whose response never arrives even though the request succeeded server-side, and any of these can trigger an at-least-once retry, meaning the same logical operation may genuinely execute more than once against the same target.

The fix has two complementary layers. First, a describe-then-act pattern: before creating any resource, the script checks whether an equivalent resource already exists, by a stable identifier like a name tag or a deterministic naming convention derived from the input, not a randomly generated ID that changes every run, and only proceeds with creation if it's genuinely missing. Second, where the underlying API supports it, use client-side idempotency tokens — many AWS APIs, like EC2's RunInstances ClientToken parameter, let you pass a unique-per-logical-operation token that the API itself uses to deduplicate: if you retry with the same token, AWS recognizes it as the same request and returns the original result instead of creating a second resource, even if your first request's response was lost due to a network failure and you have no local memory of whether it succeeded.

In production, the describe-then-act pattern needs to handle a subtle race: if two instances of the automation run concurrently, which happens more often than expected, an overlapping cron schedule or a retried Lambda invocation racing the original before it's confirmed dead, both can pass the "doesn't exist yet" check before either creates the resource, resulting in a duplicate anyway. Mature automation adds a distributed lock, a DynamoDB conditional write, a Redis-style atomic set, or a cloud-native equivalent, around the describe-then-act sequence specifically to close this race window, not just the idempotency token alone.

The non-obvious gotcha is that idempotency tokens typically have a limited validity window, EC2's ClientToken, for example, is only guaranteed to deduplicate requests within several hours, so a retry that happens much later, say a manually re-triggered pipeline run days after the original failure using a script that generates a fresh token each time rather than persisting the token used for the original attempt, will not be deduplicated by the API at all, silently reintroducing the duplicate-resource problem the token was supposed to prevent. The token only protects against retries close in time to the original attempt, not against a human manually re-running the same logical job long after.

This is like a security guard who's supposed to check every door in a building every night and radio in a count of doors checked — if the guard simply stops making rounds, or starts checking zero doors because someone locked them out of half the building, there's no alarm bell ringing anywhere; the absence of the guard's report is itself the only signal, and it's silent unless someone is specifically listening for a missing check-in.

DevOps automation scripts are especially prone to a specific and dangerous failure mode: silent success. A script that exits with code zero and prints "scan complete" is indistinguishable, from a basic cron or CI monitoring perspective, from a script that genuinely found and reported zero issues versus one whose IAM permissions were quietly revoked and now silently returns empty results for every query it makes. Standard process-level monitoring, did the job run, did it exit non-zero, is necessary but not sufficient, because it can't distinguish "nothing to report" from "couldn't check anything."

The fix is to make every automation script emit a structured outcome record on every run, not just logs, but a small piece of machine-readable telemetry, items scanned, items flagged, items remediated, duration, exit status, pushed to a metrics system or a dead-man's-switch style heartbeat service. Critically, the script should also track its own expected baseline: if a nightly untagged-instance scanner has historically found five to fifteen flagged instances per run for months, and one night it reports zero, that's worth investigating even though "zero findings" is technically a valid, non-error result, the same way a fire alarm system that suddenly stops reporting any smoke at all, ever, might mean the building is clean or might mean the sensor died.

In production, mature platforms wire this into a dead-man's-switch pattern: an external monitoring service expects a heartbeat ping from each scheduled automation job within its expected run window, and pages if the ping doesn't arrive — this catches the case where the job doesn't run at all, a cron misconfiguration, a CI schedule that got silently disabled, a Kubernetes CronJob suspended during a cluster migration and never resumed, which produces literally zero signal in application-level logs since the application never even started. Combine this with anomaly detection on the outcome counts themselves, items scanned dropping to zero, or items flagged deviating wildly from the historical range, to catch the cases where the job runs but does the wrong thing.

The non-obvious gotcha is that adding more logging alone doesn't solve this problem — a script that logs "scan complete, zero issues found" in exactly the same format whether that's because the environment is genuinely clean or because a credential expired and every API call returned an empty result due to a caught, swallowed exception, provides zero additional signal over the original silent failure. The instrumentation has to explicitly distinguish and separately track "zero because nothing was found" from "zero because we couldn't check," for example by catching and specifically counting AccessDenied or throttling exceptions rather than letting a generic try/except silently produce the same empty result as a clean scan, for this kind of monitoring to actually catch the failure class it's meant to catch.

This is like giving one person a master key that opens every door in a hospital so they can respond quickly to any room needing attention — enormously convenient for getting things done fast, and also exactly the single point of failure an attacker or a bug would most want to compromise, because one mistake or one theft compromises every room in the building at once instead of just one.

A remediation platform is different from a one-off script precisely because it's designed to act broadly and repeatedly across many teams' infrastructure, which means its own reliability and security properties become organization-wide risk, not team-local risk. This is why mature platforms treat every destructive action as requiring the same rigor as a financial transaction: a clear record of what was proposed, who or what approved it, what actually executed, and what the verified outcome was, not because engineers don't trust each other, but because at organizational scale, "trust me, it worked" doesn't scale and doesn't survive an incident post-mortem.

The pipeline design separates concerns explicitly. Discovery gathers current state read-only. Classification applies declarative policy, not ad-hoc if-statements scattered through the code, to decide what qualifies for remediation, making the policy itself reviewable and testable independent of the execution engine. Plan generation produces a diff, exactly what would change, before anything happens, in both human-readable and machine-readable form. Approval gates scale the rigor to the risk: a low-risk, easily-reversible action, such as disabling an unused API key, might auto-approve after a dry-run review window with no objections, while a high-risk, hard-to-reverse action, such as deleting a database or revoking a production access role, requires an explicit human approval tied to a specific reviewer identity. Execution runs in small batches with rate limiting, per-target error isolation so one failing target doesn't block or corrupt the rest of the batch, and immediate verification that the intended state was actually achieved, not just that the API call didn't error.

In production, the platform's own credentials become the highest-priority thing to defend, since compromising the remediation platform is strictly more valuable to an attacker than compromising any single team's resources — this argues for short-lived, action-scoped credentials generated per remediation run rather than a standing broad-access service account, and for running the execution phase in an isolated environment separate from where policy and approval logic live, so a bug or compromise in the approval UI can't directly trigger execution. What to monitor: approval-to-execution latency, since a bottleneck here means teams start routing around the platform with manual fixes, defeating its purpose, false-positive rate in classification, since teams losing trust in flagged findings stop reviewing them carefully, and the platform's own audit log completeness, since a gap in the audit trail during an actual incident is often worse than the incident itself for a security post-mortem.

The non-obvious gotcha is that centralizing remediation this way creates organizational pressure to expand its scope over time — once a platform can safely disable unused IAM roles, someone will ask it to also clean up S3 buckets, then rotate secrets, then terminate instances, and each addition, taken alone, seems reasonable, but the platform's aggregate blast radius grows quietly until it has more standing destructive capability across the entire company than any single team's on-call engineer. Senior engineers push back on scope creep by requiring each new remediation type to justify its own approval tier and audit requirements independently, rather than inheriting the platform's existing trust level by default.

Think of an automation script like a dispatcher sending technicians to data centers. If the dispatcher writes a vague instruction on a sticky note, the technician may interpret it differently, take too long, or do something dangerous. A production Python script calling kubectl, aws, helm, or terraform needs precise instructions, a deadline, and a clean incident report when something fails.

In Python, the safe default is subprocess.run with a list of arguments and shell=False. The Python documentation recommends passing arguments as a sequence because Python can handle escaping and quoting without involving a shell. That matters because shell=True turns one string into shell syntax, so characters such as semicolons, pipes, dollar expansion, and backticks become meaningful. In interview terms, this is not just style; it is a security boundary.

A strong implementation wraps subprocess calls in one function. The wrapper sets timeout, text=True, capture_output=True, check=True, cwd only when needed, and a minimal env when credentials or PATH should be controlled. It catches subprocess.TimeoutExpired and subprocess.CalledProcessError separately because a timeout is an operational failure while a non-zero exit is a command-level failure. The wrapper should log the command safely, redacting tokens and kubeconfigs, and it should never dump secrets from stderr into chat tools or tickets.

At scale, the tricky behavior is that automation failures are often partial. A kubectl apply may fail on one manifest after mutating another. An aws command may succeed but return paginated or eventually consistent data. A hung process can hold a deployment lock and block the next run. That is why serious automation includes idempotency, dry-run mode, bounded retries around known transient errors, and audit output that lets an on-call engineer prove exactly what was attempted.

The non-obvious gotcha is that avoiding shell=True does not automatically make input safe. If a user can pass --kubeconfig, --context, --namespace, or a file path, they can still redirect the automation into the wrong cluster or sensitive files. Senior candidates explain both layers: command invocation safety and business-rule validation of the arguments being passed.

Imagine asking 200 branch offices to send you a list of every laptop they own. If you call only the first office, stop after the first page, or hide which offices did not answer, the final spreadsheet looks clean but lies. AWS inventory automation has the same failure mode: the script can produce a file while silently missing accounts, regions, or pages.

The core boto3 pattern is explicit. First call STS AssumeRole into the target account using a role created for inventory. Then build service clients with those temporary credentials and the target region. For list-style APIs, use paginators because many AWS APIs return only a limited page and a continuation token. The boto3 retry guide documents standard and adaptive retry modes, and production collectors should set retry mode and max attempts intentionally rather than relying on unknown environment defaults.

Concurrency must be bounded. A script that fans out 200 accounts times 25 regions times 10 services can create tens of thousands of API calls. Unbounded ThreadPoolExecutor usage will cause throttling, noisy CloudTrail, and sometimes organization-wide guardrail alerts. A mature collector uses a small worker pool, per-service backoff, and clear retry classification: throttling and transient network errors are retried, AccessDenied is recorded as an authorization gap, and validation errors are treated as code or configuration defects.

The output format matters as much as the API calls. Each account-region-service tuple should report status, item count, elapsed time, and error class. That lets platform teams distinguish no resources from no permission. For auditability, include the assumed role ARN, collection timestamp, script version, and schema version in the artifact. Large companies ask this because inventory data drives security posture, cost cleanup, and incident response.

The non-obvious gotcha is eventual consistency. A newly created role, instance, tag, or region opt-in may not appear immediately. Senior candidates mention rerunning collectors, comparing deltas, and designing consumers to tolerate late-arriving data instead of treating one scan as absolute truth.

Think of this like checking 10,000 doors in a city. Sending one inspector door by door is too slow, but sending 10,000 inspectors at once blocks streets and creates a new incident. Good async automation is controlled parallelism: enough concurrency to finish quickly, but not so much that DNS, load balancers, proxies, or the service itself collapse.

Python asyncio is useful for I/O-bound work because the event loop can switch tasks while each request waits on the network. The design should separate three concerns: target generation, bounded execution, and result collection. A semaphore limits the number of in-flight checks. Each request has connect and read timeouts. The caller gets one result per endpoint with status, latency, error class, and retry count.

Modern Python adds structured concurrency through asyncio.TaskGroup. A TaskGroup gives a clear lifetime to a set of related tasks: create them inside the block, wait for completion, and let errors cancel sibling tasks in a controlled way. For health-check automation, many teams choose to catch per-target exceptions inside the worker so one bad endpoint does not cancel the whole scan. For deployment gates, they may do the opposite: fail fast when critical checks break.

At scale, the bottleneck may not be Python. DNS resolvers, NAT gateways, outbound proxies, and service-side rate limits all matter. A mature script includes a configurable concurrency limit, jittered retries for transient network errors, and an allowlist of target domains or clusters. It also emits metrics about timeout rate and p95 latency because the health checker itself becomes part of the deployment system.

The subtle gotcha is cancellation hygiene. If a CI job times out or a deploy is aborted, the script should stop launching new checks and close HTTP connections cleanly. Senior engineers talk about cancellation, finalizers, and bounded resource cleanup, not just await gather.

A Kubernetes watcher is like a security desk watching badge events at a large office. Asking for the full list every five seconds is wasteful and still misses context. Subscribing to a stream and remembering the last seen position is cheaper and more accurate. That is the mental model interviewers want when they ask about Kubernetes automation in Python.

The official Kubernetes Python client exposes CoreV1Api and watch.Watch. A production watcher usually starts with a list_namespaced_pod or list_pod_for_all_namespaces call to get the current resourceVersion, then uses watch.stream against the same list function. The resourceVersion is important because Kubernetes objects are versioned; it lets the watcher resume from a known point instead of starting from scratch every time.

CrashLoopBackOff is not a pod phase. It appears in container status under state.waiting.reason, often alongside restartCount and lastState. Good code inspects initContainerStatuses and containerStatuses, records the exact container, and includes namespace, pod name, node, image, restart count, and last termination reason. It deduplicates so one broken pod does not create 500 Slack messages.

At scale, the watcher must respect API server limits. Use watch timeouts so the HTTP stream refreshes periodically. Handle 410 Gone by relisting because the resourceVersion is too old. Avoid one watcher per namespace unless needed; a single all-namespaces watch is usually simpler. Emit metrics for reconnects, event lag, and handler errors so the automation itself is observable.

The non-obvious gotcha is RBAC and blast radius. A watcher running in-cluster should use a ServiceAccount with list/watch on pods only, not cluster-admin. If it creates incident objects, write those to a narrow namespace or external queue, not back into every application namespace.

A remediation script is closer to a surgical checklist than a one-off script. The dangerous part is not the API call that deletes a key; it is deleting the wrong key quickly across hundreds of repositories. Interviewers ask this to see whether you understand automation blast radius, approval gates, and auditability.

The workflow should have clear phases. Discovery reads repository keys and last-used signals where the provider exposes them. Classification marks keys stale only when they match policy, such as no usage for 90 days, not owned by a protected integration, and not tagged as break-glass. Planning writes a human-readable and machine-readable diff. Approval requires a ticket, reviewer, or signed change request. Execution deletes or disables keys in small batches. Verification confirms the key is gone and that known deployment pipelines still authenticate.

In Python, design the code around pure functions and side-effect boundaries. A classify_key function should be easy to unit test with fake key metadata. A provider client should hide GitHub, GitLab, or Bitbucket API differences. The execution layer should handle 429 rate limits, 5xx retries, and permanent 403 permission errors differently. The script should accept --dry-run by default and require --execute plus a change ID for mutation.

At scale, rollback is not magic. If the API supports disabling instead of deleting, prefer disable first. If deletion is required, capture enough metadata to recreate the key: title, public key, repository, permissions, and owner. Never log private keys because they are not retrievable and should not exist in logs. Use append-only audit storage so security teams can answer who changed what and when.

The non-obvious gotcha is ownership. A key may look stale because usage telemetry is incomplete or because a rarely used disaster recovery pipeline only runs quarterly. Senior engineers define exception handling and communication before the deletion run, not after the outage.

A Python automation repo is like a toolbox shared by many mechanics. If every mechanic has a different wrench size and nobody knows which tool was used on the last repair, the organization eventually creates inconsistent and risky operations. Packaging is what turns a script into a repeatable operational tool.

The baseline is a project file such as pyproject.toml, a src layout, pinned runtime dependencies, and a lock or constraints strategy. The script should expose a console_scripts entry point so users run inventory-scan instead of python random/path/script.py. Configuration should come from environment variables, config files, or cloud identity, not from edits inside the script. Logging should be structured enough for CI and cron logs.

Testing should match the risk. Pure logic such as filtering stale resources, rendering reports, or classifying errors should have fast unit tests with no cloud access. API boundaries should be wrapped behind clients that can be faked. Integration tests that touch AWS, Kubernetes, or Git providers should require explicit environment variables and run against sandbox accounts or ephemeral namespaces. This prevents every pull request from mutating real infrastructure.

For execution, containers remove host drift. A slim Python image with the package installed, non-root user, and pinned dependency set gives CI, cron, and Kubernetes CronJobs the same runtime. The container should not bake in secrets. Credentials should come from workload identity, IAM role, Kubernetes ServiceAccount, or secret mounts controlled by the platform.

The gotcha is dependency drift. A script can work for months and then break because a transitive dependency released a new major version or a base image moved. Senior candidates mention reproducible builds, dependency scanning, scheduled upgrade PRs, and recording tool version in every audit log.

Imagine a shared office building where every desk is supposed to have a nameplate showing which team owns it, but facilities can only fit a limited number of desks on one clipboard page — if the person doing rounds only checks the first page and calls it done, half the building's desks never get inspected, and desks without nameplates on the missed pages stay invisible forever even though they're still using power, space, and the janitorial budget.

This script exists because cloud costs quietly become unowned costs — an EC2 instance launched without a required tag, like owner or cost-center, can run for months, accumulating real spend, with nobody able to answer "who launched this and can we turn it off?" during a cost review. FinOps and platform teams write governance scripts like this specifically because manual tag audits don't scale past a handful of instances, and because tagging policy is only enforceable if something actually checks compliance on a recurring basis rather than trusting that engineers remembered the rule at launch time.

The script calls ec2.describe_instances with a filter for running state, but critically uses get_paginator('describe_instances') rather than a single call, because the EC2 API returns results in pages, using a NextToken, and any account with more than roughly 1,000 instances in scope will silently truncate results without pagination. For each instance, it builds a Python set from the Tags list, where each tag is a Key/Value dict, and checks whether the required key is a member of that set — using a set rather than scanning the list repeatedly keeps the lookup efficient even across accounts with thousands of instances and dozens of tags each.

In production, this script is typically scheduled as a daily Lambda function or CI job, with findings posted to Slack or written to a compliance dashboard rather than just printed to a console. What to monitor: the trend of untagged-instance count over time, since a sudden spike usually means a new automation or a new team started launching instances without going through the standard provisioning path that applies tags automatically, and false positives from instances tagged through mechanisms this script doesn't check, like tags applied only at the Auto Scaling Group level that don't propagate to instances unless PropagateAtLaunch is explicitly enabled.

The non-obvious gotcha is that describe_instances only reflects tags as of the API call, and tags applied moments after instance launch, common with automation that launches first and then tags in a follow-up API call, can create a race condition where a perfectly compliant, soon-to-be-tagged instance gets flagged as a false positive if the governance script runs in that narrow window. Mature versions of this script add a grace period, skipping instances launched less than 15 to 30 minutes ago, to avoid paging someone about a tagging violation that's actually just eventual consistency in the provisioning pipeline.

Think of a TLS certificate like a passport with an expiration date stamped inside — a border agent doesn't need to call the passport office to know it's expired, they just read the date printed on the document itself. This script does the equivalent of a border agent's ID check: connect, ask to see the certificate, read the expiration date directly, and flag anyone whose certificate is about to run out before anyone gets stuck at the border.

This script exists because certificate expiration is one of the most preventable categories of production outage, and yet it happens constantly — a certificate renewed manually eighteen months ago quietly expires at 2 AM, taking down an entire service's HTTPS endpoint, and the team finds out from a customer complaint instead of a dashboard. Python's standard library ssl module was chosen deliberately over an external tool because it requires zero extra dependencies, which matters for lightweight monitoring scripts that need to run reliably in constrained environments like a cron job on a bastion host or a minimal container image.

The script opens a raw TCP socket to each domain on port 443, wraps it with an SSL context via wrap_socket, which performs the actual TLS handshake including certificate presentation from the server, and calls getpeercert() to retrieve the certificate's fields as a Python dictionary. Critically, this only works with certificate verification enabled, the default in create_default_context(), because getpeercert() returns an empty dict if verification is disabled, a subtle trap engineers hit when they've disabled verification elsewhere in their codebase and copy-paste that pattern here. The notAfter field comes back as a specific string format that must be parsed with datetime.strptime using the exact matching format string, and the days-remaining calculation is a simple datetime subtraction.

In production, this check is typically wired into a Prometheus blackbox exporter or run as a scheduled job feeding a dashboard, with graduated alert thresholds — a 30-day warning gives teams time to file a renewal ticket, while a 7-day warning should page on-call directly since it indicates the earlier warning was missed. What to monitor beyond just days-remaining: the certificate's issuer chain, since an internal CA cert expiring is just as dangerous as a public one, and Subject Alternative Names coverage, since a certificate can be not-expired but still wrong for a newly added subdomain that was never included when the cert was issued.

The non-obvious gotcha is that this script only checks the certificate actively being served by the connection it makes — if a service sits behind a load balancer with multiple backend instances serving slightly different or differently-aged certificates due to a partially-rolled-out renewal, a single check from one script run might see the newly renewed cert on one backend and report all clear while a different backend instance is still serving the old, soon-to-expire certificate to some fraction of real traffic. Thorough checks connect multiple times or target specific backend IPs directly rather than trusting DNS round-robin to represent the whole fleet in one shot.

This is like a hospital triage nurse walking through a ward not to check every patient's full chart, but specifically to spot anyone whose vital signs monitor has alarmed repeatedly — a single alarm might be a fluke, but a patient whose monitor has alarmed five times in an hour needs attention immediately, and the nurse doesn't need to read every chart in detail to know which bed to go to first.

This script exists because kubectl get pods shows a restart count column, but scanning that visually across a namespace with hundreds of pods during an incident is slow and error-prone, especially when the unstable pods are scattered rather than clustered. Automating the threshold check turns a manual visual scan into an instant, repeatable triage step — this matters most during incidents, when the fastest path to root cause is often "which of these 200 pods is actually the unstable one," not a general health overview.

The script authenticates via config.load_kube_config(), reading the local kubeconfig, though load_incluster_config() would be used if this ran as a pod itself, then calls list_namespaced_pod to get every pod object in the target namespace. Each pod's status.container_statuses is a list, one entry per container in the pod, that can be None if the pod is still being scheduled or its images are still pulling, which is why the script explicitly guards against that case before iterating. For each container status, restart_count is a cumulative counter that Kubernetes maintains for the life of the pod object, resetting to zero only if the pod itself is deleted and recreated, not on a simple container restart, so a high restart count is a durable signal of instability rather than something that self-clears.

In production, this kind of check is typically run both proactively, as a daily cluster health scan feeding a dashboard, and reactively, with an on-call engineer running it directly during triage to shortcut past kubectl describe on dozens of pods. What matters operationally is combining restart count with the reason for the last termination, terminated.reason such as OOMKilled versus Error, since a high restart count alone doesn't tell you whether the fix is a memory limit increase, a code bug, or a missed dependency — the threshold script is a triage filter, not a root-cause diagnosis by itself.

The non-obvious gotcha is that restart count resets to zero when a pod is deleted and recreated, for example during a rolling deployment or when a node is replaced, so a genuinely unstable workload that's been crash-looping for days can show a deceptively low restart count immediately after its most recent pod recreation, hiding the real instability pattern from a single point-in-time check. Engineers who rely purely on restart count without also checking pod age can be misled into thinking a chronic problem is resolved right after a deployment simply reset the counter, not because anything was actually fixed.

This is like a storage unit facility running a report on units that haven't had anyone visit in 30 days, so facilities management can start the process of reclaiming and re-renting them — but a good facility manager still checks the paperwork before drilling the lock off a unit, because "nobody's visited" doesn't always mean "nobody wants it," especially if it's someone's disaster-recovery storage they intentionally never touch.

This script exists because EBS volumes left behind after an instance is terminated, rather than the volume being explicitly deleted alongside it, silently continue billing at their full provisioned rate indefinitely — this is one of the most common sources of unnoticed cloud waste, since nothing about an orphaned volume is loud or urgent, it just sits there costing money with no owner watching it. DeleteOnTermination defaults to true for the root volume but not for additional attached volumes, which is exactly how so many of these accumulate over time in accounts with frequent instance replacement.

The script filters describe_volumes for status available, which EC2 sets specifically when a volume is not currently attached to any instance, as opposed to in-use or creating. It then compares each volume's CreateTime, a timezone-aware datetime returned directly by boto3, against a cutoff computed with timedelta, using timezone.utc explicitly to avoid the classic naive-vs-aware datetime comparison error that raises a TypeError in Python. Deletion itself is a single delete_volume call gated entirely behind the DRY_RUN flag, meaning the exact same discovery and filtering logic runs whether or not anything is actually deleted, which is the safest structure for this kind of destructive automation.

In production, this should never run as a single unattended step straight to deletion — the standard operating pattern is DRY_RUN=True on a schedule generating a report for review, often posted to a cost-optimization channel, with a human or a secondary approval step confirming the list before flipping to DRY_RUN=False. What to check before deleting: whether the volume has a tag indicating intentional retention, such as a manual snapshot source, disaster-recovery data, or a volume awaiting a support ticket investigation, and whether a recent snapshot exists — teams commonly add a rule that skips deletion for volumes without at least one recent snapshot, treating "no snapshot and no attachment" as the real deletion criterion rather than age alone.

The non-obvious gotcha is that available status only means "not currently attached" — it does not mean "safe to delete." A volume can be intentionally detached and parked, common during a maintenance window, a migration in progress, or as a deliberate disaster-recovery strategy of keeping data attached to nothing until needed, and a purely age-based cleanup script with no additional context will happily delete something a team is actively relying on, with no way to recover it once gone since EBS volume deletion is permanent unless a snapshot exists. Mature versions of this script cross-reference tags, recent snapshot existence, and even a pending-deletion grace-period tag applied a week before actual deletion, giving owners a window to object.

Think of an S3 bucket like a storage room that can be unlocked two completely different ways — a physical spare key hidden under the mat, the ACL, and a separate digital keypad code shared in a group chat, the bucket policy. A security guard who only checks whether the key is under the mat, but never asks whether anyone shared the keypad code, will confidently report the room secure while half the building can still walk in through the door they didn't check.

This script exists because S3 public exposure remains one of the most common and highest-impact cloud misconfigurations, precisely because there are two independent ways a bucket becomes public — legacy Access Control Lists, a per-bucket or per-object permission grant system that predates IAM policies, and modern bucket policies, JSON documents defining who can perform which actions — and a security review that checks only the newer mechanism can completely miss an old ACL grant left over from years ago, or vice versa. AWS's own Block Public Access settings can mitigate both, but this script exists specifically to detect exposure directly rather than assume account-level settings are correctly configured everywhere.

For each bucket, the script first checks get_bucket_acl, iterating the Grants list and checking whether any grantee's URI matches the well-known AWS group URIs for AllUsers, meaning anyone on the internet, authenticated or not, or AuthenticatedUsers, meaning any AWS account holder, not just this account's users, which is a much larger and less obvious exposure than the name implies. Separately, it fetches get_bucket_policy, parses the JSON policy document, and inspects each statement for Effect Allow combined with a Principal of "*" or an AWS wildcard, which grants access to literally anyone. The script wraps both checks in try/except ClientError, since a bucket with no policy at all raises NoSuchBucketPolicy, which is an entirely normal, expected condition, not an error to alert on.

In production, this is typically deployed as a scheduled Lambda function feeding AWS Security Hub or a dedicated security Slack channel, and should run frequently, hourly or on every bucket policy change event via CloudTrail and EventBridge, not just daily, because public exposure is a "seconds matter" class of risk, not a "check it eventually" class. What to monitor beyond the binary public/not-public result: exceptions raised while checking ACLs or policies, since an AccessDenied error while scanning could mean the script's own IAM role lost permissions, which would silently blind the entire security check without anyone noticing unless that failure mode is itself alerted on.

The non-obvious gotcha, and the one that actually broke the original version of this script, is a Python syntax error: an if statement missing its trailing colon fails at parse time, meaning the entire script would refuse to even start — a reminder that in security tooling specifically, a script that silently fails to run at all is far more dangerous than one that runs and reports zero findings, because a cron job that errors out on import or syntax can go unnoticed for a long time if nothing alerts on the automation's own health, leaving buckets unmonitored while everyone assumes the nightly scan is working.

This is like a building security audit that reviews every employee badge and flags anyone who hasn't swiped into the building in three months — not because they're necessarily a threat, but because an unused badge that still works is unnecessary risk sitting around, and if it's ever stolen or misused, nobody would notice quickly since nobody was using it anyway.

This script exists because IAM roles accumulate over the life of an AWS account far faster than they get cleaned up — a role created for a one-time migration, a proof-of-concept that never shipped, or a service that was decommissioned without anyone remembering to remove its role, all continue to exist as valid, assumable identities indefinitely. Security teams treat unused roles as attack surface because every role is a potential path for privilege escalation or lateral movement if its trust policy or attached permissions are ever exploited, and reducing that surface area is one of the most effective, low-risk security improvements available.

The script paginates through list_roles, necessary because accounts with hundreds of roles would otherwise be truncated, and for each role calls get_role to access the RoleLastUsed field, which AWS populates based on actual AssumeRole activity — critically, this field can be entirely absent for a role that has genuinely never been assumed, which is why the script checks both RoleLastUsed and falls back to CreateDate for roles with no usage history at all. It explicitly skips roles under the /aws-service-role/ path, since these are managed by AWS services themselves and aren't meant to be manually cleaned up. For notification, rather than embedding a Slack webhook URL directly in code or an environment variable, both of which risk leaking a live webhook into logs or version control, it fetches the webhook from AWS Secrets Manager at runtime, keeping the actual credential out of the deployment artifact entirely.

In production, this runs as a scheduled Lambda triggered by EventBridge on a recurring cadence, with its execution role scoped narrowly to list roles, get role, and get-secret-value on the one specific secret it needs — a common mistake is granting broader Secrets Manager access than the single secret required. What to monitor: the trend in flagged-role count over time, since a sudden jump usually means a team's automation started creating throwaway roles that never got attached to a cleanup process, and Slack delivery failures themselves, since a security report that fails to send silently is functionally the same as a security report that was never generated.

The non-obvious gotcha is that RoleLastUsed data is not updated in real time — AWS documents it as being updated within a matter of hours, not instantly — so a role assumed minutes before the scan runs can still show as last-used much earlier than expected, making a strict cutoff of exactly 90 days occasionally flag a role that's actually in active, if infrequent, use. Additionally, usage tracked in one region does not always reflect activity that occurred through another region's endpoint for the same global IAM role, which can make an actively-used role appear falsely idle depending on which region's data the check happens to observe. Mature versions build in a small buffer and cross-check with CloudTrail AssumeRole events for roles near the threshold boundary before treating them as confirmed unused.

This is like a librarian periodically clearing out books from the "browsing table" that haven't been picked up in months, returning them to the shelves, except a few books on that table have a permanent "do not reshelve" sign on them, the protected branches, and the librarian double-checks the due date on every other book individually rather than assuming the whole table is fair game just because it looks cluttered.

This script exists because feature branches accumulate relentlessly in any actively developed repository — branches for abandoned experiments, branches merged via squash-merge which can leave the original branch technically "unmerged" from Git's perspective even though its changes already landed on main, and branches from contributors who left the team all pile up, making it hard to tell which branches represent real, active work versus digital clutter. Automating cleanup matters because manual branch hygiene never happens consistently — nobody wants to spend their afternoon guessing which of 400 branches are safe to delete.

The script authenticates with a GITHUB_TOKEN, paginates through the branches endpoint, again, pagination matters because GitHub caps each page at 100 branches and any repo with real history will exceed that, and for each non-protected branch, follows the branch's commit URL to fetch the actual commit object and read the committer's date — this is necessary because the branches endpoint itself doesn't directly expose a clean "last activity" timestamp without this follow-up call. Branches in the PROTECTED_BRANCHES set are skipped outright regardless of age, and the deletion step itself, a DELETE to the git refs endpoint, only executes when DRY_RUN is explicitly set to False.

In production, this typically runs as a scheduled GitHub Action with the token stored as a repository secret, generating a dry-run report for review before anyone flips it to actually delete branches. What matters operationally: rate limiting, since GitHub's REST API enforces request quotas, and a repository with thousands of branches making one extra API call per branch to fetch commit dates can approach those limits, so production versions often batch or cache commit lookups, and add explicit handling for rate-limit responses rather than letting the whole run fail partway through with no record of what was already checked.

The non-obvious gotcha is that "stale" as measured by commit date doesn't account for branches that are intentionally long-lived without frequent commits — a release branch that's cut once and then only merged from occasionally, or a long-running feature-flagged branch that's deliberately kept in sync but rarely committed to directly, can look identical to an abandoned branch by this metric alone. Teams that rely purely on commit age without also checking whether a branch has an open, active pull request end up deleting branches that were very much still in use, just not recently committed to — a mature version of this script cross-checks for open pull requests referencing the branch before ever considering it a deletion candidate.