AWS Services

Imagine a hotel that, instead of giving long-term employees a single master key that opens every room forever, issues a fresh keycard every shift that only opens the specific rooms that employee needs for that shift and automatically deactivates itself at the end of the shift. If a keycard is lost or stolen, the exposure window is just the remainder of that one shift — nobody has to track down and physically disable a master key that could open every door indefinitely, for years, until someone happens to notice it's missing.

AWS IAM's AssumeRole mechanism, backed by the Security Token Service, was designed to replace exactly the older pattern of generating a long-lived access key and secret key pair, essentially a master key that works until someone manually deletes it, with this shift-based, auto-expiring keycard model. This matters because long-lived access keys are a huge source of real breaches — they get accidentally committed to git repositories, baked into container images, or leaked in logs, and unlike a temporary credential, they don't self-destruct, so a leak from months ago can still be actively exploitable if nobody caught it in the meantime.

Internally, when a service like checkout-worker running on an EC2 instance or in a Lambda function needs to access, say, an S3 bucket or a DynamoDB table, it calls sts:AssumeRole, often transparently via the AWS SDK's default credential provider chain, which for EC2 fetches credentials from the instance metadata service, or for Lambda from environment variables injected by the Lambda execution environment, specifying the target role's ARN. STS validates that the calling identity is permitted to assume that role, checked against the role's trust policy, which explicitly lists who's allowed to assume it, and if allowed, returns a temporary AccessKeyId, SecretAccessKey, and SessionToken bundle valid for a configured duration, default one hour, configurable up to the role's max session duration. All subsequent AWS API calls include that session token alongside the temporary keys, and AWS automatically rejects any request using expired temporary credentials outright.

In production, this pattern is what underlies EC2 instance profiles, Lambda execution roles, and cross-account access. A payments-api service can be granted an instance profile role scoped tightly to exactly the S3 bucket and DynamoDB table it needs, and nothing more, with credentials that rotate automatically every hour without any human or deployment process ever touching a static secret. Teams monitor CloudTrail for AssumeRole events to track exactly which identities are assuming which roles and from where, which is also a strong security signal — an AssumeRole call from an unexpected IP range or an unusual role and service combination is a classic indicator of compromised infrastructure worth investigating immediately.

The gotcha: teams sometimes still create long-lived IAM user access keys just to get something working quickly, like a local script or a CI job, and because those keys never expire on their own, they tend to persist for years, attached to overly broad permissions granted temporarily that nobody ever tightens, turning what was meant to be a quick shortcut into a long-term, high-blast-radius credential far more dangerous than the fleet of properly-scoped, auto-expiring role-based credentials everything else uses. Auditing for and eliminating long-lived access keys entirely, replacing them with role assumption, including for CI/CD via OIDC federation, is a common senior-level security initiative.

Imagine a restaurant host who checks every 30 seconds whether a specific table is occupied or free by glancing at it, a very cheap, quick check. That host can honestly report table 12 is free while completely missing that the kitchen is actually so backed up that any customer seated there right now would wait an hour for food, or that table 12's chair just broke in a way the host's quick glance never caught. The health check and the actual dining experience are checking fundamentally different things, and one being fine says nothing reliable about the other.

AWS's Application Load Balancer health checks are deliberately lightweight and narrow by design — they hit a configured path, like /health, at a configured interval and expect a specific status code, existing purely to detect whether a target is completely down or unresponsive so the ALB can stop routing to a dead instance. They were never designed to guarantee every real request will succeed, because a full simulation of production traffic patterns on every health check interval would itself add load and complexity that defeats the purpose of a cheap, frequent liveness signal in the first place.

The actual 502 usually stems from a layer the health check doesn't touch: TCP and HTTP connection handling between the ALB and the target. ALBs keep connections to backend targets alive and reuse them for multiple requests for efficiency, governed by an idle timeout on both the ALB side, default 60 seconds, configurable, and the backend web server's own keep-alive timeout — a Node.js or nginx backend serving payments-api might have a shorter keep-alive timeout than the ALB expects. If the backend closes a connection the ALB still considers open, and then the ALB tries to reuse it for a new request, the backend responds with a connection reset, and the ALB surfaces that as a 502 to the client, a failure mode entirely invisible to a simple periodic health check hitting a fresh connection each time it runs.

In production, the ALB's access logs, delivered to S3 if enabled, include a target_status_code field and error_reason. A 502 where target_status_code shows -1 means the ALB never received a valid HTTP response from the target at all, a connection-level failure, versus a genuine 502 the application itself returned deliberately. Cross-referencing the timing of these 502s against deploy events is also critical, since a common cause is a rolling deployment where a target gets deregistered and stops accepting new connections, but in-flight requests routed to it right before deregistration get abruptly cut off if the deregistration delay, the connection draining timeout, is set too short for how long real requests to something like checkout-worker actually take to complete.

The non-obvious gotcha: the fix for the keep-alive mismatch is counterintuitive to many engineers — you must set the backend server's keep-alive or idle timeout to be LONGER than the ALB's idle timeout, not shorter, so the backend never closes a connection the ALB still considers usable. Getting this backwards, a very common misconfiguration since it feels natural to want the backend to be the more aggressive timeout, is one of the single most common causes of intermittent, hard-to-reproduce 502s that show up only under real concurrent traffic and never during simple manual health-check-style testing.

Think about a family storage unit rental where you'd want three different kinds of alerts: one for your bill went up unexpectedly, something changed in usage, one for someone unrelated to the family is now able to access the unit, a lock got left open, and one for a bunch of items were moved out overnight, unusual activity. Each of those is a genuinely different kind of problem requiring a different detection method — checking the monthly bill alone would never catch an unlocked door, and checking the lock alone would never catch a slow, creeping cost increase from someone quietly hoarding more and more storage over time.

S3 buckets accumulate exactly these three independent risk categories over time, and AWS provides separate purpose-built tools for each because they're genuinely different problems. Cost Anomaly Detection and budget alerts exist for the usage-and-spend-changed-unexpectedly category, built on AWS's cost and usage data using anomaly detection rather than fixed thresholds, so it catches gradual drift, not just hard limit breaches. CloudTrail data events and S3 Server Access Logs exist for the who-is-actually-touching-this-data category. AWS Config rules plus IAM Access Analyzer exist for the is-this-bucket's-permission-configuration-actually-safe category, continuously evaluating policy and ACL state rather than only checking at creation time and never again.

Internally, a typical detection setup enables S3 Server Access Logging, or CloudTrail data events, which are more detailed but higher-volume and higher-cost, writing to a dedicated logging bucket, feeds that into an analysis pipeline, Athena queries over the access logs or a SIEM ingestion, and separately runs AWS Config's s3-bucket-public-read-prohibited and s3-bucket-public-write-prohibited managed rules continuously, which evaluate bucket policy and ACL state on every change and flag non-compliant buckets within minutes, not at the next scheduled audit weeks later. Cost Anomaly Detection runs its own ML-based baseline over historical spend per service and linked account, firing when actual spend deviates meaningfully from the learned pattern, catching a runaway lifecycle rule, one accidentally configured to keep transitioning already-archived objects repeatedly and incurring repeated retrieval charges, that a fixed dollar-threshold budget alert might miss entirely if the absolute dollar amount involved is still small.

In production, teams route AWS Config non-compliance findings, especially public-bucket rules, to a page-immediately channel since public exposure of something like a payments-api-logs bucket is a live security incident, while cost anomalies typically route to a next-business-day review channel unless the anomaly is extreme, since cost issues are important but rarely require middle-of-the-night response. Access pattern analysis via Athena over access logs is usually a weekly or on-demand investigation tool rather than continuous alerting, because usage volume alone is noisy, but it becomes essential once an anomaly from either of the other two signals points you toward a specific bucket worth investigating further.

The non-obvious gotcha: a bucket can be perfectly private, no public policy, AWS Config green across the board, while still leaking data through an overly permissive IAM role attached to an unrelated service. If user-auth-service's IAM role was granted s3:* on all buckets months ago to unblock a deploy and never tightened afterward, that role can read a completely unrelated, correctly-locked-down bucket, and neither the bucket-policy checks nor a public-access scan would ever catch this, because the exposure lives in the IAM role's permissions, not the bucket's own configuration. A full picture requires reviewing IAM policies granting S3 access fleet-wide, not just auditing bucket policies in isolation.

Think about a bank vault door with three independent layers of protection: a policy requiring two employees to authorize any change to the vault's access list, a process control; a physical mechanism that makes it impossible to set the vault's lock to always-open no matter who tries, even a well-meaning employee under pressure, a hard technical constraint; and a silent alarm that triggers immediately if the vault's configuration is ever found in an unsafe state, even if it happened by some unexpected path, continuous detection. No single layer is trusted alone — the whole design assumes any one layer can fail or be bypassed under real-world conditions.

AWS security groups are stateful virtual firewalls attached to resources like EC2 instances or RDS databases, and because a single misconfigured rule, allowing 0.0.0.0/0, meaning any IP address on the internet, on a database port like 5432, can expose sensitive data with essentially no additional steps required, mature platform teams don't rely on any single safeguard. Instead they deliberately layer preventive controls, making the dangerous action impossible or requiring review, with detective controls, catching it fast if it happens anyway, because manual review processes do fail under time pressure, and purely reactive monitoring alone means some window of actual exposure exists before anyone notices it.

The concrete layers: security group and VPC configuration lives entirely in Terraform or CloudFormation, never manually edited in the console for production environments, since console changes are themselves flagged by CloudTrail and Config as drift from the IaC-managed state, with pull request review required before any change merges. At the AWS Organization level, a Service Control Policy attached to the production OU can explicitly deny ec2:AuthorizeSecurityGroupIngress calls where the request includes 0.0.0.0/0 as a CIDR source combined with specific sensitive ports, making that specific dangerous action fail with an access-denied error regardless of the IAM permissions of whoever's account is making the call — this is a hard organizational boundary, not a convention anyone has to remember to follow correctly. Separately, AWS Config continuously evaluates every security group's actual current rules against custom rules checking for open sensitive ports, and can be wired to an automatic remediation Lambda that reverts an unsafe rule within minutes of it appearing, whatever the source of the change.

In production, teams monitor AWS Config compliance dashboards specifically for security-group-related rules across every account in the Organization, not just production, since a lower-environment misconfiguration can be a stepping stone for lateral movement, and treat any SCP-denied API call attempt as a signal worth investigating even though it was successfully blocked. A legitimate engineer hitting that guardrail unexpectedly usually means their actual intent, opening access for a specific partner IP range, say, needs a properly scoped rule instead, while a non-legitimate attempt is itself a security signal worth escalating right away.

The non-obvious gotcha: SCPs only govern IAM API calls within accounts that are members of the AWS Organization and only apply going forward from when they're attached — they don't retroactively fix or even flag security groups that were already misconfigured before the SCP existed. Rolling out this kind of guardrail needs to be paired with an immediate one-time sweep, using AWS Config's ability to evaluate existing resources against a new rule, to find and fix any pre-existing exposed database security groups, or the SCP creates a false sense that we're now protected while an old, dangerous rule from before the policy was silently grandfathered in and still live in production.