AWS

Think of a hospital system with several buildings on the same campus. If one building loses power, patients should not lose all care because the pharmacy, oxygen supply, and emergency desk were only in that building. AWS Availability Zones are similar failure compartments inside a Region. Multi-AZ design is not just placing servers in two zones; every dependency the service needs must survive the loss of one zone.

AWS reliability guidance emphasizes strong foundations, resilient architecture, change management, and failure recovery. For a web service, this usually means an Application Load Balancer spanning subnets in multiple AZs, Auto Scaling groups balanced across those subnets, independent NAT gateways per AZ for private egress, and data stores configured for Multi-AZ durability or replication. Stateless services are easiest because replacement capacity can start elsewhere. Stateful services need replication, failover semantics, and recovery objectives.

The internal request path matters. A client resolves DNS, reaches a regional load balancer, the load balancer chooses healthy targets, app instances call databases, caches, queues, and external services, and responses return through the same dependency chain. If the app instances are balanced but all private traffic exits through one NAT gateway in the impaired AZ, healthy instances in other zones can still fail. If a cache cluster has all primary shards in one zone, the app may be technically running but unusable.

In production, engineers watch per-AZ target health, load balancer zonal metrics, database failover events, queue age, retry rates, and dependency saturation. They also run game days that intentionally remove an AZ from service, disable a subnet route, or shift traffic away from a zone. Good systems use timeouts, retries with jitter, idempotent writes, queue buffering, and autoscaling headroom so the surviving zones can absorb load without cascading.

The non-obvious gotcha is that multi-AZ can still fail as one unit when deployments, credentials, or shared control-plane assumptions are regional. A bad launch template, expired secret, or globally shared config can break all zones at once. Senior engineers separate failure domains, but they also test rollback, configuration isolation, and dependency degradation. Multi-AZ is a reliability foundation, not a substitute for disciplined operations.

Designing for startup scale is fundamentally about avoiding premature optimization while building in the right inflection points. Think of it like building a restaurant: you start with a food truck (serverless), move to a small storefront (single-region containers), then franchise (multi-region). At each stage, you only pay for what you need.

Phase 1 (0-10K users): Deploy everything serverless. API Gateway handles HTTP routing, Lambda runs your business logic, and DynamoDB provides single-digit millisecond reads. Your monthly bill stays under $50. The key insight is that Lambda's per-request pricing means zero cost at zero traffic. Use S3 for static assets behind CloudFront. Deploy with SAM or CDK for reproducibility. At this stage, a single DynamoDB table with a well-designed partition key (e.g., USER#12345) handles all access patterns through single-table design.

Phase 2 (10K-100K users): Lambda concurrency limits and cold starts begin hurting user experience. Migrate compute-heavy paths to ECS Fargate behind an Application Load Balancer. Keep event-driven workloads (image processing, notifications) on Lambda. Introduce ElastiCache Redis for session management and hot-path caching, reducing DynamoDB read costs by 60-70%. Add an SQS queue between your API tier and async processors to absorb traffic spikes. Your bill is now $500-2000/month, but you have headroom.

Phase 3 (100K-1M users): Migrate from DynamoDB to Aurora PostgreSQL with read replicas for complex query patterns. DynamoDB excels at key-value lookups but becomes expensive for analytics queries. Aurora Serverless v2 scales capacity automatically between 0.5 and 128 ACUs. Implement write-through caching: writes go to Aurora, which triggers a Lambda via DynamoDB Streams-style change capture to update ElastiCache. Deploy a NAT Gateway in each AZ (not one shared, which creates a single point of failure and cross-AZ data transfer charges).

Phase 4 (1M-10M users): Move to multi-region active-active with Aurora Global Database (under 1 second replication lag). Use Route 53 latency-based routing to direct users to the nearest region. Adopt CloudFront with Lambda@Edge for personalization at the edge. Purchase Compute Savings Plans (not EC2 Instance Savings Plans) because they cover Lambda, Fargate, and EC2, giving you flexibility as your compute mix evolves. Implement S3 Intelligent-Tiering for your data lake since access patterns are unpredictable at this scale. Use AWS Cost Explorer daily with anomaly detection alerts via SNS.

Production gotchas: Always set billing alarms at each phase boundary. NAT Gateway data processing charges are the number one surprise cost, often exceeding the compute bill. Use VPC endpoints for S3 and DynamoDB to eliminate NAT Gateway charges for AWS service traffic. Tag every resource with cost-allocation tags (team, environment, service) from day one. You cannot retroactively tag resources for cost attribution.

Zero-trust networking replaces the traditional castle-and-moat model with a 'never trust, always verify' approach. Think of the old model like an office building: once you badge in at the front door, you can walk into any room. Zero-trust is like a hospital where every room has its own lock, every person wears an ID badge that's checked at every door, and the security system logs every entry.

The foundation starts with identity. Every workload, service, and user must have a cryptographically verifiable identity. In AWS, this means IAM roles for compute resources (never long-lived access keys), OIDC federation for external services, and IAM Identity Center (successor to AWS SSO) for human access. Critical production accounts should enforce hardware MFA tokens, not virtual MFA.

Network microsegmentation is the next layer. Traditional VPC security uses NACLs and security groups at the subnet and instance level. Zero-trust goes further: security groups should reference other security groups instead of CIDR blocks. For example, your payments-api security group allows inbound on port 8443 only from the checkout-service security group. This means if an attacker compromises a machine in the same subnet, they cannot reach the payments API because their security group is not authorized. VPC Lattice, launched as a service mesh alternative, handles service-to-service authentication and authorization at the network layer without sidecars.

Encryption in transit is non-negotiable. Deploy ACM Private CA to issue short-lived certificates (24-hour validity) to every service. Mutual TLS (mTLS) ensures both client and server verify each other's identity. AWS App Mesh or Envoy sidecars terminate and originate mTLS connections, so application code does not need to handle certificate management. For east-west traffic between VPCs, use PrivateLink endpoints instead of VPC peering because PrivateLink restricts access to specific services rather than exposing entire VPC CIDR ranges.

AWS Verified Access replaces traditional VPNs for application access. Instead of granting network-level access to a VPC (where a compromised laptop can scan the entire network), Verified Access evaluates trust policies based on user identity (from IAM Identity Center or third-party IdP), device posture (from CrowdStrike, Jamf, or AWS device trust), and application-specific conditions. Each application gets its own Verified Access endpoint, so compromising access to one application does not grant access to others.

Continuous monitoring completes the zero-trust model. VPC Flow Logs feed into a SIEM (Security Lake or third-party) for anomaly detection. GuardDuty monitors for credential exfiltration, cryptocurrency mining, and unusual API patterns. CloudTrail logs every API call across all accounts. AWS Config rules verify that security groups do not have 0.0.0.0/0 ingress rules and that all ALBs enforce HTTPS.

Production gotchas: mTLS certificate rotation must be automated; manual rotation causes outages. VPC Lattice has a 500 services-per-service-network limit that bites at scale. Security group rules referencing other security groups do not work across VPC peering connections in different regions; use PrivateLink instead. Verified Access adds 5-15ms latency per request, which matters for latency-sensitive APIs. Always deploy in 'monitor only' mode first to identify legitimate traffic patterns before enforcing deny rules.

Cross-account access is one of the most misunderstood topics in AWS because there are two fundamentally different mechanisms, and the permissions evaluation logic differs between them. Think of it like two buildings: with role assumption, you walk into building B, get a visitor badge (temporary credentials), and operate under building B's rules. With resource-based policies, building B sends a package to you in building A; you never enter building B, so your building A badge is what matters.

Mechanism 1: IAM Role Assumption (sts:AssumeRole). Account B (the trusting account, e.g., 987654321098) creates an IAM role with a trust policy that specifies which principals in account A (the trusted account, e.g., 123456789012) can assume it. Account A's principal must also have an IAM policy granting sts:AssumeRole on the role ARN in account B. Both policies must align; either one missing means access denied. When the principal in account A calls sts:AssumeRole, it receives temporary credentials (access key, secret key, session token) that operate with the permissions of the role in account B. Critically, the original permissions from account A are dropped entirely. The principal now operates as if it lives in account B, subject to account B's SCPs, permission boundaries, and session policies.

Mechanism 2: Resource-Based Policies. Some AWS resources (S3 buckets, SQS queues, SNS topics, Lambda functions, KMS keys) support resource-based policies that directly grant cross-account access. When account B's S3 bucket policy grants s3:GetObject to arn:aws:iam::123456789012:role/prod-analytics-etl-role, the principal in account A can access the bucket directly without assuming a role. The critical difference: the principal retains its own identity and permissions. Both the identity-based policy in account A and the resource-based policy in account B are evaluated, and if either grants access (in the absence of an explicit deny), the request succeeds. This is called the union of permissions for cross-account resource-based policies.

SCP interactions create the most confusion. When using role assumption, SCPs in account B apply to the assumed role session. If account B's SCP denies s3:DeleteObject, the assumed role cannot delete objects even if the role's IAM policy allows it. SCPs in account A do not apply because the principal is now operating in account B's context. For resource-based policies, SCPs in both accounts apply: account A's SCPs affect the principal making the request, and account B's SCPs affect the resource being accessed.

Permission boundaries interact differently too. If principal A has a permission boundary that does not include s3:GetObject, it cannot access account B's S3 bucket via resource-based policy (because its effective permissions are the intersection of IAM policy and permission boundary). However, if principal A assumes a role in account B, the permission boundary on principal A is irrelevant; only the permission boundary on the role in account B matters.

The confused deputy problem arises when a service in account B assumes a role on behalf of an external entity. Always use the ExternalId condition in trust policies for third-party access. The ExternalId acts like a shared secret that prevents an attacker from tricking your role into granting them access through a different AWS account they control.

Production gotchas: S3 bucket policies have a 20KB size limit, which you hit quickly when granting access to many individual accounts; use aws:PrincipalOrgID instead to grant access to all accounts in your AWS Organization. KMS key policies are unique: the key policy must explicitly grant access even for same-account principals (unlike S3 where the IAM policy alone is sufficient). When using cross-account role assumption in CI/CD pipelines, implement session tags to pass metadata (environment, pipeline-id, commit-sha) that can be used in condition keys for fine-grained access control.

A well-designed data pipeline on AWS follows the medallion architecture pattern (bronze/silver/gold zones), with S3 as the storage layer, Glue as the ETL engine and metadata catalog, and Athena as the serverless query engine. Think of it like a water treatment plant: raw water (data) enters the intake (landing zone), goes through filtration stages (ETL transformations), and emerges as clean drinking water (analytics-ready tables).

The S3 layer is organized into three zones. The landing zone (s3://prod-analytics-data-lake-123456789012/landing/) receives raw data in its original format: CSV exports from RDS, JSON events from Kinesis Firehose, or Parquet files from third-party vendors. This zone has a lifecycle policy: move to Glacier Deep Archive after 90 days, delete after 365 days. The processed zone (s3://prod-analytics-data-lake-123456789012/processed/) contains transformed data in Apache Parquet format with Snappy compression, partitioned by date. Parquet is critical because Athena charges per byte scanned, and Parquet's columnar format means querying 5 columns of a 200-column table scans only 2.5% of the data. The curated zone (s3://prod-analytics-data-lake-123456789012/curated/) contains business-specific aggregations ready for dashboarding: daily revenue summaries, user cohort metrics, funnel conversion rates.

Glue Crawlers automatically infer schemas from files in S3 and register them in the Glue Data Catalog. However, in production you should avoid running crawlers on every file change. Instead, use the Glue API to manually register schemas (create_table) because crawlers can infer incorrect data types (e.g., treating ZIP codes as integers, stripping leading zeros). Crawlers are useful for initial discovery but should be replaced with explicit schema definitions for production tables.

Glue ETL jobs run Apache Spark under the hood but with AWS-managed infrastructure. For small datasets (under 1GB), use Python shell jobs (0.0625 DPU, costs roughly $0.44/hour) instead of Spark jobs (minimum 2 DPUs at $0.44/DPU/hour). For larger datasets, use Glue 4.0 with auto-scaling enabled, which adjusts DPU count based on actual workload. A critical optimization: enable job bookmarks, which track which data has already been processed. Without bookmarks, re-running a job reprocesses all historical data, inflating both cost and execution time.

Partitioning strategy determines Athena query performance. Partition by the most common query filter. For event data, partition by year/month/day (s3://bucket/events/year=2024/month=01/day=15/). For multi-tenant data, partition by tenant_id first, then date. Use Hive-style partitions (key=value in the path) so Athena automatically recognizes them. Each partition creates a separate metadata entry in the Glue Data Catalog, so avoid over-partitioning: 100,000+ partitions causes slow query planning. If you need hourly granularity, use Apache Iceberg hidden partitioning instead, which partitions data without polluting the S3 path structure.

Athena pricing is $5 per TB scanned, so query optimization is directly financial optimization. Always use columnar formats (Parquet or ORC), compress with Snappy or ZSTD, partition by the most filtered column, and use LIMIT clauses during development. Create Athena workgroups to enforce per-team query limits: the analytics team gets a 10GB per-query scan limit to prevent runaway queries that scan the entire data lake. Enable Athena query result reuse for repeated queries to avoid re-scanning data.

Orchestration ties everything together. Use Glue Workflows for simple linear pipelines (crawl, transform, validate) or AWS Step Functions for complex DAGs with conditional branching, error handling, and human approval steps. EventBridge rules trigger the pipeline when new files land in the S3 landing zone via s3:ObjectCreated events.

Production gotchas: Glue crawlers do not detect schema evolution (added/removed columns) well; use schema registry with explicit versioning. Athena has a 30-query concurrency limit per account per region (can be increased). S3 strongly consistent reads (since December 2020) eliminated eventual consistency bugs in pipelines, but older documentation still warns about them. Never use CSV for analytical workloads; the format lacks schema enforcement and columnar scan benefits.

Choosing between serverless and containers is not a binary decision; it is a spectrum, and most production systems use both. Think of serverless like a taxi: you pay per trip, there is no vehicle to maintain, but you cannot customize the car and you wait for one to arrive (cold start). Containers are like a company car: fixed monthly cost whether you drive it or not, but it is always ready and you can customize everything under the hood.

Serverless Architecture (Lambda + API Gateway + DynamoDB). API Gateway receives HTTP requests and routes them to Lambda functions. Each Lambda invocation is isolated, runs for up to 15 minutes, and scales to 1000 concurrent executions per region by default (soft limit, requestable to tens of thousands). Lambda charges per 1ms of execution time and per request ($0.20 per million requests, $0.0000166667 per GB-second). For a workload handling 10 million requests/month at 200ms average duration with 512MB memory, the Lambda cost is approximately $18.67/month. This is transformatively cheap for startups.

However, serverless has hard constraints. Cold starts add 100ms-3s depending on runtime (Java/C# are worst, Node.js/Python best, and Rust/Go with custom runtime are near-zero). Provisioned Concurrency eliminates cold starts but adds fixed cost, defeating the pay-per-request model. Lambda has a 10GB memory ceiling, 10GB ephemeral storage, and cannot maintain persistent connections (WebSockets require API Gateway WebSocket API with connection management in DynamoDB). Each Lambda invocation is stateless, so session state must live in ElastiCache or DynamoDB. Lambda functions behind API Gateway have a 29-second timeout (API Gateway limit), even though Lambda itself allows 15 minutes.

Container Architecture (ECS Fargate or EKS behind ALB). An Application Load Balancer distributes requests to Fargate tasks running your Docker containers. Containers maintain state in memory across requests, support WebSocket connections natively, and have no cold start once running. ECS Fargate charges per vCPU-hour ($0.04048) and per GB-hour ($0.004445). A service running 2 tasks with 1 vCPU and 2GB RAM 24/7 costs approximately $96.15/month. At low traffic, this is more expensive than serverless. At high traffic (millions of requests/day), the unit economics flip because you are paying for capacity, not per-request.

EKS adds Kubernetes orchestration: service mesh (Istio/Linkerd), horizontal pod autoscaling based on custom metrics, rolling deployments with canary analysis, and a vast ecosystem of operators. However, EKS adds $73/month for the control plane, plus worker node costs, and requires Kubernetes expertise. Use EKS when you need multi-cloud portability, have a platform team, or run 20+ microservices that benefit from the Kubernetes ecosystem.

Hybrid patterns are the real-world answer. Use Lambda for event-driven glue: processing S3 uploads, responding to DynamoDB Streams, handling SNS notifications, and running scheduled cron jobs. Use containers for your core API that handles synchronous HTTP traffic with predictable latency requirements. Use Lambda@Edge or CloudFront Functions for edge logic (A/B testing, geolocation routing, header manipulation). The API Gateway can route /api/* to an ALB (containers) and /webhooks/* to Lambda in the same application.

State management differs fundamentally. Containers can use in-memory caching (local LRU cache) for hot data, reducing ElastiCache calls. Lambda functions must externalize all state, which adds latency and cost for every cache lookup. For machine learning inference, containers allow loading large models into memory once and serving many requests, while Lambda must load the model on every cold start (or use Provisioned Concurrency with large memory).

Production gotchas: Lambda concurrent execution limits are per-region, not per-function, so one runaway function can starve all other functions in the account. Use reserved concurrency to isolate critical functions. ECS Fargate task startup time is 30-60 seconds for pulling images; use ECR with VPC endpoints and slim base images (Alpine, distroless) to minimize pull time. ALB health checks default to 30-second intervals; reduce to 10 seconds with a 2-unhealthy-threshold to speed up failed task replacement. Never run ECS tasks with the awsvpc network mode in a subnet without enough available IP addresses; each task consumes one ENI (and one IP).

Infrastructure compliance at scale requires a layered defense model: preventive controls stop non-compliant actions before they occur, detective controls identify drift after the fact, and corrective controls automatically remediate violations. Think of it like a highway system: SCPs are the physical guardrails that prevent cars from leaving the road, Config rules are the speed cameras that detect violations, and remediation actions are the automated toll systems that fix issues without human intervention.

Service Control Policies (SCPs) are the most powerful preventive control in AWS because they set the maximum permissions boundary for an entire account. SCPs do not grant permissions; they restrict what IAM policies can grant. This is a critical distinction: if an SCP denies ec2:RunInstances in a region, no IAM policy in that account can override it, not even the root user. SCPs apply to all principals in member accounts (except the management account itself, which is why you should never run workloads in the management account).

SCP design follows a deny-list or allow-list strategy. Deny-list is more common: start with the default FullAWSAccess SCP, then add explicit denies for non-compliant actions. For example, deny launching EC2 instances without IMDSv2 (Instance Metadata Service version 2), deny creating S3 buckets without encryption, or deny using regions outside your approved list. Allow-list is stricter: remove the default FullAWSAccess and explicitly allow only the services and actions needed. Allow-list is recommended for regulated industries (healthcare, finance) but requires significant upfront planning.

SCPs are attached to Organizational Units (OUs), not individual accounts. Structure your OUs to reflect compliance boundaries: a Production OU with the strictest SCPs (no delete operations on databases, mandatory encryption, restricted regions), a Development OU with relaxed SCPs (allow experimentation but deny production resource access), and a Sandbox OU with minimal restrictions but a spending cap via AWS Budgets actions.

AWS Config continuously records resource configurations and evaluates them against rules. Config rules come in two types: AWS-managed rules (160+ pre-built rules like s3-bucket-ssl-requests-only, encrypted-volumes, iam-root-access-key-check) and custom rules (Lambda functions that evaluate arbitrary compliance logic). For custom rules, use Config Rule Development Kit (RDK) to scaffold and test Lambda-based rules locally before deployment.

Conformance packs bundle multiple Config rules into compliance frameworks. AWS provides operational best practices packs for PCI DSS 3.2.1 (35 rules), HIPAA (55 rules), CIS AWS Foundations Benchmark (49 rules), and NIST 800-53 (100+ rules). Deploy conformance packs via AWS Organizations to roll them out across all accounts simultaneously. Each rule evaluates every relevant resource in every account, producing a compliance score per account and per rule.

Remediation actions are the corrective control layer. When a Config rule finds a non-compliant resource, it can trigger an SSM Automation document to fix the issue. For example, if s3-bucket-server-side-encryption-enabled finds an unencrypted bucket, the remediation action runs the AWS-EnableS3BucketEncryption document to apply AES-256 encryption. Use manual remediation (requires human approval via SNS) for production resources and automatic remediation for development accounts.

Config Aggregator collects compliance data from all accounts and regions into a single delegated administrator account. This gives your compliance team a centralized dashboard without granting them access to individual workload accounts. The aggregator supports both AWS Organizations-based collection (automatic for all member accounts) and individual account authorization.

Production gotchas: SCPs have a 5,120 character limit per policy, which you hit quickly with complex deny statements; use condition keys like aws:RequestedRegion with StringNotEquals instead of listing individual region denies. Config rules evaluate asynchronously, so there is a delay between resource creation and compliance evaluation (typically 1-10 minutes). Config recording charges $0.003 per configuration item recorded; in accounts with frequent Auto Scaling events, this adds up. Limit Config recording to the specific resource types you need to evaluate. SCP changes take effect immediately but can take up to 24 hours to propagate to all API endpoints globally; always test SCP changes in a sandbox OU first. Never attach a restrictive SCP to the management account OU; SCPs do not apply to the management account, creating a false sense of security.

AWS S3 offers four distinct encryption-at-rest mechanisms, each serving different security postures and compliance needs. Understanding all four is critical for any DevOps engineer working in regulated environments like PCI-DSS, HIPAA, or SOC2.

First, SSE-S3 (Server-Side Encryption with S3-Managed Keys) is the simplest option. Think of it like a hotel safe where the hotel manages the master key. AWS generates, manages, and rotates a unique AES-256 key for every object. You have zero visibility into the key material itself. This is the default encryption for all new S3 buckets since January 2023. The trade-off is that anyone with s3:GetObject permission can decrypt the data — there is no separate key permission layer.

Second, SSE-KMS (Server-Side Encryption with AWS KMS Keys) adds a permission boundary. Imagine a safety deposit box that requires both the bank's master key and your personal key to open. KMS creates a Customer Master Key (CMK), and every object encryption generates a unique Data Encryption Key (DEK) via the envelope encryption pattern. The CMK encrypts the DEK, and the DEK encrypts the object. This means decryption requires both s3:GetObject AND kms:Decrypt permissions — a powerful separation of duties. Every decryption call is logged in CloudTrail, giving you a full audit trail. The gotcha in production is KMS request throttling: each S3 GET/PUT triggers a KMS API call, and the default quota is 5,500 requests per second per region (10,000 in some regions). Teams running large Spark or EMR jobs against encrypted datasets often hit this limit and see throttling errors. The fix is to request a quota increase or use S3 Bucket Keys, which reduce KMS calls by up to 99% by caching a bucket-level key.

Third, SSE-C (Server-Side Encryption with Customer-Provided Keys) means you supply the encryption key with every PUT and GET request over HTTPS. AWS uses your key, encrypts the object, then discards the key immediately — they never store it. This is like bringing your own padlock to a storage unit. The massive operational burden is that if you lose the key, your data is gone forever. AWS cannot recover it. Every API call must include the key in the request header, which means your application layer must have a robust key management system. This is rarely used in practice but is required by some government contracts where key material must never persist on a third-party system.

Fourth, client-side encryption means you encrypt data before it ever leaves your network. AWS receives only ciphertext and has zero knowledge of the plaintext or keys. You can use the AWS Encryption SDK, which implements envelope encryption with a local master key or an HSM-backed key. This is the highest security posture but adds latency and complexity. You must manage key rotation, handle encryption errors, and ensure your client library is up to date against vulnerabilities.

In production, most organizations use SSE-KMS as the default for its balance of security, auditability, and manageability. SSE-S3 is acceptable for non-sensitive data. SSE-C and client-side encryption appear in highly regulated or multi-tenant environments where cryptographic key isolation is a compliance requirement.

A Virtual Private Cloud (VPC) is your isolated section of the AWS cloud — think of it as your own private office building within a massive shared complex. You control the floor plan (CIDR block), the rooms (subnets), the hallways between rooms (route tables), and the doors to the outside world (gateways).

CIDR allocation is the foundation. When you create a VPC, you assign a primary CIDR block, such as 10.0.0.0/16, which gives you 65,536 IP addresses. This is like choosing the size of your building lot before construction. The critical production consideration is planning for growth and peering. If your payments VPC uses 10.0.0.0/16 and your analytics VPC also uses 10.0.0.0/16, you cannot peer them because overlapping CIDR blocks create routing ambiguity. Many organizations adopt a structured allocation scheme: 10.0.0.0/16 for production, 10.1.0.0/16 for staging, 10.2.0.0/16 for development, and 172.16.0.0/12 reserved for on-premises connectivity via Direct Connect.

Subnets divide the VPC CIDR into smaller segments, each tied to a single Availability Zone. A /16 VPC might be split into /24 subnets, each with 251 usable IPs (AWS reserves 5 per subnet: network address, VPC router, DNS server, future use, and broadcast). Public subnets host resources that need direct internet access — load balancers, bastion hosts, NAT gateways. Private subnets host application servers, databases, and backend services that should never be directly reachable from the internet. The distinction between public and private is not a property of the subnet itself but entirely determined by the route table associated with it.

Route tables are the traffic control system. Every subnet must be associated with exactly one route table. A public subnet's route table has a route like 0.0.0.0/0 pointing to an Internet Gateway (IGW), which allows bidirectional internet traffic. A private subnet's route table has 0.0.0.0/0 pointing to a NAT Gateway, which allows outbound-only internet access. The local route (10.0.0.0/16 → local) is automatically present and enables intra-VPC communication. You can add more specific routes for VPC peering, Transit Gateway, VPN connections, or VPC endpoints.

NAT Gateways sit in a public subnet and enable private subnet resources to initiate outbound connections (pulling Docker images, downloading OS patches, calling external APIs) without exposing them to inbound internet traffic. Think of it as a one-way mirror: your servers can see out, but the internet cannot see in. NAT Gateways are managed, highly available within an AZ, and charged per hour plus per GB processed. A common production mistake is deploying only one NAT Gateway for the entire VPC — if that AZ goes down, all private subnets lose internet access. Best practice is one NAT Gateway per AZ with AZ-specific route tables.

The Internet Gateway is a horizontally scaled, redundant VPC component that performs network address translation for instances with public IPs. Unlike NAT Gateways, IGWs are free and have no bandwidth limits. They are attached at the VPC level but only affect subnets whose route tables direct 0.0.0.0/0 traffic to the IGW.

In production multi-account architectures, VPCs are connected via Transit Gateway, which acts as a central hub. This avoids the O(n^2) complexity of full-mesh VPC peering. Security Groups (stateful, instance-level) and Network ACLs (stateless, subnet-level) add defense-in-depth. VPC Flow Logs capture metadata about all traffic for security analysis and troubleshooting.

AWS Identity and Access Management (IAM) is the foundation of every security decision in AWS. Think of it as the security system for an entire corporate campus — it manages who people are (authentication), what badge they carry (identity), what doors they can open (authorization), and logs every door they walk through (auditing).

IAM has four core building blocks. Users are individual identities with long-term credentials (access keys and passwords). Groups are collections of users that share the same permissions — like giving all developers the same badge level. Roles are temporary identities that anyone or anything can assume — humans, EC2 instances, Lambda functions, even users from other AWS accounts. Roles issue short-lived credentials via AWS STS (Security Token Service), making them far more secure than long-term user access keys. Policies are JSON documents that define permissions — they are the actual rules that say allow or deny specific actions on specific resources under specific conditions.

Policies come in several types. Identity-based policies are attached to users, groups, or roles and define what that identity can do. Resource-based policies are attached to resources (S3 buckets, SQS queues, KMS keys) and define who can access that resource. Permissions boundaries set the maximum permissions an identity can have — they do not grant access themselves but cap what identity-based policies can grant. Think of it as a fence around what an identity is allowed to request. Organizations Service Control Policies (SCPs) set the maximum permissions for an entire AWS account — they are guardrails applied at the organizational level.

The policy evaluation logic is where most engineers get confused, and getting it wrong leads to security vulnerabilities. AWS evaluates policies in a specific order with a strict hierarchy. The algorithm works as follows:

Step 1: All policies are gathered — identity-based, resource-based, SCPs, permissions boundaries, and session policies. Step 2: AWS checks for any explicit Deny across ALL policies. If any single policy statement has Effect: Deny that matches the request, access is denied immediately — no exceptions. This is the most important rule and cannot be overridden. Step 3: AWS checks Organizations SCPs. If the action is not allowed by SCPs, the request is denied even if an identity policy allows it. Step 4: For resource-based policies, if the resource policy explicitly grants access AND is in the same account, access can be granted even without an identity-based policy allowing it. This is a critical nuance — resource-based policies can grant cross-principal access within an account. Step 5: AWS checks identity-based policies. The action must be explicitly allowed here. Step 6: If a permissions boundary exists, the action must ALSO be allowed by the boundary. The effective permission is the intersection of the identity policy and the boundary. Step 7: If session policies exist (used with assumed roles or federated sessions), the action must also be allowed by the session policy.

The default state is implicit deny — if no policy explicitly allows an action, it is denied. This is the principle of least privilege enforced at the platform level.

A common production gotcha is the confused deputy problem in cross-account access. If your account lets another account assume a role, you must use the ExternalId condition to prevent a malicious third party from tricking the trusted account into accessing your resources. Another gotcha is policy size limits: managed policies max at 6,144 characters, and inline policies max at 2,048 characters for users and 10,240 for roles. Complex environments often hit these limits and must refactor into multiple policies.

In production, best practice is to use roles exclusively (never long-term access keys), enforce MFA for console access, use permissions boundaries for delegated administration, and audit with IAM Access Analyzer to find unintended external access.

Understanding the difference between EKS and self-managed Kubernetes requires knowing what the Kubernetes control plane actually does and why managing it yourself is an operational burden that most teams underestimate.

Kubernetes has two layers: the control plane and the data plane. The control plane is the brain — it consists of the API server (receives all requests), etcd (the distributed key-value store holding all cluster state), the scheduler (decides which node runs each pod), and the controller manager (ensures desired state matches actual state). The data plane is the muscle — the worker nodes running kubelet, kube-proxy, and your actual application containers.

With self-managed Kubernetes, you own everything. You must install kubeadm or use a tool like kops to bootstrap the cluster. You manage etcd yourself — this means configuring it as a 3 or 5 node cluster for quorum, managing backups, handling disk pressure, and performing version upgrades that require careful rolling restarts. etcd is the single most critical component: if etcd loses data, your cluster state is gone. You must also handle API server high availability by running multiple replicas behind a load balancer, manage TLS certificates for all component-to-component communication (API server to kubelet, kubelet to API server, etcd peer communication), and handle Kubernetes version upgrades which require upgrading the control plane first, then worker nodes, with careful attention to version skew policies.

With EKS, AWS manages the entire control plane. The API server runs across at least three Availability Zones with automatic failover. etcd is managed, backed up, and encrypted by AWS. You never SSH into a control plane node — you cannot even see them. AWS handles Kubernetes version upgrades for the control plane (you trigger the upgrade, AWS performs the rolling update). The API server endpoint can be configured as public (accessible from the internet), private (accessible only within the VPC), or both. EKS also integrates natively with AWS services: IAM Roles for Service Accounts (IRSA) maps Kubernetes service accounts to IAM roles, the AWS Load Balancer Controller provisions ALBs and NLBs from Kubernetes Ingress resources, EBS CSI driver manages persistent volumes, and CloudWatch Container Insights provides monitoring.

However, EKS is not fully managed — you still manage worker nodes. You have three options: self-managed node groups (full EC2 control, you handle AMI updates and scaling), managed node groups (AWS handles node provisioning and lifecycle, you choose instance types), and Fargate profiles (fully serverless, AWS manages the underlying compute — no nodes to manage at all). Each option trades control for operational simplicity.

The cost comparison is nuanced. EKS charges $0.10 per hour ($73/month) for the control plane. Self-managed Kubernetes has no such fee, but you pay for the EC2 instances running control plane components, plus the engineering time to maintain them. For most organizations, the control plane management fee is trivial compared to the operational cost of maintaining etcd, handling certificate rotation, and performing safe upgrades. A single botched etcd backup or failed upgrade can cost days of downtime.

Production gotchas with EKS include: the API server has rate limits (you can hit throttling with large clusters), Kubernetes version support has a lifecycle (EKS supports four minor versions, and you must upgrade before end-of-support or AWS will auto-upgrade), and the VPC CNI plugin uses one ENI per pod by default, which limits pod density based on instance type ENI limits (a t3.medium supports only 17 pods). Self-managed Kubernetes avoids the CNI pod density issue since you can choose Calico or Cilium with overlay networking, but introduces its own networking complexity.

The decision framework: choose EKS if you want to focus on application delivery rather than infrastructure. Choose self-managed if you need full control over the control plane (rare), have a dedicated platform engineering team, or have compliance requirements that demand you manage every component.

AWS Lambda cold starts are one of the most misunderstood performance characteristics in serverless computing. Think of it like starting a car engine versus driving an already-running car. A warm invocation (engine running) just executes your handler function. A cold start (cold engine) requires Lambda to perform several heavy steps before your code can run.

The cold start sequence has distinct phases. Phase 1 is environment provisioning: Lambda allocates a microVM using Firecracker (AWS's lightweight virtualization technology), provisions CPU and memory based on your configuration, and sets up the network stack. This takes approximately 100-200ms and is outside your control. Phase 2 is code download: Lambda downloads your deployment package from S3 (or pulls your container image from ECR). A 50MB zip takes meaningfully longer than a 5MB zip. Phase 3 is runtime initialization: the language runtime starts — the Python interpreter, the Node.js V8 engine, or the JVM. The JVM is notoriously slow here, often adding 2-5 seconds. Phase 4 is your initialization code: anything outside your handler function runs — importing libraries, establishing database connections, loading ML models, reading configuration from Parameter Store. This is where most of the controllable latency lives.

After the cold start, the execution environment stays warm for 5-15 minutes (AWS does not guarantee a specific duration). Subsequent invocations reuse the warm environment and skip directly to executing the handler. This is why global variables persist between invocations — a database connection pool initialized in the cold start is reused across warm invocations.

VPC-attached Lambda functions historically had severe cold starts (10+ seconds) because Lambda had to create an Elastic Network Interface. Since 2019, AWS uses Hyperplane (the internal network function virtualization platform), which pre-creates shared ENIs. This reduced VPC cold starts to roughly the same as non-VPC cold starts, but there is still additional latency for initial security group and subnet configuration.

Optimization strategies, ordered by impact

1. Provisioned Concurrency: AWS pre-initializes a specified number of execution environments that are always warm. You pay for them whether they are used or not. This completely eliminates cold starts for that capacity. It is essential for latency-sensitive workloads like payment processing APIs where a 2-second cold start is unacceptable.

2. SnapStart (Java only): AWS takes a Firecracker microVM snapshot after your initialization code runs, then restores from this snapshot instead of re-initializing. This reduces Java cold starts from 5+ seconds to under 200ms. It has limitations — you must handle uniqueness restoration since random seeds and in-memory state are shared across restored snapshots.

3. Minimize package size: Remove unnecessary dependencies. Use Lambda Layers for shared libraries. For Node.js, use esbuild or webpack to tree-shake. For Python, strip unused libraries and avoid including test files. A 5MB package cold-starts measurably faster than a 50MB package.

4. Lazy initialization: Do not load everything at init time. If only 10% of invocations need the ML model, load it on first use rather than on every cold start. Cache it in a global variable for subsequent warm invocations.

5. Choose efficient runtimes: Python and Node.js cold-start in 100-300ms. Java cold-starts in 2-5 seconds (without SnapStart). Go and Rust (via custom runtime) cold-start in under 50ms because they compile to native binaries with no runtime overhead.

6. Increase memory allocation: Lambda allocates CPU proportionally to memory. A 128MB function gets a fraction of a vCPU. A 1769MB function gets a full vCPU. Higher CPU speeds up initialization code that is CPU-bound (parsing, compilation). Sometimes doubling memory halves cold start time and the cost stays the same because the function runs half as long.

Production monitoring: Use CloudWatch Metrics with the Init Duration metric to track cold starts. Set alarms for cold start percentage exceeding your SLA threshold. Use X-Ray to visualize the initialization breakdown.

The AWS Shared Responsibility Model is the foundational framework that defines who is responsible for what in cloud security. Think of it like renting an apartment: the landlord (AWS) is responsible for the building structure, plumbing, electrical wiring, and foundation. The tenant (you) is responsible for locking the doors, not leaving the stove on, and choosing who gets a key. If the building foundation cracks, that is the landlord's problem. If you leave the front door wide open and get robbed, that is your problem.

AWS's responsibility is security OF the cloud. This encompasses four layers. First, the physical layer: data center buildings with biometric access, 24/7 security guards, CCTV, mantraps, and physical destruction of decommissioned disks. AWS data center locations are undisclosed and undergo SOC1, SOC2, SOC3, ISO 27001, PCI-DSS, and FedRAMP audits. Second, the infrastructure layer: the global network backbone, edge locations, Availability Zones, and Regions. AWS manages the fiber optic connections between AZs (encrypted in transit), the redundant power supplies, and the cooling systems. Third, the compute layer: the Nitro hypervisor that isolates your EC2 instances from other tenants. AWS patched Spectre and Meltdown at the hypervisor level. Fourth, the managed services control plane: when you use RDS, AWS patches the database engine. When you use Lambda, AWS patches the runtime. When you use S3, AWS ensures the storage layer is resilient.

Your responsibility is security IN the cloud, and this is where most breaches happen. The exact scope depends on the service abstraction level. For IaaS services like EC2, you are responsible for: the guest operating system (patching Amazon Linux, Ubuntu, or Windows), the application stack running on it, security group and NACL configurations, host-based firewalls, antivirus, data encryption at rest and in transit, and IAM policies. For container services like ECS or EKS, AWS manages the underlying EC2 instances (if using Fargate) or you manage them (if using EC2 launch type). You always own the container images, application code, and network policies. For abstracted services like S3, Lambda, and DynamoDB, AWS handles much more — you are responsible for bucket policies, function permissions, data classification, encryption key management, and access logging configuration.

The most critical customer responsibilities that cause real breaches

1. S3 bucket misconfiguration: Setting bucket policies to public access has caused multiple high-profile data breaches. AWS added S3 Block Public Access as an account-level setting specifically because so many customers misconfigured bucket policies.

2. IAM over-permissioning: Granting AdministratorAccess or using wildcard (*) permissions instead of least privilege. A compromised application with broad IAM permissions can pivot to access any resource in the account.

3. Unpatched EC2 instances: AWS patches the hypervisor, but if you run a vulnerable version of Apache on your EC2 instance and it gets exploited, that is your responsibility. You should use AWS Systems Manager Patch Manager to automate OS patching.

4. Unencrypted data: AWS provides the encryption tools (KMS, ACM, CloudHSM), but you must choose to use them. Leaving an EBS volume or RDS instance unencrypted is your responsibility.

5. Missing logging: AWS provides CloudTrail, VPC Flow Logs, and S3 Access Logs, but you must enable and monitor them. Many organizations enable CloudTrail but never review the logs.

The model also extends to compliance. AWS provides compliance-eligible infrastructure (HIPAA, PCI, FedRAMP), but achieving compliance for your application requires you to configure services correctly and implement the required controls. AWS Artifact provides compliance reports, and services like AWS Config Rules and Security Hub help automate compliance checking. The shared responsibility model is not just a concept — it is the legal and operational framework that determines liability when something goes wrong.

Designing multi-region disaster recovery (DR) on AWS requires understanding two critical metrics: Recovery Time Objective (RTO) — how long you can afford to be down, and Recovery Point Objective (RPO) — how much data you can afford to lose. A payment processing system might need RTO of 30 seconds and RPO of zero. A marketing analytics dashboard might tolerate RTO of 4 hours and RPO of 24 hours. These requirements drive the architecture and cost.

AWS defines four DR strategies, each increasing in cost and decreasing in recovery time

Backup and Restore is the simplest and cheapest. You regularly back up data to another region — S3 Cross-Region Replication for objects, RDS automated backups or snapshots copied cross-region, DynamoDB global tables or point-in-time recovery exports, and AMI copies for EC2 configurations. During a disaster, you spin up infrastructure from these backups. Think of it like having a storage unit in another city with copies of all your important documents — if your house burns down, you can rebuild, but it takes days. RTO is typically 4-24 hours. RPO depends on backup frequency — if you back up hourly, you lose up to an hour of data. Cost is minimal because you only pay for storage in the secondary region.

Pilot Light keeps the minimal core infrastructure running in the DR region — typically just databases with continuous replication. Application servers, load balancers, and other compute resources are pre-configured but not running. Think of it like a gas stove with the pilot light always burning — you can fire up the full flame quickly but it is not heating anything until you turn the knob. During failover, you launch the pre-configured compute resources (from AMIs or Infrastructure as Code), update DNS, and the application comes online. RTO is typically 10-30 minutes. RPO depends on replication lag, usually seconds to minutes with asynchronous replication.

Warm Standby runs a fully functional but scaled-down copy of the production environment in the DR region. All components are active — web servers, application servers, databases — but at a fraction of production capacity. Think of it like having a fully furnished guest house that is smaller than your main house. During failover, you scale up the DR region to production capacity (Auto Scaling groups increase desired count), update DNS routing, and you are operational. RTO is typically 1-10 minutes. RPO is near-zero with synchronous or near-synchronous replication. The ongoing cost is significant — you are running infrastructure 24/7 in two regions.

Multi-Site Active/Active runs full production capacity in two or more regions simultaneously, with traffic split across them using Route 53 latency-based or geolocation routing. Both regions are serving live production traffic at all times. If one region fails, the other absorbs all traffic. Think of it like having two fully staffed offices — if one office has a fire, the other seamlessly handles all customers. RTO is near-zero (just DNS failover time, typically 30-60 seconds). RPO is zero with DynamoDB Global Tables or Aurora Global Database. The cost is roughly double because you run full capacity in both regions.

Critical components of any multi-region DR architecture

DNS failover with Route 53 health checks is the traffic steering mechanism. Route 53 monitors endpoints in both regions and automatically shifts traffic when the primary fails. For Active/Active, use latency-based routing. For Active/Passive, use failover routing.

Data replication strategy depends on the database. Aurora Global Database provides cross-region replication with under 1 second lag and supports managed failover. DynamoDB Global Tables provide active-active multi-region replication with eventual consistency. RDS read replicas can be promoted to standalone instances during failover, but this is a manual process.

Infrastructure as Code (Terraform or CloudFormation StackSets) ensures both regions have identical infrastructure definitions. Never manually configure the DR region — drift between regions is the number one cause of DR failures.

Regular DR testing is non-negotiable. Chaos engineering tools like AWS Fault Injection Simulator (FIS) can simulate region-level failures. Many organizations discover during an actual disaster that their DR plan does not work because they never tested it. Netflix's Chaos Monkey philosophy applies here — if you have not tested your failover, you do not have a failover.

Amazon S3's 11 nines (99.999999999%) durability is one of the most remarkable engineering achievements in cloud computing. To put this in perspective, if you stored 10 billion objects in S3, you would statistically expect to lose a single object once every 10,000 years. This durability is achieved through a combination of redundancy, continuous integrity checking, and automatic repair — not through any single technique but through defense in depth.

The foundation is multi-AZ replication. When you upload an object to S3, it is not simply stored on a single disk. S3 synchronously replicates the object across a minimum of three physically separated Availability Zones within the same region before returning a successful PUT response. Each AZ is a separate data center (or cluster of data centers) with independent power, cooling, and networking. This means that an entire data center can be destroyed — fire, flood, power failure — and your data survives intact in the remaining AZs. The key word is synchronous: S3 does not acknowledge your upload until the data is durably stored in all three locations.

S3 uses erasure coding rather than simple replication for storage efficiency. Simple three-way replication would require 3x the storage capacity. Erasure coding is a mathematical technique that splits data into fragments and generates parity fragments, such that any subset of fragments can reconstruct the original data. Think of it like a RAID-6 across data centers — even if multiple fragments are lost, the data can be fully reconstructed. The exact scheme AWS uses is not publicly documented, but it achieves the redundancy of three-copy replication while using significantly less storage capacity. This is critical at S3's scale (hundreds of exabytes).

Continuous integrity verification is the silent hero. S3 constantly reads back stored data and computes checksums (MD5, SHA-256, or CRC32C) to detect bit rot — the gradual degradation of data on storage media. Hard drives and SSDs do not fail catastrophically all at once; individual bits can silently flip due to magnetic interference, cosmic rays, or media degradation. If S3 detects a checksum mismatch (a corrupted fragment), it automatically heals the data by reconstructing the correct fragment from the remaining healthy fragments and writing it to new healthy storage. This process runs continuously across all stored data.

Automatic healing extends beyond bit rot. When an entire storage device fails (which is a daily occurrence at AWS's scale with millions of drives), S3 detects the loss and immediately begins reconstructing the affected fragments on other healthy devices. The reconstruction happens in the background without any impact to read/write operations. The system is designed to heal faster than failures accumulate — the mean time to repair is much shorter than the mean time between failures of remaining copies.

Storage class considerations affect durability differently. S3 Standard, S3 Intelligent-Tiering, S3 Glacier, and S3 Glacier Deep Archive all provide 11 nines of durability because they all replicate across three AZs. S3 One Zone-Infrequent Access stores data in only one AZ and provides 99.999999999% durability within that zone, but you lose everything if that AZ is destroyed. S3 Reduced Redundancy Storage (deprecated) offered lower durability. The storage class affects availability and retrieval time, not durability (except One Zone-IA).

At the hardware level, AWS designs custom storage servers and manages the full hardware lifecycle. Failed drives are detected automatically, decommissioned, and physically destroyed (shredded) to prevent data leakage. New drives are continuously provisioned and data is rebalanced across the fleet.

Networking integrity also matters. Data in transit between AZs during replication is protected against corruption. S3 uses Content-MD5 headers and checksums at the API level — if you enable additional checksum algorithms (SHA-256, CRC32C), S3 verifies your upload matches before storing it.

Durability versus availability is an important distinction. Durability means your data will not be lost. Availability means you can access it right now. S3 Standard offers 99.99% availability (about 53 minutes of downtime per year). A regional outage might make S3 temporarily unavailable (lowering availability), but your data is still safely stored (durability is maintained). Cross-Region Replication (CRR) adds durability against an entire region failure by maintaining copies in a second region.

AWS offers three load balancer types, each designed for different networking layers and use cases. Understanding when to use each is a fundamental networking decision that affects application latency, scalability, and architecture.

The Application Load Balancer (ALB) operates at Layer 7 (HTTP/HTTPS) of the OSI model. It understands the HTTP protocol — it reads headers, paths, query strings, and even request body content. Think of it as an intelligent receptionist who reads your appointment letter, understands which department you need, and directs you to the correct floor. ALB supports content-based routing rules: you can route /api/payments/* to one target group, /api/users/* to another, and host-based routing sends api.payments.example.com to different targets than dashboard.payments.example.com. This enables microservice architectures where a single ALB fronts dozens of services. ALB supports WebSocket connections natively, HTTP/2 for multiplexed streams, gRPC for service-to-service communication, and sticky sessions with cookies. It can perform SSL/TLS termination, authenticate users with OIDC or Amazon Cognito before forwarding requests, and return fixed responses or redirects without hitting any backend. ALB integrates with AWS WAF for web application firewall protection.

The key limitation of ALB is latency. Because it inspects HTTP content, it adds processing overhead. Typical ALB latency is 1-5ms for simple routing, but it can be higher under complex rule evaluation. ALB does not provide static IP addresses — it uses DNS names that resolve to changing IPs, which is problematic for clients that require IP whitelisting. The workaround is placing an NLB in front of an ALB or using AWS Global Accelerator for static anycast IPs.

The Network Load Balancer (NLB) operates at Layer 4 (TCP/UDP/TLS). It does not inspect HTTP content — it routes based on IP protocol data: source/destination IP and port. Think of it as a high-speed highway interchange that routes vehicles based on their lane (port) without stopping to check what is inside the vehicle. NLB is designed for extreme performance: it handles millions of requests per second with ultra-low latency (typically under 100 microseconds for the load balancer itself). It provides static IP addresses (one per AZ), supports Elastic IP assignment, and preserves the client's source IP address by default (no need for X-Forwarded-For headers).

NLB is ideal for: TCP services that are not HTTP (database connections, MQTT for IoT, custom binary protocols), applications requiring static IPs for firewall whitelisting, extreme throughput requirements (gaming, financial trading, video streaming), and as a frontend for AWS PrivateLink to expose services to other VPCs. NLB supports TLS termination but does not inspect HTTP content, so you cannot route based on URL paths or host headers.

A critical difference is health check behavior. ALB health checks are HTTP-based — it sends an HTTP GET to a path like /health and expects a 200 response. NLB health checks can be TCP (just verifying the port is open), HTTP, or HTTPS. TCP health checks are less accurate because a process might accept TCP connections but be unable to serve requests (for example, the JVM is running but the application is stuck in garbage collection).

The Classic Load Balancer (CLB) is the original AWS load balancer from 2009. It provides basic Layer 4 and Layer 7 functionality but lacks the advanced features of ALB and NLB. It does not support content-based routing, WebSockets (properly), HTTP/2, target groups, or multiple listeners with different rules. CLB uses one backend per load balancer, making microservice routing impossible. AWS has been discouraging CLB for years and recommends migrating to ALB or NLB. The only remaining reason to use CLB is EC2-Classic networking, which itself is deprecated.

Cost comparison: ALB charges per hour ($0.0225/hr) plus per Load Balancer Capacity Unit (LCU) based on new connections, active connections, and processed bytes. NLB charges per hour ($0.0225/hr) plus per Network Load Balancer Capacity Unit (NLCU) based on new connections/flows, active connections/flows, and processed bytes. NLB can be more expensive for HTTP workloads because it counts TCP connections, not HTTP requests — a single HTTP/2 connection can carry thousands of requests but NLB sees it as one flow. CLB charges per hour ($0.025/hr) plus per GB of data processed.

In modern architectures, the common pattern is: ALB for HTTP/HTTPS API traffic and web applications, NLB for non-HTTP protocols, PrivateLink services, and ultra-low-latency requirements, and CLB for nothing new — migrate away.

AWS CloudTrail is the comprehensive API activity logging service that records every action taken in your AWS account. Think of it as a security camera system for your entire AWS infrastructure — every door opened, every file accessed, every configuration changed is recorded with who did it, when, from what IP address, and whether it succeeded or failed. It is the single most important service for security auditing, compliance, and incident response.

CloudTrail captures three categories of events. Management events (also called control plane events) record operations that modify or manage AWS resources: creating an EC2 instance, modifying an IAM policy, deleting an S3 bucket, changing a security group rule. These are enabled by default on every AWS account. Data events (also called data plane events) record operations on the resources themselves: reading an object from S3, invoking a Lambda function, querying a DynamoDB table. These are not enabled by default because they generate high volume and cost. Insights events use machine learning to detect unusual API activity patterns — like a sudden spike in TerminateInstances calls that might indicate a compromised credential.

The anatomy of a CloudTrail event record contains critical forensic information. Every event includes: the event time (UTC timestamp), the event name (the specific API action like RunInstances), the event source (which AWS service, like ec2.amazonaws.com), the AWS region, the source IP address of the caller, the user identity (which IAM user, role, or AWS service made the call), the request parameters (what was requested), and the response elements (what AWS returned). For assumed roles, the event includes the role ARN, the session name, and the original identity that assumed the role — this is crucial for tracing actions back to human identities in environments that use role-based access.

CloudTrail delivers events to destinations. The primary destination is an S3 bucket, where events are stored as compressed JSON files organized by account, region, and date. A typical path looks like: s3://prod-cloudtrail-logs/AWSLogs/119283746501/CloudTrail/us-east-1/2026/06/18/. Events are delivered within approximately 5-15 minutes of the API call (this is not real-time). For near-real-time analysis, you send events to CloudWatch Logs, where you can create metric filters to trigger alarms on specific API patterns. For advanced querying, CloudTrail Lake provides a managed data lake with SQL-based querying across your event history.

Organization trails are essential for multi-account environments. An organization trail created in the management account automatically captures events from all member accounts in AWS Organizations. This gives your security team a single, centralized view of all API activity across every account. The trail writes to a central S3 bucket in the security account, and member account administrators cannot disable or modify the organization trail.

Production security patterns with CloudTrail

Unauthorized access detection: Create a CloudWatch metric filter for ConsoleLogin events where the responseElements.ConsoleLogin equals Failure. Trigger an SNS alarm when failed login attempts exceed a threshold, indicating a potential brute force attack.

Root account usage alerting: The root account should never be used for day-to-day operations. Create a metric filter for any event where userIdentity.type equals Root. Any root activity should trigger an immediate P1 alert to the security team.

Security group modification tracking: Monitor AuthorizeSecurityGroupIngress and RevokeSecurityGroupIngress events. Alert when someone opens port 22 (SSH) or port 3389 (RDP) to 0.0.0.0/0, which exposes instances to the entire internet.

IAM policy changes: Track CreatePolicy, AttachRolePolicy, PutRolePolicy, and DeletePolicy events. Any IAM change in production should go through a change management process, so out-of-band IAM modifications indicate either a process violation or compromised credentials.

Log integrity validation is critical. CloudTrail can enable log file validation, which creates a digitally signed digest file every hour. The digest contains SHA-256 hashes of all log files delivered in that period. If an attacker gains access and attempts to modify or delete CloudTrail logs to cover their tracks, the digest chain will break. You can validate the integrity using aws cloudtrail validate-logs. For defense in depth, send CloudTrail logs to a separate security account with a bucket policy that denies DeleteObject — even a compromised admin in the source account cannot delete the logs.

CloudTrail is free for management events (one copy per region). Data events and CloudTrail Lake incur charges based on the number of events. Given that a large organization can generate billions of data events monthly, be selective about which S3 buckets and Lambda functions you enable data events for — focus on sensitive data stores and critical functions rather than enabling data events globally.

Architecture

Cost Optimization Framework (Target: $150K/month savings)

Tier 1: Quick Wins (Week 1-2, ~$50K savings) - Right-size pods: Deploy VPA in recommend mode, identify over-provisioned pods. Most pods request 2-4x what they use. - Delete idle resources: Unused EBS volumes, unattached EIPs, idle load balancers, orphaned snapshots - Scale down non-prod: Dev/staging clusters scale to zero nights/weekends (saves ~60% of non-prod cost) - S3 lifecycle policies: Move infrequently accessed data to S3-IA/Glacier

Tier 2: Compute Optimization (Week 3-6, ~$70K savings) - Spot instances: Run stateless workloads (web servers, workers) on Spot (60-90% discount). Use Karpenter with diversified instance types for availability. - Karpenter over Cluster Autoscaler: Better bin-packing, faster scaling, automatic instance type selection - Graviton instances: ARM-based instances are 20% cheaper with equal or better performance - Reserved Instances / Savings Plans: 1-year commitment for baseline (always-on) capacity. Covers 40-60% of total compute.

Tier 3: Architecture Changes (Week 7-12, ~$30K savings) - Data transfer optimization: Keep services in same AZ where possible, use VPC endpoints instead of NAT Gateway for AWS services - Caching layer: Add Redis/ElastiCache to reduce database load and RDS costs - Serverless for bursty workloads: Move event-driven workloads to Lambda - Container image optimization: Smaller images = faster pulls = less EBS IOPS

IAM Concepts

• IAM Policy: A JSON document defining permissions (Allow/Deny on specific Actions for specific Resources). Policies alone do nothing — they must be attached to an identity.
• IAM Role: An identity that can be assumed by trusted entities (EC2, Lambda, EKS pods, other accounts). Unlike users, roles have no permanent credentials — they use temporary STS tokens.
• Instance Profile: A container for an IAM role that's attached to an EC2 instance. The instance's metadata service (169.254.169.254) provides temporary credentials.

EKS Least-Privilege with IRSA (IAM Roles for Service Accounts)

Without IRSA, all pods on a node share the node's IAM role — a single compromised pod can access all permissions. IRSA solves this:

1. Create an IAM OIDC provider for the EKS cluster 2. Create per-service IAM roles with trust policy referencing the Kubernetes ServiceAccount 3. Annotate the Kubernetes ServiceAccount with the IAM role ARN 4. Pod gets its own credentials via projected service account token (not instance metadata)

Putting AWS access keys in Kubernetes secrets creates a rotation and leakage problem. Relying on the node IAM role is also too broad because every pod on that node can potentially reach credentials unless IMDS access is tightly restricted. EKS Pod Identity gives the application a role through its Kubernetes service account, similar in spirit to an EC2 instance profile but scoped to the pod identity.

Operationally, each association maps one IAM role to one service account in one namespace in the cluster. When a pod uses that service account, EKS injects environment variables so supported AWS SDKs and the AWS CLI use the Pod Identity credential flow. The Pod Identity Agent runs as a DaemonSet on Linux EC2 nodes and serves credentials to pods on the same node.

A strong production answer includes the tradeoffs: restrict IMDS so pods cannot fall back to node credentials; remember that containers are not a security boundary; account for eventual consistency after creating associations; configure proxy NO_PROXY for the link-local agent address; and know that Fargate and Windows pods are not supported for EKS Pod Identity.

Picture a bank's internal pneumatic tube system connecting the teller counter to the vault in the back room — a physical, private pipe that never routes money-filled canisters out onto the public street just to travel twenty feet. The tube system is entirely self-contained within the building; there's no window it passes to get from point A to point B. A private-subnet-to-private-subnet RDS connection works the same way: as long as both endpoints live inside the VPC's private address space, AWS routes traffic through its own internal network fabric, never touching the public internet, regardless of how far apart the resources are physically.

AWS designed VPCs around this exact separation of concerns: a subnet is 'private' specifically because its route table has no route to an Internet Gateway (IGW), only to the VPC's local CIDR range (and optionally a NAT gateway for one-way outbound internet access, which is irrelevant here since RDS traffic never needs to leave the VPC at all). This means the compliance requirement isn't actually something you have to build — it's the default behavior of AWS networking, as long as you don't accidentally give either side a path to the internet, such as placing the RDS instance in a public subnet or attaching a public IP to the database (which AWS disables by default for RDS, but can be misconfigured).

Internally, when the application server issues a query, DNS resolution for the RDS endpoint returns a private IP address within the VPC's CIDR block (assuming the RDS instance has 'Publicly Accessible' set to No), and the packet is routed entirely through the VPC's internal routing table — AWS's software-defined networking layer handles this at the hypervisor level using its own backbone, never emitting the packet onto the public internet even for cross-Availability-Zone traffic within the same region, since AZs within a VPC are connected by AWS's private inter-AZ links.

At production scale for a banking client specifically, the design typically goes further than 'just use private subnets': security groups on the RDS instance are locked down to allow inbound traffic only from the application tier's specific security group (not a CIDR range, which is more precise and self-documenting), NACLs add a stateless second layer of defense at the subnet boundary, and traffic is encrypted in transit via TLS/SSL enforced through the RDS parameter group (rds.force_ssl = 1) even though it's already physically private, satisfying compliance frameworks that require encryption in transit as a defense-in-depth measure regardless of network isolation. For multi-account setups (common in banking, where prod and the database might sit in separate AWS accounts for blast-radius isolation), the same 'never touches the internet' guarantee is achieved via VPC Peering or Transit Gateway rather than a public endpoint with an IP allowlist, since IP allowlisting a public endpoint technically does route through the internet even if access is restricted.

The non-obvious gotcha: enabling a Multi-AZ RDS deployment or a read replica in a different region can silently reintroduce a compliance violation if the second region's VPC isn't connected via a private path (like a cross-region VPC peering connection or Transit Gateway peering) — engineers sometimes assume Multi-AZ failover traffic is automatically private, but a misconfigured cross-region replica without inter-region private connectivity will replicate over the public internet by default unless explicitly set up otherwise, silently breaking the exact guarantee the compliance team mandated.

Think of this like discovering your house master key is taped to a flyer stapled to a public bulletin board. Your first move isn't to carefully peel the flyer off the board — by the time you're doing that, anyone walking by could have already photographed it, and photographs don't come off the board when you remove the original. Your first move is to change the locks. Everything else — figuring out who saw it, cleaning up the bulletin board — happens after the locks are changed, because that's the only action that actually stops ongoing risk.

AWS designed root account credentials to have complete, unrestricted control over the account with no permission boundary possible — unlike IAM users, root cannot be constrained by an SCP, permission boundary, or resource policy in most cases, which is exactly why AWS's own best practice is to avoid creating root access keys at all and use IAM users or roles for everyday operations. Once root keys are public, the account is compromised in the fullest sense: an attacker could create new IAM users, delete CloudTrail logging, spin up expensive compute in unused regions for cryptomining, or exfiltrate data from every service in the account, all with zero permission checks standing in the way.

Internally, the response needs to happen in strict priority order because each step closes a different door. Deactivating the key (or deleting it and creating a new one, then deactivating the exposed one immediately) via the IAM console under 'My Security Credentials' takes effect within seconds and is the single action that stops any further API calls with that credential — this must happen before anything else, including notifying anyone or investigating scope, because scope investigation doesn't reduce ongoing risk and every minute of delay is a minute of continued exposure. Once the immediate bleeding is stopped, CloudTrail (assuming it wasn't disabled by an attacker, which is itself something to check) provides an audit trail of exactly which API calls were made with the exposed key, letting you distinguish 'exposed for 6 minutes with zero suspicious calls' from 'exposed for 6 minutes with a new IAM user created and attached to AdministratorAccess,' which are two very different incidents requiring very different follow-up.

At production scale, mature incident response here also includes checking AWS Config history and CloudTrail for any new IAM users, roles, or access keys created during the exposure window (since an attacker's first move is usually to create a persistent backdoor credential before the original one gets revoked), reviewing billing alerts and Cost Explorer for anomalous spend in unused regions (a classic sign of cryptomining), and rotating not just root but every other secret potentially reachable from root-level access, since root can read Secrets Manager, Parameter Store, and any KMS-encrypted data in the account. AWS's own Trusted Advisor and GuardDuty often flag exposed credentials automatically within minutes of a public GitHub push, since AWS actively scans public repos for leaked key patterns and may auto-quarantine the key by attaching a deny-all policy — treating this proactive AWS notification as confirmation, not as the primary detection mechanism, is a key operational maturity signal.

The non-obvious gotcha: git commit --amend or a force-push to remove the secret from the latest commit does not remove it from GitHub's history, cached forks, or any CI system that already cloned the repo and may have cached the .git objects — the secret must be treated as permanently public the moment it's pushed, and the entire remediation strategy should assume it has already been scraped by an automated bot, since these exist specifically to harvest credentials from public commits within seconds of a push, often faster than a human notices the mistake.

Think of an IAM User like an employee badge with a photo and a permanent PIN code, valid indefinitely until someone in security manually deactivates it — if that badge is lost or copied, whoever has it can walk in whenever they want until someone notices and revokes it. An IAM Role is more like a visitor badge that a receptionist hands out after checking ID at the front desk: it works for a few hours, expires automatically, and the visitor never actually possesses a permanent credential — even if someone photographs the visitor badge, it's useless the next day.

AWS designed this distinction specifically around credential lifecycle risk: IAM Users hold durable secrets (access keys that don't expire on their own, passwords protected only by rotation policy), which is exactly the shape of credential that ends up leaked in GitHub repos, laptop backups, or CI logs, because it's meant to be reused indefinitely. IAM Roles solve this by never handing out a durable secret at all — instead, a trusted principal (a user, an EC2 instance, a Lambda function, another AWS account, or an AWS service itself) calls sts:AssumeRole, and STS issues a temporary credential set (access key, secret key, and session token) that's valid for a bounded window, typically 15 minutes to 12 hours, after which it's cryptographically useless even if somehow exposed.

Internally, this is why EC2 instance profiles, ECS task roles, and Lambda execution roles are all IAM Roles rather than Users with embedded keys — an application running on an EC2 instance queries the instance metadata service, which itself talks to STS to fetch fresh temporary credentials automatically, rotating them behind the scenes without the application ever needing to store or manage a secret. This is a foundational shift in the security model: the credential simply cannot be exfiltrated in a way that remains useful for long, which is precisely why 'never use long-lived access keys for applications, only roles' is close to a universal best practice at any AWS shop with a real security program.

A Service-Linked Role exists for a narrower, more specific reason: some AWS services need to reach into other AWS services on your behalf to perform automated background actions — Auto Scaling launching and terminating EC2 instances, Elastic Load Balancing registering targets, RDS creating ENIs for Multi-AZ — and AWS predefines exactly what permissions that service needs, locks the role's trust policy and (often) its permissions policy so you cannot accidentally over-scope or under-scope it, and links the role's lifecycle to the resource using it (deleting the last Auto Scaling group referencing the role allows the role itself to be deleted, whereas a normal role could be deleted while still in active use, breaking things silently). This matters operationally because a hand-rolled custom role for the same purpose risks either being too permissive (a security finding) or missing a permission AWS added in a later API version update, which is exactly the class of bug Service-Linked Roles are designed to eliminate by having AWS itself own and maintain the permission set.

The non-obvious gotcha: you cannot always predict when a service silently requires a Service-Linked Role until you hit a permission error creating a specific resource — for example, enabling AWS Organizations' consolidated billing or certain ELB features auto-creates a service-linked role behind the scenes on first use, and in tightly locked-down accounts where IAM role creation itself is restricted via an SCP, this first-use auto-creation can fail with a cryptic access-denied error that has nothing to do with the actual resource you're trying to create, since the real blocker is the account's inability to create the prerequisite service-linked role at all.

The trade-off is storage price vs access cost/latency and durability/AZ scope. One Zone-IA is cheaper but stores in a single AZ (less resilient). Use lifecycle policies to transition objects automatically (e.g., Standard to IA after 30 days to Glacier after 90), and Intelligent-Tiering when access patterns are unpredictable so you do not have to tune it.

Pick ALB for HTTP routing, TLS termination, and WAF integration; NLB for raw TCP/UDP, extreme performance, or preserving client IP. On EKS the AWS Load Balancer Controller maps Ingress to ALBs (one ALB fronting many services) and Service type LoadBalancer to NLBs — consolidating at the ALB via Ingress is usually cheaper than one LB per service.

Start from measured requirements (CPU/memory/IO), pick the family that fits (burstable t3 for spiky low-baseline, m/c/r for steady workloads), and right-size from CloudWatch utilization after go-live. Combine with Auto Scaling for elasticity and Spot/Reserved/Savings Plans for cost. Avoid over-provisioning by iterating on real metrics.

The big-three offer near-equivalent primitives under different names. Compute: EC2 ≈ Azure VM ≈ Compute Engine. Object storage: S3 ≈ Blob ≈ Cloud Storage. Block: EBS ≈ Managed Disks ≈ Persistent Disk. SQL: RDS ≈ Azure SQL ≈ Cloud SQL. K8s: EKS ≈ AKS ≈ GKE. Registry: ECR ≈ ACR ≈ Artifact Registry. Knowing the mapping lets you translate architectures across clouds in interviews.

Identity & access: AWS IAM ≈ Microsoft Entra ID (formerly Azure AD) ≈ GCP Cloud IAM. Networking: VPC ≈ VNet ≈ VPC Network, with load balancers ELB ≈ Azure LB ≈ Cloud Load Balancing. Observability: CloudWatch ≈ Azure Monitor ≈ Cloud Monitoring (+ Log Analytics / Cloud Logging). CI/CD: CodePipeline ≈ Azure DevOps ≈ Cloud Build. These mappings are the most common cloud-round interview questions.