Docker

Think of a restaurant kitchen that uses mixers, cutting boards, and raw ingredients to make a meal, but only sends the finished plate to the customer. A bad container image ships the entire kitchen to the table. A good multi-stage Docker build uses one stage as the kitchen and a second stage as the clean plate. The final image has only what the app needs to run, which cuts size, attack surface, and surprises in production.

Docker's build best practices call for multi-stage builds, smart base image choices, a solid .dockerignore file, skipping unnecessary packages, using the build cache wisely, and rebuilding images regularly. The real concern is not just size. Every extra package, shell, compiler, credential, or leftover file in the final image gives attackers something to inspect or exploit. Smaller runtime images are easier to scan, faster to transfer, quicker to start, and simpler to reason about when things go wrong.

The build process runs Dockerfile instructions in order and can reuse cached layers when an instruction and its inputs have not changed. This is why you copy stable dependency files before frequently changing source code. With BuildKit, teams can also use cache mounts and secret mounts so dependency downloads go faster and credentials never become image layers. Multi-stage builds then copy only selected files from builder stages into the final runtime stage.

In production pipelines, engineers pin base images by digest for repeatable builds, scan images for known vulnerabilities, generate SBOMs (software bills of materials), sign artifacts, and rebuild even when app code has not changed. Rebuilds matter because pinning a tag or digest freezes the base layer, while security patches land in newer versions. Good teams track image size, critical CVE counts, startup time, pull latency, and rollback success rates as key metrics.

The tricky part is that reproducibility and freshness pull in opposite directions. Pinning python:3.12-slim@sha256:... makes builds predictable, but it also locks in vulnerabilities until someone bumps the digest. Floating tags pick up patches automatically, but they can change under you and create builds you cannot reproduce. Senior engineers solve this with automated dependency-update PRs, signed digests, scheduled CI rebuilds, and policy gates. The goal is to make image supply-chain safety boring and routine rather than heroic and manual.

Think of a multi-stage build like a factory assembly line. In a car factory, welding robots, paint booths, and heavy machinery stay on the factory floor. Only the finished car rolls out and into the showroom. A multi-stage Docker build works the same way: all the bulky compilers, build tools, and source code stay in the build stage, while only the lean, finished binary ships in the final image. The key idea is separating what you need to build from what you need to run.

A multi-stage Docker build is a Dockerfile pattern where you write multiple FROM instructions, each starting a fresh stage. The first stage usually pulls a full SDK or compiler image, installs dependencies, compiles source code, runs tests, and produces a deployable file. Later stages start from a tiny base image like alpine or distroless and use COPY --from to grab only the necessary files from earlier stages. The result is a final image that contains nothing except what the app needs to run, with no leftover build tools, package caches, or temporary files.

Under the hood, Docker treats each stage as a separate build context with its own layer history. When Docker hits a second FROM instruction, it starts a clean image context while keeping previous stages in memory for reference. The COPY --from=builder command reaches into the filesystem of the named stage and pulls out specific paths. Each stage can use a completely different base image. For example, you might build with golang:1.21 and run with gcr.io/distroless/static-debian12. The build cache works per-stage, so changes to later stages do not force earlier stages to rebuild, which makes development iterations faster.

In production, multi-stage builds are considered a must-have for several reasons. First, they shrink image size dramatically. A Go app built in a standard golang image might weigh 900MB, but the final distroless image with just the static binary can be under 15MB. Smaller images pull faster across registries, scale quicker in Kubernetes, cost less to store, and give security scanners far less to audit. Second, multi-stage builds fit cleanly into CI/CD pipelines because the entire build process lives inside the Dockerfile. No external build scripts or Makefiles needed. Third, they enable reproducible builds since every developer and every CI runner uses the exact same build environment defined in that first stage.

A common mistake is copying too many files from the build stage. Using COPY --from=builder / /app/ instead of targeting a specific directory can accidentally pull in source code, credential files, or package caches that bloat the image and create security risks. Always copy only the exact artifact you need. Another subtle issue: build arguments (ARG) defined in one stage are not available in later stages. You must redeclare ARG after each FROM if you need the same value. Finally, intermediate stages are not always cleaned up automatically, so running docker image prune regularly is important to free disk space on build servers.

Picture building a tower of colored blocks. If you want to repaint the fifth block, you have to remove every block above it, repaint the fifth one, and then stack fresh blocks on top. Docker layer caching works the same way. Each Dockerfile instruction creates a layer (a block), and if any layer changes, every layer after it must be rebuilt from scratch. That is why instruction order in your Dockerfile has a huge impact on build speed.

Docker layer caching is the system that stores the result of each build instruction as a read-only filesystem layer. When you rebuild an image, Docker walks through the Dockerfile line by line, comparing each instruction to what it cached last time. For COPY and ADD, Docker computes a checksum of the files being copied and compares it to the cached checksum. For RUN, Docker compares the exact command string. If the instruction matches and all parent layers also match, Docker skips the work and reuses the cached result. The moment any instruction causes a cache miss, every instruction after it runs fresh, even if those later instructions have not changed at all.

Under the hood, each layer is stored as a compressed tar file in the Docker storage driver, usually overlay2 on modern Linux. The layer is identified by a SHA256 hash of its contents. When Docker checks a build step, it looks up the combination of parent layer ID plus current instruction hash in its local cache. With BuildKit (the default builder since Docker 23.0), the cache supports more advanced tricks like cache mounts for package manager directories and remote cache sources stored in container registries. BuildKit can also run independent stages in parallel, so stages that do not depend on each other build at the same time.

In production CI/CD pipelines, layer caching is the single biggest lever for build speed. The golden rule is: order instructions from least-frequently changed to most-frequently changed. System package installs and dependency downloads rarely change, so put them first. Application source code changes on every commit, so put it last. For a Node.js app, copy package.json and package-lock.json before the rest of the source, so npm install stays cached unless dependencies actually change. For Python, copy requirements.txt first and run pip install before copying your code. In CI, push cache layers to a registry with docker build --cache-from or BuildKit inline cache so different CI runners can pull and reuse layers across pipeline runs.

A critical trap is that any change to a COPY instruction invalidates it and every layer below it, even if the following RUN command is identical. Accidentally copying a frequently changing file, like a build timestamp or git revision, early in the Dockerfile busts the cache for everything downstream. Another common mistake is combining too many operations in a single RUN step. While this reduces layer count, it means the entire step reruns if anything in it changes. The sweet spot is grouping logically related operations that change at the same rate. Also, always keep apt-get update and apt-get install in the same RUN instruction. If they are split, a cached apt-get update might produce stale package lists that cause version resolution failures when paired with a fresh install.

Think of COPY as a simple courier who picks up a package from your desk and places it exactly where you say. ADD is a courier who also opens boxes for you: hand it a tar.gz archive and it unpacks it automatically, and it can even fetch packages from a URL. For the other pair, ENTRYPOINT is the permanent manager of a restaurant who greets every customer, while CMD is the daily special that can be swapped out. The manager (ENTRYPOINT) always runs, but what they serve (CMD) can be changed by the customer (whoever runs docker run).

COPY is the straightforward Dockerfile instruction for moving files and directories from the build context into the image. It takes a source path relative to the build context and a destination path inside the image. ADD does everything COPY does but adds two extras: it auto-extracts recognized compressed archives (tar, gzip, bzip2, xz) into the destination, and it can download files from remote URLs. However, Docker best practices strongly recommend using COPY for all standard file transfers because its behavior is explicit and predictable. Use ADD only when you specifically need tar extraction. For remote downloads, using curl or wget inside a RUN instruction is better because you can verify checksums, retry on failure, and clean up the downloaded file in the same layer.

Under the hood, both COPY and ADD create new image layers. Docker computes a checksum of the source files to decide if the layer cache can be reused. For ADD with URLs, Docker cannot reliably cache because the remote content might change without the URL changing, which leads to unpredictable builds. For tar extraction, ADD detects the archive format from file headers and extracts in place during layer creation. CMD and ENTRYPOINT are metadata instructions that change the image config rather than the filesystem. They do not create new layers but are stored as JSON in the image manifest. When both are present in exec form (the bracket syntax), Docker joins them together: the ENTRYPOINT array becomes the command and the CMD array gets appended as arguments.

In production Dockerfiles, the ENTRYPOINT plus CMD pattern is the standard for services. You set ENTRYPOINT to the application binary so the container always runs that binary, and CMD provides sensible default flags that operators can override. For example, ENTRYPOINT ["/order-service"] with CMD ["--port=8080", "--env=production"] means running docker run order-service --port=9090 replaces only the CMD part while the entrypoint stays fixed. This pattern matters in Kubernetes because the command field overrides ENTRYPOINT while the args field overrides CMD. Health checks, init containers, and sidecar proxies all depend on this predictable behavior.

A common trap is mixing shell form and exec form. CMD npm start (shell form) runs through /bin/sh -c, which means the node process becomes a child of the shell and does not receive SIGTERM signals properly, causing slow container shutdowns. CMD ["npm", "start"] (exec form) runs npm directly as PID 1, allowing proper signal handling. Another gotcha is defining CMD in a parent base image and then setting ENTRYPOINT in your Dockerfile without re-specifying CMD. The inherited CMD from the base image becomes the argument to your new ENTRYPOINT, which is almost never what you want. Always set both explicitly when using the ENTRYPOINT plus CMD pattern. Finally, remember that docker run arguments replace CMD entirely rather than appending to it, which surprises engineers who expect additive behavior.

Picture an office building. Bridge networking gives each team their own private floor with an internal phone system. Teams on the same floor can call each other easily, but reaching the outside world means going through the front desk (a NAT gateway). Host networking removes all office walls so every employee sits in the open lobby with a public phone line. Overlay networking connects multiple office buildings with a secure private tunnel so employees in different buildings can call each other as if they were on the same floor. None networking puts someone in a soundproof room with no phone at all.

Docker networking is built on Linux network namespaces, virtual Ethernet pairs (veth), and iptables rules. When Docker starts, it creates a default bridge network called docker0, which is a virtual Linux bridge device. Every container on the bridge network gets a veth pair: one end goes inside the container's network namespace as eth0, and the other end attaches to the docker0 bridge on the host. Containers on the same bridge can talk via their internal IP addresses. Host network mode skips namespace creation entirely, so the container shares the host's network stack, uses the host's IP address, and can bind directly to host ports without port mapping. Overlay networks use VXLAN encapsulation to create a virtual Layer 2 network that spans multiple Docker hosts, letting containers on different physical machines talk as if they were on the same local network. None mode creates a network namespace but does not set up any network interfaces inside it.

At the Linux kernel level, bridge mode relies on the bridge module and iptables NAT rules. Docker adds MASQUERADE rules in the POSTROUTING chain so outbound container traffic gets translated to the host's IP. Port publishing with -p adds DNAT rules in the PREROUTING chain to forward incoming host-port traffic to a container's internal port. For overlay networks, Docker uses its libnetwork library and the VXLAN protocol, which wraps Layer 2 Ethernet frames inside UDP packets on port 4789. A distributed key-value store (either Swarm's built-in Raft store or an external etcd) keeps track of which container IPs live on which hosts. The overlay driver also creates a separate bridge on each host called the overlay ingress bridge that handles the VXLAN wrapping and unwrapping.

In production, network choice depends on the architecture. Bridge is the default for single-host development and testing. User-defined bridge networks are strongly preferred over the default bridge because they give you automatic DNS resolution by container name. The default bridge only supports the old --link flag for name resolution. Host mode is used when you need maximum network throughput, such as for high-frequency trading systems or network monitoring tools, since it eliminates the overhead of virtual bridges and NAT. Overlay networks are the backbone of Docker Swarm deployments and are also available in Kubernetes via CNI plugins. Production Swarm clusters use overlay networks with the routing mesh, which load-balances incoming requests across all nodes to reach the right container.

A common gotcha is port conflicts in host mode. Since the container shares the host's port space, two containers cannot bind to the same port, and a container might conflict with host services already using that port. On bridge networks, a subtle issue is that containers on different user-defined bridges cannot talk by default. You must explicitly connect a container to multiple networks using docker network connect. Another trap is DNS resolution: containers on the default bridge network do not get automatic DNS, so connecting by container name fails silently, causing confusing connectivity issues. Always create custom bridge networks with docker network create for any multi-container application.

Think of a small dinner party versus a massive wedding reception. Docker Compose is the dinner party planner. You write a single checklist (docker-compose.yml) that says which dishes to prepare, how many plates to set, and which wine pairs with each course. Everything happens in your own kitchen on one evening. Kubernetes is the wedding planner who coordinates multiple catering companies, manages seating across several banquet halls, automatically reorders food when a tray runs empty, and reroutes guests if one hall floods. Both plan events, but one is built for simplicity on a single venue while the other handles complex logistics across many venues.

Docker Compose uses a declarative YAML file to define services, networks, volumes, and environment variables for a multi-container application. You run docker compose up and it builds images, creates networks, starts containers in dependency order, and attaches volumes, all on the local Docker host. Each service in the YAML maps to one or more container instances and can specify build contexts, port mappings, health checks, restart policies, and resource limits. Compose is great for local development, integration testing, and small single-host deployments. Kubernetes, on the other hand, is a cluster-level orchestration platform originally designed by Google. It manages containers across pools of machines called nodes, using building blocks like Pods, Deployments, Services, ConfigMaps, Secrets, Ingresses, and StatefulSets. Kubernetes provides automated scaling, rolling updates, self-healing through liveness and readiness probes, service discovery via internal DNS, and load balancing.

Under the hood, Docker Compose talks to the Docker daemon through the Docker API to create containers, networks, and volumes on the local host. It reads the YAML, figures out dependencies using the depends_on field, and makes sequential or parallel API calls. Compose V2 (now the standard) is a Docker CLI plugin written in Go, replacing the old standalone Python binary. Kubernetes has a fundamentally different architecture: it runs a control plane with an API server, etcd key-value store, scheduler, and controller manager on master nodes. Worker nodes run the kubelet agent and a container runtime like containerd. When you apply a Deployment manifest, the API server stores the desired state in etcd, the scheduler picks nodes for Pods, and the kubelet on each node pulls images and starts containers. Controllers continuously compare actual state with desired state, restarting crashed containers and adjusting replica counts.

In production, the choice between Compose and Kubernetes depends on scale, team skill, and operational needs. Compose is the standard for local development stacks where you need a database, cache, message queue, and your application running together. Many teams also use Compose in CI pipelines to spin up integration test environments. However, Compose lacks cluster-wide features: no automatic rescheduling if the host crashes, no horizontal pod autoscaling, no rolling update strategies with canary percentages, and no ingress controllers for HTTP routing. Kubernetes is the industry standard for production workloads needing high availability, geographic distribution, and fine-grained resource management. Some teams use Compose for development and Kubernetes for production, keeping the two in sync with tools like Kompose that convert docker-compose.yml files into Kubernetes manifests.

A frequent mistake is assuming Docker Compose is never suitable for production. In reality, small teams running a few services on a single server can use Compose with restart policies set to always and get reasonable uptime without the operational overhead of a Kubernetes cluster. Another common issue is using depends_on for startup ordering without health checks. depends_on only waits for the container to start, not for the application inside to be ready, so your app may crash trying to connect to a database that is still initializing. Use condition: service_healthy with proper health check definitions. On the Kubernetes side, a trap for newcomers is confusing Pods with containers. A Pod is a group of co-located containers that share network and storage, and running a single container per Pod is the norm rather than the exception.

Imagine baking a house key into a brick wall during construction. Anyone who gets access to the wall, even years later, can dig out the key. Baking secrets into Docker images is just as dangerous: images are stored in registries, cached on build servers, and pulled to every node that needs them. Even if you delete the secret in a later Dockerfile layer, it stays in the image's layer history and can be extracted with docker history or by inspecting the layer tar archive. The right approach is to hand the key to the resident only at the moment they walk through the door. Inject secrets at runtime, not build time.

Docker secrets management is about keeping sensitive data like database passwords, API keys, TLS certificates, and tokens out of image layers and source code. The simplest approach is runtime environment variables passed via docker run -e or the environment key in Docker Compose. While this works, environment variables are visible in docker inspect output and process listings inside the container. Docker Swarm has a built-in secrets feature: you create a secret with docker secret create, reference it in your service definition, and Docker mounts it as a file at /run/secrets/secret_name inside the container with restricted permissions. The secret is encrypted at rest in the Swarm Raft log and only sent to nodes running tasks that need it. For non-Swarm setups, you can inject secrets via volume mounts from the host or tmpfs mounts that live only in memory.

Internally, Docker Swarm secrets use the Raft consensus protocol to replicate encrypted secret data across manager nodes. When a service task gets scheduled on a worker node, the manager sends the decrypted secret over a mutually authenticated TLS connection to the worker. The secret is mounted as a tmpfs filesystem inside the container, meaning it never touches disk on the worker node. For third-party tools, HashiCorp Vault uses a sidecar or init container pattern: an init container authenticates to Vault using a machine identity (like an IAM role or Kubernetes service account), fetches the secrets, and writes them to a shared tmpfs volume. The application container reads from that volume and never needs to know about Vault directly. AWS Secrets Manager and GCP Secret Manager follow similar patterns, often using cloud-native identity federation instead of static credentials.

In production, a layered approach is standard. First, use a .dockerignore file to stop secret files like .env, credentials.json, and private keys from being included in the build context. Second, use multi-stage builds so even if a secret is needed during the build (for example, to access a private package registry), it stays in the builder stage and is never copied to the final image. Third, use BuildKit secret mounts with --mount=type=secret, which makes the secret available to a single RUN instruction without storing it in any layer. Fourth, at runtime, connect to an external secret manager that supports automatic rotation. When a database password rotates, the manager updates the mounted value and the application reloads it without a container restart. This rotation ability is something environment variables cannot provide because they are fixed at container start.

The most dangerous gotcha is using ARG for secrets in a Dockerfile. Build arguments show up in image metadata and can be extracted with docker history --no-trunc. Similarly, running RUN echo $SECRET > /app/.env followed by RUN rm /app/.env does not actually remove the secret. It is still in the layer created by the echo command. Always use BuildKit secret mounts for build-time secrets. Another trap is logging: applications that print their configuration at startup may accidentally write secrets to stdout, which Docker captures and stores. Make sure your logging framework redacts sensitive fields. Finally, docker inspect on a running container reveals environment variables in plain text. Anyone with Docker socket access can read every secret passed via -e flags, which is why file-based secret injection with restricted permissions is preferred for high-security setups.

Think of Docker volumes like a safe deposit box at a bank. You tell the bank (Docker) to create a box, and the bank decides where it is stored, who can access it, and how it is secured. You interact through the bank's interface, not by walking into the vault. Bind mounts are like giving someone a key to a specific room in your house. They get direct access to that exact location on your host filesystem, with all the benefits and risks that brings. The bank's box is more controlled and portable; the house key gives more flexibility but less isolation.

Docker volumes are the preferred way to persist data from containers. When you create a volume with docker volume create, Docker manages it in a directory under /var/lib/docker/volumes/ on Linux. Volumes can be named or anonymous. Named volumes survive container restarts and removals, making them ideal for database storage, application state, and file uploads. Bind mounts map a directory or file on the host machine directly into the container's filesystem. Unlike volumes, bind mounts depend on the host's directory structure, so they are not portable across machines. Docker also supports tmpfs mounts, which exist only in host memory and are never written to the filesystem, perfect for sensitive data that should not persist.

At the storage driver level, volumes use Docker's volume driver interface, which stores data locally by default but can be extended with plugins for NFS, AWS EBS, Azure Disk, GlusterFS, and other backends. The Docker daemon manages permissions, labels, and cleanup. When a container mounts a volume, the daemon uses a Linux bind mount internally to project the volume directory into the container's mount namespace. For bind mounts, the daemon directly projects the host path into the container's namespace without any management layer in between. Changes to a file on the host are immediately visible inside the container and the other way around. Both volumes and bind mounts can be made read-only by appending :ro to the mount spec, which is a security best practice when the container only needs to read configuration files.

In production, volumes are the standard for stateful services like databases, message queues, and object stores. A PostgreSQL container mounts a named volume to /var/lib/postgresql/data so data survives container updates and restarts. Volume drivers enable cloud storage integration. The REX-Ray driver, for example, automatically provisions and attaches AWS EBS volumes to the Docker host. For CI/CD caching, volumes persist build caches like Maven's .m2 directory or Node's node_modules across pipeline runs. Bind mounts dominate in development workflows: you mount your local source code into the container so file changes trigger hot-reloading without rebuilding the image. This is the pattern behind docker compose watch and tools like nodemon or Air for Go development.

A common gotcha with bind mounts is file ownership and permission mismatches. If your app runs as UID 1000 inside the container but the host files are owned by UID 501 (common on macOS), you get permission denied errors or the app creates files the host user cannot read. This is especially painful on Linux where UIDs are shared between host and container. With named volumes, Docker handles initial permissions based on the container's filesystem state. Another trap is that anonymous volumes created by VOLUME instructions in the Dockerfile pile up over time and eat disk space. Running docker volume prune is essential for cleanup. On macOS and Windows, bind mount performance is significantly slower than on Linux because files must sync between the host OS and the Linux VM running Docker. For large projects, this latency can make bind-mounted builds noticeably slower than building inside a volume.

Think of a woodworking shop. You need saws, clamps, sandpaper, and a workbench to build a cabinet, but the customer only gets the finished cabinet. They do not take home the saw. A multi-stage Docker build works the same way: one stage has all the build tools, and the final stage holds only the finished product.

In Docker, a multi-stage build uses multiple FROM instructions in a single Dockerfile. Each FROM begins a new stage with its own base image and filesystem. Intermediate stages can install compilers, download dependencies, run tests, and produce artifacts. The final stage starts from a tiny base like distroless or alpine and copies only the compiled binaries or bundled assets from earlier stages using COPY --from. This means the production image never contains gcc, npm, pip, or any build toolchain, eliminating hundreds of megabytes and thousands of CVE-carrying packages from the runtime image.

Under the hood, Docker and BuildKit process each stage as an independent node in the build graph. BuildKit can run independent stages in parallel, which is a major speed advantage over the legacy builder. When the Dockerfile is ordered correctly (base image first, dependency manifest copy second, dependency install third, source code copy fourth, build fifth), BuildKit reuses cached layers for everything up to the point where content changes. Since dependency manifests like package.json or go.sum change far less often than source code, this ordering means most CI builds only rebuild the final compilation step instead of re-downloading all dependencies.

At production scale, teams running 50 or more microservices through CI see dramatic results. A payments-api image that was 1.2 GB with a single-stage node build drops to 85 MB with a multi-stage build using distroless as the final base. CI time drops from 8 minutes to 2 minutes because dependency layers are cached. Security scanners report 90 percent fewer vulnerabilities because the final image has no compilers, shells, or package managers. Teams should also use .dockerignore to exclude test fixtures, documentation, and local configs from the build context, which prevents unnecessary cache busting and reduces context transfer time.

The tricky gotcha is that COPY --from references are position-based by default (stage 0, stage 1), which breaks silently when someone adds a new stage. Always name stages with AS and reference by name. Another trap is copying an entire directory from the build stage instead of specific artifacts, which can accidentally include build caches, test output, or sensitive files in the production image. Architects should also know that multi-stage builds do not automatically clean up intermediate images in CI. BuildKit's garbage collection handles this, but disk pressure on CI runners can still build up if max-storage is not configured.

Think of a building with three independent security systems: a keycard system that controls who can enter which floor, a phone system that blocks certain outgoing call types, and a camera system that restricts which rooms each employee can access. Each system works on its own, but sometimes a new employee's keycard triggers the camera alert because their access pattern looks unusual. Container security layers interact the same way.

Rootless containers tackle the fundamental risk that the Docker daemon traditionally runs as root on the host. In rootless mode, dockerd runs under a regular user's UID, and user namespaces remap container UIDs so that root inside the container maps to an unprivileged UID on the host. This means even if an attacker escapes the container, they land as a non-root user on the host. Seccomp (Secure Computing Mode) is a Linux kernel feature that filters system calls. Docker applies a default Seccomp profile that blocks about 44 dangerous syscalls including mount, reboot, and kexec_load. AppArmor provides Mandatory Access Control that restricts which files, network sockets, and capabilities a container process can access, regardless of its UID.

Internally, these three mechanisms operate at different kernel layers. User namespaces (rootless) work at the namespace level, remapping UIDs and GIDs through /etc/subuid and /etc/subgid. Seccomp operates at the system call interface, using BPF filters loaded before the container process starts. AppArmor operates at the LSM (Linux Security Module) layer, enforcing path-based access control policies loaded into the kernel. Docker applies all three during container creation: it sets up user namespace mapping, loads the Seccomp BPF filter via the OCI runtime spec, and assigns the AppArmor profile through the container's security options.

At production scale, the combination provides real defense in depth. A container running the checkout-api might have rootless mode preventing host root access, a custom Seccomp profile allowing only the 150 system calls the Go binary actually uses, and an AppArmor profile restricting file writes to /tmp and /var/log/checkout only. Security teams should test Seccomp in audit mode before enforcing, generate AppArmor profiles using tools like bane or aa-logprof, and test rootless networking thoroughly because rootless mode uses slirp4netns or pasta for network isolation, which adds latency compared to native bridge networking.

The tricky part is that rootless containers need the newuidmap and newgidmap system calls for user namespace setup, and an overly strict Seccomp profile can block these. Similarly, AppArmor may deny access to /proc/self/uid_map or /etc/subuid paths that rootless mode needs. When turning on all three, architects must test the specific combination: create a Seccomp profile that includes the system calls rootless mode requires, make sure the AppArmor profile allows the filesystem paths rootless networking and UID mapping use, and verify that slirp4netns or pasta networking works under both Seccomp and AppArmor constraints. Roll these out one at a time (rootless first, then Seccomp in audit mode, then AppArmor in complain mode) to avoid silent breakage in production.

Think of three ways to connect apartments in different buildings. A bridge network is like an intercom system inside one building. Residents can talk to each other, and a doorman routes calls to the outside. An overlay network is like a private phone system across multiple buildings, where calls are tunneled through the public phone lines but feel like internal calls. A macvlan network gives each apartment its own front door on the street. No doorman, no intercom, each unit is directly reachable by anyone walking by.

Docker bridge networking creates a Linux bridge device (docker0 or a custom bridge) on the host. Containers attach to this bridge via veth pairs, which are virtual ethernet cables with one end in the container's network namespace and the other on the bridge. The host uses iptables MASQUERADE rules to NAT outbound container traffic through the host's IP, and DNAT rules to forward published ports to container IPs. DNS resolution between containers on user-defined bridges uses Docker's embedded DNS server at 127.0.0.11.

Overlay networking adds VXLAN encapsulation so containers on different hosts can talk as if they are on the same Layer 2 network. Each host gets a VTEP (VXLAN Tunnel Endpoint) that wraps container-to-container frames in UDP packets (default port 4789) addressed to the remote host's IP. A distributed key-value store (Swarm's built-in Raft store or an external one) keeps the mapping of container IPs to host IPs. When container A on host 1 sends a packet to container B on host 2, the local VTEP wraps the frame in a VXLAN header, sends it as a UDP packet to host 2's VTEP, which unwraps it and delivers it to container B. This adds about 50 bytes of overhead per packet and introduces some encapsulation latency.

At production scale, the choice depends on network needs. Bridge is the default and works for single-host deployments or development, but does not span hosts. Overlay is standard for Docker Swarm and works for services needing multi-host communication with network isolation, but VXLAN encapsulation reduces MTU (from 1500 to about 1450) and adds CPU overhead. Macvlan is essential when containers must appear as physical devices on the network, for example a legacy payments-gateway that needs a static IP on the corporate VLAN, or network appliance containers that need promiscuous mode access. Macvlan has near-native network performance because there is no NAT or encapsulation, but containers cannot communicate with the host directly (a common surprise) and DHCP integration requires careful subnet planning.

The non-obvious gotcha with overlay networks is that the VXLAN UDP port (4789) must be open between all hosts, and some cloud security groups block it by default. With macvlan, the host itself cannot reach macvlan containers because the host NIC filters traffic to its own MAC address. You need a macvlan sub-interface on the host to talk to its own containers. Bridge networks have a subtle DNS issue: the default bridge (docker0) does not provide automatic DNS resolution between containers. Only user-defined bridges do, which catches teams that never created a custom network.

Think of a chef's kitchen with three upgrades. Cache mounts are like a shared pantry that stays stocked between shifts so the next cook does not have to re-buy ingredients. Secret mounts are like a recipe book the chef can read while cooking but that gets locked away before the dish leaves the kitchen. No customer ever sees the recipe. Cross-compilation is like a kitchen that can prepare both French and Japanese cuisine from the same recipe card by adjusting technique based on who ordered.

BuildKit is Docker's modern build engine, replacing the legacy builder. Cache mounts (--mount=type=cache) tell BuildKit to mount a persistent directory into a RUN step that survives across builds but never gets included in the final image layer. This is a game-changer for package managers: Go modules, npm packages, pip wheels, and apt archives get downloaded once and reused in later builds. Without cache mounts, developers either accept slow builds that re-download dependencies every time, or use fragile tricks like copying lock files early, which still re-downloads if the lock file changes. Secret mounts (--mount=type=secret) inject files like NPM tokens, private registry credentials, or SSH keys into a RUN step without them appearing in any image layer, build history, or intermediate container.

Under the hood, BuildKit implements cache mounts as named volumes managed by the BuildKit daemon. When a RUN instruction with --mount=type=cache runs, BuildKit mounts the named volume at the specified path. The command runs with access to cached data, then the volume is unmounted. The contents persist in BuildKit's storage between builds but are never committed to image layers. Secret mounts work similarly: BuildKit mounts a tmpfs with the secret file, the RUN command reads it, and the tmpfs is unmounted and never recorded. For cross-compilation, BuildKit sets BUILDPLATFORM (the host architecture, like amd64) and TARGETPLATFORM (the desired output, like arm64) as automatic build arguments. A Go binary can be compiled with GOOS and GOARCH set from TARGETPLATFORM without running an ARM emulator on an AMD64 CI runner.

At production scale, these features compound. A payments-api Go service that took 4 minutes to build drops to 45 seconds because go mod download results are cached across builds. The same Dockerfile produces linux/amd64 and linux/arm64 images in a single buildx command, enabling deployment to both x86 EC2 instances and Graviton ARM instances. Secret mounts eliminate the risk of accidentally pushing a Docker image that contains an NPM_TOKEN in a layer, a vulnerability that has caused real credential leaks in public registries. CI pipelines should use docker buildx bake for multi-service builds, which runs builds in parallel and shares cache across services.

The non-obvious gotcha is that cache mounts are local to the BuildKit instance. In CI systems where each build runs on a fresh runner, cache mounts are empty unless the team configures an external cache backend like a registry cache (--cache-to=type=registry) or a shared S3 cache. Another trap is that secret mounts require the BuildKit syntax directive (# syntax=docker/dockerfile:1) at the top of the Dockerfile. Older Docker versions or build systems that do not enable BuildKit silently ignore the mount directives, and builds fail because the secret file is missing.

Think of two car engines designed for the same chassis. containerd is a versatile engine that powers sedans, trucks, and race cars. It does more than any single car needs, but it works everywhere. CRI-O is an engine designed for one car model only. It is lighter, has fewer parts, and is tuned for exactly that chassis. Both engines use the same fuel injectors (runc) to actually combust the fuel (run the container).

In Kubernetes, the kubelet talks to the container runtime through the Container Runtime Interface (CRI), a gRPC API that handles image pulling, container lifecycle, and sandbox management. After Kubernetes dropped dockershim in v1.24, clusters moved to either containerd (with its built-in CRI plugin) or CRI-O. containerd is maintained by the CNCF and used by Docker Desktop, AWS EKS, Google GKE, and many self-managed clusters. CRI-O is maintained by the Kubernetes SIG-Node community and used mainly by Red Hat OpenShift and Fedora CoreOS. Both are production-proven and CNCF graduated projects.

Under the hood, when the kubelet sends a CreateContainer CRI request, the high-level runtime (containerd or CRI-O) pulls the container image if needed, unpacks it into a root filesystem using a snapshotter driver (overlayfs is most common), generates an OCI runtime spec (config.json), and calls the low-level OCI runtime. runc, the reference OCI runtime, reads that config.json and creates the container: it sets up Linux namespaces (pid, net, mnt, uts, ipc, user, cgroup), configures cgroups for resource limits, pivots the root filesystem, applies Seccomp and AppArmor profiles, drops capabilities, and finally execs the container's entrypoint process. The OCI runtime spec standardizes this interface so alternatives like crun (written in C for faster startup), gVisor's runsc (kernel-level sandboxing), or Kata Containers' kata-runtime (VM-based isolation) can replace runc without changing the high-level runtime.

At production scale, the choice between containerd and CRI-O depends on your organization. containerd is the safer default for most teams because it has broader ecosystem support, more documentation, and works across managed Kubernetes providers. CRI-O offers a smaller attack surface and tighter Kubernetes alignment since it matches Kubernetes release cycles and skips features irrelevant to Kubernetes. Performance differences are negligible for most workloads, but CRI-O's smaller codebase can be an advantage in security-audited environments. Teams should monitor runtime metrics including container start latency, image pull duration, pod sandbox creation time, and runtime daemon memory use.

The non-obvious gotcha is that switching runtimes on a running cluster requires draining nodes, reconfiguring the kubelet's --container-runtime-endpoint, and potentially reformatting the container storage directory because containerd and CRI-O use different on-disk layouts. Another trap is assuming OCI compatibility means feature parity: gVisor's runsc intercepts system calls for sandboxing but has compatibility gaps with some applications, and Kata Containers add VM startup latency. Architects must test their specific workloads against alternative runtimes instead of assuming drop-in replacement.

Think of building a secure Docker image like constructing a bank vault room. During construction, workers bring in welding equipment, power tools, scaffolding, and raw materials. Once the vault is complete, every construction tool is removed from the room. The vault door is keyed to specific authorized personnel, not a master key. The room contains only what is needed for its purpose: reinforced walls, a locking mechanism, and a ventilation system. If a thief breaks in, they find no tools to use against the vault itself. Multi-stage Docker builds follow the same principle: build tools exist only during construction and never ship to production.

A multi-stage Dockerfile separates the build environment from the runtime environment using multiple FROM statements. The first stage installs compilers, package managers, testing frameworks, and build dependencies needed to compile the application. The second stage starts from a minimal base image and copies only the compiled binary or application artifacts from the build stage. For a Java banking application, the build stage might use a full JDK image with Maven, while the runtime stage uses a Distroless Java image that contains only the JRE and no shell, package manager, or system utilities. This dramatically reduces the number of packages that vulnerability scanners flag and eliminates tools that attackers could use for post-exploitation activities like installing malware or pivoting to other services.

Running containers as non-root is a fundamental security control that prevents container breakout exploits from gaining host-level root access. The Dockerfile creates a dedicated application user with a specific numeric UID and GID, changes ownership of application files to that user, and switches to that user with the USER directive before the ENTRYPOINT. In banking environments, the specific UID matters because it must match file permissions on mounted volumes and satisfy Pod Security Standards that require runAsNonRoot in Kubernetes. Using numeric UIDs instead of usernames avoids dependency on /etc/passwd, which may not exist in Distroless images. The non-root user should have no shell assigned and no home directory beyond what the application needs.

Minimal base images are the foundation of attack surface reduction. A standard Ubuntu base image contains over 100 installed packages including shells, text editors, network utilities, and package managers. An Alpine image reduces this to roughly 15 packages. A Google Distroless image contains only the application runtime and its direct dependencies, with no shell at all. For banking applications, Distroless is preferred for production because if an attacker gains code execution inside the container, they cannot open a shell, install tools, or inspect the filesystem interactively. When debugging is needed, teams use ephemeral debug containers through kubectl debug rather than shipping debug tools in production images.

The production gotcha that catches many teams is the interaction between read-only root filesystems and application behavior. Many frameworks write temporary files, session data, or compilation caches to the filesystem at runtime. When the root filesystem is read-only, these writes fail and the application crashes. Teams must identify every path the application writes to and mount emptyDir volumes at those paths. Log files should go to stdout and stderr rather than filesystem paths. Another subtle issue is layer ordering in the Dockerfile: placing frequently changing instructions like COPY of application code after rarely changing instructions like dependency installation maximizes build cache utilization and reduces build times from minutes to seconds. In regulated environments, every base image must also be scanned and approved through the organization's software supply chain process before it can be used as a FROM source.

Think of Docker like a traditional bank with a single central manager who handles every transaction, account opening, and customer request. Every teller must route their work through this manager, and if the manager calls in sick, the entire branch shuts down. Podman is like a modern bank where each teller is fully empowered to complete transactions independently. There is no single manager whose absence stops operations. Each teller operates within their own authority, processes their own work, and the overall system is more resilient because no single failure brings everything down. This architectural difference has profound implications for security, reliability, and operations in regulated banking environments.

Docker's architecture centers on the Docker daemon, dockerd, a long-running background process that runs as root. Every docker CLI command communicates with this daemon through a Unix socket. The daemon manages image pulls, container lifecycle, networking, and storage. This design means that anyone with access to the Docker socket effectively has root access to the host, because they can mount any host directory, run privileged containers, or escape container isolation. In banking environments, this creates an unacceptable security risk: a compromised application or a developer with Docker socket access could potentially access the entire host system, read secrets from other containers, or modify the host filesystem.

Podman eliminates the daemon entirely. When you run podman run, a new process is forked directly without communicating through a centralized service. Each container is a child process of the podman command, managed by the standard Linux process model. This means containers can run as regular unprivileged users without any root daemon. A developer running podman run as their own user account creates a container that runs with that user's permissions and cannot access resources beyond what the user could access directly. Podman achieves this through Linux user namespaces, which map container UIDs to unprivileged host UIDs. The security benefit is dramatic: even if an attacker breaks out of a rootless Podman container, they land in an unprivileged user context with no path to root.

For enterprise operations, Podman offers several additional advantages. Systemd integration allows containers to be managed as standard systemd services with dependency ordering, automatic restart, and logging through journald. Podman can generate systemd unit files from running containers, making production deployment consistent with how enterprises manage all other services. Podman also introduces the concept of pods, groups of containers that share network and IPC namespaces, directly mirroring Kubernetes pod semantics. This means teams can develop and test pod configurations locally with Podman before deploying to Kubernetes, reducing the gap between development and production environments. Podman is also fully OCI-compliant and uses the same image format as Docker, so existing Dockerfiles and images work without modification.

The production gotcha that enterprises discover during migration is that Docker Compose workflows do not directly translate to Podman. While podman-compose exists as a compatibility layer, it does not support all Docker Compose features. Enterprise teams often move to Podman's native pod YAML support or use Kubernetes manifests directly for orchestration. Another consideration is that rootless Podman containers use slirp4netns or pasta for networking, which adds slight latency compared to Docker's bridge networking. For most banking applications the difference is negligible, but high-frequency trading or ultra-low-latency services may need to benchmark both approaches. Build-time tooling is also different: Podman uses Buildah as its image building engine, which supports building images without a Dockerfile using scripted commands, providing additional flexibility for CI/CD pipelines in regulated environments where every build step must be auditable.

Think of container image scanning like the quality inspection process at a pharmaceutical manufacturing plant. Every batch of medicine must pass through a testing station that checks for contamination, verifies ingredient concentrations, and confirms the batch meets safety standards before it can proceed to packaging and distribution. If a batch fails any critical safety test, it is quarantined and cannot reach patients. The testing station also continuously monitors stored batches because a recall notice could arrive at any time for an ingredient that was previously considered safe. Container image scanning in CI/CD follows the same pattern: scan at build time, gate on policy, store the evidence, and rescan continuously.

The scanning process works by analyzing every layer of a container image. A container image is a stack of filesystem layers, each adding packages, libraries, configuration files, or application code. Vulnerability scanners like Trivy decompose the image into these layers, identify every installed OS package and application dependency, and compare each one against the National Vulnerability Database, vendor security advisories, and specialized databases like the GitHub Advisory Database. The scanner produces a report listing every known CVE affecting the image, with severity ratings from negligible to critical, fix availability information, and CVSS scores. For banking applications, the report also generates a Software Bill of Materials that documents every component in the image, satisfying regulatory requirements for software transparency.

The CI/CD pipeline integrates scanning as a mandatory gate between the build and push stages. After the Docker image is built, the scanner runs against the local image before it is pushed to the container registry. The pipeline configuration specifies a policy: which severity levels should block the pipeline, which CVEs are explicitly accepted through a risk exception process, and what the maximum allowable vulnerability count is per severity. In banking environments, the typical policy blocks any image with critical CVEs and requires a security review for high CVEs. The scan results are published to a centralized dashboard and stored as pipeline artifacts for SOX audit evidence. Failed scans create tickets in the vulnerability management system with the affected image, namespace, owning team, and remediation deadline.

Continuous scanning addresses the gap between build-time security and runtime exposure. An image that passes all checks today may have a new critical CVE published tomorrow affecting one of its base image packages. Registry-level scanning tools like ECR enhanced scanning, Harbor with Trivy, or Prisma Cloud continuously rescan all images in the registry against updated vulnerability databases. When a new critical CVE affects an image that is currently deployed in production, the scanning system triggers an alert to the owning team with the specific CVE details, affected pods, and remediation guidance. Teams then rebuild with a patched base image and deploy through the standard pipeline, creating a complete audit trail of the vulnerability lifecycle from discovery to remediation.

The production gotcha is managing false positives and vulnerability exceptions without creating security debt. Not every CVE is exploitable in every context: a vulnerability in a network utility is irrelevant if the container does not have network tools installed or if the affected code path is never reached. Banking teams maintain an exception registry where accepted vulnerabilities are documented with a justification, an expiration date, and an approving security engineer. Exceptions are reviewed quarterly as part of the vulnerability management program. Another common mistake is scanning only the application image while ignoring sidecar containers, init containers, and operator images that run in the same pod. A vulnerable logging sidecar or Vault Agent container provides the same attack surface as the main application container and must be scanned with the same rigor.

Think of image signing like the notarization process for financial documents at a bank. When a loan officer prepares a mortgage document, the document is reviewed, verified, and stamped by a licensed notary. The notary seal proves three things: the document came from the bank (provenance), it has not been altered since signing (integrity), and a specific authorized person vouched for it (accountability). If anyone changes even a single word after notarization, the seal becomes invalid and the document is rejected. Container image signing provides the same guarantees for software: the image was built by an authorized pipeline, it has not been tampered with since signing, and there is an auditable record of who authorized it.

Docker Content Trust is Docker's built-in signing mechanism, based on The Update Framework and the Notary project. When DCT is enabled through the DOCKER_CONTENT_TRUST environment variable, every docker push operation signs the image with the publisher's private key, and every docker pull operation verifies the signature before accepting the image. DCT uses a two-key architecture: a root key that should be stored offline in a hardware security module and a repository-specific signing key used for daily operations. While DCT provides basic signing capabilities, it has limitations in modern CI/CD environments: key management is complex, the Notary server adds infrastructure overhead, and integration with Kubernetes admission controllers requires additional tooling.

Cosign from the Sigstore project has emerged as the modern standard for container image signing in enterprise environments. Cosign's keyless signing mode uses OIDC identity providers to bind signatures to verified identities rather than manually managed private keys. In a banking CI/CD pipeline, the GitHub Actions workflow authenticates to Sigstore's Fulcio certificate authority using the workflow's OIDC token, receives a short-lived signing certificate, signs the image, and records the signature in Sigstore's Rekor transparency log. The signature is stored as an OCI artifact alongside the image in the container registry. This approach eliminates private key management entirely: there is no key to rotate, leak, or protect in a hardware security module. The transparency log provides a tamper-evident audit trail of every signing event, which banking regulators can verify independently.

Kubernetes admission controllers complete the trust chain by enforcing signature verification at deployment time. Without admission control, signing images is security theater because nothing prevents unsigned images from running. Kyverno's verifyImages rule or OPA Gatekeeper with the Cosign constraint template intercept pod creation requests, extract the image reference, query the registry for the Cosign signature, verify it against the expected identity (the CI/CD pipeline's OIDC issuer and subject), and reject the pod if verification fails. In banking environments, this policy applies to all production namespaces and blocks images from unauthorized registries, unsigned images from authorized registries, and images signed by unauthorized pipelines. The admission controller also validates image attestations, which are signed metadata about the image's build process, such as the scan results, SBOM, and build provenance.

The production gotcha is the operational complexity of bootstrapping trust across a large organization. When image signing is first enforced, every existing deployment must be updated with signed images, or pods will fail to start after a rollout. Teams must coordinate a migration period where the admission controller runs in audit mode, logging violations without blocking. During this period, all CI/CD pipelines are updated to include the signing step, all existing images in the registry are re-signed, and all Deployment manifests are updated to reference signed image digests. Another critical consideration is signature verification latency: the admission controller must contact the registry for every pod creation, adding milliseconds to pod scheduling. At scale, teams cache verification results and configure timeouts to prevent Vault or registry outages from blocking all pod creation. Banking teams also implement emergency bypass procedures for disaster recovery, where a designated security officer can temporarily disable enforcement through a documented break-glass process that is fully audited.

Think of building a house. If you repaint one room, you do not tear down and rebuild the entire house — you only repaint that room. Docker build optimization works the same way: structure your build so that changes to one part do not force rebuilding everything else.

Docker builds work layer by layer. Each instruction in a Dockerfile creates a layer, and Docker caches each layer. If nothing changed in a layer and all layers before it are also unchanged, Docker reuses the cached version instead of rebuilding it. This is why layer ordering matters — put package manager installs (which change rarely) before copying application code (which changes every commit).

The most impactful optimizations are: First, use a minimal base image like alpine:3.20 (5MB) instead of ubuntu:22.04 (77MB) — smaller images download faster and have fewer layers to process. Second, use multi-stage builds where the first stage has build tools (gcc, maven, npm) and the second stage only copies the compiled artifact into a minimal runtime image. Third, leverage cache mounts with BuildKit (--mount=type=cache) to persist package manager caches (npm cache, pip cache, maven repository) between builds so dependencies are not re-downloaded every time.

At production scale, teams also use distributed build systems like BuildKit with remote cache backends (S3, registry), Docker layer caching in CI systems (GitHub Actions cache, GitLab CI cache), and parallel multi-stage builds where independent stages run concurrently. Monitoring build times per pipeline and setting alerts when builds exceed a threshold helps catch regressions early.

The non-obvious gotcha is that COPY . . early in the Dockerfile invalidates every subsequent layer on every code change. Always copy dependency files first (package.json, requirements.txt, pom.xml), install dependencies, then copy application code. This way dependency installation is cached unless the dependency file itself changes.

Think of image scanning like an airport security checkpoint. Every bag (image layer) goes through an X-ray machine (vulnerability scanner) that compares its contents against a database of known threats (CVE databases). If the scanner finds a prohibited item above a certain threat level, the passenger (image) is stopped from boarding (deploying). Just as airports have different alert levels that determine what gets flagged, your CI pipeline has severity thresholds that control whether a vulnerability blocks deployment or merely generates a warning.

Docker image vulnerability scanning is the process of analyzing every layer of a container image to identify known security vulnerabilities in operating system packages, application dependencies, and configuration issues. Scanners decompose the image into its constituent layers, extract the package manifests — such as dpkg status files for Debian, RPM databases for Red Hat, package-lock.json for Node.js, and requirements.txt for Python — and compare every installed package version against databases like the National Vulnerability Database (NVD), GitHub Advisory Database, and distribution-specific security trackers. Each identified vulnerability is assigned a severity rating (Critical, High, Medium, Low) based on its CVSS score, and the scan report maps these to specific remediation steps such as upgrading a package or switching to a patched base image.

Under the hood, modern scanners like Trivy maintain a local or cached copy of vulnerability databases that are updated with each CI run. When Trivy scans an image, it first pulls and extracts the image manifest and layer blobs, then walks through each layer's filesystem to identify package managers and their installed packages. For OS-level scanning, it reads distribution metadata to determine the exact OS version and matches packages against the corresponding security tracker. For application-level scanning, it parses lockfiles and dependency trees to detect vulnerabilities in transitive dependencies that your code never directly imports. Grype uses a similar approach but with the Anchore vulnerability database. Both tools output machine-readable formats like JSON and SARIF that integrate with CI platforms, code scanning dashboards, and security information and event management systems.

In a production CI pipeline, image scanning should be a mandatory gate before any image is pushed to a container registry. The typical workflow is: build the image, scan it with a severity threshold, fail the pipeline if critical or high vulnerabilities are found, and only then push the tagged image to the registry. Most teams run scans at two points: during the CI build and on a scheduled basis against images already in the registry, because new CVEs are published daily against previously clean packages. Integration with GitHub Actions, GitLab CI, or Jenkins is straightforward — Trivy provides official actions and plugins. Teams also implement allowlists for false positives or vulnerabilities with no available fix, storing these in a .trivyignore or policy file committed alongside the Dockerfile.

A critical gotcha is scanning only the final image and missing vulnerabilities introduced in build stages. If your multi-stage build copies a binary from a builder stage that was compiled against a vulnerable library, the final image scan may not detect it because the vulnerable package exists only in the builder stage. Static analysis tools like Snyk can scan the Dockerfile itself to catch insecure base images and risky RUN instructions. Another common mistake is failing to update the vulnerability database before each scan — stale databases produce false negatives. Always ensure your scanner fetches the latest database as part of the CI step. Finally, remember that scanning is not a substitute for keeping base images updated; schedule regular rebuilds to pull patched base images even when your application code has not changed.

Attack Vectors

1. Vulnerable base images: Unpatched OS packages with known CVEs 2. Running as root: Container processes run as root by default — if container is compromised, attacker has root in the container namespace 3. Docker socket exposure: Mounting /var/run/docker.sock gives full control over the host Docker daemon 4. Secrets in images: API keys, passwords baked into image layers (visible via docker history) 5. Privilege escalation: --privileged flag disables ALL security boundaries 6. Supply chain attacks: Compromised base images or dependencies

Defense-in-Depth Layers

Complete Execution Flow of 'docker run nginx'

Build Strategies

1. QEMU Emulation (Universal, Slow): BuildKit uses QEMU user-mode emulation to run foreign-architecture binaries on the build host. When building an arm64 image on an amd64 host, every RUN instruction executes through QEMU's arm64 emulator. This is transparent but 5-20x slower than native execution. - Setup: docker run --privileged --rm tonistiigi/binfmt --install all - Use case: Building images where most work is in COPY/ADD (interpreted languages), not RUN (compilation)

2. Native Cross-Compilation (Fast, Language-Specific): For compiled languages (Go, Rust, C/C++), set cross-compilation flags instead of emulating the target architecture. The compiler runs natively on the host but produces target-architecture binaries. - Go: Set GOARCH=arm64 while running on amd64 - Rust: Use --target aarch64-unknown-linux-musl - C/C++: Use cross-compiler toolchain (aarch64-linux-gnu-gcc)

3. Multi-Node Build (Fastest, Complex): Use BuildKit with remote builder nodes - an amd64 builder and an arm64 builder. Each platform builds natively. BuildKit orchestrates both and merges results into a manifest list.