What should you monitor on a Docker host running a multi-container Compose stack (api, worker, redis, postgres) to catch resource contention before it causes container restarts or OOM kills?
Quick Answer
Monitor per-container CPU and memory usage against their configured limits, container restart counts, and host-level disk and memory pressure, since Compose has no scheduler to reschedule a killed container elsewhere — a resource problem on that single host directly becomes an outage. The most actionable signal is memory usage trending toward the container's `mem_limit` combined with restart_count incrementing, which predicts an OOM kill before it happens rather than after.
Detailed Answer
Running a Compose stack in production is like running a small restaurant out of a single kitchen instead of a chain with a central commissary — if the grill and the fryer both get slammed with orders at the same time, there's no second kitchen down the street to send overflow orders to. Kubernetes has a scheduler that can move a crashed workload to a healthier node; Compose has no such thing. Whatever host is running docker compose up is the entire blast radius, so noticing resource pressure early matters more here than almost anywhere else, because the fallback when something runs out of memory is simply 'the container dies and restarts in place,' not 'gets rescheduled somewhere with more headroom.'
This single-host reality is exactly why Compose stacks need tighter, more proactive monitoring than their Kubernetes equivalents. Compose was built as a local/single-node tool, not a distributed orchestrator, so it deliberately doesn't include the observability and self-healing machinery Kubernetes has — that's the trade-off for its simplicity. Teams running Compose in staging or small production environments need to bolt on their own visibility, because without it, a memory leak in checkout-worker doesn't get flagged and rescheduled elsewhere — it just silently degrades the whole host until postgres-db, which happens to share that host, also starts swapping and slowing to a crawl.
Step by step, the useful signal path looks like this: docker stats (or its scraped equivalent via cAdvisor/Prometheus node exporter) reports live CPU and memory usage per container; compare that against each service's configured mem_limit and cpus in the compose file. As a container's memory usage approaches its limit, the kernel's OOM killer inside that container's cgroup will eventually kill its main process, Docker will report the container exited with code 137, and if restart: always or on-failure is set, Compose immediately restarts it — which, if the underlying leak is still there, restarts, leaks again, and repeats in a crash loop that shows up as a climbing RestartCount in docker inspect. Host-level metrics (available memory, swap usage, disk I/O wait) matter just as much, because containers without explicit limits can consume the entire host's memory and starve every other container, not just themselves.
At production scale, teams scrape docker stats-equivalent metrics into Prometheus via cAdvisor and alert on three tiers: container memory usage above 80% of its limit for 5+ minutes (leading indicator), any container restart in the last 10 minutes (lagging but urgent), and host-level memory pressure or swap activity (systemic risk to every co-located container). Disk is the most commonly forgotten dimension — Postgres write-ahead logs and application log files filling the host's disk will eventually cause every container on that host to fail unpredictably, often with confusing 'no space left on device' errors that look unrelated to the actual disk exhaustion.
The gotcha: a service without an explicit mem_limit doesn't get 'no limit' in a good sense — it gets access to the entire host's memory, meaning one leaking service can starve out completely unrelated, otherwise-healthy containers on the same box, and the OOM killer might pick the *wrong* victim (postgres-db instead of the actual leaking checkout-worker) because Linux's OOM heuristics score based on memory usage and adjustability, not which process actually caused the problem. Engineers who set generous or absent limits on 'just the worker' are often surprised when the database gets killed instead during a memory crunch.