Meta production automation interview: You need to run 10,000 HTTP health checks from Pyth…

Meta production automation interview: You need to run 10,000 HTTP health checks from Python in under two minutes. How would you design the asyncio code so it is fast, cancellable, and does not overload dependencies?

⚡

Quick Answer

Use asyncio with bounded concurrency, per-request timeouts, structured cancellation, and a result object for every target. In modern Python, TaskGroup helps manage groups of tasks, while semaphores and client timeouts prevent the script from becoming a denial-of-service tool.

Detailed Answer

Think of this like checking 10,000 doors in a city. Sending one inspector door by door is too slow, but sending 10,000 inspectors at once blocks streets and creates a new incident. Good async automation is controlled parallelism: enough concurrency to finish quickly, but not so much that DNS, load balancers, proxies, or the service itself collapse.

Python asyncio is useful for I/O-bound work because the event loop can switch tasks while each request waits on the network. The design should separate three concerns: target generation, bounded execution, and result collection. A semaphore limits the number of in-flight checks. Each request has connect and read timeouts. The caller gets one result per endpoint with status, latency, error class, and retry count.

Modern Python adds structured concurrency through asyncio.TaskGroup. A TaskGroup gives a clear lifetime to a set of related tasks: create them inside the block, wait for completion, and let errors cancel sibling tasks in a controlled way. For health-check automation, many teams choose to catch per-target exceptions inside the worker so one bad endpoint does not cancel the whole scan. For deployment gates, they may do the opposite: fail fast when critical checks break.

At scale, the bottleneck may not be Python. DNS resolvers, NAT gateways, outbound proxies, and service-side rate limits all matter. A mature script includes a configurable concurrency limit, jittered retries for transient network errors, and an allowlist of target domains or clusters. It also emits metrics about timeout rate and p95 latency because the health checker itself becomes part of the deployment system.

The subtle gotcha is cancellation hygiene. If a CI job times out or a deploy is aborted, the script should stop launching new checks and close HTTP connections cleanly. Senior engineers talk about cancellation, finalizers, and bounded resource cleanup, not just await gather.

from __future__ import annotations # Enable modern typing features. import asyncio # Use Python's async event loop for I/O concurrency. from dataclasses import dataclass # Return structured health check results. from time import monotonic # Measure latency safely. import aiohttp # Use an async HTTP client for many concurrent requests. @dataclass(frozen=True) # Keep each result immutable. class HealthResult: # Represent one endpoint check. url: str # Endpoint that was checked. ok: bool # Whether the endpoint returned a healthy response. status: int | None # HTTP status when a response exists. ms: int # Elapsed milliseconds. error: str | None # Error class for timeout or network failures. async def check_one(session: aiohttp.ClientSession, sem: asyncio.Semaphore, url: str) -> HealthResult: # Check one URL. async with sem: # Bound total in-flight checks. start = monotonic() # Start timing before the request. try: # Convert exceptions into result records. async with session.get(url) as resp: # Run the HTTP GET asynchronously. await resp.read() # Drain the body so the connection can be reused. ms = int((monotonic() - start) * 1000) # Convert elapsed time to ms. return HealthResult(url, 200 <= resp.status < 300, resp.status, ms, None) # Return success or bad status. except asyncio.TimeoutError: # Request exceeded the configured timeout. ms = int((monotonic() - start) * 1000) # Measure timeout duration. return HealthResult(url, False, None, ms, "timeout") # Preserve timeout as data. except aiohttp.ClientError as exc: # DNS, connect, TLS, and protocol failures land here. ms = int((monotonic() - start) * 1000) # Measure failed request duration. return HealthResult(url, False, None, ms, type(exc).__name__) # Preserve error class. async def check_many(urls: list[str], concurrency: int = 200) -> list[HealthResult]: # Run all checks. timeout = aiohttp.ClientTimeout(total=8, connect=2, sock_read=5) # Bound every request phase. sem = asyncio.Semaphore(concurrency) # Protect dependencies from unbounded fanout. async with aiohttp.ClientSession(timeout=timeout) as session: # Reuse connections efficiently. async with asyncio.TaskGroup() as tg: # Give all tasks a structured lifetime. tasks = [tg.create_task(check_one(session, sem, url)) for url in urls] # Schedule checks. return [task.result() for task in tasks] # Return one result per URL. if __name__ == "__main__": # Keep script entry point explicit. targets = ["https://payments.example.com/healthz", "https://orders.example.com/healthz"] # Example targets. results = asyncio.run(check_many(targets, concurrency=100)) # Run event loop from sync main. failed = [r for r in results if not r.ok] # Summarize failed checks. raise SystemExit(1 if failed else 0) # Fail CI only when health checks fail.

Meta production automation interview: You need to run 10,000 HTTP health checks from Python in under two minutes. How would you design the asyncio code so it is fast, cancellable, and does not overload dependencies?

More Python DevOps Automation questions