At a company running 1000+ microservices on Istio, how does service discovery actually work under the hood, and what becomes the bottleneck at that scale?

Istio doesn't do per-request discovery lookups like client-side DNS — it precomputes the full service graph and pushes it down. Istiod watches the Kubernetes API server (Endpoints/EndpointSlices, Services) and any other configured registries, builds an internal model, and streams it to each Envoy sidecar over gRPC using the xDS APIs: CDS (clusters), EDS (endpoints within a cluster), LDS (listeners), RDS (routes).

At small scale this is instant and cheap. At 1000+ services with frequent pod churn (rolling deploys, autoscaling, spot instance turnover), three things start to hurt:

1. Push fan-out cost: every endpoint change in any service can trigger a push to every sidecar that references that cluster. Istiod batches and debounces changes (default ~1s debounce window), but during mass rollouts this still means thousands of pushes per second. 2. Sidecar resource consumption (the 'sidecar tax'): each Envoy holds the full configuration for every service it can reach. Without Sidecar resources scoping egress visibility per-namespace, every proxy holds state for the entire mesh, multiplying memory linearly with both cluster size and service count. 3. Convergence lag: under heavy churn, some sidecars receive stale endpoint lists for a few hundred ms to a few seconds, causing transient 503s on newly-scaled-down or newly-scaled-up pods.

Mitigations at scale: use Sidecar custom resources to restrict each workload's egress visibility to only the services it actually calls (cuts memory and push volume dramatically), tune istiod's push debounce and EDS batching, shard istiod by namespace/revision for very large meshes, and consider ambient mesh (sidecar-less, per-node ztunnel) to remove the per-pod proxy overhead entirely.