Meta Production Engineering Interview: How would you implement resource quota enforcement across 200 namespaces in a multi-tenant Kubernetes cluster, ensuring fair resource distribution while allowing burst capacity for critical services?
Quick Answer
Use hierarchical ResourceQuotas with LimitRanges for defaults, PriorityClasses for tiered preemption, and a custom admission webhook or Kyverno policies to enforce organizational quotas that span across namespaces. Implement a resource banking system where teams can borrow from a shared pool.
Detailed Answer
Multi-Layer Resource Management
Layer 1 - Namespace ResourceQuotas
Each namespace gets a ResourceQuota defining hard limits on CPU, memory, pod count, and storage. For 200 namespaces on a cluster with 1000 cores, you might allocate 4 cores per namespace base quota (800 cores committed, 200 cores buffer). LimitRanges set default requests/limits per container so developers don't need to specify resources manually.
Layer 2 - Hierarchical Quotas
Kubernetes doesn't natively support org-level quotas (e.g., 'Team Payments gets 50 cores across their 5 namespaces'). Implement this via a custom controller or use Hierarchical Namespace Controller (HNC). The controller watches ResourceQuota usage across related namespaces and prevents new pod creation if the org-level quota is exceeded.
Layer 3 - Burst Capacity with Priority
Allow critical services to burst beyond their quota by using PriorityClasses. Define tiers: critical (preempts everything), production (preempts batch), development (preempts nothing). When a critical service needs to burst, it can preempt lower-priority pods. This gives you overcommitment without overprovisioning.
Layer 4 - Vertical Pod Autoscaler (VPA)
Deploy VPA in recommendation mode to continuously suggest optimal resource requests based on actual usage. Feed VPA recommendations into a dashboard showing quota utilization efficiency. Most teams request 3-5x more resources than they actually use.
Monitoring and Chargeback
Export kube_resourcequota and kube_pod_container_resource_requests metrics to Prometheus. Build Grafana dashboards showing per-team utilization vs allocation. Implement a cost allocation system using kubecost or custom metrics to charge teams for reserved (not used) resources, incentivizing right-sizing.