Design a Kubernetes RBAC strategy for a platform team supporting 50 product teams, where each team needs isolated namespaces but shared platform services. How do you handle cross-namespace access and emergency break-glass procedures?
Quick Answer
Use a hierarchical RBAC model with ClusterRoles for common permissions, namespace-scoped RoleBindings for team isolation, aggregated ClusterRoles for platform services, and a time-limited break-glass system using impersonation or temporary RoleBindings managed by a custom operator.
Detailed Answer
RBAC Architecture for Multi-Tenant Clusters
Tier 1 - Platform Team (ClusterAdmin-scoped)
The platform team needs cluster-wide access for infrastructure management. Create a platform-admin ClusterRole with permissions to manage nodes, namespaces, CRDs, webhook configurations, and storage classes. Bind this via ClusterRoleBinding to the platform team's IdP group.
Tier 2 - Product Teams (Namespace-scoped)
Each team gets a dedicated namespace (or set of namespaces for dev/staging/prod). Create a standardized set of ClusterRoles using aggregation labels: team-developer (deploy, view pods, exec), team-lead (all developer perms + manage ResourceQuotas, NetworkPolicies), team-viewer (read-only). Bind these per-namespace using RoleBindings referencing IdP groups.
Cross-Namespace Access
Teams that need to read ConfigMaps or Secrets from shared namespaces (e.g., platform-config) use ClusterRoles with resource name restrictions. For service-to-service communication, use NetworkPolicies rather than RBAC (RBAC controls API access, not data plane traffic).
Break-Glass Procedure
Implement a custom operator that watches a CRD BreakGlassRequest. When an on-call engineer creates a request (approved via PagerDuty webhook), the operator creates a time-limited RoleBinding (using a finalizer with TTL) granting elevated access. All break-glass events are audit-logged and auto-expire after 4 hours.
Admission Control Integration
Deploy OPA Gatekeeper or Kyverno to enforce RBAC boundaries: prevent teams from creating ClusterRoleBindings, restrict service account token mounting, enforce label-based namespace ownership. Use ValidatingAdmissionWebhooks to prevent privilege escalation (e.g., binding to cluster-admin).