What is the Prometheus Operator and how does ServiceMonitor work in Kubernetes?

Imagine you hire a property manager for your apartment building. Instead of personally meeting each new tenant and updating the paperwork, you tell the manager: anyone who moves into floors 3 through 5 gets set up with water, electricity, and parking. The property manager (Prometheus Operator) watches for new tenants (services), matches them against your rules (ServiceMonitors), and handles all the configuration automatically. You never edit the master list by hand.

The Prometheus Operator is a Kubernetes controller that manages Prometheus deployments and their configuration through Custom Resource Definitions (CRDs). It was originally created by CoreOS and is the backbone of the popular kube-prometheus-stack Helm chart. The Operator watches for changes to its CRDs and keeps the actual state in sync with what you declared. The main CRDs are: Prometheus (defines a Prometheus instance), ServiceMonitor (defines which Kubernetes Services to scrape), PodMonitor (scrapes pods directly without needing a Service), PrometheusRule (defines recording and alerting rules), and Alertmanager (manages Alertmanager instances).

When you create a ServiceMonitor, the Operator reads its spec, which includes a selector (label matcher for Kubernetes Services), namespaceSelector (which namespaces to look in), and endpoints (port names, paths, and scrape intervals). The Operator then generates the equivalent Prometheus scrape_config YAML and loads it into the Prometheus pod. The important part is that the Prometheus CRD has a serviceMonitorSelector field that controls which ServiceMonitors it picks up -- if your ServiceMonitor labels do not match this selector, Prometheus will silently ignore it. The Operator also supports serviceMonitorNamespaceSelector to limit which namespaces are watched.

In production, the Prometheus Operator is the standard way to run Prometheus on Kubernetes. Teams create ServiceMonitors alongside their application deployments, often in the same Helm chart. This means the team deploying payments-api also defines how it gets monitored. The Operator supports advanced features like sharding (spreading targets across multiple Prometheus instances), Thanos sidecar integration for long-term storage, and automatic TLS for scrape endpoints. It also handles version upgrades, resource management, and persistent volumes for TSDB storage.

The most common gotcha is the label mismatch problem: you create a ServiceMonitor, but Prometheus never scrapes your targets. This happens because the Prometheus CRD's serviceMonitorSelector does not match your ServiceMonitor's labels. The kube-prometheus-stack Helm chart defaults to requiring release: kube-prometheus-stack as a label, so every ServiceMonitor must include it. Debug by running kubectl get prometheus -o yaml and checking the serviceMonitorSelector field. Another pitfall is port name mismatches -- the ServiceMonitor endpoints.port field must match the port name in the Kubernetes Service spec, not the container port number.