What's the difference between relabel_configs and metric_relabel_configs, and when would you use each?

relabel_configs operates in the service-discovery phase. Its inputs are the __meta_* labels that service discovery mechanisms attach (e.g. __meta_kubernetes_namespace, __meta_kubernetes_pod_label_app), plus the special __address__ and __scheme__ labels that determine where and how the scrape request is sent. Common uses: keep only pods with a specific annotation (action: keep), rewrite __address__ to scrape a non-default port, or attach a namespace/environment label onto every resulting series. Because this runs BEFORE the HTTP request, it can also prevent a scrape entirely (action: drop), which is the only way to avoid wasting a scrape on targets you don't want.

metric_relabel_configs operates AFTER the scrape response is parsed into samples. It sees real metric names and label values (e.g. an actual user_id="12345" label value), so it's the tool for cardinality control — dropping a metric entirely (action: drop matching on __name__), or dropping a specific high-cardinality label from a metric while keeping the rest of the metric intact (action: labeldrop). Because the scrape already happened, this doesn't save scrape bandwidth/CPU on the target, only storage and query cost in Prometheus itself.

The classic incident fix — "drop the user_id label that's exploding cardinality" — is done with metric_relabel_configs and action: labeldrop, since by the time you know a label is a problem, it's already in the scraped samples.