Mock scenario: data scientists complain the MLflow UI takes 30+ seconds to load experiments and searches time out. Diagnose and fix.
Quick Answer
It's almost always the backend store: millions of accumulated runs/metrics rows on an undersized or unindexed database, made worse by high-cardinality metric logging (per-step metrics for every batch) and no retention policy. Fix with DB right-sizing/indexes, metric-logging discipline, retention/archival, and splitting monster experiments.
Detailed Answer
Work the layers: (1) Database — check instance size, slow-query log, and table sizes; the metrics table is usually the monster (every log_metric call with step granularity is a row; a hyperparameter sweep logging per-batch loss adds millions). Verify indexes match search patterns (experiment_id, run status, start_time; attribute filters used by the UI). Upgrade MLflow — several releases carried search/UI query optimizations. (2) Logging discipline — log per-epoch not per-batch (or every Nth step), aggregate system metrics, and stop logging what nobody reads; autolog defaults deserve review since they can be chatty. (3) Retention — archive or delete runs from dead experiments (export to cold storage first if compliance needs them); mlflow gc permanently removes soft-deleted runs and their orphaned artifacts. (4) Structure — one experiment with 500K runs behaves worse than many scoped experiments; split by project/quarter. (5) Server — more gunicorn workers and connection pooling help concurrency but won't fix a slow query; measure before scaling the wrong layer. Prevent recurrence with a quarterly cleanup job and dashboards on runs/day and DB growth.
Code Example
-- find the pain SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 5; # retention: archive then purge mlflow experiments csv -x 42 > archive/exp42.csv mlflow gc --backend-store-uri $DB --experiment-ids 42
Interview Tip
Point at the metrics table and per-batch logging as the usual culprit before proposing infrastructure — 'log less, index right, retain deliberately' beats 'bigger database' as a first answer.