How do you deploy an MLflow tracking server for a multi-team production environment, and what breaks with the quickstart setup?
Quick Answer
Production: PostgreSQL/MySQL backend store, object storage (S3/GCS/Azure Blob) for artifacts, stateless tracking server replicas behind a load balancer, and auth in front. The quickstart (local files + SQLite) breaks under concurrency — SQLite corrupts with parallel writers, local artifacts don't scale or survive, and there's no access control.
Detailed Answer
The tracking server has two storage planes: the backend store (runs, params, metrics, registry state — relational) and the artifact store (models, plots, datasets — blobs). Production shape: Postgres with connection pooling and regular backups for the backend; S3-compatible object storage for artifacts, ideally with clients writing artifacts directly to object storage (or via the server proxy when you need a single egress/auth point — mind the bandwidth). The server itself is stateless, so run 2+ replicas behind a load balancer with health checks; put SSO/OIDC auth in front (MLflow's built-in auth is basic — most orgs front it with a proxy or use a managed offering for real RBAC). What breaks on quickstart setups, in the order teams hit it: SQLite corruption once parallel training jobs write concurrently; artifact loss because they lived on one VM's disk; 'works from my laptop' URI confusion (file paths logged that other machines can't read); no auth, so anyone can delete the registry; and slow UI queries once runs reach the hundreds of thousands without DB indexes/retention.
Code Example
mlflow server \ --backend-store-uri postgresql://mlflow:***@pg:5432/mlflow \ --artifacts-destination s3://ml-artifacts/mlflow \ --host 0.0.0.0 --port 5000 --workers 4 # clients export MLFLOW_TRACKING_URI=https://mlflow.internal.example.com
Interview Tip
Name the two storage planes and their different scaling problems — relational metadata vs blob artifacts. The SQLite-corruption-under-concurrency detail signals you've seen the quickstart fail for real.