Kubeflow: Zero to Hero

The machine-learning platform for Kubernetes — turn ad-hoc notebooks into reproducible, scalable ML pipelines with experiment tracking, hyperparameter tuning, and model serving. You'll understand the components, build a pipeline, run distributed training and tuning, and serve a model. Concepts-and-hands-on. Work top to bottom, or jump to a part.

The journey:

1. What Kubeflow is and the MLOps problem
2. The component map
3. Notebooks and the dev loop
4. Pipelines — reproducible ML workflows
5. Distributed training operators
6. Katib — hyperparameter tuning
7. KServe — model serving
8. Fitting into MLOps
9. Production practices

---

Prerequisites

Kubeflow is a platform, not a single tool — expect a bigger footprint than most tutorials here. Confirm your cluster: kubectl get nodes.

---

Part 1 — What Kubeflow is and the MLOps problem

Machine learning in notebooks doesn't reproduce, doesn't scale, and doesn't ship: "works on my laptop" data science, manual retraining, no experiment history, and a painful gap from a trained model to a running endpoint.

Kubeflow makes the ML lifecycle Kubernetes-native and reproducible: every step (data prep, training, tuning, serving) runs as containers on the cluster, defined as code, tracked, and scalable — using the same primitives (pods, CRDs, autoscaling) you already know. It's the MLOps toolkit for teams that outgrew notebooks and want the ML workflow to be as engineered as the rest of their infrastructure.

---

Part 2 — The component map

Kubeflow is a suite; you adopt the pieces you need:

The through-line: each turns an ML task into a Kubernetes resource. You'll rarely use all of them at once — most teams start with Notebooks + Pipelines and add Katib/serving as they mature.

---

Part 3 — Notebooks and the dev loop

Kubeflow Notebooks give each user a managed Jupyter server as a pod — with the CPU/GPU/memory they request, shared storage, and access to cluster services. It's where experimentation starts, but with cluster resources instead of a laptop.

The key discipline: a notebook is for exploration, not production. As soon as a step works, factor it into a pipeline component (Part 4) so it's reproducible, versioned, and runnable without a human clicking "Run All." Notebooks that quietly become production jobs are the anti-pattern Kubeflow exists to fix.

---

Part 4 — Pipelines — reproducible ML workflows

Kubeflow Pipelines (KFP) is the core. You define an ML workflow as a DAG of containerized steps in Python; each step's inputs/outputs are tracked, so runs are reproducible and comparable.

from kfp import dsl

@dsl.component(base_image="python:3.11")
def preprocess(data_path: str) -> str:
    # ... read, clean, write features ...
    return "/data/features.parquet"

@dsl.component(base_image="python:3.11")
def train(features: str) -> str:
    # ... train, save model ...
    return "/models/model.pkl"

@dsl.pipeline(name="train-pipeline")
def pipeline(data_path: str = "/data/raw.csv"):
    feats = preprocess(data_path=data_path)
    train(features=feats.output)          # output wiring defines the DAG edge

Each @dsl.component becomes a container that runs as a pod; wiring one step's .output into the next defines the dependency graph. The KFP UI then shows every run, its parameters, artifacts, and metrics — so "which data + hyperparameters produced this model?" is always answerable. That lineage is the whole point.

---

Part 5 — Distributed training operators

When a model won't train on one machine, the Training Operators run distributed jobs as CRDs — e.g. a PyTorchJob coordinates a master and workers:

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata: { name: train-resnet }
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template: { spec: { containers: [{ name: pytorch, image: my-train:1.0, resources: { limits: { nvidia.com/gpu: 1 } } }] } }
    Worker:
      replicas: 3
      template: { spec: { containers: [{ name: pytorch, image: my-train:1.0, resources: { limits: { nvidia.com/gpu: 1 } } }] } }

The operator sets up the master/worker topology, environment variables, and networking that the distributed framework expects — you just declare replicas and GPUs. The same pattern exists for TensorFlow (TFJob), XGBoost, and MPI. It turns "configure a distributed training cluster" into a YAML manifest.

---

Part 6 — Katib — hyperparameter tuning

Katib automates the search for good hyperparameters (learning rate, layers, batch size) — running many trials and steering toward the best result:

apiVersion: kubeflow.org/v1beta1
kind: Experiment
metadata: { name: tune-lr }
spec:
  objective: { type: maximize, objectiveMetricName: accuracy }
  algorithm: { algorithmName: bayesianoptimization }
  parameters:
    - name: lr
      parameterType: double
      feasibleSpace: { min: "0.001", max: "0.1" }
  trialTemplate: { ... }          # how to run one trial

Katib launches trials (each a training run) with different parameter values, reads back the objective metric, and uses a search algorithm (grid, random, Bayesian, Hyperband) to explore efficiently — instead of hand-tuning one run at a time. It's AutoML/HPO as a Kubernetes CRD.

---

Part 7 — KServe — model serving

Training produces a model; KServe serves it as a scalable, production endpoint with one manifest:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata: { name: sklearn-iris }
spec:
  predictor:
    model:
      modelFormat: { name: sklearn }
      storageUri: "s3://models/iris/"

KServe handles the hard serving parts: autoscaling (including scale-to-zero), canary rollouts between model versions, batching, and a standard inference protocol — across frameworks (sklearn, PyTorch, TensorFlow, ONNX, LLMs). Point it at a stored model and you get an autoscaling HTTP/gRPC endpoint, closing the gap from "trained" to "in production."

---

Part 8 — Fitting into MLOps

Kubeflow is the compute/orchestration layer of MLOps, not the whole thing. A mature stack pairs it with:

• A feature store and data versioning for reproducible inputs.
• A model registry (MLflow) for versioned, promotable models.
• CI/CD (the GitHub Actions/Argo CD tutorials) to trigger pipelines and promote models via GitOps.
• Monitoring (the Prometheus/OpenTelemetry stack) for latency and model drift.

The goal is the ML equivalent of a software delivery pipeline: data → automated training/tuning → tracked model → gated deploy → monitored serving → retrain. Kubeflow provides the Kubernetes-native engine for the training-through-serving middle.

---

Part 9 — Production practices

• Graduate notebooks into pipelines quickly — reproducibility beats "Run All".
• Version everything — data, code, model, and pipeline — so any run is reproducible.
• Track experiments (KFP metadata / MLflow) — never lose "what produced this model".
• Request GPUs and resources explicitly; use node pools/taints for expensive hardware.
• Serve with KServe for autoscaling + canary, not a hand-rolled Flask app.
• Automate via GitOps/CI — trigger pipelines and promote models through review.
• Monitor serving and drift — accuracy degrades as data shifts; alert on it.
• Mind the footprint — Kubeflow is heavy; right-size and consider managed offerings.

---

Common mistakes to avoid

• Notebooks as production. They don't reproduce or scale — factor steps into pipelines.
• No versioning of data/models. You can't reproduce or roll back a model you can't identify.
• Hand-tuning hyperparameters. Katib searches far more efficiently than a human.
• Rolling your own model server. KServe gives autoscaling, canary, and batching for free.
• Under-provisioning / no GPU scheduling. Training jobs starve or fail without explicit resources.
• Treating Kubeflow as all of MLOps. Pair it with a registry, feature store, CI/CD, and monitoring.
• Ignoring model drift. A model that was accurate degrades silently as data changes.

---

Practice: try it yourself

1. Map it. In your cluster (or a Kind-based Kubeflow), identify the running components — Pipelines, Katib, Training Operator, Notebooks.
2. Build a pipeline. Write a two-step KFP pipeline (preprocess → train) with the SDK and run it; inspect the DAG and artifacts in the UI.
3. Distributed job. Submit a small PyTorchJob with 1 master + 2 workers and watch the pods coordinate.
4. Tune. Define a Katib Experiment over one hyperparameter and compare trial results.
5. Serve. Deploy a stored model with a KServe InferenceService and curl the endpoint; scale it to zero and back.

Self-check:

• Why are notebooks the wrong home for production ML, and what replaces them?
• What does a KFP pipeline give you that a script doesn't?
• What do the Training Operators and Katib each automate?
• What serving concerns does KServe handle for you?

---

Where to go next

• Orchestrate the plumbing — the Apache Airflow tutorial for data pipelines that feed ML.
• Ship models via GitOps — the Argo CD and Kubernetes tutorials.
• Practice with MLOps/Kubernetes interview questions in the question bank.

You now have the map: notebooks → pipelines → distributed training → tuning → serving → MLOps loop. That's ML engineered on Kubernetes.