Python for DevOps: Zero to Hero

The automation language of DevOps — when a Bash script grows past ~100 lines, this is where it goes. You'll set up clean environments, script real ops tasks (files, processes, HTTP, JSON/YAML), call cloud and Kubernetes APIs, and package tools that teammates can install. Practical and hands-on. Work top to bottom, or jump to a part.

The journey:

1. Why Python for ops (and when not to)
2. Environments and dependencies done right
3. The scripting toolkit — files, subprocess, args
4. Talking to APIs — HTTP, JSON, YAML
5. Automating the cloud and Kubernetes
6. Error handling, logging, and structure
7. Testing your automation
8. Packaging and shipping a CLI
9. Production practices

---

Prerequisites

Confirm your version: python3 --version. On macOS/Linux python3 is standard; avoid the system Python for projects — use a virtual environment (Part 2).

---

Part 1 — Why Python for ops (and when not to)

Python is the default for automation because it has a huge standard library, first-class SDKs for every cloud and tool (AWS boto3, kubernetes, requests), and reads like pseudocode. Reach for it when:

• A shell script needs real data structures, error handling, or tests.
• You're calling APIs, parsing JSON/YAML, or transforming data.
• You're building a CLI teammates will install and reuse.

Stay in Bash for short glue (a few piped commands); reach for a compiled language (Go/Rust) when you need a single static binary or raw performance. Python's sweet spot is the large middle: readable automation that evolves.

---

Part 2 — Environments and dependencies done right

Never pip install into the system Python — you'll create version conflicts across projects. Use an isolated virtual environment:

python3 -m venv .venv          # create an isolated env
source .venv/bin/activate      # activate it (Windows: .venv\Scripts\activate)
pip install requests pyyaml    # installs only into .venv
pip freeze > requirements.txt  # pin what you installed

Modern projects declare dependencies in pyproject.toml and use faster tools like uv or pipx (for installing CLIs globally but isolated). The rule: dependencies live in the project, pinned and reproducible — not scattered in a shared interpreter.

---

Part 3 — The scripting toolkit

The standard library covers most ops needs. Files with pathlib, external commands with subprocess, and arguments with argparse:

#!/usr/bin/env python3
import argparse, subprocess
from pathlib import Path

def main():
    parser = argparse.ArgumentParser(description="Tidy old logs")
    parser.add_argument("dir", type=Path)
    parser.add_argument("--days", type=int, default=7)
    args = parser.parse_args()

    for log in args.dir.glob("*.log"):          # pathlib globbing
        print(f"found {log} ({log.stat().st_size} bytes)")

    # Run a shell command safely — a list, never a string, avoids injection
    result = subprocess.run(["df", "-h"], capture_output=True, text=True, check=True)
    print(result.stdout)

if __name__ == "__main__":
    main()

Key idioms: pathlib.Path for filesystem work (not string paths), subprocess.run([...], check=True) with a list of args (never shell=True on untrusted input — that's an injection hole), and argparse for a real CLI with --help.

---

Part 4 — Talking to APIs: HTTP, JSON, YAML

Most automation is calling an API and reshaping the response.

import requests, json, yaml

# Call a REST API with a timeout (always set one) and auth
resp = requests.get(
    "https://api.example.com/v1/services",
    headers={"Authorization": f"Bearer {token}"},
    timeout=10,
)
resp.raise_for_status()          # turn a 4xx/5xx into an exception
services = resp.json()

# Reshape and write YAML
config = {"services": [s["name"] for s in services if s["healthy"]]}
Path("services.yaml").write_text(yaml.safe_dump(config))

Always pass timeout= (a hung request without one blocks forever) and call raise_for_status() so failures are loud. Use yaml.safe_load/safe_dump (never plain yaml.load on untrusted input — it can execute code).

---

Part 5 — Automating the cloud and Kubernetes

The official SDKs turn infrastructure into a few lines:

# AWS with boto3 — list running EC2 instances
import boto3
ec2 = boto3.client("ec2", region_name="us-east-1")
for r in ec2.describe_instances()["Reservations"]:
    for i in r["Instances"]:
        if i["State"]["Name"] == "running":
            print(i["InstanceId"], i.get("PrivateIpAddress"))

# Kubernetes with the official client — list pods
from kubernetes import client, config
config.load_kube_config()        # or load_incluster_config() inside a pod
v1 = client.CoreV1Api()
for pod in v1.list_namespaced_pod("default").items:
    print(pod.metadata.name, pod.status.phase)

boto3 reads the same credential chain as the AWS CLI (env, profile, IAM role) — so on an EC2/EKS workload it uses the instance/pod role automatically, no keys needed. The Kubernetes client mirrors the API objects you already know from YAML.

---

Part 6 — Error handling, logging, and structure

Ops scripts fail in production; make failures clear and recoverable.

import logging, sys

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("deploy")

def deploy(env: str) -> None:
    try:
        log.info("deploying to %s", env)
        # ... work ...
    except requests.HTTPError as e:
        log.error("API call failed: %s", e)
        sys.exit(1)              # non-zero exit so CI knows it failed
    finally:
        log.info("cleanup done")

Use the logging module (not print) so you get levels, timestamps, and can route to stdout/files. Catch specific exceptions, not a bare except:. Exit non-zero on failure so CI and shell callers detect it. Type hints (env: str) document intent and catch bugs with a type checker (mypy/pyright).

---

Part 7 — Testing your automation

Automation that touches production deserves tests. pytest is the standard:

# code.py
def parse_cpu(s: str) -> int:
    """Convert '250m' or '1' to millicores."""
    return int(s[:-1]) if s.endswith("m") else int(s) * 1000

# test_code.py
from code import parse_cpu
def test_parse_cpu():
    assert parse_cpu("250m") == 250
    assert parse_cpu("2") == 2000

pip install pytest
pytest -q

Extract pure logic (parsing, transforms) into small functions and test those directly; mock the network/cloud calls (unittest.mock or responses) so tests are fast and don't hit real infrastructure. Wire pytest into CI so a broken script never merges.

---

Part 8 — Packaging and shipping a CLI

Turn a script into an installable tool with a pyproject.toml and an entry point:

# pyproject.toml
[project]
name = "opsctl"
version = "0.1.0"
dependencies = ["requests", "boto3"]

[project.scripts]
opsctl = "opsctl.cli:main"       # `opsctl` runs opsctl/cli.py:main

pipx install .          # install the CLI, isolated, on PATH
opsctl --help

Now teammates get a real command, not a script to copy around. For richer CLIs, libraries like Typer or Click add subcommands, help, and validation with far less boilerplate than raw argparse.

---

Part 9 — Production practices

• Always use a virtualenv and pin dependencies (requirements.txt or a lockfile).
• subprocess.run([...], check=True) with a list; avoid shell=True on untrusted input.
• Set timeout= on every network call and raise_for_status().
• logging, not print; exit non-zero on failure.
• yaml.safe_load / json, never yaml.load on untrusted data.
• Type-hint and run a type checker (mypy/pyright) in CI.
• Test pure logic with pytest; mock external calls.
• Package as a CLI (pyproject.toml + entry point) so tools are installable and versioned.

---

Common mistakes to avoid

• Installing into system Python. Version conflicts and broken OS tooling. Always a venv.
• shell=True with user input. Command injection. Pass a list of args instead.
• No request timeout. A hung API call blocks the script forever. Always timeout=.
• Bare except:. Hides real errors (including Ctrl-C). Catch specific exceptions.
• print instead of logging. No levels, no timestamps, hard to route in production.
• yaml.load on untrusted YAML. It can execute arbitrary code — use safe_load.
• Not exiting non-zero on failure. CI and shell callers think it succeeded.
• A 500-line script with no functions/tests. Refactor into testable units.

---

Practice: try it yourself

1. Real CLI. Write an argparse script that finds files over N MB under a directory and prints them, with --help working.
2. Safe subprocess. Call kubectl get pods -o json via subprocess.run, parse the JSON, and print pods not in Running.
3. API + reshape. Hit any public JSON API with requests (timeout + raise_for_status), transform the result, and write it as YAML.
4. Test it. Extract a parsing function, write a pytest test with two cases, and run pytest -q.
5. Package it. Add a pyproject.toml entry point and pipx install . your tool, then run it by name.

Self-check:

• Why a virtualenv instead of system Python?
• Why pass a list (not a string) to subprocess.run, and why avoid shell=True?
• What two things should every requests call include?
• When is Python the right tool vs Bash vs Go?

---

Where to go next

• Shell foundations — the Bash tutorial for the glue Python complements.
• Automate infra — pair with the Terraform and Kubernetes tutorials; boto3/the K8s client script what they declare.
• Practice with DevOps interview questions in the question bank.

You now have the loop: isolate the env → script the task → call APIs → handle errors → test → package. That's Python as DevOps teams actually use it.