architectsecurityStripe styleGitHub styleAtlassian styleShopify style

Stripe DevOps code question: How would you design a Python remediation script that can disable stale CI deploy keys across hundreds of repositories while supporting dry-run, approvals, audit logs, and safe rollback?

⚡

Quick Answer

Design the script as an idempotent workflow: discover keys, classify stale keys, produce a dry-run plan, require approval for destructive actions, execute with rate-limit-aware retries, and write append-only audit logs. Rollback should be based on captured key metadata and should be tested on a small allowlisted repository set before fleet-wide execution.

Detailed Answer

A remediation script is closer to a surgical checklist than a one-off script. The dangerous part is not the API call that deletes a key; it is deleting the wrong key quickly across hundreds of repositories. Interviewers ask this to see whether you understand automation blast radius, approval gates, and auditability.

The workflow should have clear phases. Discovery reads repository keys and last-used signals where the provider exposes them. Classification marks keys stale only when they match policy, such as no usage for 90 days, not owned by a protected integration, and not tagged as break-glass. Planning writes a human-readable and machine-readable diff. Approval requires a ticket, reviewer, or signed change request. Execution deletes or disables keys in small batches. Verification confirms the key is gone and that known deployment pipelines still authenticate.

In Python, design the code around pure functions and side-effect boundaries. A classify_key function should be easy to unit test with fake key metadata. A provider client should hide GitHub, GitLab, or Bitbucket API differences. The execution layer should handle 429 rate limits, 5xx retries, and permanent 403 permission errors differently. The script should accept --dry-run by default and require --execute plus a change ID for mutation.

At scale, rollback is not magic. If the API supports disabling instead of deleting, prefer disable first. If deletion is required, capture enough metadata to recreate the key: title, public key, repository, permissions, and owner. Never log private keys because they are not retrievable and should not exist in logs. Use append-only audit storage so security teams can answer who changed what and when.

The non-obvious gotcha is ownership. A key may look stale because usage telemetry is incomplete or because a rarely used disaster recovery pipeline only runs quarterly. Senior engineers define exception handling and communication before the deletion run, not after the outage.

from __future__ import annotations # Use modern Python typing. import argparse # Parse dry-run and execute flags explicitly. import json # Write machine-readable plans and audit records. from dataclasses import asdict, dataclass # Model remediation plans as data. from datetime import datetime, timezone # Use timezone-aware audit timestamps. @dataclass(frozen=True) # Immutable plan item for one deploy key. class KeyPlan: # Describe the intended action. repo: str # Repository name. key_id: str # Provider key identifier. title: str # Human-readable key title. action: str # Planned action such as disable or skip. reason: str # Policy reason for the action. PROTECTED_TITLES = {"prod-breakglass", "release-bot"} # Never mutate protected keys. def classify_key(repo: str, key: dict) -> KeyPlan: # Pure function that is easy to test. title = key.get("title", "") # Read key title safely. days_unused = int(key.get("days_unused", 0)) # Normalize usage age. if title in PROTECTED_TITLES: # Respect explicit protection policy. return KeyPlan(repo, key["id"], title, "skip", "protected title") # Skip protected key. if days_unused >= 90: # Apply stale-key policy. return KeyPlan(repo, key["id"], title, "disable", f"unused for {days_unused} days") # Plan remediation. return KeyPlan(repo, key["id"], title, "skip", "recently used") # Keep active keys. def write_audit(record: dict) -> None: # Append audit evidence. envelope = { # Wrap event with common metadata. "ts": datetime.now(timezone.utc).isoformat(), # Use UTC timestamp. "tool": "deploy-key-remediator", # Identify automation source. **record, # Include event-specific details. } # Envelope is JSON serializable. print(json.dumps(envelope, sort_keys=True)) # Send to stdout or log collector. def main() -> int: # Script entry point returns process status. parser = argparse.ArgumentParser() # Build CLI parser. parser.add_argument("--execute", action="store_true") # Require explicit mutation flag. parser.add_argument("--change-id", default="") # Tie changes to approval record. args = parser.parse_args() # Parse command line. if args.execute and not args.change_id: # Enforce approval evidence. raise SystemExit("--change-id is required with --execute") # Fail before mutation. sample_keys = [{"id": "123", "title": "old-preview-deploy", "days_unused": 120}] # Replace with provider API data. plan = [classify_key("payments-api", key) for key in sample_keys] # Build remediation plan. write_audit({"phase": "plan", "items": [asdict(item) for item in plan]}) # Record the plan. if not args.execute: # Dry-run is the default behavior. return 0 # Exit successfully without changes. for item in plan: # Execute approved actions. if item.action != "disable": # Only mutate planned disable actions. continue # Skip non-mutating items. write_audit({"phase": "execute", "repo": item.repo, "key_id": item.key_id, "change_id": args.change_id}) # Record mutation. return 0 # Return success. if __name__ == "__main__": # Run only when executed as a script. raise SystemExit(main()) # Convert return code to process exit.

Stripe DevOps code question: How would you design a Python remediation script that can disable stale CI deploy keys across hundreds of repositories while supporting dry-run, approvals, audit logs, and safe rollback?

More Python DevOps Automation questions