Cloud Custodian: Zero to Hero

Governance-as-code for the cloud — write YAML policies that find and fix non-compliant resources (untagged instances, public S3 buckets, idle volumes, unencrypted databases) across AWS, Azure, and GCP. You'll install it, write filter/action policies, run them in dry-run, automate them with events and schedules, and adopt guardrail patterns. Hands-on throughout.

The journey:

1. What Cloud Custodian solves
2. Install and your first policy
3. The policy model — resource, filters, actions
4. Filters — finding the resources that matter
5. Actions — remediation
6. Dry-run and safe rollout
7. Run modes — pull, event, periodic
8. Real guardrails (tags, cost, security)
9. Production practices

---

Prerequisites

Use a read-only role first and always --dryrun. Confirm the AWS basics in the AWS tutorial. Install: pip install c7n (add c7n-org for multi-account).

---

Part 1 — What Cloud Custodian solves

Cloud accounts drift out of compliance constantly: someone launches an untagged instance, opens a security group to the world, leaves a volume unattached burning money, or creates an unencrypted bucket. Clicking through the console to find and fix these doesn't scale.

Cloud Custodian ("c7n") turns governance into declarative YAML policies that run against your cloud API: find resources matching these filters, then take these actions. One tool, one language, across AWS/Azure/GCP — for tag compliance, cost control, security remediation, and cleanup. It's the automated policy enforcement layer for your cloud, run on a schedule or on events.

---

Part 2 — Install and your first policy

python3 -m venv .venv && source .venv/bin/activate
pip install c7n

A policy is YAML. This one finds EC2 instances missing an Owner tag (read-only — no action yet):

# policy.yml
policies:
  - name: ec2-missing-owner-tag
    resource: aws.ec2
    filters:
      - "tag:Owner": absent

custodian run --dryrun -s output policy.yml     # never mutates in dry-run
custodian report -s output policy.yml           # tabular view of matches

custodian run queries the account; -s output writes results and logs; report shows what matched. Start every policy this way — see what it selects before adding actions.

---

Part 3 — The policy model — resource, filters, actions

Every policy is three things:

policies:
  - name: stop-untagged-instances
    resource: aws.ec2          # 1. WHAT kind of resource
    filters:                   # 2. WHICH ones (AND-ed)
      - "tag:Owner": absent
      - "State.Name": running
    actions:                   # 3. WHAT to do to them
      - type: stop

• resource — the cloud resource type (aws.ec2, aws.s3, aws.rds, azure.vm, gcp.instance…). Hundreds are supported.
• filters — predicates that narrow the set (all must match).
• actions — what to do to the matches (stop, tag, delete, notify, encrypt…).

The mental model is exactly find these, do that. Filters without actions = an audit report; add actions and it becomes enforcement.

---

Part 4 — Filters — finding the resources that matter

Filters are the expressive core. Beyond simple tag/attribute checks:

filters:
  - or:                                   # boolean logic
      - "tag:Environment": absent
      - "tag:Owner": absent
  - type: value                           # arbitrary attribute comparison
    key: LaunchTime
    op: less-than
    value_type: age
    value: 90                             # older than 90 days
  - type: metrics                         # CloudWatch metrics
    name: CPUUtilization
    statistics: Average
    period: 86400
    value: 5
    op: less-than                         # avg CPU < 5% (idle)

Common filter types: value (any attribute with ops like less-than, regex, in), metrics (CloudWatch — find idle resources), age, security-group, image-age, and cross-resource filters. This is how you find "public buckets," "unencrypted RDS," or "instances idle for a week."

---

Part 5 — Actions — remediation

Actions turn findings into fixes:

policies:
  - name: tag-compliance-mark-and-stop
    resource: aws.ec2
    filters:
      - "tag:Owner": absent
    actions:
      - type: mark-for-op                 # tag it now, act later (grace period)
        op: stop
        days: 4
      - type: notify                      # tell someone
        to: [[email protected]]
        transport: { type: sns, topic: arn:aws:sns:...:governance }

Powerful actions include stop/terminate, tag/remove-tag, set-encryption, remove-statements (fix a bucket policy), and mark-for-op — a two-phase pattern that tags a resource with a deadline, notifies the owner, and only remediates if it's still non-compliant after the grace period. That grace period is what makes automated remediation safe.

---

Part 6 — Dry-run and safe rollout

Remediation on the cloud is irreversible — roll out like any enforcement:

1. --dryrun always first — it reports matches and intended actions without touching anything.
2. Read the report, confirm the filters select exactly what you expect (no surprises).
3. Start with soft actions — tag/notify/mark-for-op before stop/delete.
4. Scope tightly — one region, one account, a test tag — then widen.

custodian run --dryrun -s out policy.yml && custodian report -s out policy.yml

The classic disaster is a too-broad filter with a terminate action run for real — dry-run and soft actions exist precisely to prevent it.

---

Part 7 — Run modes — pull, event, periodic

A policy's mode decides when it runs:

• pull (default) — run on demand from your CLI/CI.
• periodic — deploy as a Lambda on a cron schedule (schedule: "rate(1 day)").
• cloudtrail — event-driven: react the instant a resource is created (e.g. tag or reject a new public bucket in real time).

policies:
  - name: block-public-bucket
    resource: aws.s3
    mode:
      type: cloudtrail
      events: [{ source: s3.amazonaws.com, event: PutBucketAcl, ids: "requestParameters.bucketName" }]
    filters:
      - type: global-grants          # bucket granted to "everyone"
    actions:
      - type: set-permissions
        remove: [Read, ReadAcp]

Event mode is real-time guardrails; periodic mode is scheduled sweeps; pull mode is ad-hoc audits and CI checks. Custodian deploys the Lambda + event wiring for you.

---

Part 8 — Real guardrails (tags, cost, security)

The three workhorse use cases:

• Tag governance — require Owner/Environment/CostCenter; mark-for-op and notify, then stop. Makes cost allocation and ownership real.
• Cost control — find idle instances (low CPU metrics), unattached EBS volumes, old snapshots, and unused Elastic IPs; stop/delete on a schedule.
• Security remediation — public S3 buckets, security groups open to 0.0.0.0/0, unencrypted RDS/EBS, and disabled logging; fix or alert in real time via cloudtrail mode.

These encode the guardrails from the AWS tutorial — private-by-default, least privilege, tagged, cost-aware — as continuously-enforced policy instead of hope.

---

Part 9 — Production practices

• --dryrun first, every time, and read the report before enabling actions.
• Soft actions before hard ones — mark-for-op/notify with a grace period, then stop/delete.
• Least-privilege IAM for Custodian — only the read/act permissions its policies need.
• Version policies in Git, review them, and run them in CI against a sandbox.
• c7n-org to run across many accounts/regions from one place.
• cloudtrail mode for real-time security guardrails; periodic for cost sweeps.
• Notify owners and give a grace window before remediating — governance with humans in the loop.
• Monitor the Lambdas (errors, invocations) — a broken policy silently stops enforcing.

---

Common mistakes to avoid

• Running with actions before dry-run. A broad filter + terminate for real is a self-inflicted outage.
• Too-broad filters. Always confirm the exact match set in report first.
• Hard actions with no grace period. Use mark-for-op + notify so owners can react.
• Over-privileged Custodian role. Scope IAM to exactly what the policies do.
• Policies not in Git/CI. Governance code deserves review and testing like any code.
• Ignoring Lambda failures. A silently-failing periodic policy means no enforcement.
• Learning remediation on production. Start in a sandbox account.

---

Practice: try it yourself

1. Audit. Write a policy that finds EC2 instances missing an Owner tag; run --dryrun and report.
2. Find idle. Add a metrics filter for average CPU < 5% over a day to find idle instances.
3. Safe remediation. Add mark-for-op (stop after 4 days) + notify; confirm the tag appears (in a sandbox).
4. Real-time. Write a cloudtrail-mode policy that flags a newly-public S3 bucket.
5. Cost sweep. Write a policy that finds unattached EBS volumes older than 30 days.

Self-check:

• What are the three parts of every Custodian policy?
• Why is mark-for-op safer than a direct stop/terminate?
• What do pull, periodic, and cloudtrail modes each do?
• Which two safety habits prevent a remediation disaster?

---

Where to go next

• The cloud fundamentals it enforces — the AWS tutorial (tags, least privilege, private-by-default).
• Analyze IAM before you grant it with the IAM Policy Analyzer.
• Practice with cloud/security interview questions in the question bank.

You now have the loop: resource → filters → actions → dry-run → soft-then-hard → automate with modes. That's cloud governance as continuously-enforced code.