How do you build reliable 'did my cron job actually run and succeed' monitoring, given cron has no built-in success or failure tracking?
Quick Answer
Wrap each cron job so it actively checks in with an external monitoring endpoint both when it starts and when it completes successfully, a dead man's switch or heartbeat pattern, using a service like Healthchecks.io, Cronitor, or a self-hosted equivalent, and have that external service alert if the expected check-in doesn't arrive within a grace period past the scheduled time. This inverts the monitoring responsibility from cron telling you it failed, which it can't do reliably, to an outside system noticing you went silent, which catches every failure mode including the job not running at all, hanging forever, or crashing before it could report its own error.
Detailed Answer
Think about how many home security systems handle a man-down scenario for someone living alone. Rather than waiting for the person to actively press a panic button when something goes wrong, which assumes they're conscious and able to act, the system requires the person to check in periodically, pressing a button once a day to confirm I'm fine, and if that routine check-in doesn't happen within an expected window, the system assumes something's wrong and alerts someone, without needing the person in trouble to do anything at all themselves.
This dead man's switch pattern, named after a railway or machinery safety mechanism that only stays in the safe state while someone actively holds it down, is exactly the right model for cron monitoring, because it solves the fundamental problem: cron cannot tell you a job failed if the failure mode is severe enough to prevent the job from even reaching its own error-reporting code, a hung process, a segfault, the entire host being down, the crontab entry silently deleted. Any monitoring approach that depends on the job itself successfully reporting failure has a blind spot exactly matching its own most catastrophic failure modes.
The implementation: each critical cron job is wrapped, or modified, to send an HTTP request to a unique monitoring URL at the start of execution and another at successful completion, commonly implemented by prefixing the actual command with a curl call. The monitoring service, self-hosted or SaaS, is configured with the job's expected schedule and a grace period, expect a successful ping every 24 hours, allow up to 2 hours of lateness before alerting, and it fires an alert specifically when an expected ping does NOT arrive in time. This means the alert triggers on absence of a signal, not presence of an error signal, which is what makes it robust against every failure mode, including the host being completely down and unable to send anything at all.
In production, teams typically also capture the job's actual exit code and include it in the completion ping, many of these services accept a status code, treating a non-zero exit as a distinct job-ran-but-failed signal versus job-never-even-started, giving two useful alert categories: job-didn't-run-or-complete-at-all, an infrastructure-level problem, versus job-ran-but-exited-with-an-error, an application-level problem, which route to different responders and severity levels. Dashboards typically show every monitored job's last successful check-in time and current status, giving an at-a-glance view across dozens of scheduled jobs that individual crontab inspection could never provide on its own.
The non-obvious gotcha: teams sometimes implement only a completion ping and skip the start ping, which seems like it saves an HTTP call but actually loses the ability to distinguish the job hung and never finished from the job never started at all — both look identical, no completion ping arrives, without a start ping to prove the job at least began. The start ping is also what lets you separately alert on job-has-been-running-suspiciously-long, started but no completion ping well past the job's normal runtime, a distinct and often earlier warning sign than waiting for the full scheduled-interval grace period to elapse before assuming failure.