A nightly billing-reconciler cron job didn't run last night and nobody was paged — how do you find out why, given cron fails silently by default?
Quick Answer
Check /var/log/cron, or /var/log/syslog, or journalctl -u cron depending on distro, first — cron itself logs every job it actually attempted to launch, so if there's no entry at all for the expected time, the crontab entry itself is missing, disabled, or has a schedule or timezone mismatch. If there IS a launch entry but the job failed, the actual error output went wherever cron's MAILTO was configured to send it, often an unmonitored local mailbox nobody reads, or was silently discarded entirely if the script's own output wasn't captured anywhere. The root cause is almost always one of: the crontab entry was accidentally removed during an unrelated edit, the script depends on environment variables or a PATH that only exists in an interactive shell and fails silently under cron's minimal environment, or the error output genuinely was generated but sent to an address nobody monitors.
Detailed Answer
Imagine a night-shift security guard whose job is to make rounds and note in a logbook started round at 2am every time they begin, but who was never told to write anything if something goes wrong mid-round. They just quietly stop and go back to the guard station without further explanation if they hit a problem. If you show up the next morning wondering why a door wasn't checked, the logbook will at least confirm whether the guard started that round at all. If they did start it, you then have to go find the guard directly to learn what actually went wrong, because the logbook itself won't tell you anything more than that they began.
This is fundamentally how cron's failure reporting was designed: cron's own log, via syslog, typically /var/log/cron or accessible through journalctl, faithfully records every time it launches a job, this is the started-round-at-2am entry, but cron was never designed to capture or interpret the job's actual success or failure. That's considered the job script's own responsibility, and cron's only built-in mechanism for surfacing errors is emailing whatever the script wrote to stdout and stderr to the address configured in MAILTO, or the crontab owner's local mail by default, which many modern systems don't even have a functioning mail transport configured for, meaning that email often goes absolutely nowhere at all.
The diagnostic sequence: first check cron's own log for whether the job was even launched at the expected time. A missing entry means the crontab entry itself has a problem, accidentally deleted during an unrelated crontab -e edit by someone else, a schedule field typo, or, commonly overlooked, a system timezone change or DST transition shifting when 2am actually falls, especially on systems using local time in crontabs rather than UTC. If the log shows the job WAS launched, the next step is checking whether output was actually captured anywhere — if the crontab entry didn't explicitly redirect stdout and stderr to a log file, the only record of what went wrong is whatever MAILTO mail delivery target exists, which needs to actually be checked, or better, forwarded to a monitored address or Slack webhook instead of a mailbox nobody opens.
In production, mature teams don't rely on manually checking /var/log/cron after the fact at all — they instrument each critical cron job to explicitly report its own success or failure to an external monitoring system, discussed further in dead-man's-switch style heartbeat monitoring, specifically because relying on humans to notice a missing overnight job the next morning is exactly the failure mode that just happened here. The immediate fix for this specific incident is redirecting the job's output to a captured log file and wiring MAILTO, or better, the script's own exit-code handling, to an actively monitored alert channel, not just a mailbox that sits unread for weeks.
The non-obvious gotcha: even after fixing logging and alerting, a very common recurring cause of the-job-silently-failed is that cron jobs run with a drastically minimal environment, often just PATH=/usr/bin:/bin and none of the environment variables or shell customizations present in an interactive login session. A script that works perfectly when an engineer runs it manually from their own shell can fail immediately under cron because it depends on a tool only found via a PATH entry set in .bashrc, or an environment variable normally exported by a login profile. Any cron job should be tested by explicitly running it with an empty, minimal environment, env -i /path/to/script.sh, rather than trusting that it works when I run it manually means anything about how it'll behave under cron's stripped-down execution context.