Linux Cron

Picture a train station where a train departs exactly on the printed schedule every 10 minutes, no matter what, even if the previous train hasn't actually finished unloading passengers on the platform yet. The schedule board doesn't check whether the last train finished its business; it only checks whether 10 minutes have passed since the last scheduled departure time. Normally this is fine because trains reliably finish in under 10 minutes. But the one day a train breaks down and takes 25 minutes to clear the platform, the next scheduled train arrives anyway, and now there are two trains trying to use the same platform at once, a collision the schedule itself never anticipated or prevented in any way.

Cron was built in the earliest days of Unix specifically as a simple, minute-resolution time-based trigger, a design intentionally kept dumb and stateless, checking only wall-clock time against a schedule, with zero awareness of whether a previously-launched job is still executing. This simplicity is exactly why cron has remained essentially unchanged and universally reliable for decades: there's very little to go wrong in the scheduling logic itself. But that same simplicity means cron offers no built-in protection against the previous-run-is-still-going scenario, unlike more modern schedulers that track job state explicitly as part of their design.

Internally, crond wakes up every 60 seconds, reads the parsed crontab entries, from per-user crontabs typically stored under /var/spool/cron/, the system-wide /etc/crontab, and drop-in files under /etc/cron.d/, and for each entry evaluates whether the current minute, hour, day-of-month, month, and day-of-week fields match. Any entry that matches gets forked as a brand-new child process running under the environment and permissions of whichever user owns that crontab. Critically, cron doesn't check a PID file, doesn't ask whether a previous instance of this command is already running, and doesn't wait; it just forks and moves on to evaluate the next matching entry in sequence.

In production, this becomes dangerous for jobs whose runtime is close to or sometimes exceeds their schedule interval. For example, a billing-reconciler job scheduled every 5 minutes that normally finishes in 2 minutes but occasionally takes 8 minutes under heavy load. During that one slow run, cron launches a second instance at the next 5-minute mark anyway, and now two instances are both reading and writing the same reconciliation state, which can produce duplicate charges, double-counted records, or lock contention against a shared database table that cascades into broader slowness for other unrelated jobs and queries hitting the same tables.

The gotcha most engineers miss until they've been burned by it: adding a simple pgrep-based check inside the job script feels like a fix but has a subtle race condition. Between the pgrep check and actually starting the real work, a nearly-simultaneous second cron-triggered invocation can pass the same check before the first instance's process name shows up in the process table, so both proceed anyway. The robust fix is flock, a proper kernel-level file lock, wrapping the job's execution, which atomically ensures only one instance can hold the lock at a time, with the second invocation either exiting immediately or waiting, rather than trusting a process-name check that has an inherent timing gap built into it.

Imagine a night-shift security guard whose job is to make rounds and note in a logbook started round at 2am every time they begin, but who was never told to write anything if something goes wrong mid-round. They just quietly stop and go back to the guard station without further explanation if they hit a problem. If you show up the next morning wondering why a door wasn't checked, the logbook will at least confirm whether the guard started that round at all. If they did start it, you then have to go find the guard directly to learn what actually went wrong, because the logbook itself won't tell you anything more than that they began.

This is fundamentally how cron's failure reporting was designed: cron's own log, via syslog, typically /var/log/cron or accessible through journalctl, faithfully records every time it launches a job, this is the started-round-at-2am entry, but cron was never designed to capture or interpret the job's actual success or failure. That's considered the job script's own responsibility, and cron's only built-in mechanism for surfacing errors is emailing whatever the script wrote to stdout and stderr to the address configured in MAILTO, or the crontab owner's local mail by default, which many modern systems don't even have a functioning mail transport configured for, meaning that email often goes absolutely nowhere at all.

The diagnostic sequence: first check cron's own log for whether the job was even launched at the expected time. A missing entry means the crontab entry itself has a problem, accidentally deleted during an unrelated crontab -e edit by someone else, a schedule field typo, or, commonly overlooked, a system timezone change or DST transition shifting when 2am actually falls, especially on systems using local time in crontabs rather than UTC. If the log shows the job WAS launched, the next step is checking whether output was actually captured anywhere — if the crontab entry didn't explicitly redirect stdout and stderr to a log file, the only record of what went wrong is whatever MAILTO mail delivery target exists, which needs to actually be checked, or better, forwarded to a monitored address or Slack webhook instead of a mailbox nobody opens.

In production, mature teams don't rely on manually checking /var/log/cron after the fact at all — they instrument each critical cron job to explicitly report its own success or failure to an external monitoring system, discussed further in dead-man's-switch style heartbeat monitoring, specifically because relying on humans to notice a missing overnight job the next morning is exactly the failure mode that just happened here. The immediate fix for this specific incident is redirecting the job's output to a captured log file and wiring MAILTO, or better, the script's own exit-code handling, to an actively monitored alert channel, not just a mailbox that sits unread for weeks.

The non-obvious gotcha: even after fixing logging and alerting, a very common recurring cause of the-job-silently-failed is that cron jobs run with a drastically minimal environment, often just PATH=/usr/bin:/bin and none of the environment variables or shell customizations present in an interactive login session. A script that works perfectly when an engineer runs it manually from their own shell can fail immediately under cron because it depends on a tool only found via a PATH entry set in .bashrc, or an environment variable normally exported by a login profile. Any cron job should be tested by explicitly running it with an empty, minimal environment, env -i /path/to/script.sh, rather than trusting that it works when I run it manually means anything about how it'll behave under cron's stripped-down execution context.

Think about how many home security systems handle a man-down scenario for someone living alone. Rather than waiting for the person to actively press a panic button when something goes wrong, which assumes they're conscious and able to act, the system requires the person to check in periodically, pressing a button once a day to confirm I'm fine, and if that routine check-in doesn't happen within an expected window, the system assumes something's wrong and alerts someone, without needing the person in trouble to do anything at all themselves.

This dead man's switch pattern, named after a railway or machinery safety mechanism that only stays in the safe state while someone actively holds it down, is exactly the right model for cron monitoring, because it solves the fundamental problem: cron cannot tell you a job failed if the failure mode is severe enough to prevent the job from even reaching its own error-reporting code, a hung process, a segfault, the entire host being down, the crontab entry silently deleted. Any monitoring approach that depends on the job itself successfully reporting failure has a blind spot exactly matching its own most catastrophic failure modes.

The implementation: each critical cron job is wrapped, or modified, to send an HTTP request to a unique monitoring URL at the start of execution and another at successful completion, commonly implemented by prefixing the actual command with a curl call. The monitoring service, self-hosted or SaaS, is configured with the job's expected schedule and a grace period, expect a successful ping every 24 hours, allow up to 2 hours of lateness before alerting, and it fires an alert specifically when an expected ping does NOT arrive in time. This means the alert triggers on absence of a signal, not presence of an error signal, which is what makes it robust against every failure mode, including the host being completely down and unable to send anything at all.

In production, teams typically also capture the job's actual exit code and include it in the completion ping, many of these services accept a status code, treating a non-zero exit as a distinct job-ran-but-failed signal versus job-never-even-started, giving two useful alert categories: job-didn't-run-or-complete-at-all, an infrastructure-level problem, versus job-ran-but-exited-with-an-error, an application-level problem, which route to different responders and severity levels. Dashboards typically show every monitored job's last successful check-in time and current status, giving an at-a-glance view across dozens of scheduled jobs that individual crontab inspection could never provide on its own.

The non-obvious gotcha: teams sometimes implement only a completion ping and skip the start ping, which seems like it saves an HTTP call but actually loses the ability to distinguish the job hung and never finished from the job never started at all — both look identical, no completion ping arrives, without a start ping to prove the job at least began. The start ping is also what lets you separately alert on job-has-been-running-suspiciously-long, started but no completion ping well past the job's normal runtime, a distinct and often earlier warning sign than waiting for the full scheduled-interval grace period to elapse before assuming failure.

Think about upgrading a factory from a wall clock that rings a bell at fixed times, with each department expected to independently know what to do when they hear their bell, to a proper shift-management system. The new system knows which shifts depend on which other shifts finishing first, refuses to start a new shift on a machine that's still actively running the previous one, keeps a detailed log of exactly when each shift started, finished, and whether it hit any problems, and can cap how many resources any single shift is allowed to consume. The bell-based system technically worked for decades, but every one of those improvements requires the department itself to build custom logic; the shift-management system provides it all as a built-in platform feature instead.

Systemd timers were introduced specifically to bring init-system-level features to job scheduling, addressing exactly the gaps that make raw cron risky at scale: no native overlap protection, no structured queryable logs, cron output goes to whatever ad-hoc redirection or MAILTO you configured, if anything at all, no dependency awareness, cron can't express don't-run-this-backup-job-until-the-database-mount-is-confirmed-ready, and no resource governance, a runaway cron job can consume unbounded CPU and memory with nothing else in the system aware of or limiting it. Because systemd already manages every other service on a modern Linux host, extending it to also manage scheduled jobs means scheduled jobs get the same operational tooling, journald logging, systemctl status, cgroup resource limits, that every other service already benefits from.

A safe migration converts one job at a time. For a job like order-fulfillment-sync, you create /etc/systemd/system/order-fulfillment-sync.service, defining ExecStart, the User= to run as, and any Environment= variables previously relied upon implicitly from a login shell, and a paired .timer unit with OnCalendar= expressing the schedule, which supports cron-like patterns but also richer expressions. Then systemctl enable --now order-fulfillment-sync.timer while leaving the old crontab entry commented out, not deleted, as a documented rollback path. You let both the new timer and the disabled-but-present old crontab entry sit for at least one full schedule cycle, confirming via journalctl -u order-fulfillment-sync.service that it ran successfully and on time, before removing the old crontab entry entirely.

In production, once migrated, Type=oneshot combined with default systemd behavior means a new timer-triggered start is simply skipped, not queued, not force-started, if the service unit is still marked active from a previous run, solving the classic cron overlap problem without any custom flock wrapper needed. Dependency ordering, After= and Requires=, lets you correctly sequence something like a reconciliation job that must not start until a database backup unit has completed, which cron has no native way to express at all. Resource limits, MemoryMax= and CPUQuota= in the service unit, mean a single runaway scheduled job can be capped from starving the rest of the host, which raw cron-launched processes never had unless manually wrapped in cgroup tooling separately by hand.

The non-obvious gotcha: OnCalendar scheduling in systemd timers is NOT minute-by-minute polling like cron. Systemd calculates the next trigger time analytically and can, by default, add a small randomized delay, RandomizedDelaySec, or align timers to reduce simultaneous wake-ups across many services, AccuracySec, default 1 minute of slack, meaning a timer's actual fire time can be a little later than the exact OnCalendar value unless you explicitly set AccuracySec=1s for jobs where precise timing genuinely matters. Teams migrating from cron's exact-minute behavior are frequently surprised by this when a job that always ran at exactly 2:00:00 under cron starts firing at 2:00:47 under a naively-configured timer instead.