How does systemd work — units, services, targets, and journalctl?

Think of systemd as an airport control tower managing everything needed to get the airport running. Each unit (service, mount, timer) is a department like baggage handling, fueling, or security. Targets are operational stages: 'pre-flight checks done' (multi-user.target), 'ready for passengers' (graphical.target). The control tower knows the dependencies between departments and starts them in parallel where possible, instead of calling each one sequentially like the old SysV init scripts did.

The basic building block in systemd is the unit, which is any resource systemd manages. There are twelve types: service (daemons), socket (for socket-based activation), target (grouping and sync points), mount (filesystem mounts), automount (mount on demand), timer (scheduled activation, like cron), path (file path monitoring), device (kernel devices), swap (swap space), slice (cgroup resource grouping), scope (externally started processes), and snapshot (saved states). Unit files live in /usr/lib/systemd/system/ (vendor defaults), /etc/systemd/system/ (admin overrides), and /run/systemd/system/ (runtime units). Each file has sections: [Unit] for description and dependencies (After=, Requires=, Wants=), [Service] for how to run it (ExecStart=, Restart=, Type=), and [Install] for boot enablement (WantedBy=).

Internally, systemd runs as PID 1 and builds a dependency graph of all units at boot. A clever feature is socket activation: a socket unit listens on a port, and when a connection comes in, systemd starts the matching service and hands it the connection. This eliminates startup ordering headaches because clients just connect and wait. The Type= directive tells systemd how to track when a service is ready: Type=simple means the main process is the service, Type=forking means it forks and the parent exits, Type=notify means the service sends a 'ready' signal via sd_notify(), and Type=oneshot is for scripts that run once and exit. systemd also puts each service in its own cgroup (system.slice/servicename.service), enabling per-service resource tracking and limits through directives like MemoryMax=, CPUQuota=, and IOWeight= right in the unit file.

In production, systemctl and journalctl are daily tools. Use systemctl start/stop/restart/reload for immediate control, enable/disable for boot persistence, status for a quick health check with recent logs, and list-units/list-unit-files for inventory. To override vendor unit files without editing them directly, use systemctl edit, which creates a drop-in file at /etc/systemd/system/servicename.service.d/override.conf. journalctl reads the binary journal maintained by systemd-journald and offers powerful filtering: journalctl -u payments-api.service for one service, -p err for error-level only, --since '1 hour ago' for time ranges, -f to follow like tail -f, and -o json for structured output. The journal automatically captures stdout/stderr, syslog messages, and kernel messages in one indexed store.

A critical gotcha is the difference between Requires= and Wants=. Requires= means if the dependency fails, your unit fails too. Wants= is a soft dependency that does not drag your unit down. Using Requires= carelessly can create cascading failures where a non-critical dependency takes down your main service. Another common mistake is forgetting to run systemctl daemon-reload after changing unit files; systemd caches the parsed config and will not see changes without it. Services with Restart=always will respawn forever after crashes, which can cause rapid restart loops burning CPU. Use StartLimitIntervalSec= and StartLimitBurst= to cap restarts (for example, 5 restarts within 60 seconds). Finally, journald defaults to storing logs in memory on some distros, meaning logs vanish on reboot unless you create /var/log/journal/ and set Storage=persistent in /etc/systemd/journald.conf.