Authoring Systemd Service Units: Unit Files, Dependencies & systemctl

Modern Linux distributions—including Ubuntu, Debian, Red Hat Enterprise Linux (RHEL), AlmaLinux, Fedora, and openSUSE—utilize systemd as their default system and service manager. Replacing legacy SysVinit scripts, systemd provides parallelized service booting, aggressive dependency resolution, process tracking via Linux Control Groups (cgroups), and centralized logging. For systems engineers, authoring custom systemd service units is mandatory for production workload deployment.

Systemd Unit File Architecture & Filesystem Hierarchy

A systemd unit is a declarative configuration file ending in .service (or .target, .timer, .socket). Understanding where unit files reside and how systemd resolves conflicting definitions is fundamental to system administration.

Unit file locations are structured by administrative authority and evaluated in strict order of precedence:

1. /etc/systemd/system/ (Highest Precedence): Reserved for system administrators. Administrator-created units, local custom services, and unit drop-in configuration overrides (e.g. service.d/*.conf) should normally be placed here. Files located here supersede unit files of the same name in all other directories, ensuring local customizations are never overwritten during operating system upgrades.
2. /run/systemd/system/ (Runtime Precedence): Transient runtime units generated dynamically during execution by systemd generators (e.g. mount units or session slices). These exist purely in volatile memory and do not survive system reboots.
3. Vendor and Distribution Package Directories (Lowest Precedence): Default units supplied by upstream software packages and distribution maintainers (installed via apt, dnf, pacman, or zypper). Importantly, the exact filesystem path varies across Linux distributions:
   • /usr/lib/systemd/system/: The primary location for packaged units in modern distributions that utilize a merged-/usr layout (such as RHEL 8/9, Fedora, Arch Linux, and modern Debian/Ubuntu releases).
   • /lib/systemd/system/: Commonly used in traditional Debian/Ubuntu systems (or present as a compatibility symlink pointing to /usr/lib/systemd/system/ in merged-/usr systems).
   Administrators should never directly edit vendor units in /usr/lib/systemd/system/ or /lib/systemd/system/, as package manager updates will overwrite any modifications.

The Anatomy of a Production .service Unit

A standard service unit is divided into three core sections: [Unit], [Service], and [Install]. Below is an enterprise-grade service definition for an internal Go/Node/Python API service:

[Unit]
Description=Enterprise API Background Worker Daemon
Documentation=https://themjtechhub.site/docs
After=network-online.target remote-fs.target
Wants=network-online.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/api-worker
ExecStart=/opt/api-worker/bin/worker --config=/etc/api-worker/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s

# Security Hardening & Sandboxing
ProtectSystem=full
ProtectHome=true
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Section Deep Dive & Key Directives

1. The [Unit] Section: Dependencies & Ordering

• After= vs Requires=:
  • After=network-online.target dictates ordering only. It guarantees that if both units are queued to start, systemd starts network-online before starting this service. It does not force the network to start if it wasn't requested.
  • Requires= creates a strict operational dependency: if the required unit fails or terminates, this service immediately terminates as well.
  • Wants= establishes a soft dependency: systemd attempts to activate the wanted unit, but failure of that unit will not abort this service.

2. The [Service] Section: Process Lifecycle

• Type=:
  • simple (Default): systemd assumes the service starts immediately upon spawning the process specified in ExecStart. Standard for modern web servers and background workers that do not daemonize.
  • forking: Used for traditional UNIX daemons that call fork() and exit the parent process. Requires PIDFile= so systemd tracks the child daemon.
  • oneshot: Used for scripts that execute tasks and exit immediately. Useful in combination with systemd timers.
  • notify: The daemon sends a notification signal via sd_notify() to systemd when initialization is complete.
• User= & Group=: Drops root privileges, running the executable under a dedicated non-privileged service account.
• Restart=on-failure: Automatically relaunches the daemon if the process exits with a non-zero exit code, unhandled exception, or fatal signal.

3. The [Install] Section: Boot Activation

• WantedBy=multi-user.target: Specifies that enabling this service links it to runlevel equivalent multi-user.target (standard non-graphical multi-user mode). When the server boots into multi-user target, this service launches automatically.

Controlling Services with systemctl

After creating or editing any unit file, you must instruct the systemd manager daemon to reload its configuration cache:

# 1. Reload systemd daemon to pick up new unit files
sudo systemctl daemon-reload

# 2. Enable service to start at system boot
sudo systemctl enable api-worker.service

# 3. Start the service immediately
sudo systemctl start api-worker.service

# 4. Inspect real-time status, PID, memory consumption, and recent logs
sudo systemctl status api-worker.service

● api-worker.service - Enterprise API Background Worker Daemon
     Loaded: loaded (/etc/systemd/system/api-worker.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2026-09-09 10:15:30 UTC; 15min ago
   Main PID: 48210 (worker)
      Tasks: 8 (limit: 4915)
     Memory: 42.4M
        CPU: 1.250s
     CGroup: /system.slice/api-worker.service
             └─48210 /opt/api-worker/bin/worker --config=/etc/api-worker/config.yaml

Troubleshooting with journalctl

Because systemd intercepts standard output (stdout) and standard error (stderr) streams from managed processes, application logs are aggregated directly into the systemd journal:

# Follow live logs in real time (-f) for a specific service unit (-u)
sudo journalctl -u api-worker.service -f

# View errors only (-p err) recorded since the last system boot (-b)
sudo journalctl -u api-worker.service -b -p err --no-pager

Summary

Authoring native systemd unit files enables administrators to deploy resilient, self-healing services with granular security isolation and unified journal logging. By understanding unit file sections, choosing appropriate service types, and applying daemon-reload workflows, systems engineers master modern Linux application orchestration.