systemd Delete Edit

List all available type=services and their current status

systemctl list-units -t service --all

See which services a target pulls in. ( example the multi-user.target )

systemctl show -p "Wants" multi-user.target

Examine what gets started when when booted into a specific target. ( example the multi-user.target )

/usr/lib/systemd/systemd --test --system --unit=multi-user.target

Reloads /etc/fstab and creates units for remote fs

systemctl daemon-reload;systemctl restart remote-fs.target

Reloads /etc/fstab and creates units for local mounts

systemctl daemon-reload;systemctl restart local-fs.target

To find the systemd services which failed to start:

systemctl --state=failed

To identify slow boot and look for the jobs that are “running” those jobs are the ones where boot waits for completion on and the ones that listed as “waiting” will be executed only after those which are “running” are completed.

systemd-analyze blame
systemctl list-jobs

To visualize as SVG graphics:

systemd-analyze plot

To analyze the security of a service:

systemd-analyze security fubar.service

Troubleshooting

consider enabling SysRQ

Enable debug log

Enable debug boot log to both console and serial port, append to kernel boot:

systemd.log_level=debug systemd.log_target=console systemd.journald.forward_to_console=1 console=ttyS0,38400 console=tty1
# or
systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on

Booting into Rescue or Emergency Targets

To boot directly into rescue target add systemd.unit=rescue.target or just 1 to the kernel command line.

To boot directly into emergency shell add systemd.unit=emergency.target or emergency to the kernel command line. Note that in the emergency shell you will have to remount the root filesystem read-write by yourself before editing any files:

mount -o remount,rw /

Early Debug Shell

You can enable shell access to be available very early in the startup process to fall back on and diagnose systemd related boot up issues with various systemctl commands. Enable it using:

systemctl enable debug-shell.service

or by specifying on the kernel command line:

systemd.debug_shell=1

Diagnosing a service

Add a drop-in file for the service adding the two lines:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

Or equivalently, set the environment variable manually:

SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd

Shutdown Completes Eventually

If normal reboot or poweroff work, but take a suspiciously long time,

• then boot with the debug options: systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on enforcing=0
• save the following script as /usr/lib/systemd/system-shutdown/debug.sh and make it executable:

#!/bin/sh
mount -o remount,rw /
dmesg > /shutdown-log.txt
mount -o remount,ro /

• reboot

Look for timeouts logged in the resulting file shutdown-log.txt and/or attach it to a bugreport.

Shutdown Never Finishes

If normal reboot or poweroff never finish even after waiting a few minutes, the above method to create the shutdown log will not help and the log must be obtained using other methods. Two options that are useful for debugging boot problems can be used also for shutdown problems:

• use a serial console
• use a debug shell - not only is it available from early boot, it also stays active until late shutdown.

inspect what’s taking so long with the commands at the top of this document

Tips

Tip: If you find yourself in a situation where you cannot use systemctl to communicate with a running systemd (e.g. when setting this up from a different booted system), you can avoid communication with the manager by specifying –root=: systemctl --root=/ enable debug-shell.service

Other boot varibles

• systemd.confirm_spawn=
  • Takes a boolean argument. If true, asks for confirmation when spawning processes. Defaults to false.