Your first dive into systemd!

Your first dive into systemd!
1
ver1.5 Etsuji Nakai
Twitter @enakai00
Open Cloud Campus
Your first dive
into systemd!

Open Cloud Campus
2
$ who am i
– The author of “Professional Linux Systems” series.
• Available only in Japanese. Translation offering from publishers are welcomed ;-)
Self-study Linux
Deploy and Manage by yourself
Professional Linux Systems
Deployment and Management
Network Management
 Etsuji Nakai
– Senior solution architect and cloud
evangelist at Red Hat.
Technology for Next Decade

Open Cloud Campus
3
Contents
 Review: SysVinit & Upstart
 System boot process with systemd
 Operating systemd
 Log management with journald
 Unit configuration files
 Hint and tips
 References
(*) These contents are based on Fedora 19

4
Review: SysVinit & Upstart

Open Cloud Campus
5
Linux System Boot Process (1)
 “System BIOS” reads the boot loader program (GRUB) into memory from the boot disk.
 GRUB displays the boot kernel selection menu, reads the specified kernel and initrd
into memory, and start the kernel execution.
 Kernel extracts the contents of initrd into the RAM disk area of memory, and kicks the
“init script.”
– initrd contains device drivers necessary for root file system access and the “init
script.”
Boot Loader （GRUB）
/boot filesystem
(2) Read by the boot loader
(3) Extracted into
the RAM disk area
Boot DiskPhysical Memory
Linux Kernel
initrd
Linux Kernel
initrd
RAM disk area
- Device Drivers
- init script
- Other commands
(1) Executed by System BIOS

Open Cloud Campus
6
Linux System Boot Process (2)
 “init script” loads appropriate device drivers and mounts the root filesystem. Then it
executes the initial process “/sbin/init”.
– Strictly speaking, “init script” is replaced by “/sbin/init” with “exec” command.
 This “/sbin/init” is the main body of the traditional SysVinit or Upstart. Under the
systemd environment, this will be “/usr/bin/systemd” instead.
(*) Since initrd is a cpio+gzip archive file, you can extract it with pax command.
– # pax -rzf /boot/initramfs-2.6.32-131.0.15.el6.x86_64.img
#!/bin/sh
#
# Licensed under the GPLv2
#
# Copyright 2008-2009, Red Hat, Inc.
# Harald Hoyer <harald@redhat.com>
# Jeremy Katz <katzj@redhat.com>
...snip...
# start up udev and trigger cold plugs
udevd --daemon –resolve-names=never
...snip...
exec switch_root "$NEWROOT" "$INIT" $initargs || {
warn "Something went very badly wrong in the initramfs. Please "
warn "file a bug against dracut."
emergency_shell
}
fi
udevd loads necessary device drivers.
This process is replaced by /sbin/init with exec.
Example of init script from RHEL6

Open Cloud Campus
7
How SysVinit works
 SysVinit / Upstart does user land system initialization and service startups.
 RHEL5's /sbin/init (SysVinit) does the following tasks according to “/etc/inittab”.
id:5:initdefault:
# System initialization.
si::sysinit:/etc/rc.d/rc.sysinit
l0:0:wait:/etc/rc.d/rc 0
...snip...
# Run gettys in standard runlevels
1:2345:respawn:/sbin/mingetty tty1
# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
Example of /etc/inittab from RHEL5
– Execute the system initialization script “/etc/rc.d/rc.sysinit”
which does fsck and mount of filesystems, and other various
initialization works.
– Execute the service startup script “/etc/rc.d/rc” which
starts services according to the current runlevel.
• The actual startup tasks are done by the separate
service scripts under /etc/init.d/.
– Start mingetty processes which accepts the console user
login. And start the GUI login screen for runlevel 5.
mingetty
prefdm
rc.sysinit
System initialization Service startups Accept user login
rc
# /etc/init.d/<service> start

Open Cloud Campus
8
Upstart's Job flow
 RHEL6's ”/sbin/init” is an event based job management system called “Upstart.”
– Only the default runlevel is written in “/etc/inittab”. Various jobs are executed according to
the job configuration files under /etc/init/ on the system startup.
 However, the actual tasks done by those jobs are almost the same as in RHEL5.
Job: "rcS"
rc.sysinit
telinit $runlevel
Event "startup" is issued
/etc/init/rcS.conf
Event "runlevel X" is issued
/etc/init/rc.conf
Event "stopped rc" is issued
Job: "start-ttys"
initctl start tty TTY=$tty
/etc/init/start-ttys.conf
Job: "prefdm"
/etc/X11/prefdm -nodaemon
/etc/init/prefdm.conf
If RUNLEVEL=5
Job: "tty"
/sbin/mingetty TTY
Job: "rc"
/etc/rc.d/rc $RUNLEVEL
/etc/init/tty.conf
Execute /sbin/init

Open Cloud Campus
9
Service startup scripts
 Service startup scripts “/etc/init.d/<service>” should accept the following options at
least.
– start : Starting the service.
– stop : Stopping the service.
– status : Replying the service status through the return code.
 Since they are mere shell scripts, they could accept non-standard options, or could be
used for purposes other than the service(daemon) startup.
– PostgreSQL's database cluster initialization. (Non-standard option.)
• # service postgresql initdb
– Running the configuration wizard at the first boot time. (Not for service startup.)
• # service firstboot start

Open Cloud Campus
10
Objectives of systemd
 Reducing the system startup time.
– Shell scripts execute commands in serial order, which is not efficient under the modern multi
core processors. If various tasks in the script can be split and executed in parallel, it will
reduce the system startup time.
 Handling dynamic system configuration changes.
– Not only at the system startup time, but also at system configuration changes, services need
to be dynamically started/stopped.
 Providing the standard method for process shutdown.
– There has been no standard method to stop processes in the service script. Some use the PID
file while others rely on a process name to determine which processes should be stopped.
systemd should provide the standard methods with tracking the forked daemons by itself.
 Controlling the process execution environment
– Process execution environment such as resource management with cgroups and directory
access control should be centrally managed by systemd.
http://0pointer.de/blog/projects/systemd.html

11
System boot process with systemd

Open Cloud Campus
12
The minimum execution unit of systemd
 Various tasks in the SysVinit's system initialization scripts are extracted and defined
as independent “Unit”s.
rc.sysinit rc
systemd-remount-fs.service
systemd-udevd.service
・・・ chronyd.service
crond.service
dbus.service
irqbalance.service
mdmonitor.service
NetworkManager.service
rngd.service
rpcbind.service
rsyslog.service
sshd.service
・・・
console-getty.service
systemd-logind.service
dev-hugepages.mount
proc-sys-fs-binfmt_misc.mount
tmp.mount
・・・
mingetty
prefdm
# /etc/init.d/<service> start
sys-devices-pci00...0:00:03.0-virtio0-net-eth0.device
sys-devices-pci00...4.0-virtio1-block-vda-vda1.device
sys-devices-pci00...4.0-virtio1-block-vda-vda2.device
sys-devices-pci00...:00:04.0-virtio1-block-vda.device
dev-dmx2d1.swap
・・・
These are all “Unit”.

Open Cloud Campus
13
Unit types
 Unit has various types which are indicated by their name extensions. The followings
are some major units.
– .service：Service type
• When activated, an associated daemon is started.
– .target：Target type
• Do nothing. This is used to group other units when defining unit dependencies, or to
provide a timing synchronization point when defining order relationships.
– .mount：Mount point type
• When activated, an associated filesystem is mounted.
– .swap：Swap area type
• When activated, an associated swap area is enabled.
– .device：Device type
• When udev recognizes a new device, the associated unit is defined and activated
automatically.
 Some units are dynamically generated. You don't need to define all units by hand.
– .service and .target are defined through config files by hand.
– .mount and .swap are automatically generated from /etc/fstab.
– .device is automatically generated by udev.

Open Cloud Campus
14
Location of unit configuration files
 Unit configuration files are placed in two locations.
– Under /etc/systemd/system/ : Admin customized files.
– Under /usr/lib/systemd/system/ : System default files installed from RPM packages.
 If configuration files of the same filename are in both places, one in
/etc/systemd/system is used. (One in /usr/lib/systemd/system is simply ignored.)
– When modifying the system default configuration, you need to copy the configuration file
in /usr/lib/systemd/system to /etc/systemd/system, and modify the copied one.
 The name of configuration file corresponds to its unit name.
– So, you can create an alias name just by creating a symlink to the original configuration file.
– Directories “<unit name>.wants” are used to define unit dependencies, which is explained
later.
basic.target.wants
dbus-org.fedoraproject.FirewallD1.service -> /usr/lib/systemd/system/firewalld.service
dbus-org.freedesktop.Avahi.service -> /usr/lib/systemd/system/avahi-daemon.service
dbus-org.freedesktop.NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
default.target -> /lib/systemd/system/multi-user.target
default.target.wants
getty.target.wants
multi-user.target.wants
sockets.target.wants
sysinit.target.wants
syslog.service -> /usr/lib/systemd/system/rsyslog.service
system-update.target.want
Unit configuration files under /etc/systemd/system (Sample)

Open Cloud Campus
15
Unit dependencies and orders
 There are two independent definitions among units. One is “dependencies” and the
other is “orders”
– Dependency means “if unit A is activated, unit B must be activated, too.”
– Order means “unit A must be activated after unit B.”
 At a startup, systemd tries to activate the unit named “default.target”. Then all units
under its dependency tree are automatically activated.
– default.target is a symlink to another target such as multi-user.target or graphical.target.
Changing this symlink corresponds to changing the “default runlevel.”
– The drawing shows a typical dependency tree where the backbone is made of target units and
other real units are under those targets.
default.target multi-user.target basic.target
NetworkManager.service
avahi-daemon.service
irqbalance.service
remote-fs.target
rsyslog.service
・・・
symlink
fedora-autorelabel-mark.service
fedora-autorelabel.service
fedora-configure.service
fedora-loadmodules.service
sys-kernel-config.mount
sys-kernel-debug.mount
systemd-journald.service
systemd-modules-load.service
systemd-random-seed-load.service
systemd-sysctl.service
systemd-udev-trigger.service
systemd-udevd.service
・・・
sysinit.target

Open Cloud Campus
16
Overview of major dependencies
multi-user.target
symlink
graphical.target
rescue.target
Services activated
under runlevel 1
Services activated
under runlevel 3
Services activated
under runlevel 5
basic.target
default.target
sysinit.target
Services independent
of the runlevel
runlevelに依存せず
起動するサービス
Tasks traditionally
Handled by rc.sysinit
symlink can be
changed.
local-fs.target
swap.target
Enabling swap ares
Mounting
filesystems

Open Cloud Campus
17
Defining dependencies and orders (1)
 At startup, systemd tries to figure out all units which should be activated as a whole,
based on their dependencies.
– Note that the other relationship “orders” is defined independent of “dependencies.” They are
totally orthogonal.
– systemd tries to activate multiple (dependent) units in parallel unless they have some order
relationships in order to accelerate the system startup speed.
 Dependencies are defined in following ways.
– Use “Wants=” or “Requires=” to specify the pre-req units (i.e. units which must be activated
together) in the [Unit] section of the configuration file.
– Create a symlink to the configuration files of the pre-req units in “<unit name>.wants” or
“<unit name>.requires” directory.
– “Requires” means if the pre-req unit fails to start, this unit should not start either.
– “Wants” means even if the pre-req unit fails to start, this unit should start anyway.
– In addition, you can use “Conflicts=” option to specify the conflicting units which should not
be activated at the same time with this unit.

Open Cloud Campus
18
 “<unit name>.wants” directories are used to
configure autostartup of services.
– When autostartup is enabled with the
systemctl command, a symlink to the
service's configuration file is created
under the .wants directory specified by
“WantedBy=” option.
# systemctl disable sshd.service
rm '/etc/systemd/system/multi-user.target.wants/sshd.service'
# systemctl enable sshd.service
ln -s '/usr/lib/systemd/system/sshd.service' '/etc/systemd/system/multi-user.target.wants/sshd.service'
# ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx. 1 root root 46 Oct 30 12:35 NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx. 1 root root 35 Oct 30 12:36 atd.service -> /usr/lib/systemd/system/atd.service
lrwxrwxrwx. 1 root root 38 Oct 30 12:36 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root 44 Oct 30 12:35 avahi-daemon.service -> /usr/lib/systemd/system/avahi-daemon.service
lrwxrwxrwx. 1 root root 39 Oct 30 12:36 chronyd.service -> /usr/lib/systemd/system/chronyd.service
...snip...
lrwxrwxrwx. 1 root root 36 Nov 27 20:38 sshd.service -> /usr/lib/systemd/system/sshd.service
Sample of enabling / disabling autostartup of sshd.service
[Unit]
Description=OpenSSH server daemon
After=syslog.target network.target auditd.service
[Service]
EnvironmentFile=/etc/sysconfig/sshd
ExecStartPre=/usr/sbin/sshd-keygen
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
[Install]
WantedBy=multi-user.target
/usr/lib/systemd/system/sshd.service

Open Cloud Campus
19
 Unit orders are defined with “After=” and “Before=” options in the unit configuration files.
– “After=A B C” means this unit needs to be activated after “A”, “B” and “C”.
– “Before=A B C” means this unit needs to be activated before “A”, “B” and “C”.
 target units are used to define synchronization points of multiple units.
– For example, “network.target” is used as a synchronization point of “units providing network
functions” and “units using network functions”.
network.target
Units to provide
network functions
Units to use
network functions
Before=network.targetAfter=network.target
[Unit]
Description=firewalld - dynamic firewall daemon
Before=network.target
Before=libvirtd.service
Before=NetworkManager.service
・・・
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
・・・
/usr/lib/systemd/system/firewalld.service/usr/lib/systemd/system/httpd.service
httpd is assured to start
after firewall has been configured.

Open Cloud Campus
20
 The following commands show the dependencies and orders of currently active units.
– # systemctl list-dependencies <unit name>
• Show units which are pre-req of the specified unit (“default target” unless specified).
• If the pre-req unit is of target type, its pre-req units are shown recursively.
• –-all option shows all units recursively.
# systemctl list-dependencies
default.target
├─atd.service
├─auditd.service
...snip...
├─basic.target
│ ├─fedora-autorelabel-mark.service
...snip...
│ ├─firewalld.service
│ ├─paths.target
│ ├─sockets.target
│ │ ├─avahi-daemon.socket
│ │ ├─dbus.socket
...snip...
│ ├─sysinit.target
│ │ ├─dev-hugepages.mount
│ │ ├─dev-mqueue.mount
│ │ ├─lvm2-monitor.service
│ │ ├─plymouth-read-write.service
│ │ ├─plymouth-start.service
...snip...
├─getty.target
│ └─getty@tty1.service
└─remote-fs.target
– # systemctl list-dependencies <unit name> --after
• Show units which are activated before the specified
unit. (i.e. the specified one is activated after them.)
• --all option works as above.
– # systemctl list-dependencies <Unit名> --before
• Show units which are activated after the specified
unit. (i.e. the specified one is activated before them.)
• --all option works as above.

Open Cloud Campus
21
Dynamic unit generation (1)
 Generator modules under /usr/lib/systemd/system-generators/ provide dynamic unit
generation and modification according to system configuration changes.
– Example:
– systemd-cryptsetup-generator
• Generate systemd-cryptsetup@.service according to /etc/crypttab.
– systemd-fstab-generator
• Generate mount and swap type units according to /etc/fstab.
• For a mount type unit, its name corresponds to the mount point directory path.
(/ is replaced with _.)
– systemd-rc-local-generator
• Enable autostart of rc-local.service if /etc/rc.d/rc.local exists as an executable file.
 Configuration files of dynamically generated units are placed under
/run/systemd/generator/.
# ls /run/systemd/generator/
-.mount
boot.mount
dev-disk-byx2duuid-89cd76bex2d8d59x2d441cx2d9165x2dfe8ff338266b.device.wants
dev-mapper-fedorax2dswap.device.wants
dev-mapper-fedorax2dswap.swap
local-fs.target.requires
swap.target.wants

Open Cloud Campus
22
Dynamic unit generation (2)
 When udev assigns “systemd” tag to a new device, the corresponding unit is generated
and activated.
– Its name corresponds to the device's /sys filesystem path. (/ is replaced with _.)
 bluetooth.target, printer.target, smartcard.target and sound.target will be set as pre-
req of dynamically generated device units - bluetooth controllers, printers,
smartcards and soundcards, respectively.
– This means, for example, you can configure some units which requires soundcards as pre-req
of sound.target so that those units are automatically activated when a soundcard device is
attached to the system.
# systemctl list-units --type=device --full
UNIT LOAD ACTIVE SUB DESCR
sys-devices-pci0000:00-0000:00:01.1-ata2-host1-target1:0:0-1:0:0:0-block-sr0.device loaded active plugged QEMU_
sys-devices-pci0000:00-0000:00:03.0-virtio0-net-eth0.device loaded active plugged Virti
sys-devices-pci0000:00-0000:00:04.0-virtio1-block-vda-vda1.device loaded active plugged /sys/
sys-devices-pci0000:00-0000:00:04.0-virtio1-block-vda-vda2.device loaded active plugged /sys/
sys-devices-pci0000:00-0000:00:04.0-virtio1-block-vda.device loaded active plugged /sys/
sys-devices-platform-serial8250-tty-ttyS1.device loaded active plugged /sys/
sys-devices-pnp0-00:04-tty-ttyS0.device loaded active plugged /sys/
sys-devices-virtual-block-dmx2d0.device loaded active plugged /sys/
sys-devices-virtual-block-dmx2d1.device loaded active plugged /sys/
sys-module-configfs.device loaded active plugged /sys/
sys-subsystem-net-devices-eth0.device loaded active plugged Virti

23
Operating systemd

Open Cloud Campus
24
Listing units (1)
 # systemctl list-unit-files
– List all defined units and their statuses.
– --type option lists only the specified type.
Status Description
enabled “WantedBy=” entry exists.
Autostart enabled.
disabled “WantedBy=” entity exists.
Autostart disabled.
static “WantedBy=” entity doesn't
exists.
# systemctl list-unit-files --type=service
UNIT FILE STATE
arp-ethers.service disabled
atd.service enabled
auditd.service enabled
autovt@.service disabled
avahi-daemon.service enabled
blk-availability.service disabled
chrony-wait.service disabled
chronyd.service enabled
console-getty.service disabled
console-shell.service disabled
crond.service enabled
dbus-org.fedoraproject.FirewallD1.service enabled
dbus-org.freedesktop.Avahi.service enabled
dbus-org.freedesktop.hostname1.service static
dbus-org.freedesktop.locale1.service static
dbus-org.freedesktop.login1.service static
dbus-org.freedesktop.NetworkManager.service enabled
dbus-org.freedesktop.timedate1.service static
dbus.service static
...snip...
Similar to the traditional
“chkconfig --list”
You can use systemctl command
to enable/disable autostart.

Open Cloud Campus
25
Listing units (2)
 # systemctl list-units （”list-units” can be omitted.)
– List currently active (or should be active) units and their statuses.
– --type option lists only the specified type.
# systemctl --type=service
UNIT LOAD ACTIVE SUB DESCRIPTION
atd.service loaded active running Job spooling tools
auditd.service loaded active running Security Auditing Service
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
chronyd.service loaded active running NTP client/server
crond.service loaded active running Command Scheduler
dbus.service loaded active running D-Bus System Message Bus
fedora-readonly.service loaded active exited Configure read-only root supp
firewalld.service loaded active running firewalld - dynamic firewall
getty@tty1.service loaded active running Getty on tty1
irqbalance.service loaded active running irqbalance daemon
lvm2-lvmetad.service loaded active running LVM2 metadata daemon
lvm2-monitor.service loaded active exited Monitoring of LVM2 mirrors, s
mcelog.service loaded active running Machine Check Exception Loggi
NetworkManager.service loaded active running Network Manager
polkit.service loaded active running Authorization Manager
rngd.service loaded failed failed Hardware RNG Entropy Gatherer
rpcbind.service loaded active running RPC bind service
・・・
systemd-v...le-setup.service loaded active exited Setup Virtual Console
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
31 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Open Cloud Campus
26
Basic unit operations (1)
 # systemctl enable/disable <unit name>
– Enable/disable autostart of the specified unit. (It defines/undefines dependency from the
unit specified with “WantedBy=” option in the background.)
 # systemctl start/stop/restart <unit name>
– Start/stop/restart the specified unit.
– “reload” can be used only when “ExecReload=” is defined in the unit configuration file.
 # systemctl status <unit name>
– Show running status of the specified unit.
# systemctl status httpd.service
httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
Active: active (running) since Sun 2013-11-03 15:59:37 JST; 4 weeks 2 days ago
Main PID: 4621 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: name=systemd:/system/httpd.service
├─4621 /usr/sbin/httpd -DFOREGROUND
└─4626 /usr/sbin/httpd -DFOREGROUND
Nov 03 15:59:36 fedora19 systemd[1]: Starting The Apache HTTP Server...
Nov 03 15:59:36 fedora19 httpd[4621]: AH00557: httpd: apr_sockaddr_info_get...19
Nov 03 15:59:36 fedora19 httpd[4621]: AH00558: httpd: Could not reliably de...ge
Nov 03 15:59:37 fedora19 systemd[1]: Started The Apache HTTP Server.
Recent entries of daemon logs
daemon processes of this service

Open Cloud Campus
27
Basic unit operations (2)
 # systemctl daemon-reload
– Let systemd reload configuration files. You need to run this when you modified a unit
configuration file.
 chkconfig and service commands are translated to corresponding systemctl commands.
 Non-standard options in service startup scripts cannot be used with systemd
command.
– For example, you need to use postgresql-setup for database cluster initialization of
PostgreSQL.
• RHEL6: # service postgresql initdb
• Fedora19: # postgresql-setup initdb
# chkconfig sshd off
Note: Forwarding request to 'systemctl disable sshd.service'.
rm '/etc/systemd/system/multi-user.target.wants/sshd.service'
# chkconfig sshd on
Note: Forwarding request to 'systemctl enable sshd.service'.
ln -s '/usr/lib/systemd/system/sshd.service' '/etc/systemd/system/multi-user.target.wants/sshd.service'
# service sshd stop
Redirecting to /bin/systemctl stop sshd.service
# service sshd start
Redirecting to /bin/systemctl start sshd.service

Open Cloud Campus
28
Listing cgroups
 systemd creates a cgroups' group for each service.
– # systemd-cgls : Show cgroups' group tree.
– # systemd-cgtop : Display resource usage per group in real time.
# systemd-cgls
├─system
│ ├─1 /usr/lib/systemd/systemd --system --deserialize 17
│ ├─httpd.service
│ │ ├─4621 /usr/sbin/httpd -DFOREGROUND
│ │ └─4626 /usr/sbin/httpd -DFOREGROUND
│ ├─sm-client.service
│ │ └─560 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
│ ├─sendmail.service
│ │ └─495 sendmail: accepting connections
│ ├─rpcbind.service
│ │ └─461 /sbin/rpcbind -w
│ ├─sshd.service
│ │ └─4440 /usr/sbin/sshd -D
・・・
│ └─systemd-journald.service
│ └─285 /usr/lib/systemd/systemd-journald
└─user
└─0.user
├─108.session
│ ├─4521 sshd: root@pts/1
│ ├─4525 -bash
│ ├─4654 systemd-cgls
│ └─4655 cat
Subgroup for each unit is created
under this parent group.
Subgroup for each user account is
created under this parent group.

Open Cloud Campus
29
Sending signal to process groups
 You can send a process signal to all processes in the same cgroups' group.
– Example：
– # systemctl kill -s9 sshd.service
• Send KILL signal (signal# 9) to all processes in the group for sshd.service.
– You can kill all the processes related to a specific service with this.
– With “--kill-who=main” option, it sends the signal only to the main process which has started
first in the group.
 This mechanism is used when stopping a unit with “systemctl stop”, too.
– systemd first execute the command specified with “ExecStop=” in the configuration file. If
there remains some processes in the group, it sends “SIGTERM” to them. If there still remains
some processes, it sends “SIGKILL”.

30
Log management with journald

Open Cloud Campus
31
Log management mechanism of systemd
 systemd intercepts messages to stdout/stderr and rsyslogd from daemon processes
which have been started as service units. The messages are sent to its own logging
service “systemd-journald.service”, or “journald” in short.
– journald records those messages in binary files under /var/log/journal/ with adding some
meta information of sender processes.
– You can use the “journalctl” command to search the binary log.
 rsyslogd should be configured accordingly.
– In the traditional environment, rsyslogd receives processes' system log messages through the
unix socket “/dev/log”. Under the systemd environment, systemd receives messages from
“/dev/log” and send it again to “/run/systemd/journal/syslog”. So rsyslogd needs to receive it
through “/run/systemd/journal/syslog”.
rsyslogd
/dev/log
Process
/dev/log
Process
rsyslogd
/run/systemd/journal/syslog
systemd
$SystemLogSocketName /run/systemd/journal/syslog
/etc/rsyslog.d/listen.conf
Traditional system log flow
System log flow
under systemd

Open Cloud Campus
32
imjournal module of rsyslogd
 The latest rsyslogd can directly access journald's database using “imjournal” module.
In this case, it doesn't need to read “/run/systemd/journal/syslog”.
– RHEL7 uses imjournal module by default.
– See the following site for details.
• http://www.rsyslog.com/doc/imjournal.html

Open Cloud Campus
33
Log search with journalctl
 The followings are major options of journalctl command.
– -u <unit name> : Show log messages related to this unit.
– --since="YYYY-MM-DD hh:mm:ss" : Show log messages since this date/time.
– --until="YYYY-MM-DD hh:mm:ss" : Show log messages until this date/time.
– -b : Show log messages since the last system boot.
– -f : Works as “tail -f”
– --no-pager: Don't use the pager(less)
– -a : Don't cut long messages.
# journalctl -u sshd.service -b --no-pager -a
-- Logs begin at Wed 2013-10-30 12:40:32 JST, end at Tue 2013-12-03 21:18:59 JST. --
Oct 30 12:59:39 fedora19 systemd[1]: Starting OpenSSH server daemon...
Oct 30 12:59:39 fedora19 systemd[1]: Started OpenSSH server daemon.
Oct 30 12:59:40 fedora19 sshd[454]: Server listening on 0.0.0.0 port 22.
Oct 30 12:59:40 fedora19 sshd[454]: Server listening on :: port 22.
Oct 30 12:59:50 fedora19 sshd[824]: Accepted password for root from 192.168.122.1 port 42680 ssh2
Nov 01 16:08:57 fedora19 sshd[2037]: Accepted password for root from 192.168.122.1 port 52666 ssh2
Nov 03 15:49:23 fedora19 systemd[1]: Stopping OpenSSH server daemon...
Nov 03 15:49:23 fedora19 sshd[454]: Received signal 15; terminating.
Nov 03 15:49:24 fedora19 systemd[1]: Stopped OpenSSH server daemon.
Nov 03 15:49:25 fedora19 systemd[1]: Starting OpenSSH server daemon...
Nov 03 15:49:25 fedora19 systemd[1]: Started OpenSSH server daemon.
Nov 03 15:49:25 fedora19 sshd[4440]: Server listening on 0.0.0.0 port 22.
Nov 03 15:49:25 fedora19 sshd[4440]: Server listening on :: port 22.
Nov 03 15:56:39 fedora19 sshd[4461]: Accepted password for root from ::1 port 51491 ssh2

34
Unit configuration files

Open Cloud Campus
35
General configuration options (1)
 Unit configuration file has multiple sections such as [Unit], [Install], [Service], etc.
– [Unit] : Common options for all unit types such as dependency and order.
– [Install]：Options related to “systemctl enable/disable” operation.
– [Service]：Options specific to the service type.
– Each type has its own section in general.
 Major options in [Unit] section.
– If you need to specify multiple items in the same option, they can be space separated. You can
use the same option multiple times, too. (This rule applies to all sections.)
Option Description
Description Short documentation of this unit.
Documentation Document URL
Requires/Wants(*)
Pre-req units of this unit.
After This unit will be activated after these units.
Before This unit will be activated before these units.
(*) ”Requires” means this unit won't be activated if pre-req units failed to start.
　 “Wants” means this unit will be activated even if pre-req units fails to start.

Open Cloud Campus
36
General configuration options (2)
 Major options in [Install] section.
A
– “WantedBy” and “RequiredBy” is to specify the unit to which this unit becomes a pre-req
when autostart is enabled.
Option Description
WantedBy When enabled, symlink to the configuration files is created under this
unit's .wants directory.
RequiredBy When enabled, symlink to the configuration files is created under this unit's
.required directory.
Also When enabled/disabled, this unit is enabled/disabled, too.
[Unit]
[Service]
KillMode=process
[Install]
This service is activated after
syslog/network/auditd is ready.
This service is enabled as
a pre-req of multi-user.target

Open Cloud Campus
37
Configuration options for service type unit (1)
 Major options in [service] section (1)
– “ExecXXX” options specify the commands to start/reload/stop the service. Environment
variables from “EnvironmentFile” and the special variable “$MAINPID” can be used. Details are
explained later.
– systemd judges if the service is successfully started based on the result of ExecStart
command. The result of ExecStartPre/ExecStartPost commands are not considered.
– ExecStopPost command is executed not only when the service is stopped with ExecStop but
also when the service aborts.
Option Description
ExecStart Command to start the service
ExecReload Command to reload the service
ExecStop Command to stop the service
ExecStartPre
ExecStartPost
Additional commands to be executed before/after ExecStart.
ExecStopPost Command to be executed when the service stops including the case of abort.
EnvironmentFile Read environment variables from this file.
KillMode Specify how remaining processes are handled after ExecStop.

Open Cloud Campus
38
– Type option specifies the timing when systemd considers that the service is successfully
started.
• Type=simple : Used when the specified command runs in foreground. systemd considers
it's successfully started just when the command is executed.
• Type=forking : Used when the specified command forks a child process and exits.　
systemd considers it's successfully started when the command exits.
• Type=oneshot : Used when the specified command does oneshot task and exits. systemd
considers it's successfully executed and the service has been finished. (If
“RemainAfterExit=yes” is set, it considers that the service is still active.）
• Type=notify : Used when the specified command uses the systemd's library function
“sd_notify()”. The process needs to be designed to use sd_notify()(*)
.
• Type=dbus : Used when the service uses D-Bus (Inter process messaging bus). systemd
considers it's successfully started whtn the connection name specified with “BusName” is
registered in D-Bus.
Option Description
Type Specify how the service startup is detected.（Default: simple)
PIDFile PID file of the main process of “fork” type service.
BusName Bus connection name of “D-Bus” type service.
Restart Specify whether the service is restarted when it aborts.（Default: no)
(*) https://fedoraproject.org/wiki/User:Johannbg/QA/Systemd/Sd_notify

Open Cloud Campus
39
 systemd tracks PID of the main process which has been started first in the process
group of the service. This PID can be referred with $MAINPID in ExecXXX options.
– You use it for sending HUP signal to the main process in ExecReload, for example.
– For “type=simple”, the main process is the one started with ExecStart.
– For “type=forking”, systemd detects it from the PID file specified with PIDFile option. (The
service startup command needs to create the PID file by itself.）
 When the process aborts, systemd takes the following actions. (Corresponds to the
respawn configuration in the traditional case.)
– Restart option specified if systemd tries to restart service.
• Restart=no : It doesn't restart service.
• Restart=always : It tries to restart service.
– In default, if the service restarted more than five times in ten seconds, systemd doesn't
restart it for next ten seconds. In general, if the service restarted more than
“StartLimitBurst” times in “StartLimitInterval”, systemd doesn't restart it for next
“StartLimitInterval”. (i.e. The default is StartLimitInterval=10s and StartLimitBurst=5.)

Open Cloud Campus
40
 systemd creates a cgroups' group for each service to track the related processes.
– When the service is stopped with ExecStop command, systemd takes the following actions
according to “KillMode” option if there are some remaining processes in the group.
• KillMode=none : The remaining processes are left untouched.
• KillMode=process : If the main process is still running, systemd stops it with
SIGTERM/SIGKILL. Other processes are left untouched.
• KillMode= control-group : All remaining processes in the group are stopped with
SIGTERM/SIGKILL.
[Unit]
[Service]
KillMode=process
[Install]
Generate hostkey before starting the service.
Reload is done by sending
HUP to the main process.
With -D option (daemon mode), the initial
command keeps running in foreground.
Hence, “Type=simple” (default) should be used.
Child processes are left
when the service is stopped.

Open Cloud Campus
41
 Use of “Type=simple/notify” is recommended instead of “Type=forking” in order to
avoid the use of PID file.
– For example, httpd.service uses “Type=forking” in Fedora17, but it's replaced with
“Type=notify” in Fedora19.
[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop
KillSignal=SIGCONT
PrivateTmp=true
/usr/lib/systemd/system/httpdd.service in Fedora19
[Service]
Type=forking
PIDFile=/var/run/httpd/httpd.pid
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -k start
ExecReload=/usr/sbin/httpd $OPTIONS -t
ExecStop=/usr/sbin/httpd $OPTIONS -k stop
PrivateTmp=true
/usr/lib/systemd/system/httpdd.service in Fedora17
The main process is tracked with PID file.
This command exits after forking
child process in background.
httpd is integrated with systemd.(*)
With -DFOREGROUND option, this command
keeps running in foreground. The service startup
is notified with sd_notify().
(*) httpd uses mod_systemd module for systemd integration. http://bit.ly/1bTIoKg

Open Cloud Campus
42
– User/Group specifies user/group to execute commands in “ExecXXX=” options.
• If “PermissionsStartOnly=yes” is set, only “ExecStart=” command is executed with this
user/group.
– Other options are used to limit directory access from the processes of this service.
• These features use “filesystem namespace” separation mechanism in background. Because
of this, the bind-mount done by the processes in this service are invisible from other
processes(*)
.
(*) For example, if processes in the service use “ip netns”, this may cause a problem because this command bind-
mounts network namespace information under /proc onto /var/run/netns to share it with other processes.
Bug 872689 - Quantum: root cannot access network namespaces created by Quantum service
https://bugzilla.redhat.com/show_bug.cgi?id=872689
Option Description
User / Group user/group to start processes.
PrivateTmp Prepare private /tmp and /var/tmp for this service.
ReadOnlyDirectories The specified directory becomes read only.
InaccessibleDirectories The specified directory becomes inaccessible.
RootDirectory chroot-ed to the specified directory.

43
Hint and tips

Open Cloud Campus
44
Disable with masking
 # systemctl mask/unmask <unit name>
– Disable/enable the specified unit with masking it. When disabled with masking, it cannot be
activated with “systemctl start”.
– When disabled with masking, its configuration file is created as a symlink to /dev/null
under /etc/systemd/system. If a configuration files already exits in /etc/systemd/system/, it
cannot be masked.
# systemctl mask firewalld.service
ln -s '/dev/null' '/etc/systemd/system/firewalld.service'
# systemctl unmask firewalld.service
rm '/etc/systemd/system/firewalld.service'

Open Cloud Campus
45
Template style configuration file
 A configuration file with the name “foo@.service” works as a configuration for
multiple services as “foo@<arbitrary string>.service”
– # systemctl start foo@bar.service
• Service “foo@bar.service” is activated according to the configuration file “foo@.service”.
The parameter “%I” in the configuration file is replaced by “<arbitrary sting>” (“bar” in
this case).（”%i” can also be used. It is an escaped version of “%I”.）
– # systemctl enable foo@bar.service
• A symlink from “foo@bar.service” to “foo@.server” is created in the .wants directory.
When systemd tires to autostart “foo@bar.service”, it actually uses the configuration file
“foo@.service”.
– In the example below, “getty@tty1.service”, “getty@tty2.service”, etc. can be used.
[Service]
# the VT is cleared by TTYVTDisallocate
ExecStart=-/sbin/agetty --noclear %I 38400 linux
Type=idle
Restart=always
RestartSec=0
UtmpIdentifier=%I
TTYPath=/dev/%I
TTYReset=yes
TTYVHangup=yes
TTYVTDisallocate=yes
KillMode=process
IgnoreSIGPIPE=no
/lib/systemd/system/getty@.service（Snippet from [service] section)

46
References

Open Cloud Campus
47
References
 freedesktop.org - systemd System and Service Manager
– Project home of systemd
– http://www.freedesktop.org/wiki/Software/systemd/
 The systemd for Administrators Blog Series
– Blog series by the original developer Lennart (Links are on the project home.)
 Rethinking PID 1
– Lennart's blog about the background idea of systemd
– http://0pointer.de/blog/projects/systemd.html
 man page
– systemd is supplemented with useful man pages. In addition to a man page for each command,
you can follow “SEE ALSO” section of systemd(1).

48
Etsuji Nakai
Twitter @enakai00
Open Cloud Campus
Let's study the latest
Linux features with Fedora!

Your first dive into systemd!

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Your first dive into systemd!

Similar to Your first dive into systemd! (20)

More from Etsuji Nakai

More from Etsuji Nakai (20)

Recently uploaded

Recently uploaded (20)

Your first dive into systemd!