systemd recap++ (part 2 of 3) fixin' it

[Ed]

Systems Administrators make things work and systemd gives you the
opportunity to configure systems in many different ways[1]. A good
example is unbound which I like to run as a validating recursive
caching DNS on all my systems, with a little spoofing thrown in for
good measure. Call it GDFI_dev style: unbound runs on the loopback,
and that is the only nameserver listed in resolv.conf. I like it
because you don't touch the network much at all, and when you do,
dnssec makes sure you get accurate answers. The spoofing is handy for
developing; you can work on things with the proper network context,
just like it will be when deployed. (also good for guaranteeing
doublclick never hears from you & yours, but i digress)

As an example, unbound.service
(/usr/lib/systemd/system/unbound.service) is worth the look. Other
than showing which units unbound should follow and which to precede
(unbound-keygen, nss-lookup.target) the unit shows which applications
should front-run unbound - there are two: unbound-anchor and
unbound-checkconf.

unbound-checkconf is obvious - a sanity check on
/etc/unbound/unbound.conf. unbound-anchor is interesting and brings up
an issue that you may want to add to the unbound unit and I will use
this as the example. "Unbound-anchor  performs setup or update of the
root trust anchor for DNSSEC validation" and runs prior to unbound
every time - everything dnssec validates hangs off this certificate
and unbound will only validate from a good certificate.

But what happens if your clock is way wrong when unbound starts? What
if the dnssec trust anchor gets a timestamp from the future? Too old,
or certs from future will put unbound sideways and nothing will
validate. That's a problem we can fix: make sure we only start unbound
after ntpd is running. (ok, you could be running chronyd, but it will
shut down when ntpd starts and besides chrond isn't the example here)

This calls for a "drop-in". Drop-ins are files that end in .conf and
are in a directory with the unit name plus ".d". Drop-ins for unbound,
like ntpd_first.conf, would be found in the
/etc/systemd/system/unbound.service.d directory. Create that directory
and the file ntpd_first.conf with the following lines in it:

[Unit]
After=ntpd.service
Wants=ntpd.service

The Wants= is a bit extra, but creates a less rigid relationship than
the implied Requires= and is the recommended way to link one unit to
another. Feel free to use the ntpd_first.conf file whenever you want
to make sure your systems time is good before an app runs. Also
remember to include section headers (ie [Unit]) before any directives
or you will see "Assignment outside of section. Ignoring" in your
Haiku.

Over time you will want to keep track of your adjustments. Run the
command "systemd-delta" to see the changes that make up the units
currently in effect:

[EQUIVALENT] /etc/systemd/system/default.target →
/usr/lib/systemd/system/default.target
[EXTENDED]   /run/systemd/system/session-1.scope →
/run/systemd/system/session-1.scope.d/90-SendSIGHUP.conf
[EXTENDED]   /run/systemd/system/session-1.scope →
/run/systemd/system/session-1.scope.d/90-TimeoutStopUSec.conf
[EXTENDED]   /run/systemd/system/session-1.scope →
/run/systemd/system/session-1.scope.d/90-KillMode.conf
[EXTENDED]   /run/systemd/system/session-1.scope →
/run/systemd/system/session-1.scope.d/90-After-systemd-user-sessions\x2eservice.conf
[EXTENDED]   /run/systemd/system/session-1.scope →
/run/systemd/system/session-1.scope.d/90-Description.conf
[EXTENDED]   /run/systemd/system/session-1.scope →
/run/systemd/system/session-1.scope.d/90-Slice.conf
[EXTENDED]   /usr/lib/systemd/system/unbound.service →
/etc/systemd/system/unbound.service.d/ntpd_first.conf

8 overridden configuration files found.

In the above response we see some modifications from Fedora that are
mostly runtime as well as the changes (of type [EXTENDED]) to the
unbound.service we just made with the drop-in - for a total of 8.
Also, the unbound Haiku (systemctl status unbound) gets a bit longer:

unbound.service - Unbound recursive Domain Name Server
   Loaded: loaded (/usr/lib/systemd/system/unbound.service; enabled)
  Drop-In: /etc/systemd/system/unbound.service.d
           └─ntpd_first.conf
   Active: active (running) since Mon 2014-03-17 19:28:04 MST; 4min 5s ago
  Process: 5739 ExecStartPre=/usr/sbin/unbound-checkconf (code=exited,
status=0/SUCCESS)
  Process: 5737 ExecStartPre=/sbin/runuser --shell /bin/sh -c
/usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c
/etc/unbound/icannbundle.pem unbound (code=exited, status=0/SUCCESS)
 Main PID: 5742 (unbound)
   CGroup: /system.slice/unbound.service
           └─5742 /usr/sbin/unbound -d

Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] debug: target fetch
policy for level 1 is 2
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] debug: target fetch
policy for level 2 is 1
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] debug: target fetch
policy for level 3 is 0
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] debug: target fetch
policy for level 4 is 0
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] debug: Forward zone
server list:
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] info:
DelegationPoint<.>: 0 names (0 missing), 1 addrs (0 result, 1 avail)
parentNS
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] debug: Reading root
hints from /named.cache
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] info:
DelegationPoint<.>: 13 names (4 missing), 22 addrs (0 result, 22
avail) parentNS
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] debug: cache memory
msg=66072 rrset=66072 infra=2600 val=66344
Mar 17 19:28:04 malibu.xb unbound[5742]: [5742:0] info: start of
service (unbound 1.4.21).

You can customize until you get them all just perfect, just remember
configuration changes made after enabling a unit need a "systemctl
daemon-reload" to be incorporated into systemd. Don't worry, you will
get a hint and it will be a twitch reflex in no time. Next, a bit
extra on how dmesg has changed...

[1] unbound builds a unit from what it finds in the following
locations(in order): /usr/lib/systemd/system/* (as installed),
/run/systemd/system/*, and then /etc/systemd/system/* among others.
see man systemd.unit for more details.