README

system-helpers - Collection of helper systemd services, systemd timers and shell scripts for common tasks.

It can help you out for small servers where you think a proper monitoring and alerting system like Prometheus and AlertManager is too much.

The notifier script can send notifications via mail or gotify.

system-helpers collection provides the following helpers

notifier which sends notifications via mail or gotify (see see notifier)
checking if systemd services and timers are running (see man systemd_check)
borgmatic backup (see exposed systemd services borgmatic@ which require a valid borgmatic.yml file in /etc/borgmatic or your $HOME/.config/borgmatic)
restic backup (see /usr/share/doc/system-helpers/restic for an example mybackup)
checking if docker containers are running (see man docker_check)
batch docker-compose upgrades (see man docker_compose_update)
dynamic dns update (see man docker_compose_update)
memory and disk checks (see man disk_space_alert and man memory_usage_alert)
docker housekeeping for removing unused docker containers, volumes and images
update check for pacman (ArchLinux specific, see man check_updates)

Provided systemd services mostly support mail notifications if they have failed via systemd_failure_notify.

See /usr/share/doc/system-helpers for example configurations which should be copied inside $HOME or /etc depending on the helper script.

Installation

Package repository for Arch Linux

Use the custom provided ArchLinux repository hosted at aur.myservermanager.com

[repo-aur-myservermanager-com]
SigLevel = Never
Server = https://aur.myservermanager.com

Package build for Arch Linux

Clone this repository
Build a pacman package for ArchLinux via makepkg -csi inside the _pkg/ folder

system-helpers-sync script to synchronize files to locations

Clone this repository
Run ./system-helpers-sync and pick an action which should be self-explaining

Usage

Services and scripts don't depend on each other, except for notifier which is commonly used. See more about notifier in the next section. All scripts include hints that they can be combined (out-commented line in service files for systemd_failure_notify@.service for example).

Examples are given inside usr/share/doc/system-helpers and man pages are also available.

Making use of notifier

Scripts make heavy use the bundled notifier script (see man notifier) which should be configured properly, e.g. add a /etc/notifier.conf:

NOTIFIER_MAIL_ADDRESS=monitoring@yourdomain.tld
NOTIFIER_MAIL_ENABLED=true
# Enable gotify by running gotify init first, you need a custom serverfor this
NOTIFIER_GOTIFY_ENABLED=false

Important: You can also change notifier behavior on a per-command basis, e.g. disable mail or gotify by providing ALL notifier variables as environment variables where the command is issued, for example in a systemd .service file.

# ...

[Service]
# ...
Environment="NOTIFIER_GOTIFY_ENABLED=false"
Environment="NOTIFIER_MAIL_ENABLED=true"
Environment="NOTIFIER_MAIL_ADDRESS=another-mail-different-from-your-global-config@domain.tld"

# ...

Another option is to set those inside the respective config files of another script. Let's have an example with the check_updates script.

# /etc/check_updates.conf
NOTIFIER_GOTIFY_ENABLED=false
NOTIFIER_MAIL_ENABLED=true
NOTIFIER_MAIL_ADDRESS=updatemail@domain.tld

This will not send gotify messages for check_updates and send a mail to a different address updatemail@domain.tld.

FAQ

User cannot use network-online.target

By default, no user unit can depend on system events like network-online.target. To enable this, you need to link the proper .target which you like to use inside the user service before.

systemctl --user link /usr/lib/systemd/system/network-online.target

This would make network-online.target available after a reboot to the user who issued the command.

`network-online.target` doesn't recognize DNS correctly

If you like to set Persistent=true, then you should probably equip the related .service file with a ExecStartPre=/bin/bash -c 'until host captiveportal.myservermanager.com; do sleep 1; done' or any domain you think is 100% reachable.

4.4 KiB Raw Blame History