system-helpers/README.md

# README

system-helpers - Collection of helper systemd services, systemd timers and
shell scripts for common tasks.

It can help you out for small servers where you think a proper monitoring and
alerting system like [Prometheus](https://prometheus.io) and 
[AlertManager](https://prometheus.io/docs/alerting/latest/alertmanager/) is too much.

The **notifier** script can send notifications via mail or [gotify](https://gotify.net).

**system-helpers** collection provides the following helpers

- notifier which sends notifications via mail or gotify (see **see notifier**)
- checking if systemd services and timers are running (see **man systemd_check**)
- borgmatic backup (see exposed systemd services *borgmatic@* which require a valid *borgmatic.yml* file in */etc/borgmatic* or your **$HOME/.config/borgmatic**)
- restic backup (see */usr/share/doc/system-helpers/restic* for an example *mybackup*)
- checking if docker containers are running (see **man docker_check**)
- batch docker-compose upgrades (see **man docker_compose_update**)
- dynamic dns update (see **man docker_compose_update**)
- memory and disk checks (see **man disk_space_alert** and **man memory_usage_alert**)
- docker housekeeping for removing unused docker containers, volumes and images
- update check for pacman (ArchLinux specific, see **man check_updates**)

Provided systemd services mostly support mail notifications if they have failed
via `systemd_failure_notify`.

See `/usr/share/doc/system-helpers` for example configurations which should 
be copied inside $HOME or /etc depending on the helper script.

## Installation

### Package repository for Arch Linux

Use the custom provided ArchLinux repository hosted at `aur.myservermanager.com`

```shell
[repo-aur-myservermanager-com]
SigLevel = Never
Server = https://aur.myservermanager.com
```

### Package build for Arch Linux

1. Clone this repository
2. Build a `pacman` package for ArchLinux via `makepkg -csi` inside the `_pkg/` folder

### system-helpers-sync script to synchronize files to locations

1. Clone this repository
2. Run `./system-helpers-sync` and pick an action which should be self-explaining

## Usage

Services and scripts don't depend on each other, except for `notifier` which is commonly used. See more about `notifier` in the next section. All scripts include hints that they can be combined (out-commented line in service files for `systemd_failure_notify@.service` for example).

Examples are given inside `usr/share/doc/system-helpers` and *man pages* are also available.

### Making use of notifier

Scripts make heavy use the bundled **notifier** script (see `man notifier`) which should be configured properly, e.g. add a `/etc/notifier.conf`:

```shell
NOTIFIER_MAIL_ADDRESS=monitoring@yourdomain.tld
NOTIFIER_MAIL_ENABLED=true
# Enable gotify by running gotify init first, you need a custom serverfor this
NOTIFIER_GOTIFY_ENABLED=false
```

**Important:** You can also change notifier behavior on a per-command basis, e.g. disable mail or gotify by providing ALL notifier variables as environment variables where the command is issued, for example in a systemd .service file.

```shell
# ...

[Service]
# ...
Environment="NOTIFIER_GOTIFY_ENABLED=false"
Environment="NOTIFIER_MAIL_ENABLED=true"
Environment="NOTIFIER_MAIL_ADDRESS=another-mail-different-from-your-global-config@domain.tld"

# ...
```

Another option is to set those inside the respective config files of another script. Let's have an example with the `check_updates` script.

```shell
# /etc/check_updates.conf
NOTIFIER_GOTIFY_ENABLED=false
NOTIFIER_MAIL_ENABLED=true
NOTIFIER_MAIL_ADDRESS=updatemail@domain.tld
```

This will not send gotify messages for `check_updates` and send a mail to a different address `updatemail@domain.tld`.

## FAQ

### User cannot use network-online.target

By default, no user unit can depend on system events like `network-online.target`. 
To enable this, you need to link the proper `.target` which you like to use inside
the user service before.

```shell
systemctl --user link /usr/lib/systemd/system/network-online.target
```

This would make `network-online.target` available after a _reboot_ to the user who issued the command.

### `network-online.target` doesn't recognize DNS correctly

If you like to set `Persistent=true`, then you should probably equip the related `.service` file with a `ExecStartPre=/bin/bash -c 'until host captiveportal.myservermanager.com; do sleep 1; done'` or any domain you think is 100% reachable.