Varakh 0194429c80 Adapted logging which defaults to JSON encoding

2024-01-24 23:21:48 +01:00

27 KiB

Raw Blame History

README

upda - Update Dashboard in Go. Please see motivation and concepts what this application does.

There's also a upda web interface. It's recommended to take a look (at least at the screenshots).

In addition, there's a commandline tool called upda-cli. For more information, download it and run ./upda-cli help for further instructions. This is especially useful, if you have an upda (server) running and like to invoke webhooks from CLI. upda-cli is also bundled in the docker images.

See the deployment instructions for examples on how to deploy upda and upda-ui

The main git repository is hosted at https://git.myservermanager.com/varakh/upda. Other repositories are mirrors and pull requests, issues, and planning are managed there.

Contributions are very welcome!

Motivation
Concepts
Configuration
3rd party integrations
- Webhooks
- Prometheus Metrics
Deployment
- Native
- Docker
  - Build docker image
Development & contribution
- Getting started
  - Windows hints
- Release

Motivation

duin can determine which OCI images have updates available. Argus can query other sources like GitHub and even invoke actions when an update has been found, but there's no convenient way of having one dashboard or source of truth for all of them across different hosts without tinkering with collecting them somewhere in one place. This application is the result of that tinkering. :-)

Managing various application or OCI container image updates can be a tedious task:

A lot of hosts to operate with a lot of different applications being deployed
A lot of different OCI containers to watch for updated images
No convenient dashboard to see and manage all the available updates in one place

upda manages a list of updates with attributes attached to it. For new updates to arrive, upda needs to be called via a webhook call (created within upda) from other applications, such as a bash script, an application like duin or simply by using the upda-cli.

After an update is being tracked, upda provides a convenient way to have everything in one place. In addition, it exposes managed updates as prometheus metrics, so that you can easily build a dashboard in Grafana, or even attach alerts to pending updates via alertmanager.

In addition, you can use upda's UI to manage updates, e.g. approve them when they have been rolled out to a host.

Important to note:

upda is NOT a scraper to watch docker registries or GitHub releases, it simply collects and consolidates updates from different sources via webhooks. If you like to watch GitHub releases, write a scraper and use upda-cli to report back to upda.
upda uses basic auth for administrative tasks like viewing available updates or setting up the initial webhooks.

Concepts

upda retrieves new updates when webhooks of upda are invoked, e.g., duin invokes it or any other application which can reach the instance. Tracked updates are unique for the attributes (application,provider,host) which means that subsequent updates for an identical application, provider and host simply updates the version and metadata attributes for that tracked update (regardless if the version or metadata payload actually changed - reasoning behind this is to get reflected metadata updates independent if version attribute has changed).

State management of tracked updates:

On first creation, state is set to pending.
When an update is in approved state, an invocation for it resets its state to pending.
Ignored updates are skipped entirely and no attribute is updated.

The `application` attribute

The application attribute is an arbitrary identifier, name or label of a subject you like to track, e.g., docker.io/varakh/upda for an OCI image.

The `provider` attribute

The provider attribute is an arbitrary name or label. During webhook invocation the provider attribute is derived in priority:

For the generic webhook:

If the incoming payload contains a non-blank provider attribute, it's taken from the request.
If the incoming payload contains a blank or missing provider attribute, the issuing webhook's label is taken.

For the diun webhook:

If the issuing webhook's label is blank, then oci is used.
In any other case, the webhook's label is used.

Because the first priority is the issuing webhook's label, setting the same label for all webhooks results in a grouping. Also see the ignore host setting for host below.

Remember that changing a webhook's label won't be reflected in already created/tracked updates!

The `host` attribute

host should be set to the originating host name a webhook has been issued from. The host attribute can also be "ignored" (a setting in each webhook). If set to ignored, upda sets host to global, thus update versions can be grouped independent of the originating host. If set for all webhooks, you'll end up with a host independent update dashboard.

The `version` attribute

The version attribute is an arbitrary name or label and subject to change across invocations of webhooks. This can be a version number, a number of total updates, anything.

The `metadata` attribute

An update can hold any additional metadata information provided by request payload metadata. Metadata can be inspected via web interface or API.

Configuration

The following environment variables can be used to modify application behavior.

Variable	Purpose	Default/Description
`TZ`	The time zone (recommended to set it properly, background tasks depend on it)	Defaults to `Europe/Berlin`, can be any time zone according to tz database
`ADMIN_USER`	Admin user name for login	Not set by default, you need to explicitly set it to user name
`ADMIN_PASSWORD`	Admin password for login	Not set by default, you need to explicitly set it to a secure random

`DB_TYPE`	The database type (Postgres is recommended)	Defaults to `sqlite`, possible values are `sqlite` or `postgres`
`DB_SQLITE_FILE`	Path to the SQLITE file	Defaults to `<XDG_DATA_DIR>/upda/upda.db`, e.g. `~/.local/share/upda/upda.db`
`DB_POSTGRES_HOST`	The postgres host	Postgres host address, defaults to `localhost`
`DB_POSTGRES_PORT`	The postgres port	Postgres port, defaults to `5432`
`DB_POSTGRES_NAME`	The postgres database name	Postgres database name, needs to be set
`DB_POSTGRES_TZ`	The postgres time zone	Postgres time zone settings, defaults to `Europe/Berlin`
`DB_POSTGRES_USER`	The postgres user	Postgres user name, needs to be set
`DB_POSTGRES_PASSWORD`	The postgres password	Postgres user password, needs to be set

`SERVER_PORT`	Port	Defaults to `8080`
`SERVER_LISTEN`	Server's listen address	Defaults to empty which equals `0.0.0.0`
`SERVER_TLS_ENABLED`	If server uses TLS	Defaults `false`
`SERVER_TLS_CERT_PATH`	When TLS enabled, provide the certificate path
`SERVER_TLS_KEY_PATH`	When TLS enabled, provide the key path
`SERVER_TIMEOUT`	Timeout the server waits before shutting down to end any pending tasks	Defaults to `1s` (1 second), qualifier can be `s = second`, `m = minute`, `h = hour` prefixed with a positive number
`CORS_ALLOW_ORIGIN`	CORS configuration	Defaults to `*`
`CORS_ALLOW_METHODS`	CORS configuration	Defaults to `GET, POST, PUT, PATCH, DELETE, OPTIONS`
`CORS_ALLOW_HEADERS`	CORS configuration	Defaults to `Authorization, Content-Type`

`LOGGING_LEVEL`	Logging level. Possible are `debug`, `info`, `warn`, `error`, `dpanic`, `panic`, `fatal`. Setting to `debug` enables high verbosity output.	Defaults to `info`
`LOGGING_ENCODING`	Logging encoding. Possible are `console` and `json`	Defaults to `json`

`WEBHOOKS_TOKEN_LENGTH`	The length of the token	Defaults to `16`, positive number

`TASK_UPDATE_CLEAN_STALE_ENABLED`	If background task should run to do housekeeping of stale (ignored/approved) updates from the database	Defaults to `false`
`TASK_UPDATE_CLEAN_STALE_INTERVAL`	Interval at which a background task does housekeeping by deleting stale (ignored/approved) updates from the database	Defaults to `1h` (1 hour), qualifier can be `s = second`, `m = minute`, `h = hour` prefixed with a positive number
`TASK_UPDATE_CLEAN_STALE_MAX_AGE`	Number defining at which age stale (ignored/approved) updates are deleted by the background task (updatedAt attribute decides)	Defaults to `168h` (168 hours = 1 week), qualifier can be `s = second`, `m = minute`, `h = hour` prefixed with a positive number
`TASK_EVENT_CLEAN_STALE_ENABLED`	If background task should run to do housekeeping of stale (old) events from the database	Defaults to `false`
`TASK_EVENT_CLEAN_STALE_INTERVAL`	Interval at which a background task does housekeeping by deleting stale (old) events from the database	Defaults to `8h` (8 hours), qualifier can be `s = second`, `m = minute`, `h = hour` prefixed with a positive number
`TASK_EVENT_CLEAN_STALE_MAX_AGE`	Number defining at which age stale (old) events are deleted by the background task (updatedAt attribute decides)	Defaults to `2190h` (2190 hours = 3 months), qualifier can be `s = second`, `m = minute`, `h = hour` prefixed with a positive number
`TASK_PROMETHEUS_REFRESH_INTERVAL`	Interval at which a background task updates custom metrics	Defaults to `60s` (60 seconds), qualifier can be `s = second`, `m = minute`, `h = hour` prefixed with a positive number

`LOCK_REDIS_ENABLED`	If locking via REDIS (multiple instances) is enabled. Requires REDIS. Otherwise uses in-memory locks.	Defaults to `false`
`LOCK_REDIS_URL`	If locking via REDIS is enabled, this should point to a resolvable REDIS instance, e.g. `redis://<user>:<pass>@localhost:6379/<db>`.

`PROMETHEUS_ENABLED`	If Prometheus metrics are exposed	Defaults to `false`
`PROMETHEUS_METRICS_PATH`	Defines the metrics endpoint path	Defaults to `/metrics`
`PROMETHEUS_SECURE_TOKEN_ENABLED`	If Prometheus metrics endpoint is protected by a token when enabled (recommended)	Defaults to `true`
`PROMETHEUS_SECURE_TOKEN`	The token securing the metrics endpoint when enabled (recommended)	Not set by default, you need to explicitly set it to a secure random

3rd party integrations

Webhooks

This is the core mechanism of upda and why it exists. Webhooks are the central piece of how upda gets notified about updates.

In order to configure a 3rd party application like duin to send updates to upda with the duin webhook notification configuration, create a new upda webhook token via upda's web interface or via API call. This gives you

a unique upda URL to configure in the notification part of duin, e.g., /api/v1/webhooks/<a unique identifier>
a corresponding token for the URL which must be sent as X-Webhook-Token header when calling upda's URL

Expected payload is derived from the type of the webhook which has been created in upda.

Example for duin Webhook notification notif:

notif:
    webhook:
        endpoint: https://upda.domain.tld/api/v1/webhooks/ee03cd9e-04d0-4c7f-9866-efe219c2501e
        method: POST
        headers:
            content-type: application/json
            X-Webhook-Token: <the token from webhook creation in upda>
        timeout: 10s

Prometheus Metrics

When PROMETHEUS_ENABLED is set to true, default metrics about memory utilization, but also custom metrics specific to upda are exposed under the PROMETHEUS_METRICS_PATH endpoint.

A Prometheus scrape configuration might look like the following if PROMETHEUS_SECURE_TOKEN_ENABLED is set to true.

scrape_configs:
  - job_name: 'upda'
    static_configs:
      - targets: ['upda:8080']
    bearer_token: 'VALUE_OF_PROMETHEUS_SECURE_TOKEN'

Custom exposed metrics are exposed under the upda_ namespace.

Examples:

# HELP upda_updates details for all updates, -1=deleted (deleted next restart), 0=pending, 1=approved, 2=ignored
upda_updates{application="codeberg.org/forgejo/forgejo",host="myserver",provider="oci"} 0
upda_updates{application="docker.io/library/mysql",host="myserver",provider="oci"} 2
upda_updates{application="quay.io/navidys/prometheus-podman-exporter",host="myserver",provider="oci"} 1
upda_updates{application="quay.io/navidys/prometheus-podman-exporter",host="myserver2",provider="oci"} 1
# HELP upda_updates_all amount of all updates
upda_updates_all 4
# HELP upda_updates_approved amount of all updates in approved state
upda_updates_approved 2
# HELP upda_updates_ignored amount of all updates in ignored state
upda_updates_ignored 1
# HELP upda_updates_pending amount of all updates in pending state
upda_updates_pending 1
# HELP upda_webhooks amount of all webhooks
upda_webhooks 2
# HELP upda_events amount of all events
upda_events 146

There's an example Grafana dashboard in the _doc/ folder.

Alertmanager could check for the following:

- name: update_checks
  rules:
    - alert: UpdatesAvailable
      expr: upda_updates == 0 and upda_updates_pending > 0
      for: 4w
      labels:
        severity: high
        class: update
      annotations:
        summary: "Updates available from upda for {{ $labels.job }}"
        description: "Updates available from upda for {{ $labels.job }}"

Deployment

Native

Use the released binary for your platform or run make clean build-server-{your-platform} and the binary will be placed into the bin/ folder.

Docker

For examples how to run, look into deployment instructions which contains examples for docker-compose files.

Build docker image

To build docker images, do the following

docker build --rm --no-cache -t upda:latest .

Development & contribution

Ensure to set the following environment variables for proper debug logs during development

DEVELOPMENT=true
LOGGING_ENCODING=console
LOGGING_LEVEL=debug

Code guidelines
- Each entity has its own repository
- Each entity is only used in repository and service (otherwise, mapping happens)
- Presenter layer is constructed from the entity, e.g., in REST responses and mapped
- No entity is directly returned in any REST response
- All log calls should be handled by zap.L()
- Configuration is bootstrapped via separated struct types which are given to the service which need them
- Error handling
  - Always throw an error with NewServiceError
  - Always wrap the cause error with fmt.Errorf
  - Forward/bubble up the error directly, when original error is already a NewServiceError (most likely internal calls)
  - Always abort handler chain with AbortWithError
  - Utils can throw any error

Please look into the _doc/ folder for OpenAPI specification and a Postman Collection.

Getting started

Run make clean dependencies to fetch dependencies
Start git.myservermanager.com/varakh/upda/cmd/server (or cli) as Go application and ensure to have required environment variables set

If you like to test with PSQL and/or REDIS for task locking, here are some useful docker commands to have containers up and running quickly. Set necessary environment variables properly.

# postgres
docker run --rm --name=upda-db \
  -p 5432:5432 \
  --restart=unless-stopped \
  -e POSTGRES_USER=upda \
  -e POSTGRES_PASSWORD=upda \
  -e POSTGRES_DB=upda \
  postgres:16-alpine
  
# redis
docker run --rm --name some-redis \
  -p 6379:6379 \
  redis redis-server --save 60 1 --loglevel warning

Windows hints

On Windows, you need a valid gcc, e.g., https://jmeubank.github.io/tdm-gcc/download/ and add the \bin folder to your path.

For any go command you run, ensure that your PATH has the gcc binary and that you add CGO_ENABLED=1 as environment.

Release

Releases are handled by the SCM platform and pipeline. Creating a new git tag, creates a new release in the SCM platform, uploads produced artifacts to that release and publishes docker images automatically. Before doing so, please ensure that the commit on master has the correct version settings and has been built successfully:

Adapt constants_app.go and change Version to the correct version number
Adapt CHANGELOG.md to reflect changes and ensure a date is properly set in the header, also add a reference link in footer (link to scm git tag source)
Adapt api.yaml: version attribute must reflect the to be released version
Adapt env: VERSION_* in .forgejo/workflows/release.yaml

After the release has been created, ensure to change the following settings for the next development cycle:

Adapt constants_app.go and change Version to the next version number
Adapt CHANGELOG.md and add an UNRELEASED section
Adapt api.yaml: version attribute must reflect the next version number
Adapt env: VERSION_* in .forgejo/workflows/release.yaml to next version number

27 KiB Raw Blame History