Previously, all container volumes initialized via the `container-volume`
service would have local rotating backups performed by pushing `tar`
archives to the `/var/lib/backups/coreos-home-server` directory.
This proved to be a simple and effective mechanism for storing historic
volume state locally; however, the use-case for historic backup is
usually data loss, either partial (e.g. by deleting files inadvertently)
or complete (e.g. by loss of disk), which is likely better mitigated by
more concrete mechanisms of retention.
In addition, this need to store historic volume state locally, in its
totality, proved to be a barrier for performing partial backups, which
is an issue especially for larger volumes.
This commit deprecates this simple, generic volume backup/restore
mechanism, and instead has us rely directly on Rclone reading from the
volume in question.
Container builds using the `container-build@` systemd service will
generally tag any container image built with the `latest` tag, which is
then referred to pervasively in container executions.
However, this tag is overwritten when building new images, and, combined
with how `podman auto-update` will prune old image digests, may cause us
to lack the ability to roll back, automatically or otherwise.
This commit sets a `previous` tag on container re-builds, which should
only generally happen when source files change (due to the `ExecCondition`)
present on the service, which in turn should ensure that images are not
spuriously tagged as such.
This commit adds an `ExecCondition` directive on the `container-build@`
service, used as a pre-requisite for all other Podman-based services,
skipping `podman build` invocations unless local `Containerfile` or any
files in the `container` sub-directories have changed.
Container builds are responsible for the majority of time taken during
boot, even with cache in place; this will help alleviate pressure and
hopefully speed up boot considerably.
Container auto-updates are scheduled for 30 minutes past every hour, or
approximately 30 minutes after `coreos-home-server-update` runs, in
order to give enough time for container builds to complete; only
containers with auto-updates enabled are eligible, however.
This also enables health-checks for Redis, and updates the version to
7.2.
Podman-generated systemd unit files aren't usually generated into
`/etc/systemd/system`, and this directory is preferred for any
pre-existing unit files.
This commit extends our `coreos-home-server-update` script with support
for updating host directories with configuration collected across
multiple remote directories of the same name. This will, essentially,
allow for extending systemd services with custom configuration, as
sometimes required of base systemd service files.
Container volume backup logic has been moved to a (largely equivalent)
external script, allowing for future expansion of functionality. In
addition, a `rclone-pull@` service has been added and set up as a default
dependency for the `container-volume-restore@` service, allowing for
automatic set up of servers based on latest remote backups.
Pointers to the latest backup are useful for restore operations, but
should be ignored in all other cases, and thus are more appropriately
handled as symbolic links.
We no longer copy service directories into `/etc/coreos-home-server` if
these have not had their respective `spec.bu` files included; these
directories are not needed in these cases, and would be erroneously
considered as eligible in subsequent `coreos-home-server-update`
invocations.
Container volume backups will now be skipped if no changes have been
made in source files against the latest backup. In addition the default
timer has been changed for performing backups once a day, at 02:00,
likely a time where there's less traffic on the server.
This commit contains a fairly large diff for a fairly small change:
moving the `config/common` directory to `host/base` to better reflect
its intended use, and promoting `config/service` to the root directory.
These changes unlock some improvements in `coreos-home-server-update`
processes, which will (assuming `/etc/coreos-home-server/base` exists)
keep host-wide systemd services in sync in addition to service-specific
ones.
Changes have been make to the `Makefile` and a few other places where
`config/common` was referenced, but most of this work is renames that
are not intended to break compatibility with new or running servers.
Alex Palaistras