git clone git://git.garbash.com/alex/www
Log | Files | Refs | README | LICENSE

commit 8ea58519766d91da48052999e92db28b2b4e2947
parent 3b7728a0ab1ec0ed188cedaf1a4b99b0b64fe0f2
Author: alex <alex@garbash.com>
Date:   Wed, 13 Oct 2021 23:24:18 -0400

notes: Add notes on backups with dumpster(8)

This was a ton of fun to write!

Mindex.html | 2+-
Anotes/011-backups.txt | 65+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/index.html b/index.html @@ -22,7 +22,6 @@ this site along with my field notes on how I set each thing up! <h3>TODO:</h3> <ul> <li>Wildcard cert for internal services</li> -<li>Basic backup solution</li> <li>Start inviting people</li> <li>Automate account creation</li> <li>Homepage</li> @@ -44,6 +43,7 @@ this site along with my field notes on how I set each thing up! <li><a href="notes/008-local-irc.txt">Set up IRC for tilde members</a></li> <li><a href="notes/009-wireguard.txt">Set up wireguard</a></li> <li><a href="notes/010-irc-bouncer.txt">Set up IRC bouncer</a></li> +<li><a href="notes/011-backups.txt">Basic backup solution</a></li> </ul> </body> </html> diff --git a/notes/011-backups.txt b/notes/011-backups.txt @@ -0,0 +1,65 @@ +011-backups -- October 12, 2021 + +My usual take on server backups is "don't put anything worth +backing up on the server that's not stored in git elsewhere". + +This has treated me pretty well in the past. Source code, +configuration files, and even documentation on setup are all +stored in git both on the server and on my laptop, and so +I can sleep at night knowing a catastrophic disk failure wouldn't +mean I lost any serious work. + +This strategy breaks down when thinking about a tilde. First, +the array of services we're providing is _much_ more complex +than my normal blog server. Second, there are more people +involved! + +I want to guarantee any tilde members that I will at least try +my best to keep backups of their data in case of failure or +accidental deletion. + +There are tons of backup tools, but a lot of them are fairly +complex (with good reason I suppose.. compression, deduplication, +etc). Since this tilde is about exploring OpenBSD, I took the +opportunity to home-roll a simple backup solution with dump(8) +and restore(8). + +The meat of it is in a script I'm calling "dumpster" that runs +via cron with the day of the week (1-7) as the dump level +and a weekly job dumping the whole system fresh: + + #!/bin/sh + # dumpster -- taking out the garbash with dump(8) + + # %u is 1=mon 7=sun (unless given in ARGV) + LVL=${1:-"$(date +%u)"} + BAKDIR="/bak/$(date +%F)_$LVL" + + mkdir -p "$BAKDIR" + dump -$LVL -auf "$BAKDIR/root.dump.$LVL" / + dump -$LVL -auf "$BAKDIR/home.dump.$LVL" /home + dump -$LVL -auf "$BAKDIR/var.dump.$LVL" /var + +This dumps to /bak, which is a separate Linode Volume, which +has better data redundancy guarantees than the VPS volume and +can be detached/attached to other hosts in the event of VPS +failure. + +As you can see, I'm only really dumping areas that have user +data (/var for git, /home, and / for configs). /usr/* can be +rebuilt from /var/backups/pkglist for the most part! + +A note to anyone trying this: the Linode Volume was a bit hacky +to get set up, since it expects to be mounting against a Linux +machine. Linode's console will error on attaching, but I found +that rebooting the host made the drive appear as wdN and from +there I was able to format it, etc. + +As a bonus, I took the opportunity to set up /altroot backups, +which is a brilliantly simple way to ensure you can boot into +a known-good state of your host even if something goes very +wrong with the main drive! + +Overall, I went from a backup-avoider to a backup-fan in the +process :) it's so cool to watch the daily script create dump +files of things that changed!