Snapshot before you change things

Most server disasters start with a change that was supposed to be small. A restore point taken five minutes earlier turns a lost evening into a rollback. Here are four kinds, from whole-disk to single files, and how to know they work.

Why bother

A package upgrade, a firewall rule, an edit to sshd_config: each one can leave the server unbootable, unreachable or broken in a way that takes hours to trace. If you have a restore point from just before the change, you don't have to trace it. You go back and try again more carefully.

Pick the restore point to match the change. A kernel or distribution upgrade needs the whole disk. A config edit needs only the files you touch. Often it's worth having both.

Provider snapshots

Most VPS and cloud providers can snapshot a server's disk from their control panel or API. It captures everything, it's the easiest to restore, and it lives outside the server, so it survives even if the server won't boot. The exact steps and costs differ by provider; look for "snapshot" or "backup" in your provider's documentation.

Things to know before you rely on one:

LVM snapshots

If the server's filesystems sit on LVM (common on installs you set up yourself, rare on provider images), you can snapshot a logical volume from the server itself. Check first:

sudo lvs
sudo vgs   # VFree must have room for a snapshot

A classic (copy-on-write) snapshot needs free space in the volume group. It stores the old version of every block that changes after the snapshot. The lvcreate man page says 20% of the origin is often enough:

sudo lvcreate --snapshot --size 5G --name pre-upgrade vg0/root

Watch how full it gets while you work. The Data% column in lvs shows usage, and lvextend can grow it if space runs low. Thin-provisioned volumes take snapshots without a size: lvcreate --snapshot -n pre-upgrade vg0/thinroot.

To roll back, merge the snapshot into the origin:

sudo lvconvert --mergesnapshot vg0/pre-upgrade

For a volume that's in use, like the root filesystem, the merge waits until the next time the volume is activated, which normally means a reboot. When the merge finishes, the snapshot is removed. If the change went fine, remove the snapshot yourself with lvremove vg0/pre-upgrade so it stops using space in the volume group.

An LVM snapshot lives on the same disks as the original. It protects you from a bad change, not from a failed disk or a deleted server.

Copy the config you're about to touch

For most changes, the files in /etc are what you need back. A tar archive of /etc takes seconds:

sudo tar --xattrs --acls -czpf /root/etc-$(date +%F-%H%M).tar.gz -C / etc

-p keeps permissions, and when root extracts the archive, ownership is restored too. The archive contains secrets (/etc/shadow, private keys, passwords in config files), so keep it readable by root only, and don't copy it anywhere less protected than the server.

To get one file back, list the archive and extract just that path:

sudo tar -tzf /root/etc-2026-10-30-1400.tar.gz | grep sshd_config
sudo tar -xzpf /root/etc-2026-10-30-1400.tar.gz -C /tmp etc/ssh/sshd_config
sudo diff /tmp/etc/ssh/sshd_config /etc/ssh/sshd_config

For a single file, a plain copy is enough: sudo cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.

etckeeper: history for /etc

etckeeper keeps /etc in a version control repository and commits automatically before and after apt installs or upgrades packages. That gives you a history of every config change, including the ones packages made.

sudo apt install etckeeper
sudo etckeeper vcs status          # is there a repository yet?
sudo etckeeper init                # only if there isn't
sudo etckeeper commit "before ssh changes"

After your change, see exactly what moved with sudo etckeeper vcs diff. Commit before you start each change with a message that says why, so you can find the point to go back to later.

A restore point you haven't tested is a hope. Snapshots fail to restore and archives turn out to be empty. Test each method once, on a server you can afford to break, before you need it on one you can't.

Test that you can restore

Then make the change, and keep a way back in while you do it: a second SSH session, or your provider's web console. For SSH changes in particular, see SSH hardening without locking yourself out.

Stenvakt takes the snapshot for you. Its fixes run in safe mode: a snapshot first, then a rollback timer, then a test over a fresh connection, and the change is undone if you can't get back in. You see each fix in a Preview before it runs. It's a Mac app and isn't out yet: get early access.

Sources

  1. DigitalOcean docs: live snapshots and data consistency (an example of provider guidance)
  2. lvcreate(8): creating COW and thin snapshots
  3. lvconvert(8): --mergesnapshot
  4. Debian manpages: etckeeper(8)
  5. Debian manpages: tar(1)
Next guideWho tried to log in over SSH