Snapshot before you change things
Most server disasters start with a change that was supposed to be small. A restore point taken five minutes earlier turns a lost evening into a rollback. Here are four kinds, from whole-disk to single files, and how to know they work.
Published . Generic Linux advice; the LVM and etckeeper commands assume Debian or Ubuntu with the lvm2 and etckeeper packages.
- Why bother
- Provider snapshots
- LVM snapshots
- Copy the config you're about to touch
- etckeeper: history for /etc
- Test that you can restore
Why bother
A package upgrade, a firewall rule, an edit to sshd_config: each one can leave the server unbootable, unreachable or broken in a way that takes hours to trace. If you have a restore point from just before the change, you don't have to trace it. You go back and try again more carefully.
Pick the restore point to match the change. A kernel or distribution upgrade needs the whole disk. A config edit needs only the files you touch. Often it's worth having both.
Provider snapshots
Most VPS and cloud providers can snapshot a server's disk from their control panel or API. It captures everything, it's the easiest to restore, and it lives outside the server, so it survives even if the server won't boot. The exact steps and costs differ by provider; look for "snapshot" or "backup" in your provider's documentation.
Things to know before you rely on one:
- A live snapshot can catch data half-written. Providers allow snapshots of a running server, but warn that databases don't guarantee a consistent state on disk unless they're stopped. Power off from the command line first if you can, or at least stop the database, or take a proper database dump as well.
- Check what restoring does. If it replaces the current disk, anything written after the snapshot is gone.
- Check what's included. If you have extra block volumes attached, find out whether the snapshot covers them or only the main disk.
- They aren't instant. Wait until the panel says the snapshot is complete before you start the change.
LVM snapshots
If the server's filesystems sit on LVM (common on installs you set up yourself, rare on provider images), you can snapshot a logical volume from the server itself. Check first:
sudo lvs
sudo vgs # VFree must have room for a snapshot
A classic (copy-on-write) snapshot needs free space in the volume group. It stores the old version of every block that changes after the snapshot. The lvcreate man page says 20% of the origin is often enough:
sudo lvcreate --snapshot --size 5G --name pre-upgrade vg0/root
Watch how full it gets while you work. The Data% column in lvs shows usage, and lvextend can grow it if space runs low. Thin-provisioned volumes take snapshots without a size: lvcreate --snapshot -n pre-upgrade vg0/thinroot.
To roll back, merge the snapshot into the origin:
sudo lvconvert --mergesnapshot vg0/pre-upgrade
For a volume that's in use, like the root filesystem, the merge waits until the next time the volume is activated, which normally means a reboot. When the merge finishes, the snapshot is removed. If the change went fine, remove the snapshot yourself with lvremove vg0/pre-upgrade so it stops using space in the volume group.
An LVM snapshot lives on the same disks as the original. It protects you from a bad change, not from a failed disk or a deleted server.
Copy the config you're about to touch
For most changes, the files in /etc are what you need back. A tar archive of /etc takes seconds:
sudo tar --xattrs --acls -czpf /root/etc-$(date +%F-%H%M).tar.gz -C / etc
-p keeps permissions, and when root extracts the archive, ownership is restored too. The archive contains secrets (/etc/shadow, private keys, passwords in config files), so keep it readable by root only, and don't copy it anywhere less protected than the server.
To get one file back, list the archive and extract just that path:
sudo tar -tzf /root/etc-2026-10-30-1400.tar.gz | grep sshd_config
sudo tar -xzpf /root/etc-2026-10-30-1400.tar.gz -C /tmp etc/ssh/sshd_config
sudo diff /tmp/etc/ssh/sshd_config /etc/ssh/sshd_config
For a single file, a plain copy is enough: sudo cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.
etckeeper: history for /etc
etckeeper keeps /etc in a version control repository and commits automatically before and after apt installs or upgrades packages. That gives you a history of every config change, including the ones packages made.
sudo apt install etckeeper
sudo etckeeper vcs status # is there a repository yet?
sudo etckeeper init # only if there isn't
sudo etckeeper commit "before ssh changes"
After your change, see exactly what moved with sudo etckeeper vcs diff. Commit before you start each change with a message that says why, so you can find the point to go back to later.
A restore point you haven't tested is a hope. Snapshots fail to restore and archives turn out to be empty. Test each method once, on a server you can afford to break, before you need it on one you can't.
Test that you can restore
- Provider snapshot: create a new server from it, rather than restoring over the original. Check that it boots and your services start. Then delete the test server.
- tar archive: list it with
tar -tzfand extract it into a scratch directory. If the list is short or empty, the backup isn't what you think. - etckeeper: run
sudo etckeeper vcs logand check that recent changes are there. - Database: restore the dump into a new, empty database. See database backups before changes.
Then make the change, and keep a way back in while you do it: a second SSH session, or your provider's web console. For SSH changes in particular, see SSH hardening without locking yourself out.
Stenvakt takes the snapshot for you. Its fixes run in safe mode: a snapshot first, then a rollback timer, then a test over a fresh connection, and the change is undone if you can't get back in. You see each fix in a Preview before it runs. It's a Mac app and isn't out yet: get early access.
Sources
- DigitalOcean docs: live snapshots and data consistency (an example of provider guidance)
- lvcreate(8): creating COW and thin snapshots
- lvconvert(8): --mergesnapshot
- Debian manpages: etckeeper(8)
- Debian manpages: tar(1)