Backup and Disaster Recovery

Backups aren't "zip up the server". They're grading by importance, then choosing a strategy per grade.

Start with one list — the single step you can't skip:

| Data | What losing it costs | Grade |
|---|---|---|
| Site source files or build output | Rebuildable, but takes time | Medium |
| data/*.json content | The whole site's content; losing it undoes the work | High |
| Database (if any) | User data, not reproducible | Highest |
| Server / Nginx config | Tedious to redo | Medium |
| Logs | Nobody cares | Low |

Strategy per grade:

• Highest: daily full backup plus hourly incremental, two off-site copies, and a restore you have rehearsed
• High: daily backup, 30-day retention, one off-site copy
• Medium: weekly backup, 4 weeks retention
• Low: no backup, or whatever the server snapshot covers

The key insight: a backup you have never restored from is not a backup. Plenty of people discover during an incident that the backup job has been failing silently for six months.

Compliance: encrypt any backup containing personal data, and state the retention period.

A backup script does four things: archive, ship, log, and alert on failure.

#!/bin/bash
# backup.sh · runs daily at 03:10
set -u
SRC="/www/wwwroot/yourdomain"
DATE=$(date +%Y%m%d)
OUT="/tmp/site-$DATE.tar.gz"
tar -czf "$OUT" -C "$(dirname $SRC)" "$(basename $SRC)" || { echo "archive failed"; exit 1; }
coscli cp "$OUT" cos://your-bucket/backup/ || { echo "upload failed"; exit 1; }
echo "$DATE ok $(du -h $OUT | cut -f1)" >> /var/log/backup.log
find /tmp -name 'site-*.tar.gz' -mtime +3 -delete

Scheduling: panel → scheduled tasks → shell script → daily 03:10; or add a line with crontab -e.

Three hard rules:

1. Off-site: a backup that stays on the same machine dies with it when the disk dies
2. Alert on failure: send a notification — a chat webhook or email — when the script fails. Silent failure is the worst failure
3. Keep several copies: at least 7 historical versions. Today's backup might be broken, and so might yesterday's

Choosing object storage: any S3-compatible service is fine. Turn on versioning to protect against accidental deletion and ransomware.

90% of people skip this step, and then discover during the real incident that a config file was missing from the archive.

How to drill (once a quarter, one hour):

1. Get a clean environment: spin up a cheap pay-as-you-go server, or use a local VM or Docker
2. Restore only from the backup files — pretend the original server exploded and copy nothing from it
3. Time the whole thing: record how long each stage takes (download, extract, restore data, edit config, start, verify)
4. Verify the site works: click through the pages. Not "the files exist" — actually usable
5. Write it up: total restore time / where you got stuck / what the backup was missing

What a drill always exposes (expect these):

• Nginx config wasn't included → add it
• The database wasn't backed up, or versions don't match → add a logical dump
• Secrets and environment variables aren't in the backup → store them separately, encrypted
• Restore took far longer than expected → consider a server image or snapshot

Metric: write down a recovery time objective — say 2 hours. If the drill exceeds it, improve the plan.

Tear the environment down afterwards. Don't leave a server running and forget about it.

Monitoring (the backup itself must be monitored):

• There must be a success line in the backup log for today — no line means alert (see the Server Monitoring Dashboard scenario)
• Alert on sudden size changes in the stored backup: 50 MB yesterday, 1 KB today is almost always a failed archive
• Once a month, download a random backup and extract it to check the files are complete. Cheap and effective

Cost:

| Item | How to keep it cheap |
|---|---|
| Storage | Lifecycle rules: move to infrequent access after 30 days, delete after 90 |
| Requests | Don't sync every minute; once a day is plenty |
| Traffic | Upload over the internal network or within the same region — usually free |

Cleanup policy (bake it into the script):

• Keep local temp archives for 3 days
• Object storage: daily backups for 30 days, weekly for 12 weeks, monthly for 12 months
• Don't keep everything forever: unbounded retention is pure cost, and holding personal data indefinitely is itself a risk

Finally: write the backup policy, the restore procedure and the contact person on one page, and put it somewhere others can find — not only on your own laptop. During a real incident you need instructions to follow, not memories to reconstruct.