RugMunchMedia/rmi-backend
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
rmi-backend/scripts/ops/README.md

2.6 KiB
Raw Permalink Blame History

ops/ — Server-side operations scripts

Scripts that run on netcup (not in the FastAPI container). Deploy via:

scp scripts/ops/backup.sh        root@netcup:/root/scripts/backup.sh
scp scripts/ops/restore_test.sh  root@netcup:/root/scripts/restore_test.sh
ssh netcup "chmod +x /root/scripts/*.sh"

backup.sh

Daily backup of all RMI data + system configs to /backups/<timestamp>/.

What it backs up:

  1. Postgres (postgres.sql.gz)
  2. Neo4j (neo4j.dump)
  3. Qdrant (qdrant/<collection>/<snapshot>)
  4. MinIO (minio/)
  5. GlitchTip Postgres (glitchtip.sql.gz)
  6. System configs (etc_root.tar.gz) — DR-critical, ~19 MB, 2440 files:
    • /etc/prometheus/ — alert rules, prometheus.yml, alertmanager configs
    • /root/scripts/ — this backup script, restore_test, cron health check
    • /root/.hermes/ — Hermes gateway config (NOT sessions — too large)
    • /root/.ssh/ — SSH keys for git/auth
    • /root/.bashrc, /root/.profile — shell config
    • /root/backend/.env — backend secrets (gopass-backed in source)
  7. Old backups rotated (>7 days deleted)
  8. ntfy notification on success

Cadence: Runs via cron (daily, see crontab -l on netcup).

Restore test: /root/scripts/restore_test.sh spins up test Postgres on alt port 15432, restores, verifies table count ≥ 10, samples row counts, now also validates etc_root.tar.gz contains etc/prometheus/prometheus.yml.

restore_test.sh

Monthly restore-test cron (1st of month) that proves a backup can actually be restored, not just written.

What it checks:

  1. Postgres table count ≥ 10 (fail if too few)
  2. Row counts on tokens, wallets, news_items tables
  3. etc_root.tar.gz contains etc/prometheus/prometheus.yml
  4. etc_root.tar.gz file count > 1000 (catches truncated backups)
  5. Notifies via ntfy topic rmi-critical on failure, rmi-info on success

Cron health check (cron_health_check.py)

Python watchdog that monitors cron job health and auto-restarts failed crons.

T10 (RMIV5) — Backup /etc + /root

Implemented in backup.sh step 6 (etc_root tarball). Verified 2026-06-22:

  • Tarball: 19 MB, 2440 files
  • Validates etc/prometheus/prometheus.yml exists
  • Excludes: .cache, .ollama, .hermes/sessions, .cargo, .rustup
  • Restore test validates tarball contents before declaring success

Bug found and fixed 2026-06-22: Original script tried to back up /etc/caddy/Caddyfile which does not exist on netcup (we use nginx, not caddy). The validation silently passed because grep returned nothing. Fixed to validate /etc/prometheus/prometheus.yml instead, which actually exists and is the file we need for DR.