[Runbook] Operational runbooks: Incident response, deployment, backup, and maintenance procedures

[OneByJorah]

Runbook Checklist

• Create docs/runbooks/ directory
• Incident Response Runbooks:
  • Service down: SearXNG
  • Service down: Camofox
  • Service down: Honcho API
  • Service down: PostgreSQL/Honcho DB
  • Service down: Redis
  • Service down: Qdrant
  • Service down: Obsidian
  • Service down: Qdrant
  • High CPU/Memory/Disk usage
  • Disk full
  • Network connectivity issues
  • Secret compromise / rotation emergency
  • Data corruption / recovery
• Deployment Runbooks:
  • Initial deployment (bootstrap)
  • Rolling update / zero-downtime deploy
  • Rollback procedure
  • Blue-green deployment (if applicable)
  • Database migration procedures
• Backup & Restore Runbooks:
  • PostgreSQL backup (pg_dump / pg_basebackup)
  • Redis backup (RDB/AOF)
  • Qdrant backup (snapshots)
  • Obsidian vault backup
  • SearXNG settings backup
  • Full stack backup procedure
  • Restore procedures for each service
  • Disaster recovery test schedule
• Maintenance Runbooks:
  • Routine maintenance (apt update, docker image pull, reboot)
  • Certificate renewal (if using TLS)
  • Log rotation
  • Database vacuum/analyze
  • Dependency updates (monthly)
  • Security patching schedule
• Runbook template standardization:
  • Standard format: Overview, Prerequisites, Steps, Verification, Rollback, Contacts
  • Link runbooks from monitoring alerts
  • Regular runbook review schedule (quarterly)
• Add runbook index: docs/runbooks/README.md