[Runbook] Create comprehensive operational runbooks

[OneByJorah]

Runbook Gap

Current Issues:

1. No operational runbooks - Only minimal MAINTENANCE.md (19 lines)
2. No incident response procedures - No runbooks for common failure scenarios
3. No backup/restore procedures - Only one-line tar command in MAINTENANCE.md
4. No disaster recovery plan - No RTO/RPO defined
5. No capacity planning - No scaling guidelines
6. No on-call procedures - No escalation paths
7. No deployment runbook - Deploy is automated but no manual fallback
8. No security incident runbook - No breach response
9. No service-specific runbooks - Each service needs its own ops guide

Required Runbooks:

1. Bootstrap/Deployment Runbook

   • Fresh server setup
   • Production deployment
   • Rollback procedures
   • Blue-green deployment

2. Service Operations Runbooks (one per service)

   • SearXNG: reindex, update instances, debug search quality
   • Camofox: browser restart, session cleanup, resource tuning
   • Honcho: migration, backup, token rotation, LLM provider switching
   • Qdrant: snapshot, restore, collection management
   • Obsidian: vault backup, plugin management, permission fixes
   • Portainer: backup/restore, RBAC management

3. Incident Response Runbooks

   • Service down troubleshooting
   • High memory/CPU investigation
   • Disk space exhaustion
   • Network connectivity issues
   • Database corruption recovery
   • Secret compromise response

4. Maintenance Runbooks

   • Scheduled maintenance windows
   • Base image updates
   • Security patching
   • Certificate renewal (if TLS added)
   • Log rotation
   • Backup verification

5. Disaster Recovery Runbook

   • RTO: 4 hours, RPO: 1 hour (define targets)
   • Full stack restore from backup
   • Partial service restore
   • Cross-region failover (if applicable)

6. Security Runbook

   • Vulnerability patching
   • Secret rotation
   • Access review
   • Audit logging review

Format:

• Markdown with clear steps
• Prerequisites
• Expected outcomes
• Rollback steps
• Contact/escalation info
• Runbook testing schedule