Adding Observability to `backend/` on Grafana Cloud

[dudu-theman]

Starting point

• telemetry/app/ already proves out the pattern locally (OTel + grafana/otel-lgtm).
• This plan reuses that same instrumentation approach, pointed at Grafana Cloud instead of a local collector.

High level Steps

1. Sign up for Grafana Cloud account, free tier for now
2. Point the Otel SDK at Grafana Cloud instead of docker-otel-lgtm
3. Add some logging into the backend app

In-detail Steps

1. Grafana Cloud account — sign up for free tier, grab the OTLP endpoint + instance ID + API key.
2. Pick what to collect — traces (per-request + Athena/S3 calls), logs (structured, key identifiers only), metrics (RED: rate/errors/duration, auto-instrumented).
3. Add OTel dependencies — same packages as telemetry/app, plus a botocore instrumentation package since backend/ calls AWS via boto3.
4. Point the SDK at Grafana Cloud — env vars for OTLP endpoint, auth header, service name. No local collector needed.
5. Instrument the app — auto-instrumentation covers most of it; add a couple of manual spans around the Athena query / S3 presign in posts.py, plus light request logging.
6. Verify against Grafana Cloud — run locally, hit each endpoint (including an error path), confirm traces/logs/metrics show up in the hosted Grafana instance.
7. Basic dashboard + one alert — request rate, error rate, latency per route; alert on error rate spikes.
8. Document it — add a runbook alongside the existing docs/runbooks/demo-app-*.md ones.

[coderabbitai]

🔗 Related PRs

#24 - Feat/v1 telemetry stack [closed]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[mark-torres10]

LGTM as for the basic shape @dudu-theman , LMK when the PR is up so I can take a closer look.