Plan v0.3 hardening sequence before realtime/dashboard

[Justinabox]

v0.3 Hardening Sequence Implementation Plan

For Hermes: Use subagent-driven-development skill to implement this plan task-by-task.

Goal: Deliver v0.3 in a safe sequence: secure network write surfaces first, then complete robust SMS ingest/reassembly, then expose realtime/metrics without expanding the attack surface or hiding modem quirks.

Architecture: Keep Modem + service classes + typed EventBus as the core boundary. Treat server.py and any future WebSocket/dashboard as authenticated adapters over stable service events, not as places to invent modem state. Add SMS reassembly inside the SMS layer with deterministic mock-transport tests before requiring real hardware validation.

Tech Stack: Python 3.11, asyncio, aiohttp, aiosqlite, pyserial-asyncio, pytest, pytest-asyncio, pytest-aiohttp, mock transports.

---

Recent context

• README.md and ROADMAP.md now show v0.2 foundation as mostly complete: SIM PIN, API-key middleware/rate limiting, delivery report plumbing, DTMF send, USSD, SMS prompt handling, packaging discovery, BER descriptions, and multipart UDH parser groundwork.
• Open issues/PRs as of this planning pass:
  • Secure HTTP server startup instead of disabling auth by default #4 — secure HTTP server startup instead of disabling auth by default.
  • Handle VOICE CALL: BEGIN arriving during answer without double ACTIVE transition #5 / fix: handle answer active URC race #7 — answer-path VOICE CALL: BEGIN URC race; PR is open and clean but draft.
  • Prevent silent corruption of non-ASCII outbound SMS bodies #6 / fix: encode outbound SMS with GSM charset #8 — prevent silent non-ASCII SMS corruption; PR is open and clean but draft.
• Local mock suite on the current PR branch: git diff --check && PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q → 296 passed in 3.95s.

Dependency order

1. Land/review narrow correctness fixes first: fix: handle answer active URC race #7 and fix: encode outbound SMS with GSM charset #8 reduce real-modem state/SMS corruption risk and should stay ahead of feature expansion.
2. Resolve Secure HTTP server startup instead of disabling auth by default #4 before any realtime/dashboard/network expansion: WebSocket feeds and dashboards should inherit an auth model that is secure by default.
3. Do SMS multipart reality discovery before service integration: The existing UDH parser is useful, but text-mode modem behavior may hide UDH or auto-join parts. Confirm whether reliable reassembly requires PDU-mode receive support.
4. Add realtime/metrics only after event semantics are stable: WebSocket and Prometheus should expose cleaned service events/metrics, not raw partial modem lines with PII-heavy labels.
5. Add modem auto-detection/capability profiles before broader voice/PBX assumptions: Voice/audio commands and port assignments vary by device family.

Slice 1 — HTTP startup is secure by default

Objective: Close #4 and establish the policy future /ws and dashboard endpoints must follow.

Files likely touched:

• server.py
• tests/test_api_auth.py
• README.md or a new small HTTP server usage section

Acceptance criteria:

• Starting server.py on 0.0.0.0 without configured API keys fails closed, or explicitly binds to loopback-only in a documented dev mode.
• Tests cover: no-key non-loopback startup, loopback/dev behavior if allowed, authenticated SMS send, authenticated USSD send, and unauthenticated rejection.
• Documentation explains how to provide API keys without logging or committing them.
• No secrets, SIM numbers, or message bodies are printed in startup/auth errors.

Risk notes:

• This is the one place where a product/security choice may be needed: fail startup without keys versus permit loopback-only dev mode.
• Prefer fail-closed for Raspberry Pi deployments because write endpoints can send SMS/USSD.

Slice 2 — Finish outbound SMS charset boundary

Objective: Land #8 or equivalent so outbound SMS cannot silently mutate bodies.

Files likely touched:

• callstack/sms/service.py
• tests/test_sms_service.py
• README.md troubleshooting/SMS limitations section if not already updated

Acceptance criteria:

• GSM 03.38 basic and extension-table characters are sent as exact modem payload bytes.
• Unsupported UCS2-required bodies are rejected before AT+CMGS, with a clear error that PDU/UCS2 send is not implemented yet.
• Stored sent-message metadata cannot claim success for a body different from what was handed to the modem.
• Mock tests assert raw bytes for representative ASCII, accented GSM characters, extension-table characters, and rejected Unicode.

Risk notes:

• Rejecting UCS2 is safer than corruption, but it is not a complete international SMS solution. Track UCS2/PDU send as a future capability if real users need it.

Slice 3 — Multipart SMS receive/reassembly

Objective: Turn UDH metadata groundwork into one complete public incoming-message event per logical long SMS.

Files likely touched:

• callstack/sms/types.py
• callstack/sms/service.py
• callstack/sms/store.py
• callstack/sms/pdu.py
• tests/test_sms_service.py
• tests/test_sms_pdu.py

Acceptance criteria:

• A discovery/test task records whether the supported receive path exposes concatenation metadata in text mode; if not, the implementation uses PDU-mode receive for multipart messages.
• Parts are keyed by sender + concatenation reference + total parts, tolerate out-of-order arrival, and emit exactly one IncomingSMSEvent when complete.
• Partial groups have deterministic expiry/cleanup so the process cannot grow unbounded.
• Durable storage can distinguish partial parts from completed messages, or explicitly documents that partial persistence is out of scope for v0.3.
• Tests cover: 8-bit refs, 16-bit refs, out-of-order parts, duplicate parts, missing part timeout, and normal single-part SMS unaffected.

Risk notes:

• Do not rely on one modem family's text-mode behavior unless capability profiles document it.
• Avoid emitting partial bodies to webhooks/WebSocket subscribers as final messages.

Slice 4 — Authenticated WebSocket realtime feed

Objective: Add /ws only after Slice 1 auth policy is merged.

Files likely touched:

• server.py or a new callstack/http.py if the server module is split
• tests/test_websocket.py
• README.md

Acceptance criteria:

• /ws requires the same auth policy as write endpoints.
• Broadcasts sanitized event envelopes for SMS, delivery reports, call state, signal quality, and USSD responses.
• Message bodies and phone numbers are included only where intentionally needed by the API contract; logs avoid PII.
• Tests use pytest-aiohttp and mock events, no hardware.

Risk notes:

• A realtime feed without auth is effectively a message/call metadata exfiltration endpoint.
• WebSocket should not become a second event system; it should adapt the typed EventBus.

Slice 5 — PII-safe metrics endpoint

Objective: Add /metrics with operational counters/gauges that help unattended Pi deployments without leaking private data.

Files likely touched:

• server.py or a new HTTP metrics helper
• tests/test_metrics.py
• README.md

Acceptance criteria:

• Exposes counters/gauges for sent/received/failed SMS, delivery-report outcomes, active call state, signal quality, reconnect count, and uptime.
• No phone numbers, SMS bodies, webhook URLs, SIM identifiers, or API keys appear in metric labels/values.
• Tests assert both metric presence and absence of PII-like dynamic labels.

Risk notes:

• Metrics endpoints are often scraped without auth on LANs; decide whether this endpoint follows the same auth policy or is loopback-only by default.

Slice 6 — Modem detection + capability profiles

Objective: Make hardware-specific assumptions explicit before broader voice/PBX work.

Files likely touched:

• callstack/config.py
• callstack/modem.py
• new callstack/hardware/ module if useful
• tests/test_modem_detection.py
• README.md troubleshooting/hardware section

Acceptance criteria:

• Detection scans candidate serial ports, probes with safe commands such as ATI, and reports model/vendor/candidate roles without sending SMS/calls.
• A capability profile records which features are known/supported/unknown: audio port role, DTMF command shape, CPCMREG availability, SMS mode, delivery reports, USSD, GNSS.
• Default behavior remains explicit and overridable; auto-detection must not break current ModemConfig(at_port=..., audio_port=...) users.
• Tests use fake ports/transports and do not require real hardware.

Risk notes:

• Avoid baking SIMCOM-only assumptions into the public API before Quectel/Huawei/Sierra-class profiles exist.

Human decisions requested

1. For Secure HTTP server startup instead of disabling auth by default #4/Slice 1, confirm preferred no-key startup policy for python server.py:
   • Recommended: fail closed when binding non-loopback without API keys.
   • Alternative: automatically bind loopback-only in a documented development mode.
2. For metrics, decide whether /metrics should require API-key auth or default to loopback-only.

Definition of done for v0.3

• Secure HTTP server startup instead of disabling auth by default #4, Handle VOICE CALL: BEGIN arriving during answer without double ACTIVE transition #5/fix: handle answer active URC race #7, and Prevent silent corruption of non-ASCII outbound SMS bodies #6/fix: encode outbound SMS with GSM charset #8 are resolved or intentionally superseded.
• Multipart SMS reassembly is implemented or explicitly deferred behind documented PDU/text-mode limitations.
• WebSocket and metrics do not ship before secure HTTP defaults.
• Full mock suite and whitespace gate pass:

git diff --check
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q

[Justinabox]

Visionist coherence update — 2026-06-24

Current main is converging in the right direction, but the v0.3 plan needs a small sequencing refresh now that several risk-reduction PRs landed and new parser/HTTP issues were filed.

Recent state changes

• Merged: fix: handle answer active URC race #7 fixed the answer-path VOICE CALL: BEGIN race.
• Merged: fix: encode outbound SMS with GSM charset #8 rejects non-GSM-03.38 outbound SMS instead of silently corrupting unsupported Unicode.
• Merged: feat: add safe modem capability profiles #12 added conservative modem capability profile dataclasses/classification, but it is not yet full safe port scanning/auto-detection.
• Merged: fix: match AT final result codes exactly #16 fixed exact AT final-result matching, closing the SMS-body truncation risk in Match AT final result codes exactly to avoid truncating SMS bodies #13.
• New open defects that should influence sequencing: Preserve multiline SMS bodies when parsing CMGR/CMGL responses #17 multiline CMGR/CMGL bodies, Let IVR play_and_collect finish prompts before starting no-input timeout #18 IVR prompt/no-input timeout, and Back HTTP SMS and delivery-report listings with SMSStore instead of process globals #19 HTTP SMS/report listings backed by globals instead of SMSStore.
• Product/security decision for Secure HTTP server startup instead of disabling auth by default #4 is now recorded: with no API key, allow only loopback binding with a clear warning.

Verified local health for this planning pass:

git diff --check
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q
# 311 passed in 4.02s

Updated dependency order

1. Secure and durable HTTP surface before realtime/dashboard

   • Implement Secure HTTP server startup instead of disabling auth by default #4 first using the approved policy: no-key startup may bind only loopback; non-loopback requires configured API keys.
   • Then implement Back HTTP SMS and delivery-report listings with SMSStore instead of process globals #19 so GET /sms/messages and delivery-report listing read bounded/durable service/store state instead of process globals.
   • Only after those should /ws, dashboard, webhook retries, or public metrics expand the network surface.

2. SMS ingest correctness before multipart reassembly

   • Fix Preserve multiline SMS bodies when parsing CMGR/CMGL responses #17 multiline parser preservation and Honor CMTI/CDSI SMS storage before reading or deleting slots #14 storage selection/deletion semantics before Reassemble inbound multipart SMS before public delivery #10.
   • Then Reassemble inbound multipart SMS before public delivery #10 can safely reassemble multipart SMS without building on a parser that may flatten/drop body lines or a storage path that may read/delete the wrong slot.

3. Voice call UX correctness before bigger PBX features

   • Fix Let IVR play_and_collect finish prompts before starting no-input timeout #18 before voicemail/IVR expansion. A PBX menu that cuts off a long prompt before the caller hears it will make higher-level IVR work look flaky even if call state is correct.

4. Treat capability profiles as groundwork, not completed auto-detection

   • feat: add safe modem capability profiles #12 is useful and aligned with the north star, but the roadmap item “Modem Auto-Detection” still needs a follow-up that actually scans injectable candidate ports, probes safe AT identity commands, and emits a redacted ModemDiscoveryReport with confidence/notes.
   • Keep all probes non-mutating: no SMS, USSD, calls, SIM unlock, storage changes, or sensitive identifiers in logs.

Concrete next slice recommendation

The highest-leverage next implementation slice is HTTP production-safety and durable listings:

• First PR: close Secure HTTP server startup instead of disabling auth by default #4 with loopback-only no-key dev mode and non-loopback fail-closed behavior.
• Second PR: close Back HTTP SMS and delivery-report listings with SMSStore instead of process globals #19 by routing HTTP message/report listings through bounded SMSStore/service queries and restart-safe tests.

Acceptance gates for those slices should include:

git diff --check
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/test_api_auth.py tests/test_sms_store.py tests/ -q
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q

Remaining human/product choice

• Metrics policy remains unresolved: should /metrics require the same API-key auth, or be loopback-only by default for local Prometheus/node exporters? Recommended default: loopback-only unless an explicit authenticated public bind is configured.

[Justinabox]

Visionist coherence update — 2026-06-25

Current main is still converging toward the north-star PBX/SMS gateway, but the next autonomous slices should stay boring and production-facing before realtime/dashboard work expands the surface area.

Recent state changes

• Merged since the last planning comment:
  • fix: preserve multiline stored SMS bodies #20 preserved multiline stored SMS bodies.
  • feat: add minimal callstack CLI #24 added the minimal callstack CLI (send / status-oriented groundwork).
  • fix: decode UCS2 USSD responses #28 decodes UCS2/GSM CUSD responses before returning USSD messages.
  • fix: route inbound SMS through stored notifications #29 routes direct multiline +CMT SMS through stored notifications.
• Open draft fix: declare HTTP server install extra #34 addresses Declare aiohttp server dependency for documented HTTP mode #33 by adding a documented [server] extra for aiohttp; this is aligned with DX and should be safe to land through the normal review/merge lane once gates are current.
• Open draft fix: add public SIM PUK recovery #23 / issue Make PUK-locked SIM recovery reachable through public API #15 remains a human/security decision, not an autonomous landing candidate: public SIM PUK recovery sends hardware-affecting credential commands and now also has a merge conflict after recent main changes.
• New HTTP/realtime planning issues now exist and should be sequenced behind the HTTP safety baseline: Add PII-safe health and Prometheus metrics endpoints #30 metrics/health, Add authenticated WebSocket realtime event stream #31 authenticated WebSocket, Return 4xx JSON errors for malformed HTTP API requests #32 JSON 4xx request errors, Declare aiohttp server dependency for documented HTTP mode #33 server dependency.

Verified local health for this planning pass on the current autonomous worktree:

git diff --check
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q
# 329 passed in 4.04s

Updated dependency order

1. Finish HTTP production baseline before realtime/dashboard

   • Land Declare aiohttp server dependency for documented HTTP mode #33/fix: declare HTTP server install extra #34 so documented server installs work in clean environments.
   • Fix Secure HTTP server startup instead of disabling auth by default #4 using the approved policy: no-key startup may bind only loopback with a warning; non-loopback requires configured API keys.
   • Fix Return 4xx JSON errors for malformed HTTP API requests #32 so malformed JSON / invalid route payloads return safe 4xx JSON instead of tracebacks.
   • Fix Back HTTP SMS and delivery-report listings with SMSStore instead of process globals #19 so HTTP SMS/report listings use bounded durable store-backed state instead of process globals.
   • Only then implement Add authenticated WebSocket realtime event stream #31 WebSocket or dashboard work.

2. Keep SMS ingest correctness ahead of multipart public delivery

   • Reassemble inbound multipart SMS before public delivery #10 is still the right v0.3 SMS feature, but it should build on the recent direct/stored multiline fixes and should explicitly verify whether the active receive path exposes UDH or needs PDU-mode receive.
   • Do not let webhook/WebSocket clients see multipart fragments as final messages.

3. Expose observability without leaking secrets/PII

   • Add PII-safe health and Prometheus metrics endpoints #30 is worthwhile after Secure HTTP server startup instead of disabling auth by default #4/Return 4xx JSON errors for malformed HTTP API requests #32 because health/metrics will help unattended Pi deployments.
   • Metrics labels must stay static/low-cardinality: no phone numbers, SMS bodies, webhook URLs, SIM identifiers, IMEI/serial values, API keys, or raw error strings.

4. Treat hardware-capability work as a safety rail for PBX expansion

   • Capability profiles are groundwork, but full safe discovery (Add safe modem discovery reports and capability profiles #11) is still needed before broader modem-family assumptions, voicemail, transfer/conference, or dashboard UX imply the hardware state is known.

Next highest-leverage implementation theme

HTTP operational hardening should remain the next theme:

1. Land fix: declare HTTP server install extra #34/Declare aiohttp server dependency for documented HTTP mode #33 (server extra + README install instructions).
2. Implement Secure HTTP server startup instead of disabling auth by default #4 (secure startup policy already approved).
3. Implement Return 4xx JSON errors for malformed HTTP API requests #32 (safe request validation / JSON 4xx responses).
4. Implement Back HTTP SMS and delivery-report listings with SMSStore instead of process globals #19 (durable, bounded HTTP list endpoints).

Acceptance gates for this theme should include:

git diff --check
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/test_api_auth.py tests/test_packaging_metadata.py tests/ -q
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q

Human/product decisions still needed

• Confirm whether public SIM PUK recovery (fix: add public SIM PUK recovery #23/Make PUK-locked SIM recovery reachable through public API #15) should exist at all, and if yes what safety policy it needs (explicit API shape, logging redaction standard, warnings, retry/lockout guardrails, and whether it must remain library-only rather than HTTP/CLI exposed).
• Metrics policy remains unresolved: recommended default is loopback-only when unauthenticated, or API-key protected for non-loopback binds, matching the Secure HTTP server startup instead of disabling auth by default #4 server-startup policy.

[Justinabox]

Visionist coherence update — 2026-06-25 08:20 UTC

Current main is healthy and still converging toward the north-star PBX/SMS gateway, but the next autonomous slices should keep closing correctness/security boundaries before adding more networked surfaces.

Evidence from this pass

• Repo head: 29bf801 (fix: make SMS store initialization idempotent).
• Local gates on clean main:

git diff --check
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q
# 361 passed in 4.20s

• Recently merged risk-reduction work includes: fix: declare HTTP server install extra #34 server extra, fix: return JSON 4xx for invalid HTTP requests #35 JSON 4xx validation, fix: validate USSD commands before modem writes #36 USSD validation, fix: return full TPDU length for submit PDUs #42 TPDU length, fix: parse text-mode delivery report references #43 delivery-report parsing, fix: normalize quoted DTMF URCs #44 quoted DTMF normalization, fix: preserve AT timeout classification #48 timeout classification, and fix: make SMS store initialization idempotent #51 idempotent SMS store initialization.
• Open draft fix: validate SMS webhook subscription URLs #52 is aligned with Validate webhook subscription URLs before posting SMS payloads #47 and should remain the immediate webhook hardening candidate. It validates subscription-time URL admission, but its own PR notes a future dispatcher/resolution hardening slice for DNS rebinding or public domains that later resolve private/link-local.
• Open draft fix: add public SIM PUK recovery #23 / Make PUK-locked SIM recovery reachable through public API #15 remains a human/security decision because it exposes credential-bearing SIM PUK recovery through public API, even with redaction and mock-only tests.
• New open planning/defect issues that should influence sequence: Add durable scheduled SMS queue and /sms/schedule endpoint #49 scheduled SMS queue, Add callstack monitor for PII-safe live event tailing #50 PII-safe callstack monitor, Preserve GSM 03.38 extension-table characters in PDU encoding #53 PDU GSM 03.38 extension-table preservation, and Reject invalid SMS recipient strings before /sms/send reaches the modem #54 stricter SMS recipient validation.

Updated v0.3 sequencing recommendation

1. Close network write/input boundaries before new surfaces.

   • Land/fix fix: validate SMS webhook subscription URLs #52/Validate webhook subscription URLs before posting SMS payloads #47 before relying on webhook subscriptions.
   • Close Reject invalid SMS recipient strings before /sms/send reaches the modem #54 so /sms/send rejects non-recipient strings before modem.sms.send().
   • Keep Secure HTTP server startup instead of disabling auth by default #4's established policy: unauthenticated/no-key startup must stay loopback-only or fail closed for non-loopback.

2. Fix lower-level SMS encoding before PDU-backed multipart receive.

   • Close Preserve GSM 03.38 extension-table characters in PDU encoding #53 before any implementation that depends on PDU-mode receive/reassembly.
   • Then return to Reassemble inbound multipart SMS before public delivery #10: reassemble inbound multipart SMS in SMSService, with deterministic tests for 8-bit refs, 16-bit refs, duplicate/out-of-order parts, missing-part expiry, and single-part SMS unaffected.

3. Make webhook delivery durable only after message finality is clear.

   • Add reliable signed SMS webhooks with retry/backoff #21 should consume completed public incoming-message events, not raw partial modem lines.
   • Acceptance should include retry/backoff, signatures, idempotency, PII-safe logs, and dispatcher-time protections against private/link-local/redirect targets in addition to subscription-time validation.

4. Prefer local DX observability before remote realtime.

   • Add callstack monitor for PII-safe live event tailing #50 (callstack monitor) is a good next DX slice because it helps hardware bring-up without opening another HTTP/WebSocket surface.
   • Add PII-safe health and Prometheus metrics endpoints #30 metrics can follow once counters/gauges are derived from stable service events/stores and avoid phone numbers, message bodies, webhook URLs, SIM identifiers, or API keys.
   • Add authenticated WebSocket realtime event stream #31 /ws should stay behind auth policy, event-envelope stability, and SMS reassembly semantics.

5. Do not accelerate dashboard/PBX expansion past capability and call-state reliability.

   • Dashboard remains Phase 4+ until auth/metrics/WebSocket foundations are safe.
   • Voice/PBX work such as Add pre-answer inbound call routing decisions #40/Add VoicemailBox helper for greeting and caller recording #41 should stay behind call-state correctness (Let IVR play_and_collect finish prompts before starting no-input timeout #18) and the conservative capability-profile direction (Add safe modem discovery reports and capability profiles #11/feat: add safe modem capability profiles #12), so SIMCOM-only audio/DTMF assumptions do not leak into public abstractions.

Architectural risk notes

• server.py on main still has module-global webhook_urls, received_messages, and delivery_reports; future /ws, /metrics, and scheduling work should avoid deepening those globals and instead adapt EventBus/SMSStore/small dedicated stores.
• PDU helpers currently have useful structure (PDUDecoder.parse_concatenation_udh) but are not yet safe enough for full PDU-mode text fidelity until Preserve GSM 03.38 extension-table characters in PDU encoding #53 lands.
• Hardware discovery currently provides pure identity/capability dataclasses and conservative classification; active port scanning/probing should remain a separate safe-probe slice with fake-port tests and no calls/SMS side effects.

Human decision status

No new product decision is needed for the v0.3 order. The outstanding human/security decision remains #15/#23: whether Callstack should expose public SIM PUK recovery at all, given that it sends credential-bearing, hardware-affecting commands even if logs are redacted.