Decode UCS2/GSM CUSD responses before returning USSD messages

[Justinabox]

Summary

ATResponseParser.parse_cusd() returns the raw quoted CUSD payload regardless of the modem-reported encoding/DCS. Many real modems/carriers return USSD balance/menu text as UCS2 hex (for example DCS/encoding 72), so USSDService.send() and /ussd/send surface unreadable hex instead of the carrier message.

This weakens the documented USSD balance-check/carrier-menu feature and makes automation brittle on non-ASCII carriers.

Affected files / lines

• callstack/protocol/parser.py:91-113 — _CUSD_RE captures message and encoding, but parse_cusd() returns message unchanged.
• callstack/ussd.py:35-63 — USSDService.send() returns the parsed USSDResponseEvent directly to callers.
• server.py:125-137 — /ussd/send serializes resp.message directly in JSON.
• tests/test_ussd.py:33-52 — coverage only checks already-readable ASCII payloads and does not cover UCS2/GSM-DCS decoding.

Evidence

Baseline is healthy:

git diff --check -> exit 0
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q
311 passed in 4.03s

Minimal safe reproduction (does not contact hardware):

PYTHONPATH=. uv run --no-project --with pyserial-asyncio python /tmp/callstack_issue_scout_repros.py

Relevant output:

CUSD UCS2 parse: (0, '0059006F00750072002000620061006C0061006E00630065002000690073002000240035002E00300030', 72)

That hex payload is UTF-16BE/UCS2 for:

Your balance is $5.00

Expected behavior

For known CUSD encodings, parse_cusd() / USSDService should expose decoded human-readable text while preserving the raw payload and encoding where useful for diagnostics.

Actual behavior

The event/message returned to Python callers and the HTTP endpoint is the raw hex string, despite the parser retaining encoding=72.

Duplicate check

No existing issues/PRs found with these searches:

gh issue list --repo Justinabox/Callstack --state all --search 'USSD UCS2 CUSD decode encoding in:title,body'
gh issue list --repo Justinabox/Callstack --state all --search 'parse_cusd hex balance carrier menu in:title,body'
gh pr list --repo Justinabox/Callstack --state all --search 'USSD UCS2 CUSD decode encoding'

Suggested fix direction

• Add a CUSD decoder helper with tests for at least:
  • UCS2/UTF-16BE hex payloads for common DCS values such as 72.
  • Existing ASCII/plain responses.
  • Malformed hex: return a clear fallback or structured error without crashing the URC dispatcher.
• Consider extending USSDResponseEvent with raw_message if preserving raw modem payload matters.
• Keep logs safe: do not include private account/menu content at INFO level.

Acceptance criteria

• ATResponseParser.parse_cusd('+CUSD: 0,"0059006F...",72') returns decoded text for the message field or a typed event with decoded message.
• /ussd/send returns decoded carrier text in JSON for UCS2 responses.
• Existing ASCII +CUSD tests continue to pass.
• Malformed encoded CUSD payloads are handled deterministically and do not kill URC dispatch.

Verification gates

git diff --check
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/test_ussd.py -q
PYTHONPATH=. uv run --no-project --with pytest --with pytest-asyncio --with pytest-aiohttp --with pyserial-asyncio --with aiosqlite pytest tests/ -q