Context
Live validation of compression.cache_hints on the anthropic provider (claude-sonnet-4-6, k8s triage agent, compression enabled) confirmed the current breakpoints work exactly as designed: the ephemeral cache_control on the last system block + last tool definition produced a 3,447-token prefix write on call 1 and clean 3,447-token cache reads (cache_creation: 0) on every subsequent call — the deterministic-compression prefix stayed byte-stable across the whole invocation.
| Call |
messages |
cache_creation |
cache_read |
uncached input |
| 1 |
1 |
3,447 |
0 |
359 |
| 2 |
4 |
0 |
3,447 |
4,026 |
| 3 |
7 |
0 |
3,447 |
8,017 |
| 4 |
9 |
0 |
3,447 |
9,232 |
Gap
Only system + tools sit behind a breakpoint. Conversation history is re-billed at the full input rate on every agent-loop iteration — uncached input grew 359 → 9,232 tokens across 4 calls in a short triage run. Long tool-heavy invocations (dozens of loop iterations) pay quadratic-ish full-rate token costs on a history prefix that is append-only and therefore ideally cacheable.
Proposal
Anthropic allows up to 4 cache_control breakpoints per request. Add a third breakpoint on the conversation prefix — e.g. on the last content block of the most recent stable message (everything before the newest turn). Each iteration then reads the previous iteration's history from cache and only pays full rate for the newly appended turn.
Design notes:
- The compression
llm.Client wrapper already guarantees a deterministic live zone (query pinned to the first user message, protected recent turns), so the history prefix is byte-stable between iterations — validated live by the cache_creation: 0 behavior above. The precondition for prefix caching already holds.
- Breakpoint placement must respect the marker-carrying messages: a breakpoint after a message whose compressed content later gets expanded in-place would invalidate the cache — place it only behind turns outside the recompression window (
protect_recent).
- Keep it gated behind the existing
ClientConfig.PromptCaching / compression.cache_hints flag; no new config surface needed.
- Bedrock passthrough (
auth_scheme: aws_sigv4) inherits the same request shape, so this benefits Bedrock deployments too.
Measured stake
In the 4-call validation run, calls 2–4 carried 21,275 uncached history tokens that a conversation-prefix breakpoint would have served at the 90%-discounted cache-read rate (minus each call's newly appended turn).
Context
Live validation of
compression.cache_hintson the anthropic provider (claude-sonnet-4-6, k8s triage agent, compression enabled) confirmed the current breakpoints work exactly as designed: the ephemeralcache_controlon the last system block + last tool definition produced a 3,447-token prefix write on call 1 and clean 3,447-token cache reads (cache_creation: 0) on every subsequent call — the deterministic-compression prefix stayed byte-stable across the whole invocation.Gap
Only system + tools sit behind a breakpoint. Conversation history is re-billed at the full input rate on every agent-loop iteration — uncached input grew 359 → 9,232 tokens across 4 calls in a short triage run. Long tool-heavy invocations (dozens of loop iterations) pay quadratic-ish full-rate token costs on a history prefix that is append-only and therefore ideally cacheable.
Proposal
Anthropic allows up to 4
cache_controlbreakpoints per request. Add a third breakpoint on the conversation prefix — e.g. on the last content block of the most recent stable message (everything before the newest turn). Each iteration then reads the previous iteration's history from cache and only pays full rate for the newly appended turn.Design notes:
llm.Clientwrapper already guarantees a deterministic live zone (query pinned to the first user message, protected recent turns), so the history prefix is byte-stable between iterations — validated live by thecache_creation: 0behavior above. The precondition for prefix caching already holds.protect_recent).ClientConfig.PromptCaching/compression.cache_hintsflag; no new config surface needed.auth_scheme: aws_sigv4) inherits the same request shape, so this benefits Bedrock deployments too.Measured stake
In the 4-call validation run, calls 2–4 carried 21,275 uncached history tokens that a conversation-prefix breakpoint would have served at the 90%-discounted cache-read rate (minus each call's newly appended turn).