Skip to content

[P2] Public testing kit: scriptable fake LLM provider, fake memory, and fake tools (v1beta/agenttest) #152

Description

@kunalkushwaha

Problem

Users have no supported way to unit-test agents without hitting a real LLM:

  • A mock provider exists (ProviderTypeMock, internal/llm/factory.go:26,477-484; internal/llm/mock_adapter.go) and is technically reachable via provider = "mock", but it is undocumented, internal-flavored, and not scriptable (fixed responses only).
  • No fake memory, fake tool, or stream-assertion helpers are exported. The repo's own tests live in test/ and internal/ and aren't reusable by consumers.
  • Frameworks users will compare against (Genkit, LangChainGo) ship test helpers; for a framework whose selling point is production Go, a testing story is table stakes.

Proposed change

Add a v1beta/agenttest package:

  1. Scriptable fake provider: enqueue responses, including tool-call turns and streamed chunk sequences; assert on received messages/params.
    fake := agenttest.NewProvider().
        ExpectUserContains("weather").
        RespondWithToolCall("get_weather", args).
        ThenRespond("It's sunny in Pune.")
  2. Fake memory (in-memory, inspectable store/query log) and fake tools (canned results, call recording).
  3. Stream test helpers: collect chunks, assert ordering/types (ChunkTypeDelta, ChunkTypeToolCall, ChunkTypeDone).
  4. Document in a "Testing your agents" guide under docs/v1beta/.

From docs/AGNOSTIC_FRAMEWORK_ANALYSIS.md (branch claude/go-agnostic-framework-analysis-ekyl3a), §5 & §6.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions