Skip to content

Add search quality fixtures and ranking benchmarks #4

@dipeshbabu

Description

@dipeshbabu

CommandGraph already has lightweight ranking, synonyms, slots, templates, and a 30-command seed set. The next leftover piece should be a small quality harness so search changes do not regress the core apropos experience.

Scope:

  • Add fixture queries for the highest-value intents, for example permissions, port conflicts, disk usage, file search, package install, DNS/network checks, git recovery, archives, and containers.
  • Assert expected top result or top-N membership where exact ordering should stay flexible.
  • Cover both curated command cards and optional man-index merge behavior where practical.
  • Keep this deterministic and local-first; no ML or external service dependency.

Acceptance criteria:

  • A test or benchmark fixture file defines representative intent queries and expected commands.
  • CI can run the checks with the normal test suite.
  • Failures make ranking regressions easy to diagnose without requiring manual CLI testing.

Out of scope for this issue:

  • Embeddings, RL, cloud models, full learning-to-rank, or broad roadmap work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions