Skip to content

Add downstream task evaluation workflows #40

Description

@Tinnci

Goal

Evaluate whether enhancement actually helps downstream tasks instead of only improving audio metrics.

Scope

  • Add a downstream eval interface for comparing low-quality input vs enhanced output.
  • Track ASR WER/CER before and after enhancement.
  • Track speaker embedding similarity so enhancement does not destroy speaker identity.
  • Track VAD or endpoint accuracy where labeled fixtures are available.
  • Leave room for keyword spotting accuracy and TTS/voice-conversion preprocessing checks.
  • Make downstream models optional and gated.

Acceptance criteria

  • Downstream eval output records baseline input score, enhanced score, delta, evaluator version, and dataset identity.
  • ASR WER/CER has a documented adapter plan or first optional adapter.
  • Speaker similarity and VAD/KWS are represented in the schema even if initially deferred.
  • Documentation warns that SR improvements that worsen WER or speaker similarity are regressions for some use cases.
  • Default tests use mocks or tiny generated fixtures only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestevaluationEvaluation, benchmark, and regression harness work

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions