Skip to content

Improve tool/config surface detection for missed agent signals #4

@EgemennSahin

Description

@EgemennSahin

Problem

The agent repo lab found source files and framework config that look like agent-relevant surfaces but were not represented strongly enough in the scan map. These are missed signals, not vulnerabilities in the tested repos.

Evidence from tested repos

From .agentdiff/agent-repo-lab/latest/report.md:

  • langchain-ai/agents-from-scratch-ts: langgraph.json (LangGraph config file)
  • vercel-labs/github-tools: packages/github-tools/src/types.ts (AI SDK import; agent operation vocabulary)
  • vercel-labs/github-tools: apps/chat/nuxt.config.ts (AI tool definition syntax)
  • vercel-labs/github-tools: apps/chat/server/workflows/chat.ts (AI SDK import; state-changing or tool-like operation name; agent operation vocabulary)
  • langchain-ai/langgraphjs: libs/sdk-vue/src/selectors.ts (AI tool definition syntax; state-changing or tool-like operation name; agent operation vocabulary)

Lab context:

  • 7/10 repos scanned
  • 0 crashes
  • 23 useful findings
  • 46 missed signals
  • 3/3 useful synthetic PR tests

Proposed fix

Add narrow detectors for common JS/TS agent framework files and tool definitions that appeared in the lab evidence:

  • LangGraph config files such as langgraph.json
  • Mastra runtime/config/index files where they establish agents, tools, or workflows
  • AI SDK tool-definition syntax such as tool(...), defineTool, createTool, tools:, and schema-like tool declarations
  • OpenAI/Anthropic tool schema definitions where they are local repo surfaces

Keep this evidence-based and scoped. Do not build a universal analyzer.

Acceptance criteria

  • npm test passes.
  • npm run lab:agent-repos still has 0 crashes.
  • At least one cited missed signal is now represented as an agent-relevant surface or entrypoint evidence.
  • Report wording calls these agent-relevant surfaces, not bugs or vulnerabilities.
  • Docs/tests/config are not promoted to action-required unless configured or reachable from runtime agent code.

Non-goals

  • No live model calls.
  • No dependency installs in external repos.
  • No Python/Java import graph work.
  • No broad TypeScript compiler integration.
  • No claims that the cited external repos are unsafe.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions