OOM crash on large monorepos (10k+ files) at passDefinitions stage

[halindrome]

Bug Report

Summary

Indexing a large monorepo (10 000+ files, ~888 MB source) causes an out-of-memory crash during
the passDefinitions pipeline stage.

Environment

• Go 1.23+
• Linux / macOS
• Monorepo size: ~8991 files, ~888 MB

Observed behaviour

Process exits with OOM kill or runtime heap exhaustion. Peak heap observed ~7.8 GB before crash.

Root cause (identified)

Three concurrent memory spikes at passDefinitions:

1. adaptivePool.maxLimit = numCPU*8 (default) — workers × 8× CPU goroutines flood RAM
2. No MaxFileSize guard in full-index mode — large binary/generated files read entirely into RAM
3. extractionCache (~50 KB/file) held through passDefinitions and all downstream passes

Workaround

None known. Setting GOMEMLIMIT helps delay but does not prevent the crash.

Fix (in progress)

Branch fix/large-monorepo-crash on fork halindrome/codebase-memory-mcp contains:

• Cap adaptivePool.maxLimit to numCPU*2
• Add 2 MB MaxFileSize guard for full mode
• Release extractionCache immediately after passDefinitions
• Memory-pressure backpressure watcher (polls HeapInuse, throttles pool at 80% GOMEMLIMIT)

Peak heap after fix: 146 MB on the same 888 MB / 8991-file repo.

[halindrome]

This has been resolved with the rewrite in C.

[DeusData]

@halindrome so did the C rewrite now also show better efficiency on your end? Just try to gather some feedback :)