[ARCH] Rust core via PyO3: high-performance engine for graph + indexing + queries

[Wolfvin]

Vision

CodeLens aims to be all-in-one: Navigator + Explorer + Guardian + Inspector.
To drop the need for external tools (like codebase-memory-mcp), CodeLens needs to match their indexing speed and query performance while keeping all existing Python logic intact.

The solution: PyO3 — Rust extensions callable from Python, distributed as a standard .whl.

Architecture

CodeLens/
├── codelens_core/           <- Rust (PyO3)
│   ├── graph_engine.rs      <- node/edge traversal, BFS/DFS, Louvain clustering
│   ├── index_pipeline.rs    <- RAM-first bulk indexing, LZ4 compression
│   ├── query_engine.rs      <- Cypher-like query execution
│   └── grammar_loader.rs    <- tree-sitter bindings for all languages
└── scripts/                 <- Python (unchanged)
    ├── codelens.py          <- CLI entry point
    ├── taint_engine.py      <- complex logic stays in Python
    ├── vulnscan_engine.py
    └── ...

Rust handles the hot path. Python handles the logic. Users still pip install codelens — they never know Rust is inside.

Expected Performance Gains

Operation	Python now	Rust target
Full scan (500 files)	~2.4s	<200ms
Symbol query	~188ms	<5ms
Graph traversal (trace)	O(n) linear	<1ms BFS
Memory (large repo)	high	10x lower

Why PyO3 over full rewrite

• Python logic (taint, CVE, guard hooks, MCP) stays untouched
• Distribute as pre-built wheels — zero Rust toolchain required for users
• Can migrate module by module, not all at once
• Rust + Python boundary is clean and well-documented via PyO3

Phased Implementation

1. Add codelens_core Rust crate with PyO3 bindings
2. Port graph engine first (highest impact)
3. Port index pipeline
4. Port query engine
5. Port grammar loader (enables 158-language support)
6. Python falls back gracefully if Rust extension not available (pure Python fallback)

References

• PyO3: https://pyo3.rs
• maturin (build tool): https://www.maturin.rs
• codebase-memory-mcp achieves 532ms full index in pure C — this is the target to match