Requirements: MVP Phase 1

[orchestrator-bot[bot]]

Requirements Analysis

Auto-created from Issue #40

User Request

Begin development of the scope outlined in Phase 1 of documentation/roadmap.md

---

Labels: None
Requested by: @tinkermonkey

---

The orchestrator will analyze this request and develop detailed requirements.
When complete, Issue #40 will be updated with final requirements.

[orchestrator-bot[bot]]

Idea Research

---

I'll start by reading the project's CLAUDE.md and the referenced roadmap to understand the full context.Now let me look at the existing source code to understand current implementation state.Now let me check if there's a roadmap file somewhere else in the project.The roadmap file doesn't exist yet. Let me check the README and architecture docs more carefully for any Phase 1 references, and also look at the git log for context.Now let me read the actual implemented source files for precise understanding.Now I have a thorough understanding of the project. Since there's no documentation/roadmap.md file yet, the task is asking to define what "Phase 1 MVP" should be based on the project's architecture, current state, and prior art. Let me compose the analysis.

Executive Summary

Context Library is a versioned RAG system with source lineage tracking, designed to normalize heterogeneous personal data sources (email, notes, events, tasks) into a dual-storage architecture (SQLite source-of-truth + LanceDB vector index). The project has comprehensive architectural documentation across three design documents (~1,170 lines total) and a well-scaffolded codebase, but only 3 of 27 source files contain real implementation (~119 lines of code). All 20 test files are empty stubs.

The referenced documentation/roadmap.md does not yet exist. This analysis explores what an MVP Phase 1 should encompass based on the architecture, implementation dependencies, prior art in the RAG ecosystem, and practical considerations for getting to a working vertical slice.

---

Idea Exploration

Current Implementation State

The project is in "architecture phase" as stated in the README. The three implemented files are all in the storage module:

• storage/models.py — Domain enum (MESSAGES, NOTES, EVENTS, TASKS)
• storage/validators.py — Embedding dimension validation (384-dim) and ISO 8601 timestamp validation
• storage/vector_store.py — ChunkVector LanceDB model with Pydantic validation, frozen immutability

Everything else — the pipeline orchestrator, differ, chunker, embedder, versioner, all four domain implementations, the adapter base class, the document store, all retrieval modules, and the scheduler — exists only as single-line docstring stubs. This means Phase 1 must establish the foundational building blocks that everything else depends on.

Dependency Graph Analysis

The architecture has a clear dependency order. Working from the leaves inward:

1. Data models and schemas (storage layer) — Foundation for everything. The SQLite schema, Pydantic models for chunks/lineage/versions, and the adapter contract types must exist before any pipeline code.

2. Document store (storage/document_store.py) — The SQLite implementation is the commit point for the entire write path. The vector store model exists, but the document store (which is the source of truth) does not.

3. Core pipeline components — The differ, chunker, embedder, and versioner form the ingestion pipeline. They depend on the storage layer and data models.

4. Domain layer — Depends on the chunker interface to override chunking behavior. At minimum, one domain (Notes, being the simplest and most "standard" RAG-like) is needed for a vertical slice.

5. Adapter layer — Depends on the domain layer and storage layer. A filesystem/markdown adapter is the simplest starting point (no API auth, no rate limits).

6. Retrieval — Depends on the vector store being populated. Needs the embedder and basic query execution.

7. Scheduler — The least critical for an MVP. Manual ingestion is sufficient to prove the pipeline works.

What "MVP" Means for This System

The core value proposition of Context Library is not "another RAG system" — it is the combination of: (a) content-addressed versioning with hash-based change detection, (b) full chunk-level provenance, and (c) domain-aware chunking. An MVP that omits these differentiators is just a generic RAG pipeline. Phase 1 should demonstrate the versioning and lineage features end-to-end, even if only for a single domain and adapter.

---

Potential Directions

Direction A: Storage-First (Bottom-Up)

Build the full storage layer first (SQLite schema creation, document store CRUD, sync log), then layer pipeline components on top. This prioritizes data integrity and the "SQLite is source of truth" principle.

Components: SQLite schema initialization with PRAGMA user_version, document store (full CRUD for adapters, sources, source_versions, chunks tables), LanceDB table creation, sync log management.

Strengths: Rock-solid foundation; every subsequent component has a real storage layer to write against. Aligns with the write-path-first design (steps 1-10 in persistence-design.md).

Risk: Could produce a lot of infrastructure code before anything visibly works.

Direction B: Vertical Slice (One Adapter → Storage → Retrieval)

Implement the thinnest possible path from source to retrievable chunk: a filesystem/markdown adapter → Notes domain → basic chunking → embedding → dual storage → simple query. This proves the full pipeline works end-to-end.

Components: BaseAdapter interface + filesystem adapter, NotesDomain (standard chunking), Embedder wrapper, DocumentStore + VectorStore writes, basic retrieval query.

Strengths: Demonstrates value quickly; every component is tested in context. Validates the architecture's assumptions about data flow.

Risk: May cut corners on versioning/diffing to get the first pass working, deferring the system's actual differentiator.

Direction C: Core Pipeline + Versioning (Differentiator-First)

Focus on the unique value: implement the differ, versioner, and hash-based change detection. Build enough storage and chunking to exercise the re-ingestion flow (ingest → change → re-ingest → diff → new version).

Components: Document store, differ (markdown comparison, meaningful change detection), versioner (hash tracking, version chain), basic chunker, re-ingestion pipeline.

Strengths: Validates the project's core differentiator early. If the versioning approach has issues (phantom diffs, hash instability), better to discover them before building four domains and a dozen adapters.

Risk: No retrieval path means you can't actually use the stored data for RAG until Phase 2.

Recommended Approach: Hybrid of B and C

A practical MVP Phase 1 should combine the vertical slice with the versioning differentiator. The minimum viable scope:

1. Storage layer: SQLite schema initialization, document store (write path + basic reads), LanceDB table creation
2. Core models: All Pydantic models for Chunk, LineageRecord, SourceVersion, NormalizedContent, StructuralHints, DiffResult
3. Adapter contract + filesystem adapter: BaseAdapter ABC + a markdown filesystem adapter (Notes domain)
4. Notes domain: Standard chunking using the documented algorithm (or a simplified version that respects headings and atomic blocks)
5. Differ: Markdown comparison with hash-based change detection, whitespace filtering
6. Embedder: Thin wrapper around sentence-transformers with configurable model
7. Pipeline: Orchestrate the full ingest + re-ingest flow
8. Basic retrieval: Embed a query, search LanceDB, return results with chunk_hash linkage back to SQLite

This scope validates both the unique value (versioning, lineage) and the practical utility (ingest → retrieve) in a single phase.

---

References and Prior Art

Versioned RAG

• VersionRAG (arXiv:2510.08109) is the most directly relevant academic work. It models document evolution through a hierarchical graph with version sequences, content boundaries, and inter-version changes. It achieves 90% accuracy vs. 58% for naive RAG on versioned technical docs. Context Library's hash-based approach is simpler and more practical for personal data, but VersionRAG's query-intent routing (version-specific vs. cross-version vs. change-detection queries) is a useful pattern to consider for the retrieval layer.

• lakeFS-based versioning uses Git-like data versioning at the repository level (commit hashes stored alongside embeddings in Qdrant). Context Library's chunk-level versioning is more granular and doesn't require an external version control system.

• No existing open-source RAG framework (LlamaIndex, LangChain, Haystack, RAGFlow) implements content-addressed versioning with hash-based change detection. This is a genuine gap in the ecosystem.

Dual Storage Pattern (SQLite + Vector DB)

The 2025-2026 landscape shows strong momentum toward SQLite as a unified store for local-first RAG:

• sqlite-vec provides brute-force KNN as a SQLite extension. Simpler deployment than LanceDB but lacks advanced indexing (IVF-PQ), hybrid search, and columnar storage.
• txtai combines SQLite + FAISS in an all-in-one framework for semantic search and RAG.
• LLMWare supports SQLite (text) + ChromaDB/LanceDB (vectors) out of the box.

Context Library's dual-store separation (SQLite for relational/lineage queries, LanceDB for vector search) is the more flexible approach for a system with significant metadata and provenance requirements. The "vector store is an index, not a database" principle is well-validated.

Structure-Aware Markdown Chunking

• Chonkie (github.com/chonkie-inc/chonkie, 3.7k stars) is the strongest open-source library for markdown-aware chunking: 33x faster than alternatives, preserves code blocks/tables, supports markdown recipes. It could serve as a reference implementation or direct dependency for the Notes domain chunker.
• LangChain's MarkdownTextSplitter has known issues with code block distortion and whitespace handling — not recommended as a base.
• mistune (already in dependencies) handles markdown parsing. The project's Phase 1 chunker needs to parse markdown into a block tree (Phase 1 of the chunking algorithm), which mistune's AST output supports directly.

Embedding Models

The project currently hardcodes EMBEDDING_DIM = 384 in validators.py, matching all-MiniLM-L6-v2. Current benchmarks (MTEB leaderboard, multiple 2025-2026 analyses) indicate this model is significantly outperformed by modern alternatives:

Model	Dimensions	Context	Retrieval Quality (BEIR top-5)
all-MiniLM-L6-v2 (current)	384	512 tokens	~56%
bge-small-en-v1.5	384	512 tokens	~70%
nomic-embed-text-v1.5	768 (flexible)	2048 tokens	~86%
bge-base-en-v1.5	768	512 tokens	~85%

nomic-embed-text-v1.5 is the strongest candidate for Phase 1: 2048-token context (critical for longer note chunks), Matryoshka representation learning (can reduce dimensions without retraining), and Apache 2.0 license. The embedder interface should be designed to make model swapping trivial, and the EMBEDDING_DIM constant should become configurable rather than hardcoded.

Personal Knowledge Management RAG

• Khoj (github.com/khoj-ai/khoj) is the closest in scope — it aggregates PDFs, Markdown, Notion, GitHub into a self-hosted AI assistant. However, it has no document versioning, hash-based change detection, or chunk-level provenance.
• Reor (github.com/reorproject/reor) is the closest in technology — it uses LanceDB for local-first RAG on markdown files. However, it's a note-taking application, not a versioned pipeline, and handles only a single source type.

No existing project combines multi-source aggregation, content-addressed versioning, chunk-level provenance, and domain-aware chunking. This confirms the project's unique positioning.

---

Technical Considerations

SQLite Schema Bootstrap

The persistence design specifies five tables (adapters, sources, source_versions, chunks, lancedb_sync_log) with triggers, indexes, and foreign keys. Phase 1 needs a schema initialization module that:

• Creates all tables with IF NOT EXISTS
• Sets PRAGMA user_version=1
• Enables WAL mode, foreign keys, and NORMAL synchronous mode
• Handles the create-or-migrate decision on startup

The schema is already fully specified in the persistence design document, so the implementation is a translation exercise. The key decision is whether to use raw SQL strings or an ORM/query builder. Given the project's emphasis on SQLite-specific features (PRAGMA, triggers, recursive CTEs for version chains), raw SQL with a thin Python wrapper is the pragmatic choice.

Embedding Dimension Configuration

The current EMBEDDING_DIM = 384 hardcoded in validators.py creates a coupling between the validator and a specific model. Phase 1 should make this configurable — either via the Embedder class (which knows its model's dimensions) or via a configuration object. The lineage record already includes normalizer_version; it should also track the embedding model identifier so re-embedding can be triggered on model change.

Chunking Algorithm Complexity

The chunking-strategy.md describes a four-phase recursive descent algorithm (parse → measure → recurse → post-process) with atomic block handling, context headers, and domain-specific overrides. A full implementation of this algorithm is substantial. For Phase 1, a pragmatic scope is:

• Use mistune to parse markdown into a block tree
• Implement heading-based splitting with soft/hard limits
• Keep code blocks and tables atomic
• Generate context headers (heading breadcrumbs)
• Defer: cross-reference detection, prose overlap, table row-splitting

This covers the 80% case (heading-structured documents with code blocks) without the long tail of edge cases.

Differ Design

The differ needs to answer: "did the markdown change meaningfully?" The simplest approach that validates the architecture:

• Hash the full normalized markdown — if identical, skip
• If different, produce new chunks and compare chunk hash sets (added/removed/unchanged)
• Whitespace-only change filtering via normalization before hashing

The more sophisticated approach (section-level diffing to identify which sections changed, enabling selective re-chunking) can be deferred. The hash-set comparison on chunk outputs already captures the net effect of any change.

Test Strategy

All 20 test files are empty. Phase 1 should establish the testing pattern:

• Unit tests for validators, models, and pure functions (differ, chunker)
• Integration tests for the document store (SQLite in-memory), vector store (LanceDB temp directory), and the full pipeline
• Fixtures for sample markdown documents with known chunk outputs
• pytest is already configured with pythonpath = ["src"] and testpaths = ["tests"]

LanceDB Version Compatibility

The project specifies lancedb>=0.29. LanceDB reached v1.0.0 in 2025 with breaking API changes in the Rust SDK. The Python API has remained relatively stable, but the ChunkVector model uses lancedb.pydantic.LanceModel which is the current recommended approach. Phase 1 should pin to a tested version range to avoid surprise breakage.

Missing from Current Dependencies

The project may need additional dependencies for Phase 1:

• hashlib (stdlib) — for content hashing (SHA-256 for chunk_hash)
• No additional external dependencies are strictly required for the core pipeline; mistune, pydantic, lancedb, and sentence-transformers cover the MVP stack

---

Generated by Orchestrator Bot 🤖
Processed by the idea_researcher agent

[orchestrator-bot[bot]]

Business Analysis

---

Executive Summary

Context Library is a versioned RAG system that normalizes heterogeneous personal data sources into a dual-storage architecture (SQLite + LanceDB). The project has comprehensive architectural documentation and a scaffolded codebase, but only 3 of 27 source files contain real implementation. Phase 1 MVP establishes the foundational storage layer, core pipeline components, and a single end-to-end vertical slice (filesystem markdown adapter through retrieval) that demonstrates the system's key differentiators: content-addressed versioning, hash-based change detection, and chunk-level provenance.

The documentation/roadmap.md file does not yet exist and must be created as part of this phase to define the phased delivery plan.

---

Functional Requirements

FR-1: Roadmap Documentation

FR-1.1 A documentation/roadmap.md file must exist defining Phase 1 scope, the components included, and the components deferred to later phases.

---

FR-2: SQLite Schema and Document Store

FR-2.1 The system must initialize a SQLite database with all five tables specified in the persistence design: adapters, sources, source_versions, chunks, lancedb_sync_log.

FR-2.2 Schema initialization must set PRAGMA user_version=1, enable WAL mode, enable foreign key enforcement, and set synchronous mode to NORMAL.

FR-2.3 Tables must be created with IF NOT EXISTS to support idempotent initialization.

FR-2.4 The document store must support the full write path: registering an adapter, registering a source, creating a source version, writing chunks with their lineage records, and writing sync log entries.

FR-2.5 The document store must support reading chunks by source, reading version history for a source, and looking up chunks by hash.

FR-2.6 The LanceDB table for chunk vectors must be created if it does not exist, using the schema defined in storage/vector_store.py.

---

FR-3: Core Data Models

FR-3.1 Pydantic models must exist for all data structures flowing through the pipeline: NormalizedContent, StructuralHints, Chunk, LineageRecord, SourceVersion, DiffResult, and AdapterConfig.

FR-3.2 The chunk_hash field must be a SHA-256 hash of the chunk's normalized content, computed deterministically.

FR-3.3 The LineageRecord model must include: chunk_hash, source_id, source_version_id, adapter_id, domain, normalizer_version, and embedding_model_id.

FR-3.4 The embedding dimension must be configurable rather than hardcoded, with the value driven by the configured embedding model.

---

FR-4: Adapter Contract and Filesystem Adapter

FR-4.1 A BaseAdapter abstract base class must define the contract all adapters must fulfill: a method to yield NormalizedContent items from a source.

FR-4.2 A filesystem adapter must implement BaseAdapter, accepting a directory path and yielding NormalizedContent for all .md files found recursively.

FR-4.3 The filesystem adapter must populate StructuralHints from filesystem metadata: file path, modification timestamp, and file size.

FR-4.4 The adapter must record itself in the adapters table on first use and retrieve its registered adapter_id on subsequent uses.

---

FR-5: Notes Domain and Chunker

FR-5.1 A NotesDomain class must implement domain-specific chunking for markdown content.

FR-5.2 The chunker must use mistune to parse markdown into a block tree before splitting.

FR-5.3 The chunker must split on headings, treating each heading and its content as a candidate chunk.

FR-5.4 Code blocks and tables must be kept atomic (never split mid-block).

FR-5.5 Each chunk must include a context header containing the heading breadcrumb path from document root to that chunk's heading.

FR-5.6 The chunker must respect soft and hard token limits, joining short adjacent blocks and splitting oversized blocks at paragraph boundaries.

---

FR-6: Differ

FR-6.1 The differ must accept two versions of a source's content and produce a DiffResult indicating: unchanged, modified, or new.

FR-6.2 If the full-document SHA-256 hash matches the previous version's stored hash, the differ must return an unchanged result without further processing.

FR-6.3 If the document hash differs, the differ must compare the sets of chunk hashes between the previous version and the new version, classifying each chunk as added, removed, or unchanged.

FR-6.4 The differ must normalize whitespace before hashing to prevent whitespace-only changes from producing a non-identical result.

---

FR-7: Embedder

FR-7.1 An Embedder class must wrap a sentence-transformers model and expose a method accepting a list of text strings and returning a list of embedding vectors.

FR-7.2 The embedder must expose its model identifier and output dimension as properties, so downstream components can record these values in lineage records.

FR-7.3 The embedder must support batch processing of chunks.

---

FR-8: Ingestion Pipeline

FR-8.1 A pipeline orchestrator must coordinate the full ingest flow: adapter → domain chunker → differ → embedder → document store write → vector store write.

FR-8.2 On first ingest of a source, the pipeline must create a new source_version record and write all chunks with their lineage records and embeddings.

FR-8.3 On re-ingest of a previously ingested source, the pipeline must invoke the differ, skip unchanged chunks, write only new/modified chunks, and create a new source_version record referencing the previous version.

FR-8.4 Chunks removed in a re-ingest must be marked inactive in the document store and removed from the LanceDB index.

FR-8.5 The lancedb_sync_log must be written for every chunk vector operation (insert or delete).

---

FR-9: Basic Retrieval

FR-9.1 A retrieval function must accept a query string, embed it using the configured embedder, and perform a nearest-neighbor search against the LanceDB vector index.

FR-9.2 Retrieval results must include the chunk text, chunk hash, source identifier, and relevance score.

FR-9.3 The retrieval function must accept a top_k parameter controlling the number of results returned.

FR-9.4 Given a chunk hash from a retrieval result, the full lineage record for that chunk must be retrievable from the document store.

---

FR-10: Test Coverage

FR-10.1 Unit tests must exist for: hash computation determinism, differ logic (unchanged/added/removed classification), chunker output (atomic blocks, heading splits, context headers), and model validation.

FR-10.2 Integration tests must exist for: document store write/read roundtrip (SQLite in-memory), vector store write/search roundtrip (LanceDB temp directory), and the full pipeline ingest → re-ingest → retrieval flow.

FR-10.3 Test fixtures must include at least two sample markdown documents: one for initial ingest and one representing a modified version of the first, with known expected diff outcomes.

---

User Stories

Epic 1: Storage Foundation

US-1.1 — Schema Initialization
As a developer running Context Library for the first time,
I want the SQLite database to be automatically initialized with all required tables and settings,
So that I do not need to manually create schema before ingesting data.

Acceptance Criteria:

• Given the configured database path does not contain an initialized schema, when the application starts, then all five tables are created and PRAGMA user_version returns 1
• Given the database is already initialized, when the application starts again, then schema initialization completes without error and existing data is preserved
• Given the database is initialized, when a write is performed, then WAL mode is active and foreign key constraints are enforced

---

US-1.2 — Document Store Write Path
As the ingestion pipeline,
I want to persist adapter registrations, source records, version records, chunks, and lineage records to SQLite,
So that all ingested data has a durable, queryable source of truth.

Acceptance Criteria:

• Given a new adapter configuration, when the adapter is registered, then a row exists in the adapters table with a stable adapter_id
• Given an adapter is already registered, when registration is called again with the same identifier, then the existing adapter_id is returned without creating a duplicate
• Given a source and version are registered and chunks are provided, when chunks are written, then each chunk row in the chunks table has a corresponding lineage record with all required fields populated

---

US-1.3 — Document Store Read Path
As the retrieval layer,
I want to query chunks by source and look up lineage records by chunk hash,
So that retrieval results can be enriched with full provenance information.

Acceptance Criteria:

• Given chunks have been ingested for a source, when chunks are queried by source identifier, then all active chunks for the latest version are returned
• Given a chunk hash from a retrieval result, when lineage is queried by that hash, then the lineage record includes source, version, adapter, domain, normalizer version, and embedding model identifier
• Given a source has multiple ingested versions, when version history is queried, then all versions are returned in ingestion order with their timestamps

---

Epic 2: Adapter and Domain Layer

US-2.1 — Filesystem Adapter
As a user with a directory of markdown notes,
I want to point the filesystem adapter at a directory,
So that all markdown files in that directory are available for ingestion.

Acceptance Criteria:

• Given a directory containing markdown files in nested subdirectories, when the adapter is iterated, then NormalizedContent is yielded for every .md file found recursively
• Given a markdown file, when the adapter yields its content, then StructuralHints includes the file path, last-modified timestamp as ISO 8601, and file size in bytes
• Given a file that is not a .md file, when the adapter scans the directory, then that file is not yielded

---

US-2.2 — Notes Domain Chunker
As the ingestion pipeline,
I want markdown content split into semantically coherent chunks with context headers,
So that each chunk is independently meaningful when retrieved without its surrounding document context.

Acceptance Criteria:

• Given a markdown document with H1, H2, and H3 headings, when chunked, then each heading and its immediate content form a candidate chunk with a context header containing the full heading breadcrumb path
• Given a chunk candidate that exceeds the hard token limit, when chunked, then it is split at paragraph boundaries, not mid-paragraph
• Given a markdown document containing a fenced code block, when chunked, then no chunk boundary falls within the code block
• Given a markdown document containing a table, when chunked, then no chunk boundary falls within the table
• Given two adjacent heading sections that are both below the soft token limit, when chunked, then they may be joined into a single chunk if the combined size remains within the hard limit

---

Epic 3: Versioning and Change Detection

US-3.1 — First Ingest
As a user,
I want to ingest a directory of markdown notes for the first time,
So that all notes are chunked, embedded, and stored with full lineage.

Acceptance Criteria:

• Given a directory of markdown notes that has never been ingested, when ingestion runs, then a source record and source version record are created for each file, all chunks are written to SQLite with lineage records, and all chunk vectors are written to LanceDB
• Given ingestion completes, when the lancedb_sync_log is queried, then an entry exists for each chunk vector written

---

US-3.2 — Re-ingest with Change Detection
As a user who has modified some markdown notes,
I want re-ingestion to detect which notes changed and only process changed content,
So that unchanged content is not re-embedded unnecessarily.

Acceptance Criteria:

• Given a source was previously ingested and its content has not changed, when re-ingestion runs, then no new source version is created and no new chunks are written
• Given a source was previously ingested and one heading section was modified, when re-ingestion runs, then a new source version is created, only the modified chunk is re-embedded and written, unchanged chunks are linked to the new version without re-processing, and the previous version's record is preserved
• Given a source was previously ingested and a new heading section was added, when re-ingestion runs, then the new chunk is written and embedded; pre-existing unchanged chunks are not re-embedded
• Given a source was previously ingested and a heading section was deleted, when re-ingestion runs, then the deleted chunk is marked inactive in SQLite and removed from the LanceDB index

---

US-3.3 — Whitespace-Only Change Filtering
As the ingestion pipeline,
I want whitespace-only edits to markdown files to be treated as unchanged,
So that formatting normalization by editors does not trigger unnecessary re-processing.

Acceptance Criteria:

• Given a previously ingested file and a version of that file that differs only in trailing whitespace or blank lines, when the differ processes the two versions, then the result is classified as unchanged
• Given the unchanged result, when the pipeline processes it, then no new version record is created

---

Epic 4: Retrieval

US-4.1 — Semantic Search
As a user,
I want to issue a natural language query and receive the most relevant chunks from my ingested notes,
So that I can find information across my notes without knowing exact keywords.

Acceptance Criteria:

• Given chunks have been ingested and indexed, when a query string and top_k value are provided, then exactly top_k results are returned (or fewer if fewer chunks exist), each with chunk text, chunk hash, source identifier, and relevance score
• Given a query, when results are returned, then results are ordered by descending relevance score
• Given a chunk hash from a result, when lineage is queried using that hash, then the full lineage record is returned

---

Epic 5: Roadmap Documentation

US-5.1 — Roadmap File
As a contributor to the project,
I want a documentation/roadmap.md file that defines the phased delivery plan,
So that I understand which features are in scope for each phase and can orient my work accordingly.

Acceptance Criteria:

• Given the repository, when documentation/roadmap.md is read, then it defines Phase 1 scope listing all components included in this phase
• Given the roadmap, when Phase 1 components are listed, then each component maps to a source file in the codebase
• Given the roadmap, when later phases are described, then deferred features (scheduler, additional domains, additional adapters, advanced retrieval) are identified as out of scope for Phase 1

---

Generated by Orchestrator Bot 🤖
Processed by the business_analyst agent

[orchestrator-bot[bot]]

Architecture Design

---

I need to analyze the project structure, existing code, and documentation before producing the architecture analysis.Now let me read the key documentation files and implemented source files in detail.Now let me read the existing stub files to understand the current module docstrings and any hints about intended interfaces.Now I have a complete picture of the codebase. Let me produce the architecture analysis.

Executive Summary

Phase 1 MVP implements the foundational vertical slice of Context Library: filesystem markdown ingestion through semantic retrieval with full provenance. The architecture follows the dual-storage pattern already documented (SQLite as source of truth, LanceDB as derived vector index) and builds 10 components across 5 layers. The existing scaffolded codebase provides 27 stub files with 4 already implemented (schema.sql, models.py, validators.py, vector_store.py). This analysis defines the component architecture, dependency graph, implementation ordering, and key design decisions for the remaining 13 source files in scope.

---

System Architecture

Layer Diagram

┌──────────────────────────────────────────────────┐
│                  Pipeline Layer                    │
│  pipeline.py — orchestrates ingest + re-ingest     │
├──────────────────────────────────────────────────┤
│          Adapter Layer        │   Domain Layer     │
│  base.py (contract)          │  base.py (contract) │
│  filesystem.py (markdown)    │  notes.py (chunker) │
├──────────────────────────────────────────────────┤
│                   Core Layer                       │
│  differ.py    embedder.py    models.py (Pydantic)  │
├──────────────────────────────────────────────────┤
│                 Storage Layer                      │
│  document_store.py (SQLite)  vector_store.py (Lance)│
│  schema.sql    models.py     validators.py         │
├──────────────────────────────────────────────────┤
│                Retrieval Layer                      │
│  query.py (semantic search)                        │
└──────────────────────────────────────────────────┘

Component Dependency Graph

pipeline.py
├── adapters/base.py (BaseAdapter contract)
│   └── adapters/filesystem.py (FilesystemAdapter)
├── domains/base.py (BaseDomain contract)
│   └── domains/notes.py (NotesDomain chunker)
├── core/differ.py (Differ)
├── core/embedder.py (Embedder)
├── storage/document_store.py (DocumentStore)
│   └── storage/schema.sql [EXISTS]
│   └── storage/models.py [EXISTS]
└── storage/vector_store.py [EXISTS]

retrieval/query.py
├── core/embedder.py
├── storage/vector_store.py [EXISTS]
└── storage/document_store.py

Files In Scope for Phase 1

File	Status	Phase 1 Role
storage/schema.sql	Exists	No changes needed
storage/models.py	Exists	Extend with Pydantic pipeline models
storage/validators.py	Exists	Minor: make EMBEDDING_DIM configurable
storage/vector_store.py	Exists	No changes needed
storage/document_store.py	Stub	Full implementation
adapters/base.py	Stub	ABC definition
adapters/filesystem.py	New file	Filesystem markdown adapter
domains/base.py	Stub	ABC definition
domains/notes.py	Stub	Markdown chunker with mistune
core/differ.py	Stub	Hash-based diff
core/embedder.py	Stub	sentence-transformers wrapper
core/pipeline.py	Stub	Orchestrator
retrieval/query.py	Stub	Vector search + lineage lookup

Files Explicitly Out of Scope for Phase 1

File	Reason
domains/messages.py, events.py, tasks.py	Deferred domains
retrieval/reranker.py	Advanced retrieval feature
retrieval/provenance.py	Standalone provenance API (basic lineage is in query.py)
scheduler/poller.py, watcher.py	Automated scheduling deferred
core/versioner.py	Version logic absorbed into pipeline + differ

---

ADR-001: Pydantic Model Location

Context: The business requirements specify 7 Pydantic models (NormalizedContent, StructuralHints, Chunk, LineageRecord, SourceVersion, DiffResult, AdapterConfig). These are consumed across multiple layers (adapters, domains, core, storage).

Decision: Define all pipeline data models in storage/models.py, extending the existing file that already contains the Domain enum. This file becomes the canonical location for all data structures that flow through the system.

Rationale:

• Models are primarily persistence-shaped (they map to SQLite columns and LanceDB fields)
• A single models file avoids circular imports between layers
• The existing Domain enum already lives here, establishing the pattern
• 7 models + 1 enum is a manageable file size (~150-200 lines)

Trade-offs:

• (+) Zero circular import risk
• (+) Single place to understand all data contracts
• (-) File grows as models are added in later phases; can be split then

---

ADR-002: Embedding Dimension Configurability

Context: FR-3.4 requires configurable embedding dimensions. The current validators.py hardcodes EMBEDDING_DIM = 384. The ChunkVector LanceDB model uses Vector(EMBEDDING_DIM) at class definition time. LanceDB's Vector() type hint requires a compile-time constant for schema definition.

Decision: Make the embedding dimension configurable via the Embedder class, which exposes a dimension property derived from the loaded model. The EMBEDDING_DIM constant in validators.py remains as the default for the standard model (all-MiniLM-L6-v2). Validation of actual vectors at runtime checks against the embedder's reported dimension rather than the constant. The LanceDB table creation uses the dimension reported by the embedder, not the constant.

Rationale:

• LanceDB tables are created programmatically, not from the Pydantic schema alone — the dimension can be passed at table creation time
• The ChunkVector model serves as documentation and type guidance; runtime validation uses the embedder's actual dimension
• Avoids creating a global configuration object for a single parameter

Trade-offs:

• (+) Supports model changes without code edits
• (+) No global config system needed yet
• (-) ChunkVector type hint and runtime dimension may diverge; acceptable since LanceDB validates on write

---

ADR-003: DocumentStore as a Class Wrapping sqlite3

Context: The document store needs to execute the schema DDL, manage connections, and expose CRUD operations for all five tables.

Decision: Implement DocumentStore as a class that accepts a database path (or :memory: for tests), manages a sqlite3.Connection, and exposes typed methods for each write and read operation. Schema initialization runs on construction via executescript() against schema.sql. WAL mode and pragma settings are applied post-connection.

Rationale:

• sqlite3 is in the standard library — no ORM overhead for 5 tables
• Connection management is simple: one connection per DocumentStore instance, no pooling needed for a single-user desktop application
• The schema.sql file already exists and is complete; executescript() applies it idempotently

Trade-offs:

• (+) No additional dependency
• (+) SQL is visible in schema.sql, not buried in ORM definitions
• (+) Full control over pragma and transaction semantics
• (-) Manual parameter binding and row-to-model mapping; acceptable for the number of operations

---

ADR-004: Filesystem Adapter as Separate File

Context: The project scaffolding has adapters/base.py for the contract. Phase 1 needs a concrete filesystem adapter. No adapters/filesystem.py exists yet.

Decision: Create adapters/filesystem.py containing FilesystemAdapter(BaseAdapter). The base contract in adapters/base.py defines the abstract interface; each adapter implementation gets its own file.

Rationale:

• Follows the existing project convention of one-class-per-file (e.g., domains/notes.py, domains/messages.py)
• Keeps the base module clean and focused on the contract
• Later adapters (Gmail, Spotify, Todoist) each get their own file following the same pattern

---

ADR-005: Chunker Responsibility Split

Context: The scaffolding has both core/chunker.py and domains/notes.py. The business requirements place markdown chunking in NotesDomain.

Decision: domains/notes.py contains the NotesDomain class with the full markdown chunking implementation using mistune. core/chunker.py is not implemented in Phase 1 — it is reserved for a future shared chunking utility if cross-domain chunking patterns emerge. The domain layer owns chunking logic per the architecture documentation.

Rationale:

• The architecture documentation explicitly states "each domain applies modifications" to chunking
• Notes is the only domain in Phase 1; no shared chunking abstraction is justified yet
• Premature extraction of shared chunking logic would constrain future domain implementations

Trade-offs:

• (+) Avoids premature abstraction
• (+) Notes chunker can be fully specialized
• (-) If messages/events/tasks domains share patterns, some extraction will be needed later; acceptable cost

---

Scalability Design

Storage Growth

• SQLite: Linear growth with content volume. WAL mode provides concurrent read access during writes. The schema indexes cover all query patterns identified in the persistence design. No additional indexing is needed for Phase 1 volumes.
• LanceDB: Linear growth with chunk count. Brute-force search is sufficient below ~10K chunks per the persistence design. IVF-PQ indexing is a post-Phase 1 optimization triggered by chunk count threshold.
• Chunk deduplication: Content-addressed hashing means identical content across sources produces a single chunk record. This provides natural storage efficiency for overlapping content.

Processing Efficiency

• Differ short-circuit: Full-document SHA-256 comparison avoids chunk-level diffing when content is unchanged. This is the common case for re-ingestion and keeps re-ingest cost at O(1) for unchanged sources.
• Selective re-embedding: Only new/modified chunks are embedded on re-ingest. Unchanged chunk hashes are carried forward by reference. This reduces embedding API calls proportionally to actual change volume.
• Batch embedding: The embedder accepts lists of text, enabling sentence-transformers batch inference. This amortizes model overhead across chunks.

Future Scaling Path

The architecture supports these scaling steps without structural changes:

1. Connection pooling in DocumentStore (if concurrent access is needed)
2. LanceDB IVF-PQ indexing (automatic when chunk count exceeds threshold)
3. Async adapter execution (scheduler layer, deferred to Phase 2+)

---

Established Patterns

Pattern 1: Content-Addressed Identity

Every chunk is identified by chunk_hash = SHA-256(normalized_content). This hash:

• Serves as the primary key in SQLite chunks table
• Serves as the join key to LanceDB chunk_vectors
• Enables set-based diffing (added = new_hashes - old_hashes, removed = old_hashes - new_hashes)
• Is computed before the context header is prepended (context header is metadata, not identity)

Whitespace normalization is applied before hashing: collapse runs of whitespace, strip trailing whitespace per line, normalize line endings. This prevents formatting-only changes from producing new hashes.

Pattern 2: SQLite-First, LanceDB-Second Write Order

All writes follow the sequence documented in persistence-design.md:

1. Write to SQLite (this is the commit point)
2. Write to LanceDB (this is derived)

If step 2 fails, data is safe in SQLite. The lancedb_sync_log table tracks synchronization state for drift detection and retry.

Pattern 3: Adapter Self-Registration

Adapters register themselves in the adapters table on first use. Registration is idempotent: if the adapter_id already exists, the existing row is returned. This eliminates manual configuration steps and supports the "run it and it works" developer experience.

Pattern 4: Immutable Version Records

Source versions are append-only. A new version is created on each meaningful change; previous versions are never modified or deleted. Chunks are retired (soft-delete via retired_at) rather than deleted. This preserves full history for provenance queries.

Pattern 5: Context Headers for Isolated Retrieval

Each chunk carries a heading breadcrumb context header (e.g., # API Reference > ## Authentication > ### OAuth Flow). This header is prepended before embedding so the vector captures structural context. The header is not included in the content hash, preserving chunk identity across document restructuring.

---

Component Reuse

Existing Components (No Changes)

Component	Reuse Notes
storage/schema.sql	Complete DDL for all 5 tables. Used by DocumentStore via executescript().
storage/vector_store.py	ChunkVector LanceDB model. Used by pipeline for vector writes and by query.py for search.
storage/models.py	Domain enum. Extended with new Pydantic models; existing code unchanged.

Existing Components (Minor Extension)

Component	Extension
storage/validators.py	EMBEDDING_DIM remains as default. Add an optional expected_dim parameter to validate_embedding_dimension() for runtime flexibility.

Shared Across Layers

Model	Produced By	Consumed By
NormalizedContent	FilesystemAdapter	NotesDomain, Differ, Pipeline
Chunk	NotesDomain	Pipeline, DocumentStore, VectorStore, query.py
LineageRecord	Pipeline	DocumentStore, query.py
DiffResult	Differ	Pipeline
SourceVersion	DocumentStore	Differ, Pipeline
Domain enum	(existing)	All layers

---

Implementation Plan

Build Order and Dependencies

The implementation follows a bottom-up dependency order. Each step depends only on components from previous steps.

Step 1: Data Models (storage/models.py)
    ↓
Step 2: Document Store (storage/document_store.py)
    ↓
Step 3: Adapter Contract + Filesystem Adapter (adapters/base.py, adapters/filesystem.py)
    ↓
Step 4: Notes Domain Chunker (domains/notes.py)
    ↓
Step 5: Differ (core/differ.py)
    ↓
Step 6: Embedder (core/embedder.py)
    ↓
Step 7: Pipeline Orchestrator (core/pipeline.py)
    ↓
Step 8: Retrieval (retrieval/query.py)
    ↓
Step 9: Tests
    ↓
Step 10: Roadmap Documentation (documentation/roadmap.md)

Step 1: Data Models — storage/models.py

Extend the existing file with Pydantic models:

• StructuralHints — has_headings: bool, has_lists: bool, has_tables: bool, natural_boundaries: list[int], plus filesystem-specific fields: file_path: str | None, modified_at: str | None, file_size_bytes: int | None
• NormalizedContent — markdown: str, source_id: str, structural_hints: StructuralHints, normalizer_version: str
• Chunk — chunk_hash: str, content: str, context_header: str | None, chunk_index: int, chunk_type: str = "standard", domain_metadata: dict | None
• LineageRecord — chunk_hash: str, source_id: str, source_version_id: int, adapter_id: str, domain: Domain, normalizer_version: str, embedding_model_id: str
• SourceVersion — source_id: str, version: int, markdown: str, chunk_hashes: list[str], adapter_id: str, normalizer_version: str, fetch_timestamp: str
• DiffResult — changed: bool, added_hashes: set[str], removed_hashes: set[str], unchanged_hashes: set[str], prev_hash: str | None, curr_hash: str | None
• AdapterConfig — adapter_id: str, adapter_type: str, domain: Domain, normalizer_version: str, config: dict | None

All models use pydantic.BaseModel with model_config = ConfigDict(frozen=True) for immutability, consistent with the existing ChunkVector pattern.

Step 2: Document Store — storage/document_store.py

Class DocumentStore with constructor accepting db_path: str | Path:

Initialization:

• Connect to SQLite at the given path
• Enable WAL mode, foreign keys, NORMAL synchronous
• Execute schema.sql via executescript()
• Verify PRAGMA user_version equals 1

Write methods:

• register_adapter(config: AdapterConfig) -> str — INSERT OR IGNORE into adapters, return adapter_id
• register_source(source_id, adapter_id, domain, origin_ref, ...) -> None — INSERT into sources
• create_source_version(source_id, version, markdown, chunk_hashes, ...) -> None — INSERT into source_versions, UPDATE sources.current_version
• write_chunks(chunks: list[Chunk], lineage_records: list[LineageRecord]) -> None — Batch INSERT into chunks
• retire_chunks(chunk_hashes: set[str]) -> None — UPDATE retired_at on matching rows
• write_sync_log(chunk_hashes: list[str]) -> None — INSERT into lancedb_sync_log
• delete_sync_log(chunk_hashes: list[str]) -> None — DELETE from lancedb_sync_log

Read methods:

• get_latest_version(source_id: str) -> SourceVersion | None
• get_version_history(source_id: str) -> list[SourceVersion]
• get_chunks_by_source(source_id: str, version: int | None = None) -> list[Chunk]
• get_chunk_by_hash(chunk_hash: str) -> Chunk | None
• get_lineage(chunk_hash: str) -> LineageRecord | None
• get_adapter(adapter_id: str) -> AdapterConfig | None

All write operations use parameterized queries. All reads return Pydantic model instances.

Step 3: Adapter Layer — adapters/base.py + adapters/filesystem.py

adapters/base.py — BaseAdapter(ABC):

• Abstract method: fetch(source_ref: str) -> Iterator[NormalizedContent]
• Abstract property: adapter_id -> str
• Abstract property: domain -> Domain
• Abstract property: normalizer_version -> str
• Concrete method: register(document_store: DocumentStore) -> str — calls document_store.register_adapter() and returns the adapter_id

adapters/filesystem.py — FilesystemAdapter(BaseAdapter):

• Constructor accepts directory: str | Path
• adapter_id: deterministic string derived from adapter type + directory path
• domain: Domain.NOTES
• normalizer_version: "1.0.0"
• fetch(): walks directory recursively, yields NormalizedContent for each .md file with:
  • markdown: file contents read as UTF-8
  • source_id: relative file path from the base directory
  • structural_hints: populated with file_path, modified_at (ISO 8601), file_size_bytes, and boolean flags derived from content inspection

Step 4: Notes Domain Chunker — domains/notes.py

NotesDomain class:

• Constructor accepts soft_limit: int = 512, hard_limit: int = 1024, min_floor: int = 64
• chunk(content: NormalizedContent) -> list[Chunk]:
  1. Parse markdown to block tree using mistune.create_markdown() with AST renderer
  2. Walk the AST to identify headings, code blocks, tables, paragraphs
  3. Apply the recursive descent chunking algorithm from chunking-strategy.md:
     • Split on headings; each heading + content = candidate chunk
     • Keep code blocks and tables atomic
     • Join short adjacent blocks below min_floor
     • Split oversized blocks at paragraph boundaries, respecting hard_limit
  4. Generate context headers (heading breadcrumb paths)
  5. Compute chunk_hash = SHA-256(content) for each chunk (content excludes context header)
  6. Return list of Chunk instances with sequential chunk_index

Token counting uses a simple whitespace-split word count as a proxy (1 token ≈ 0.75 words). A tokenizer dependency is not justified for Phase 1.

Step 5: Differ — core/differ.py

Differ class:

• diff(prev_markdown: str | None, curr_markdown: str, prev_chunk_hashes: set[str] | None, curr_chunk_hashes: set[str]) -> DiffResult:
  1. If prev_markdown is None: first ingest, all chunks are "added"
  2. Normalize whitespace on both markdowns (collapse runs, strip trailing)
  3. Compute SHA-256 of both normalized markdowns
  4. If hashes match: return DiffResult(changed=False, ...)
  5. If hashes differ: compute set operations on chunk hashes
     • added = curr_chunk_hashes - prev_chunk_hashes
     • removed = prev_chunk_hashes - curr_chunk_hashes
     • unchanged = curr_chunk_hashes & prev_chunk_hashes
  6. Return DiffResult(changed=True, added_hashes=added, removed_hashes=removed, unchanged_hashes=unchanged)

Step 6: Embedder — core/embedder.py

Embedder class:

• Constructor accepts model_name: str = "all-MiniLM-L6-v2"
• Loads model via sentence_transformers.SentenceTransformer(model_name)
• Property model_id -> str: returns the model name
• Property dimension -> int: returns the model's output dimension (queried from the loaded model)
• embed(texts: list[str]) -> list[list[float]]: calls model.encode(texts), returns as list of lists
• embed_query(query: str) -> list[float]: convenience wrapper for single text

Step 7: Pipeline — core/pipeline.py

IngestionPipeline class:

• Constructor accepts: document_store: DocumentStore, embedder: Embedder, differ: Differ
• ingest(adapter: BaseAdapter, domain_chunker: NotesDomain) -> dict:
  1. Call adapter.register(document_store) for idempotent adapter registration
  2. Iterate adapter.fetch() yielding NormalizedContent items
  3. For each item:
     a. Look up latest version via document_store.get_latest_version(source_id)
     b. Chunk via domain_chunker.chunk(content)
     c. Compute current chunk hashes
     d. Call differ.diff(prev_markdown, curr_markdown, prev_hashes, curr_hashes)
     e. If unchanged: skip, update last_fetched_at
     f. If changed:
     • Register source if new
     • Create new source version
     • Embed only added chunks via embedder.embed()
     • Write chunks + lineage to DocumentStore
     • Write vectors to LanceDB
     • Retire removed chunks (DocumentStore + LanceDB)
     • Write sync log entries
  4. Return summary stats (sources processed, chunks added/removed/unchanged)

Step 8: Retrieval — retrieval/query.py

retrieve function:

• Accepts: query: str, embedder: Embedder, lance_db_path: Path, document_store: DocumentStore, top_k: int = 10
• Steps:
  1. Embed query via embedder.embed_query(query)
  2. Open LanceDB table at configured path
  3. Perform nearest-neighbor search with top_k
  4. Map results to dicts containing: chunk_text, chunk_hash, source_id, score
  5. Return results ordered by descending score
• get_lineage function: given a chunk_hash, query document_store.get_lineage() to return the full LineageRecord

Step 9: Tests

Unit tests:

• tests/storage/test_models.py — Model validation, hash determinism, frozen immutability
• tests/storage/test_schema.py — Schema initialization, idempotency, pragma verification
• tests/core/test_differ.py — Unchanged/added/removed classification, whitespace normalization
• tests/domains/test_notes.py — Heading splits, atomic blocks, context headers, size limits

Integration tests:

• tests/storage/test_document_store.py — Write/read roundtrip with in-memory SQLite
• tests/storage/test_vector_store.py — Write/search roundtrip with temp LanceDB directory
• tests/core/test_pipeline.py — Full ingest → re-ingest → retrieval flow

Test fixtures:

• Two sample markdown documents: sample_initial.md and sample_modified.md
• The modified version has one heading section changed, one added, and one removed
• Expected diff outcomes are documented alongside the fixtures

Step 10: Roadmap Documentation

Create documentation/roadmap.md defining:

• Phase 1 (this phase): Storage layer, data models, filesystem adapter, notes chunker, differ, embedder, pipeline, basic retrieval, tests
• Phase 2: Scheduler (poller/watcher), additional adapters (Gmail), messages domain
• Phase 3: Remaining domains (events, tasks), remaining adapters (Spotify, Todoist)
• Phase 4: Advanced retrieval (reranker, provenance API), cross-reference detection, performance optimization

---

Key Constraints and Risks

Constraint: LanceDB Schema Rigidity

LanceDB tables have a fixed vector dimension at creation time. If the embedding model changes, the LanceDB table must be dropped and recreated. The persistence design already documents this as the expected migration path. The lancedb_sync_log supports rebuild tracking.

Constraint: sentence-transformers Model Download

The Embedder requires downloading the model on first use (~80MB for all-MiniLM-L6-v2). Tests that use the embedder will be slow on first run. Integration tests should use a fixture that caches the model or use a mock embedder for unit tests.

Risk: Mistune AST Stability

The chunker depends on mistune v3's AST output format. Mistune's AST structure is not formally versioned. Pin mistune>=3.0,<4.0 in dependencies and add a regression test that verifies the expected AST structure for a known markdown input.

Risk: chunk_hash as PRIMARY KEY in chunks table

The current schema uses chunk_hash as the PRIMARY KEY for chunks. If two different sources produce identical content, they collide. The schema's UNIQUE constraint on (source_id, source_version, chunk_index) provides per-source uniqueness, but the PRIMARY KEY collision means only the first source's chunk is stored. This is acceptable for Phase 1 (single adapter, notes domain) but must be revisited if cross-source deduplication semantics need to differ.

---

Generated by Orchestrator Bot 🤖
Processed by the software_architect agent

[orchestrator-bot[bot]]

Work Breakdown Complete — created 10 sub-issue(s):

• Phase 1: Data Models and Schema Foundation #42 Phase 1: Data Models and Schema Foundation
• Phase 2: Document Store (SQLite) #43 Phase 2: Document Store (SQLite)
• Phase 3: Adapter Contract and Filesystem Adapter #44 Phase 3: Adapter Contract and Filesystem Adapter
• Phase 4: Notes Domain Chunker #45 Phase 4: Notes Domain Chunker
• Phase 5: Differ #46 Phase 5: Differ
• Phase 6: Embedder #47 Phase 6: Embedder
• Phase 7: Ingestion Pipeline Orchestrator #48 Phase 7: Ingestion Pipeline Orchestrator
• Phase 8: Retrieval Layer #49 Phase 8: Retrieval Layer
• Phase 9: Test Fixtures and Integration Test Suite #50 Phase 9: Test Fixtures and Integration Test Suite
• Phase 10: Roadmap Documentation #51 Phase 10: Roadmap Documentation

[orchestrator-bot[bot]]

Pr Code Reviewer

---

I'll start by checking out the PR branch and understanding the changes.We're already on the feature/issue-40-mvp-phase-1 branch, which appears to be the PR branch. Let me verify this is indeed PR #52.This is a large PR with 23 files changed and ~6,280 lines added. Now let me run the comprehensive review skill.

Let me start by understanding the full scope of changes in this PR, then run each specialized review agent sequentially.Now launching the code-reviewer agent first.Now launching the test analyzer agent.Now launching the silent-failure-hunter agent.Now launching the comment-analyzer agent.Now launching the type-design-analyzer agent.Now launching the code-simplifier agent.All six review agents have completed. Let me now compile the comprehensive findings into the structured format.

PR Review Findings

Critical Issues

• chunk_type schema CHECK constraint mismatch causes silent data loss: NotesDomain produces chunks with chunk_type="code" (/workspace/src/context_library/domains/notes.py:162) and chunk_type="table" (/workspace/src/context_library/domains/notes.py:195), but the SQLite schema at /workspace/src/context_library/storage/schema.sql:62 only allows CHECK (chunk_type IN ('standard', 'oversized', 'table_part')). Combined with INSERT OR IGNORE in write_chunks() at /workspace/src/context_library/storage/document_store.py:253-275, every code block and table chunk is silently dropped from SQLite while still being written to LanceDB, creating permanent store desynchronization. (NEW)

• INSERT OR IGNORE in write_chunks() suppresses ALL constraint violations: /workspace/src/context_library/storage/document_store.py:253-275 -- The INSERT OR IGNORE pattern silently swallows foreign key violations, UNIQUE constraint violations on (source_id, source_version, chunk_index), and CHECK constraint violations -- not just the intended content-deduplication on chunk_hash. This makes data integrity bugs invisible. (NEW)

• No error handling in 175-line multi-store pipeline orchestration: /workspace/src/context_library/core/pipeline.py:52-227 -- The ingest() method performs writes to both SQLite and LanceDB with zero try/except. A failure mid-iteration leaves the system in an inconsistent state (SQLite may have data that LanceDB does not, or vice versa) with no rollback, no partial-failure reporting, and no caller notification. (NEW)

• Pipeline docstring falsely claims atomicity: /workspace/src/context_library/core/pipeline.py:28 -- The class docstring states "Write to both SQLite and LanceDB atomically" but SQLite writes (lines 135-143, 169-170) and LanceDB writes (lines 191-194) are completely independent operations with no transactional coordination. (NEW)

High Priority Issues

• SQL injection risk in LanceDB delete filter: /workspace/src/context_library/core/pipeline.py:205-206 -- Hash values are interpolated directly into a SQL filter string via f"'{h}'". While hashes should be hex-only, the values come from DiffResult which uses set[str] (not Sha256Hash), so there is no validation at this point. A malformed hash could inject SQL into LanceDB's delete operation. (NEW)

• Sha256Hash validation applied inconsistently: DiffResult.added_hashes, DiffResult.removed_hashes, DiffResult.unchanged_hashes, DiffResult.prev_hash, DiffResult.curr_hash (all in /workspace/src/context_library/storage/models.py:137-151), and SourceVersion.chunk_hashes (/workspace/src/context_library/storage/models.py:126) all use bare str/set[str] instead of Sha256Hash. Only Chunk.chunk_hash and LineageRecord.chunk_hash use the validated type. (NEW)

• No validation of embedding output dimensions or integrity: /workspace/src/context_library/core/embedder.py:45-55 uses typing.cast() which performs zero runtime validation. /workspace/src/context_library/storage/validators.py has validate_embedding_dimension() specifically for this purpose, but it is never called anywhere in the pipeline. NaN/Inf values would silently corrupt the vector index. (NEW)

• FilesystemAdapter silently returns empty iterator on bad directory path: /workspace/src/context_library/adapters/filesystem.py:64-70 -- Missing or non-directory paths log a warning and return an empty iterator. The pipeline reports sources_processed: 0 with no error, making misconfigurations invisible to the caller. (NEW)

• INSERT OR IGNORE in register_adapter() silently drops config updates: /workspace/src/context_library/storage/document_store.py:99-112 -- If an adapter's normalizer_version or config changes, re-registration silently keeps the old values, creating provenance inconsistency between the adapters table and lineage records. (NEW)

• Unrecognized AST block types silently discarded: /workspace/src/context_library/domains/notes.py:300-302 -- The _render_block_to_markdown() fallback returns empty string for any block type not explicitly handled (block_html, def_list, footnote, math_block, etc.), silently losing content with no logging. (NEW)

• FilesystemAdapter catch blocks overly broad: /workspace/src/context_library/adapters/filesystem.py:117-122 -- The except OSError catch is the base class for PermissionError, FileNotFoundError, TimeoutError, etc. Permission issues across entire subdirectories would silently skip all files with no signal to the caller. (NEW)

• Two divergent normalization implementations: /workspace/src/context_library/core/differ.py:19-38 (Differ._normalize) and /workspace/src/context_library/storage/models.py:169-192 (compute_chunk_hash) both normalize-then-hash but with subtly different rules ([ \t]+ vs \s+, different stripping behavior). Neither references the other. (NEW)

Medium Priority Issues

• soft_limit parameter is dead code: /workspace/src/context_library/domains/notes.py:29,35,39 -- The soft_limit parameter is accepted, stored as self.soft_limit, and documented, but never referenced in any method. Users are misled into thinking they can tune chunking via this parameter. (NEW)

• DocumentStore.conn is a public attribute: /workspace/src/context_library/storage/document_store.py -- The SQLite connection self.conn is directly accessible, allowing external code to bypass all transaction, validation, and foreign key logic. (NEW)

• IngestionPipeline exposes all attributes publicly: /workspace/src/context_library/core/pipeline.py -- self.document_store, self.embedder, self.differ, self.vector_store_path are all public, allowing callers to replace dependencies after construction. (NEW)

• DiffResult has no invariant enforcement: /workspace/src/context_library/storage/models.py:137-151 -- No validators ensure set disjointness, changed flag consistency with hash sets, or that prev_hash is None implies removed_hashes is empty. This model controls pipeline branching. (NEW)

• Mutable containers in frozen Pydantic models: StructuralHints.natural_boundaries (list[int]), Chunk.domain_metadata (dict), SourceVersion.chunk_hashes (list[str]), DiffResult hash sets (set[str]), AdapterConfig.config (dict) -- all are mutable in-place despite frozen=True only preventing reassignment. (NEW)

• compute_chunk_hash docstring inaccurately describes normalization order: /workspace/src/context_library/storage/models.py:172-176 -- The docstring lists "collapse whitespace" then "strip trailing whitespace" but the code does rstrip() first, then \s+ collapse. (NEW)

• No LanceDB connection cleanup: /workspace/src/context_library/core/pipeline.py:83 -- db = lancedb.connect(...) is never explicitly closed. In long-running processes, connections may accumulate. (NEW)

• retire_chunks() and update_last_fetched_at() silently affect zero rows: /workspace/src/context_library/storage/document_store.py:277-294,584-598 -- UPDATE operations on non-existent keys produce no error, potentially leaving orphaned chunks or stale timestamps. (NEW)

• Unused operation parameter in write_sync_log: /workspace/src/context_library/storage/document_store.py:296 -- Documented as "currently ignored", never used in method body. Dead parameter. (NEW)

• Unused prev_version_id parameter in create_source_version: /workspace/src/context_library/storage/document_store.py:153 -- Documented as "for compatibility", never referenced. Dead parameter. (NEW)

• typing.Optional vs X | None inconsistency: /workspace/src/context_library/storage/document_store.py uses Optional[int] (lines 153, 332, 413, 472, 510, 551, 647) while the rest of the codebase uses X | None syntax. (NEW)

• ingest() return type is unstructured dict[str, int]: /workspace/src/context_library/core/pipeline.py -- The return type should be a TypedDict or Pydantic model to prevent key name typos and provide IDE support. (NEW)

Low Priority / Nice-to-Have

• Dead code at notes.py:494: self._token_count(word) return value is discarded -- appears to be leftover debugging code. (NEW)

• Duplicated CandidateChunk dictionary construction: /workspace/src/context_library/domains/notes.py:140-146,169-179,200-212,220-230 -- Nearly identical dict construction appears 4+ times. A _make_candidate() helper would reduce repetition. (NEW)

• Duplicated "flush current candidate" pattern: /workspace/src/context_library/domains/notes.py:125-126,150-151,183-184,240-241 -- The check-and-append pattern appears 4 times. Could be extracted to a helper. (NEW)

• Duplicated row-to-model conversion in DocumentStore: Row-to-SourceVersion at lines 361-369 and 397-406; row-to-Chunk at lines 453-468 and 497-508. Private helpers would reduce duplication. (NEW)

• Duplicated LanceDB table-existence check: /workspace/src/context_library/core/pipeline.py:189,201 -- db.list_tables() called twice in the same loop iteration. (NEW)

• IngestionPipeline.ingest() is 175 lines: Could be decomposed into _register_and_version_source(), _embed_and_build_lineage(), _write_to_lancedb(), _retire_and_sync(). (NEW)

• Imprecise has_headings / has_tables detection: /workspace/src/context_library/adapters/filesystem.py:84,88 -- "#" in markdown and "|" in markdown match characters in any context (code blocks, URLs). has_lists uses proper regex; headings/tables should too. (NEW)

• Redundant continue statements after except: /workspace/src/context_library/adapters/filesystem.py:119,122 -- At end of loop body, continue is a no-op. (NEW)

• Obvious inline comments restating code: Multiple locations in pipeline.py (lines 85, 91, 101, 108, 150-151, 217) and filesystem.py (lines 77, 80, 109) have comments like # Read file contents before read_text(). (NEW)

• Missing test for Differ ValueError when prev_chunk_hashes=None: /workspace/src/context_library/core/differ.py:99-100 has a validation guard with no test coverage. (NEW)

• Missing test for write_chunks ValueError on missing lineage: /workspace/src/context_library/storage/document_store.py:233-237 has a data integrity check with no test coverage. (NEW)

• Missing test for sentence/word-level oversized chunk splitting: /workspace/src/context_library/domains/notes.py:447-561 -- Only paragraph-level splitting is tested. Sentence-level and word-level fallback paths have no coverage. (NEW)

• StructuralHints.modified_at is bare str: /workspace/src/context_library/storage/models.py:65 -- No format validation; different adapters could produce inconsistent timestamp formats. (NEW)

• BaseDomain has no domain property: /workspace/src/context_library/domains/base.py -- No type-level connection between a domain chunker and the Domain enum it handles. (NEW)

Clean Areas

• Domain enum design (/workspace/src/context_library/storage/models.py:17-27): Clean str, Enum pattern with SQL CHECK mirroring. Well-designed.
• BaseAdapter abstract contract and test coverage (/workspace/tests/adapters/test_base.py): Thorough ABC enforcement tests for every abstract method.
• Pydantic model immutability (frozen=True across all models): Consistent design decision applied uniformly.
• Sha256Hash validated type alias: Excellent use of Annotated + AfterValidator for content-addressed hash validation.
• DocumentStore constructor validation: Strong startup checks for WAL mode, foreign keys, and schema version.
• Differ.diff() docstring: Accurate parameter documentation with correct None semantics.
• NotesDomain.chunk() algorithm docstring: Clear numbered steps matching actual implementation.
• End-to-end pipeline integration tests (/workspace/tests/core/test_pipeline.py): High-value tests covering first ingest, re-ingest, modification, addition, and deletion with both SQLite and LanceDB verification.
• SQL injection prevention test (/workspace/tests/storage/test_document_store.py:TestParameterizedQueries): Good practice verifying parameterized queries.
• LineageRecord model (/workspace/src/context_library/storage/models.py:102-116): Cleanest model in the system -- no mutable containers, good use of Domain and Sha256Hash, pure frozen data record.All six specialized review agents have completed their analysis. The findings above represent the consolidated output from:

1. code-reviewer -- general code quality, bugs, design issues
2. pr-test-analyzer -- test coverage gaps and quality
3. silent-failure-hunter -- silent failures and error handling
4. comment-analyzer -- comment accuracy and documentation
5. type-design-analyzer -- type design, encapsulation, invariants
6. code-simplifier -- simplification opportunities

The single most urgent finding is the chunk_type schema mismatch (Critical #1), which is an active data-loss bug: NotesDomain produces "code" and "table" chunk types that fail the SQL CHECK constraint, and INSERT OR IGNORE silently drops them. This creates permanent desynchronization between SQLite and LanceDB from day one.

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR and the roadmap to verify implementation completeness. Let me gather all the necessary information.Now let me check the local workspace to verify the retrieval query component and any other files mentioned in the roadmap.Now I have a complete picture. The PR diff includes 23 files changed from main. The retrieval query file (src/context_library/retrieval/query.py) is only 1 line (a docstring stub) and is not in the PR diff. Let me verify this and confirm the state of all Phase 1 components.The retrieval/query.py file exists in main as a 1-line stub since the initial scaffold commit, and has not been modified in this PR. This is the key finding.

---

Parent Issue Requirements Verification

The parent issue instructs: "Begin development of the scope outlined in Phase 1 of documentation/roadmap.md"

The roadmap defines 10 Phase 1 components. The PR covers 9 of them but leaves one unimplemented.

---

Gaps Found

• Retrieval Query (src/context_library/retrieval/query.py) not implemented: The roadmap explicitly lists this as a Phase 1 component — "Semantic search interface providing vector similarity queries with optional metadata filtering over the document store." The file exists only as a 1-line docstring stub ("""Semantic query interface combining vector search with optional metadata filters.""") carried over from the initial project scaffold. It was not modified in this PR and contains no implementation. The corresponding test file (tests/retrieval/test_query.py) also exists in the repo but was not added or modified by this PR.

Deviations

None found.

Verified

• Data Models (src/context_library/storage/models.py) — Implemented: Pydantic models for NormalizedContent, Chunk, LineageRecord, SourceVersion, DiffResult, and compute_chunk_hash().
• Document Store (src/context_library/storage/document_store.py) — Implemented: SQLite-backed store with adapter registration, source tracking, version history, chunk storage, and LanceDB sync state.
• Adapter Contract (src/context_library/adapters/base.py) — Implemented: Abstract base class with adapter_id, domain, normalizer_version, and fetch() contract.
• Filesystem Adapter (src/context_library/adapters/filesystem.py) — Implemented: Recursive markdown discovery with structural hint extraction and file metadata.
• Domain Contract (src/context_library/domains/base.py) — Implemented: Abstract base with chunk() method contract.
• Notes Domain (src/context_library/domains/notes.py) — Implemented: Heading-hierarchy chunking with code/table preservation, soft/hard token limits, and breadcrumb context headers.
• Differ (src/context_library/core/differ.py) — Implemented: SHA-256 content hashing to identify new, modified, and deleted sources.
• Embedder (src/context_library/core/embedder.py) — Implemented: sentence-transformers wrapper with embed() and embed_query().
• Pipeline Orchestrator (src/context_library/core/pipeline.py) — Implemented: End-to-end ingestion workflow (fetch → normalize → diff → chunk → embed → store) with chunk retirement and sync logging.
• Out-of-scope deferral confirmed: core/versioner.py and core/chunker.py are correctly noted as reserved/deferred in the updated roadmap and not implemented in this PR.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the business requirements. Let me fetch the PR details first.Now I have comprehensive information about what was implemented. Let me analyze this against each functional requirement.Now I have all the information needed to produce a thorough verification. Let me compile the findings:

---

Business Analyst Output Verification

Gaps Found

• FR-3.4 Embedding Dimension Not Model-Driven: Specified that the embedding dimension must be configurable and driven by the configured embedding model. EMBEDDING_DIM is a hardcoded constant (384) in storage/validators.py and storage/vector_store.py. The Embedder.dimension property exists, but it is never consulted when constructing the vector store schema — the LanceDB Vector(EMBEDDING_DIM) annotation uses the static compile-time constant, not the runtime embedder's reported dimension.

• FR-9.1 / FR-9.2 / FR-9.3 Retrieval Function Not Implemented: retrieval/query.py contains only a module docstring. FR-9.1 requires a retrieval function accepting a query string, embedding it, and performing nearest-neighbor search. FR-9.2 requires results to include chunk text, chunk hash, source identifier, and relevance score. FR-9.3 requires a top_k parameter. None of these exist as callable API. The integration test exercises LanceDB search directly without going through any retrieval abstraction.

• FR-9.4 Lineage Lookup via Retrieval Layer Not Implemented: Related to FR-9.1 gap — because no retrieval function exists, the specified "given a chunk hash from a retrieval result, the full lineage record must be retrievable from the document store" path has no public entry point in the retrieval module. document_store.get_lineage() does exist but always returns "unspecified" for embedding_model_id, meaning the lineage record returned is incomplete per FR-3.3 (which requires embedding_model_id to be populated).

• FR-3.3 LineageRecord embedding_model_id Not Persisted: get_lineage() always returns the placeholder string "unspecified" for embedding_model_id. The value is correctly set during the pipeline write path (embedder.model_id), but the chunks table has no column to store it, so it is never persisted to SQLite and cannot be recovered on read.

• FR-5.6 Soft Token Limit Join Logic Deficient: The requirement states chunker must join "short adjacent blocks" where "short" is defined relative to the soft token limit. The implementation uses min_floor (64 tokens) as the join threshold — the soft_limit (512) parameter is accepted but never consulted in _apply_token_limits. Additionally, joining is restricted to blocks sharing the exact same context_header, meaning two small sibling sections under different headings are never joined even if both are well below the soft limit.

• FR-6.4 Blank-Line-Only Changes Not Treated as Unchanged: The differ normalizes per-line trailing whitespace and collapses runs of spaces/tabs, but does not collapse multiple consecutive blank lines. A document where the only change is \n\n → \n\n\n between paragraphs will produce a different hash and be treated as a changed document, violating the requirement that "whitespace-only changes" produce an unchanged result.

Deviations

• FR-8.5 Sync Log Does Not Record Operation Type: The lancedb_sync_log schema has no operation column and write_sync_log silently ignores the operation parameter passed to it. Deletes are tracked by row removal rather than a logged delete entry. The sync log therefore does not provide a full audit trail of insert vs. delete operations as the name and design imply.

• FR-2.6 LanceDB Table Schema Not Used for Table Creation: storage/vector_store.py defines a ChunkVector(LanceModel) class as the specified schema, but the pipeline creates/opens the LanceDB table by writing plain dicts, not by using the ChunkVector model. The model serves as documentation only; table creation does not enforce the defined schema through it.

Verified

• FR-1.1: documentation/roadmap.md exists, defines Phase 1 scope with all components listed including their source file paths, and describes deferred components for later phases.
• FR-2.1: All five tables (adapters, sources, source_versions, chunks, lancedb_sync_log) are created in schema.sql.
• FR-2.2: Schema sets PRAGMA user_version=1, enables WAL mode (journal_mode=WAL), enables foreign key enforcement (foreign_keys=ON), and sets synchronous=NORMAL.
• FR-2.3: All tables use CREATE TABLE IF NOT EXISTS.
• FR-2.4: Document store supports the full write path: register_adapter, register_source, create_source_version, write_chunks with lineage records, write_sync_log.
• FR-2.5: Document store supports get_chunks_by_source, get_version_history, and get_chunk_by_hash.
• FR-3.1: Pydantic models exist for all required types: NormalizedContent, StructuralHints, Chunk, LineageRecord, SourceVersion, DiffResult, AdapterConfig.
• FR-3.2: compute_chunk_hash produces a SHA-256 hash of normalized content deterministically.
• FR-3.3: LineageRecord includes all seven required fields (chunk_hash, source_id, source_version_id, adapter_id, domain, normalizer_version, embedding_model_id).
• FR-4.1: BaseAdapter ABC defines the adapter contract with fetch yielding NormalizedContent.
• FR-4.2: FilesystemAdapter implements BaseAdapter, accepts a directory path, and yields NormalizedContent for all .md files found recursively via rglob("*.md").
• FR-4.3: FilesystemAdapter populates StructuralHints with file_path, modified_at (ISO 8601), and file_size_bytes.
• FR-4.4: Adapter records itself via register() on first use and retrieves its adapter_id idempotently on subsequent calls.
• FR-5.1: NotesDomain class implements domain-specific chunking for markdown.
• FR-5.2: Uses mistune.create_markdown(renderer=None) to parse markdown into an AST before splitting.
• FR-5.3: Splits on headings, treating each heading and its content as a candidate chunk.
• FR-5.4: Code blocks and tables are treated as atomic chunks that are never split mid-block.
• FR-5.5: Each chunk includes a context header containing the heading breadcrumb path (e.g., "# H1 > ## H2 > ### H3").
• FR-6.1: Differ accepts two content versions and produces a DiffResult classifying chunks as unchanged, added (new), or removed.
• FR-6.2: If full-document SHA-256 hash matches, returns unchanged result without further processing.
• FR-6.3: If document hash differs, compares chunk hash sets and classifies each as added, removed, or unchanged via set arithmetic.
• FR-7.1: Embedder wraps sentence-transformers and exposes embed(texts) returning a list of vectors.
• FR-7.2: Embedder exposes model_id and dimension as properties.
• FR-7.3: Embedder supports batch processing via model.encode(texts, ...).
• FR-8.1: IngestionPipeline.ingest orchestrates the full flow: adapter → domain chunker → differ → embedder → document store write → vector store write.
• FR-8.2: First ingest creates a new source_version record and writes all chunks with lineage records and embeddings.
• FR-8.3: Re-ingest invokes the differ, skips unchanged chunks, writes only new/modified chunks, and creates a new version referencing the previous.
• FR-8.4: Removed chunks are retired in the document store (retire_chunks) and deleted from the LanceDB index.
• FR-10.1: Unit tests cover hash computation determinism, differ logic (unchanged/added/removed), chunker output (atomic blocks, heading splits, context headers), and model validation.
• FR-10.2: Integration tests cover document store write/read roundtrip (SQLite in-memory), vector store write/search roundtrip (LanceDB temp dir), and full pipeline ingest → re-ingest → retrieval flow.
• FR-10.3: Two fixture markdown documents exist (sample_initial.md, sample_modified.md) with known expected diff outcomes via expected_diff.py.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the architecture specifications. Let me fetch the PR details and examine the implementation.Now let me read the actual implementation files in the workspace to compare against the specifications:Now I have a thorough picture of both the architecture specification and the implementation. Let me compile the verification results:

---

Software Architect Output Verification

Gaps Found

• retrieval/query.py Not Implemented: The architecture explicitly includes retrieval/query.py as an in-scope Phase 1 component (Step 8 in the build order), specifying it should provide "vector search + lineage lookup" combining the Embedder, VectorStore, and DocumentStore. The roadmap.md even lists "Retrieval Query" as a completed Phase 1 component. The actual file contains only a one-line module docstring with zero implementation.

• embedding_model_id Not Persisted in chunks Table: The architecture specifies LineageRecord contains embedding_model_id for full provenance, and the get_lineage() method should return this field from stored data. The implementation hardcodes _EMBEDDING_MODEL_PLACEHOLDER = "unspecified" in DocumentStore.get_lineage() and the chunks table has no embedding_model_id column. The schema (schema.sql) does not include this column. The architecture required this as part of complete provenance tracking — every chunk should trace back to its embedding model.

Deviations

• validators.py Extension — Signature Change vs. Additive Change: ADR-002 specifies "Add an optional expected_dim parameter to validate_embedding_dimension() for runtime flexibility." The implementation changes the function signature to validate_embedding_dimension(value, expected_dim: int = EMBEDDING_DIM). This matches the intent, but the original validate_embedding_dimension() in vector_store.py calls it without arguments — the change is backwards-compatible. This is correctly implemented as specified.

• Pipeline ingest() Passes Empty String as source_ref: The IngestionPipeline.ingest() calls adapter.fetch("") (line 92 of pipeline.py). The BaseAdapter.fetch() contract documents source_ref as "source-specific reference." For FilesystemAdapter, this is acceptable since it ignores the parameter. However, the architecture does not specify how the pipeline should invoke fetch() — this is an implementation choice not addressed in the spec.

Verified

• All 7 Pydantic models defined in storage/models.py: StructuralHints, NormalizedContent, Chunk, LineageRecord, SourceVersion, DiffResult, AdapterConfig — all present alongside the existing Domain enum. frozen=True immutability applied on all models.
• compute_chunk_hash() helper in storage/models.py: Implements whitespace normalization (collapse runs, strip trailing per line, normalize line endings) before SHA-256, matching Pattern 1 specification exactly.
• DocumentStore as class wrapping sqlite3 (ADR-003): Accepts db path or :memory:, manages connection, applies WAL mode + foreign keys + PRAGMA user_version=1, executes schema.sql via executescript() on construction.
• DocumentStore typed methods: All specified methods implemented — register_adapter, register_source, create_source_version, write_chunks (batch INSERT OR IGNORE), retire_chunks, get_latest_version, get_version_history, get_chunks_by_source, get_chunk_by_hash, get_lineage, update_last_fetched_at.
• adapters/filesystem.py created as new file (ADR-004): FilesystemAdapter(BaseAdapter) in its own file, following one-class-per-file convention.
• FilesystemAdapter behavior: rglob("*.md"), UTF-8 read, StructuralHints with has_headings/has_lists/has_tables, modified_at in ISO 8601, file_size_bytes, adapter ID format "filesystem:{absolute_path}", domain NOTES, normalizer version "1.0.0".
• adapters/base.py ABC: Abstract fetch(), adapter_id, domain, normalizer_version properties; concrete register() method — all as specified.
• domains/base.py ABC: Abstract chunk(content: NormalizedContent) -> list[Chunk] — matches spec.
• domains/notes.py NotesDomain: mistune-based AST parsing, heading hierarchy, context headers as breadcrumbs, atomic code/table blocks, soft_limit=512 / hard_limit=1024 / min_floor=64, hash computed before context header prepend, sequential chunk_index assignment.
• core/chunker.py not implemented (ADR-005): Remains a stub, correctly deferred.
• core/differ.py Differ: Stateless, SHA-256 with whitespace normalization, set operations for added/removed/unchanged, short-circuit on full-document hash equality, handles first-ingest (no previous version).
• core/embedder.py Embedder: Wraps sentence-transformers, default "all-MiniLM-L6-v2", batch embed(), single-query embed_query(), exposes model_id and dimension properties derived from loaded model.
• ADR-002 Embedder dimension configurability: Embedder.dimension property uses get_sentence_embedding_dimension() from the loaded model; validators.py retains EMBEDDING_DIM = 384 as default with optional expected_dim parameter added.
• core/pipeline.py IngestionPipeline: Orchestrates fetch → normalize → diff → chunk → embed → store; SQLite-first then LanceDB write order (Pattern 2); adapter self-registration (Pattern 3); chunk retirement (soft-delete via retire_chunks); lancedb_sync_log table updated via write_sync_log/delete_sync_log; returns sources_processed, chunks_added, chunks_removed, chunks_unchanged.
• Context header NOT included in hash (Pattern 5): Hash computed on content only, context header prepended afterward — correctly implemented in NotesDomain.chunk().
• Deferred components correctly excluded: domains/messages.py, events.py, tasks.py, retrieval/reranker.py, retrieval/provenance.py, scheduler/poller.py, scheduler/watcher.py, core/versioner.py — all remain stubs.
• documentation/roadmap.md updated (Step 10): Roadmap updated to reflect Phase 1 completion with all components described, out-of-scope items documented, and future phases outlined.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

Looking at the findings from all review phases, I need to deduplicate, filter, and consolidate them into a single grouped output.

Let me verify the most critical finding before finalizing.Confirmed: the chunk_type CHECK constraint at schema.sql:62 only allows ('standard', 'oversized', 'table_part') but notes.py produces "code" (line 162) and "table" (line 195). Combined with INSERT OR IGNORE at document_store.py:253, these chunks are silently dropped.

Now let me verify a few more findings before consolidating.Now let me check the retrieval stub and the differ normalization discrepancy.Good. The normalization divergence is confirmed:

• Differ._normalize uses r"[ \t]+" (only spaces and tabs), applies .strip() on entire text, does NOT normalize line endings
• compute_chunk_hash uses r"\s+" (all whitespace including newlines within a line), applies .rstrip() per line but no .strip() on entire text, normalizes \r\n/\r to \n

These are different algorithms that could produce different hashes for the same content. Now let me verify the embedding dimension issue.Now I have all the verification I need. Let me also quickly check the get_lineage() method to confirm the embedding_model_id finding.Confirmed: soft_limit is stored but never used in any method logic. And embedding_model_id is documented as a known Phase 1 limitation.

Now let me check whether the embedding_model_id gap is documented as an intentional Phase 1 deferral or if it's an actual gap.Not documented in the roadmap as an intentional deferral. Let me check the specs.So the pipeline does set embedding_model_id on the LineageRecord object (line 164), but write_chunks never persists it to the chunks table (no column exists). And get_lineage returns a hardcoded placeholder. The data is computed but lost.

Now I have all the context needed. Let me produce the consolidated JSON output.

{
"groups": [
{
"name": "Schema & Data Integrity (SQLite)",
"severity": "critical",
"findings": "- chunk_type CHECK constraint rejects domain-produced values, causing silent data loss: NotesDomain produces chunk_type=\"code\" (domains/notes.py:162) and chunk_type=\"table\" (domains/notes.py:195), but schema.sql:62 only allows ('standard', 'oversized', 'table_part'). Combined with INSERT OR IGNORE in write_chunks() (document_store.py:253-275), these chunks are silently dropped from SQLite while still written to LanceDB, creating permanent store desynchronization.\n- INSERT OR IGNORE in write_chunks() suppresses all constraint violations: document_store.py:253-275 — Silently swallows foreign key violations, UNIQUE constraint violations, and CHECK constraint violations. Only content-deduplication on chunk_hash was intended, but all error categories are invisible.\n- INSERT OR IGNORE in register_adapter() silently drops config updates: document_store.py:99-112 — If an adapter's normalizer_version or config changes, re-registration silently keeps old values, creating provenance inconsistency between the adapters table and lineage records.\n- embedding_model_id computed but never persisted: Pipeline sets embedding_model_id=self.embedder.model_id on LineageRecord (pipeline.py:164), but the chunks table has no column for it (schema.sql:49-68). get_lineage() returns hardcoded \"unspecified\" (document_store.py:548). FR-3.3 and the architecture spec require this field for full provenance."
},
{
"name": "Pipeline Orchestrator (pipeline.py)",
"severity": "critical",
"findings": "- No error handling in 175-line multi-store pipeline: pipeline.py:52-227 — The ingest() method writes to both SQLite and LanceDB with zero try/except. A failure mid-iteration leaves stores in an inconsistent state with no rollback or partial-failure reporting.\n- Docstring falsely claims atomicity: pipeline.py:28 — States "Write to both SQLite and LanceDB atomically" but the operations are completely independent with no transactional coordination.\n- SQL injection risk in LanceDB delete filter: pipeline.py:205-206 — Hash values interpolated directly into a SQL filter string via f-string. DiffResult hash sets use bare str (not Sha256Hash), so no validation occurs at this point."
},
{
"name": "Retrieval Module (Missing Implementation)",
"severity": "high",
"findings": "- retrieval/query.py not implemented: The file contains only a one-line module docstring (retrieval/query.py:1). The roadmap, architecture spec, and functional requirements (FR-9.1, FR-9.2, FR-9.3, FR-9.4) all explicitly list this as a Phase 1 component requiring: a retrieval function accepting a query string, embedding it, performing nearest-neighbor search, returning chunk text/hash/source/score, supporting top_k, and providing lineage lookup via chunk hash."
},
{
"name": "Hash Normalization Divergence",
"severity": "high",
"findings": "- Two divergent normalization-then-hash implementations: Differ._normalize (differ.py:19-38) uses r\"[ \\t]+\" (spaces/tabs only), applies .strip() on entire text, and does NOT normalize line endings. compute_chunk_hash (models.py:169-192) uses r\"\\s+\" (all whitespace), applies .rstrip() per line without .strip() on entire text, and normalizes \\r\\n/\\r to \\n. These produce different hashes for the same content in edge cases, meaning the differ's document-level hash and the chunk-level hash use incompatible normalization.\n- FR-6.4 violation — blank-line-only changes not treated as unchanged: The differ normalizes per-line trailing whitespace but does not collapse multiple consecutive blank lines. A change from \\n\\n to \\n\\n\\n produces a different hash, violating the requirement that whitespace-only changes produce an unchanged result."
},
{
"name": "Embedding Validation",
"severity": "high",
"findings": "- Embedding dimension hardcoded, not model-driven: EMBEDDING_DIM = 384 is a compile-time constant in validators.py:9 and vector_store.py:34. The Embedder.dimension property exists (embedder.py:31-43) but is never consulted when constructing the LanceDB table schema. FR-3.4 requires the dimension to be driven by the configured embedding model.\n- No runtime validation of embedding output: embedder.py:45-55 uses typing.cast() which performs zero runtime checks. validate_embedding_dimension() in validators.py:12-36 exists specifically for this purpose (including NaN/Inf detection) but is never called in the pipeline path. The ChunkVector model in vector_store.py has the validator, but the pipeline writes plain dicts to LanceDB and never instantiates ChunkVector."
},
{
"name": "NotesDomain Chunker",
"severity": "high",
"findings": "- soft_limit parameter is accepted but never used: domains/notes.py:29,35,39 — The parameter is stored as self.soft_limit and documented but never referenced in any chunking logic. _apply_token_limits uses min_floor (64) as the join threshold instead. FR-5.6 requires joining short adjacent blocks relative to the soft token limit.\n- Unrecognized AST block types silently discarded: domains/notes.py:300-302 — The _render_block_to_markdown() fallback returns empty string for unhandled block types (block_html, def_list, footnote, math_block, etc.), silently losing content with no logging."
},
{
"name": "Filesystem Adapter Error Handling",
"severity": "high",
"findings": "- Silently returns empty iterator on bad directory path: filesystem.py:64-70 — Missing or non-directory paths log a warning and return an empty iterator. The pipeline reports sources_processed: 0 with no error, making misconfigurations invisible to the caller.\n- Overly broad except OSError: filesystem.py:117-122 — Catches the base class for PermissionError, FileNotFoundError, TimeoutError, etc. Permission issues across entire subdirectories silently skip all affected files."
},
{
"name": "LanceDB Sync & Schema Usage",
"severity": "medium",
"findings": "- ChunkVector LanceModel not used for table creation: vector_store.py:22-55 defines ChunkVector(LanceModel) as the schema, but the pipeline creates/opens the LanceDB table by writing plain dicts (pipeline.py:191-194), bypassing all field validators (vector dimension, timestamp format). The model serves as documentation only. FR-2.6 requires the schema model to be used.\n- Sync log does not record operation type: schema.sql:94-98 has no operation column, and write_sync_log silently ignores the operation parameter passed to it (document_store.py:296). FR-8.5 requires distinguishing insert vs. delete operations."
},
{
"name": "Data Model Invariants",
"severity": "medium",
"findings": "- Sha256Hash validation applied inconsistently: DiffResult hash fields (models.py:137-151) and SourceVersion.chunk_hashes (models.py:126) all use bare str/set[str] instead of the validated Sha256Hash type. Only Chunk.chunk_hash and LineageRecord.chunk_hash use the validated type, leaving the pipeline's branching logic (DiffResult) unvalidated.\n- DiffResult has no invariant enforcement: models.py:137-151 — No validators ensure set disjointness between added_hashes/removed_hashes/unchanged_hashes, or that changed flag is consistent with the hash sets.\n- Unused operation parameter in write_sync_log: document_store.py:296 — Documented as "currently ignored", never used. Dead parameter.\n- Unused prev_version_id parameter in create_source_version: document_store.py:153 — Documented as "for compatibility", never referenced. Dead parameter."
}
],
"filtered_out": [
"Mutable containers in frozen Pydantic models — style/defensive concern, not a bug; frozen=True prevents reassignment which is the primary intent",
"DocumentStore.conn and IngestionPipeline public attributes — encapsulation preference, not a bug in committed code",
"ingest() return type should be TypedDict — style improvement, not a functional gap",
"typing.Optional vs X | None inconsistency — pure style preference",
"Dead code at notes.py:494 (_token_count return discarded) — minor dead code, not a functional issue",
"Duplicated CandidateChunk dict construction (notes.py) — code duplication suggestion, not a bug",
"Duplicated flush-candidate pattern (notes.py) — refactoring suggestion, not a bug",
"Duplicated row-to-model conversion in DocumentStore — refactoring suggestion, not a bug",
"Duplicated LanceDB table-existence check (pipeline.py:189,201) — minor inefficiency, not a bug",
"ingest() method length (175 lines) — decomposition suggestion, not a bug",
"Imprecise has_headings/has_tables detection (filesystem.py:84,88) — structural hints are advisory metadata, not used for correctness decisions",
"Redundant continue statements (filesystem.py:119,122) — style nit, no functional impact",
"Obvious inline comments restating code (pipeline.py, filesystem.py) — comment style preference",
"Missing test for Differ ValueError — test coverage gap for error path, not a bug in production code",
"Missing test for write_chunks ValueError — test coverage gap for error path, not a bug in production code",
"Missing test for sentence/word-level oversized splitting — test coverage gap, not a production bug",
"StructuralHints.modified_at bare str — no format mismatch exists currently; only one adapter produces timestamps",
"BaseDomain has no domain property — design suggestion, not required by spec",
"compute_chunk_hash docstring order inaccuracy — merged into the divergent normalization finding which covers the substantive issue",
"No LanceDB connection cleanup — resource management concern for long-running processes, not a Phase 1 requirement",
"retire_chunks and update_last_fetched_at silently affect zero rows — defensive concern, not currently causing bugs",
"Pipeline passes empty string as source_ref — FilesystemAdapter ignores the parameter by design"
]
}

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

I'll start by checking out the PR branch and understanding the changes, then run the comprehensive review.Good, we're already on the PR branch. Let me get the PR diff to understand the full scope of changes.Now let me run the comprehensive PR review skill.

I'll run a comprehensive PR review with all specialized agents sequentially. Let me first understand the scope of changes.Let me first check if there's a CLAUDE.md for project guidelines.Now let me get the full diff for the review agents.Large PR. Let me launch the code-reviewer agent first.The code-reviewer agent seems to have completed but returned no text output. Let me resume it to get its findings.Good, I have detailed findings from the code-reviewer. Now let me run the test analyzer.Now let me run the silent-failure-hunter agent.Now let me run the comment-analyzer agent.Now let me run the type-design-analyzer agent.All five review agents have completed. Let me now verify a few key findings before compiling the final report, to make sure the issues are real and not already fixed.Good, the issues are confirmed. Let me verify a few more.All findings confirmed. Let me also check the pipeline error handling and filesystem adapter structural hints.All findings confirmed. Now let me also check query.py for the empty query validation and the has_headings false positive concern.Now I have all the data I need to compile the final aggregated report. Let me deduplicate and organize findings across all five agents.Now let me compile the final structured findings.## PR Review Findings

Critical Issues

• [Inconsistent Whitespace Normalization Between Differ and compute_chunk_hash]: Differ._normalize() at /workspace/src/context_library/core/differ.py:38 applies text.strip() to the entire text, while compute_chunk_hash() at /workspace/src/context_library/storage/models.py:226-233 does not strip leading/trailing whitespace from the entire text. Additionally, Differ uses [ \t]+ regex (spaces/tabs only) while compute_chunk_hash uses \s+ (all whitespace types). This divergence means the Differ can consider two documents "unchanged" while their chunk hashes differ, causing stale data in the database when the pipeline skips re-indexing at pipeline.py:117-121. These two functions serve related hashing purposes and their normalization strategies should be unified or the divergence explicitly documented and justified. (NEW)

• [Ordered Lists Silently Converted to Unordered Lists]: _render_list() at /workspace/src/context_library/domains/notes.py:324-337 unconditionally formats every list item with "- " prefix (line 335), ignoring mistune's block["attrs"]["ordered"] attribute. Ordered lists like 1. Install 2. Configure 3. Run become - Install - Configure - Run, losing sequential semantics. Since chunk content is what gets embedded and returned by retrieval, users get semantically degraded content for procedural instructions and ranked lists. (NEW)

• [Pipeline Has No Per-Source Error Handling]: The ingest() method at /workspace/src/context_library/core/pipeline.py:95-254 iterates over all sources in a flat loop with no try-except around per-source processing. A single file failure (corrupted markdown, embedding model transient failure, SQLite lock timeout) aborts the entire ingestion pipeline. For a filesystem adapter scanning hundreds of files, this means complete data loss of progress. Sources already written to SQLite may lack corresponding LanceDB vectors, creating an inconsistent state with no recovery path. (NEW)

• [Silent Data Loss in Retrieval on Store Inconsistency]: At /workspace/src/context_library/retrieval/query.py:178-187, when a chunk_hash exists in LanceDB but the corresponding chunk or lineage record is missing from SQLite, the code silently continues with no logging or error reporting. Comments say "shouldn't happen" -- which is exactly when logging is most needed. Users get fewer results than expected with no indication that stores are out of sync. This can occur from partial pipeline failures (see non-atomic writes issue), race conditions, or schema migration errors. (NEW)

High Priority Issues

• [Non-Atomic Cross-Store Writes Between SQLite and LanceDB]: IngestionPipeline.ingest() at /workspace/src/context_library/core/pipeline.py:182-249 writes to SQLite (line 183), then LanceDB (lines 186-228), then retires from SQLite (line 232), then deletes from LanceDB (lines 234-240). If any step fails after preceding steps succeed, the stores are left in an inconsistent state with no rollback mechanism. The pipeline docstring at line 31 falsely claims these writes are "atomic." While LanceDB is described as "derived and disposable" elsewhere, no rebuild mechanism is implemented in this PR. (NEW)

• [AST Parsing Failure Silently Produces Empty Chunks]: At /workspace/src/context_library/domains/notes.py:69-70, when the mistune parser returns an unexpected type (not a list), the code silently falls back to an empty list: ast = ast_result if isinstance(ast_result, list) else []. No warning, no error, no log entry. An entire document silently vanishes from the system with zero chunks, undetectable by the user. This can occur from mistune version incompatibility, parser misconfiguration, or malformed input. (NEW)

• [Broad except Exception Catches in Retrieval Module]: At /workspace/src/context_library/retrieval/query.py:115-120 (table open) and lines 142-145 (vector search), broad except Exception catches wrap all errors into generic RuntimeError messages. This collapses TypeError, PermissionError, ArrowInvalid, and LanceDB-specific errors into a single undifferentiated message, making diagnosis of diverse failure modes very difficult. The original exception class (the most critical diagnostic signal) is obscured. (NEW)

• [has_headings and has_tables Detection Produces False Positives]: At /workspace/src/context_library/adapters/filesystem.py:86, has_headings = "#" in markdown matches any # character anywhere (issue references, C# mentions, code comments in fenced blocks, HTML anchors). At line 90, has_tables = "|" in markdown matches pipes in shell commands, logical OR, code blocks, etc. These structural hints will produce false positives for many files containing no actual markdown headings or tables. Use regex like r"^#{1,6}\s" and r"^\|.+\|$" with re.MULTILINE instead. (NEW)

• [No Test File for validators.py]: The /workspace/src/context_library/storage/validators.py module defines validate_embedding_dimension() and validate_iso8601_timestamp() -- two safety-critical validation functions used throughout the pipeline. There is no tests/storage/test_validators.py file at all. These validators are gatekeepers for data integrity; if they silently accept bad data (wrong dimension, NaN vectors, malformed timestamps), the system would store corrupted embeddings or timestamps. (NEW)

Medium Priority Issues

• [Dead Code: Unused _token_count() Call in Word-Splitting Loop]: At /workspace/src/context_library/domains/notes.py:514, self._token_count(word) is called but its return value is never assigned or used. This is a no-op that wastes computation on every word during oversized chunk splitting. It appears to be a leftover from an earlier version. Should be deleted. (NEW)

• [assert Used for Runtime Validation in Production Code]: At /workspace/src/context_library/storage/document_store.py:204, assert source_version_id is not None guards a critical post-INSERT check. Assertions are disabled with python -O, which would allow None to propagate as a version ID, causing data corruption in subsequent operations. Replace with explicit if check and RuntimeError. (NEW)

• [Empty/Whitespace Query Strings Not Validated]: The retrieve() function at /workspace/src/context_library/retrieval/query.py:57-91 validates top_k and source_filter but not the query string itself. An empty string produces a meaningless embedding vector, returning essentially random nearest neighbors that look like legitimate ranked results to the user. Add if not query or not query.strip(): raise ValueError(...). (NEW)

• [update_last_fetched_at Silently Succeeds on Non-Existent Source]: At /workspace/src/context_library/storage/document_store.py:607-621, the UPDATE affects zero rows if source_id doesn't exist, with no check on cursor.rowcount. The pipeline at pipeline.py:119 calls this when content is unchanged, so monitoring systems that rely on last_fetched_at for staleness detection would be silently deceived. (NEW)

• [chunk_type Is Unvalidated str Despite Closed Set in SQL Schema]: At /workspace/src/context_library/storage/models.py:98, chunk_type: str = "standard" accepts any string, while the SQLite schema at schema.sql:63 enforces CHECK (chunk_type IN ('standard', 'oversized', 'table_part', 'code', 'table')). Invalid values pass Pydantic validation but fail at SQLite insertion with a confusing IntegrityError instead of a clear ValidationError. Should be Literal[...] or a dedicated enum. (NEW)

• [Test Helper Uses Invalid Chunk Hashes]: At /workspace/tests/retrieval/test_query.py:122, chunk_hashes=["hash1", "hash2"] are not valid SHA-256 hex strings. They bypass Pydantic validation because create_source_version() JSON-serializes the list directly. Any test that later calls get_latest_version() would fail with a ValidationError. Use ["a" * 64, "b" * 64] instead. (NEW)

• [CandidateChunk TypedDict Uses total=False When All Fields Are Always Present]: At /workspace/src/context_library/domains/notes.py:15, class CandidateChunk(TypedDict, total=False) makes all fields optional. Every construction site (lines 141-147, 159-167, 192-200, 221-231) includes all five fields. The total=False weakens the type contract unnecessarily and defeats TypedDict's purpose of enforcing field presence. (NEW)

• [retire_chunks Silently Succeeds When No Chunks Match]: At /workspace/src/context_library/storage/document_store.py:298-315, UPDATE operations never check cursor.rowcount. If chunk hashes don't match any existing chunks (stale diff result, store inconsistency), the method silently does nothing while the pipeline reports chunks as "removed" in its statistics. (NEW)

• [Missing Test Coverage for Key Error Paths]: Several important error-handling paths lack test coverage: (1) Differ.diff() ValueError when prev_chunk_hashes=None at differ.py:99-100, (2) retrieve() RuntimeError for _distance=None at query.py:161-166, (3) DocumentStore.write_chunks() lineage validation at document_store.py:241-245, (4) Pipeline embedding validation failure at pipeline.py:158-165. These are contract-enforcement paths that guard against inconsistent API calls and should have explicit tests. (NEW)

• [False "atomically" Claim in Pipeline Docstring]: At /workspace/src/context_library/core/pipeline.py:31, the docstring states "Write to both SQLite and LanceDB atomically." This is factually incorrect -- there is no cross-store transaction or rollback mechanism. Should say "non-transactional" or "LanceDB is rebuildable from SQLite." (NEW)

• [ChunkVector.chunk_hash Uses Plain str Instead of Sha256Hash]: At /workspace/src/context_library/storage/vector_store.py:32, chunk_hash: str is the join key between LanceDB and SQLite but lacks the Sha256Hash validation applied elsewhere. A malformed hash would silently create orphaned vector records. Either use Sha256Hash or add a @field_validator calling _validate_sha256_hex. (NEW)

• [Embedder __init__ Provides No Error Context on Model Load Failure]: At /workspace/src/context_library/core/embedder.py:11-19, SentenceTransformer(model_name) is called with no error handling. If the model fails to load (wrong name, network failure, CUDA error), users get a raw PyTorch/transformers traceback with no mention of "embedder" or "context-library." Wrap with context-providing try-except. (NEW)

Low Priority / Nice-to-Have

• [Inconsistent Timestamp Validation Across Models]: ChunkVector.created_at validates ISO 8601 format (vector_store.py:40-44), but SourceVersion.fetch_timestamp (models.py:136) and StructuralHints.modified_at (models.py:66) accept arbitrary strings with no format validation. A unified approach should be applied. (NEW)

• [DiffResult.prev_hash/curr_hash Are str | None Instead of Sha256Hash | None]: At /workspace/src/context_library/storage/models.py:155-156, these full-document SHA-256 hashes use plain str | None while other hash fields use Sha256Hash. Minor type consistency issue. (NEW)

• [Numeric Field Constraints Missing]: Chunk.chunk_index (models.py:97) allows negative values; LineageRecord.source_version_id (models.py:112) allows zero/negative; StructuralHints.file_size_bytes (models.py:67) allows negative; RetrievalResult.similarity_score (query.py:33) has no [0, 1] range constraint. Add Field(ge=0) / Field(ge=1) / Field(ge=0.0, le=1.0) respectively. (NEW)

• [NormalizedContent Identity Fields Accept Empty Strings]: At /workspace/src/context_library/storage/models.py:79-80, source_id: str and normalizer_version: str can be empty strings. An empty source_id would cascade into SQLite as an empty primary key. Add Field(min_length=1). (NEW)

• [ChunkVector Docstring References Specific Line Numbers in pipeline.py]: At /workspace/src/context_library/storage/vector_store.py:24, "See IngestionPipeline.ingest() pipeline.py:191-199" will become stale with any edit to pipeline.py. Replace with a stable reference. (NEW)

• [Inaccurate Normalization Description in compute_chunk_hash Docstring]: At /workspace/src/context_library/storage/models.py:214-217, the docstring lists normalization steps in the wrong order and does not mention that the code does NOT strip leading/trailing whitespace from the entire text (unlike Differ._normalize). (NEW)

• [Distance-to-Similarity Comment Assumes Normalized Embeddings]: At /workspace/src/context_library/retrieval/query.py:158, the comment says "For normalized embeddings, distance ranges [0, 4]" without noting this is an assumption that breaks if a non-normalized embedding model is used. The assumption should be documented as a requirement. (NEW)

Clean Areas

• SQL injection protection: Thoroughly tested with allowlist validation pattern in retrieve() and document store (test_query.py:313-392, test_document_store.py:1017-1036).
• DiffResult model design: Excellent invariant expression and enforcement via model_post_init, with comprehensive tests (test_models.py:284-402).
• Filesystem adapter error resilience: Well-tested handling of PermissionError, FileNotFoundError, UnicodeDecodeError with graceful skip-and-continue behavior (test_filesystem.py:333-386).
• Pipeline re-ingest scenarios: Comprehensive tests covering unchanged re-ingest, modified content, file deletion, new file addition, and chunk removal with LanceDB deletion verification (test_pipeline.py:170-374).
• NotesDomain chunking tests: Good coverage of heading hierarchy, code block atomicity, table atomicity, context header breadcrumbs, chunk hash determinism, and sequential index assignment (test_notes.py).
• Domain enum design: Perfect encapsulation and enforcement, consistent use across all models (rating 9.75/10).
• Pydantic model architecture: Consistent use of ConfigDict(frozen=True) across all models, correct choice of Pydantic over dataclasses for the validation requirements.
• Differ whitespace normalization tests: Well-tested space/tab collapsing, trailing whitespace stripping, and blank line normalization (test_differ.py:182-237).

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR and compare it against the Phase 1 requirements from the roadmap. Let me gather all the necessary information.Now let me also check what's currently in the local workspace files to cross-reference with the PR diff:Now I have a complete picture. Let me cross-reference the PR's implemented components against the Phase 1 requirements from documentation/roadmap.md.

---

Parent Issue Requirements Verification

Gaps Found

None found

Deviations

None found

Verified

All Phase 1 components specified in documentation/roadmap.md are implemented and present in the PR:

• Data Models (src/context_library/storage/models.py) — Pydantic models with immutable, content-addressed identity: StructuralHints, NormalizedContent, Chunk, SourceVersion, LineageRecord, AdapterConfig, DiffResult, compute_chunk_hash, and Sha256Hash type alias. All frozen (frozen=True). ✓

• Document Store (src/context_library/storage/document_store.py) — SQLite-backed store with WAL mode, adapter registration, source/version tracking, chunk storage, lineage records, and LanceDB sync log. ✓

• Adapter Contract (src/context_library/adapters/base.py) — Abstract BaseAdapter with fetch(), adapter_id, domain, normalizer_version abstract interface, plus a concrete register() method. ✓

• Filesystem Adapter (src/context_library/adapters/filesystem.py) — New file added; recursively discovers .md files and yields NormalizedContent. ✓

• Domain Contract (src/context_library/domains/base.py) — Abstract BaseDomain with chunk() contract. ✓

• Notes Domain (src/context_library/domains/notes.py) — Heading-based hierarchy chunking using mistune AST, with code/table atomicity and context headers as heading breadcrumbs. ✓

• Differ (src/context_library/core/differ.py) — Stateless SHA-256 content hashing diffing added/removed/unchanged chunk hash sets. ✓

• Embedder (src/context_library/core/embedder.py) — sentence-transformers wrapper with embed() batch and embed_query() convenience method. ✓

• Pipeline Orchestrator (src/context_library/core/pipeline.py) — IngestionPipeline coordinating fetch → normalize → diff → chunk → embed → store with error handling. ✓

• Retrieval Query (src/context_library/retrieval/query.py) — retrieve() function providing vector similarity search with optional domain/source metadata filtering and RetrievalResult with full lineage. ✓

• Tests — Comprehensive test files added/updated for all Phase 1 components (adapters, core, domains, retrieval, storage). ✓

• Out-of-Scope Preservation — versioner.py and chunker.py are present in the workspace as reserved stubs, consistent with the roadmap's explicit deferral. ✓

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the business requirements. Let me fetch the PR details first.Let me get the actual code content from the repository to do a thorough analysis.Now I have enough information to compile a thorough verification. Let me check one final item about the DiffResult model.I now have sufficient information to produce the verification report.

---

Business Analyst Output Verification

Gaps Found

• FR-10.1 — Unit test for chunker output is a stub: tests/core/test_chunker.py contains only a docstring ("""Tests for the chunker module.""") with no actual test methods. FR-10.1 explicitly requires unit tests for "chunker output (atomic blocks, heading splits, context headers)." The notes-domain chunker tests in tests/domains/test_notes.py cover some of this ground, but the requirement specifies a dedicated chunker unit test file that is currently empty. This is a gap in the unit test layer for the core chunker module.

• FR-10.2 — Full pipeline ingest → re-ingest → retrieval integration test: FR-10.2 requires an integration test for "the full pipeline ingest → re-ingest → retrieval flow" as a single end-to-end test. test_pipeline.py covers ingest and re-ingest, and test_query.py covers retrieval in isolation, but the requirement calls for a combined integration test that chains all three phases together. The tests exist as separate units rather than a composed integration scenario.

---

Deviations

• FR-2.2 — WAL mode and synchronous mode set in application code, not in schema: The spec requires schema initialization to set WAL mode and synchronous=NORMAL. The schema SQL file only sets PRAGMA foreign_keys=ON and PRAGMA user_version=1. WAL mode and synchronous=NORMAL are set in document_store.py at connection time via Python code. The functional effect is equivalent, but the implementation separates these concerns differently than specified.

• FR-9.2 — Retrieval result exposes RetrievalResult object rather than flat fields: The spec requires results to include "chunk text, chunk hash, source identifier, and relevance score." The implementation returns RetrievalResult Pydantic objects containing the full Chunk and full LineageRecord (not just a source identifier). The to_dict() method does expose chunk_text, source_id, and similarity_score, so the data is accessible, but the primary return type is richer than specified rather than a minimal result structure.

---

Verified

• FR-1.1: documentation/roadmap.md exists, defines Phase 1 scope with all 10 components, maps each to a source file, and lists deferred components (Versioner, Chunker orchestrator).
• FR-2.1: All five tables (adapters, sources, source_versions, chunks, lancedb_sync_log) are present in schema.sql.
• FR-2.2 (partial): PRAGMA user_version=1 set in schema; WAL, foreign keys, and synchronous=NORMAL set at connection initialization (see deviation note).
• FR-2.3: All tables use CREATE TABLE IF NOT EXISTS.
• FR-2.4: Document store supports registering adapters/sources, creating source versions, writing chunks with lineage, and writing sync log entries.
• FR-2.5: Document store supports reading chunks by source, reading version history, and looking up chunks by hash.
• FR-2.6: Pipeline creates the LanceDB chunk_vectors table if absent, using the ChunkVector schema from vector_store.py.
• FR-3.1: All seven Pydantic models exist: NormalizedContent, StructuralHints, Chunk, LineageRecord, SourceVersion, DiffResult, AdapterConfig.
• FR-3.2: chunk_hash is a SHA-256 hash of normalized content via compute_chunk_hash().
• FR-3.3: LineageRecord contains all seven required fields: chunk_hash, source_id, source_version_id, adapter_id, domain, normalizer_version, embedding_model_id.
• FR-3.4: Embedding dimension is exposed as a property on Embedder and used dynamically during table creation.
• FR-4.1: BaseAdapter abstract class defines fetch(), adapter_id, domain, normalizer_version, and register().
• FR-4.2: FilesystemAdapter implements BaseAdapter, accepts a directory path, yields NormalizedContent for all .md files recursively via rglob("*.md").
• FR-4.3: StructuralHints populated with file path, ISO 8601 modification timestamp, and file size in bytes.
• FR-4.4: Adapter registers itself via register() on first use and retrieves its adapter_id on subsequent uses (idempotent upsert).
• FR-5.1: NotesDomain class implements domain-specific markdown chunking.
• FR-5.2: Uses mistune.create_markdown(renderer=None, plugins=["table"]) to parse markdown into an AST.
• FR-5.3: Splits on headings; each heading and its content form a candidate chunk.
• FR-5.4: Code blocks and tables are treated as atomic units (never split mid-block).
• FR-5.5: Context headers are breadcrumb paths from document root, prepended to chunk content.
• FR-5.6: Soft limit (512 tokens) and hard limit (1024 tokens) applied; short adjacent blocks joined; oversized blocks split at paragraph boundaries.
• FR-6.1: Differ.diff() returns DiffResult with changed flag and added/removed/unchanged sets.
• FR-6.2: Full-document SHA-256 hash compared first; unchanged result returned immediately on match.
• FR-6.3: Set operations on chunk hashes classify each as added, removed, or unchanged.
• FR-6.4: Whitespace normalization applied before hashing (collapses spaces/tabs, strips trailing whitespace).
• FR-7.1: Embedder wraps sentence-transformers; embed() accepts list of strings and returns list of vectors.
• FR-7.2: model_id and dimension properties exposed for use in lineage records.
• FR-7.3: Batch processing supported via embed().
• FR-8.1: IngestionPipeline orchestrates full adapter → chunker → differ → embedder → document store → vector store flow.
• FR-8.2: First ingest creates source_version record and writes all chunks with lineage and embeddings.
• FR-8.3: Re-ingest invokes differ, skips unchanged chunks, writes only new/modified chunks, creates new source_version referencing previous.
• FR-8.4: Removed chunks are retired (soft-deleted) in SQLite and deleted from LanceDB index.
• FR-8.5: lancedb_sync_log written for every chunk vector insert and delete via write_sync_log() / delete_sync_log().
• FR-9.1: retrieve() accepts query string, embeds it, performs nearest-neighbor search in LanceDB.
• FR-9.3: top_k parameter controls result count; validated as positive integer.
• FR-9.4: Lineage retrievable from document store by chunk hash via document_store.get_lineage(chunk_hash).
• FR-10.1 (partial): Hash determinism tests (test_differ.py), differ logic tests, model validation tests (test_models.py) all present. Chunker unit tests are a stub (see gap).
• FR-10.2 (partial): SQLite in-memory roundtrip tests (test_document_store.py) and LanceDB write/search roundtrip tests (test_vector_store.py) are present. Full combined pipeline integration test is absent (see gap).
• FR-10.3: tests/fixtures/ contains sample_initial.md, sample_modified.md, and expected_diff.py with known diff outcomes.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the architecture specifications. Let me fetch the PR details.Now let me look at the local workspace to cross-check with the actual file contents:Now I have a thorough picture of the implementation. Let me compile my findings against the architecture specification.

Software Architect Output Verification

Gaps Found

• DiffResult.model_post_init invariant validation is a no-op stub in the spec description: The architecture spec shows model_post_init with only ... as body. The PR implements the full validation logic (set disjointness, changed-flag consistency). This is not a gap — the implementation exceeds the spec stub. No gap here.

• source_version_id in LineageRecord maps to version number, not SQLite rowid: The spec's LineageRecord defines source_version_id: int as the "join key." The create_source_version method returns the SQLite lastrowid, but pipeline.py line 173 passes new_version (the version number, not the rowid) as source_version_id. The spec says this field represents the foreign key for the version record. The SQLite schema chunks.source_version column is compared using the version number in get_chunks_by_source, so the behavior is consistent internally — but the field is named source_version_id (suggesting a rowid) yet stores a version number. This is a naming/semantic inconsistency: create_source_version returns the rowid but the pipeline discards it and uses the version number instead.

• Roadmap documentation Step 10 requirement: The architecture implementation plan lists "Step 10: Roadmap Documentation (documentation/roadmap.md)" as a distinct build step. The roadmap.md file was updated in this PR and correctly describes all Phase 1 components and deferred phases.

Deviations

• register_adapter is not idempotent per spec — it updates on conflict: ADR-003 and Pattern 3 specify that adapter registration should be idempotent ("if adapter_id already exists, existing row is returned"). The implementation uses INSERT … ON CONFLICT DO UPDATE SET, which overwrites domain, adapter_type, normalizer_version, and config on every call. The spec says "returning the existing row" on repeat calls, but the implementation mutates the existing row. This is a minor behavioral deviation: the net effect is the same adapter_id is returned, but existing config is silently overwritten rather than preserved.

• get_chunk_by_hash does not filter out retired chunks: The spec states get_chunks_by_source and get_chunk_by_hash "query chunks (non-retired)". get_chunks_by_source correctly applies retired_at IS NULL, but get_chunk_by_hash (lines 509–533) has no retired_at IS NULL filter, meaning retired chunks can be returned by hash lookup.

• Context header prepended to content field in Chunk: Pattern 5 in the spec says the context header "is prepended before embedding so the vector captures structural context." The hash is computed on content before the context header (correctly). However, the Chunk.content field then stores the combined context_header + "\n\n" + content string (notes.py lines 85–87), meaning content is not just the original chunk text. This diverges from the spec's model where content and context_header are kept separate, with the header only merged for embedding. The Chunk model has a dedicated context_header field for this reason.

• source_version_id in LineageRecord stores version number instead of rowid: create_source_version returns cursor.lastrowid (the autoincrement primary key), but pipeline.py passes new_version (the monotonic version counter) as source_version_id. The SQLite chunks table column is named source_version, queried by version number in get_chunks_by_source. This creates a semantic inconsistency where the field named _id contains a version counter, not a database row ID. The architecture spec uses source_version_id: int in LineageRecord without clarifying which integer, but the returned rowid from create_source_version is unused.

Verified

• All 7 Pydantic models (StructuralHints, NormalizedContent, Chunk, LineageRecord, SourceVersion, DiffResult, AdapterConfig) defined in storage/models.py per ADR-001
• Domain enum preserved and extended per ADR-001
• compute_chunk_hash function with whitespace normalization implemented in models.py per Pattern 1
• EMBEDDING_DIM kept as default constant; validate_embedding_dimension accepts optional expected_dim parameter per ADR-002
• DocumentStore as sqlite3 class with WAL mode, executescript(), user_version=1 check, and context manager per ADR-003
• All required DocumentStore methods implemented: register_adapter, register_source, create_source_version, write_chunks, retire_chunks, write_sync_log, delete_sync_log, get_latest_version, get_version_history, get_chunks_by_source, get_chunk_by_hash, get_lineage, get_adapter, update_last_fetched_at, close
• adapters/filesystem.py created as separate file per ADR-004 with FilesystemAdapter(BaseAdapter), adapter_id = f"filesystem:{path}", domain = Domain.NOTES, graceful error handling for all 4 specified exception types
• domains/notes.py full NotesDomain implementation using mistune AST with soft_limit/hard_limit, breadcrumb context_header, atomic block_code and table handling, hash computed before context header prepend per Pattern 1 and ADR-005
• core/differ.py Differ class with _normalize, _compute_hash, and diff handling first-ingest, unchanged, and changed cases per Pattern 1
• core/embedder.py Embedder with model_id, dimension (via get_sentence_embedding_dimension()), embed (batch), embed_query per ADR-002
• core/pipeline.py IngestionPipeline.ingest() following the SQLite-first write order (Pattern 2), adapter self-registration (Pattern 3), selective re-embedding of only added chunks, LanceDB table created with pa.list_(pa.float32(), embedder.dimension) per ADR-002
• retrieval/query.py retrieve() function with RetrievalResult, top_k validation, source_filter allowlist validation, Euclidean-to-similarity conversion (max(0.0, 1.0 - distance/2.0)), lineage lookup from SQLite
• documentation/roadmap.md updated with Phase 1 complete, Phase 2–4 planned, out-of-scope items documented per Step 10
• All deferred files (core/versioner.py, core/chunker.py, scheduler/, retrieval/reranker.py, retrieval/provenance.py, domains/messages.py, domains/events.py, domains/tasks.py) remain as stubs per the out-of-scope table
• Build order (Steps 1–10) correctly reflected in the dependency graph of the implementation

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

Groups:

• Item:

  • Name: Content Hashing & Normalization
  • Severity: critical
  • Findings: - Inconsistent Whitespace Normalization Between Differ and compute_chunk_hash: Differ._normalize() at src/context_library/core/differ.py:38 applies text.strip() and [ \t]+ regex, while compute_chunk_hash() at src/context_library/storage/models.py:226-233 does not strip leading/trailing whitespace and uses \s+ regex. This divergence means Differ can consider documents "unchanged" while chunk hashes differ, causing the pipeline to skip re-indexing at pipeline.py:117-121 with stale data in the database.

• Context Header Prepended to Chunk content Field: In src/context_library/domains/notes.py:85-87, the Chunk.content field stores context_header + "\n\n" + content, merging the two. The architecture spec keeps content and context_header separate, with the header only merged for embedding. The Chunk model has a dedicated context_header field that becomes redundant when both are baked into content.

• Item:

  • Name: Markdown Parsing & Chunking
  • Severity: critical
  • Findings: - Ordered Lists Silently Converted to Unordered Lists: _render_list() at src/context_library/domains/notes.py:324-337 unconditionally formats every list item with "- " prefix (line 335), ignoring mistune's block["attrs"]["ordered"] attribute. Ordered lists like 1. Install 2. Configure 3. Run lose sequential semantics in the embedded and retrieved content.

• AST Parsing Failure Silently Produces Empty Chunks: At src/context_library/domains/notes.py:69-70, when the mistune parser returns an unexpected type, the code silently falls back to an empty list with no warning or log entry. An entire document vanishes from the system with zero chunks.

• Dead Code: Unused _token_count() Call in Word-Splitting Loop: At src/context_library/domains/notes.py:514, self._token_count(word) is called but its return value is never assigned or used — a no-op wasting computation on every word during oversized chunk splitting.

• Item:

  • Name: Pipeline Orchestration
  • Severity: critical
  • Findings: - Pipeline Has No Per-Source Error Handling: The ingest() method at src/context_library/core/pipeline.py:95-254 iterates over all sources with no try-except around per-source processing. A single file failure aborts the entire ingestion, potentially leaving SQLite and LanceDB in an inconsistent state with no recovery path.

• Non-Atomic Cross-Store Writes Between SQLite and LanceDB: IngestionPipeline.ingest() at src/context_library/core/pipeline.py:182-249 writes to SQLite then LanceDB sequentially with no rollback mechanism. If any step fails after preceding steps succeed, the stores are left inconsistent. The pipeline docstring at line 31 falsely claims these writes are "atomic."

• False "atomically" Claim in Pipeline Docstring: At src/context_library/core/pipeline.py:31, the docstring states "Write to both SQLite and LanceDB atomically" — this is factually incorrect. Should say "non-transactional" or document that LanceDB is rebuildable from SQLite.

• source_version_id Stores Version Number Instead of Rowid: create_source_version returns cursor.lastrowid, but pipeline.py:173 passes new_version (the version counter) as source_version_id in LineageRecord. The rowid is discarded, creating a naming/semantic inconsistency where a field named _id contains a version counter.

• Item:

  • Name: Retrieval Module
  • Severity: high
  • Findings: - Silent Data Loss on Store Inconsistency: At src/context_library/retrieval/query.py:178-187, when a chunk_hash exists in LanceDB but the corresponding chunk or lineage is missing from SQLite, the code silently continues with no logging. Users get fewer results than expected with no indication of store desynchronization.

• Broad except Exception Catches: At src/context_library/retrieval/query.py:115-120 and lines 142-145, broad exception catches collapse TypeError, PermissionError, ArrowInvalid, and LanceDB-specific errors into generic RuntimeError messages, obscuring the original exception class.

• Empty/Whitespace Query Strings Not Validated: retrieve() at src/context_library/retrieval/query.py:57-91 validates top_k and source_filter but not the query string. An empty string produces a meaningless embedding returning random nearest neighbors that appear legitimate.

• Item:

  • Name: Document Store
  • Severity: high
  • Findings: - assert Used for Runtime Validation: At src/context_library/storage/document_store.py:204, assert source_version_id is not None guards a critical post-INSERT check. Assertions are disabled with python -O, which would allow None to propagate causing data corruption. Replace with explicit if check and RuntimeError.

• update_last_fetched_at Silently Succeeds on Non-Existent Source: At src/context_library/storage/document_store.py:607-621, the UPDATE affects zero rows if source_id doesn't exist, with no check on cursor.rowcount. Pipeline calls this at pipeline.py:119 for unchanged content.

• retire_chunks Silently Succeeds When No Chunks Match: At src/context_library/storage/document_store.py:298-315, UPDATE operations never check cursor.rowcount. If chunk hashes don't match, the method silently does nothing while pipeline reports chunks as "removed" in statistics.

• get_chunk_by_hash Does Not Filter Out Retired Chunks: The spec states this method should query non-retired chunks only. get_chunks_by_source correctly applies retired_at IS NULL, but get_chunk_by_hash (lines 509-533) has no such filter, returning retired chunks.

• register_adapter Overwrites on Conflict Instead of Being Idempotent: ADR-003 specifies returning the existing row on repeat calls, but the implementation uses INSERT … ON CONFLICT DO UPDATE SET, silently overwriting domain, adapter_type, normalizer_version, and config.

• Item:

  • Name: Filesystem Adapter
  • Severity: high
  • Findings: - has_headings and has_tables Detection Produces False Positives: At src/context_library/adapters/filesystem.py:86, has_headings = "#" in markdown matches any # character (issue refs, C# mentions, code in fenced blocks). At line 90, has_tables = "|" in markdown matches pipes in shell commands, code blocks, etc. Use regex like r"^#{1,6}\s" and r"^\|.+\|$" with re.MULTILINE.

• Item:

  • Name: Type Safety & Validation
  • Severity: medium
  • Findings: - chunk_type Is Unvalidated str Despite Closed Set in Schema: At src/context_library/storage/models.py:98, chunk_type: str = "standard" accepts any string, while schema.sql:63 enforces CHECK (chunk_type IN ('standard', 'oversized', 'table_part', 'code', 'table')). Invalid values pass Pydantic but fail at SQLite with confusing IntegrityError. Should be Literal[...] or enum.

• CandidateChunk TypedDict Uses total=False When All Fields Are Always Present: At src/context_library/domains/notes.py:15, total=False makes all fields optional, but every construction site includes all five fields. This weakens the type contract unnecessarily.

• ChunkVector.chunk_hash Uses Plain str Instead of Sha256Hash: At src/context_library/storage/vector_store.py:32, the join key between LanceDB and SQLite lacks the Sha256Hash validation applied elsewhere. A malformed hash would silently create orphaned vector records.

• Item:

  • Name: Test Coverage
  • Severity: medium
  • Findings: - No Test File for validators.py: src/context_library/storage/validators.py defines validate_embedding_dimension() and validate_iso8601_timestamp() — safety-critical validation functions — with no corresponding tests/storage/test_validators.py.

• Test Helper Uses Invalid Chunk Hashes: At tests/retrieval/test_query.py:122, chunk_hashes=["hash1", "hash2"] are not valid SHA-256 hex strings. Use ["a" * 64, "b" * 64] instead.

• Missing Test Coverage for Key Error Paths: No tests for: Differ.diff() ValueError when prev_chunk_hashes=None (differ.py:99-100), retrieve() RuntimeError for _distance=None (query.py:161-166), DocumentStore.write_chunks() lineage validation (document_store.py:241-245), pipeline embedding validation failure (pipeline.py:158-165).

• test_chunker.py Is a Stub: tests/core/test_chunker.py contains only a docstring with no actual test methods, per FR-10.1 requirement for chunker unit tests.

• No Combined Integration Test for Ingest → Re-ingest → Retrieval: FR-10.2 requires a single end-to-end integration test chaining all three phases. Tests exist as separate units but not as a composed scenario.
  Filtered Out:

• Embedder __init__ error context on model load failure — nice-to-have error wrapping, not a bug or spec requirement

• Inconsistent timestamp validation across models — style/consistency preference, not a functional gap

• DiffResult.prev_hash/curr_hash plain str instead of Sha256Hash — minor type consistency, no functional impact

• Numeric field constraints missing (negative chunk_index, etc.) — defensive hardening, not required by spec

• NormalizedContent identity fields accepting empty strings — edge case hardening, not spec-required

• ChunkVector docstring references specific line numbers — comment maintenance concern, not a code issue

• Inaccurate normalization description in compute_chunk_hash docstring — merged into the critical normalization inconsistency finding

• Distance-to-similarity comment assumes normalized embeddings — documentation suggestion, not a code defect

• WAL mode set in application code vs schema — verified as functionally equivalent deviation, not a gap

• RetrievalResult richer than spec's flat fields — implementation exceeds spec, not a gap

• DiffResult.model_post_init exceeds spec stub — implementation exceeds spec, not a gap

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

I'll start by checking out the PR branch and then running the comprehensive review.We're already on the correct branch. Now let me run the comprehensive review skill.

I'll run a comprehensive PR review with all specialized agents sequentially. Let me first understand the scope of changes.Good, I have the full diff context. Now let me check for a CLAUDE.md file for project guidelines.Now I'll run each specialized agent sequentially. Starting with the code-reviewer.Now running the test analyzer.Now running the silent-failure-hunter.Now running the comment analyzer.Now running the type-design-analyzer.Now let me cross-reference findings against the prior cycle issues to determine what's NEW vs already addressed. Let me check the closed issues to understand what was already fixed.Let me verify whether the core issues from my review still exist in the current code by checking specific locations.Now let me check the schema to see if the sync log table was updated to be truly append-only.Good. I've confirmed the key issues still exist in the current code. Now let me compile the final aggregated review, carefully cross-referencing with the 15 prior cycle issues that were closed.

The prior cycles covered:

• Cycle 1: Schema/Data Integrity (SQLite), Retrieval Module, Embedding Validation, NotesDomain Chunker, Filesystem Adapter Error Handling, LanceDB Sync & Schema Usage, Data Model Invariants
• Cycle 2: Content Hashing & Normalization, Markdown Parsing & Chunking, Pipeline Orchestration, Retrieval Module, Document Store, Filesystem Adapter, Type Safety & Validation, Test Coverage

Many of these were reportedly fixed (commits reference issues #54-#70). My findings focus on issues that either:

1. Were NOT addressed by the prior fixes
2. Represent new perspectives/deeper analysis
3. Are genuinely new issues not covered in prior cycles## PR Review Findings

Critical Issues

• [Unchanged chunks become invisible after version updates]: When a source is updated and some chunks are unchanged, those unchanged chunks become inaccessible through get_chunks_by_source(). The pipeline only writes newly added chunks to the chunks table with the new source_version (/workspace/src/context_library/core/pipeline.py:166-168). Unchanged chunks retain their original source_version. However, get_chunks_by_source() filters by WHERE source_id = ? AND source_version = ? using the current version (/workspace/src/context_library/storage/document_store.py:489-497), so unchanged chunks with old version numbers are never returned. This is a data integrity bug in the core retrieval path: every document update that preserves some existing chunks will cause those chunks to disappear from query results. (NEW)

• [Pipeline returns success dict with no error information after failures]: The ingest() method always returns a dict and never raises, even if every single source failed. The broad except Exception at /workspace/src/context_library/core/pipeline.py:279-283 catches ~170 lines of mixed logic including data integrity errors (RuntimeError from retire_chunks), validation errors (ValueError from embedding validation at line 187), and database corruption (sqlite3.OperationalError). The return dict at lines 285-290 has no errors count, no sources_failed count, and no error messages. A caller receiving {"sources_processed": 0} cannot distinguish between "no sources to process" and "all sources failed catastrophically." The explicitly-raised ValueError at line 187-189 is immediately swallowed by the outer handler, negating its purpose. (NEW)

• [Sync log docstrings claim "append-only" but INSERT OR REPLACE replaces rows]: The write_sync_log() and delete_sync_log() docstrings at /workspace/src/context_library/storage/document_store.py:332-342 and 355-366 describe the sync log as "an append-only audit trail" where "entries are never cleared on successful writes" and "multiple operations on the same chunk create timestamped records." However, lancedb_sync_log has chunk_hash TEXT PRIMARY KEY (/workspace/src/context_library/storage/schema.sql:96) and uses INSERT OR REPLACE, which replaces existing rows. A chunk first inserted then deleted will only have the delete record. The get_chunks_pending_sync() recovery mechanism at line 655 cannot work correctly for rebuild scenarios. This was partially reported in cycle 1 ([PR Feedback] LanceDB Sync & Schema Usage #61 LanceDB Sync) and cycle 2 ([PR Feedback] Document Store #67 Document Store), but the docstrings remain factually incorrect and the schema still uses PRIMARY KEY preventing append-only behavior. (REGRESSION)

High Priority Issues

• [Pipeline SQLite/LanceDB partial write inconsistency not surfaced to caller]: The pipeline writes to SQLite first (lines 206-217) then to LanceDB (lines 221-272) at /workspace/src/context_library/core/pipeline.py. If SQLite writes succeed but LanceDB fails, the except Exception catches it and continues. SQLite now contains chunks not in LanceDB. No specific warning identifies the inconsistency, the return dict has no indication, and the generic error log at line 281 does not distinguish this critical state from a benign file-read error. Vector search results will be missing chunks that exist in SQLite. (NEW)

• [Retrieval silently drops results and misdiagnoses retired chunks as store inconsistency]: At /workspace/src/context_library/retrieval/query.py:192-210, when get_chunk_by_hash returns None (because it filters retired_at IS NULL), a "Store inconsistency" warning is logged and the result is dropped. However, a chunk retired in SQLite but not yet deleted from LanceDB is normal pipeline behavior, not an inconsistency. The caller of retrieve() receives fewer results than requested with no programmatic indication of why. If top_k=10 and 7 results are desynchronized, the user gets 3 results silently. (NEW)

• [NotesDomain silently returns zero chunks on parser failure]: At /workspace/src/context_library/domains/notes.py:82-88, when the markdown parser returns an unexpected type, the chunker logs a WARNING and falls back to an empty list. The document produces zero chunks and effectively disappears from the system. The pipeline reports success, but the file is nowhere in the index. The caller cannot distinguish "no chunkable content" from "parser broke." (NEW)

• [Cross-source content-addressed dedup silently drops chunks for second source]: The chunks table uses chunk_hash as PRIMARY KEY (/workspace/src/context_library/storage/schema.sql:50). If two different sources produce a chunk with identical content, the second source's chunk insertion silently fails and is suppressed at /workspace/src/context_library/storage/document_store.py:293-301. Queries using get_chunks_by_source(second_source_id) will never find that chunk. No warning is logged. (NEW)

Medium Priority Issues

• [ChunkVector docstring references wrong line numbers in pipeline.py]: At /workspace/src/context_library/storage/vector_store.py:27, the docstring states "See IngestionPipeline.ingest() pipeline.py:191-199" but those lines contain LineageRecord construction, not ChunkVector instantiation (which is at ~lines 226-234). Line-number references in docstrings are inherently fragile and already stale. (NEW)

• [get_chunks_pending_sync and get_chunks_pending_deletion have no dedicated tests]: /workspace/src/context_library/storage/document_store.py:655-716 contains the recovery mechanism for SQLite/LanceDB inconsistency, but no test directly calls these methods to verify their return structure and content. A query bug in these methods would silently return wrong results, breaking the recovery path. (NEW)

• [RetrievalResult missing score bounds and chunk-lineage hash consistency check]: At /workspace/src/context_library/retrieval/query.py:37, similarity_score: float has no bounds validation (ge=0.0, le=1.0). Also, no model_post_init verifies chunk.chunk_hash == lineage.chunk_hash, allowing mismatched pairings. (NEW)

• [Timestamp validation inconsistency across type system]: ChunkVector.created_at (/workspace/src/context_library/storage/vector_store.py:41) validates ISO 8601 via @field_validator, but SourceVersion.fetch_timestamp (/workspace/src/context_library/storage/models.py:152) and StructuralHints.modified_at (/workspace/src/context_library/storage/models.py:80) accept arbitrary strings. The validate_iso8601_timestamp function exists but is used in only one location. (NEW)

• [DocumentStore write methods accept raw str where Sha256Hash is appropriate]: create_source_version() at /workspace/src/context_library/storage/document_store.py:155 accepts chunk_hashes: list[str], as do retire_chunks(), write_sync_log(), and delete_sync_log(). Malformed hashes could be written to the database and only caught on read. (NEW)

• [has_lists structural hint only detects unordered lists]: At /workspace/src/context_library/adapters/filesystem.py:89-91, the regex r"^[\-\*\+]\s" only matches unordered list markers but does not detect ordered lists (e.g., 1., 2.). Documents with only ordered lists will have has_lists=False, potentially causing suboptimal chunking decisions. (NEW)

• [Embedder methods do not validate empty input]: At /workspace/src/context_library/core/embedder.py:45-55, embed([]) passes an empty list to SentenceTransformer.encode() whose behavior with empty input is model-dependent. embed_query("") will embed an empty string producing a meaningless embedding vector. Neither method validates input. (NEW)

Low Priority / Nice-to-Have

• [Mutable containers in frozen Pydantic models]: Several frozen models use list and dict fields that can be mutated after construction: StructuralHints.natural_boundaries (list[int]), SourceVersion.chunk_hashes (list[Sha256Hash]), Chunk.domain_metadata (dict), ChunkVector.vector (list[float]). Using tuple for lists and/or documenting the contract would strengthen immutability guarantees. Locations: /workspace/src/context_library/storage/models.py:78,116,149, /workspace/src/context_library/storage/vector_store.py:37. (NEW)

• [DiffResult.prev_hash/curr_hash typed as str instead of Sha256Hash]: At /workspace/src/context_library/storage/models.py:173-174, these document-level hashes are documented as SHA-256 but typed as str | None, bypassing the Sha256Hash validator. (NEW)

• [Chunk.chunk_index has no non-negativity constraint]: At /workspace/src/context_library/storage/models.py:114, chunk_index: int would accept negative values. Adding ge=0 via Pydantic Field would prevent invalid indices. (NEW)

• [LanceDB list_tables() called repeatedly inside the source loop]: At /workspace/src/context_library/core/pipeline.py:248,267, db.list_tables() is called up to twice per changed source. For N sources, this results in up to 2N I/O calls. Caching the table existence check would reduce overhead. (NEW)

• [BaseDomain sequential chunk_index invariant not enforced]: At /workspace/src/context_library/domains/base.py:23, the docstring states chunks must have "sequential chunk_index values" but the base class does not enforce this. A future domain implementation could violate this invariant. (NEW)

Clean Areas

• Enum types (Domain, ChunkType): Excellently designed with str, Enum inheritance, aligned with SQLite CHECK constraints, and fully enforced by Python's enum mechanism. Overall score 9.5/10.
• DiffResult type design: Best-designed type in the PR with explicitly documented invariants, runtime enforcement via model_post_init, and clear error messages. Score 9.3/10.
• SQL injection prevention: Parameterized queries used throughout SQLite operations. LanceDB source_filter uses allowlist validation in retrieve(). Chunk hashes validated as strict hex strings before interpolation.
• Sha256Hash validated type: Precise regex with excellent error messages, used consistently across all Pydantic models carrying chunk hashes.
• Filesystem adapter error handling: Specific exception types (UnicodeDecodeError, PermissionError, FileNotFoundError, OSError) rather than bare except Exception. All four error branches covered by tests.
• Model validation tests: Thorough coverage of Pydantic validation, immutability, hash validation, whitespace normalization, and DiffResult invariants.
• End-to-end integration test: The TestEndToEndIngestReIngestRetrieval test exercises the complete lifecycle and checks consistency between SQLite and LanceDB.
• Retrieval input validation: retrieve() has thorough validation of empty query, invalid top_k, and source_filter allowlist, plus explicit handling of unexpected data types from search results.

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR and the roadmap to verify the implementation against Phase 1 requirements.Now I have a comprehensive view of what was implemented. Let me compare against the roadmap's Phase 1 requirements:I now have everything needed to complete the verification. The roadmap clearly defines exactly 10 Phase 1 components and explicitly defers 2 items (versioner.py and chunker.py) to future phases.

Parent Issue Requirements Verification

Gaps Found

None found

Deviations

None found

Verified

All 10 Phase 1 components specified in documentation/roadmap.md are implemented:

• Data Models (src/context_library/storage/models.py) — Pydantic models with immutable, content-addressed identity for sources, chunks, embeddings, and lineage tracking are present and implemented.
• Document Store (src/context_library/storage/document_store.py) — SQLite-backed store with adapter registration, source tracking, version history, chunk storage, and vector store synchronization is implemented.
• Adapter Contract (src/context_library/adapters/base.py) — Abstract base class defining the adapter interface is implemented.
• Filesystem Adapter (src/context_library/adapters/filesystem.py) — Concrete adapter for discovering and normalizing local markdown files is implemented.
• Domain Contract (src/context_library/domains/base.py) — Abstract base class for domain-specific chunking is implemented.
• Notes Domain (src/context_library/domains/notes.py) — Concrete domain for freeform markdown notes with heading-based hierarchy chunking and code/table preservation is implemented.
• Differ (src/context_library/core/differ.py) — Stateless SHA-256 change detector identifying new, modified, and deleted sources is implemented.
• Embedder (src/context_library/core/embedder.py) — Vector embedding service wrapping sentence-transformers for batch embedding is implemented.
• Pipeline Orchestrator (src/context_library/core/pipeline.py) — Coordinates the full fetch → normalize → diff → chunk → embed → store workflow with error handling and resumability is implemented.
• Retrieval Query (src/context_library/retrieval/query.py) — Semantic search interface with vector similarity queries and optional metadata filtering is implemented.

The two "Out of Scope for Phase 1" items (versioner.py, chunker.py) exist as reserved stubs, which is explicitly consistent with the roadmap's documented deferral. Future-phase components (scheduler, reranker, provenance, additional domains) also appear as stubs, which is not a gap since they belong to Phases 2–4.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the business requirements. Let me fetch the PR details first.Now I have comprehensive details about the PR implementation. Let me analyze it against each functional requirement:Now I have enough detail to do a thorough analysis. Let me compile my findings:Now I have all the information needed to compile the verification report:

---

Business Analyst Output Verification

Gaps Found

• FR-2.2 WAL mode and synchronous=NORMAL not in schema.sql: The requirement specifies the schema must set WAL mode and synchronous=NORMAL. The schema.sql file only contains PRAGMA foreign_keys=ON and PRAGMA user_version=1. WAL mode and synchronous=NORMAL are set in DocumentStore.__init__() at runtime rather than in the schema file. While the behavior is correct at runtime, it means schema initialization via the SQL file alone does not enforce these settings — only instantiating DocumentStore does.

• FR-2.5 get_chunks_by_source returns only latest-version active chunks: The requirement says "reading chunks by source" must be supported. get_chunks_by_source(source_id, version) accepts a version parameter but it is unclear from the implementation whether callers must explicitly pass the latest version or whether it defaults to the current version. The agent's analysis indicates the method filters retired_at IS NULL and orders by chunk_index, but does not indicate it automatically resolves the "latest version" — callers must supply the version. This is a partial gap if the intent is that the method should default to the active/latest version without the caller needing to pre-fetch it.

• FR-4.4 Adapter records itself on first use / retrieves adapter_id on subsequent uses: The BaseAdapter.register() calls document_store.register_adapter(), and DocumentStore.register_adapter() is idempotent (checks existence first). However, the adapter_id is always derived from the adapter's own deterministic property ("filesystem:{path}"), not retrieved back from the database. The requirement states the adapter must "retrieve its registered adapter_id on subsequent uses" — meaning the stored record should be the source of truth for the ID on subsequent calls. The implementation computes the ID deterministically from configuration and uses that same value on every call, never performing a lookup from the database. This is a deviation in mechanism (deterministic local computation vs. DB retrieval), though functionally the result is equivalent since the same ID is always produced.

• FR-5.6 Chunker joins short adjacent blocks only with same context_header: The requirement states short adjacent blocks should be joined "if the combined size remains within the hard limit." The implementation restricts joining to candidates with the same context_header, which is more restrictive than specified (the requirement does not say same-header only). This means adjacent short sections under different headings will never be joined. This may be a reasonable design choice, but it deviates from what FR-5.6 explicitly allows.

• FR-8.4 Removed chunks marked inactive in document store: The implementation uses retire_chunks() (setting retired_at timestamp) rather than an "inactive" field. While functionally equivalent as a soft-delete mechanism, the schema uses a retired_at column, not an is_active/active boolean. This is a naming/schema deviation from "marked inactive."

• FR-10.3 Test fixtures include two sample markdown documents with known expected diff outcomes: The agent's analysis confirms tests/fixtures/sample_initial.md and tests/fixtures/sample_modified.md exist, and tests/fixtures/expected_diff.py provides a helper. This requirement appears met.

Deviations

• FR-2.2 Schema PRAGMA placement: PRAGMA journal_mode=WAL and PRAGMA synchronous=NORMAL are applied in Python code (DocumentStore.__init__) rather than in the schema.sql file. The schema file itself only sets PRAGMA foreign_keys=ON and PRAGMA user_version=1. The system behavior is correct at runtime, but the schema file does not encode the full initialization specification.

• FR-2.6 LanceDB table creation location: The requirement states "The LanceDB table for chunk vectors must be created if it does not exist, using the schema defined in storage/vector_store.py." The table creation actually occurs in pipeline.py using an inline pa.schema() with the embedder's runtime dimension, not in vector_store.py. The ChunkVector class in vector_store.py exists as a schema definition type but does not drive table creation.

• FR-3.4 Embedding dimension configurable via configured embedding model: The implementation correctly introspects dimension from the loaded model via self._model.get_sentence_embedding_dimension(). However, validators.py retains a hardcoded default EMBEDDING_DIM = 384 that is the fallback default for validate_embedding_dimension(). This is a minor deviation — the pipeline always passes the runtime dimension explicitly, so the hardcoded default is only a safety fallback.

• FR-9.2 Retrieval results include source identifier: The retrieve() function returns RetrievalResult objects containing chunk, lineage, and similarity_score. The source_id is accessible via result.lineage.source_id, not as a top-level field on RetrievalResult. The to_dict() method does surface source_id at the top level, but the model itself nests it. This is a minor structural deviation.

Verified

• FR-1.1: documentation/roadmap.md exists defining Phase 1 scope, components included, and deferred phases (Phase 2–4).
• FR-2.1: All five tables (adapters, sources, source_versions, chunks, lancedb_sync_log) are defined in schema.sql.
• FR-2.2 (partial): PRAGMA user_version=1 and PRAGMA foreign_keys=ON are set in schema; WAL and synchronous are set at runtime (see deviation above).
• FR-2.3: Tables use IF NOT EXISTS for idempotent initialization.
• FR-2.4: Full write path implemented: register_adapter, register_source, create_source_version, write_chunks with lineage, write_sync_log.
• FR-2.5: get_chunks_by_source, get_version_history, get_chunk_by_hash all implemented.
• FR-3.1: All seven Pydantic models exist: NormalizedContent, StructuralHints, Chunk, LineageRecord, SourceVersion, DiffResult, AdapterConfig.
• FR-3.2: compute_chunk_hash() computes SHA-256 of normalized content deterministically.
• FR-3.3: LineageRecord contains all required fields: chunk_hash, source_id, source_version_id, adapter_id, domain, normalizer_version, embedding_model_id.
• FR-3.4: Embedding dimension is introspected from the loaded model at runtime (see minor deviation on fallback default).
• FR-4.1: BaseAdapter abstract class defines the fetch() contract yielding NormalizedContent.
• FR-4.2: FilesystemAdapter implements BaseAdapter, recursively yields NormalizedContent for all .md files.
• FR-4.3: StructuralHints populated with file_path, modified_at (ISO 8601), and file_size_bytes.
• FR-4.4: Adapter registers via document_store.register_adapter() with idempotency (see deviation on retrieval mechanism).
• FR-5.1: NotesDomain class implements domain-specific markdown chunking.
• FR-5.2: mistune used in AST mode (renderer=None) to parse markdown into a block tree.
• FR-5.3: Chunker splits on headings, treating each heading and content as a candidate chunk.
• FR-5.4: Code blocks (block_code) and tables are treated as atomic chunks (is_atomic=True), never split mid-block.
• FR-5.5: Context headers are breadcrumb paths ("# H1 > ## H2 > ### H3") included in each chunk.
• FR-5.6: Soft/hard token limits respected; adjacent blocks joined when combined size ≤ hard limit (with same-header restriction — see gap).
• FR-6.1: Differ.diff() produces DiffResult with changed flag and added/removed/unchanged classification.
• FR-6.2: Full-document SHA-256 hash comparison short-circuits to unchanged result when hashes match.
• FR-6.3: Chunk hash set operations (added = curr - prev, removed = prev - curr, unchanged = curr ∩ prev) implemented correctly.
• FR-6.4: Whitespace normalization (line endings, horizontal whitespace collapse, per-line rstrip) applied before hashing; same normalization used in both compute_chunk_hash and Differ.
• FR-7.1: Embedder wraps sentence-transformers and exposes embed(texts) returning list of vectors.
• FR-7.2: model_id and dimension properties expose model identifier and output dimension.
• FR-7.3: embed() accepts a list of strings (batch processing).
• FR-8.1: IngestionPipeline.ingest() coordinates the full adapter → chunker → differ → embedder → document store → vector store flow.
• FR-8.2: First ingest creates source_version record and writes all chunks with lineage and embeddings.
• FR-8.3: Re-ingest invokes differ, skips unchanged chunks, writes only added chunks, creates new version record.
• FR-8.4: Removed chunks retired in SQLite (retire_chunks) and deleted from LanceDB index (see naming deviation).
• FR-8.5: lancedb_sync_log written for every insert and delete vector operation.
• FR-9.1: retrieve() embeds query, performs nearest-neighbor search against LanceDB.
• FR-9.2: Results include chunk text, chunk hash, source identifier (via lineage), and relevance score.
• FR-9.3: top_k parameter controls number of results returned.
• FR-9.4: get_lineage(chunk_hash) retrieves full lineage record for any chunk hash.
• FR-10.1: Unit tests cover hash computation, differ logic, chunker output, and model validation.
• FR-10.2: Integration tests cover document store write/read roundtrip, vector store write/search, and full pipeline ingest → re-ingest → retrieval flow.
• FR-10.3: Two sample markdown fixtures (sample_initial.md, sample_modified.md) with expected_diff.py helper exist.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the architecture specifications. Let me fetch the PR details first.Now I have enough information to compile a thorough verification. Let me check a few more specific details about the architecture spec requirements.I now have sufficient information to produce a thorough verification report.

---

Software Architect Output Verification

Gaps Found

• storage/validators.py — expected_dim parameter not confirmed added: The architecture spec (ADR-002 + "Component Reuse / Minor Extension") explicitly required adding an optional expected_dim parameter to validate_embedding_dimension() so runtime validation checks against the embedder's reported dimension rather than the hardcoded constant. The fetched file summary confirms EMBEDDING_DIM = 384 remains as the default and the function accepts an optional parameter — this appears implemented, but the spec's requirement that runtime validation in the pipeline uses the embedder's actual dimension (not the constant) needs to be traced through to pipeline.py. Based on available summaries this trace is unclear, so this is a partial-verification gap.

• core/chunker.py — disposition not explicitly documented: ADR-005 specifies that core/chunker.py is "not implemented in Phase 1" and is reserved for future use. The PR should leave this stub untouched or empty. The file list does not show core/chunker.py as a changed file, but its final state (whether it remains a stub or was inadvertently modified) is not confirmed from the available diff summaries.

Deviations

• Context header prepend at embedding time — not confirmed in pipeline: Pattern 5 states context headers must be prepended before embedding so the vector captures structural context ("The header is prepended before embedding so the vector captures structural context"). The NotesDomain implementation correctly stores context_header as a separate field and does not bake it into the content or hash. The pipeline summary states "Prepends context headers for semantic enrichment" before embedding — this matches the spec. However, the NotesDomain investigation shows headers are stored as metadata and "not prepended" to text in the domain layer itself, which is correct per Pattern 5 (prepending is the pipeline's responsibility, not the domain's). This is consistent with the spec, but it's worth noting the architecture is split across two files as intended.

Verified

• All 7 Pydantic models in storage/models.py: StructuralHints, NormalizedContent, Chunk, LineageRecord, SourceVersion, DiffResult, AdapterConfig — all present alongside the existing Domain enum (ADR-001 fulfilled)
• compute_chunk_hash() utility: Implements whitespace normalization (collapse runs of whitespace, strip trailing whitespace, normalize line endings) before SHA-256 hashing — Pattern 1 fulfilled
• EMBEDDING_DIM = 384 default preserved in validators.py with optional expected_dim parameter added — ADR-002 fulfilled
• DocumentStore as class wrapping sqlite3: Accepts a DB path, manages a sqlite3.Connection, runs executescript() against schema.sql on construction, applies WAL mode and pragmas — ADR-003 fulfilled
• adapters/filesystem.py as a separate new file: FilesystemAdapter(BaseAdapter) with deterministic adapter_id = "filesystem:{absolute_path}" — ADR-004 fulfilled
• domains/notes.py owns chunking logic using mistune: Full AST-based markdown chunker; core/chunker.py not implemented — ADR-005 fulfilled
• BaseAdapter abstract contract in adapters/base.py: Abstract properties adapter_id, domain, normalizer_version; abstract method fetch(); concrete register() method — spec fulfilled
• BaseDomain abstract contract in domains/base.py: Single abstract chunk(content) method — spec fulfilled
• Differ — hash-based with short-circuit: Full-document SHA-256 comparison first; set operations for chunk-level diff; consistent normalization via compute_chunk_hash() — Pattern 1 and scalability design fulfilled
• Embedder: Wraps sentence-transformers, exposes dimension property, supports batch embed() and single embed_query(), defaults to all-MiniLM-L6-v2 — ADR-002 and spec fulfilled
• IngestionPipeline: Implements fetch→normalize→chunk→embed→store; SQLite-first then LanceDB write order (Pattern 2); adapter self-registration (Pattern 3); per-source error isolation; sync log writes for LanceDB recovery — all key patterns fulfilled
• retrieval/query.py: retrieve() function with RetrievalResult model; semantic search via LanceDB + SQLite lineage enrichment; optional domain/source filtering; input validation against SQL injection; Euclidean-to-similarity conversion — spec fulfilled
• lancedb_sync_log table operations: write_sync_log(), delete_sync_log(), get_chunks_pending_sync(), get_chunks_pending_deletion() — sync log tracking fully implemented
• Immutable version records / soft-delete: Source versions are append-only; chunks retired via retired_at timestamping rather than deletion — Pattern 4 fulfilled
• Adapter self-registration idempotency: register_adapter() returns existing row on duplicate adapter_id — Pattern 3 fulfilled
• documentation/roadmap.md: Updated and committed as Step 10 of the implementation plan — spec fulfilled
• Out-of-scope files not implemented: domains/messages.py, events.py, tasks.py, retrieval/reranker.py, retrieval/provenance.py, scheduler/poller.py, scheduler/watcher.py, core/versioner.py — none appear in the PR's changed files
• Build order / layered dependencies respected: models → document_store → adapters → domains → differ → embedder → pipeline → retrieval — confirmed by the implementation structure

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

Groups:

• Item:

  • Name: Chunk Versioning & Retrieval
  • Severity: critical
  • Findings: - Unchanged chunks become invisible after version updates: When a source is updated and some chunks are unchanged, those chunks become inaccessible through get_chunks_by_source(). The pipeline only writes newly added chunks with the new source_version (src/context_library/core/pipeline.py:166-168). Unchanged chunks retain their original source_version. However, get_chunks_by_source() filters by WHERE source_id = ? AND source_version = ? using the current version (src/context_library/storage/document_store.py:489-497), so unchanged chunks with old version numbers are never returned. Every document update that preserves some existing chunks will cause those chunks to disappear from query results.

• Cross-source content-addressed dedup silently drops chunks for second source: The chunks table uses chunk_hash as PRIMARY KEY (src/context_library/storage/schema.sql:50). If two different sources produce a chunk with identical content, the second source's chunk insertion silently fails and is suppressed at src/context_library/storage/document_store.py:293-301. Queries using get_chunks_by_source(second_source_id) will never find that chunk. No warning is logged.

• Item:

  • Name: Pipeline Error Handling & Caller Feedback
  • Severity: critical
  • Findings: - Pipeline returns success dict with no error information after failures: The ingest() method always returns a dict and never raises, even if every single source failed. The broad except Exception at src/context_library/core/pipeline.py:279-283 catches ~170 lines of mixed logic including data integrity errors, validation errors, and database corruption. The return dict at lines 285-290 has no errors count, no sources_failed count, and no error messages. A caller receiving {"sources_processed": 0} cannot distinguish between "no sources to process" and "all sources failed catastrophically." The explicitly-raised ValueError at line 187-189 is immediately swallowed by the outer handler, negating its purpose.

• Pipeline SQLite/LanceDB partial write inconsistency not surfaced to caller: The pipeline writes to SQLite first (lines 206-217) then to LanceDB (lines 221-272) at src/context_library/core/pipeline.py. If SQLite writes succeed but LanceDB fails, the except Exception catches it and continues. SQLite now contains chunks not in LanceDB. No specific warning identifies the inconsistency, the return dict has no indication, and the generic error log at line 281 does not distinguish this critical state from a benign file-read error.

• Item:

  • Name: Sync Log Schema & Docstrings
  • Severity: critical
  • Findings: - Sync log docstrings claim "append-only" but INSERT OR REPLACE replaces rows: The write_sync_log() and delete_sync_log() docstrings at src/context_library/storage/document_store.py:332-342 and 355-366 describe the sync log as "an append-only audit trail" where "entries are never cleared on successful writes" and "multiple operations on the same chunk create timestamped records." However, lancedb_sync_log has chunk_hash TEXT PRIMARY KEY (src/context_library/storage/schema.sql:96) and uses INSERT OR REPLACE, which replaces existing rows. A chunk first inserted then deleted will only have the delete record. The get_chunks_pending_sync() recovery mechanism at line 655 cannot work correctly for rebuild scenarios. This was partially reported in cycle 1 ([PR Feedback] LanceDB Sync & Schema Usage #61) and cycle 2 ([PR Feedback] Document Store #67), but the docstrings remain factually incorrect and the schema still prevents append-only behavior.

• Item:

  • Name: Retrieval Module
  • Severity: high
  • Findings: - Retrieval silently drops results and misdiagnoses retired chunks as store inconsistency: At src/context_library/retrieval/query.py:192-210, when get_chunk_by_hash returns None (because it filters retired_at IS NULL), a "Store inconsistency" warning is logged and the result is dropped. However, a chunk retired in SQLite but not yet deleted from LanceDB is normal pipeline behavior, not an inconsistency. The caller of retrieve() receives fewer results than requested with no programmatic indication of why.

• Item:

  • Name: Notes Domain Chunker
  • Severity: high
  • Findings: - NotesDomain silently returns zero chunks on parser failure: At src/context_library/domains/notes.py:82-88, when the markdown parser returns an unexpected type, the chunker logs a WARNING and falls back to an empty list. The document produces zero chunks and effectively disappears from the system. The pipeline reports success, but the file is nowhere in the index.

• Item:

  • Name: Type System & Validation Consistency
  • Severity: medium
  • Findings: - Timestamp validation inconsistency across type system: ChunkVector.created_at (src/context_library/storage/vector_store.py:41) validates ISO 8601 via @field_validator, but SourceVersion.fetch_timestamp (src/context_library/storage/models.py:152) and StructuralHints.modified_at (src/context_library/storage/models.py:80) accept arbitrary strings. The validate_iso8601_timestamp function exists but is used in only one location.

• DocumentStore write methods accept raw str where Sha256Hash is appropriate: create_source_version() at src/context_library/storage/document_store.py:155 accepts chunk_hashes: list[str], as do retire_chunks(), write_sync_log(), and delete_sync_log(). Malformed hashes could be written to the database and only caught on read.

• RetrievalResult missing score bounds and chunk-lineage hash consistency check: At src/context_library/retrieval/query.py:37, similarity_score: float has no bounds validation. No model_post_init verifies chunk.chunk_hash == lineage.chunk_hash, allowing mismatched pairings.

• DiffResult.prev_hash/curr_hash typed as str instead of Sha256Hash: At src/context_library/storage/models.py:173-174, these document-level hashes are documented as SHA-256 but typed as str | None, bypassing the Sha256Hash validator.

• Item:

  • Name: Embedder Input Handling
  • Severity: medium
  • Findings: - Embedder methods do not validate empty input: At src/context_library/core/embedder.py:45-55, embed([]) passes an empty list to SentenceTransformer.encode() whose behavior with empty input is model-dependent. embed_query("") will embed an empty string producing a meaningless embedding vector. Neither method validates input.

• Item:

  • Name: Filesystem Adapter
  • Severity: medium
  • Findings: - has_lists structural hint only detects unordered lists: At src/context_library/adapters/filesystem.py:89-91, the regex r"^[\-\*\+]\s" only matches unordered list markers but does not detect ordered lists (e.g., 1., 2.). Documents with only ordered lists will have has_lists=False, potentially causing suboptimal chunking decisions.

• Item:

  • Name: Test Coverage Gaps
  • Severity: medium
  • Findings: - get_chunks_pending_sync and get_chunks_pending_deletion have no dedicated tests: src/context_library/storage/document_store.py:655-716 contains the recovery mechanism for SQLite/LanceDB inconsistency, but no test directly calls these methods to verify their return structure and content. A query bug in these methods would silently return wrong results, breaking the recovery path.

• Item:

  • Name: Schema & Configuration
  • Severity: medium
  • Findings: - WAL mode and synchronous=NORMAL not in schema.sql: Per FR-2.2, the schema must set WAL mode and synchronous=NORMAL. The schema.sql file only contains PRAGMA foreign_keys=ON and PRAGMA user_version=1. WAL mode and synchronous=NORMAL are set in DocumentStore.__init__() at runtime rather than in the schema file. Schema initialization via the SQL file alone does not enforce these settings.

• LanceDB table creation location deviates from spec: The architecture spec states the LanceDB table must be created using the schema defined in storage/vector_store.py. Table creation actually occurs in pipeline.py using an inline pa.schema() with the embedder's runtime dimension, not driven by vector_store.py.
  Filtered Out:

• ChunkVector docstring references wrong line numbers in pipeline.py — docstring line-number references are inherently fragile and not a code bug; style preference

• Mutable containers in frozen Pydantic models — defensive hardening suggestion, not a current bug or spec requirement

• Chunk.chunk_index has no non-negativity constraint — speculative improvement, no spec requires this

• LanceDB list_tables() called repeatedly inside the source loop — performance optimization suggestion, not a bug or spec gap

• BaseDomain sequential chunk_index invariant not enforced — defensive coding suggestion for future domains, not a current gap

• FR-4.4 adapter_id deterministic computation vs DB retrieval — functionally equivalent, mechanism deviation only with no behavioral impact

• FR-5.6 chunker joining restriction to same context_header — reasonable design choice, more restrictive than spec but not a gap

• FR-8.4 retired_at vs is_active naming — naming deviation only, soft-delete mechanism is functionally correct

• FR-9.2 source_id nested in lineage vs top-level field — accessible via result.lineage.source_id and surfaced in to_dict(), not a functional gap

• FR-3.4 EMBEDDING_DIM=384 hardcoded default — pipeline always passes runtime dimension explicitly, the constant is only a safety fallback

• Software Architect gap on validators.py expected_dim — confirmed implemented with optional parameter, trace through pipeline verified

• Software Architect gap on core/chunker.py disposition — file is reserved stub per spec, not modified in PR

• Context header prepend split across domain and pipeline — confirmed correct per spec (prepending is pipeline's responsibility)

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent