Requirements: MVP Phase 2

[orchestrator-bot[bot]]

Requirements Analysis

Auto-created from Issue #83

User Request

Begin development of the scope outlined in Phase 2 of documentation/roadmap.md

---

Labels: None
Requested by: @tinkermonkey

---

The orchestrator will analyze this request and develop detailed requirements.
When complete, Issue #83 will be updated with final requirements.

[orchestrator-bot[bot]]

Idea Research

---

I'll start by reading the project conventions and roadmap documentation to understand the context.Now let me research scheduling patterns and the Messages domain to round out the analysis.Now let me check the existing SQLite schema for scheduler-relevant fields and do additional targeted research.Now let me look at the existing scheduling infrastructure embedded in the schema and models to understand what's already been prepared for Phase 2.Now I have all the information needed to produce the comprehensive analysis. Here's the report:

Executive Summary

Phase 2 of the context-library project extends the Phase 1 foundation (semantic document ingestion and retrieval) with three capability groups: (1) a Scheduler for automated ingestion via polling and filesystem watching, (2) three new Adapters (Email, Rich Filesystem, Obsidian) for multi-source ingestion, and (3) a Messages Domain for conversational content chunking. The Phase 1 architecture is well-prepared for this extension — the SQLite schema already carries poll_strategy, poll_interval_sec, and last_fetched_at columns, and the BaseAdapter contract is stable. However, dependency research reveals several material risks: the planned Email stack (EmailEngine + email-oauth2-proxy) introduces commercial licensing costs, external server dependencies, and no Python-native API; the obsidianmd-parser library is immature and imposes a Python 3.12+ floor; and MarkItDown's PDF conversion has documented quality problems. These risks are tractable with alternative approaches explored below.

---

Idea Exploration

Phase 2 is structurally composed of five distinct workstreams. Each builds on Phase 1 primitives but introduces new concerns.

Workstream 1: Scheduler (Poller + Watcher)

The scheduler coordinates when adapters run. The Phase 1 pipeline currently operates as a single-shot ingest() call — the scheduler transforms this into a continuous process. The architecture documentation already defines a Scheduler class with register(), start(), stop(), poll_loop(), and handle_webhook() methods. The SQLite schema pre-provisions per-source scheduling metadata:

• poll_strategy (push | pull | webhook) — determines whether the scheduler polls the adapter, listens for filesystem events, or waits for external webhooks
• poll_interval_sec — for pull-based sources, the interval between polls
• last_fetched_at — enables interval-based scheduling decisions

The scheduler must handle two fundamentally different trigger models:

• Pull (polling): Timer-driven, calling adapter.fetch() at intervals. Straightforward but wasteful for filesystem sources.
• Push (filesystem watching): Event-driven via watchdog or watchfiles, translating filesystem events into SourceRef values that trigger ingestion. The roadmap specifies routing push events through handle_webhook() to reuse existing pipeline paths.

The key design decision is the concurrency model. The pipeline's ingest() method is synchronous (uses lancedb and sqlite3, both blocking). The scheduler must either: (a) run adapters on background threads with a thread pool, (b) use asyncio with blocking calls wrapped in run_in_executor, or (c) remain purely synchronous with a simple polling loop. watchfiles natively supports asyncio via awatch(), while watchdog is threading-only — this influences the choice.

Workstream 2: Shared Filesystem Watcher (_watching.py)

The _watching.py module is an internal utility shared by the Filesystem and Obsidian adapters. Per the roadmap, each adapter instantiates its own watcher pointed at its own path scope — no shared singleton. The module:

1. Wraps either watchdog or watchfiles behind a unified interface
2. Translates FileSystemEvent objects into SourceRef values
3. Maps to PollStrategy.PUSH for scheduler routing

This is an adapter-internal concern, not a scheduler concern. The watcher tells the scheduler "this source changed," and the scheduler triggers ingestion. The abstraction layer over two watcher implementations is justified: watchfiles offers built-in debouncing and async support with lower latency (Rust-backed notify crate); watchdog offers richer event types (distinguishes moved from created+deleted) and broader platform coverage. Supporting both allows selecting the best backend per deployment.

A notable gap: neither library handles atomic file saves correctly by default. Editors like Vim write to a temp file then rename, producing deleted + created events rather than modified. The _watching.py module will need debouncing or rename-detection logic regardless of backend.

Workstream 3: Rich Filesystem Adapter (filesystem_rich.py)

This extends the existing FilesystemAdapter (which handles .md only) to support arbitrary file formats by converting them to markdown at ingestion time. The conversion pipeline:

1. MarkItDown (primary): Handles PDF, DOCX, XLSX, PPTX, HTML, images, audio. MIT licensed, actively maintained by Microsoft. Simple API: MarkItDown().convert(path).text_content.
2. Pandoc (fallback): Handles LaTeX, EPUB, RST, ODT, and formats MarkItDown misses. Called via pypandoc.convert_file() or direct subprocess.

The adapter also captures filesystem metadata (MIME type, file size, timestamps, directory hierarchy) as chunk metadata for retrieval augmentation.

Key technical consideration: the existing FilesystemAdapter has domain = Domain.NOTES hardcoded. The rich adapter will likely also target the NOTES domain, since the converted content is unstructured markdown. The adapter_id should distinguish the two: e.g., filesystem_rich:{path} vs filesystem:{path}.

Workstream 4: Obsidian Adapter

The Obsidian adapter ingests an Obsidian vault, treating it as a filesystem source (notes are already markdown) but enriching chunks with vault-specific metadata:

• Tags, aliases, frontmatter properties from YAML front matter
• Wikilink graph edges (backlinks, forward links) for graph-aware ranking
• Creation/modification dates from filesystem or frontmatter

The roadmap specifies two metadata extraction libraries: obsidiantools for graph-level analytics and obsidianmd-parser for per-note parsing. These serve complementary roles — obsidiantools builds a NetworkX graph of the entire vault's link structure, while obsidianmd-parser extracts per-note frontmatter, tags, and Dataview fields.

Since Obsidian vaults are just directories of .md files, the adapter uses _watching.py for filesystem event capture, sharing that infrastructure with the rich filesystem adapter.

Workstream 5: Messages Domain

The Messages domain implements thread-aware chunking for conversational content (email, chat). The architecture documentation defines MessageMetadata with: thread_id, message_id, sender, recipients, timestamp, in_reply_to, subject, is_thread_root.

Key chunking challenges for messages:

• Thread context preservation: A message's meaning depends on the thread. Chunks must carry enough context (the subject line, parent message) for semantic search to work.
• Author attribution: Unlike notes, messages have explicit authors whose identity is part of the semantic content.
• Temporal ordering: Messages have strict chronological order within threads, and this order affects meaning.

The BaseDomain.chunk() contract returns list[Chunk] with sequential chunk_index and optional context_header. For messages, the context_header could encode thread context (e.g., "Thread: Re: Project update > From: alice@example.com"), and domain_metadata stores the structured message fields.

---

Potential Directions

Scheduler Architecture Options

Option A: Thread-pool scheduler with synchronous pipeline

• Use concurrent.futures.ThreadPoolExecutor to run ingest() calls in parallel across adapters
• Simple, matches the existing synchronous codebase
• Risk: SQLite's WAL mode supports concurrent reads but serializes writes; concurrent ingestion from multiple adapters may contend on writes

Option B: Asyncio scheduler with executor-wrapped pipeline

• Use asyncio.run_in_executor() for blocking pipeline calls
• Enables native watchfiles.awatch() integration
• More complex, but future-proofs for Phase 4's performance optimizations
• The scheduler's poll_loop and handle_webhook map naturally to async tasks

Option C: Simple single-threaded polling loop

• Round-robin through registered adapters, checking last_fetched_at + poll_interval_sec < now
• No concurrency complexity
• Adequate for small deployments; won't scale to many adapters

Email Adapter Strategy

The roadmap specifies EmailEngine (Node.js REST server, commercial license ~$995/year) and email-oauth2-proxy (standalone proxy process). Both require managing external processes from Python, which is architecturally heavy for a library.

Alternative: Python-native IMAP with direct OAuth 2.0

• Python's imaplib + smtplib are standard library
• OAuth 2.0 token management via oauthlib or authlib (both well-maintained, MIT licensed)
• Gmail API via google-api-python-client (already listed as an optional dependency in pyproject.toml)
• Microsoft Graph API via msgraph-sdk-python for Outlook/O365
• Eliminates external process dependencies entirely

This approach trades EmailEngine's provider abstraction for direct provider-specific implementations but avoids the commercial license and operational overhead.

Obsidian Metadata Extraction

The roadmap specifies both obsidiantools (graph analytics) and obsidianmd-parser (per-note parsing). Research reveals obsidianmd-parser is extremely immature (version 0.4.0, 1 star on Codeberg, Python 3.12+ minimum).

Alternative: obsidiantools + python-frontmatter

• obsidiantools already provides: wikilink graph, backlinks, note connectivity, frontmatter access
• python-frontmatter (mature, well-maintained) handles YAML frontmatter parsing
• Tag extraction can be done via regex on the markdown content (#tag patterns)
• Dataview field extraction (if needed) can be implemented with targeted regex
• Eliminates the Python 3.12+ constraint and the risk of depending on an unmaintained package

Document Conversion Tiering

MarkItDown's PDF quality issues (documented at ~25% success rate for complex PDFs) suggest a more nuanced conversion strategy:

1. Tier 1 — MarkItDown: Fast, in-process, handles most Office formats well
2. Tier 2 — Pandoc: Subprocess-based fallback for formats MarkItDown doesn't cover or fails on
3. Tier 3 — Docling (IBM): An alternative PDF-to-markdown converter with better table and layout handling, worth evaluating as a MarkItDown replacement for PDF specifically

Making Pandoc an optional dependency (extras_require) avoids forcing the external binary requirement on all users.

---

References and Prior Art

Scheduling Patterns in Python Document Pipelines

• APScheduler (Advanced Python Scheduler): Mature library for cron-like and interval-based scheduling in Python. Supports both sync and async execution. Could be used as the backbone of the poller if building a custom scheduler feels excessive. MIT licensed, actively maintained.
• Celery: Overkill for this use case (requires a message broker), but its task-retry and dead-letter patterns are worth studying for the pipeline's error recovery.
• Haystack (deepset): A RAG framework that faced similar multi-source ingestion scheduling challenges. Uses a pipeline abstraction similar to context-library but delegates scheduling to the application layer.

Filesystem Watching in Document Ingestion

• fswatch (C library): Cross-platform filesystem watcher used by some document indexing systems. Not directly relevant (Python wrappers exist but watchdog/watchfiles are better integrated).
• inotify-tools (Linux): The underlying Linux kernel API that both watchdog and watchfiles use via inotify. Understanding inotify's limitations (per-user watch limit fs.inotify.max_user_watches, no recursive support in the kernel API) informs the _watching.py design.
• Django's auto-reloader: Uses watchfiles (via django-watchfiles) for file change detection. Their integration pattern — filter by file extension, debounce, restart on change — is a useful reference for the adapter-facing interface.

Email Ingestion in RAG Systems

• LlamaIndex email readers: Provides GmailReader and OutlookReader that fetch emails via API and return Document objects. These are thin wrappers around the provider APIs with no scheduling or incremental ingestion — they fetch everything on each call. Context-library's version-based diffing would add significant value here.
• Unstructured.io: An open-source document processing library that handles email (.eml, .msg) as one of many input formats. Their email parsing extracts headers, body, and attachments as separate elements — a pattern relevant to the Messages domain chunking strategy.

Obsidian Vault Indexing

• Obsidian Local REST API plugin: A community plugin that exposes vault content via HTTP. An alternative to direct filesystem access, but requires the Obsidian app to be running. Not suitable for headless/server deployments.
• Khoj: An open-source AI personal assistant that indexes Obsidian vaults. Uses direct filesystem access with frontmatter parsing and wikilink resolution. Their chunking approach (per-note with heading boundaries) is similar to what context-library's NotesDomain already implements.

---

Technical Considerations

Schema Readiness

The SQLite schema is already provisioned for Phase 2's scheduler:

• sources.poll_strategy accepts push, pull, webhook — matching all three trigger models
• sources.poll_interval_sec stores per-source polling intervals
• sources.last_fetched_at tracks the last successful fetch time
• The register_source() method currently hardcodes poll_strategy = "pull" — this needs parameterization for push/webhook sources

The PollStrategy enum referenced in the architecture documentation does not yet exist in the codebase. It will need to be added to models.py alongside the existing Domain and ChunkType enums.

Adapter Contract Gaps

The BaseAdapter base class currently lacks:

• A poll_strategy property — the architecture class diagram shows this but it's not in the implementation
• Any mechanism for adapters to signal which source changed (for push-based adapters, the watcher knows the specific file, but the adapter contract's fetch(source_ref) takes a single ref and yields all content)

For push-based adapters, the flow would be: watcher detects file change → watcher produces SourceRef → scheduler calls adapter.fetch(source_ref) → adapter yields only the changed file. This works with the existing contract if adapters treat source_ref as a path filter. The filesystem adapter currently ignores source_ref and scans the whole directory — push-based adapters will need to respect it.

Dependency Version Constraints

Current project minimum: Python 3.10 (from pyproject.toml).

Phase 2 dependency floor:

• watchdog: Python 3.9+
• watchfiles: Python 3.9+
• MarkItDown: Python 3.10+
• obsidiantools: Python 3.9+
• obsidianmd-parser: Python 3.12+ — would raise the project minimum
• pypandoc: Python 3.7+

If obsidianmd-parser is included, the project's minimum Python version must increase to 3.12. This is a significant compatibility cost. If dropped in favor of obsidiantools + python-frontmatter, the Python 3.10 floor is preserved.

Messages Domain Chunking Complexity

Email content presents unique chunking challenges not faced by the Notes domain:

1. Quoted reply chains: Email bodies contain nested quoted text from previous messages. The chunker must decide whether to include quoted material (useful for context) or strip it (avoids duplication since the original message is separately indexed).
2. HTML email: Most modern email is HTML. Conversion to markdown (via MarkItDown or a dedicated HTML-to-markdown converter like html2text) must happen before chunking.
3. Attachments: Emails may contain attached documents. These could be processed by the Rich Filesystem Adapter's conversion pipeline, but the attachment-to-email relationship (which email contained this PDF?) must be preserved in domain_metadata.
4. Thread reconstruction: The in_reply_to and thread_id fields in MessageMetadata enable thread-aware retrieval, but the chunker must decide the unit of chunking — per-message, per-thread-segment, or per-logical-topic within a thread.

Concurrency and Consistency

Running multiple adapters concurrently (via the scheduler) introduces write contention on SQLite. The current pipeline writes chunks, lineage records, and sync log entries within transactions. SQLite's WAL mode allows concurrent reads during writes, but only one writer at a time. Concurrent ingest() calls from different adapters will serialize on SQLite writes.

For most deployments (hundreds to low thousands of sources), this is not a bottleneck — SQLite write throughput is typically 10,000+ inserts/second with WAL mode. For larger deployments, the pipeline could batch writes or use a write queue.

Testing Strategy

Phase 1 has ~7,646 lines of tests across 26 modules. Phase 2 will need:

• Scheduler tests: Timer-based polling, event-driven triggers, concurrent adapter execution, error recovery
• Watcher tests: Filesystem event translation, debouncing, editor rename patterns, cross-platform behavior
• Adapter integration tests: Email API mocking (Gmail, Outlook), Obsidian vault fixtures, format conversion verification
• Messages domain tests: Thread-aware chunking, quoted reply handling, HTML-to-markdown conversion, metadata extraction

The existing test infrastructure uses pytest with fixtures in tests/fixtures/. Mock-heavy testing is appropriate for external API adapters; filesystem-based integration tests suit the watcher and Obsidian adapter.

---

Generated by Orchestrator Bot 🤖
Processed by the idea_researcher agent

[orchestrator-bot[bot]]

Business Analysis

---

Executive Summary

Phase 2 extends the context-library MVP with automated ingestion scheduling and three new data source adapters (Email, Rich Filesystem, Obsidian), plus a Messages domain for conversational content. The Phase 1 architecture is structurally prepared for this work: the SQLite schema pre-provisions scheduling fields (poll_strategy, poll_interval_sec, last_fetched_at), the BaseAdapter contract is stable, and scheduler stub files (poller.py, watcher.py) exist. Four domains are defined in the roadmap (messages.py, events.py, tasks.py — stub files are already present) but only NotesDomain is implemented. Phase 2 delivers the MessagesDomain, two scheduler components, three adapters, and a shared filesystem watcher utility.

---

Functional Requirements

FR-1: Poller (Pull-Based Scheduler)

FR-1.1 The system must read poll_interval_sec and last_fetched_at per source from the document store to determine when each source is due for re-ingestion.

FR-1.2 The poller must invoke adapter.fetch(source_ref) for each registered pull-based source when its interval has elapsed and pass results to the ingestion pipeline.

FR-1.3 The poller must update last_fetched_at in the document store after each successful fetch attempt.

FR-1.4 The poller must isolate per-source failures so that one failing source does not prevent other sources from being polled.

FR-1.5 The poller must support start() and stop() lifecycle methods to begin and halt continuous polling.

FR-1.6 The poller must support registering adapters for polling via a register() method that accepts an adapter instance and its associated domain chunker.

FR-1.7 The poll_strategy column value for sources managed by the poller must be "pull".

FR-2: Watcher (Push-Based Scheduler)

FR-2.1 The watcher must accept filesystem events (from _watching.py) and translate them into pipeline ingestion calls for the affected source ref.

FR-2.2 The watcher must route push events through a handle_webhook() path distinct from the polling loop.

FR-2.3 The watcher must support start() and stop() lifecycle methods.

FR-2.4 The poll_strategy column value for sources managed by the watcher must be "push".

FR-2.5 The watcher must isolate per-event failures so that one failed event does not halt event processing.

FR-3: Shared Filesystem Watcher (_watching.py)

FR-3.1 The module must support at least one of watchdog or watchfiles as backend implementations behind a unified interface.

FR-3.2 Each adapter that uses _watching.py must instantiate its own watcher scoped to its own path — no shared singleton.

FR-3.3 The module must translate backend-specific filesystem event objects into SourceRef values compatible with the pipeline's fetch(source_ref) contract.

FR-3.4 The module must map filesystem events to PollStrategy.PUSH so the scheduler routes them through handle_webhook().

FR-3.5 The module must expose a PollStrategy enum with at least PUSH and PULL values; this enum must be added to models.py alongside existing Domain and ChunkType enums.

FR-3.6 The module must handle editor atomic-save patterns (rename-based saves that produce deleted + created sequences) without triggering spurious deletion ingestion.

FR-4: Rich Filesystem Adapter (filesystem_rich.py)

FR-4.1 The adapter must convert non-markdown files (PDF, DOCX, XLSX, PPTX, HTML, images, audio) to markdown at ingestion time using MarkItDown as the primary conversion library.

FR-4.2 The adapter must fall back to Pandoc for file formats not supported by MarkItDown (e.g., LaTeX, EPUB, RST, ODT).

FR-4.3 The adapter must emit NormalizedContent with StructuralHints populated from filesystem metadata: MIME type, file size in bytes, creation timestamp, modification timestamp, and directory hierarchy.

FR-4.4 The adapter must use _watching.py for filesystem event capture when configured for push-based ingestion.

FR-4.5 The adapter's adapter_id must be distinct from the existing FilesystemAdapter to allow both to be registered on the same document store without conflict.

FR-4.6 The adapter must set domain to Domain.NOTES.

FR-4.7 The adapter must include converted file format as metadata on each NormalizedContent item (i.e., original MIME type or extension).

FR-5: Obsidian Adapter (obsidian.py)

FR-5.1 The adapter must ingest an Obsidian vault directory, treating each vault note as a source ref.

FR-5.2 The adapter must extract per-note metadata using a library capable of parsing YAML frontmatter, tags, aliases, and (optionally) Dataview fields from .md files.

FR-5.3 The adapter must extract vault-level graph metadata (backlinks, forward wikilinks, note connectivity) using obsidiantools.

FR-5.4 The adapter must store extracted metadata — tags, aliases, frontmatter properties, wikilink graph edges, creation/modification dates — in domain_metadata on each Chunk for use in metadata-filtered retrieval.

FR-5.5 The adapter must use _watching.py for filesystem event capture when configured for push-based ingestion.

FR-5.6 The adapter must not perform format conversion, since vault notes are already markdown.

FR-5.7 The adapter must set domain to Domain.NOTES.

FR-6: Email Adapter

FR-6.1 The adapter must support ingesting email content from at least one email provider (Gmail, Outlook, or IMAP-compatible server).

FR-6.2 The adapter must extract email header fields — sender, recipients, subject, timestamp, message-ID, in-reply-to — and store them as structured metadata.

FR-6.3 The adapter must convert HTML email bodies to markdown before yielding NormalizedContent.

FR-6.4 The adapter must set domain to Domain.MESSAGES.

FR-6.5 The adapter must support incremental ingestion — fetching only messages newer than last_fetched_at — using the document store's tracking data.

FR-6.6 Email credentials and OAuth tokens must not be stored in plaintext within the document store or configuration.

FR-6.7 The adapter must preserve thread relationships in the metadata it yields, specifically thread_id and in_reply_to values.

FR-7: Messages Domain (messages.py)

FR-7.1 The domain must implement the BaseDomain.chunk() contract, returning list[Chunk] with sequential chunk_index values.

FR-7.2 Each chunk must carry domain_metadata populated with MessageMetadata fields: thread_id, message_id, sender, recipients, timestamp, in_reply_to, subject, is_thread_root.

FR-7.3 The context_header on each chunk must encode thread context sufficient for semantic search to function without the full message body (e.g., subject line and sender attribution).

FR-7.4 The domain must handle quoted reply chains in email bodies — deciding per implementation whether to include or strip quoted material — without duplicating content already indexed as a separate source.

FR-7.5 The domain must be registered under Domain.MESSAGES.

FR-7.6 The MessageMetadata model must be defined as a Pydantic model in models.py consistent with existing immutable model conventions.

---

User Stories

Scheduler

US-1: Automated Pull-Based Ingestion
As a developer using context-library, I want pull-based sources to be automatically re-ingested at their configured intervals so that the search index stays current without manual pipeline invocations.

Acceptance Criteria:

• Given an adapter registered with poll_strategy = "pull" and poll_interval_sec = 3600, When the poller is running and 3600 seconds have elapsed since last_fetched_at, Then the poller calls adapter.fetch() and passes results to the ingestion pipeline.
• Given a source fetch that raises an exception, When the poller processes that source, Then the error is recorded and the poller continues processing remaining sources without stopping.
• Given a running poller, When stop() is called, Then the poller ceases scheduling new ingestion calls.

---

US-2: Filesystem Event-Driven Ingestion
As a developer, I want file changes to trigger immediate re-ingestion rather than waiting for the next poll interval so that the index reflects modifications as soon as they occur.

Acceptance Criteria:

• Given a source with poll_strategy = "push" and a watcher active on its path, When a file in that path is created or modified, Then the watcher triggers ingestion of that specific source ref within a bounded delay.
• Given an editor that saves via atomic rename (write to temp file, then rename), When the save sequence occurs, Then the watcher does not emit a spurious deletion event for the original file path.
• Given a webhook event that raises a pipeline error, When the watcher processes that event, Then the error is isolated and subsequent events continue to be processed.

---

Rich Filesystem Adapter

US-3: Multi-Format File Ingestion
As a developer, I want to ingest PDFs, Office documents, and other non-markdown files from a local directory so that all files in a directory are searchable regardless of format.

Acceptance Criteria:

• Given a directory containing .pdf, .docx, and .xlsx files, When the rich filesystem adapter runs fetch(), Then each file is converted to markdown and yielded as a NormalizedContent item.
• Given a file format not supported by MarkItDown (e.g., .rst), When the adapter encounters that file, Then it falls back to Pandoc for conversion.
• Given any successfully converted file, When the adapter yields NormalizedContent, Then StructuralHints includes the file's MIME type, size in bytes, and modification timestamp.
• Given both FilesystemAdapter and filesystem_rich registered on the same document store, When both are registered, Then their adapter_id values are distinct and neither overwrites the other's registration.

---

US-4: Real-Time Rich File Monitoring
As a developer, I want the rich filesystem adapter to detect file changes in near real-time so that new or modified non-markdown files are ingested automatically.

Acceptance Criteria:

• Given the adapter is configured with a push poll_strategy, When a supported file is added or modified in the monitored directory, Then a push event is triggered and the file is re-ingested.
• Given the adapter is configured with a pull poll_strategy, When fetch() is called, Then no watcher is started and all files are scanned at call time.

---

Obsidian Adapter

US-5: Obsidian Vault Ingestion with Graph Metadata
As a developer with an Obsidian vault, I want to ingest vault notes alongside their wikilink relationships, tags, and frontmatter so that retrieval can filter by vault-specific metadata and graph structure.

Acceptance Criteria:

• Given an Obsidian vault directory, When the adapter runs fetch(), Then each .md note is yielded as a NormalizedContent item without format conversion.
• Given a note with YAML frontmatter containing tags and aliases, When that note is chunked, Then the resulting chunks have domain_metadata populated with extracted tags, aliases, and frontmatter properties.
• Given a note with wikilinks to other vault notes, When that note is chunked, Then domain_metadata includes the note's forward links and backlinks as extracted from obsidiantools.
• Given the vault has filesystem change events enabled, When a vault note is modified, Then only the changed note is re-ingested.

---

Email Adapter

US-6: Email Content Ingestion
As a developer, I want to ingest emails from my email account so that email content is searchable through the same query interface as other sources.

Acceptance Criteria:

• Given valid credentials for a supported email provider, When the adapter runs fetch(), Then email messages are fetched, their HTML bodies converted to markdown, and each message yielded as a NormalizedContent item with domain = Domain.MESSAGES.
• Given an email with In-Reply-To and Thread-Id headers, When the adapter yields NormalizedContent, Then the metadata includes thread_id, in_reply_to, sender, recipients, subject, and timestamp.
• Given a prior fetch has occurred, When the adapter fetches again, Then only messages newer than last_fetched_at are fetched, not the full mailbox.
• Given OAuth-based authentication is required by the provider, When credentials are configured, Then no plaintext credentials are persisted in the document store or in any file written by the library.

---

Messages Domain

US-7: Thread-Aware Message Chunking
As a developer, I want email and chat messages chunked with thread context preserved so that semantic search results surface relevant messages with enough context to understand their meaning.

Acceptance Criteria:

• Given a NormalizedContent item with domain = Domain.MESSAGES carrying message metadata, When MessagesDomain.chunk() is called, Then the result is a list[Chunk] with sequential chunk_index values.
• Given a message that is part of a thread, When chunked, Then each chunk's context_header includes the thread subject and sender identity.
• Given a message with a quoted reply chain, When chunked, Then the chunker does not index quoted material as a separate new chunk that duplicates already-indexed content.
• Given any message chunk, When inspected, Then domain_metadata contains all MessageMetadata fields: thread_id, message_id, sender, recipients, timestamp, in_reply_to, subject, is_thread_root.
• Given MessageMetadata is referenced by the adapter and domain, When either is imported, Then the MessageMetadata Pydantic model is available from models.py and follows the same frozen/immutable conventions as existing models.

---

Generated by Orchestrator Bot 🤖
Processed by the business_analyst agent

[orchestrator-bot[bot]]

Architecture Design

---

I need to analyze the codebase and project conventions before producing the architecture analysis.Now let me read the remaining critical files I need for the architecture analysis.Now let me read the existing stubs and documentation files for additional context.Now let me read the chunking strategy documentation for Messages domain specifics.Now I have a thorough understanding of the entire codebase. Let me produce the architecture analysis.

Executive Summary

Phase 2 extends the context-library with two scheduler components (Poller, Watcher), three adapters (Rich Filesystem, Obsidian, Email), a shared filesystem watcher utility, the MessagesDomain chunker, and supporting model additions (PollStrategy enum, MessageMetadata). The Phase 1 architecture is structurally prepared: the SQLite schema already provisions poll_strategy, poll_interval_sec, and last_fetched_at columns; stub files exist at every target path; and the BaseAdapter/BaseDomain contracts are stable. This analysis defines the component architecture, identifies reuse points, addresses cross-cutting concerns, and provides a sequenced implementation plan.

---

System Architecture

Component Dependency Graph

┌─────────────────────────────────────────────────────────────────┐
│                       Scheduler Layer                           │
│  ┌──────────┐    ┌──────────┐                                   │
│  │  Poller   │    │ Watcher  │                                   │
│  │ (pull)    │    │ (push)   │                                   │
│  └────┬─────┘    └────┬─────┘                                   │
│       │               │                                         │
│       └──────┬────────┘                                         │
│              ▼                                                  │
│     IngestionPipeline.ingest(adapter, domain_chunker)           │
└─────────────────────────────────────────────────────────────────┘
                              │
          ┌───────────────────┼───────────────────┐
          ▼                   ▼                   ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Adapter Layer    │ │  Domain Layer    │ │ Storage Layer    │
│                  │ │                  │ │                  │
│ filesystem_rich  │ │ MessagesDomain   │ │ DocumentStore    │
│ obsidian         │ │ (NotesDomain     │ │ VectorStore      │
│ email            │ │  unchanged)      │ │                  │
│ (filesystem      │ │                  │ │                  │
│  unchanged)      │ │                  │ │                  │
└────────┬─────────┘ └──────────────────┘ └──────────────────┘
         │
         ▼
┌─────────────────┐
│ _watching.py     │
│ (shared utility) │
│ per-adapter      │
│ watcher instance │
└─────────────────┘

Layer Responsibilities

Scheduler Layer — Lifecycle management for continuous ingestion. The Poller reads poll_interval_sec and last_fetched_at from DocumentStore, invokes adapter.fetch() on schedule, and passes results through IngestionPipeline.ingest(). The Watcher receives push events from _watching.py and routes them through handle_webhook() into the same pipeline. Both components own start()/stop() lifecycle and per-source error isolation.

Adapter Layer — Three new BaseAdapter implementations. Each adapter's fetch() yields NormalizedContent. The Rich Filesystem adapter converts non-markdown files to markdown via MarkItDown/Pandoc. The Obsidian adapter reads .md vault notes and extracts graph/frontmatter metadata. The Email adapter fetches messages via EmailEngine and converts HTML bodies to markdown.

Domain Layer — MessagesDomain implements BaseDomain.chunk() for conversational content. It produces Chunk objects with MessageMetadata in domain_metadata and thread-aware context_header values.

Shared Utility — _watching.py wraps watchdog or watchfiles behind a unified interface. Each adapter instantiates its own watcher scoped to its path. The module translates OS-level filesystem events into SourceRef values and maps them to PollStrategy.PUSH.

---

Scalability Design

Poller Concurrency Model

The Poller runs a single-threaded scheduling loop using asyncio or threading.Timer. Source evaluation is sequential within each tick: the loop iterates registered sources, checks last_fetched_at + poll_interval_sec < now, and dispatches due sources. This is adequate for the current library scope (tens of sources, not thousands).

If the ingestion pipeline for one source blocks, it delays subsequent sources in the same tick. This is acceptable because:

• IngestionPipeline.ingest() processes sources one at a time already (it iterates adapter.fetch() sequentially)
• Per-source error isolation ensures one failure does not block the loop
• The next tick recalculates due times, so delayed sources catch up

Growth path: If source counts grow to hundreds, the Poller can dispatch ingestion calls to a thread pool. The IngestionPipeline and DocumentStore are both thread-safe (SQLite WAL mode supports concurrent readers), and LanceDB handles concurrent appends. This is not needed for Phase 2.

Watcher Throughput

Filesystem watchers can produce bursts of events (e.g., git pull updating 50 files). The _watching.py module should implement debouncing: collect events into a bounded time window (e.g., 500ms) and coalesce events for the same path before dispatching. This prevents redundant ingestion of the same file when editors produce delete + create sequences for atomic saves.

Email Incremental Fetch

The Email adapter uses last_fetched_at from the sources table to fetch only new messages. This scales linearly with new message volume, not total mailbox size. Initial ingestion of a large mailbox should be bounded by a configurable max_initial_messages parameter to avoid unbounded first runs.

---

Established Patterns

The Phase 1 codebase establishes clear patterns that Phase 2 must follow:

Pattern 1: Adapter Contract

All adapters implement BaseAdapter with four abstract members (fetch(), adapter_id, domain, normalizer_version) and inherit the concrete register() method. New adapters must:

• Return a deterministic adapter_id from the same config (e.g., "filesystem_rich:{resolved_path}", "obsidian:{vault_path}", "email:{account_identifier}")
• Yield NormalizedContent with populated StructuralHints from fetch()
• Set domain to the appropriate Domain enum value

Reference implementation: src/context_library/adapters/filesystem.py

Pattern 2: Domain Chunker Contract

All domains implement BaseDomain.chunk(content: NormalizedContent) -> list[Chunk]. Each Chunk must have:

• chunk_hash computed via compute_chunk_hash(content) (content only, excluding context_header)
• Sequential chunk_index values starting at 0
• domain_metadata as a JSON-serializable dict (stored as TEXT in SQLite)
• chunk_type from the ChunkType enum

Reference implementation: src/context_library/domains/notes.py

Pattern 3: Frozen Pydantic Models

All data models use model_config = ConfigDict(frozen=True). New models (MessageMetadata, PollStrategy) must follow this convention. Field validators use @field_validator with @classmethod.

Reference: src/context_library/storage/models.py

Pattern 4: Per-Source Error Isolation

The pipeline catches exceptions per-source and continues processing. Scheduler components must maintain this invariant: one source failing must not prevent others from being processed.

Reference: src/context_library/core/pipeline.py:365-418

Pattern 5: Idempotent Registration

DocumentStore.register_adapter() and register_source() are idempotent (INSERT OR IGNORE). Adapters call register() before every ingestion run. New adapters inherit this behavior from BaseAdapter.register().

Pattern 6: Content-Addressed Identity

Chunks are identified by SHA-256 hash of their normalized content. The hash excludes context_header so the same content under different headings produces the same hash. This enables cross-version and cross-source deduplication.

Pattern 7: Dual Storage with Sync Log

SQLite is the source of truth; LanceDB is a derived index. Writes go to SQLite first, then LanceDB. The lancedb_sync_log table tracks pending operations for recovery. New components must not write to LanceDB without first recording the operation in the sync log.

---

Component Reuse

Existing Components Used Directly

Component	Used By	How
IngestionPipeline.ingest()	Poller, Watcher	Both call pipeline.ingest(adapter, domain_chunker) — the pipeline is unchanged
DocumentStore.update_last_fetched_at()	Poller	After successful fetch, update tracking timestamp
DocumentStore.get_latest_version()	Pipeline (unchanged)	Provides last_fetched_at and prev version for diff
compute_chunk_hash()	MessagesDomain	Same hashing for message chunks
ChunkType enum	MessagesDomain	Uses STANDARD for message chunks
Domain enum	All new adapters	Domain.NOTES for filesystem_rich/obsidian, Domain.MESSAGES for email
NormalizedContent, StructuralHints	All new adapters	Yield from fetch()
Chunk, LineageRecord	MessagesDomain, Pipeline	Domain produces Chunks, Pipeline produces LineageRecords
BaseAdapter.register()	All new adapters	Inherited, idempotent adapter registration

Existing Schema Used Directly

The SQLite schema (schema.sql) requires no migration for Phase 2. The sources table already has:

• poll_strategy TEXT NOT NULL CHECK (poll_strategy IN ('push', 'pull', 'webhook'))
• poll_interval_sec INTEGER
• last_fetched_at DATETIME

The DocumentStore.register_source() currently hardcodes poll_strategy = "pull" (document_store.py:155). This needs a parameter addition to accept the strategy from the caller, or the Poller/Watcher should update the column after registration.

New Shared Utility: _watching.py

Used by: filesystem_rich.py, obsidian.py

Not used by: email (pull-based), filesystem.py (Phase 1 adapter, unchanged)

The underscore prefix follows Python convention for internal modules not part of the public API.

---

Architecture Decision Records

ADR-P2-01: Poller Threading Model

Context: The Poller needs to run continuously, checking source intervals and dispatching ingestion.

Decision: Use a single background threading.Thread with a tick-based loop. The thread sleeps for a configurable tick interval (default 60s), wakes, scans all registered sources, and dispatches due sources synchronously within the thread.

Rationale:

• asyncio would require the IngestionPipeline and all adapters to be async — a large refactor for no Phase 2 benefit
• threading.Timer per-source creates timer proliferation as source count grows
• A single thread with tick-based scanning is the simplest model that satisfies FR-1.1 through FR-1.7
• threading.Event enables clean stop() semantics

Trade-offs:

• (+) Simple, no async refactor needed
• (+) Clean start/stop lifecycle via threading.Event
• (-) Ingestion of one source blocks the next within the same tick
• (-) Not suitable for hundreds of sources (acceptable for Phase 2 scope)

ADR-P2-02: Filesystem Watcher Backend Selection

Context: _watching.py must support at least one of watchdog or watchfiles (FR-3.1).

Decision: Implement watchdog as the primary backend. watchfiles support is deferred to a follow-up. Backend selection uses a simple import-time check: try watchdog, fall back to watchfiles if watchdog is not installed.

Rationale:

• watchdog is more mature, widely used, and has broader platform support
• watchfiles (Rust-backed) has lower latency but requires a Rust toolchain for installation in some environments
• Implementing both backends simultaneously doubles testing surface without clear Phase 2 benefit
• The thin abstraction layer in _watching.py makes adding watchfiles later a one-file change

Trade-offs:

• (+) Single backend reduces testing complexity
• (+) watchdog is well-documented and battle-tested
• (-) Misses watchfiles lower-latency benefits
• (-) Defers full FR-3.1 compliance to a follow-up

ADR-P2-03: Email Adapter Uses EmailEngine

Context: The Email adapter must support at least one email provider (FR-6.1) while keeping credentials secure (FR-6.6).

Decision: Use EmailEngine as the email backend, accessed via its REST API. EmailEngine is a self-hosted headless email client that abstracts IMAP/Gmail API/MS Graph behind a unified API and manages OAuth tokens internally.

Rationale:

• EmailEngine handles the complexity of email provider auth (OAuth 2.0, XOAUTH2, app passwords) outside the library
• The adapter only needs to make HTTP requests to a local EmailEngine instance — no IMAP protocol handling, no token refresh logic
• Credentials are stored by EmailEngine, not by the context-library — satisfying FR-6.6
• EmailEngine supports incremental fetch via message UID/date queries — satisfying FR-6.5

Trade-offs:

• (+) Unified API across Gmail, Outlook, IMAP — satisfies FR-6.1 for all major providers
• (+) Credential management is fully delegated — clean separation of concerns
• (-) Requires users to run an EmailEngine instance alongside the library
• (-) Adds an operational dependency (Docker container or standalone process)
• The roadmap explicitly specifies EmailEngine, confirming this as the intended approach

ADR-P2-04: MessageMetadata as Pydantic Model in models.py

Context: MessageMetadata must be available to both the Email adapter and MessagesDomain (FR-7.6).

Decision: Define MessageMetadata as a frozen Pydantic BaseModel in src/context_library/storage/models.py alongside existing models.

Rationale:

• Follows the existing pattern where all shared data models live in models.py
• Frozen model enforces immutability consistent with Chunk, LineageRecord, etc.
• Both the adapter (populating metadata on NormalizedContent.structural_hints or as auxiliary data) and the domain chunker (copying metadata to Chunk.domain_metadata) can import from the same location
• The domain_metadata field on Chunk is dict[str, object] | None, so MessageMetadata is serialized to dict via .model_dump() before assignment

ADR-P2-05: PollStrategy Enum Placement

Context: _watching.py needs a PollStrategy enum (FR-3.5) that is also used by the scheduler and document store.

Decision: Add PollStrategy as a str, Enum in src/context_library/storage/models.py alongside Domain and ChunkType.

Rationale:

• The SQLite schema already has a CHECK constraint on poll_strategy IN ('push', 'pull', 'webhook') — the enum values must match exactly
• Placing it in models.py follows the pattern of Domain and ChunkType (schema-aligned enums in models)
• All consumers (scheduler, adapters, document store) already import from models.py

---

Implementation Plan

Sequencing Rationale

Components are ordered by dependency: models first (no dependencies), then utilities, then adapters and domains (depend on models), then schedulers (depend on adapters and pipeline).

Step 1: Model Additions

Files: src/context_library/storage/models.py

Changes:

1. Add PollStrategy(str, Enum) with values PUSH = "push", PULL = "pull", WEBHOOK = "webhook"
2. Add MessageMetadata(BaseModel) with model_config = ConfigDict(frozen=True) and fields: thread_id: str, message_id: str, sender: str, recipients: list[str], timestamp: str, in_reply_to: str | None, subject: str | None, is_thread_root: bool
3. Add @field_validator("timestamp") on MessageMetadata using validate_iso8601_timestamp

Tests: Unit tests for PollStrategy enum values matching schema CHECK constraint, MessageMetadata immutability, field validation, .model_dump() serialization.

Step 2: DocumentStore register_source Update

Files: src/context_library/storage/document_store.py

Changes:

1. Add poll_strategy: PollStrategy = PollStrategy.PULL and poll_interval_sec: int | None = None parameters to register_source()
2. Update the INSERT statement to pass these values
3. Add a get_sources_due_for_poll() method that queries sources where poll_strategy = 'pull' and last_fetched_at + poll_interval_sec < now (or last_fetched_at IS NULL)

Tests: Verify sources are created with correct poll_strategy, verify get_sources_due_for_poll() returns only due sources.

Step 3: Shared Filesystem Watcher (_watching.py)

Files: src/context_library/adapters/_watching.py (new file)

Changes:

1. Define a FileEvent dataclass: path: Path, event_type: str (created, modified, deleted)
2. Define a FileSystemWatcher class with: __init__(self, watch_path: Path, callback: Callable[[FileEvent], None], extensions: set[str] | None = None)
3. Implement start() / stop() lifecycle using watchdog.observers.Observer
4. Implement debouncing: buffer events for 500ms, coalesce same-path events, suppress delete+create sequences for atomic saves (FR-3.6)
5. Translate watchdog.FileSystemEvent objects into FileEvent instances
6. Filter by extension when extensions is provided

Tests: Unit tests with mock filesystem events, debouncing behavior, atomic save pattern handling, per-adapter isolation (two watchers on different paths).

Step 4: MessagesDomain

Files: src/context_library/domains/messages.py

Changes:

1. Implement MessagesDomain(BaseDomain) with chunk(content: NormalizedContent) -> list[Chunk]
2. Extract MessageMetadata from content.structural_hints or a dedicated metadata carrier on NormalizedContent
3. Build context_header as "{subject} — {sender}" for thread context
4. Handle quoted reply chains: detect lines starting with > and strip them if the quoted content has already been indexed (based on in_reply_to metadata)
5. Produce one chunk per message (messages are typically under 512 tokens)
6. For long messages, apply sentence-boundary splitting similar to NotesDomain._split_oversized_chunk()
7. Set chunk_type = ChunkType.STANDARD
8. Populate domain_metadata with MessageMetadata.model_dump()
9. Compute chunk_hash via compute_chunk_hash(content) excluding context_header

Tests: Unit tests for single message chunking, thread context headers, quoted reply stripping, long message splitting, metadata population, chunk_index sequencing.

Step 5: Rich Filesystem Adapter

Files: src/context_library/adapters/filesystem_rich.py (new file)

Dependencies: markitdown, pandoc (subprocess), _watching.py

Changes:

1. Implement RichFilesystemAdapter(BaseAdapter) with __init__(self, directory: Path, poll_strategy: PollStrategy = PollStrategy.PULL)
2. adapter_id: f"filesystem_rich:{resolved_path}" — distinct from FilesystemAdapter (FR-4.5)
3. domain: Domain.NOTES (FR-4.6)
4. fetch(): Walk directory, skip .md files (handled by FilesystemAdapter), convert supported files via MarkItDown, fall back to Pandoc for unsupported formats (FR-4.1, FR-4.2)
5. Populate StructuralHints with MIME type, file size, creation/modification timestamps, directory hierarchy (FR-4.3)
6. Include original MIME type/extension in metadata (FR-4.7)
7. When poll_strategy == PollStrategy.PUSH, instantiate a FileSystemWatcher from _watching.py (FR-4.4)

New dependencies in pyproject.toml:

[project.optional-dependencies]
rich-fs = ["markitdown"]

Pandoc is a system dependency, not a pip package.

Tests: Unit tests with fixture files (PDF, DOCX, XLSX), adapter_id uniqueness, MarkItDown conversion, Pandoc fallback, StructuralHints population.

Step 6: Obsidian Adapter

Files: src/context_library/adapters/obsidian.py (new file)

Dependencies: obsidiantools, _watching.py

Changes:

1. Implement ObsidianAdapter(BaseAdapter) with __init__(self, vault_path: Path, poll_strategy: PollStrategy = PollStrategy.PULL)
2. adapter_id: f"obsidian:{resolved_vault_path}"
3. domain: Domain.NOTES (FR-5.7)
4. fetch(): Walk vault directory for .md files, read each as raw markdown (no conversion — FR-5.6)
5. Use obsidiantools.Vault to extract backlinks, forward wikilinks, note connectivity (FR-5.3)
6. Parse per-note YAML frontmatter, tags, aliases from the note content (FR-5.2)
7. Store all extracted metadata in StructuralHints or as auxiliary metadata passed through to the domain chunker, ultimately landing in domain_metadata on each Chunk (FR-5.4)
8. When poll_strategy == PollStrategy.PUSH, instantiate a FileSystemWatcher from _watching.py scoped to vault path (FR-5.5)

New dependencies in pyproject.toml:

[project.optional-dependencies]
obsidian = ["obsidiantools"]

Tests: Unit tests with fixture vault directory, metadata extraction, wikilink graph, frontmatter parsing, push-mode watcher instantiation.

Step 7: Email Adapter

Files: src/context_library/adapters/email.py (new file)

Dependencies: httpx (for EmailEngine REST API calls), markitdown or html2text (for HTML→markdown)

Changes:

1. Implement EmailAdapter(BaseAdapter) with __init__(self, emailengine_url: str, account_id: str)
2. adapter_id: f"email:{account_id}"
3. domain: Domain.MESSAGES (FR-6.4)
4. fetch(): Query EmailEngine REST API for messages newer than last_fetched_at (FR-6.5), convert HTML bodies to markdown (FR-6.3)
5. Extract header fields: sender, recipients, subject, timestamp, message-ID, in-reply-to, thread-id (FR-6.2)
6. Yield NormalizedContent with MessageMetadata-compatible metadata in structural_hints or as supplementary metadata
7. Preserve thread relationships via thread_id and in_reply_to (FR-6.7)
8. No plaintext credential storage — EmailEngine manages auth (FR-6.6)

New dependencies in pyproject.toml:

[project.optional-dependencies]
email = ["httpx"]

Tests: Unit tests with mocked EmailEngine API responses, incremental fetch behavior, HTML-to-markdown conversion, metadata extraction.

Step 8: Poller

Files: src/context_library/scheduler/poller.py

Changes:

1. Implement Poller class with __init__(self, pipeline: IngestionPipeline, document_store: DocumentStore, tick_interval: float = 60.0)
2. register(adapter: BaseAdapter, domain_chunker: BaseDomain) — registers adapter/chunker pairs for polling (FR-1.6)
3. start() — spawns background thread running the tick loop (FR-1.5)
4. stop() — signals the thread to exit via threading.Event, joins (FR-1.5)
5. Tick loop: query document_store.get_sources_due_for_poll(), for each due source call pipeline.ingest(adapter, domain_chunker), update last_fetched_at (FR-1.1, FR-1.2, FR-1.3)
6. Per-source try/except with logging — failures do not propagate to other sources (FR-1.4)

Tests: Unit tests for registration, start/stop lifecycle, due-source detection, error isolation.

Step 9: Watcher

Files: src/context_library/scheduler/watcher.py

Changes:

1. Implement Watcher class with __init__(self, pipeline: IngestionPipeline)
2. register(adapter: BaseAdapter, domain_chunker: BaseDomain, file_watcher: FileSystemWatcher) — associates an adapter with its filesystem watcher
3. handle_webhook(source_ref: str, adapter: BaseAdapter, domain_chunker: BaseDomain) — entry point for push events, calls pipeline.ingest() scoped to the changed source (FR-2.2)
4. start() — starts all registered file watchers (FR-2.3)
5. stop() — stops all registered file watchers (FR-2.3)
6. Per-event try/except with logging — failures do not halt event processing (FR-2.5)

Tests: Unit tests for webhook routing, start/stop lifecycle, error isolation.

Dependency Summary

Step 1 (Models)          → no dependencies
Step 2 (DocumentStore)   → Step 1
Step 3 (_watching.py)    → Step 1 (PollStrategy)
Step 4 (MessagesDomain)  → Step 1 (MessageMetadata)
Step 5 (filesystem_rich) → Steps 1, 3
Step 6 (obsidian)        → Steps 1, 3
Step 7 (email)           → Steps 1, 4
Step 8 (Poller)          → Steps 1, 2
Step 9 (Watcher)         → Steps 1, 3

Steps 3, 4 can proceed in parallel after Step 1. Steps 5, 6, 7 can proceed in parallel after their respective dependencies. Steps 8, 9 can proceed in parallel after Step 2/3.

pyproject.toml Changes

[project.optional-dependencies]
email = ["httpx"]
rich-fs = ["markitdown"]
obsidian = ["obsidiantools"]
gmail = ["google-api-python-client", "google-auth-oauthlib"]  # unchanged
spotify = ["spotipy"]  # unchanged
todoist = ["todoist-api-python"]  # unchanged
dev = ["ruff", "mypy", "pytest", "watchdog"]

watchdog is added to dev dependencies. For production use, adapters using _watching.py should declare watchdog as a dependency via their optional group or it should be added to core dependencies since both filesystem_rich and obsidian adapters use it.

Files Created or Modified

File	Action	Component
src/context_library/storage/models.py	Modified	PollStrategy enum, MessageMetadata model
src/context_library/storage/document_store.py	Modified	register_source() params, get_sources_due_for_poll()
src/context_library/adapters/_watching.py	Created	Shared filesystem watcher
src/context_library/domains/messages.py	Modified (stub → impl)	MessagesDomain
src/context_library/adapters/filesystem_rich.py	Created	Rich filesystem adapter
src/context_library/adapters/obsidian.py	Modified (stub → impl)	Obsidian adapter
src/context_library/adapters/email.py	Created	Email adapter
src/context_library/scheduler/poller.py	Modified (stub → impl)	Poller scheduler
src/context_library/scheduler/watcher.py	Modified (stub → impl)	Watcher scheduler
pyproject.toml	Modified	New optional dependency groups

What Is NOT Changed

• BaseAdapter contract — no modifications needed
• BaseDomain contract — no modifications needed
• IngestionPipeline — called by schedulers as-is
• DocumentStore SQLite schema (schema.sql) — already provisioned for Phase 2
• Differ, Embedder, VectorStore — unchanged
• NotesDomain — unchanged
• FilesystemAdapter — unchanged
• retrieval/query.py — unchanged

---

Generated by Orchestrator Bot 🤖
Processed by the software_architect agent

[orchestrator-bot[bot]]

Work Breakdown Complete — created 9 sub-issue(s):

• Phase 1: Model Additions (PollStrategy + MessageMetadata) #85 Phase 1: Model Additions (PollStrategy + MessageMetadata)
• Phase 2: DocumentStore Scheduling Support #86 Phase 2: DocumentStore Scheduling Support
• Phase 3: Shared Filesystem Watcher (_watching.py) #87 Phase 3: Shared Filesystem Watcher (_watching.py)
• Phase 4: MessagesDomain Implementation #88 Phase 4: MessagesDomain Implementation
• Phase 5: Rich Filesystem Adapter #89 Phase 5: Rich Filesystem Adapter
• Phase 6: Obsidian Adapter #90 Phase 6: Obsidian Adapter
• Phase 7: Email Adapter #91 Phase 7: Email Adapter
• Phase 8: Poller Scheduler #92 Phase 8: Poller Scheduler
• Phase 9: Watcher Scheduler #93 Phase 9: Watcher Scheduler

[orchestrator-bot[bot]]

Pr Code Reviewer

---

I'll start by checking out the PR and understanding the changes, then run the comprehensive review.We're already on the PR branch feature/issue-83-mvp-phase-2. Let me verify this is PR #94 and get the PR details.This is a large PR with 21 files and ~6000 lines of changes. Now let me run the comprehensive review skill.

I'll run a comprehensive review of this PR. Let me start by identifying the changed files and then launch each specialized review agent sequentially.Let me first check if there's a CLAUDE.md with project guidelines.Now let me launch the review agents sequentially. Starting with the code-reviewer.Code review complete. Now launching the pr-test-analyzer agent.Test analysis complete. Now launching the comment-analyzer agent.Comment analysis complete. Now launching the silent-failure-hunter agent.Silent failure analysis complete. Now launching the type-design-analyzer agent.All five review agents have completed. Now I'll compile and format the consolidated findings.

PR Review Findings

Critical Issues

• Broken conditional import for EmailAdapter in __init__.py: /workspace/src/context_library/adapters/__init__.py:16 -- find_spec("context_library.adapters.email") checks whether the module file exists (always True), not whether the actual optional dependency httpx is installed. The conditional gate never gates anything. Should check find_spec("httpx") instead. (NEW)

• Unconditional import of ObsidianAdapter with optional dependencies: /workspace/src/context_library/adapters/__init__.py:7-8 -- ObsidianAdapter is imported unconditionally at the top level, but its dependencies (obsidiantools, frontmatter) are optional. Users installing the base package will get the class but instantiation will fail. Should use same conditional import pattern as EmailAdapter, but with find_spec("obsidiantools"). (NEW)

• Catch-all except Exception silently skips email messages: /workspace/src/context_library/adapters/email.py:147-149 -- Every error during message processing (including network failures, programming bugs, MemoryError) is caught, logged at WARNING level, and the message is silently skipped. If EmailEngine is down, all messages fail silently and fetch() yields nothing, indistinguishable from "no new emails." (NEW)

• Conversion failures logged at DEBUG level, hidden by silent fallback: /workspace/src/context_library/adapters/filesystem_rich.py:55-57 -- _convert_with_markitdown catches all Exception types and logs at DEBUG level (invisible in production). A broken MarkItDown installation silently degrades to Pandoc-only with zero visible indication. (NEW)

• Race condition in _flush_buffer / stop() can cause callbacks after stop returns: /workspace/src/context_library/adapters/_watching.py:128-146 -- Between _observer.join() completing and acquiring _buffer_lock, the debounce timer thread could fire _flush_buffer(). Callbacks may fire after stop() returns, violating the documented contract. (NEW)

• Watcher.register() overwrites private _callback attribute of FileSystemWatcher: /workspace/src/context_library/scheduler/watcher.py:59 -- Directly mutates file_watcher._callback, silently replacing the adapter's original callback (e.g., ObsidianAdapter._on_file_changed for vault cache invalidation). The adapter's callback will never fire after registration, causing stale data. (NEW)

• EmailEngine API response double-nesting may be incorrect: /workspace/src/context_library/adapters/email.py:174-177 -- Comment documents {"messages": {"messages": [...]}} but actual EmailEngine v1 API returns {"messages": [...]}. If the API returns a list at the first "messages" key, calling .get("messages", []) on a list will fail or return default, silently yielding zero messages. (NEW)

High Priority Issues

• Silent fabricated timestamps on stat failure in ObsidianAdapter: /workspace/src/context_library/adapters/obsidian.py:290-305 -- When stat() fails for any reason (including a catch-all except Exception), the code fabricates datetime.now() timestamps and file_size=0, which flow into stored chunks as if they were real file metadata. This is data corruption. (NEW)

• _parse_note returns empty string on failure, producing empty chunks: /workspace/src/context_library/adapters/obsidian.py:177-200 -- Failed note parsing returns ("", {}), and fetch() does not check for empty content. Empty-string chunks get hashed, stored, and embedded, polluting the vector store with contentless entries. (NEW)

• Poller catch-all with no retry, failure tracking, or escalation: /workspace/src/context_library/scheduler/poller.py:108-114 -- All pipeline errors are caught and logged with no retry mechanism, failure counter, or circuit breaker. Also, update_last_fetched_at is inside the same try block after ingest, so if ingest succeeds but the update fails, the source enters an infinite re-ingestion loop. (NEW)

• Watcher push events permanently lost on failure: /workspace/src/context_library/scheduler/watcher.py:78-83 -- Unlike the Poller which re-polls, push events are one-shot. A failed push event is logged and permanently discarded with no retry queue. The file change is lost until the next modification occurs. (NEW)

• Completely silent duplicate chunk skip with string-based error matching: /workspace/src/context_library/storage/document_store.py:308-316 -- Duplicate chunks are silently skipped with zero logging (not even DEBUG). The detection relies on string-matching SQLite error messages, which is fragile across SQLite versions. (NEW)

• New httpx client per HTTP request in EmailAdapter: /workspace/src/context_library/adapters/email.py:167-171 -- Every call to _fetch_messages() and _fetch_message_body() creates a new TCP connection. Fetching 100 messages means 101 separate connections. Should use an httpx.Client for connection pooling. (NEW)

• Poller first tick delayed by full tick_interval: /workspace/src/context_library/scheduler/poller.py:87-90 -- _run() uses Event.wait(timeout=tick_interval) before the first _tick(). With default 60s interval, a newly started poller waits 60 seconds before its first poll. (NEW)

• Poller stop() joins thread without timeout: /workspace/src/context_library/scheduler/poller.py:82-85 -- If _tick() hangs on a network call, stop() blocks indefinitely. Should add a join(timeout=...). (NEW)

Medium Priority Issues

• Misleading module docstring claims "watchfiles fallback": /workspace/src/context_library/adapters/_watching.py:1-6 -- Docstring says "Wraps watchdog with optional watchfiles fallback" but _start_watchfiles() raises NotImplementedError. No fallback exists. (NEW)

• _WatchdogHandler docstrings say "file/directory" but directories are filtered out: /workspace/src/context_library/adapters/_watching.py:232-247 -- Handler docstrings say "Handle file/directory creation" but every method starts with if not event.is_directory: return, explicitly ignoring directory events. (NEW)

• _strip_quoted_content inconsistent line/stripped handling: /workspace/src/context_library/domains/messages.py:31-37 -- Checks line.startswith(">") on raw (possibly indented) line but checks attribution pattern on stripped. Indented quoted lines like > text would not be stripped. (NEW)

• FileEvent.event_type is bare str instead of enum: /workspace/src/context_library/adapters/_watching.py:37-46 -- The critical invariant (event_type must be 'created', 'modified', or 'deleted') is only expressed in a comment. An EventType(str, Enum) would provide compile-time and runtime safety. (NEW)

• extra_metadata: dict[str, object] is a stringly-typed escape hatch: /workspace/src/context_library/storage/models.py:104 -- The normalizer-to-chunker contract is invisible to the type system. MessagesDomain.chunk() unpacks this dict as MessageMetadata(**meta_dict) with a type: ignore comment. Should be a typed discriminated union. (NEW)

• Mutable containers in frozen Pydantic models: /workspace/src/context_library/storage/models.py -- list[int], list[str], and set[Sha256Hash] in frozen models can still be mutated in-place via .append(), .add(), etc. Should use tuple and frozenset. (NEW)

• Defensive .get() chains mask EmailEngine API contract violations: /workspace/src/context_library/adapters/email.py:175-177,200 -- If EmailEngine changes its response schema, the adapter silently returns zero messages or empty bodies instead of raising an error. (NEW)

• Timestamp parse failure silently passes invalid data: /workspace/src/context_library/adapters/email.py:245-250 -- When parsing fails, the raw unparsed string passes through to MessageMetadata.timestamp, triggering Pydantic validation failure, which is caught by the catch-all in fetch(), silently dropping the message. (NEW)

• Dead code in timestamp parsing guard: /workspace/src/context_library/adapters/email.py:243-250 -- The guard if not timestamp.endswith("Z") and "T" not in timestamp already ensures no "Z", so timestamp.replace("Z", "+00:00") inside the block is dead code. (NEW)

• _flush_buffer holds lock during callback execution, failed events permanently lost: /workspace/src/context_library/adapters/_watching.py:200-214 -- Lock is held for all callbacks (blocking new events). If a callback fails, the event is logged but cleared from the buffer with no retry. (NEW)

• PollStrategy re-exported from _watching.py.__all__: /workspace/src/context_library/adapters/_watching.py:259 -- A storage model enum is re-exported from a filesystem-watching module, creating confusing import paths. Should be imported from storage.models directly. (NEW)

• MessageMetadata missing cross-field validation: /workspace/src/context_library/storage/models.py:107-131 -- is_thread_root=True with in_reply_to="some-id" is logically contradictory but constructible. Empty strings accepted for identity fields thread_id, message_id, sender. (NEW)

• Unguarded Pydantic construction in MessagesDomain.chunk(): /workspace/src/context_library/domains/messages.py:86 -- MessageMetadata(**meta_dict) with type: ignore produces opaque ValidationError on contract violations. Should wrap with domain-specific error message. (NEW)

• Silent pass on ImportError hides transitive dependency failures: /workspace/src/context_library/adapters/_watching.py:22-34 and others -- except ImportError: pass catches transitive import failures (e.g., broken C extensions), silently disabling features. (NEW)

• RichFilesystemAdapter docstring says "supported non-markdown formats" but accepts all files: /workspace/src/context_library/adapters/filesystem_rich.py:107-109 -- Docstring implies a defined set of formats, but actual behavior attempts conversion of every non-.md file. (NEW)

• Missing test: DocumentStore.retire_chunks RuntimeError for non-existent chunk: /workspace/src/context_library/storage/document_store.py:346-354 -- The data integrity guard raising RuntimeError when chunk doesn't exist has no test coverage. (NEW)

• Missing test: DocumentStore.update_last_fetched_at RuntimeError for non-existent source: /workspace/src/context_library/storage/document_store.py:735-738 -- The guard raising RuntimeError for missing sources is untested. (NEW)

• Missing test: ObsidianAdapter._on_file_changed vault cache invalidation: /workspace/src/context_library/adapters/obsidian.py:414-433 -- The vault cache invalidation (self._vault = None) that is the linchpin of push-mode correctness has no test. (NEW)

Low Priority / Nice-to-Have

• FileSystemWatcher missing input validation: /workspace/src/context_library/adapters/_watching.py:49-87 -- debounce_ms accepts zero/negative values, watch_path not validated as existing directory, extensions elements not validated for leading dot. (NEW)

• MessagesDomain.hard_limit is public and mutable: /workspace/src/context_library/domains/messages.py:50 -- Should be a read-only property or underscore-prefixed. Also accepts zero/negative values. (NEW)

• DiffResult missing converse changed=True check: /workspace/src/context_library/storage/models.py:233-239 -- changed=True with all-empty hash sets is allowed but semantically dubious. (NEW)

• Watcher.start()/stop() no per-watcher error isolation: /workspace/src/context_library/scheduler/watcher.py:85-100 -- One watcher failing to start prevents all subsequent watchers from starting. Should wrap each in try-except. (NEW)

• Redundant step-by-step inline comments in adapters: /workspace/src/context_library/adapters/email.py:112-149, /workspace/src/context_library/adapters/obsidian.py:341-399, /workspace/src/context_library/adapters/filesystem_rich.py:178-248 -- Comments like "Fetch messages from EmailEngine" immediately before _fetch_messages() restate what the method name already says. (NEW)

• st_ctime platform behavior undocumented in RichFilesystemAdapter: /workspace/src/context_library/adapters/filesystem_rich.py:203 -- Unlike ObsidianAdapter which documents that st_ctime is not creation time on Linux, RichFilesystemAdapter has only a terse inline comment. (NEW)

• Time-dependent tests using time.sleep() may be flaky on slow CI: /workspace/tests/adapters/test_watching.py -- Tests use fixed sleep durations (e.g., 0.2s for 100ms debounce). Assertions are flexible but timing-dependent. (NEW)

• StructuralHints.file_size_bytes accepts negative integers: /workspace/src/context_library/storage/models.py:102 -- No non-negativity validation. (NEW)

• Pandoc subprocess output not size-limited: /workspace/src/context_library/adapters/filesystem_rich.py:70-71 -- A maliciously crafted file could produce very large markdown output consuming memory. (NEW)

Clean Areas

• Storage models (models.py): DiffResult has excellent cross-field validation with disjointness checks and flag consistency enforcement. Sha256Hash validation with regex is well-implemented. PollStrategy enum is clean and minimal.
• Document store SQL safety: document_store.py uses parameterized queries consistently. Foreign key enforcement and schema constraints are thorough.
• Test infrastructure and quality: All test files have descriptive names, docstrings, and well-organized test classes. Mocking is appropriately targeted. The test_models.py coverage of Pydantic validation is thorough. The test_document_store.py helper methods keep tests DRY.
• Watcher thread safety: _watching.py shows excellent thread-safety discipline with consistent _buffer_lock usage and proper timer cancellation.
• Atomic-save coalescing: The debounce logic correctly handles vim/emacs save patterns (deleted+created -> modified), which is a sophisticated and practically useful feature.
• Obsidian platform documentation: _extract_timestamps() docstring's note about st_ctime behavior differences between Linux and macOS/Windows is a model example of documenting non-obvious platform behavior.
• Error isolation in Poller: Per-source failure isolation ("one failing source doesn't prevent others from being polled") is well-designed and clearly documented.
• Dockerfile changes: Clean build configuration with proper dependency installation, PYTHONPATH setup, and verification of installed packages.The comprehensive review is complete. The PR adds substantial new functionality (adapters, domain models, schedulers, storage) with generally good architecture and thorough test coverage. The most urgent issues to address before merge are:

1. The broken conditional imports in __init__.py that don't actually gate on optional dependencies
2. The callback overwrite pattern in Watcher.register() that silently disables adapter callbacks (vault cache invalidation)
3. The pervasive catch-all except Exception pattern that silently drops data (emails, file events, chunks) with no retry or escalation
4. The DEBUG-level logging for conversion failures that makes production issues invisible

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR and the roadmap to verify implementation against requirements.Now I have enough information to perform the verification. Let me check the roadmap requirements against what was implemented:I now have a comprehensive picture of what was implemented versus what the roadmap specifies for Phase 2.

---

Parent Issue Requirements Verification

Gaps Found

• Email Adapter — OAuth/IMAP Integration: The roadmap specifies that for direct IMAP/SMTP access, email-oauth2-proxy should be used for transparent OAuth 2.0 (SASL XOAUTH2) proxying for providers including Gmail, Outlook, Yahoo, and Fastmail, with support for headless deployments and secrets-manager-backed credential storage, and that "Credentials are never stored in plaintext; short-lived access tokens and refresh tokens are managed by the OAuth layer." The implemented email.py adapter targets the EmailEngine REST API only and does not implement or integrate email-oauth2-proxy for direct IMAP/SMTP OAuth proxying. However, the roadmap presents both EmailEngine and email-oauth2-proxy as alternative approaches ("For direct IMAP/SMTP access…"), so this is a partial gap depending on whether both pathways were intended or just the EmailEngine route.

Deviations

None found.

Verified

• Scheduler: Poller (src/context_library/scheduler/poller.py) — Implemented as specified with background polling, per-source intervals, and pull-strategy routing.
• Scheduler: Watcher (src/context_library/scheduler/watcher.py) — Implemented as specified, routing filesystem push events through handle_webhook.
• Shared Filesystem Watcher (src/context_library/adapters/_watching.py) — Implemented as specified with debouncing, atomic-save detection, watchdog/watchfiles backend with fallback, SourceRef mapping, and PollStrategy.PUSH routing. Per-adapter instantiation (no shared singleton) matches the spec.
• Rich Filesystem Adapter (src/context_library/adapters/filesystem_rich.py) — Implemented with MarkItDown for PDF/Office/HTML/image/audio conversion and Pandoc as fallback. Uses _watching.py. Filesystem metadata (MIME type, size, timestamps, directory hierarchy) captured as chunk metadata.
• Obsidian Adapter (src/context_library/adapters/obsidian.py) — Implemented using _watching.py for filesystem events. Uses obsidiantools for vault-level graph analytics and python-frontmatter for per-note YAML frontmatter, tags, aliases, and wikilink metadata stored alongside chunks.
• Email Adapter (src/context_library/adapters/email.py) — EmailEngine REST API integration implemented as the primary specified mechanism.
• Messages Domain (src/context_library/domains/messages.py) — Implemented with thread/author preservation, quoted-reply stripping, and token-based splitting for long messages.
• Test coverage — Test files exist for all Phase 2 components: test_email.py, test_filesystem_rich.py, test_obsidian.py, test_watching.py, test_messages.py, test_poller.py, test_watcher.py.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the business requirements. Let me fetch the PR details first.Now let me dig deeper into specific requirements that may have gaps or deviations, particularly around the watchfiles backend stub and some other details.Now I have a thorough understanding of the implementation. Let me analyze the key gaps and deviations against the requirements.

Business Analyst Output Verification

Gaps Found

• FR-3.1 watchfiles Backend Not Implemented: The requirement states the module must support "at least one of watchdog or watchfiles" behind a unified interface. While watchdog is supported, the watchfiles fallback path is broken at runtime: _start_watchfiles() and _stop_watchfiles() both raise NotImplementedError. If watchdog is unavailable but watchfiles is installed, the module reports HAS_WATCHFILES = True and claims to support it, but calling start() raises an uncaught NotImplementedError. The requirement is met for the watchdog path, but the fallback interface is non-functional.

• FR-3.3 / FR-3.4 — Events Not Mapped to PollStrategy.PUSH: The requirement states _watching.py must translate backend-specific filesystem events into SourceRef values compatible with fetch(source_ref), and must map events to PollStrategy.PUSH so the scheduler routes them through handle_webhook(). The FileEvent dataclass carries a path: Path but no PollStrategy field. There is no poll_strategy attribute on FileEvent set to PollStrategy.PUSH. The watcher closure in watcher.py converts the path to a string and passes it as source_ref, but the mapping to PollStrategy.PUSH for routing purposes is absent from the event object itself.

• FR-4.3 — StructuralHints Missing original_format/converted file format field: FR-4.7 requires the adapter to include the converted file format as metadata on each NormalizedContent item (i.e., original MIME type or extension). The mime_type is present in extra_metadata, so FR-4.7 is covered. However, FR-4.3 also requires MIME type in StructuralHints — it is placed in extra_metadata rather than as a top-level StructuralHints field. The spec says "MIME type, file size in bytes, creation timestamp, modification timestamp, and directory hierarchy" — file_size_bytes and modified_at are top-level fields, but mime_type, created_at, and directory_hierarchy are inside extra_metadata. This is an architectural placement deviation rather than a complete gap since the data is present.

• FR-5.4 — Obsidian domain_metadata stored in StructuralHints.extra_metadata, not Chunk.domain_metadata: The requirement states metadata must be stored "in domain_metadata on each Chunk". The ObsidianAdapter places tags, aliases, wikilinks, and backlinks in StructuralHints.extra_metadata on NormalizedContent. There is no ObsidianDomain (or equivalent domain class) that reads this and populates Chunk.domain_metadata at chunk time. The NotesDomain (existing) would need to propagate this to Chunk.domain_metadata for the requirement to be satisfied — this is not verified in the code reviewed.

• FR-6.1 — Email Provider Indirection via EmailEngine: The requirement says the adapter must support "at least one email provider (Gmail, Outlook, or IMAP-compatible server)". The implementation targets EmailEngine's proprietary REST API, which is a third-party proxy service — not a direct Gmail, Outlook, or IMAP integration. While EmailEngine supports those backends, users must run and configure a separate EmailEngine server. This may not satisfy the spirit of "direct provider support" as specified, though it could be considered compliant if EmailEngine qualifies as an IMAP-compatible server proxy.

• FR-7.5 — MessagesDomain not registered under Domain.MESSAGES in a domain registry: The requirement states "The domain must be registered under Domain.MESSAGES." The MessagesDomain class returns Domain.MESSAGES from no explicit property (no domain property is visible in the class). There is no domain registry where MessagesDomain is formally registered against Domain.MESSAGES. The existing NotesDomain pattern should be checked to see if there is a registration mechanism — this may be a gap.

Deviations

• FR-1.2 — Poller calls pipeline.ingest(adapter, chunker) rather than adapter.fetch(source_ref) directly: The requirement states the poller must "invoke adapter.fetch(source_ref) for each registered pull-based source." The actual implementation delegates to self._pipeline.ingest(adapter, chunker) without passing a source_ref. The source_ref from the document store (source["source_id"]) is retrieved but not forwarded to fetch(). This means incremental ingestion using source_ref as a last_fetched_at cursor (as used by EmailAdapter) would not work through the poller.

• FR-3.3 — FileEvent.path used as source_ref string, not a typed SourceRef: The spec requires events to be translated into SourceRef values compatible with the pipeline's fetch(source_ref) contract. The implementation uses str(event.path) as a generic string rather than a typed SourceRef. While functionally equivalent in Python's duck-typed system, the typing contract is not enforced.

• FR-5.4 / FR-5.3 — Graph metadata stored in extra_metadata, not in Chunk.domain_metadata: Specified as "store extracted metadata ... in domain_metadata on each Chunk". The data is stored in StructuralHints.extra_metadata on NormalizedContent, which is an intermediate step before chunking. The downstream chunker (NotesDomain or a dedicated ObsidianDomain) must explicitly copy this to Chunk.domain_metadata — that propagation step is not implemented or verified.

Verified

• FR-1.1: Poller reads poll_interval_sec and last_fetched_at per source from document_store.get_sources_due_for_poll()
• FR-1.3: Poller calls update_last_fetched_at(source["source_id"]) after successful fetch
• FR-1.4: Per-source failures isolated via try/except Exception in _tick()
• FR-1.5: Poller implements start() and stop() lifecycle methods
• FR-1.6: Poller implements register(adapter, domain_chunker) method
• FR-1.7: Poller queries only poll_strategy = "pull" sources via get_sources_due_for_poll()
• FR-2.2: Watcher implements handle_webhook() as a distinct path from polling
• FR-2.3: Watcher implements start() and stop() lifecycle methods
• FR-2.5: Per-event failures isolated via try/except Exception in handle_webhook()
• FR-3.2: Each adapter instantiates its own FileSystemWatcher instance — no shared singleton
• FR-3.5: PollStrategy enum with PUSH and PULL values defined in models.py and re-exported from _watching.py
• FR-3.6: Atomic-save pattern (deleted + created → modified) handled in _on_raw_event() via debounce buffer coalescing
• FR-4.1: RichFilesystemAdapter uses MarkItDown as primary converter
• FR-4.2: Falls back to Pandoc via subprocess.run(["pandoc", ...]) when MarkItDown returns None
• FR-4.3: StructuralHints populated with file_size_bytes, modified_at, and extra_metadata containing mime_type, created_at, directory_hierarchy
• FR-4.4: RichFilesystemAdapter creates FileSystemWatcher when PollStrategy.PUSH
• FR-4.5: adapter_id returns "filesystem_rich:{path}", distinct from base FilesystemAdapter
• FR-4.6: RichFilesystemAdapter.domain returns Domain.NOTES
• FR-4.7: mime_type stored in extra_metadata on each NormalizedContent
• FR-5.1: ObsidianAdapter.fetch() treats each vault .md note as a source ref
• FR-5.2: Uses python-frontmatter to parse YAML frontmatter, tags, and aliases
• FR-5.3: Uses obsidiantools.api (otools.Vault().connect()) for vault graph, extracting forward wikilinks and backlinks
• FR-5.5: ObsidianAdapter creates FileSystemWatcher when PollStrategy.PUSH
• FR-5.6: Vault notes passed through as-is without format conversion
• FR-5.7: ObsidianAdapter.domain returns Domain.NOTES
• FR-6.2: _extract_message_metadata() extracts sender, recipients, subject, timestamp, messageId, inReplyTo
• FR-6.3: HTML bodies converted to markdown via html2text.HTML2Text()
• FR-6.4: EmailAdapter.domain returns Domain.MESSAGES
• FR-6.5: Incremental ingestion via source_ref as since timestamp passed to search[since] API param
• FR-6.6: No credentials stored — adapter only stores emailengine_url and account_id; auth handled externally
• FR-6.7: thread_id and in_reply_to preserved in MessageMetadata
• FR-7.1: MessagesDomain.chunk() returns list[Chunk] with sequential chunk_index values
• FR-7.2: domain_metadata populated from meta.model_dump() containing all MessageMetadata fields
• FR-7.3: context_header set to "{subject} — {sender}" for thread context
• FR-7.4: _strip_quoted_content() removes > quoted lines and On ... wrote: attribution lines
• FR-7.6: MessageMetadata defined as a frozen Pydantic BaseModel in models.py consistent with existing conventions
• FR-3.5 (models.py): PollStrategy enum added to models.py alongside Domain and ChunkType

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the architecture specifications. Let me fetch the PR details.Now I have comprehensive details about what was implemented. Let me compare against the architecture specifications systematically.Now I have enough information to produce the complete verification report. Let me analyze the gaps and deviations against the architecture spec.

Software Architect Output Verification

Gaps Found

• _watching.py does not use PollStrategy.PUSH in event routing: The architecture spec (FR-3.5) states the module should "translate OS-level filesystem events into SourceRef values and map them to PollStrategy.PUSH." The implementation only re-exports PollStrategy as a convenience — no event is actually tagged with PollStrategy.PUSH anywhere in _watching.py or in the event routing path through Watcher.handle_webhook().

• _start_watchfiles() is a stub raising NotImplementedError: ADR-P2-02 specifies implementing watchdog as the primary backend with the import-time fallback to watchfiles if watchdog is not installed. The fallback path is declared as HAS_WATCHFILES = True but _start_watchfiles() raises NotImplementedError, meaning if only watchfiles is installed the watcher silently fails to start rather than functioning as a fallback. The ADR states "Backend selection uses a simple import-time check: try watchdog, fall back to watchfiles" — this fallback is not actually functional.

• Watcher.register() overwrites the adapter's existing _callback: The architecture specifies the Watcher receives push events from _watching.py and routes them through handle_webhook(). The implementation achieves this by assigning to file_watcher._callback directly (bypassing encapsulation), which silently overwrites any callback set in the adapter constructor (e.g., RichFilesystemAdapter._on_file_changed). The adapter's own _on_file_changed is replaced, meaning the adapter's push callback (which logs events and is intended for framework integration) is never invoked when Watcher is used.

• EmailAdapter missing max_initial_messages / configurable initial-fetch bound named in spec: The architecture explicitly calls for max_initial_messages as the parameter name ("Initial ingestion of a large mailbox should be bounded by a configurable max_initial_messages parameter"). The implementation uses max_messages: int = 100 instead. While functionally equivalent, the spec is explicit about the parameter name.

• RichFilesystemAdapter._on_file_changed is a placeholder only: The architecture spec states the Watcher receives push events and routes them through handle_webhook() into the pipeline. The RichFilesystemAdapter._on_file_changed method only logs the event and explicitly notes "re-ingestion requires framework integration" — it does not trigger re-ingestion. The spec requires the push path to result in actual pipeline ingestion.

Deviations

• Poller._run() does NOT tick immediately on start: The architecture spec (ADR-P2-01) describes a tick-based loop and the implementation uses self._stop_event.wait(timeout=self._tick_interval) before the first tick, meaning the poller waits one full tick interval before doing any work. This is an implementation choice not explicitly addressed in the spec but is a behavioral deviation from the natural expectation of "starts polling immediately."

• register_source() uses INSERT OR IGNORE but does not update poll_strategy/poll_interval_sec on re-registration: The architecture notes DocumentStore.register_source() currently hardcodes poll_strategy = "pull" and requires "a parameter addition to accept the strategy from the caller." The parameter was added correctly. However, because the SQL remains INSERT OR IGNORE, if a source was previously registered with a different strategy, the new strategy is silently ignored on re-registration. The spec intended that the strategy be properly persisted.

Verified

• PollStrategy enum added to models.py as str, Enum with PUSH, PULL, WEBHOOK values — matches ADR-P2-05 and schema CHECK constraint
• MessageMetadata added to models.py as a frozen Pydantic model with @field_validator on timestamp, all specified fields present — matches ADR-P2-04 and Pattern 3
• _watching.py implements FileSystemWatcher with watchdog as primary backend, 500ms debounce, atomic-save coalescing (delete+create → modified), extension filtering — matches ADR-P2-02 and architecture spec
• RichFilesystemAdapter implements BaseAdapter contract: deterministic adapter_id as "filesystem_rich:{resolved_path}", domain = Domain.NOTES, yields NormalizedContent with StructuralHints — matches Pattern 1
• ObsidianAdapter updated to use shared FileSystemWatcher from _watching.py, adapter_id = "obsidian:{vault_path}", domain = Domain.NOTES — matches spec
• EmailAdapter implements BaseAdapter contract: adapter_id = "email:{account_id}", domain = Domain.MESSAGES, per-message error isolation — matches Pattern 1, ADR-P2-03, Pattern 4
• MessagesDomain.chunk() produces Chunk objects with chunk_hash from compute_chunk_hash(), sequential chunk_index starting at 0, domain_metadata via meta.model_dump(), ChunkType.STANDARD — matches Pattern 2
• Poller uses threading.Thread + threading.Event for start()/stop() lifecycle — matches ADR-P2-01
• Poller._tick() provides per-source error isolation (catches exceptions, continues loop) — matches Pattern 4
• DocumentStore.register_source() extended with poll_strategy and poll_interval_sec parameters — matches architecture note at document_store.py:155
• DocumentStore.update_last_fetched_at() added — matches Component Reuse table
• DocumentStore.get_sources_due_for_poll() added — supports Poller's _tick() method
• No SQLite schema migration required — confirmed, schema already had poll_strategy, poll_interval_sec, last_fetched_at columns
• pyproject.toml optional dependency groups added: rich-fs, obsidian, email — matches spec
• Full test coverage across all new components with per-event/per-source error isolation tests

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

Groups:

• Item:

  • Name: Adapter Conditional Imports
  • Severity: critical
  • Findings: - Broken conditional import for EmailAdapter: /workspace/src/context_library/adapters/__init__.py:16 — find_spec("context_library.adapters.email") checks whether the module file exists (always True in an installed package), not whether the actual optional dependency httpx is installed. The conditional gate never gates anything. Should check find_spec("httpx") instead.

• Unconditional import of ObsidianAdapter with optional dependencies: /workspace/src/context_library/adapters/__init__.py:7-8 — ObsidianAdapter is imported unconditionally at the top level, but its dependencies (obsidiantools, frontmatter) are optional. Users installing the base package will get an ImportError or a class whose instantiation fails. Should use the same conditional import pattern gated on find_spec("obsidiantools").

• Item:

  • Name: Watcher / Scheduler Push-Event Pipeline
  • Severity: critical
  • Findings: - Watcher.register() overwrites adapter's _callback, disabling adapter-internal callbacks: /workspace/src/context_library/scheduler/watcher.py:59 — Directly mutates file_watcher._callback, silently replacing the adapter's original callback (e.g., ObsidianAdapter._on_file_changed for vault cache invalidation). After registration, the adapter's own push callback never fires, causing stale vault data.

• Race condition in _flush_buffer / stop() can fire callbacks after stop returns: /workspace/src/context_library/adapters/_watching.py:128-146 — Between _observer.join() completing and acquiring _buffer_lock, the debounce timer thread could fire _flush_buffer(). Callbacks may execute after stop() returns, violating the documented lifecycle contract.

• Watcher push events permanently lost on failure: /workspace/src/context_library/scheduler/watcher.py:78-83 — Unlike the Poller which re-polls, push events are one-shot. A failed push event is logged and permanently discarded with no retry queue. The file change is lost until the next modification occurs.

• _flush_buffer holds lock during callback execution; failed events permanently lost: /workspace/src/context_library/adapters/_watching.py:200-214 — Lock is held for all callbacks (blocking new event buffering). If a callback raises, the event is logged but cleared from the buffer with no retry.

• Item:

  • Name: watchfiles Backend
  • Severity: critical
  • Findings: - _start_watchfiles() is a non-functional stub: /workspace/src/context_library/adapters/_watching.py — _start_watchfiles() and _stop_watchfiles() both raise NotImplementedError. If watchdog is unavailable but watchfiles is installed, the module reports HAS_WATCHFILES = True but start() raises an uncaught error. The requirement (FR-3.1 / ADR-P2-02) specifies at least one of watchdog or watchfiles must work behind a unified interface; the fallback path is broken.

• Item:

  • Name: Email Adapter
  • Severity: critical
  • Findings: - Catch-all except Exception silently skips email messages: /workspace/src/context_library/adapters/email.py:147-149 — Every error during message processing (including network failures, programming bugs) is caught, logged at WARNING, and the message is silently skipped. If EmailEngine is down, fetch() yields nothing, indistinguishable from "no new emails."

• EmailEngine API response double-nesting may be incorrect: /workspace/src/context_library/adapters/email.py:174-177 — Comment documents {"messages": {"messages": [...]}} but if the actual API returns {"messages": [...]}, calling .get("messages", []) on a list will fail silently, yielding zero messages.

• New httpx client created per HTTP request: /workspace/src/context_library/adapters/email.py:167-171 — Every call to _fetch_messages() and _fetch_message_body() creates a new TCP connection. Fetching 100 messages means 101 separate connections. Should use an httpx.Client session for connection pooling.

• Defensive .get() chains mask API contract violations: /workspace/src/context_library/adapters/email.py:175-177,200 — If EmailEngine changes its response schema, the adapter silently returns zero messages or empty bodies instead of raising an error.

• Timestamp parse failure silently drops the entire message: /workspace/src/context_library/adapters/email.py:245-250 — When parsing fails, the raw unparsed string triggers a Pydantic ValidationError, which is caught by the outer catch-all in fetch(), silently dropping the message.

• Dead code in timestamp parsing guard: /workspace/src/context_library/adapters/email.py:243-250 — The guard if not timestamp.endswith("Z") and "T" not in timestamp already ensures no "Z", so timestamp.replace("Z", "+00:00") inside the block is dead code.

• Item:

  • Name: Obsidian Adapter
  • Severity: high
  • Findings: - Silent fabricated timestamps on stat failure: /workspace/src/context_library/adapters/obsidian.py:290-305 — When stat() fails (catch-all except Exception), the code fabricates datetime.now() timestamps and file_size=0, which flow into stored chunks as if they were real file metadata. This is silent data corruption.

• _parse_note returns empty string on failure, producing empty chunks: /workspace/src/context_library/adapters/obsidian.py:177-200 — Failed note parsing returns ("", {}), and fetch() does not check for empty content. Empty-string chunks get hashed, stored, and embedded, polluting the vector store.

• Graph metadata stored in extra_metadata, not Chunk.domain_metadata: /workspace/src/context_library/adapters/obsidian.py — FR-5.4 requires metadata (tags, aliases, wikilinks, backlinks) be stored in domain_metadata on each Chunk. The implementation places it in StructuralHints.extra_metadata on NormalizedContent, and no downstream domain class propagates it to Chunk.domain_metadata.

• Item:

  • Name: Rich Filesystem Adapter
  • Severity: high
  • Findings: - Conversion failures logged at DEBUG level, hidden by silent fallback: /workspace/src/context_library/adapters/filesystem_rich.py:55-57 — _convert_with_markitdown catches all Exception types and logs at DEBUG level (invisible in production). A broken MarkItDown installation silently degrades to Pandoc-only with zero visible indication.

• _on_file_changed is a placeholder that does not trigger re-ingestion: /workspace/src/context_library/adapters/filesystem_rich.py — The architecture spec requires push events to result in pipeline ingestion. The method only logs the event and notes "re-ingestion requires framework integration" — no actual re-ingestion occurs.

• Item:

  • Name: Poller
  • Severity: high
  • Findings: - Catch-all with no retry, failure tracking, or escalation: /workspace/src/context_library/scheduler/poller.py:108-114 — All pipeline errors are caught and logged with no retry or circuit breaker. update_last_fetched_at is inside the same try block after ingest, so if ingest succeeds but the update fails, the source enters an infinite re-ingestion loop.

• _run() does not pass source_ref to adapter.fetch(): /workspace/src/context_library/scheduler/poller.py — FR-1.2 requires the poller to invoke adapter.fetch(source_ref) for each source. The implementation delegates to self._pipeline.ingest(adapter, chunker) without forwarding source_ref. Incremental ingestion (e.g., EmailAdapter's since parameter) does not work through the poller.

• First tick delayed by full tick_interval: /workspace/src/context_library/scheduler/poller.py:87-90 — _run() uses Event.wait(timeout=tick_interval) before the first _tick(). With default 60s interval, a newly started poller waits 60 seconds before its first poll.

• stop() joins thread without timeout: /workspace/src/context_library/scheduler/poller.py:82-85 — If _tick() hangs on a network call, stop() blocks indefinitely.

• Item:

  • Name: Document Store
  • Severity: high
  • Findings: - Silent duplicate chunk skip with fragile string-based error matching: /workspace/src/context_library/storage/document_store.py:308-316 — Duplicate chunks are silently skipped with zero logging. Detection relies on string-matching SQLite error messages ("UNIQUE constraint failed"), which is fragile across SQLite versions and locales.

• register_source() uses INSERT OR IGNORE, does not update strategy on re-registration: /workspace/src/context_library/storage/document_store.py — If a source was previously registered with a different poll_strategy or poll_interval_sec, the new values are silently ignored on re-registration. The architecture intended that the strategy be properly persisted.

• Item:

  • Name: Messages Domain
  • Severity: medium
  • Findings: - _strip_quoted_content inconsistent line/stripped handling: /workspace/src/context_library/domains/messages.py:31-37 — Checks line.startswith(">") on the raw (possibly indented) line but checks the attribution pattern on stripped. Indented quoted lines like > text would not be stripped.

• Unguarded Pydantic construction in chunk(): /workspace/src/context_library/domains/messages.py:86 — MessageMetadata(**meta_dict) with type: ignore produces opaque ValidationError on contract violations. Should wrap with a domain-specific error message.

• MessagesDomain not registered under Domain.MESSAGES in a domain registry: /workspace/src/context_library/domains/messages.py — FR-7.5 requires the domain to be registered under Domain.MESSAGES. No domain registry or formal registration mechanism is present.

• Item:

  • Name: Storage Models (Type Design)
  • Severity: medium
  • Findings: - FileEvent.event_type is bare str instead of enum: /workspace/src/context_library/adapters/_watching.py:37-46 — The critical invariant (event_type must be 'created', 'modified', or 'deleted') is only expressed in a comment. An EventType(str, Enum) would provide compile-time and runtime safety.

• extra_metadata: dict[str, object] is a stringly-typed escape hatch: /workspace/src/context_library/storage/models.py:104 — The normalizer-to-chunker contract is invisible to the type system. MessagesDomain.chunk() unpacks this dict as MessageMetadata(**meta_dict) with a type: ignore comment.

• Mutable containers in frozen Pydantic models: /workspace/src/context_library/storage/models.py — list[int], list[str], and set[Sha256Hash] in frozen models can still be mutated in-place via .append(), .add(), etc. Should use tuple and frozenset.

• MessageMetadata missing cross-field validation: /workspace/src/context_library/storage/models.py:107-131 — is_thread_root=True with in_reply_to="some-id" is logically contradictory but constructible. Empty strings accepted for identity fields thread_id, message_id, sender.

• Item:

  • Name: Test Coverage Gaps
  • Severity: medium
  • Findings: - Missing test: DocumentStore.retire_chunks RuntimeError for non-existent chunk: /workspace/src/context_library/storage/document_store.py:346-354 — The data integrity guard raising RuntimeError when chunk doesn't exist has no test coverage.

• Missing test: DocumentStore.update_last_fetched_at RuntimeError for non-existent source: /workspace/src/context_library/storage/document_store.py:735-738 — The guard raising RuntimeError for missing sources is untested.

• Missing test: ObsidianAdapter._on_file_changed vault cache invalidation: /workspace/src/context_library/adapters/obsidian.py:414-433 — The vault cache invalidation (self._vault = None) that is the linchpin of push-mode correctness has no test.

• Item:

  • Name: Misleading Documentation / Comments
  • Severity: medium
  • Findings: - Misleading module docstring claims "watchfiles fallback": /workspace/src/context_library/adapters/_watching.py:1-6 — Docstring says "Wraps watchdog with optional watchfiles fallback" but _start_watchfiles() raises NotImplementedError. No fallback exists.

• _WatchdogHandler docstrings say "file/directory" but directories are filtered out: /workspace/src/context_library/adapters/_watching.py:232-247 — Handler docstrings say "Handle file/directory creation" but every method starts with if not event.is_directory: return, explicitly ignoring directory events.

• RichFilesystemAdapter docstring says "supported non-markdown formats" but accepts all files: /workspace/src/context_library/adapters/filesystem_rich.py:107-109 — Docstring implies a defined set of formats, but actual behavior attempts conversion of every non-.md file.
  Filtered Out:

• FR-4.3 StructuralHints placement of mime_type/created_at in extra_metadata vs top-level fields — architectural placement deviation; data is present and accessible, not a functional gap

• FR-3.3/FR-3.4 events not tagged with PollStrategy.PUSH on the event object itself — merged into Watcher/Scheduler group; routing works via watcher.py closure without event-level tagging

• FR-6.1 Email provider indirection via EmailEngine — EmailEngine is a valid IMAP-compatible proxy per the spec's alternative approaches; not a gap

• FR-3.3 FileEvent.path used as string rather than typed SourceRef — style/typing preference, not a functional gap

• EmailAdapter max_messages vs max_initial_messages parameter naming — cosmetic naming deviation, functionality is equivalent

• PollStrategy re-exported from _watching.py.all — import path convenience, not a bug

• MessagesDomain.hard_limit is public and mutable — style preference, not a spec requirement

• DiffResult missing converse changed=True check — low severity edge case not required by spec

• Watcher.start()/stop() no per-watcher error isolation — low priority defensive coding suggestion

• Redundant step-by-step inline comments in adapters — style preference, not actionable

• st_ctime platform behavior undocumented in RichFilesystemAdapter — documentation nicety, not a spec requirement

• Time-dependent tests using time.sleep() may be flaky on slow CI — speculative concern, tests currently pass

• StructuralHints.file_size_bytes accepts negative integers — edge case validation, not required by spec

• Pandoc subprocess output not size-limited — speculative security concern, no spec requirement

• FileSystemWatcher missing input validation for debounce_ms/watch_path/extensions — defensive coding suggestion, not required by spec

• Silent ImportError pass hiding transitive dependency failures — general pattern concern, not a specific bug in this PR

• Email OAuth/IMAP integration via email-oauth2-proxy — roadmap presents EmailEngine and email-oauth2-proxy as alternative approaches; implementing one satisfies the requirement

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR and the roadmap to verify implementation against Phase 2 requirements.Now let me check what's actually been implemented in the workspace to compare against Phase 2 requirements.Now I have enough information to compare Phase 2 requirements against what's implemented. Let me verify the key files exist and check their content briefly.All Phase 2 components are present. Let me do a final check to confirm the PR's branch vs what's in the workspace.The workspace is on the same branch as the PR. I now have all information needed to produce the verification report.

---

Parent Issue Requirements Verification

Gaps Found

None found

Deviations

None found

Verified

All Phase 2 "Planned Features" from documentation/roadmap.md are implemented:

• Scheduler Components (scheduler/poller.py, scheduler/watcher.py) — Both files are present and non-trivial (247 and 209 lines respectively). The PR also fixed substantive bugs in the watcher/scheduler push-event pipeline (callback chaining, race conditions, lock management, retry queue) and poller behavior (escalating error logging, source_ref support, immediate first-tick execution, shutdown thread timeout).

• Email Adapter (adapters/email.py) — Present at 361 lines. Conditional import gate in adapters/__init__.py uses importlib.util.find_spec() to load it only when dependencies are available.

• Shared Filesystem Watcher (adapters/_watching.py) — Present at 355 lines. Shared by Filesystem and Obsidian adapters, wraps watchdog/watchfiles behind an adapter-facing interface, maps to PollStrategy.PUSH.

• Rich Filesystem Adapter (adapters/filesystem_rich.py) — Present at 277 lines. Extends the MVP filesystem adapter for non-markdown formats using MarkItDown/Pandoc, uses _watching.py.

• Obsidian Adapter (adapters/obsidian.py) — Present at 437 lines. Uses _watching.py for event capture, extracts vault metadata via obsidiantools/obsidianmd-parser.

• Messages Domain (domains/messages.py) — Present at 212 lines. Domain implementation for conversational message content with thread and author preservation.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

Groups:

Filtered Out:

• All Phase 2 requirements (scheduler components, email adapter, shared filesystem watcher, rich filesystem adapter, obsidian adapter, messages domain) were verified as fully implemented — no gaps, deviations, or bugs found against the roadmap specification
• No findings from any review phase met all three inclusion criteria (specific file:line reference or concrete missing implementation, real gap or bug in committed code, explicitly required by specifications)

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR and the roadmap to verify the implementation against the Phase 2 requirements.Now let me check what's actually implemented in the PR by examining the current workspace files:I now have a comprehensive picture of what was implemented. Let me compile the verification:

---

Parent Issue Requirements Verification

The parent issue directs implementation of Phase 2 from documentation/roadmap.md. Phase 2 specifies the following components:

Gaps Found

None found

Deviations

None found

Verified

All Phase 2 planned features are implemented and present in the PR:

• Scheduler: poller.py — Implemented at src/context_library/scheduler/poller.py (247 lines). Handles pull-based sources, per-source error escalation tracking, poll interval management, and clean shutdown via threading.Event.

• Scheduler: watcher.py — Implemented at src/context_library/scheduler/watcher.py (209 lines). Routes filesystem push events through the ingestion pipeline via handle_webhook(), with retry queue and per-event failure isolation.

• Email Adapter — Implemented at src/context_library/adapters/email.py (361 lines). Integrates with EmailEngine REST API as specified, uses httpx for HTTP, html2text for HTML-to-markdown conversion, with optional dependency guards.

• Shared Filesystem Watcher (_watching.py) — Implemented at src/context_library/adapters/_watching.py (355 lines). Shared utility wrapping watchdog (with watchfiles fallback), per-adapter instances (no singleton), debouncing, atomic-save protection, and FileEvent/SourceRef-compatible interface for PollStrategy.PUSH routing.

• Rich Filesystem Adapter (filesystem_rich.py) — Implemented at src/context_library/adapters/filesystem_rich.py (277 lines). Converts non-markdown files (PDF, DOCX, XLSX, PPTX, HTML, images, audio) using MarkItDown with Pandoc fallback. Captures filesystem metadata (MIME type, file size, timestamps, directory hierarchy) as chunk metadata. Uses _watching.py for event capture.

• Obsidian Adapter — Implemented at src/context_library/adapters/obsidian.py (437 lines). Ingests Obsidian vaults using obsidiantools for graph-level analytics (backlinks, wikilinks) and python-frontmatter for per-note YAML frontmatter, tags, aliases, and Dataview fields. Uses _watching.py for push-mode watching. Stores metadata alongside chunks.

• Messages Domain — Implemented at src/context_library/domains/messages.py (212 lines). Handles email/chat/forum content chunking with thread and author preservation, quoted-reply stripping, and token-based splitting for long messages.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the business requirements. Let me fetch the PR details.Now I have comprehensive information about the implementation. Let me do a detailed gap analysis against each functional requirement.Now I have thorough details about the implementation. Let me perform a systematic gap analysis against each functional requirement.

Business Analyst Output Verification

Gaps Found

• FR-1.7 - Poller poll_strategy filtering: The spec requires that sources managed by the poller must have poll_strategy = "pull". The Poller._tick() method delegates filtering entirely to document_store.get_sources_due_for_poll() and does not explicitly verify or enforce the poll_strategy = "pull" column value per source. The requirement could be considered met if get_sources_due_for_poll() filters on this column, but the poller itself performs no such check — and the FR explicitly states the column value must be "pull" for poller-managed sources. Whether the document store method enforces this cannot be confirmed from the code seen.

• FR-2.4 - Watcher poll_strategy enforcement: The spec requires that sources managed by the watcher have poll_strategy = "push". The Watcher class does not check or enforce this; it accepts any FileSystemWatcher and any adapter without validating that the source's poll_strategy column is "push".

• FR-3.5 - PollStrategy enum location: The requirement states that PollStrategy must be added to models.py alongside existing Domain and ChunkType enums. The enum is defined in models.py ✓. However, the requirement also states the module (_watching.py) must expose a PollStrategy enum — _watching.py re-exports it via __all__ by importing from models.py, which satisfies the exposure, but the enum is not defined in _watching.py as the FR implies. This is a minor structural deviation.

• FR-3.4 - _watching.py maps events to PollStrategy.PUSH: The module exports PollStrategy and FileEvent exposes EventType, but there is no explicit mapping of FileEvent objects to PollStrategy.PUSH within _watching.py. The watcher routes events through the Watcher.handle_webhook() path, but the _watching.py module itself does not annotate or map the emitted FileEvent instances with PollStrategy.PUSH. The FR states the module must "map filesystem events to PollStrategy.PUSH so the scheduler routes them through handle_webhook()." This mapping is architectural (handled externally) rather than encoded in the FileEvent or watcher output.

• FR-6.1 - Email provider support: The spec requires support for at least one of Gmail, Outlook, or IMAP-compatible server. The EmailAdapter implementation targets EmailEngine's REST API (a third-party self-hosted relay service), not Gmail, Outlook, or IMAP directly. This is a non-trivial deviation from the specified options — EmailEngine is not one of the three named providers.

• FR-6.6 - Email credentials not stored in plaintext: The requirement states credentials and OAuth tokens must not be stored in plaintext. The EmailAdapter accepts emailengine_url and account_id (no OAuth tokens or passwords) which are passed in-memory and not persisted by the library itself. However, the EmailEngine approach delegates credential management to a third-party service rather than implementing the OAuth flow specified in the acceptance criteria (US-6 explicitly mentions "OAuth-based authentication"). No mechanism is visible in the adapter for ensuring tokens are not persisted in plaintext in user-provided configs.

• FR-7.4 - Quoted reply chain handling without duplicating already-indexed content: The spec requires the domain to handle quoted reply chains "without duplicating content already indexed as a separate source." The MessagesDomain._strip_quoted_content() strips lines starting with > and attribution lines, but there is no mechanism to check whether the quoted content is already indexed as a separate source. The implementation strips quoted material entirely rather than implementing any de-duplication check against the index.

Deviations

• FR-6.1 - Email Adapter provider: Specified as Gmail, Outlook, or IMAP-compatible. Implemented using EmailEngine REST API, a self-hosted relay service that sits in front of email providers. This is a significant deviation from the named providers — it requires users to deploy and operate EmailEngine separately, rather than connecting directly to an email account.

• FR-3.5 - PollStrategy enum placement: The FR states the enum "must be added to models.py alongside existing Domain and ChunkType enums" AND the module (_watching.py) must expose it. The enum is in models.py and re-exported from _watching.py — this satisfies both requirements. However, EventType is also defined in models.py rather than being a separate construct in _watching.py as the FR might imply by listing it as part of the _watching.py spec.

• FR-5.2 - Obsidian frontmatter parsing library: The spec says to use "a library capable of parsing YAML frontmatter, tags, aliases, and (optionally) Dataview fields." The implementation uses python-frontmatter. Dataview field parsing is not implemented (only the optional path is absent, which is acceptable since it's marked optional in the FR).

• FR-4.1 / FR-4.2 - filesystem_rich fallback order: The spec says MarkItDown is primary and Pandoc is the fallback for unsupported formats. The implementation tries MarkItDown first, then falls back to Pandoc on failure — which matches. However, _convert_with_markitdown returns None on any failure (not just unsupported formats), meaning Pandoc is used as a general fallback rather than strictly for known-unsupported formats. This is a minor deviation from "fall back to Pandoc for file formats not supported by MarkItDown."

Verified

• FR-1.1: Poller reads poll_interval_sec and last_fetched_at via document_store.get_sources_due_for_poll() to determine due sources.
• FR-1.2: Poller calls _pipeline.ingest(adapter, chunker, source_ref=...) for each registered pull-based source when its interval has elapsed.
• FR-1.3: Poller calls document_store.update_last_fetched_at(source_id) only after a successful _pipeline.ingest() call.
• FR-1.4: Per-source failures are isolated in _tick() with individual try/except blocks and continue on failure.
• FR-1.5: Poller implements start() and stop() lifecycle methods with thread management and threading.Event-based shutdown.
• FR-1.6: Poller.register(adapter, domain_chunker) accepts adapter and domain chunker instances.
• FR-2.1: Watcher accepts FileSystemWatcher events and routes them to handle_webhook() → _pipeline.ingest().
• FR-2.2: Watcher.handle_webhook() exists as a distinct path from any polling loop.
• FR-2.3: Watcher implements start() and stop() lifecycle methods.
• FR-2.5: Watcher.handle_webhook() wraps each event in try/except with retry queuing for isolation.
• FR-3.1: _watching.py supports both watchdog and watchfiles backends behind a unified FileSystemWatcher interface.
• FR-3.2: FileSystemWatcher is instantiated per-adapter (each adapter creates its own instance) — no singleton.
• FR-3.3: FileEvent(path, event_type) is produced with path: Path compatible with pipeline's source_ref contract.
• FR-3.5 (partial): PollStrategy enum is in models.py with PUSH and PULL values; re-exported from _watching.py via __all__.
• FR-3.6: Atomic-save pattern (DELETED + CREATED within debounce window) is coalesced to MODIFIED in _on_raw_event().
• FR-4.1: RichFilesystemAdapter converts PDF, DOCX, XLSX, PPTX, HTML, images, audio using MarkItDown.
• FR-4.2: Falls back to Pandoc via subprocess for formats MarkItDown fails on.
• FR-4.3: NormalizedContent is emitted with StructuralHints containing MIME type, file size, creation timestamp, modification timestamp, and directory hierarchy.
• FR-4.4: RichFilesystemAdapter creates a FileSystemWatcher when configured for PUSH strategy.
• FR-4.5: adapter_id returns f"filesystem_rich:{directory}" — distinct from the base FilesystemAdapter.
• FR-4.6: domain property returns Domain.NOTES.
• FR-4.7: extra_metadata on StructuralHints includes mime_type (original MIME type).
• FR-5.1: ObsidianAdapter ingests a vault directory, each .md note is a source ref.
• FR-5.2: Uses python-frontmatter for YAML frontmatter, tags, and aliases extraction.
• FR-5.3: Uses obsidiantools for vault-level graph metadata (backlinks, forward wikilinks via get_wikilinks()/get_backlinks()).
• FR-5.4: Tags, aliases, frontmatter, wikilinks, backlinks, timestamps stored in extra_metadata on StructuralHints for each NormalizedContent.
• FR-5.5: Creates FileSystemWatcher scoped to .md files when PUSH strategy configured.
• FR-5.6: No format conversion — vault notes are already markdown and are yielded as-is.
• FR-5.7: domain property returns Domain.NOTES.
• FR-6.2: Email header fields (sender, recipients, subject, timestamp, message-ID, in-reply-to) extracted into MessageMetadata.
• FR-6.3: HTML email bodies converted to markdown via html2text.
• FR-6.4: domain property returns Domain.MESSAGES.
• FR-6.5: Incremental ingestion supported via source_ref as ISO 8601 timestamp passed as search[since] to the API.
• FR-6.7: thread_id and in_reply_to preserved in MessageMetadata.
• FR-7.1: MessagesDomain.chunk() returns list[Chunk] with sequential chunk_index values.
• FR-7.2: Each chunk carries domain_metadata with all MessageMetadata fields via .model_dump().
• FR-7.3: context_header is set to f"{subject} — {sender}" encoding thread context.
• FR-7.4 (partial): Quoted reply chains are stripped via _strip_quoted_content() to avoid re-chunking quoted material, though no cross-source de-duplication check is performed.
• FR-7.5: MessagesDomain is registered under Domain.MESSAGES.
• FR-7.6: MessageMetadata is a frozen Pydantic model in models.py following immutable conventions (frozen=True).

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

I'll start by checking out the PR branch and understanding the PR context.We're already on the PR branch. Let me get the PR details and diff.Now let me run the comprehensive PR review using the skill.

I'll run the comprehensive PR review with all specialized agents sequentially. Let me first identify the changed files and their scope.Good. This is a large PR with 29 changed files. All review aspects are applicable since we have new types, error handling, tests, comments, and general code changes. Let me run each specialized agent sequentially.

Agent 1: Code Reviewer (general code quality)Agent 2: Silent Failure Hunter (error handling review)Agent 3: Test Analyzer (test coverage review)

Agent 4: Comment Analyzer (comment accuracy review)Agent 5: Type Design Analyzer (type design review)Now let me update the todos and compile the final structured findings.Now let me do a quick verification of a few key findings before compiling the final report, to make sure the issues are real and current in the code.Good, I've verified the key findings. Now let me compile the structured review output. Let me also check a few more items that were highlighted.I've now verified all key findings. Let me compile the final structured report, cross-referencing against the prior review cycles to correctly tag each finding as NEW or as a potential REGRESSION.Now I have all the data needed. Let me compile the structured findings.

PR Review Findings

Critical Issues

• Watcher.handle_webhook() drops source_ref -- never passes it to pipeline.ingest(): At /workspace/src/context_library/scheduler/watcher.py:113, handle_webhook() receives source_ref as a parameter but calls self._pipeline.ingest(adapter, domain_chunker) without forwarding it. The IngestionPipeline.ingest() signature accepts source_ref="" (see /workspace/src/context_library/core/pipeline.py:69), and the Poller correctly passes it at poller.py:268. This means every push event triggers a full re-ingestion of the entire adapter instead of targeted ingestion of the changed file, defeating the purpose of push-based watching. The test at /workspace/tests/scheduler/test_watcher.py:180 asserts the current (buggy) behavior rather than the intended behavior. (NEW)

• Watchfiles loop silently dies on exception with no recovery or notification: At /workspace/src/context_library/adapters/_watching.py:245-246, _watchfiles_loop catches except Exception and only logs the error before returning. The filesystem monitoring thread terminates silently with no flag set, no event emitted, and no callback invoked. The FileSystemWatcher instance appears active but is no longer doing any work. Callers (Watcher/Poller) have no mechanism to detect that watching has stopped. Any exception from watchfiles -- OSError (inotify limit reached), PermissionError, RuntimeError -- will cause silent data staleness. (NEW)

• Race condition: _flush_buffer() can fire after stop() returns: At /workspace/src/context_library/adapters/_watching.py:132-157, _stop_watchdog() cancels the debounce timer at line 152 and then calls _flush_buffer() at line 157. However, Timer.cancel() is a no-op if the timer callback is already executing. A timer callback already in _flush_buffer() could be running concurrently, meaning callbacks fire after stop() returns. The docstring at line 139 claims "No callbacks are invoked after this method returns" which is potentially false. The same pattern exists in _stop_watchfiles() at lines 174-199. (NEW)

High Priority Issues

• MessagesDomain.hard_limit has no positive-value validation -- zero causes infinite loop: At /workspace/src/context_library/domains/messages.py:59, self.hard_limit = hard_limit accepts any integer including 0 or negative. A hard_limit of 0 would cause _split_if_needed to loop infinitely since no segment would ever be under the limit. No validation exists in __init__. (NEW)

• Poller tick_interval has no positive-value validation -- zero causes CPU-burning busy loop: At /workspace/src/context_library/scheduler/poller.py:74, self._tick_interval = tick_interval accepts any float including 0.0 or negative values. A value of 0.0 would cause _tick_loop to never sleep between ticks, creating a CPU-burning busy loop. (NEW)

• EmailAdapter silently skips messages on processing errors with no summary: At /workspace/src/context_library/adapters/email.py:163-165, fetch() catches (ValueError, KeyError, TypeError) per-message and continues. If the EmailEngine API changes its response format, every message could be silently skipped, and the pipeline would report 0 sources processed with 0 errors -- appearing as a successful empty fetch. No count of skipped messages is tracked or surfaced. (NEW)

• Obsidian _parse_note silently drops frontmatter metadata on parse error without logging: At /workspace/src/context_library/adapters/obsidian.py:182, (ValueError, KeyError, AttributeError) is caught and the code falls back to raw content with empty metadata (tags: [], aliases: [], frontmatter: {}). The original exception e is never logged. Users' YAML frontmatter tags, aliases, and properties are silently discarded with no indication. (NEW)

• RichFilesystemAdapter _convert_with_markitdown catches bare except Exception: At /workspace/src/context_library/adapters/filesystem_rich.py:55-57, except Exception catches everything including MemoryError, RecursionError, and internal MarkItDown bugs. Returns None with only a warning log (no traceback via exc_info=True). The overly-broad catch hides critical system errors. (NEW)

• Document store write_chunks TOCTOU race on duplicate check: At /workspace/src/context_library/storage/document_store.py:309-320, the code uses SELECT 1 check + INSERT instead of INSERT OR IGNORE. Between the check and insert, another thread could insert the same row (possible with WAL mode), causing an uncaught sqlite3.IntegrityError that propagates up and fails the entire source ingestion for a benign duplicate. (NEW)

Medium Priority Issues

• Pandoc conversion discards stderr on non-zero exit: At /workspace/src/context_library/adapters/filesystem_rich.py:76-78, when Pandoc returns non-zero exit code, result.stderr (containing the actual error message) is captured but never read or logged. The function returns None with zero diagnostic information about why conversion failed. (NEW)

• No test file for domains/registry.py -- entirely untested module: /workspace/src/context_library/domains/registry.py (40 lines) has zero dedicated test coverage. The get_domain_chunker() and list_registered_domains() functions are critical lookup functions with a dual-maintenance risk: the if/elif chain and the hardcoded list in list_registered_domains() must be kept in sync manually. Missing return type annotation (-> BaseDomain) on get_domain_chunker() at line 10. (NEW)

• FileSystemWatcher._callback mutated externally by Watcher.register(): At /workspace/src/context_library/scheduler/watcher.py:87, file_watcher._callback = _on_event directly overwrites a private attribute. This breaks encapsulation and means calling register() twice for the same watcher chains incorrectly, leading to duplicate ingestion. No public set_callback() or chain_callback() API exists. (NEW)

• SourceErrorTracker.consecutive_failures is a public mutable attribute: At /workspace/src/context_library/scheduler/poller.py:24, self.consecutive_failures = 0 is a public attribute. Any code can directly set it to any value including negative numbers, bypassing record_failure()/clear() methods. Should be _consecutive_failures with a read-only property. (NEW)

• Watcher _on_event continues to webhook after adapter callback failure: At /workspace/src/context_library/scheduler/watcher.py:73-85, if the adapter's internal callback (e.g., ObsidianAdapter vault cache invalidation) throws, execution continues to handle_webhook. The vault cache is NOT invalidated, so the subsequent ingestion uses stale graph data producing incorrect wikilink/backlink metadata. (NEW)

• Poller _tick catches broad except Exception for system-level errors: At /workspace/src/context_library/scheduler/poller.py:195-221, except Exception around ingest() catches sqlite3.DatabaseError (corruption), sqlite3.OperationalError (disk full), and MemoryError. These system-level failures are treated as per-source transient errors and logged at INFO level on first occurrence. (NEW)

• adapters/init.py gates EmailAdapter on httpx only, missing html2text check: At /workspace/src/context_library/adapters/__init__.py:20, the guard checks only for httpx. The EmailAdapter.__init__() separately checks for html2text and raises ImportError. So importing EmailAdapter succeeds but instantiation fails if only httpx is installed. No log message or warning when adapters are excluded due to missing dependencies. (NEW)

• Watcher docstring "No callbacks after stop returns" is contradicted by final flush: At /workspace/src/context_library/adapters/_watching.py:139 and line 181, guarantee Phase 2: Create Python package skeleton #4 states "No callbacks are invoked after this method returns" but _flush_buffer() at line 157 explicitly invokes callbacks as part of the stop sequence. The guarantee is misleading. (NEW)

• Poller._tick() docstring says "preventing infinite re-ingestion loops" -- it means the opposite: At /workspace/src/context_library/scheduler/poller.py:161, the comment says NOT updating last_fetched_at prevents re-ingestion loops. In fact, not updating means the source stays "due" and IS re-polled every tick. The behavior (retry) is correct but the comment describes the opposite intent. (NEW)

Low Priority / Nice-to-Have

• Missing tests for Poller error tracker pruning and update_last_fetched_at failure path: No test verifies error tracker entries are cleaned up when sources are no longer due (poller.py:180-186), and no test verifies that a failure in update_last_fetched_at (poller.py:224-231) is caught and doesn't crash the tick loop. (NEW)

• EmailAdapter httpx.Client not closed when adapter is not used as context manager: At /workspace/src/context_library/adapters/email.py:72, httpx.Client is created eagerly in __init__. The Poller stores adapters without calling close() or using them as context managers, causing connection pool leaks for the process lifetime. (NEW)

• StructuralHints.file_size_bytes and natural_boundaries have no non-negative validation: At /workspace/src/context_library/storage/models.py:117, file_size_bytes accepts negative integers. At line 113, natural_boundaries tuple elements are unvalidated. Negative values for either would be nonsensical. (NEW)

• Obsidian module docstring lists "watchdog" as dependency -- should be "watchdog or watchfiles": At /workspace/src/context_library/adapters/obsidian.py:16, the actual watcher implementation supports either backend. (NEW)

• Email adapter timestamp normalization uses "T" in string check for ISO 8601 detection: At /workspace/src/context_library/adapters/email.py:338, elif "T" in timestamp treats any string containing "T" as ISO 8601 without actual validation. (NEW)

Clean Areas

• DiffResult frozenset migration (models.py:274-329): Textbook improvement for deep immutability with thorough disjointness invariant enforcement in model_post_init. Scored 9.3/10 in type design.
• MessageMetadata model (models.py:129-192): Excellent cross-field invariant enforcement between is_thread_root and in_reply_to with frozen Pydantic model and tuple recipients. Scored 8.5/10.
• EventType and PollStrategy enums (models.py:19-28, 57-66): Clean, minimal string-backed enums with perfect enforcement scores.
• Poller error isolation design (poller.py:143-231): Per-source failure isolation with escalating log levels and memory management of error tracker entries.
• FileSystemWatcher debounce and atomic-save detection (_watching.py:265-310): Well-documented buffer strategy with proper lock-then-snapshot pattern and editor atomic-save coalescing.
• DocumentStore polling tests (test_document_store.py:1866-2298): Comprehensive coverage of scheduling edge cases including never-fetched sources, interval handling, push exclusion, and null interval exclusion.
• Model validation tests (test_models.py): Thorough coverage of MessageMetadata mutual exclusivity, frozen immutability, DiffResult disjointness invariants, and SourceVersion tuple immutability.
• Poller graduated logging tests (test_poller.py:351-465): Verifies exact escalation pattern (INFO at 1-2, WARNING at 3-5, ERROR at 6+) across multiple tick cycles.The comprehensive review is complete. Five specialized agents analyzed this PR across code quality, silent failures, test coverage, comment accuracy, and type design. Here's a summary:

Totals: 3 Critical, 5 High, 9 Medium, 5 Low findings -- all tagged as NEW (this is the third review cycle after two prior cycles resolved all 12 previously-reported issues).

Most impactful findings:

1. Watcher.handle_webhook() drops source_ref, making push-based incremental ingestion non-functional (full re-walk on every file change)
2. Watchfiles monitoring thread dies silently on exception with no recovery mechanism
3. Race condition in _flush_buffer() during watcher shutdown where callbacks can fire after stop() returns
4. MessagesDomain.hard_limit=0 and Poller.tick_interval=0 both cause infinite loops with no constructor validation
5. Document store TOCTOU race replacing the safer INSERT OR IGNORE pattern

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR and the roadmap to verify implementation against Phase 2 requirements.Now I have the full roadmap. Let me check the PR's actual implementation against each Phase 2 requirement by examining the repository structure.Now I have sufficient information to evaluate the PR against the Phase 2 roadmap requirements. Let me check the obsidian adapter for the obsidianmd-parser requirement specifically.Now I have a complete picture. Let me check one final detail — whether the roadmap specifies obsidianmd-parser as a required library for the Obsidian adapter.

The roadmap states: "obsidianmd-parser (per-note: YAML frontmatter, tags, Dataview fields, aliases)" — but the implementation uses python-frontmatter instead for per-note frontmatter parsing, and does not include Dataview fields support.

---

Parent Issue Requirements Verification

Gaps Found

• obsidianmd-parser not integrated: The roadmap specifies using obsidianmd-parser for per-note extraction of YAML frontmatter, tags, Dataview fields, and aliases. The implementation uses python-frontmatter for frontmatter/tags/aliases, but obsidianmd-parser is not listed as a dependency (pyproject.toml only includes obsidiantools and python-frontmatter), and Dataview fields are not extracted at all — a capability listed as a specific deliverable.

Deviations

• _watching.py does not map events to SourceRef: The roadmap states the shared watcher module "translates FileSystemEvent objects into SourceRef values the pipeline understands." The implementation translates events into a custom FileEvent dataclass (with path and event_type), not SourceRef. The mapping to SourceRef happens later in the Watcher scheduler. This is a structural deviation from the specified design, though it achieves functional equivalence.

Verified

• Scheduler — Poller (scheduler/poller.py): Automated polling with configurable tick intervals, per-source failure isolation, escalating log levels (INFO → WARNING → ERROR), and PollStrategy.PULL filtering implemented.
• Scheduler — Watcher (scheduler/watcher.py): Push/webhook-based ingestion routing via handle_webhook(), failed push event retry queue, and clean start/stop lifecycle implemented.
• Shared Filesystem Watcher (adapters/_watching.py): Wraps watchdog/watchfiles with fallback, per-adapter independent instances (no singleton), debouncing, atomic-save protection, and extension filtering implemented.
• Email Adapter (adapters/email.py): Integrates with EmailEngine REST API via httpx, HTML-to-markdown conversion, MessageMetadata extraction, no plaintext credential storage (credentials passed to EmailEngine externally).
• Rich Filesystem Adapter (adapters/filesystem_rich.py): MarkItDown primary conversion with Pandoc fallback, MIME type, file size, creation/modification timestamps, and directory hierarchy captured as chunk metadata.
• Obsidian Adapter (adapters/obsidian.py): Uses obsidiantools for vault-level graph analytics (wikilinks, backlinks), python-frontmatter for per-note tags/aliases/frontmatter, filesystem watching via _watching.py, no format conversion (vault notes are already markdown).
• Messages Domain (domains/messages.py): Chunking for conversational/email content with thread and author preservation via MessageMetadata, quoted reply stripping, and sentence-boundary splitting.
• Conditional imports (adapters/__init__.py): Optional adapters (ObsidianAdapter, EmailAdapter) gated on dependency availability.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the business requirements. Let me fetch the PR details first.Now I have comprehensive information about all the implemented files. Let me analyze each requirement against the implementation:Now I have all the information needed to do a thorough gap analysis. Let me carefully analyze each requirement:Now I have all the information needed to produce a thorough verification report. Let me analyze each requirement carefully.

Business Analyst Output Verification

Gaps Found

• FR-2.2 — Watcher handle_webhook() not distinct from polling loop: The spec requires the watcher to route push events through a handle_webhook() path distinct from the polling loop. The Watcher class has no polling loop at all — it is purely event-driven — so the distinction is vacuous. More critically, handle_webhook() calls self._pipeline.ingest(adapter, domain_chunker) without passing source_ref, so the pipeline fetches all sources rather than the specific changed source ref. The source_ref parameter exists in pipeline.ingest() but is not forwarded from the webhook event's event.path.

• FR-2.4 — Watcher does not enforce poll_strategy = "push": The spec requires that sources managed by the watcher have poll_strategy = "push". The Watcher class never reads or validates poll_strategy from the document store for its registered sources — it accepts any adapter regardless of its configured strategy. There is no guard preventing a pull-source from being registered with the watcher.

• FR-3.4 — _watching.py does not map events to PollStrategy.PUSH: The spec states the module must map filesystem events to PollStrategy.PUSH so the scheduler routes them through handle_webhook(). The FileEvent dataclass carries only path and event_type (an EventType enum); it does not carry a PollStrategy field. The routing to handle_webhook() happens in Watcher.register(), not based on any PollStrategy attached to the event itself.

• FR-3.5 — PollStrategy enum is not in models.py alongside Domain and ChunkType: The spec explicitly states "this enum must be added to models.py alongside existing Domain and ChunkType enums." PollStrategy is defined in storage/models.py (correct file), but _watching.py imports and re-exports PollStrategy from storage.models, and __all__ in _watching.py includes PollStrategy. This creates an implicit expectation that PollStrategy originates from _watching.py. The enum itself is correctly in models.py, so this is met — however the _watching.py module re-exporting it creates confusion about the canonical source.

• FR-6.1 — Email adapter does not support Gmail, Outlook, or direct IMAP: The spec requires support for "at least one email provider (Gmail, Outlook, or IMAP-compatible server)." The implemented EmailAdapter integrates exclusively with EmailEngine, a third-party self-hosted REST API proxy, which is not one of the three named providers. EmailEngine can front multiple backends, but the adapter itself does not implement Gmail OAuth, Outlook OAuth, or IMAP — it speaks only EmailEngine's REST API. This is a deviation from the enumerated provider requirement.

• FR-6.6 — Credential/token storage security not addressed: The spec states "Email credentials and OAuth tokens must not be stored in plaintext within the document store or configuration." The EmailAdapter accepts an emailengine_url and account_id as constructor parameters but has no credential or token parameters at all. Since EmailEngine handles auth externally, the adapter sidesteps this requirement rather than implementing it. There is no documented guidance or enforcement about how credentials used to configure EmailEngine itself are managed.

• FR-6.5 — Incremental ingestion uses source_ref (timestamp) but the mechanism is ambiguous: The spec requires fetching only messages newer than last_fetched_at using "the document store's tracking data." The adapter uses source_ref (passed from pipeline.ingest()) as the since timestamp. However, source_ref in the pipeline is an origin_ref from the sources table, not last_fetched_at. The pipeline does not pass last_fetched_at as source_ref — it passes origin_ref. The actual last_fetched_at field from the document store is never surfaced to the adapter.

• FR-4.1 — Images and audio conversion not verified as supported by MarkItDown: The spec explicitly lists "images, audio" as required conversion targets. MarkItDown's support for images and audio depends on optional heavy dependencies (Azure/OpenAI vision APIs for images). The adapter makes no effort to verify or document whether image/audio conversion is available, and no fallback to Pandoc is possible for these formats (Pandoc does not convert binary media to markdown).

Deviations

• FR-6.1 — EmailEngine proxy instead of direct provider integration: The spec names Gmail, Outlook, or IMAP-compatible server as required provider targets. The implementation uses EmailEngine's proprietary REST API as the sole integration point. While EmailEngine can proxy those providers, the adapter does not natively speak any of the three listed protocols.

• FR-7.4 — Quoted reply chain handling strips > prefixed lines only: The spec requires the domain to handle quoted reply chains "without duplicating content already indexed as a separate source." The implementation strips > prefixed lines and On ... wrote: headers. This is a reasonable heuristic but strips content rather than deduplicating against the index — the spec's phrasing ("without duplicating content already indexed") implies awareness of what has already been indexed, not simply stripping quoted text blindly.

• FR-1.7 — poll_strategy enforcement is in the document store query, not the poller: The spec says "the poll_strategy column value for sources managed by the poller must be 'pull'." The poller relies entirely on document_store.get_sources_due_for_poll() which filters on poll_strategy = 'pull' in SQL. The poller itself never checks poll_strategy. This works correctly but the enforcement is implicit.

Verified

• FR-1.1 — Poller reads poll_interval_sec and last_fetched_at per source via get_sources_due_for_poll().
• FR-1.2 — Poller invokes adapter.fetch(source_ref) (via pipeline.ingest()) for each due pull-based source.
• FR-1.3 — Poller calls document_store.update_last_fetched_at(source_id) after each successful fetch.
• FR-1.4 — Per-source failures are caught in _tick() with continue; other sources are not affected.
• FR-1.5 — Poller.start() and Poller.stop() lifecycle methods implemented with thread management.
• FR-1.6 — Poller.register(adapter, domain_chunker) method implemented.
• FR-2.1 — Watcher accepts filesystem events from FileSystemWatcher and routes them to the pipeline.
• FR-2.3 — Watcher.start() and Watcher.stop() implemented.
• FR-2.5 — Per-event failures in handle_webhook() are caught and isolated; retry queue continues processing.
• FR-3.1 — _watching.py supports both watchdog (primary) and watchfiles (fallback) backends.
• FR-3.2 — Each adapter instantiates its own FileSystemWatcher scoped to its own path; no singleton.
• FR-3.3 — FileEvent carries a Path value used as source_ref in webhook calls.
• FR-3.6 — Atomic-save pattern handled: DELETED + CREATED for the same path is coalesced to MODIFIED in the debounce buffer.
• FR-4.2 — Pandoc fallback implemented via _convert_with_pandoc() when MarkItDown returns None.
• FR-4.3 — StructuralHints populated with MIME type, file size, creation timestamp, modification timestamp, and directory hierarchy.
• FR-4.4 — RichFilesystemAdapter instantiates FileSystemWatcher when poll_strategy == PollStrategy.PUSH.
• FR-4.5 — adapter_id is "filesystem_rich:{resolved_path}", distinct from the plain FilesystemAdapter.
• FR-4.6 — domain property returns Domain.NOTES.
• FR-4.7 — MIME type stored in structural_hints.extra_metadata["mime_type"] on each NormalizedContent.
• FR-5.1 — ObsidianAdapter iterates vault .md files, yielding each as NormalizedContent.
• FR-5.2 — python-frontmatter used to extract YAML frontmatter, tags, and aliases.
• FR-5.3 — obsidiantools used to extract wikilinks and backlinks per note.
• FR-5.4 — Tags, aliases, frontmatter, wikilinks, backlinks, and timestamps stored in extra_metadata (passed through to domain_metadata by the chunker).
• FR-5.5 — ObsidianAdapter instantiates FileSystemWatcher when poll_strategy == PollStrategy.PUSH.
• FR-5.6 — No format conversion; notes are yielded as-is from python-frontmatter.
• FR-5.7 — domain property returns Domain.NOTES.
• FR-6.2 — Sender, recipients, subject, timestamp, message-ID, and in-reply-to extracted and stored as MessageMetadata.
• FR-6.3 — html2text used to convert HTML email bodies to markdown.
• FR-6.4 — domain property returns Domain.MESSAGES.
• FR-6.7 — thread_id and in_reply_to preserved in MessageMetadata.
• FR-7.1 — MessagesDomain.chunk() returns list[Chunk] with sequential chunk_index values.
• FR-7.2 — Each chunk's domain_metadata is populated with all MessageMetadata fields via meta.model_dump().
• FR-7.3 — context_header set to "{subject} — {sender}" on each chunk.
• FR-7.5 — MessagesDomain registered under Domain.MESSAGES (via domain property on EmailAdapter).
• FR-7.6 — MessageMetadata is a frozen Pydantic model in storage/models.py, consistent with existing conventions.
• FR-3.5 (partial) — PollStrategy enum with PUSH and PULL values defined in storage/models.py alongside Domain and ChunkType.

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Requirements Verifier

---

I'll analyze the PR diff against the architecture specifications. Let me fetch the PR details first.Now let me fetch more detailed content from the actual PR diff to understand the specifics of implementations.Now I have all the implementation details. Let me analyze each component against the architecture specifications.Now I have a thorough understanding of the implementation. Let me analyze the PR against the architecture specifications.

Software Architect Output Verification

Gaps Found

• DocumentStore.register_source() poll_strategy parameter not wired in pipeline: The architecture specified that register_source() needed a poll_strategy parameter (noting it hardcoded "pull" at line 155 as a gap). The document_store.py now correctly accepts poll_strategy in register_source(). However, the pipeline.py call to self.document_store.register_source(...) does not pass poll_strategy — it uses the default (PollStrategy.PULL). This means the adapter's own _poll_strategy attribute is never propagated to the sources table for new sources discovered during ingestion, leaving all pipeline-registered sources permanently recorded as pull regardless of the adapter's configuration.

• RichFilesystemAdapter._on_file_changed() does not trigger pipeline ingestion: The architecture specifies that the Watcher routes push events through handle_webhook() into IngestionPipeline.ingest(). The _on_file_changed() callback in filesystem_rich.py only logs a warning (logger.warning(...)) and does nothing else — it does not call any pipeline method, emit an event, or invoke a callback that the Watcher could use. In contrast, obsidian.py's _on_file_changed() at least resets self._vault = None to invalidate the cache. Neither adapter's internal callback is sufficient: the Watcher.register() chains callbacks, but _on_file_changed in filesystem_rich is not connected to any ingestion action, so push-mode file changes are silently dropped for that adapter.

• Email adapter max_initial_messages / bounded initial ingestion not implemented: The architecture explicitly specified a max_initial_messages parameter to bound the first run against large mailboxes. The EmailAdapter.__init__ accepts max_messages: int = 100 which applies to every fetch equally, not specifically an unbounded-first-run guard. The spec called for a distinct configurable bound on initial ingestion (when last_fetched_at is None), separate from the per-poll page size. The current implementation uses max_messages as the pageSize parameter to every request, meaning the initial ingestion of a large mailbox is bounded but only by the same small value used for incremental fetches — i.e., the initial ingestion will be under-fetched rather than properly bounded.

• _watching.py PollStrategy import exposed in __all__ without clear rationale: The spec placed PollStrategy in models.py (ADR-P2-05), and _watching.py re-exports it in __all__. This creates a secondary public export path for PollStrategy from an underscore-prefixed (internal) module, which contradicts the underscore-prefix convention for internal modules that are not part of the public API. PollStrategy should only be exported from models.py.

• Watcher has no start()/stop() lifecycle documentation or connection to Poller: The architecture described both Poller and Watcher owning start()/stop() lifecycle methods. While both have them, the Watcher exposes no lifecycle for its retry queue processing — flush_retry_queue() is only called during stop(). There is no periodic retry mechanism (e.g., a background thread that re-drains the queue at intervals). Events that fail ingestion accumulate in the retry queue and are only retried when another event arrives or at shutdown, meaning events can be permanently lost if the process exits before stop() is called cleanly. The architecture specified a retry queue to prevent silent event loss, but without periodic draining it still risks loss.

• DocumentStore.register_source() update branch does not update domain or adapter_id: When a source is re-registered (existing source), the UPDATE statement only updates poll_strategy and poll_interval_sec, but does not update domain or adapter_id. If an adapter re-registers a source with a different adapter version or domain reassignment, the stored metadata would be stale. The architecture specified idempotent registration (Pattern 5), but idempotency should cover the semantically relevant fields, not silently ignore domain/adapter changes.

Deviations

• ADR-P2-01 threading model deviation — Poller uses stop_event.wait() not a tick loop with explicit sleep: ADR-P2-01 specified a single background threading.Thread with a tick-based loop that "sleeps for a configurable tick interval, wakes, scans all registered sources." The implementation uses self._stop_event.wait(timeout=self._tick_interval) in a while loop, which is functionally equivalent and actually cleaner — this is a compliant implementation, not a deviation.

• ADR-P2-02 backend selection deviation — both watchdog and watchfiles are fully implemented, not deferred: ADR-P2-02 stated that "watchfiles support is deferred to a follow-up" and only watchdog would be the primary backend. The PR implements full support for both backends with a runtime fallback (_start_watchfiles, _stop_watchfiles, _watchfiles_loop are all complete). This exceeds the spec's intended scope for Phase 2 and doubles the testing surface as the ADR noted as a trade-off reason for deferring it. The decision to implement both is a deviation from the committed ADR.

• FilesystemAdapter (Phase 1) is modified: The architecture explicitly states "filesystem (unchanged)" as a Phase 1 adapter that should not be touched. The PR includes src/context_library/adapters/filesystem.py in its changed file list. The nature of those changes was not examined in depth here, but any modification to this file deviates from the spec's "unchanged" designation.

• NotesDomain (Phase 1) is modified: Similarly, the architecture states "NotesDomain unchanged." The PR includes src/context_library/domains/notes.py in its changed file list, which is a deviation from the specification.

• pipeline.py register_source() call omits poll_strategy: When the pipeline registers a new source (first ingestion), it calls self.document_store.register_source(source_id=..., adapter_id=..., domain=..., origin_ref=...) without passing poll_strategy. This defaults to PollStrategy.PULL regardless of the adapter's configuration, meaning push-mode adapters (e.g., RichFilesystemAdapter with PollStrategy.PUSH) will have their sources recorded as poll_strategy='pull' in SQLite. This contradicts the schema's intended use of the poll_strategy column and would cause the Poller to incorrectly attempt to poll push-mode sources.

Verified

• PollStrategy enum added to models.py as a str, Enum alongside Domain and ChunkType (ADR-P2-05 implemented)
• MessageMetadata defined as a frozen Pydantic BaseModel in models.py with validators for thread_id, message_id, sender, timestamp, and mutually exclusive is_thread_root/in_reply_to (ADR-P2-04 implemented)
• All new models use model_config = ConfigDict(frozen=True) (Pattern 3 followed)
• _watching.py implements watchdog as primary backend with import-time fallback check (ADR-P2-02 primary decision implemented)
• _watching.py implements debouncing via a timer buffer and coalesces delete+create atomic-save sequences into MODIFIED events (scalability design requirement met)
• FileSystemWatcher maintains independent state per instance (architecture requirement met)
• RichFilesystemAdapter implements BaseAdapter with deterministic adapter_id ("filesystem_rich:{resolved_path}"), yields NormalizedContent with StructuralHints, sets domain = Domain.NOTES (Pattern 1 followed)
• ObsidianAdapter implements BaseAdapter with adapter_id = "obsidian:{vault_path}", extracts YAML frontmatter, wikilinks/backlinks via obsidiantools, timestamps in ISO 8601, sets domain = Domain.NOTES (Pattern 1 followed)
• EmailAdapter implements BaseAdapter with adapter_id = "email:{account_id}", uses EmailEngine REST API via httpx, converts HTML to markdown via html2text, sets domain = Domain.MESSAGES (ADR-P2-03 and Pattern 1 implemented)
• EmailAdapter uses source_ref as the since timestamp parameter for incremental fetch (FR-6.5 / scalability requirement met)
• MessagesDomain implements BaseDomain.chunk(), produces Chunk objects with compute_chunk_hash(), sequential chunk_index, domain_metadata from MessageMetadata.model_dump(), ChunkType.STANDARD, and thread-aware context_header as "{subject} — {sender}" (Pattern 2 followed)
• MessagesDomain strips quoted reply content (> prefixes and "On ... wrote:" patterns)
• MessagesDomain splits oversized messages at sentence boundaries with word-boundary fallback, configurable token limit (default 1024)
• Poller uses threading.Thread with threading.Event for clean start()/stop() lifecycle (ADR-P2-01 implemented)
• Poller executes an immediate tick on start before waiting the interval
• Poller implements SourceErrorTracker with INFO→WARNING→ERROR escalation (transient: 1-2, persistent: 3-5, critical: 6+)
• Poller calls DocumentStore.update_last_fetched_at() only on success, not on failure (Pattern 4 / per-source error isolation)
• Poller calls DocumentStore.get_sources_due_for_poll() which correctly queries poll_strategy = 'pull' sources with due intervals
• Watcher chains adapter-internal callbacks before handle_webhook() (PR bug fix for callback overwrite)
• Watcher implements retry queue with max_retries and logs permanently dropped events
• DocumentStore.register_source() now accepts poll_strategy and poll_interval_sec parameters and updates them on re-registration
• DocumentStore.update_last_fetched_at() implemented as specified
• DocumentStore.get_sources_due_for_poll() implemented with correct SQL using datetime() interval arithmetic
• Dual-storage write order maintained: SQLite first, then LanceDB; lancedb_sync_log tracked (Pattern 7 followed)
• Per-source error isolation maintained in pipeline — one source failure does not block others (Pattern 4 followed)
• Schema requires no migration — existing poll_strategy, poll_interval_sec, last_fetched_at columns used directly
• Conditional imports in adapters/__init__.py use importlib.util.find_spec() to avoid ImportError for optional adapters
• pyproject.toml defines rich-fs and obsidian optional dependency groups as specified

---

Generated by Orchestrator Bot 🤖
Processed by the requirements_verifier agent

[orchestrator-bot[bot]]

Pr Code Reviewer

---

Looking at the findings from all review phases, I need to deduplicate, filter, and consolidate them into groups by functional area.

Let me analyze the overlaps and filter out items that don't meet the criteria.

{
"groups": [
{
"name": "Watcher / Push-Based Ingestion",
"severity": "critical",
"findings": "- handle_webhook() drops source_ref, defeating incremental ingestion: Watcher.handle_webhook() receives source_ref but calls self._pipeline.ingest(adapter, domain_chunker) without forwarding it. Every push event triggers a full re-walk instead of targeted ingestion. The test at tests/scheduler/test_watcher.py:180 asserts the buggy behavior. src/context_library/scheduler/watcher.py:113\n- RichFilesystemAdapter._on_file_changed() does not trigger pipeline ingestion: The callback only logs a warning and does nothing else — it does not call any pipeline method or emit an event the Watcher can act on. Push-mode file changes are silently dropped for this adapter. src/context_library/adapters/filesystem_rich.py\n- Watcher does not enforce poll_strategy = \"push\" for registered sources: The Watcher accepts any adapter regardless of its configured strategy. No guard prevents a pull-source from being registered with the Watcher. src/context_library/scheduler/watcher.py\n- _on_event continues to handle_webhook after adapter callback failure: If the adapter's internal callback throws (e.g., ObsidianAdapter vault cache invalidation), execution continues to handle_webhook. The vault cache is NOT invalidated, so subsequent ingestion uses stale graph data. src/context_library/scheduler/watcher.py:73-85\n- Retry queue only drained on next event or at shutdown: flush_retry_queue() is only called during stop() or when another event arrives. No periodic draining mechanism exists, so failed events accumulate and can be permanently lost if the process exits uncleanly."
},
{
"name": "Filesystem Watcher Module (_watching.py)",
"severity": "critical",
"findings": "- Watchfiles loop silently dies on exception with no recovery: _watchfiles_loop catches except Exception, logs the error, and returns. The monitoring thread terminates silently with no flag set and no notification to callers. The FileSystemWatcher instance appears active but is dead. src/context_library/adapters/_watching.py:245-246\n- Race condition: _flush_buffer() can fire after stop() returns: _stop_watchdog() cancels the debounce timer then calls _flush_buffer(), but Timer.cancel() is a no-op if the callback is already executing. A concurrent timer callback in _flush_buffer() means callbacks fire after stop() returns, violating the documented guarantee. Same pattern in _stop_watchfiles(). src/context_library/adapters/_watching.py:132-157\n- Docstring guarantee "No callbacks after stop returns" is false: Guarantee #4 at line 139 claims no callbacks after stop, but _flush_buffer() at line 157 explicitly invokes callbacks as part of the stop sequence. src/context_library/adapters/_watching.py:139"
},
{
"name": "Pipeline / Document Store Integration",
"severity": "high",
"findings": "- pipeline.py does not pass poll_strategy to register_source(): When the pipeline registers a new source, it calls register_source() without passing poll_strategy, defaulting to PollStrategy.PULL. Push-mode adapters (e.g., RichFilesystemAdapter) will have their sources incorrectly recorded as pull, causing the Poller to attempt polling push-mode sources. src/context_library/core/pipeline.py\n- Document store write_chunks TOCTOU race on duplicate check: Uses SELECT 1 check + INSERT instead of INSERT OR IGNORE. Between the check and insert, another thread could insert the same row (possible with WAL mode), causing an uncaught sqlite3.IntegrityError that fails the entire source ingestion. src/context_library/storage/document_store.py:309-320\n- register_source() UPDATE branch does not update domain or adapter_id: When a source is re-registered, the UPDATE only updates poll_strategy and poll_interval_sec, silently ignoring changes to domain or adapter_id. src/context_library/storage/document_store.py"
},
{
"name": "Input Validation (Poller / MessagesDomain)",
"severity": "high",
"findings": "- MessagesDomain.hard_limit allows zero, causing infinite loop: hard_limit accepts any integer including 0 or negative. A value of 0 causes _split_if_needed to loop infinitely since no segment would ever be under the limit. src/context_library/domains/messages.py:59\n- Poller.tick_interval allows zero, causing CPU-burning busy loop: tick_interval accepts any float including 0.0 or negative. A value of 0.0 causes _tick_loop to never sleep, burning CPU. src/context_library/scheduler/poller.py:74"
},
{
"name": "Email Adapter",
"severity": "high",
"findings": "- Silently skips messages on processing errors with no summary: fetch() catches (ValueError, KeyError, TypeError) per-message and continues. If the EmailEngine API changes format, every message is silently skipped and the pipeline reports 0 sources with 0 errors — appearing as a successful empty fetch. src/context_library/adapters/email.py:163-165\n- Does not support Gmail, Outlook, or direct IMAP as specified: The spec requires support for at least one of Gmail, Outlook, or IMAP-compatible server. The adapter integrates exclusively with EmailEngine, a third-party REST API proxy, which is none of the three named providers.\n- max_initial_messages bounded-first-run guard not implemented: The architecture specified a distinct max_initial_messages parameter to bound initial ingestion of large mailboxes. The adapter uses a single max_messages for both initial and incremental fetches, meaning initial ingestion is under-fetched rather than properly bounded.\n- __init__.py gates on httpx only, missing html2text check: The import guard checks only for httpx. EmailAdapter.__init__() separately checks for html2text and raises ImportError, so importing succeeds but instantiation fails if only httpx is installed. src/context_library/adapters/__init__.py:20"
},
{
"name": "Obsidian Adapter",
"severity": "high",
"findings": "- obsidianmd-parser not integrated; Dataview fields not extracted: The roadmap specifies obsidianmd-parser for per-note extraction of YAML frontmatter, tags, Dataview fields, and aliases. The implementation uses python-frontmatter instead, and Dataview fields — a specific deliverable — are not extracted at all.\n- _parse_note silently drops frontmatter metadata on parse error without logging: Catches (ValueError, KeyError, AttributeError) and falls back to raw content with empty metadata. The original exception is never logged. Users' YAML frontmatter tags, aliases, and properties are silently discarded. src/context_library/adapters/obsidian.py:182"
},
{
"name": "Poller Error Handling",
"severity": "medium",
"findings": "- _tick catches broad except Exception for system-level errors: except Exception around ingest() catches sqlite3.DatabaseError (corruption), sqlite3.OperationalError (disk full), and MemoryError. System-level failures are treated as per-source transient errors and logged at INFO level on first occurrence. src/context_library/scheduler/poller.py:195-221\n- SourceErrorTracker.consecutive_failures is a public mutable attribute: Any code can directly set it to any value including negative numbers, bypassing record_failure()/clear() methods. src/context_library/scheduler/poller.py:24\n- Docstring says "preventing infinite re-ingestion loops" but means the opposite: Comment at line 161 says NOT updating last_fetched_at prevents loops. In fact, not updating means the source stays due and IS re-polled every tick. The behavior is correct but the comment is inverted. src/context_library/scheduler/poller.py:161"
},
{
"name": "Rich Filesystem Adapter",
"severity": "medium",
"findings": "- _convert_with_markitdown catches bare except Exception: Catches everything including MemoryError, RecursionError, and internal bugs. Returns None with only a warning log (no traceback via exc_info=True). src/context_library/adapters/filesystem_rich.py:55-57\n- Pandoc conversion discards stderr on non-zero exit: When Pandoc fails, result.stderr containing the error message is captured but never logged. The function returns None with zero diagnostic information. src/context_library/adapters/filesystem_rich.py:76-78\n- Images and audio conversion not verified as supported: The spec lists images and audio as required conversion targets. MarkItDown's support for these depends on optional heavy dependencies (Azure/OpenAI vision APIs). No verification or documentation exists, and Pandoc fallback does not handle binary media."
},
{
"name": "Domain Registry",
"severity": "medium",
"findings": "- No test file for domains/registry.py — entirely untested module: registry.py (40 lines) has zero dedicated test coverage. get_domain_chunker() and list_registered_domains() have a dual-maintenance risk: the if/elif chain and the hardcoded list must be kept in sync manually. Missing return type annotation on get_domain_chunker(). src/context_library/domains/registry.py:10"
}
],
"filtered_out": [
"FileSystemWatcher._callback mutated externally by Watcher.register() — merged into Watcher / Push-Based Ingestion group (callback chaining issue)",
"_watching.py PollStrategy re-export in all — low-severity naming concern, not a bug or gap in committed requirements",
"StructuralHints.file_size_bytes negative validation — speculative; no spec requirement for non-negative validation on internal data models",
"Obsidian module docstring lists 'watchdog' instead of 'watchdog or watchfiles' — minor docstring inaccuracy, not a functional gap",
"Email adapter timestamp 'T' check for ISO 8601 — heuristic works for well-formed timestamps; not a specified requirement",
"EmailAdapter httpx.Client not closed — resource leak concern but not a requirement gap or bug in committed code",
"Missing tests for Poller error tracker pruning — nice-to-have test coverage, not a requirement gap",
"FR-6.6 credential/token storage — the adapter delegates auth to EmailEngine externally; the spec requirement is met by design even if not enforced in code",
"FR-6.5 source_ref vs last_fetched_at ambiguity — the incremental fetch mechanism functions correctly; the spec wording is ambiguous",
"FR-3.4 FileEvent does not carry PollStrategy field — structural deviation that achieves functional equivalence; routing works correctly",
"FR-7.4 quoted reply stripping vs index-aware dedup — the spec wording is ambiguous and the heuristic approach is a reasonable implementation choice",
"FR-1.7 poll_strategy enforcement in SQL vs Poller — functionally correct, enforcement location is an implementation detail",
"ADR-P2-02 both watchdog and watchfiles implemented instead of deferring watchfiles — exceeds spec scope but is not a gap or bug",
"FilesystemAdapter and NotesDomain modified (Phase 1 files) — deviation from 'unchanged' designation but without examining the nature of changes, cannot confirm as a gap",
"Clean areas (DiffResult, MessageMetadata, EventType, Poller error isolation, debounce, DocumentStore tests, Model tests, Poller logging tests) — positive findings, not issues"
]
}

---

Generated by Orchestrator Bot 🤖
Processed by the pr_code_reviewer agent