Phase 2: DocumentStore Scheduling Support

[tinkermonkey]

Extend DocumentStore.register_source() to accept scheduling parameters and add a get_sources_due_for_poll() query method needed by the Poller scheduler.

Requirements

• Add poll_strategy: PollStrategy = PollStrategy.PULL and poll_interval_sec: int | None = None parameters to DocumentStore.register_source()
• The INSERT statement in register_source() must pass these values into the sources table columns poll_strategy and poll_interval_sec
• The existing INSERT OR IGNORE idempotency behavior must be preserved
• Add get_sources_due_for_poll() -> list[dict] method that queries sources where: poll_strategy = 'pull' AND (last_fetched_at IS NULL OR datetime(last_fetched_at, '+' || poll_interval_sec || ' seconds') < datetime('now'))
• Returned dicts must include at minimum: source_id, adapter_id, poll_interval_sec, last_fetched_at

Design Guidance

File to modify: src/context_library/storage/document_store.py

Current register_source signature (from codebase exploration):

def register_source(self, source_id: str, adapter_id: str, domain: Domain, origin_ref: str) -> None

New signature:

def register_source(
    self,
    source_id: str,
    adapter_id: str,
    domain: Domain,
    origin_ref: str,
    poll_strategy: PollStrategy = PollStrategy.PULL,
    poll_interval_sec: int | None = None,
) -> None

The SQLite sources table already has these columns (from the existing schema):

• poll_strategy TEXT NOT NULL CHECK (poll_strategy IN ('push', 'pull', 'webhook'))
• poll_interval_sec INTEGER
• last_fetched_at DATETIME

No schema migration is required — the columns already exist.

get_sources_due_for_poll SQL pattern:

SELECT source_id, adapter_id, poll_interval_sec, last_fetched_at
FROM sources
WHERE poll_strategy = 'pull'
  AND poll_interval_sec IS NOT NULL
  AND (
    last_fetched_at IS NULL
    OR datetime(last_fetched_at, '+' || poll_interval_sec || ' seconds') < datetime('now')
  )

Import: Add from context_library.storage.models import PollStrategy (already imports Domain).

Existing update_last_fetched_at method is already implemented and used by the pipeline — the Poller will call this after each successful fetch.

Acceptance Criteria

• register_source(..., poll_strategy=PollStrategy.PUSH) correctly persists 'push' in the sources.poll_strategy column
• register_source(..., poll_interval_sec=3600) correctly persists 3600 in sources.poll_interval_sec
• Calling register_source() twice with the same source_id does not raise an error (idempotency preserved)
• get_sources_due_for_poll() returns a source with last_fetched_at IS NULL and poll_strategy='pull'
• get_sources_due_for_poll() returns a source whose last_fetched_at + poll_interval_sec < now
• get_sources_due_for_poll() does NOT return a source whose last_fetched_at + poll_interval_sec >= now
• get_sources_due_for_poll() does NOT return sources with poll_strategy='push'
• get_sources_due_for_poll() does NOT return sources with poll_interval_sec IS NULL
• All existing DocumentStore tests continue to pass

Dependencies

Phase 1: Model Additions (PollStrategy + MessageMetadata)

Parent Issue

Part of #83

Discussion

This work is detailed in discussion 84

[tinkermonkey]

Implementation

---

I'll start by understanding the current codebase structure and then implement the required changes. Let me first explore the repository to see the current state.Great! Now I have a comprehensive understanding of the codebase. Let me create a todo list and start implementing the required changes.Now let me read the DocumentStore file to understand the exact implementation:Now let me check the models file to understand the PollStrategy enum:Perfect! Now I understand the structure. Let me mark the first todo as completed and move on to implementing the changes:Now let me modify the register_source method to accept the new parameters. First, I need to add the import for PollStrategy:Now let me update the register_source method signature and implementation:Great! Now let me mark that as completed and implement the get_sources_due_for_poll method:Now let me add the get_sources_due_for_poll method. I'll add it after the update_last_fetched_at method:Perfect! Now let me mark that as completed and move to writing the tests:Now let me read the existing test file to understand the test structure and add comprehensive tests:Let me continue reading the test file to see more test patterns:Good! Now I have a sense of the test structure. Let me append comprehensive tests for the new functionality at the end of the test file. First, let me find the end of the file:Let me read the end of the file:Perfect! Now I'll add comprehensive tests for the new poll functionality at the end of the test file. Let me append the tests:Excellent! Now let me mark the test writing as completed and run all the tests to verify everything works:Now let's run the tests to ensure everything works correctly:Perfect! All tests pass. Let me mark the final todo as completed:Excellent! Let me run a few more test suites to make sure nothing else broke:Perfect! All 367 tests pass! Now let me verify that the code changes are correct by reviewing what we've implemented:Great! Let me verify the register_source method to ensure all the changes are correct:Perfect! Now let me check the get_sources_due_for_poll method:Perfect! Let me create a summary of the implementation. First, let me run a final comprehensive test to be absolutely sure everything works:Excellent! All 13 new tests pass. Now let me review my memory file and save key learnings:Perfect! Let me create a final verification by checking the acceptance criteria one more time:Let me fix the import path:Let me use pytest instead to run the acceptance criteria verification:Perfect! All tests pass. Now let me create a summary document:

Summary

I have successfully implemented Phase 2: DocumentStore Scheduling Support (Issue #83). Here's what was completed:

Changes Made

File: /workspace/src/context_library/storage/document_store.py

1. Added Import

   • Added PollStrategy to the imports from .models

2. Modified register_source() Method (lines 130-162)

   • Added poll_strategy: PollStrategy = PollStrategy.PULL parameter
   • Added poll_interval_sec: int | None = None parameter
   • Updated INSERT statement to pass these values to the database
   • Preserved INSERT OR IGNORE idempotency behavior
   • Enhanced docstring to document the new parameters

3. Implemented get_sources_due_for_poll() Method (lines 740-780)

   • Queries sources where poll_strategy = 'pull' AND poll_interval_sec IS NOT NULL
   • Returns sources that are either:
     • Never fetched (last_fetched_at IS NULL), or
     • Overdue (datetime(last_fetched_at, '+' || poll_interval_sec || ' seconds') < datetime('now'))
   • Returns list of dicts with: source_id, adapter_id, poll_interval_sec, last_fetched_at

File: /workspace/tests/storage/test_document_store.py

Added comprehensive test suite TestSourceScheduling with 13 tests covering:

• Registration with PULL, PUSH, WEBHOOK strategies
• Default strategy (PULL)
• Idempotency preservation with new parameters
• Sources never fetched
• Sources with passed intervals
• Sources with intervals not passed
• Exclusion of PUSH strategy sources
• Exclusion of WEBHOOK strategy sources
• Exclusion of sources with NULL intervals
• Mixed source scenarios
• Correct field returns

Acceptance Criteria Met

✅ All 9 acceptance criteria verified:

1. register_source(..., poll_strategy=PollStrategy.PUSH) correctly persists 'push'
2. register_source(..., poll_interval_sec=3600) correctly persists 3600
3. Calling register_source() twice with same source_id is idempotent
4. get_sources_due_for_poll() returns source with last_fetched_at IS NULL and poll_strategy='pull'
5. get_sources_due_for_poll() returns source with last_fetched_at + poll_interval_sec < now
6. get_sources_due_for_poll() does NOT return source with last_fetched_at + poll_interval_sec >= now
7. get_sources_due_for_poll() does NOT return sources with poll_strategy='push'
8. get_sources_due_for_poll() does NOT return sources with poll_interval_sec IS NULL
9. All existing DocumentStore tests continue to pass

Test Results

• New tests: 13 tests covering all scheduling scenarios - All pass
• Existing tests: 367 total tests (65 DocumentStore tests) - All pass
• Code quality: Uses parameterized queries, follows project patterns, proper error handling

The implementation is production-ready and fully integrated with the existing DocumentStore architecture.

---

Generated by Orchestrator Bot 🤖
Processed by the senior_software_engineer agent

[tinkermonkey]

🔄 Starting Review Cycle

Reviewer: Code Reviewer
Maker: Senior Software Engineer
Max Iterations: 5

The automated maker-checker review cycle is now starting. The reviewer will evaluate the work, and if changes are needed, the maker will be automatically re-invoked with feedback.

---

Review cycle initiated by Claude Code Orchestrator

[tinkermonkey]

Code Review

---

I'll start by examining the actual changes, the project's CLAUDE.md, and relevant source files.Now let me check if there's a CLAUDE.md file in the project root and look at the models file for PollStrategy.Now let me run the existing tests to verify they pass.All 65 tests pass. Let me also check the test file's import patterns and the existing test structure for consistency.Now I have a thorough understanding of the changes. Let me compile the review.

Status

CHANGES NEEDED

Issues Found

Critical (Must Fix)

None

High Priority (Should Fix)

• Test-scoped imports of PollStrategy instead of module-level import: Every test method in TestSourceScheduling does from context_library.storage.models import PollStrategy inline (e.g., tests/storage/test_document_store.py:1835, :1852, :1870, etc.). The existing test file establishes a clear pattern at lines 21-26 where all model imports are module-level. PollStrategy should be added to the existing from context_library.storage.models import (...) block at line 21, and all the per-method imports should be removed. Same applies to the from datetime import datetime, timedelta, timezone imports at lines 1933 and 1958 — timedelta is not imported at the module level but should be added to avoid inline re-imports.

Advisory (Out of Scope / FYI)

• List comprehension could replace explicit loop in get_sources_due_for_poll: The result = []; for row in rows: result.append(...) pattern at document_store.py:768-776 could be a list comprehension, which is more idiomatic. However, the existing get_chunks_pending_sync method at line 812 uses the same result = []; for ...: result.append(...) pattern, so this is consistent with the codebase style. No change needed.

• datetime and timezone re-imported in test methods: from datetime import datetime, timedelta, timezone appears inside test methods at lines 1933 and 1958. While datetime and timezone are already imported at module level in the production code, the test file doesn't import them at all at module level. This is a pre-existing pattern issue in the test file, not introduced by this PR specifically.

Summary

The implementation is clean and correct. The register_source signature extension, the INSERT OR IGNORE idempotency preservation, and the get_sources_due_for_poll SQL query all match the requirements precisely. All 65 tests pass (13 new + 52 existing). Test coverage is thorough — it covers every acceptance criterion including never-fetched sources, overdue sources, recently-fetched sources, push/webhook exclusion, null-interval exclusion, mixed-source scenarios, and field validation.

The single high-priority item is a code hygiene issue: the new tests import PollStrategy (and datetime/timedelta/timezone) inside each test method body rather than at the module level alongside the existing imports, breaking the file's established convention. This should be straightforward to fix.

---

Generated by Orchestrator Bot 🤖
Processed by the code_reviewer agent

[tinkermonkey]

Implementation

---

I'll analyze the feedback and make the targeted revisions needed. Let me start by examining the current test file structure and understanding the exact imports to fix.Now let me see what imports are currently at the top of the test file and what needs to be added:Now I understand the issue. I need to:

1. Add PollStrategy to the module-level imports at line 21-26
2. Add datetime, timedelta, timezone to the module-level imports
3. Remove all the inline imports from the test methods

Let me make these changes:Now I need to remove the inline imports from all the test methods. Let me do this systematically:Now let me verify the changes are correct by running the tests:Perfect! Now let me run all the DocumentStore tests to verify nothing broke:Excellent! All tests pass. Now let me verify the final state of the imports at the top of the test file:Perfect! Now let me verify one of the test methods to confirm the inline imports were removed:Perfect! Let me verify one more test that had the datetime imports:Excellent! All the inline imports have been successfully removed. Now let me run a comprehensive test to make sure everything still works:Perfect! Now let me create the revision notes and output. Let me also verify the production code hasn't been touched:Good - the production code is unchanged. Now let me prepare the final revision output:

Revision Notes

• ✅ Test-scoped imports of PollStrategy and datetime modules: Moved PollStrategy, datetime, timedelta, and timezone from inline per-method imports to module-level imports at the top of the test file (lines 19, 27), following the established pattern in the file. Removed all 10 inline from context_library.storage.models import PollStrategy statements and 3 inline from datetime import datetime, timedelta, timezone statements from individual test methods.

---

Implementation

I have successfully implemented Phase 2: DocumentStore Scheduling Support (Issue #83). Here's what was completed:

Changes Made

File: /workspace/src/context_library/storage/document_store.py

1. Added Import

   • Added PollStrategy to the imports from .models

2. Modified register_source() Method (lines 130-162)

   • Added poll_strategy: PollStrategy = PollStrategy.PULL parameter
   • Added poll_interval_sec: int | None = None parameter
   • Updated INSERT statement to pass these values to the database
   • Preserved INSERT OR IGNORE idempotency behavior
   • Enhanced docstring to document the new parameters

3. Implemented get_sources_due_for_poll() Method (lines 740-780)

   • Queries sources where poll_strategy = 'pull' AND poll_interval_sec IS NOT NULL
   • Returns sources that are either:
     • Never fetched (last_fetched_at IS NULL), or
     • Overdue (datetime(last_fetched_at, '+' || poll_interval_sec || ' seconds') < datetime('now'))
   • Returns list of dicts with: source_id, adapter_id, poll_interval_sec, last_fetched_at

File: /workspace/tests/storage/test_document_store.py

1. Updated Module-Level Imports (lines 18-28)

   • Added from datetime import datetime, timedelta, timezone
   • Added PollStrategy to the existing models import block

2. Added Comprehensive Test Suite TestSourceScheduling with 13 tests covering:

   • Registration with PULL, PUSH, WEBHOOK strategies
   • Default strategy (PULL)
   • Idempotency preservation with new parameters
   • Sources never fetched
   • Sources with passed intervals
   • Sources with intervals not passed
   • Exclusion of PUSH strategy sources
   • Exclusion of WEBHOOK strategy sources
   • Exclusion of sources with NULL intervals
   • Mixed source scenarios
   • Correct field returns

3. Code Hygiene Fix

   • Removed all inline imports from test methods (10× from context_library.storage.models import PollStrategy and 3× from datetime import ... statements)
   • Consolidated all imports to module level, following the established convention in the test file

Acceptance Criteria Met

✅ All 9 acceptance criteria verified:

1. register_source(..., poll_strategy=PollStrategy.PUSH) correctly persists 'push'
2. register_source(..., poll_interval_sec=3600) correctly persists 3600
3. Calling register_source() twice with same source_id is idempotent
4. get_sources_due_for_poll() returns source with last_fetched_at IS NULL and poll_strategy='pull'
5. get_sources_due_for_poll() returns source with last_fetched_at + poll_interval_sec < now
6. get_sources_due_for_poll() does NOT return source with last_fetched_at + poll_interval_sec >= now
7. get_sources_due_for_poll() does NOT return sources with poll_strategy='push'
8. get_sources_due_for_poll() does NOT return sources with poll_interval_sec IS NULL
9. All existing DocumentStore tests continue to pass

Test Results

• New tests: 13 tests covering all scheduling scenarios - All pass
• Existing tests: 65 total tests (52 existing + 13 new) - All pass
• Code quality: Follows project conventions, module-level imports aligned with file patterns, proper error handling

The implementation is production-ready and fully integrated with the existing DocumentStore architecture.

---

Generated by Orchestrator Bot 🤖
Processed by the senior_software_engineer agent

[tinkermonkey]

Code Review

---

I'll start by examining the diff and the previous review context.The diff is clear. Let me also quickly verify the test file to make sure the top-level imports are correct and tests pass.All 13 scheduling tests pass. The changes are straightforward — inline imports were moved to module-level. Here's my review:

Status

APPROVED

Previous Issues Status

• ✅ Inline imports should be moved to module-level - RESOLVED: All from context_library.storage.models import PollStrategy and from datetime import datetime, timedelta, timezone inline imports within test methods have been hoisted to the top of the file as proper module-level imports.

New Issues Found

Critical (Must Fix)

None

High Priority (Should Fix)

None

Advisory (Out of Scope / FYI)

None

Summary

The single commit (70d8909) cleanly addresses the previous review feedback. All inline imports (PollStrategy, datetime, timedelta, timezone) have been moved to module-level imports at the top of the test file, consistent with Python conventions and the rest of the codebase. No functional changes were made — only import relocation. All 13 TestSourceScheduling tests pass.

---

Generated by Orchestrator Bot 🤖
Processed by the code_reviewer agent

[tinkermonkey]

🔄 Review Cycle Complete

Status: APPROVED
Iterations: 2/5
Maker: Senior Software Engineer
Reviewer: Code Reviewer

Review approved after 2 iteration(s)

---

Automated review cycle by Claude Code Orchestrator

[tinkermonkey]

🔧 Starting Repair Cycle (Testing)

Agent: Senior Software Engineer
Test Types: compilation, unit, integration, ci
Max Iterations: 100
Container: Isolated Docker container
Branch: feature/issue-83-mvp-phase-2

The automated test-fix-validate cycle is now starting in a containerized environment. Tests will be run, and if failures are detected, the agent will automatically fix them and re-run tests until all tests pass.

Container Features:

• ✅ Survives orchestrator restarts
• ✅ Automatic checkpointing every 5 iterations
• ✅ Progress preserved on restart

---

Repair cycle initiated by Claude Code Orchestrator

[tinkermonkey]

✅ Repair Cycle Complete

Container: repair-cycle-context-library-86-d98e33c2
Exit Code: 0
Total Agent Calls: 4
Duration: 198.6s

• compilation: ✅ PASSED (1 iterations)
• unit: ✅ PASSED (1 iterations)
• integration: ✅ PASSED (1 iterations)
• ci: ✅ PASSED (1 iterations)

---

Repair cycle executed by Claude Code Orchestrator (containerized)