Skip to content

"Newest copy wins" de-duplication is not actually honored during indexing #2

Description

@aasis21

Bug

The README states:

Sessions appearing in multiple files (e.g. both .json and .jsonl) are de-duplicated by session id; the newest copy wins.

But the implementation does not guarantee this. iter_chat_files yields files in os.scandir order, and upsert_session uses ON CONFLICT(id) so the last file processed wins — not the newest by mtime. A stale .json can overwrite a freshly-written .jsonl for the same session id within a single run.

Expected

When the same session_id appears in multiple files, the copy with the newest mtime should win deterministically.

Suggested fix

Sort files by mtime before upserting, or collect candidates per session id and upsert only the newest.

Affected

  • engram.pyiter_chat_files, cmd_index, upsert_session

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions