[BUG] PermissionError during project scan triggers mass index deletion

[alejandroiglesias]

Bug Description

When Basic Memory cannot read a project's root directory, a PermissionError is treated as a successful scan of an empty directory. The sync then interprets every indexed path as deleted and removes the entities from the database.

I hit this repeatedly with an Obsidian vault under macOS iCloud Drive. One MCP process was launched by an app that did not have permission to access ~/Library/Mobile Documents. The Markdown source files were not deleted, but the Basic Memory index collapsed and required a full reindex.

This is more than a local permission/configuration problem: a transient or process-specific filesystem access failure should not be converted into destructive index reconciliation.

Steps To Reproduce

1. Create and index a project containing multiple files.
2. Start another Basic Memory MCP process that uses the same configuration/database but cannot read the project root (for example, launch it from a macOS app without access to iCloud Drive, or otherwise make the root raise PermissionError).
3. Let the MCP startup background sync run.
4. Inspect the entity count or the sync log.

Expected Behavior

• The sync aborts and reports that the project root is inaccessible.
• Existing entities and index data remain unchanged.
• Scan watermarks/file counts are not updated from an incomplete scan.

Actual Behavior

The failed scan returns zero files, triggers full_deletions, and deletes indexed entities:

Permission denied scanning directory: ~/Library/Mobile Documents/iCloud~md~obsidian/Documents/Second Brain
File count decreased (84 -> 0), running full scan to detect deletions
Permission denied scanning directory: ~/Library/Mobile Documents/iCloud~md~obsidian/Documents/Second Brain
Completed full_deletions scan ... found 69 changes (new=0, modified=0, deleted=69, moves=0)

The same failure happened again two days later:

Permission denied scanning directory: ~/Library/Mobile Documents/iCloud~md~obsidian/Documents/Second Brain
File count decreased (89 -> 0), running full scan to detect deletions
Permission denied scanning directory: ~/Library/Mobile Documents/iCloud~md~obsidian/Documents/Second Brain
Completed full_deletions scan ... found 74 changes (new=0, modified=0, deleted=74, moves=0)

There were also smaller repeated deletions under the same condition (for example, 3 entities on a later run).

Environment

• OS: macOS 15.7.4 (24G517)
• Python used by MCP: 3.12.13
• Basic Memory version used by MCP: 0.22.1
• Installation/launch method: uvx basic-memory mcp
• Project storage: Obsidian vault in iCloud Drive (~/Library/Mobile Documents/...)
• Database backend: SQLite
• Multiple MCP clients shared the same project/database; the failing process lacked filesystem permission

Root Cause in Current main

The current control flow appears to make the failure destructive:

1. _quick_count_files() handles a failed find by falling back to scan_directory() and returns the resulting count.
2. scan_directory() catches PermissionError, logs a warning, and returns normally without yielding files.
3. The resulting count of 0 is lower than project.last_file_count, so scan() selects full_deletions.
4. _scan_directory_full() again receives no files from scan_directory().
5. Deletion detection marks every database path absent from the empty scanned_paths set as deleted.

Relevant code:

• src/basic_memory/sync/sync_service.py, _quick_count_files()
• src/basic_memory/sync/sync_service.py, scan_directory()
• src/basic_memory/sync/sync_service.py, full-scan deletion detection

Possible Solution

• Propagate root-level scan errors instead of representing them as an empty successful scan.
• Abort reconciliation and preserve the existing database state whenever a scan is incomplete or the root is inaccessible.
• Add a defensive guard before mass deletion when the root scan failed.
• Add a regression test where aiofiles.os.scandir(project_root) raises PermissionError and assert that no entities are deleted and watermarks remain unchanged.

Related but distinct: #792 concerns multiple MCP server instances and tool namespacing. Multiple clients made this easier to trigger in my setup, but the data-loss behavior is caused by treating an inaccessible directory as empty.

[phernandez]

@alejandroiglesias thanks for submitting this issue. I think we would need to see your logs to look at what kind of error you got before we can fix this. You can add them here or email them to me directly at paul@basicmemory.com.