Skip to content

feat: incremental import — skip unchanged chunks via manifest chunkHashes #261

Description

@nigel-hall-codes

Parent: #259 (techdesign) — see sdlc/#259/design.md, lever L1. References #60.
Reanalyzed 2026-07-04: groundwork premise corrected; scope otherwise stands.

Why

The importer re-imports every chunk unconditionally on every run — still true on master (no fingerprint/skip logic anywhere in SFMapImporterWindow.cs). The real workflow is many re-imports after localized data fixes, so this remains the highest-value iteration lever: a re-import after a small fix should drop from minutes to seconds.

⚠ Correction (2026-07-04): the groundwork does NOT exist

The earlier body claimed GeneratedAssets.ChunkFingerprint(...) / GeneratorVersion "are already defined in PipelineTypes.cs". False — they were added in 61cc74e2 (#66) and deleted in a08384f2 (#79) together with the old C# generator, before this ticket was written. Nothing on master computes a chunk fingerprint today. Prior art is git history only:

  • 61cc74e2 — fingerprint helper (FNV-1a over inputs + params + version)
  • f75769e5 — skip-loop pattern (fingerprint match + prefab exists → InstantiatePrefab + continue)
  • the feat(pipeline): write chunkHashes into manifest.json #68 branch — not re-landable (targets the deleted SFMapPipelineWindow.cs); reference pattern only.

What

  1. Fingerprint per chunk: either Python writes chunkHashes into manifest.json (small serialize.py add), or the importer hashes the chunk's .bin and sidecars (_buildings.json, _parked.json, _names.json, overrides) — sidecars now shape the output (templates, decals), so they must dirty the chunk.
  2. In RunImport(): fingerprint match and ChunkPrefabPath(coord) exists → InstantiatePrefab the existing prefab and continue (flow per sdlc/#60/design.md).
  3. Reintroduce a GeneratorVersion constant; bump when import code changes.
  4. A skipped chunk must also skip the feat: gate per-building thumbnail/backdrop renders and server uploads behind an import toggle #447 authoring uploads (no re-render of unchanged buildings).

Done when

A re-import with unchanged source data skips already-current chunks and completes in a fraction of cold-import time, with identical resulting assets.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions