DataJoint 2.3 Released — Provenance Trinity, SparkAdapter, PostgreSQL CDC

[dimitri-yatsenko]

We are happy to announce the release of DataJoint 2.3 for Python.

For a comprehensive overview, see What's New in DataJoint 2.3.

Highlights

The provenance trinity

Three interlocking features make a computation's data lineage explicit and, optionally, enforced:

• Diagram.trace() (feat(#1423): Diagram.trace() for upstream restriction propagation #1471) — the upstream mirror of Diagram.cascade(): walk the FK graph from any result to every ancestor that contributed to it, with OR convergence and renamed-FK support.
• self.upstream (feat(#1424): self.upstream property for pre-restricted ancestor access #1473) — inside make(), read declared ancestors pre-restricted to the current key (e.g. self.upstream[Recording].fetch1(...)), no boilerplate.
• strict_provenance (feat(#1425): strict_provenance config flag for runtime enforcement #1474) — an opt-in runtime guardrail that checks a make() reads only from declared ancestors and writes only to self (and its Parts). Defaults off; best-effort by design.

Together they turn DataJoint's core provenance promise from a convention into something the framework helps construct and check.

SparkAdapter Codec Protocol

Typed codecs can now expose their decoded values as Spark-native types (#1472), opening lakehouse / Delta-Sharing consumers (e.g. Databricks) to columns that were previously opaque blobs.

PostgreSQL CDC: dj.deploy.set_replica_identity

A new dj.deploy module (#1466) configures PostgreSQL REPLICA IDENTITY for change-data-capture pipelines that need the full pre-image of updated/deleted rows — with dry-run preview and idempotent application.

Cascade fix for Part-of-Part and renamed FKs

part_integrity="cascade" now propagates correctly up through renamed foreign keys and Part-of-Part chains (#1468).

Additional fixes

• ~lineage rows self-heal on every @schema decoration (fix(#1454): refresh ~lineage on every @schema decoration #1467)
• Staged-insert metadata shape converges with ObjectCodec.encode (fix(staged_insert): converge metadata shape with ObjectCodec.encode #1465)

Install or upgrade

pip install datajoint==2.3.0

No breaking changes — strict_provenance is opt-in and everything else is additive.

Full release notes: https://github.com/datajoint/datajoint-python/releases/tag/v2.3.0