[Tracking][Catalog] Catalog SPI migration — decouple built-in connectors from FE core into loadable plugins

[morningman]

Development branch: branch-catalog-spi — all migration PRs target this branch; it merges to master once the migration completes.

Background

Today Apache Doris FE hard-codes every external data source (hive, iceberg, paimon,
hudi, trino-connector, maxcompute, jdbc, es, …) inside fe/fe-core/.../datasource/,
wired into the catalog and planner layers through switch-case and
instanceof *ExternalTable checks. Adding or changing a connector means touching FE
core.

This tracking issue covers the effort to decouple each concrete connector into an
independent, loadable plugin module under fe/fe-connector/*, leaving only generic
infrastructure and a stable catalog SPI bridge in fe-core.

End state: each connector ships as a self-contained plugin zip; fe-core has no
compile-time knowledge of any specific connector; the runtime discovers and loads
connectors through the catalog SPI (PluginDrivenExternalCatalog / …Table /
…ScanNode).

In scope: FE-side connector code under
datasource/{hive,iceberg,paimon,hudi,trinoconnector,maxcompute,jdbc,es,…}, plus the
reverse couplings (instanceof *ExternalTable, connector-specific sinks / scan-nodes)
scattered across nereids / planner / tablefunction / alter.

Out of scope: BE-side readers; fe-fs-spi file-system plugin work (separate
workstream); extension-spi; kafka / kinesis / odbc / Doris-to-Doris federation
(tracked separately); lakesoul / RemoteDoris legacy types (handled at the end or
separately).

All work lands on the branch-catalog-spi
feature branch and will be merged to master once the migration is complete.

Design invariants (acceptance criteria)

These three invariants must hold at every step:

1. One-way dependency fe-connector → fe-core. No connector module may
   import org.apache.doris.{catalog,common,datasource,qe,analysis,nereids,planner}.
   Enforced by a CI grep gate.
2. Metadata backward compatibility. Old FE images holding GSON types such as
   IcebergExternalCatalog / PaimonExternalDatabase must deserialize and migrate
   transparently to PluginDrivenExternalCatalog.
3. No user-visible behavior regression. SHOW CREATE CATALOG, SHOW TABLE STATUS,
   information_schema.tables, EXPLAIN output, error messages, and catalog type /
   engine names are all preserved.

Progress — phases

Each phase lands as one or more PRs against branch-catalog-spi.

• P0 — SPI foundation — fill SPI gaps (DDL / partition, procedures, MVCC
  snapshot, vended credentials, sys-tables, statistics, write-config) + connector
  import-gate CI. ([feat](connector) P0 SPI baseline + DDL/Partition + import gate (T03-T27) #63582)
• P1 — Scan-node consolidation — route plugin-driven scans first in the nereids
  translator; remove duplicated fe-core code. ([P1-T03-T05] route plugin-driven scans first in nereids translator #63641)
• P2 — trino-connector — first end-to-end connector migration; establishes the
  reusable migration playbook. ([feat](connector) P2 migrate trino-connector to catalog SPI (T01-T13) #64096)
• P3 — hudi (hybrid) — connector hardening + test baseline + DLA dispatch
  design merged; live cutover deferred to P7 (hudi is parasitic on HMS
  metadata). ([feat](connector) P3 hudi connector hardening + test baseline + dispatch design (hybrid, T02-T08) #64143)
• P4 — maxcompute — migrate + remove legacy subsystem from fe-core (fe-core is
  now odps-free). ([refactor](connector) P4 maxcompute: remove legacy subsystem from fe-core + make fe-core odps-free (T07-T09) #64300)
• P5 — paimon — migrate + cutover + remove legacy subsystem (fe-core is now
  paimon-SDK-free). ([refactor](catalog) P5 paimon: migrate to catalog SPI + cutover (#64446) #64446, [refactor](catalog) P5 paimon: remove legacy subsystem from fe-core + make fe-core paimon-SDK-free (T29) #64653)
• P6 — iceberg (in progress) — 7 catalog flavors + read metadata +
  scan / MVCC / cache / vended-credentials + write path + procedures + sys-tables,
  then routing cutover. ([refactor](catalog) P6 iceberg: migrate to catalog SPI + cutover + remove legacy fe-core subsystem #64688)
• P7 — hive (+HMS) (in progress) — HMS metadata ops, metastore-event pipeline,
  ACID transaction write path; absorbs the hudi live cutover. ([refactor](catalog) Catalog spi 11 hive #65473)
• P8 — cleanup (not started) — remove the SPI_READY_TYPES allowlist, legacy
  *_EXTERNAL_TABLE enum values, and remaining reverse instanceof sites.

Supporting refactor (landed alongside the phases above):

• P3b — kerberos consolidation into fe-kerberos. ([refactor](catalog) P3b: consolidate kerberos authentication into fe-kerberos #64655)

Connector status

Connector	Target module	Status	Landed in
jdbc	fe-connector-jdbc	✅ Done (SPI baseline / reference)	pre-existing
es	fe-connector-es	✅ Done (SPI baseline / reference)	pre-existing
trino-connector	fe-connector-trino	✅ Done	P2 (#64096)
maxcompute	fe-connector-maxcompute	✅ Done	P4 (#64300)
paimon	fe-connector-paimon	✅ Done	P5 (#64446, #64653)
hudi	fe-connector-hudi	🔵 In progress	P3 (#64143), P7 (#65473)
iceberg	fe-connector-iceberg	✅ Done	P6 (#64688)
hive (+HMS)	fe-connector-hive / fe-connector-hms	🔵 In progress	P7 (#65473)

---

This is an umbrella tracking issue; individual PRs reference it for detailed
per-phase design and review.