Upgrade Testcontainers for modern Docker Engine compatibility (docker-java API negotiation)

[alexander-yevsyukov]

Summary

Our pinned Testcontainers 1.19.0 (which brings docker-java 3.3.3) cannot
communicate with recent Docker Engine releases. This surfaced while adding a
"Docker is required for the emulator tests" gate: previously the Datastore-emulator
tests silently skipped when Docker looked unavailable, which masked the
incompatibility and let a green-but-empty build pass.

Recommended fix (validated this session, included in the Docker-test-gate change):
bump Testcontainers 1.19.0 → 1.21.4 (docker-java 3.4.2), which negotiates the Docker
Remote API version with the daemon.

This issue also tracks keeping Testcontainers current going forward — in particular
evaluating the 2.x line (latest 2.0.5), deferred during the session as a major
release with potential breaking changes.

What was discovered

Environment: macOS + Docker Desktop, Docker Engine 29.5.3 (Remote API 1.54,
minimum 1.40).

With Testcontainers 1.19.0 / docker-java 3.3.3, every emulator test failed at Docker
discovery:

org.testcontainers.dockerclient.DockerClientProviderStrategy
  - Could not find a valid Docker environment. Attempted configurations were:
      UnixSocketClientProviderStrategy:    failed with BadRequestException (Status 400 ...)
      DockerDesktopClientProviderStrategy: failed with BadRequestException (Status 400 ...)

Root cause: docker-java 3.3.3 defaults to a Docker Remote API version below the
daemon's minimum (1.40), so Docker Engine 29 rejects the /info request with HTTP
400. The docker CLI works because it auto-negotiates the API version; docker-java
3.3.3 does not. (The Gradle-level gate uses docker info via the CLI, so it reports
"Docker available" — the failure is specific to docker-java inside the test JVM.)

Confirmed fixes:

• Pinning docker-java's api.version into [1.40, 1.54] (e.g. via
  ~/.docker-java.properties) — proves the diagnosis, but a band-aid.
• Upgrading to Testcontainers 1.21.4 (docker-java 3.4.2) — clean root-cause fix;
  docker-java negotiates the API version automatically. No pins or env vars needed.

Verification (Testcontainers 1.21.4, on Docker Engine 29)

• ./gradlew build SUCCESSFUL, Ryuk enabled, no pins/env hacks.
• Emulator suites: :datastore 280 passed / 0 failed (19 expected off-CI assumeTrue
  skips), :testutil-gcloud 18 / 0.
• Dependency reports (docs/dependencies/dependencies.md, docs/dependencies/pom.xml)
  regenerated to reflect testcontainers 1.21.4 / docker-java 3.4.2.

Proposed follow-up

• Evaluate Testcontainers 2.x (currently 2.0.5): review breaking changes
  (minimum JDK, module/artifact reorganisation, and the DatastoreEmulatorContainer
  API used by testutil-gcloud's EmulatorContainer).
• Upgrade once vetted, or fold Testcontainers into the regular dependency-update
  cadence so we don't fall behind Docker Engine's minimum API version again.

Why this matters

docker-java compatibility tracks Docker Engine's minimum Remote API version, which
rises over time. Staying current on Testcontainers avoids a recurrence where a developer
or CI runner on a newer Docker silently can't run the emulator tests.