Driver does not fail over from a single slow coordinator under CL=QUORUM (OperationTimedOutException with all replicas up)

[yarongilor]

Argus: https://argus.scylladb.com/tests/scylla-cluster-tests/bdf874cc-f209-4a36-a7d3-7e7e525bb8b1

Summary

Under CL=QUORUM with RF=3 and all replicas up, a single overloaded-but-alive coordinator node caused the Java driver (as used by cassandra-stress, -mode cql3 native) to fail the operation with OperationTimedOutException after the 12 s client timeout, rather than completing the request through the other available replicas. The request appears to have been pinned to the one slow coordinator (connection in-flight: 1, pool total in-flight: 1) with no failover/speculative retry onto another host before the client timeout fired.

The cluster was healthy at the node level (no node down, no decommission); one coordinator was simply slow/saturated. QUORUM with RF=3 should tolerate one replica being unavailable, but it offers no protection against a single slow coordinator -- and here the driver did not route around it.

Observed exception

com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.4.8.102:9042] Timed out waiting for server response [configured timeout: 12000ms, connection in-flight: 1, pool pending borrows: 0, pool total in-flight: 1]
at com.datastax.driver.core.exceptions.OperationTimedOutException.copy(OperationTimedOutException.java:160)

Stress command (4 MV / 5 query profile, QUORUM):
cassandra-stress user profile=/tmp/c-s_profile_4mv_5queries.yaml
ops'(insert=15,read1=1,read2=1,read3=1,read4=1,read5=1)'
cl=QUORUM duration=1440m -mode cql3 native -rate threads=10

Environment

• ScyllaDB: 2026.3.0~dev
• Cluster: 6x i8g.8xlarge, AWS eu-west-1, RF=3 (NetworkTopologyStrategy, 3 racks, tablets enabled), keyspaces with materialized views and secondary indexes.
• Client: cassandra-stress (-mode cql3 native) -> ScyllaDB / DataStax Java driver, default load-balancing and retry policy, request read timeout 12000 ms.
• Consistency level: QUORUM (quorum for RF=3 = 2 replicas).
• Failure time: 2026-06-27 ~04:41:23 UTC.

What happened (server-side context)

A single coordinator node (10.4.8.102) became overloaded for ~10 minutes leading into the failure:

• Read rate spiked to ~532k reads/s on this node vs ~130-380k on peers; queued reads reached ~1073 vs <=153 on peers; reactor utilization ~78% vs 59-64%.
• The node was under continuous sl:default scheduling-group saturation (all 30 shards, 1000+ queued tasks), driven by the materialized-view write path, and went silent for ~31 s right at the timeout.

That server-side overload is tracked separately as a ScyllaDB engine bug: scylladb/scylladb#27068 (MV/SI write path floods the default scheduling group and starves read coordination). This issue is not a duplicate of that -- it is about the driver's behavior when one coordinator is slow but up.

Driver-team question / expected behavior

With RF=3, QUORUM, and all nodes up, a single slow coordinator should not necessarily fail the client request:

1. Should the default load-balancing / retry behavior route around (or speculatively retry on) a different coordinator/replica when the chosen coordinator does not respond within the request timeout, instead of surfacing OperationTimedOutException to the application?
2. Is speculative execution expected to be off by default here, and if so, is there guidance for c-s / longevity-style workloads to enable it so a single hot coordinator does not fail an otherwise-tolerable QUORUM request?
3. The exception shows only a single in-flight request to the one host (pool total in-flight: 1). Confirm whether, under default policy, the driver attempts the same logical operation on another host after a coordinator-side timeout, and at which point the 12 s client timeout pre-empts any such retry.

The goal is to understand whether this is expected driver behavior (and thus a configuration/recommendation matter for the workload) or a driver-side gap in failover-on-slow-coordinator under QUORUM.

Reproduction context

• Argus run: bdf874cc-f209-4a36-a7d3-7e7e525bb8b1
• Jenkins: scylla-master/tier1/longevity-mv-si-4days-streaming-test Add docs generation to scylla-4.x branch #109
• Test: longevity_test.LongevityTest.test_custom_time (longevity-mv-si-4days-streaming-test)

Related

• ScyllaDB engine bug producing the slow coordinator: C-S systematically fails when no nemesis is running scylladb#27068
• SCT tracking ticket for this run: SCT-571

[Lorak-mmk]

Should the default load-balancing / retry behavior route around (or speculatively retry on) a different coordinator/replica when the chosen coordinator does not respond within the request timeout, instead of surfacing OperationTimedOutException to the application?

Depends. Does cassandra-stress set requests as idempotent?
In Java Driver 3.x, client-side timeout is per-attempt, so if it triggers, then the retry behavior is invoked.
What is the retry behavior? For this specific case, driver doesn't know if request was executed on server-side or not, so it checks if statement is marked as idempotent by the user. If it is not, error is surfaced. If it is, retry policy is invoked.

Is speculative execution expected to be off by default here, and if so, is there guidance for c-s / longevity-style workloads to enable it so a single hot coordinator does not fail an otherwise-tolerable QUORUM request?

SE is generally off by default in the drivers, because it may increase load om the server. You can enable it, but need to be aware of this tradeoff.

Note that SE also needs statements to be marked as idempotent.

The exception shows only a single in-flight request to the one host (pool total in-flight: 1). Confirm whether, under default policy, the driver attempts the same logical operation on another host after a coordinator-side timeout, and at which point the 12 s client timeout pre-empts any such retry.

If think it was already answered - it won't be retried unless statement is marked as idempotent.