PHP Cassandra random busy I/O threads, request timed out - php

Question

I have a 3 node Cassandra v3.11.4 cluster. Replication factor = 3 and around 70GB data in each cluster.
Node hardware: m5.2xlarge (8 vCPU, 32 GB RAM, 500GB SSD)
Some YAML values:
num_tokens: 256
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
endpoint_snitch: SimpleSnitch
PHP connection from load balanced compute engines:
$cluster = Cassandra::cluster()->withPort(PORT)->withDefaultConsistency(Cassandra::CONSISTENCY_LOCAL_QUORUM)->withContactPoints(HOST_VAL)->withIOThreads(5)->withCredentials(CASS_USER, CASS_PASS)->build();
$session = $cluster->connect(KEYSPACE);
$statement = $session->prepare($query);
$stmt = $session->execute($statement, ['arguments' => $bindParams]);
The Cassandra service runs smoothly for most of the time but for 5-10 minutes every 5-6 hours it starts giving errors from PHP operations:
Cassandra\Exception\RuntimeException: All connections on all I/O threads are busy
Cassandra\Exception\RuntimeException: All hosts in current policy attempted and were either unavailable or failed
Cassandra\Exception\TimeoutException: Request timed out
I am guessing the issue is with PHP Connections either stalling Cassandra nodes or generating too many connections.
Please help me where to look for possible reasons or if there is any SHOW PROCESSLIST like command to monitor current connections like in MySQL.

Answer 1

Those errors indicate that the nodes are getting overloaded and become unresponsive leading to the TimeoutException -- the replicas don't respond to the coordinator within the request timeout.
When nodes are busy, the requests queue up waiting to be served. There comes a point when the queues reach the maximum size and any new requests from the client are no longer queued.
Check for long GCs which are indicative of nodes being overloaded. Correlate those times with the amount of read/write traffic from your application (you'll get these metrics from your app monitoring).
My guess is that your application doesn't have a smooth amount of traffic but instead is peaking every few hours. It is during app traffic peaks that the cluster gets overloaded. If this is the case, you need to size your cluster to cope with the peak traffic by adding more nodes. Cheers!