Error using SQS with multiple Laravel queue readers - php

Question

I am using Laravel Jobs to read messages from an SQS queue (Laravel version 5.7)
Following Laravel indications I am using supervisor to run multiple queue:work processes at the same time.
All goes well until I get this SQS error related to the message availability:
InvalidParameterValue (client): Value
... for parameter ReceiptHandle is invalid. Reason: Message does not exist or
is not available for visibility timeout change. - <?xml version="1.0"?>
<ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error>
<Type>Sender</Type><Code>InvalidParameterValue</Code><Message>Value ...
for parameter ReceiptHandle is invalid. Reason: Message does not exist or is
not available for visibility timeout change.</Message><Detail/></Error>
<RequestId>8c1d28b7-a02c-5059-8b65-7c6292a0e56e</RequestId></ErrorResponse>
{"exception":"[object] (Aws\\Sqs\\Exception\\SqsException(code: 0): Error
executing \"ChangeMessageVisibility\" on \"https://sqs.eu-central-
1.amazonaws.com/123123123123/myQueue\"; AWS HTTP error: Client error: `POST
https://sqs.eu-central-1.amazonaws.com/123123123123/myQueue` resulted in a
`400 Bad Request` response:
In particular, the strange thing is Message does not exist or is not available for visibility timeout change.
Each supervisor process calls command=php /home/application/artisan queue:work without a --sleep=3 (I'd like the process to be reactive and not waiting for 3 seconds in case nothing was in the queue) nor a --tries=3 (I need all the tasks to be completed, so I don't put a limit to the tries parameter)
In case the message is not existing (and I can't exclude this possibility) why does the process fetches it from the queue ? Is there anything I can do to prevent it ?

Answer 1

I've seen this error intermittently in production too, where we run a good number of consumers for a single SQS queue. In our case, I'm pretty convinced that the error is due to SQS's at-least-once delivery semantics. Essentially, a message can be delivered twice or more on rare occasions.
Laravel's queue worker command isn't strictly idempotent because it will throw an exception when trying to release or delete an SQS message that is no longer available (i.e., because it has been deleted by another queue worker process, which received a duplicate of the message from SQS).
Our workaround is to try to detect when a duplicate message has been received, and then attempt to safely release the message back onto the queue. If the other queue worker that is currently working on the message succeeds, it will delete the message, and it won't be received again. If the other queue worker fails, then the message will be released and received again later. Something like this:
<?php
use Aws\Sqs\Exception\SqsException;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Str;
class ProcessPodcast implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
private $jobId;
public function __construct($jobId)
{
$this->jobId = $jobId;
}
public function handle()
{
$acquired = Cache::lock("process-podcast-$this->jobId")->get(function () {
// Process the podcast (NB: this should be idempotent)
});
if (!$acquired) {
$this->releaseDuplicateMessage($delay = 60);
}
}
private function releaseDuplicateMessage($delay)
{
try {
$this->release($delay);
} catch (Exception $ex) {
if (!$this->causedByMessageNoLongerAvailable($ex)) {
throw $ex;
}
}
}
private function causedByMessageNoLongerAvailable(Exception $ex): bool
{
return $ex instanceof SqsException &&
Str::contains(
$ex->getAwsErrorMessage(),
"Message does not exist or is not available for visibility timeout change"
);
}
}

Answer 2

Another potential for these duplicate messages is that SQS has a default Visibility Timeout of 30secs.
Visibility timeout sets the length of time that a message received from a queue (by one consumer) will not be visible to the other message consumers.
So if one worker reads a message from the queue and it takes longer than 30secs to process, the message will become visible again and another worker will start processing.
When the first worker finishes, it will delete it from the queue. Then when the second worker finishes processing the same message and tries to delete it, it can't because the first worker already deleted it.
We're having the same issue at the moment and are implementing a fix/workaround similar to Louis. Will post our version when done and confirmed working.
Note: You can increase the Visibility Timeout on SQS.