Is there a way of overwriting retries for an individual call in AWS SDK for PHP?
The following code explains the question:
// Create client with a default of 2 retries
$sqsClient = new sqsClient('2012-11-05', ['retries' => 2]);
// This will retry twice to get the queue attributes (perfect)
try {
$sqsClient->getQueueAttributes();
} catch(Exception $e) {
}
// I want the following to NEVER retry
try {
$sqsClient->turnOffRetryLogic(???);
$sqsClient->receiveMessages(['WaitTimeSeconds' => 5]);
} catch(Exception $e) {
}
// Now set the retries back to as before.
Retries are handled by Middleware - but as the Middleware class is marked "final" I need to pass in a "decider"? This means we need to hook into one of the handlers but none appear to be connected to retries.
Edit:
I have managed to prove the concept of a new "decider" by directly editing the AWS SDK as follows:
final class Middleware
{
public static function retry(
callable $decider = null,
callable $delay = null,
$stats = false
) {
....
$decider = function() {
echo 'retries cancelled';
return false;
};
....
So the question is how to do this without editing the SDK. Have tried various middleware hooks as follows, without success.
$decider = function() {
echo 'No retries';
return false;
};
$SqsClient->getHandlerList()->appendSign(\AWS\Middleware::retry($decider, null), 'retry');
$result = $SqsClient->receiveMessage($aParams);
(Code samples snipped to only show relevant parts)
Next code removes retry handler
$sqsClient->getHandlerList()->remove('retry');
Sqs client isn't going to retry after that. To restore default behavior you can attach default handler back
$decider = RetryMiddleware::createDefaultDecider(3);
$sqsClient->getHandlerList()->appendSign(
Middleware::retry($decider, null, false),
'retry'
);
Though, two separate clients with retries enabled and disabled sound more transparent for me.
Related
Disclaimer: This is my first time working with ReactPHP and "php promises", so the solution might just be staring me in the face 🙄
I'm currently working on a little project where I need to create a Slack bot. I decided to use the Botman package and utilize it's Slack RTM driver. This driver utilizes ReactPHP's promises to communicate with the RTM API.
My problem:
When I make the bot reply on a command, I want to get the get retrieve the response from RTM API, so I can cache the ID of the posted message.
Problem is that, the response is being returned inside one of these ReactPHP\Promise\Promise but I simply can't figure out how to retrieve the data.
What I'm doing:
So when a command is triggered, the bot sends a reply Slack:
$response = $bot->reply($response)->then(function (Payload $item) {
return $this->reply = $item;
});
But then $response consists of an (empty?) ReactPHP\Promise\Promise:
React\Promise\Promise^ {#887
-canceller: null
-result: null
-handlers: []
-progressHandlers: & []
-requiredCancelRequests: 0
-cancelRequests: 0
}
I've also tried using done() instead of then(), which is what (as far as I can understand) the official ReactPHP docs suggest you should use to retrieve data from a promise:
$response = $bot->reply($response)->done(function (Payload $item) {
return $this->reply = $item;
});
But then $response returns as null.
The funny thing is, during debugging, I tried to do a var_dump($item) inside the then() but had forgot to remove a non-existing method on the promise. But then the var_dump actually returned the data 🤯
$response = $bot->reply($response)->then(function (Payload $item) {
var_dump($item);
return $this->reply = $item;
})->resolve();
So from what I can fathom, it's like I somehow need to "execute" the promise again, even though it has been resolved before being returned.
Inside the Bot's reply method, this is what's going on and how the ReactPHP promise is being generated:
public function apiCall($method, array $args = [], $multipart = false, $callDeferred = true)
{
// create the request url
$requestUrl = self::BASE_URL . $method;
// set the api token
$args['token'] = $this->token;
// send a post request with all arguments
$requestType = $multipart ? 'multipart' : 'form_params';
$requestData = $multipart ? $this->convertToMultipartArray($args) : $args;
$promise = $this->httpClient->postAsync($requestUrl, [
$requestType => $requestData,
]);
// Add requests to the event loop to be handled at a later date.
if ($callDeferred) {
$this->loop->futureTick(function () use ($promise) {
$promise->wait();
});
} else {
$promise->wait();
}
// When the response has arrived, parse it and resolve. Note that our
// promises aren't pretty; Guzzle promises are not compatible with React
// promises, so the only Guzzle promises ever used die in here and it is
// React from here on out.
$deferred = new Deferred();
$promise->then(function (ResponseInterface $response) use ($deferred) {
// get the response as a json object
$payload = Payload::fromJson((string) $response->getBody());
// check if there was an error
if (isset($payload['ok']) && $payload['ok'] === true) {
$deferred->resolve($payload);
} else {
// make a nice-looking error message and throw an exception
$niceMessage = ucfirst(str_replace('_', ' ', $payload['error']));
$deferred->reject(new ApiException($niceMessage));
}
});
return $deferred->promise();
}
You can see the full source of it here.
Please just point me in some kind of direction. I feel like I tried everything, but obviously I'm missing something or doing something wrong.
ReactPHP core team member here. There are a few options and things going on here.
First off then will never return the value from a promise, it will return a new promise so you can create a promise chain. As a result of that you do a new async operation in each then that takes in the result from the previous one.
Secondly done never returns result value and works pretty much like then but will throw any uncaught exceptions from the previous promise in the chain.
The thing with both then and done is that they are your resolution methods. A promise a merely represents the result of an operation that isn't done yet. It will call the callable you hand to then/done once the operation is ready and resolves the promise. So ultimately all your operations happen inside a callable one way or the other and in the broadest sense. (Which can also be a __invoke method on a class depending on how you set it up. And also why I'm so excited about short closures coming in PHP 7.4.)
You have two options here:
Run all your operations inside callable's
Use RecoilPHP
The former means a lot more mind mapping and learning how async works and how to wrap your mind around that. The latter makes it easier but requires you to run each path in a coroutine (callable with some cool magic).
It seems that only the DynamoDB and S3Clients have retry logic enabled.
It seems like the retries config value has no effect on other services. Is there an easy way to enable this on others (e.g. SQS), or have I misunderstood this functionality?
I've located the clientConfig.setUseThrottleRetries(true); option in the Java SDK, but have yet to find an equivalent in the PHP SDK.
You have misunderstood: SQS does have retries enabled.
To test: try the following:
$sqsClient = new sqqClient('2012-11-05', ['retries' => 2]);
$startTime = time();
try {
$sqsClient->receiveMessages(['WaitTimeSeconds' => 5]);
} catch(Exception $e) {
}
$timeTaken = time() - $startTime;
echo $timeTaken;
and do not send any messages. You'll see
10
as that is #WaitTime * #retries
If you get no messages, it's considered a failure and will retry to get some.
S3 and DynamoDB have special cases - which is what you noticed.
Edit:
This works: hacking AWS code.
class final class Middleware
{
public static function retry(
callable $decider = null,
callable $delay = null,
$stats = false
) {
echo 'Forcing retries to false';
$decider = function() {return false;};
...
This doesn't: in my code.
$decider = function() {
echo 'No retries';
return false;
};
$client->getHandlerList()->appendSign(\AWS\Middleware::retry($decider, null), 'retry');
I was trying to use Server Side Events mechanics in my project. (This is like Long Polling on steroids)
Example from "Sending events from the server" subtitle works beautifully. After few seconds, from disconnection, the apache process is killed. This method works fine.
BUT! If I try to use RabbitMQ, Apache does't get the process killed after browser disconnects from server (es.close()). And process leaves as is and gets killed only after the docker container restarts.
connection_aborted and connection_status don't work at all. connection_aborted returns only 0 and connection_status returns CONNECTION_NORMAL even after disconnect. It happens only when I use RabbitMQ. Without RMQ this functions works well.
ignore_user_abort(false) doesn't work either.
Code example:
<?php
use PhpAmqpLib\Channel\AMQPChannel;
use PhpAmqpLib\Connection\AbstractConnection;
use PhpAmqpLib\Exception\AMQPTimeoutException;
use PhpAmqpLib\Message\AMQPMessage;
class RequestsRabbit
{
protected $rabbit;
/** #var AMQPChannel */
protected $channel;
public $exchange = 'requests.events';
public function __construct(AbstractConnection $rabbit)
{
$this->rabbit = $rabbit;
}
public function getChannel()
{
if ($this->channel === null) {
$channel = $this->rabbit->channel();
$channel->exchange_declare($this->exchange, 'fanout', false, false, false);
$this->channel = $channel;
}
return $this->channel;
}
public function send($message)
{
$channel = $this->getChannel();
$message = json_encode($message);
$channel->basic_publish(new AMQPMessage($message), $this->exchange);
}
public function subscribe(callable $callable)
{
$channel = $this->getChannel();
list($queue_name) = $channel->queue_declare('', false, false, true, false);
$channel->queue_bind($queue_name, $this->exchange);
$callback = function (AMQPMessage $msg) use ($callable) {
call_user_func($callable, json_decode($msg->body));
};
$channel->basic_consume($queue_name, '', false, true, false, false, $callback);
while (count($channel->callbacks)) {
if (connection_aborted()) {
break;
}
try {
$channel->wait(null, true, 5);
} catch (AMQPTimeoutException $exception) {
}
}
$channel->close();
$this->rabbit->close();
}
}
What happens:
Browser establishes SSE connection to the server. var es = new EventSource(url);
Apache2 spawns new process to handle this request.
PHP generates a new Queue and connects to it.
Browser closes connection es.close()
Apache2 doesn't kill process and it stays as is. Queue of RabbitMQ will not be deleted. If I do some reconnections, it spawns a bunch of processes and a bunch of queues (1 reconnection = 1 process = 1 queue).
I close all tabs -- processes alive. I close browser -- the same situation.
Looks line some kind of PHP bug. Or of Apach2?
What I use:
Last Docker and docker-compose
php:7.1.12-apache or php:5.6-apache image (this happens on both versions of PHP)
Some screenshots:
Please, help me to figure out what's going on...
P.S. Sorry for my English. If you can find a mistake or typo, point to it in the comments. I'll be very grateful :)
You don't say if you're using send() or subscribe() (or both) during your server-side events. Assuming you're using subscribe() there is no bug. This loop:
while (count($channel->callbacks)) {
if (connection_aborted()) {
break;
}
try {
$channel->wait(null, true, 5);
} catch (AMQPTimeoutException $exception) {
}
}
Will run until the process is killed or the connection is remotely closed from RabbitMQ. This is normal when listening for queued messages. If you need to stop the loop at some point you can set a variable to check in the loop or throw an exception when the SSE is ended (although I find this awkward).
I am quite new in RabbitMQ. I'm working with php-amqplib library with codeigniter, and still wondering about some knowledge which I am lacking.
Why $channel->wait() is used?
Why it is always reside inside an endless while loop?
How to/Can I bypass the Infinite while loop.
Like In a situation when one user of my project wants to broadcast new campaign to 100k leads, the second user gets effected if he has some 100 mails to be sent, The second has to wait for 100k mails to get delivered first then the last user gets his turn.
I need a solution for Concurrent Consumers, who works smoothly without affecting the other
Here is my code snippet:
public function campaign2(){
$this->load->library('mylibrary');
for( $i=1;$i<=5;$i++ ) {
$url = "http://localhost/myproject/rabbit/waiting";
$param = array('index' => $i);
$this->waiting($i);
}
}
public function waiting($i)
{
ini_set('memory_limit','400M');
ini_set('max_execution_time', 0);
ini_set('display_errors', 1);
${'conn_'.$i} = connectRabbit();
${'channel_'.$i} = ${'conn_'.$i}->channel();
${'channel_'.$i}->exchange_declare('ha-local-campaign-'.$i.'-exchange', 'fanout', false, true, false);
$q = populateQueueName('campaign-'.$i);
${'channel_'.$i}->queue_declare($q, false, true, false, false);
${'channel_'.$i}->queue_bind($q, 'ha-local-campaign-'.$i.'-exchange', 'priority.'.$i);
$consumer_tag = 'campaign_consumer' ;
function process_message($msg) {
echo 'Mail Sent';
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
}
function shutdown($channel, $conn){
echo '['.date('H:i:s').'] Campaign consumer - Shutdown!!';
}
${'channel_'.$i}->basic_consume($q, $consumer_tag, false, false, true, false,'process_message');
while(1) {
${'channel_'.$i}->wait();
}
register_shutdown_function('shutdown', ${'channel_'.$i}, ${'conn_'.$i});
}
If anyone kindly guide me through the process I will be grateful.
When you call $channel->wait() you are:
Inspecting the channel's queues to see if there are pending messages.
For each message you are going to call the registered callback for the corresponding channel's callback.
From the "hello world example", step by step::
// First, you define `$callback` as a function receiving
// one parameter (the _message_).
$callback = function($msg) {
echo " [x] Received ", $msg->body, "\n";
};
// Then, you assign `$callback` the the "hello" queue.
$channel->basic_consume('hello', '', false, true, false, false, $callback);
// Finally: While I have any callbacks defined for the channel,
while(count($channel->callbacks)) {
// inspect the queue and call the corresponding callbacks
//passing the message as a parameter
$channel->wait();
}
// This is an infinite loop: if there are any callbacks,
// it'll run forever unless you interrupt script's execution.
Have your second user send use a different queue. You can have as many queues as you want.
I'm developing a system consisting in frontend built in CakePHP framework and Java based backend. The communication between this two ecosystems is carried out by sending JSON messages from CakePHP controller to RabbitMQ broker. When message is consumed, the result is being send back to the frontend.
Now, I need to consume the message and push the result from the controller to user browser. For the PHP part I'm using a phpamqplib, but it needs to have an infinite loop when listening for new messages:
$channel->basic_consume('AMQP.COMMAND.OUTPUT.QUEUE',
'consumer',
false,
false,
false,
false,
array($this, 'processMessage'));
function shutdown($ch, $conn){
$ch->close();
$conn->close();
}
register_shutdown_function('shutdown', $channel, $conn);
while (count($channel->callbacks)) {
$read = array($conn->getSocket()); // add here other sockets that you need to attend
$write = null;
$except = null;
if (false === ($num_changed_streams = stream_select($read, $write, $except, 60))) {
/* Error handling */
} elseif ($num_changed_streams > 0) {
$channel->wait();
}
}
In my controller this is provoking Apache Server to throw an error because maximum execution of 30 seconds is exceeded.
I really need help here. What's the best solution to listen for new messages and then pushing the result to the view?
Thanks
Cheers.
I highly recommend converting this to an AJAX-based infrastructure, and re-factor your code to do this:
CakePHP makes an AJAX call to load the page every x seconds
The AJAX URL gets the remaining elements from the queue, and outputs them
Your code doesn't look complete, so I can't refactor it completely, but you could change the AJAX URL to do something like this:
if (count($channel->callbacks)) {
$read = array($conn->getSocket()); // add here other sockets that you need to attend
$write = null;
$except = null;
if (false === ($num_changed_streams = stream_select($read, $write, $except, 60))) {
/* Error handling */
}
}
and close the channel when done.
Your other option, if you really want to use push, is to use web sockets. Do a search, or this tutorial might help you get started.