I am quite new in RabbitMQ. I'm working with php-amqplib library with codeigniter, and still wondering about some knowledge which I am lacking.
Why $channel->wait() is used?
Why it is always reside inside an endless while loop?
How to/Can I bypass the Infinite while loop.
Like In a situation when one user of my project wants to broadcast new campaign to 100k leads, the second user gets effected if he has some 100 mails to be sent, The second has to wait for 100k mails to get delivered first then the last user gets his turn.
I need a solution for Concurrent Consumers, who works smoothly without affecting the other
Here is my code snippet:
public function campaign2(){
$this->load->library('mylibrary');
for( $i=1;$i<=5;$i++ ) {
$url = "http://localhost/myproject/rabbit/waiting";
$param = array('index' => $i);
$this->waiting($i);
}
}
public function waiting($i)
{
ini_set('memory_limit','400M');
ini_set('max_execution_time', 0);
ini_set('display_errors', 1);
${'conn_'.$i} = connectRabbit();
${'channel_'.$i} = ${'conn_'.$i}->channel();
${'channel_'.$i}->exchange_declare('ha-local-campaign-'.$i.'-exchange', 'fanout', false, true, false);
$q = populateQueueName('campaign-'.$i);
${'channel_'.$i}->queue_declare($q, false, true, false, false);
${'channel_'.$i}->queue_bind($q, 'ha-local-campaign-'.$i.'-exchange', 'priority.'.$i);
$consumer_tag = 'campaign_consumer' ;
function process_message($msg) {
echo 'Mail Sent';
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
}
function shutdown($channel, $conn){
echo '['.date('H:i:s').'] Campaign consumer - Shutdown!!';
}
${'channel_'.$i}->basic_consume($q, $consumer_tag, false, false, true, false,'process_message');
while(1) {
${'channel_'.$i}->wait();
}
register_shutdown_function('shutdown', ${'channel_'.$i}, ${'conn_'.$i});
}
If anyone kindly guide me through the process I will be grateful.
When you call $channel->wait() you are:
Inspecting the channel's queues to see if there are pending messages.
For each message you are going to call the registered callback for the corresponding channel's callback.
From the "hello world example", step by step::
// First, you define `$callback` as a function receiving
// one parameter (the _message_).
$callback = function($msg) {
echo " [x] Received ", $msg->body, "\n";
};
// Then, you assign `$callback` the the "hello" queue.
$channel->basic_consume('hello', '', false, true, false, false, $callback);
// Finally: While I have any callbacks defined for the channel,
while(count($channel->callbacks)) {
// inspect the queue and call the corresponding callbacks
//passing the message as a parameter
$channel->wait();
}
// This is an infinite loop: if there are any callbacks,
// it'll run forever unless you interrupt script's execution.
Have your second user send use a different queue. You can have as many queues as you want.
Related
I want to log working status from workers' callbacks and include a number of messages in the queue left.
The only solution I found so far is getting the second member of queue_declare result array, but this should be called once per worker launch, and I need info to be updated every new message.
UPD:
Solution based on IMSoP's answer:
<?php
require_once __DIR__ . '/../vendor/autoload.php';
use PhpAmqpLib\Connection\AMQPStreamConnection;
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('test1');
echo "[*] Waiting for messages. To exit press CTRL+C\n";
$callback = function ($msg) use ($channel) {
list (, $cn) = $channel->queue_declare('test1', true);
echo ' [x] Received ', $msg->body, " $cn left";
for ($i = 0; $i < $msg->body; ++$i) {
sleep(1);
echo '.';
}
echo "\n";
};
$channel->basic_qos(null, 1, null);
$channel->basic_consume('test1', '', false, true, false, false, $callback);
while (count($channel->callbacks)) {
$channel->wait();
}
For some reason always gives 0 as message count.
The queue_declare method has a parameter called "passive" which can be used for this purpose: it checks if the queue exists, by name only, and ignores any other parameters.
According to the AMQP documentation:
If set, the server will reply with Declare-Ok if the queue already exists with the same name, and raise an error if not. The client can use this to check whether a queue exists without modifying the server state. When set, all other method fields except name and no-wait are ignored. A declare with both passive and no-wait has no effect. Arguments are compared for semantic equivalence.
Note that Declare-Ok is not just a status, but the name of the full response structure, with fields queue, message-count, and consumer-count.
In PHP-AMQPLib, you can use this to log the status of a set of queues something like this:
foreach ( $this->registeredQueues as $queueName ) {
// The second parameter to queue_declare is $passive
// When set to true, everything else is ignored, so need not be passed
list($queueName, $messageCount, $consumerCount)
= $this->rabbitChannel->queue_declare($queueName, true);
$this->logger->info(
"Queue $queueName has $messageCount messages and $consumerCount active consumers."
);
}
Is there a way of overwriting retries for an individual call in AWS SDK for PHP?
The following code explains the question:
// Create client with a default of 2 retries
$sqsClient = new sqsClient('2012-11-05', ['retries' => 2]);
// This will retry twice to get the queue attributes (perfect)
try {
$sqsClient->getQueueAttributes();
} catch(Exception $e) {
}
// I want the following to NEVER retry
try {
$sqsClient->turnOffRetryLogic(???);
$sqsClient->receiveMessages(['WaitTimeSeconds' => 5]);
} catch(Exception $e) {
}
// Now set the retries back to as before.
Retries are handled by Middleware - but as the Middleware class is marked "final" I need to pass in a "decider"? This means we need to hook into one of the handlers but none appear to be connected to retries.
Edit:
I have managed to prove the concept of a new "decider" by directly editing the AWS SDK as follows:
final class Middleware
{
public static function retry(
callable $decider = null,
callable $delay = null,
$stats = false
) {
....
$decider = function() {
echo 'retries cancelled';
return false;
};
....
So the question is how to do this without editing the SDK. Have tried various middleware hooks as follows, without success.
$decider = function() {
echo 'No retries';
return false;
};
$SqsClient->getHandlerList()->appendSign(\AWS\Middleware::retry($decider, null), 'retry');
$result = $SqsClient->receiveMessage($aParams);
(Code samples snipped to only show relevant parts)
Next code removes retry handler
$sqsClient->getHandlerList()->remove('retry');
Sqs client isn't going to retry after that. To restore default behavior you can attach default handler back
$decider = RetryMiddleware::createDefaultDecider(3);
$sqsClient->getHandlerList()->appendSign(
Middleware::retry($decider, null, false),
'retry'
);
Though, two separate clients with retries enabled and disabled sound more transparent for me.
I was trying to use Server Side Events mechanics in my project. (This is like Long Polling on steroids)
Example from "Sending events from the server" subtitle works beautifully. After few seconds, from disconnection, the apache process is killed. This method works fine.
BUT! If I try to use RabbitMQ, Apache does't get the process killed after browser disconnects from server (es.close()). And process leaves as is and gets killed only after the docker container restarts.
connection_aborted and connection_status don't work at all. connection_aborted returns only 0 and connection_status returns CONNECTION_NORMAL even after disconnect. It happens only when I use RabbitMQ. Without RMQ this functions works well.
ignore_user_abort(false) doesn't work either.
Code example:
<?php
use PhpAmqpLib\Channel\AMQPChannel;
use PhpAmqpLib\Connection\AbstractConnection;
use PhpAmqpLib\Exception\AMQPTimeoutException;
use PhpAmqpLib\Message\AMQPMessage;
class RequestsRabbit
{
protected $rabbit;
/** #var AMQPChannel */
protected $channel;
public $exchange = 'requests.events';
public function __construct(AbstractConnection $rabbit)
{
$this->rabbit = $rabbit;
}
public function getChannel()
{
if ($this->channel === null) {
$channel = $this->rabbit->channel();
$channel->exchange_declare($this->exchange, 'fanout', false, false, false);
$this->channel = $channel;
}
return $this->channel;
}
public function send($message)
{
$channel = $this->getChannel();
$message = json_encode($message);
$channel->basic_publish(new AMQPMessage($message), $this->exchange);
}
public function subscribe(callable $callable)
{
$channel = $this->getChannel();
list($queue_name) = $channel->queue_declare('', false, false, true, false);
$channel->queue_bind($queue_name, $this->exchange);
$callback = function (AMQPMessage $msg) use ($callable) {
call_user_func($callable, json_decode($msg->body));
};
$channel->basic_consume($queue_name, '', false, true, false, false, $callback);
while (count($channel->callbacks)) {
if (connection_aborted()) {
break;
}
try {
$channel->wait(null, true, 5);
} catch (AMQPTimeoutException $exception) {
}
}
$channel->close();
$this->rabbit->close();
}
}
What happens:
Browser establishes SSE connection to the server. var es = new EventSource(url);
Apache2 spawns new process to handle this request.
PHP generates a new Queue and connects to it.
Browser closes connection es.close()
Apache2 doesn't kill process and it stays as is. Queue of RabbitMQ will not be deleted. If I do some reconnections, it spawns a bunch of processes and a bunch of queues (1 reconnection = 1 process = 1 queue).
I close all tabs -- processes alive. I close browser -- the same situation.
Looks line some kind of PHP bug. Or of Apach2?
What I use:
Last Docker and docker-compose
php:7.1.12-apache or php:5.6-apache image (this happens on both versions of PHP)
Some screenshots:
Please, help me to figure out what's going on...
P.S. Sorry for my English. If you can find a mistake or typo, point to it in the comments. I'll be very grateful :)
You don't say if you're using send() or subscribe() (or both) during your server-side events. Assuming you're using subscribe() there is no bug. This loop:
while (count($channel->callbacks)) {
if (connection_aborted()) {
break;
}
try {
$channel->wait(null, true, 5);
} catch (AMQPTimeoutException $exception) {
}
}
Will run until the process is killed or the connection is remotely closed from RabbitMQ. This is normal when listening for queued messages. If you need to stop the loop at some point you can set a variable to check in the loop or throw an exception when the SSE is ended (although I find this awkward).
When I run my program through command line, then my PHP script works fine and gives output properly.
But when I run it through my browser, then it processes fine but the output I printed - is not showing.
( Note : My script is running in a while loop and it will run forever )
Code - I am writing some part of my class file
public function call($n) {
global $argv;
$this->response = null;
$this->corr_id = uniqid();
$msgBody = 'test msg';
//Create queue
$this->createQueue($queueName);
$msg = new AMQPMessage(
(string) $msg_body,
array('correlation_id' => $this->corr_id,
'reply_to' => $this->callback_queue,
'priority' => 2)
);
$this->channel->basic_publish($msg, '', $queueName);
while(!$this->response) {
$this->channel->wait();
}
return $this->response;
}
$response = call();
createQueue() function is in another class
function createQueue($queueName='')
{
$exchange = 'router';
$queue = 'msgs';
$consumer_tag = 'consumer';
$connection = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $connection->channel();
$channel->queue_declare($queueName, false, false, false, false);
echo " [x] Awaiting for message\n";
$callback = function($req) {
$n = $req->body;
echo "$n\n";
$msg = new AMQPMessage(
'msg',
array('correlation_id' => $req->get('correlation_id'))
);
$req->delivery_info['channel']->basic_publish(
$msg, '', $req->get('reply_to'));
$req->delivery_info['channel']->basic_ack(
$req->delivery_info['delivery_tag']);
};
$channel->basic_qos(null, 1, null);
$channel->basic_consume($queueName, '', false, false, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
}
Yes we can :)
Have a look at http://www.php.net/manual/en/function.flush.php - this is the way to go for permanently running scripts when you need to output before the response is complete.
Also have a look at the first user comment there - it is about sending in 4KB chunks with str_pad() as browsers tend to not display anything at all until a specific amount of data is reached.
That is bound to happen when you have a script running in a while loop and run forever.
Running the script in browser will timeout and will not output stuff since the script is not completed.
while you run a script on Command Line, the script will be executed line by line and output will be shown
Why does the loop need to run forever and ever? If it never finishes, it will never actually return a response to the client.
As the php runs, it builds an HTML document in a buffer as you add to it, and then all at once it sends the page.
If the loop generates content for the page on every iteration, you may want to stop it after a certain number of cycles through the loop, so that you are not sending the user an infinite stream of data.
If you are doing something such as a calculation, you should look into using JavaScript, and making AJAX calls to the php scripts.
I'm developing a system consisting in frontend built in CakePHP framework and Java based backend. The communication between this two ecosystems is carried out by sending JSON messages from CakePHP controller to RabbitMQ broker. When message is consumed, the result is being send back to the frontend.
Now, I need to consume the message and push the result from the controller to user browser. For the PHP part I'm using a phpamqplib, but it needs to have an infinite loop when listening for new messages:
$channel->basic_consume('AMQP.COMMAND.OUTPUT.QUEUE',
'consumer',
false,
false,
false,
false,
array($this, 'processMessage'));
function shutdown($ch, $conn){
$ch->close();
$conn->close();
}
register_shutdown_function('shutdown', $channel, $conn);
while (count($channel->callbacks)) {
$read = array($conn->getSocket()); // add here other sockets that you need to attend
$write = null;
$except = null;
if (false === ($num_changed_streams = stream_select($read, $write, $except, 60))) {
/* Error handling */
} elseif ($num_changed_streams > 0) {
$channel->wait();
}
}
In my controller this is provoking Apache Server to throw an error because maximum execution of 30 seconds is exceeded.
I really need help here. What's the best solution to listen for new messages and then pushing the result to the view?
Thanks
Cheers.
I highly recommend converting this to an AJAX-based infrastructure, and re-factor your code to do this:
CakePHP makes an AJAX call to load the page every x seconds
The AJAX URL gets the remaining elements from the queue, and outputs them
Your code doesn't look complete, so I can't refactor it completely, but you could change the AJAX URL to do something like this:
if (count($channel->callbacks)) {
$read = array($conn->getSocket()); // add here other sockets that you need to attend
$write = null;
$except = null;
if (false === ($num_changed_streams = stream_select($read, $write, $except, 60))) {
/* Error handling */
}
}
and close the channel when done.
Your other option, if you really want to use push, is to use web sockets. Do a search, or this tutorial might help you get started.