We are using RabbitMQ v3.7.5, AMQP 0.9.1, Erlang 20.1.7.1, PHP 7, centos 7
Our consumers are written using php-amqplib v2.9.1
We start our consumers by calling a URL using curl. The consumers work as expected for sometime, but after sometime all consumers gets doubled i.e it shows two connection running for same consumer. Once this happens, none of the consumers respond when there is data on the queue but all the data received on the queue is lost.
Has anyone faced the same issue?
Consumer Code (I am using the same code given on the official site: https://www.rabbitmq.com/tutorials/tutorial-two-php.html)
require_once __DIR__ . '/vendor/autoload.php';
use PhpAmqpLib\Connection\AMQPStreamConnection;
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('task_queue', false, true, false, false);
echo " [*] Waiting for messages. To exit press CTRL+C\n";
$callback = function ($msg) {
echo ' [x] Received ', $msg->body, "\n";
sleep(substr_count($msg->body, '.'));
echo " [x] Done\n";
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
};
$channel->basic_qos(null, 1, null);
$channel->basic_consume('task_queue', '', false, false, false, false, $callback);
while (count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
Expected Result: The consumer should not be duplicated.
Related
I'm trying to send some data from php application to the user's browser using websockets. Therefore I've decided to use Swoole in combination with RabbitMQ.
It's the first time I'm working with websockets and after reading some posts about Socket.IO, Ratchet, etc. I've decided to halt on Swoole because it's written in C and handy to use with php.
This is how I understood the idea of enabling data transfer using websockets:
1) Start RabbitMQ worker and Swoole server in CLI
2) php application sends data to RabbitMQ
3) RabbitMQ sends message with data to worker
4) Worker receives message with data + establishes socket connection with Swoole socket server.
5) Swoole server broadcasts data to all connections
The question is how to bind Swoole socket server with RabbitMQ? Or how to make RabbitMQ to establish connection with Swoole and send data to it?
Here is the code:
Swoole server (swoole_sever.php)
$server = new \swoole_websocket_server("0.0.0.0", 2345, SWOOLE_BASE);
$server->on('open', function(\Swoole\Websocket\Server $server, $req)
{
echo "connection open: {$req->fd}\n";
});
$server->on('message', function($server, \Swoole\Websocket\Frame $frame)
{
echo "received message: {$frame->data}\n";
$server->push($frame->fd, json_encode(["hello", "world"]));
});
$server->on('close', function($server, $fd)
{
echo "connection close: {$fd}\n";
});
$server->start();
Worker which receives message from RabbitMQ, then makes connection to Swoole and broadcasts the message via socket connection (worker.php)
$connection = new AMQPStreamConnection('0.0.0.0', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('task_queue', false, true, false, false);
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$callback = function($msg){
echo " [x] Received ", $msg->body, "\n";
sleep(substr_count($msg->body, '.'));
echo " [x] Done", "\n";
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
// Here I'm trying to make connection to Swoole server and sernd data
$cli = new \swoole_http_client('0.0.0.0', 2345);
$cli->on('message', function ($_cli, $frame) {
var_dump($frame);
});
$cli->upgrade('/', function($cli)
{
$cli->push('This is the message to send to Swoole server');
$cli->close();
});
};
$channel->basic_qos(null, 1, null);
$channel->basic_consume('task_queue', '', false, false, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
New task where the message will be send to RabbitMQ (new_task.php):
$connection = new AMQPStreamConnection('0.0.0.0', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('task_queue', false, true, false, false);
$data = implode(' ', array_slice($argv, 1));
if(empty($data)) $data = "Hello World!";
$msg = new AMQPMessage($data,
array('delivery_mode' => AMQPMessage::DELIVERY_MODE_PERSISTENT)
);
$channel->basic_publish($msg, '', 'task_queue');
echo " [x] Sent ", $data, "\n";
$channel->close();
$connection->close();
After starting both swoole server and worker I'm triggering new_task.php from command line:
php new_task.php
In command line prompt where a RabbitMQ Worker is running (worker.php) I can see that a message is delivered to the worker ("[x] Received Hello World!" message is appearing).
However in command line prompt where Swoole server is running happens nothing.
So the questions are:
1) Is the idea of this approach right?
2) What am I doing wrong?
In the callback(in worker.php) that fires when a message is received you're using swoole_http_client which is async only. This seems to results in the code never being fully executed as the callback function returns before the async code is triggered.
A synchronous method of doing the same thing will solve the problem. Here is a simple example:
$client = new WebSocketClient('0.0.0.0', 2345);
$client->connect();
$client->send('This is the message to send to Swoole server');
$recv = $client->recv();
print_r($recv);
$client->close();
Check out the WebSocketClient class and example usage at github.
You can also wrap it in a coroutine, like this:
go(function () {
$client = new WebSocketClient('0.0.0.0', 2345);
$client->connect();
$client->send('This is the message to send to Swoole server');
$recv = $client->recv();
print_r($recv);
$client->close();
});
I am trying to achieve mechanism where PHP pushes message to RabbitMQ (I don't want RabbitMQ to be directly exposed to user), RabbitMQ connects to RatchetPHP and Ratchet broadcasts it via websocket connections to users.
The issue I have is with accually making Ratchet server to simultanously listen for queue messages and transfer them further. Ratchet documentation assumes using ZeroMQ and after a long search through outdated documentations and libraries which do not have such methods anymore (eg. React\Stomp) I need fresh eyes from someone who has experience with these solutions.
What I have is pusher.php (standard example from RabbitMQ docs):
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('hello', false, false, false, false);
$msg = new AMQPMessage('Hello World!');
$channel->basic_publish($msg, '', 'hello');
echo " [x] Sent 'Hello World!'\n";
$channel->close();
$connection->close();
Just to simplify reproducing scenario I include also Chat class:
use Ratchet\ConnectionInterface;
use Ratchet\MessageComponentInterface;
class Chat implements MessageComponentInterface
{
protected $clients;
public function __construct()
{
$this->clients = new \SplObjectStorage;
}
public function onOpen(ConnectionInterface $connection)
{
// Store the new connection to send messages to later
$this->clients->attach($connection);
echo "New connection! ({$connection->resourceId})\n";
}
public function onMessage(ConnectionInterface $from, $msg)
{
$numRecv = count($this->clients) - 1;
echo sprintf('Connection %d sending message "%s" to %d other connection%s'."\n"
, $from->resourceId, $msg, $numRecv, $numRecv == 1 ? '' : 's');
foreach($this->clients as $client)
{
/** #var \SplObjectStorage $client */
if($from !== $client)
{
// The sender is not the receiver, send to each client connected
$client->send($msg);
}
}
}
public function onClose(ConnectionInterface $conn)
{
// The connection is closed, remove it, as we can no longer send it messages
$this->clients->detach($conn);
echo "Connection {$conn->resourceId} has disconnected\n";
}
public function onError(ConnectionInterface $conn, \Exception $e)
{
echo "An error has occurred: {$e->getMessage()}\n";
$conn->close();
}
}
And Ratchet server.php (standard Ratchet example and RabbitMQ receiver example):
use PhpAmqpLib\Connection\AMQPStreamConnection;
use Src\Chat;
// RABBIT_RECEIVER
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('hello', false, false, false, false);
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$callback = function ($msg)
{
echo " [x] Received ", $msg->body, "\n";
};
$channel->basic_consume('hello', '', false, true, false, false, $callback);
while(count($channel->callbacks))
{
$channel->wait();
}
$channel->close();
$connection->close();
// RABBIT_RECEIVER END
$server = new \Ratchet\App('sockets.dev');
$server->route('/', new Chat());
$server->run();
Current versions are basically 2 separate mechanisms listening for messages and they work great alone (so no issue there) except that they block each other and do not transfer messages between.
Question is how to make server.php to make RabbitMQ receive message and plug it into running Ratchet server.
I figured it out so adding answer for posterity. Solution is not fully real time but it is very close, seems to have good performance and is non-blocking for Ratchet websocket server. Solution was in Ratchet custom loop and addPeriodicTimer method.
So code for server.php should look like this:
$loop = React\EventLoop\Factory::create();
$chat = new Chat($loop);
$server = new \Ratchet\App('sockets.dev', 8080, '127.0.0.1', $loop);
$server->route('/', $chat);
$server->run();
And Chat.php class (only __constructor because rest is the same):
public function __construct(LoopInterface $loop)
{
$this->loop = $loop;
$this->clients = new \SplObjectStorage();
$this->loop->addPeriodicTimer(0, function ()
{
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('hello', false, false, false, false);
echo ' [*] Checking for for messages from RabbitMQ', "\n";
$max_number_messages_to_fetch_per_batch = 10000;
do
{
$message = $channel->basic_get('hello', true);
if($message)
{
foreach($this->clients as $client)
{
$client->send($message->body);
}
$max_number_messages_to_fetch_per_batch--;
}
}
while($message && $max_number_messages_to_fetch_per_batch > 0);
$channel->close();
$connection->close();
});
}
Initiating Ratchet server will attach periodic event (in example every 0 sec) which will check RabbitMQ for new messages and process them.
Just for better performance control allowing websocket to take a breath number of messages from RabbitMQ processed in one batch was limited to 10k. Processed messages are removed from queue and next batch is processed in next iteration.
You can also fine tune frequency of updates with addPeriodicTimer interval parameter. 0 seconds is the closest to real time you will get but it may not be required for your needs and can be changed to higher value.
I have a RabbitMQ installation on a PuPHPet-generated virtual box (Ubuntu 16.04 x64) on top of Windows 10.
After the setup was complete, I configured a new user using rabbitmqctl:
# rabbitmqctl add_user root root
# rabbitmqctl set_permissions -p / root ".*" ".*" ".*"
# rabbitmqctl set_user_tags root administrator
Following the PHP & RabbitMQ Tutorial, I set up a sender and receiver script.
The sender script is the following:
<?php
require_once 'vendor/autoload.php';
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;
$connection = new AMQPStreamConnection('localhost', 5672, 'root', 'root');
$channel = $connection->channel();
$channel->queue_declare('my_queue', false, false, false, false);
$msg = new AMQPMessage('Hello World!');
$channel->basic_publish($msg, '', 'hello');
echo "Sent 'Hello World!'\n";
$channel->close();
$connection->close();
And the receiver script is the following:
<?php
require_once __DIR__ . '/vendor/autoload.php';
use PhpAmqpLib\Connection\AMQPStreamConnection;
$connection = new AMQPStreamConnection('localhost', 5672, 'root', 'root');
$channel = $connection->channel();
$channel->queue_declare('my_queue', false, false, false, false);
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$callback = function($msg) {
echo " [x] Received ", $msg->body, "\n";
};
$channel->basic_consume('my_queue', '', false, true, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
?>
I opened a screen for the receiver script and fired off a few executions of the sender script:
$ php sender.php
Sent 'Hello World!'
The sender script didn't run into any errors (and I was able to verify that the queue was declared in rabbit), but the receiver did not output that it had received / consumed any messages.
Furthermore, a quick check using the admin manager plugin shows that the queue has no messages at all:
$ curl -i -u root:root http://localhost:15672/api/queues
...
{
"messages_details":{
"rate":0.0
},
"messages":0,
"messages_unacknowledged_details":{
"rate":0.0
},
"messages_unacknowledged":0,
"messages_ready_details":{
"rate":0.0
},
"messages_ready":0,
"reductions_details":{
"rate":0.0
},
"reductions":9294,
"node":"rabbit#leon",
"arguments":{
},
"exclusive":false,
"auto_delete":false,
"durable":false,
"vhost":"/",
"name":"my_queue",
"message_bytes_paged_out":0,
"messages_paged_out":0,
"backing_queue_status":{
"avg_ack_egress_rate":0.0,
"avg_ack_ingress_rate":0.0,
"avg_egress_rate":0.0,
"avg_ingress_rate":0.0,
"delta":[
"delta",
"undefined",
0,
0,
"undefined"
],
"len":0,
"mode":"default",
"next_seq_id":0,
"q1":0,
"q2":0,
"q3":0,
"q4":0,
"target_ram_count":"infinity"
},
"head_message_timestamp":null,
"message_bytes_persistent":0,
"message_bytes_ram":0,
"message_bytes_unacknowledged":0,
"message_bytes_ready":0,
"message_bytes":0,
"messages_persistent":0,
"messages_unacknowledged_ram":0,
"messages_ready_ram":0,
"messages_ram":0,
"garbage_collection":{
"minor_gcs":12,
"fullsweep_after":65535,
"min_heap_size":233,
"min_bin_vheap_size":46422,
"max_heap_size":0
},
"state":"running",
"recoverable_slaves":null,
"consumers":0,
"exclusive_consumer_tag":null,
"effective_policy_definition":[
],
"operator_policy":null,
"policy":null,
"consumer_utilisation":null,
"idle_since":"2018-01-28 15:21:22",
"memory":9640
},
...
It looks like the message is accepted but is immediately discarded & not logged. Speaking of logs, there's nothing suspicious in the rabbit log either. Just a lot of this:
2018-01-28 15:47:43.654 [info] <0.1417.0> accepting AMQP connection <0.1417.0> ([::1]:49058 -> [::1]:5672)
2018-01-28 15:47:43.696 [info] <0.1417.0> connection <0.1417.0> ([::1]:49058 -> [::1]:5672): user 'root' authenticated and granted access to vhost '/'
2018-01-28 15:47:43.742 [info] <0.1417.0> closing AMQP connection <0.1417.0> ([::1]:49058 -> [::1]:5672, vhost: '/', user: 'root')
Why aren't the messages coming through?
Your code seems perfectly fine except for that you have sent the message to a wrong queue. When you send a message to a queue that doesn't exist, RabbitMQ will simply discard the message since it doesn't know where to send it.
In your code, you use the default exchange to send the message. That is:
$channel->basic_publish($msg, '', 'hello');
Here we use the default or nameless exchange: messages are routed to
the queue with the name specified by routing_key, if it exists. The
routing key is the third argument to basic_publish
https://www.rabbitmq.com/tutorials/tutorial-three-php.html
so when you use the default exchange, you have to specify the routing key as the third parameter, which is your queue name. RabbitMQ creates routing keys with the same name as queues when you use the default exchange.
To fix your code, simply change hello to your queue name, i.e my_queue and it will start sending and receiving.
Hope it helps :)
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
Since you did not bind my_queue to any exchange, you must publish to the default exchange using my_queue as the routing key. All queues are bound to the default topic exchange using the queue name as the routing key.
In your code, you are using hello as the routing key.
How can I return message back to the queue if processing result did not suit me. Found only information about message acknowledgments but I think that it does not suit me. I need that if as a result of processing I get the parameter RETRY message is added back to the queue. And then this worker or another one picks it up again and tries to process it.
For example:
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$connection = new AMQPStreamConnection($AMQP);
$channel = $connection->channel();
$channel->queue_declare('test', false, false, false, false);
$callback = function($msg) {
$condition = json_decode($msg->body);
if (!$condition) {
# return to the queue
}
};
$channel->basic_consume('test', '', false, true, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
?>
set auto no_ack flag to false
queue: Queue from where to get the messages
consumer_tag: Consumer identifier
no_local: Don't receive messages published by this consumer.
no_ack: Tells the server if the consumer will acknowledge the messages.
exclusive: Request exclusive consumer access, meaning only this consumer can access the queue
nowait:
callback: A PHP Callback
$channel->basic_consume('test', '', false, false, false, false, $callback);
you must use acknowledgments , if your proccess not work you can ignore ack
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$connection = new AMQPStreamConnection($AMQP);
$channel = $connection->channel();
$channel->queue_declare('test', false, false, false, false);
$callback = function($message) {
$condition = json_decode($message->body);
if (!$condition) {
// return to the queue
$message->delivery_info['channel']->basic_nack($message->delivery_info['delivery_tag']);
}else{
// send ack , remove from queue
$message->delivery_info['channel']->basic_ack($message->delivery_info['delivery_tag']);
}
};
$channel->basic_consume('test', '', false, false, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
Of course with this approach you will face with the message always in the head of the queue, there is also another possibility,
if you really want to have a track of retry you can follow the below approach
defining a queue for retry, preferably your queue-name -retry and define a dead-letter queue preferably: -dlq
Then you can do something like below:
How to set up -retry queue:
this is the most important part of it. you need to declare queue with the following features:
x-dead-letter-exchange: should be same as your main queue routing key
x-dead-letter-routing-key: should be same as your main queue routing key
x-message-ttl: the delay between retries
the codes are sudo code, please do not copy-paste, this is just a hint to give you the idea about it
$maximumRetry = 5;
$callback = function($message) {
$body = json_decode($message->body);
try {
// process result is your condition
$message->delivery_info['channel']->basic_ack($message->delivery_info['delivery_tag']);
} catch(Exception $e) {
// return to the queue
$body['try_attempt'] = !empty($body['try_attempt'])? int($body['try_attempt']) + 1: 1
if ($body['try_attempt'] >= $maximumRetry ){
$message->delivery_info['channel']->basic_nack($message->delivery_info['delivery_tag']);
return
}
$msg = new AMQPMessage(json_encode($message));
$channel->basic_publish($msg, '', 'test-retry');
}
};
We gonna need 3 queues for retying.
queue.example
bindings:
exchange: queue.exchange
routing: queue.example
features:
x-dead-letter-exchange: queue.exchange
x-dead-letter-routing-key: queue.example-dlq
queue.example-dlq
bindings:
exchange: queue.exchange
routing: queue.example-dlq
queue.example-retry
bindings:
exchange: queue.exchange
routing: queue.example-retry
features:
x-dead-letter-exchange: queue.exchange
x-dead-letter-routing-key: queue.example-added
x-message-ttl: 10000
------------- Update -------------
Quorum queues are providing ability out of the box so in the consumer, you can understand how many times each message was retried and you can also define a dead-letter queue for it easily, for more information you can read more about quorom queues and poison message handling
The solution turned out to be easier than I thought, it turns out the task was not specifically about RabbitMQ, but about the scope of variables. If anyone is interested in a solution, here:
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$connection = new AMQPStreamConnection($AMQP);
$channel = $connection->channel();
$channel->queue_declare('test', false, false, false, false);
$callback = function($msg) {
global $channel;
$condition = json_decode($msg->body);
if (!$condition) {
$msg = new AMQPMessage(json_encode(array(
'condition' => false
)));
$channel->basic_publish($msg, '', 'test');
}
};
$channel->basic_consume('test', '', false, true, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
?>
This may be a bit late, but this is how you should do it with this version of php-amqplib "php-amqplib/php-amqplib": "^3.1" You need to set the no_ack parameter of basic consume method to false(which is the default) then explicitly specify it in the callback using the method nack on the AMQPMessage object passed to the callback
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$connection = new AMQPStreamConnection($AMQP);
$channel = $connection->channel();
$channel->queue_declare('test', false, false, false, false);
$callback = function($msg) {
$condition = json_decode($msg->body);
if (!$condition) {
// message will be added back to the queue
$msg->nack(true);
}
};
$channel->basic_consume('test', '', false, false, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
?>
Currently, I'm using https://github.com/php-amqplib/php-amqplib and I've read a lot of examples in this repository but I still don't understand how to get all the messages from the queue?
I just need to receive some messages, group them by value and perform an action.
Is it possible to do with RabbitMQ at all?
How can I implement this in php?
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
$connection = new AMQPStreamConnection('rabbitmq', 5672, 'guest', 'guest');
$channel = $connection->channel();
$queueName = 'task_queue';
$channel->queue_declare($queueName, false, true, false, false);
$result = $channel->basic_get($queueName);
var_dump($result);
$channel->close();
$connection->close();
You cannot get all messages currently available by calling single method.
Closest possible solution is by using basic_consume method.
For example:
function process_message($message)
{
echo "Received message '" . $message->body . "'\n";
/** Do your grouping here **/
}
$channel->basic_consume($queue, '', false, false, false, false, 'process_message');
// Loop as long as the channel has callbacks registered
while (count($channel->callbacks)) {
$channel->wait();
}
You can check official RabbitMQ PHP tutorial or demo from php-amqplib.