Currently, I'm using https://github.com/php-amqplib/php-amqplib and I've read a lot of examples in this repository but I still don't understand how to get all the messages from the queue?
I just need to receive some messages, group them by value and perform an action.
Is it possible to do with RabbitMQ at all?
How can I implement this in php?
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
$connection = new AMQPStreamConnection('rabbitmq', 5672, 'guest', 'guest');
$channel = $connection->channel();
$queueName = 'task_queue';
$channel->queue_declare($queueName, false, true, false, false);
$result = $channel->basic_get($queueName);
var_dump($result);
$channel->close();
$connection->close();
You cannot get all messages currently available by calling single method.
Closest possible solution is by using basic_consume method.
For example:
function process_message($message)
{
echo "Received message '" . $message->body . "'\n";
/** Do your grouping here **/
}
$channel->basic_consume($queue, '', false, false, false, false, 'process_message');
// Loop as long as the channel has callbacks registered
while (count($channel->callbacks)) {
$channel->wait();
}
You can check official RabbitMQ PHP tutorial or demo from php-amqplib.
Related
We are using RabbitMQ v3.7.5, AMQP 0.9.1, Erlang 20.1.7.1, PHP 7, centos 7
Our consumers are written using php-amqplib v2.9.1
We start our consumers by calling a URL using curl. The consumers work as expected for sometime, but after sometime all consumers gets doubled i.e it shows two connection running for same consumer. Once this happens, none of the consumers respond when there is data on the queue but all the data received on the queue is lost.
Has anyone faced the same issue?
Consumer Code (I am using the same code given on the official site: https://www.rabbitmq.com/tutorials/tutorial-two-php.html)
require_once __DIR__ . '/vendor/autoload.php';
use PhpAmqpLib\Connection\AMQPStreamConnection;
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('task_queue', false, true, false, false);
echo " [*] Waiting for messages. To exit press CTRL+C\n";
$callback = function ($msg) {
echo ' [x] Received ', $msg->body, "\n";
sleep(substr_count($msg->body, '.'));
echo " [x] Done\n";
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
};
$channel->basic_qos(null, 1, null);
$channel->basic_consume('task_queue', '', false, false, false, false, $callback);
while (count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
Expected Result: The consumer should not be duplicated.
I'm trying to send some data from php application to the user's browser using websockets. Therefore I've decided to use Swoole in combination with RabbitMQ.
It's the first time I'm working with websockets and after reading some posts about Socket.IO, Ratchet, etc. I've decided to halt on Swoole because it's written in C and handy to use with php.
This is how I understood the idea of enabling data transfer using websockets:
1) Start RabbitMQ worker and Swoole server in CLI
2) php application sends data to RabbitMQ
3) RabbitMQ sends message with data to worker
4) Worker receives message with data + establishes socket connection with Swoole socket server.
5) Swoole server broadcasts data to all connections
The question is how to bind Swoole socket server with RabbitMQ? Or how to make RabbitMQ to establish connection with Swoole and send data to it?
Here is the code:
Swoole server (swoole_sever.php)
$server = new \swoole_websocket_server("0.0.0.0", 2345, SWOOLE_BASE);
$server->on('open', function(\Swoole\Websocket\Server $server, $req)
{
echo "connection open: {$req->fd}\n";
});
$server->on('message', function($server, \Swoole\Websocket\Frame $frame)
{
echo "received message: {$frame->data}\n";
$server->push($frame->fd, json_encode(["hello", "world"]));
});
$server->on('close', function($server, $fd)
{
echo "connection close: {$fd}\n";
});
$server->start();
Worker which receives message from RabbitMQ, then makes connection to Swoole and broadcasts the message via socket connection (worker.php)
$connection = new AMQPStreamConnection('0.0.0.0', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('task_queue', false, true, false, false);
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$callback = function($msg){
echo " [x] Received ", $msg->body, "\n";
sleep(substr_count($msg->body, '.'));
echo " [x] Done", "\n";
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
// Here I'm trying to make connection to Swoole server and sernd data
$cli = new \swoole_http_client('0.0.0.0', 2345);
$cli->on('message', function ($_cli, $frame) {
var_dump($frame);
});
$cli->upgrade('/', function($cli)
{
$cli->push('This is the message to send to Swoole server');
$cli->close();
});
};
$channel->basic_qos(null, 1, null);
$channel->basic_consume('task_queue', '', false, false, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
New task where the message will be send to RabbitMQ (new_task.php):
$connection = new AMQPStreamConnection('0.0.0.0', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('task_queue', false, true, false, false);
$data = implode(' ', array_slice($argv, 1));
if(empty($data)) $data = "Hello World!";
$msg = new AMQPMessage($data,
array('delivery_mode' => AMQPMessage::DELIVERY_MODE_PERSISTENT)
);
$channel->basic_publish($msg, '', 'task_queue');
echo " [x] Sent ", $data, "\n";
$channel->close();
$connection->close();
After starting both swoole server and worker I'm triggering new_task.php from command line:
php new_task.php
In command line prompt where a RabbitMQ Worker is running (worker.php) I can see that a message is delivered to the worker ("[x] Received Hello World!" message is appearing).
However in command line prompt where Swoole server is running happens nothing.
So the questions are:
1) Is the idea of this approach right?
2) What am I doing wrong?
In the callback(in worker.php) that fires when a message is received you're using swoole_http_client which is async only. This seems to results in the code never being fully executed as the callback function returns before the async code is triggered.
A synchronous method of doing the same thing will solve the problem. Here is a simple example:
$client = new WebSocketClient('0.0.0.0', 2345);
$client->connect();
$client->send('This is the message to send to Swoole server');
$recv = $client->recv();
print_r($recv);
$client->close();
Check out the WebSocketClient class and example usage at github.
You can also wrap it in a coroutine, like this:
go(function () {
$client = new WebSocketClient('0.0.0.0', 2345);
$client->connect();
$client->send('This is the message to send to Swoole server');
$recv = $client->recv();
print_r($recv);
$client->close();
});
I am trying to achieve mechanism where PHP pushes message to RabbitMQ (I don't want RabbitMQ to be directly exposed to user), RabbitMQ connects to RatchetPHP and Ratchet broadcasts it via websocket connections to users.
The issue I have is with accually making Ratchet server to simultanously listen for queue messages and transfer them further. Ratchet documentation assumes using ZeroMQ and after a long search through outdated documentations and libraries which do not have such methods anymore (eg. React\Stomp) I need fresh eyes from someone who has experience with these solutions.
What I have is pusher.php (standard example from RabbitMQ docs):
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('hello', false, false, false, false);
$msg = new AMQPMessage('Hello World!');
$channel->basic_publish($msg, '', 'hello');
echo " [x] Sent 'Hello World!'\n";
$channel->close();
$connection->close();
Just to simplify reproducing scenario I include also Chat class:
use Ratchet\ConnectionInterface;
use Ratchet\MessageComponentInterface;
class Chat implements MessageComponentInterface
{
protected $clients;
public function __construct()
{
$this->clients = new \SplObjectStorage;
}
public function onOpen(ConnectionInterface $connection)
{
// Store the new connection to send messages to later
$this->clients->attach($connection);
echo "New connection! ({$connection->resourceId})\n";
}
public function onMessage(ConnectionInterface $from, $msg)
{
$numRecv = count($this->clients) - 1;
echo sprintf('Connection %d sending message "%s" to %d other connection%s'."\n"
, $from->resourceId, $msg, $numRecv, $numRecv == 1 ? '' : 's');
foreach($this->clients as $client)
{
/** #var \SplObjectStorage $client */
if($from !== $client)
{
// The sender is not the receiver, send to each client connected
$client->send($msg);
}
}
}
public function onClose(ConnectionInterface $conn)
{
// The connection is closed, remove it, as we can no longer send it messages
$this->clients->detach($conn);
echo "Connection {$conn->resourceId} has disconnected\n";
}
public function onError(ConnectionInterface $conn, \Exception $e)
{
echo "An error has occurred: {$e->getMessage()}\n";
$conn->close();
}
}
And Ratchet server.php (standard Ratchet example and RabbitMQ receiver example):
use PhpAmqpLib\Connection\AMQPStreamConnection;
use Src\Chat;
// RABBIT_RECEIVER
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('hello', false, false, false, false);
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$callback = function ($msg)
{
echo " [x] Received ", $msg->body, "\n";
};
$channel->basic_consume('hello', '', false, true, false, false, $callback);
while(count($channel->callbacks))
{
$channel->wait();
}
$channel->close();
$connection->close();
// RABBIT_RECEIVER END
$server = new \Ratchet\App('sockets.dev');
$server->route('/', new Chat());
$server->run();
Current versions are basically 2 separate mechanisms listening for messages and they work great alone (so no issue there) except that they block each other and do not transfer messages between.
Question is how to make server.php to make RabbitMQ receive message and plug it into running Ratchet server.
I figured it out so adding answer for posterity. Solution is not fully real time but it is very close, seems to have good performance and is non-blocking for Ratchet websocket server. Solution was in Ratchet custom loop and addPeriodicTimer method.
So code for server.php should look like this:
$loop = React\EventLoop\Factory::create();
$chat = new Chat($loop);
$server = new \Ratchet\App('sockets.dev', 8080, '127.0.0.1', $loop);
$server->route('/', $chat);
$server->run();
And Chat.php class (only __constructor because rest is the same):
public function __construct(LoopInterface $loop)
{
$this->loop = $loop;
$this->clients = new \SplObjectStorage();
$this->loop->addPeriodicTimer(0, function ()
{
$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $connection->channel();
$channel->queue_declare('hello', false, false, false, false);
echo ' [*] Checking for for messages from RabbitMQ', "\n";
$max_number_messages_to_fetch_per_batch = 10000;
do
{
$message = $channel->basic_get('hello', true);
if($message)
{
foreach($this->clients as $client)
{
$client->send($message->body);
}
$max_number_messages_to_fetch_per_batch--;
}
}
while($message && $max_number_messages_to_fetch_per_batch > 0);
$channel->close();
$connection->close();
});
}
Initiating Ratchet server will attach periodic event (in example every 0 sec) which will check RabbitMQ for new messages and process them.
Just for better performance control allowing websocket to take a breath number of messages from RabbitMQ processed in one batch was limited to 10k. Processed messages are removed from queue and next batch is processed in next iteration.
You can also fine tune frequency of updates with addPeriodicTimer interval parameter. 0 seconds is the closest to real time you will get but it may not be required for your needs and can be changed to higher value.
How can I return message back to the queue if processing result did not suit me. Found only information about message acknowledgments but I think that it does not suit me. I need that if as a result of processing I get the parameter RETRY message is added back to the queue. And then this worker or another one picks it up again and tries to process it.
For example:
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$connection = new AMQPStreamConnection($AMQP);
$channel = $connection->channel();
$channel->queue_declare('test', false, false, false, false);
$callback = function($msg) {
$condition = json_decode($msg->body);
if (!$condition) {
# return to the queue
}
};
$channel->basic_consume('test', '', false, true, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
?>
set auto no_ack flag to false
queue: Queue from where to get the messages
consumer_tag: Consumer identifier
no_local: Don't receive messages published by this consumer.
no_ack: Tells the server if the consumer will acknowledge the messages.
exclusive: Request exclusive consumer access, meaning only this consumer can access the queue
nowait:
callback: A PHP Callback
$channel->basic_consume('test', '', false, false, false, false, $callback);
you must use acknowledgments , if your proccess not work you can ignore ack
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$connection = new AMQPStreamConnection($AMQP);
$channel = $connection->channel();
$channel->queue_declare('test', false, false, false, false);
$callback = function($message) {
$condition = json_decode($message->body);
if (!$condition) {
// return to the queue
$message->delivery_info['channel']->basic_nack($message->delivery_info['delivery_tag']);
}else{
// send ack , remove from queue
$message->delivery_info['channel']->basic_ack($message->delivery_info['delivery_tag']);
}
};
$channel->basic_consume('test', '', false, false, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
Of course with this approach you will face with the message always in the head of the queue, there is also another possibility,
if you really want to have a track of retry you can follow the below approach
defining a queue for retry, preferably your queue-name -retry and define a dead-letter queue preferably: -dlq
Then you can do something like below:
How to set up -retry queue:
this is the most important part of it. you need to declare queue with the following features:
x-dead-letter-exchange: should be same as your main queue routing key
x-dead-letter-routing-key: should be same as your main queue routing key
x-message-ttl: the delay between retries
the codes are sudo code, please do not copy-paste, this is just a hint to give you the idea about it
$maximumRetry = 5;
$callback = function($message) {
$body = json_decode($message->body);
try {
// process result is your condition
$message->delivery_info['channel']->basic_ack($message->delivery_info['delivery_tag']);
} catch(Exception $e) {
// return to the queue
$body['try_attempt'] = !empty($body['try_attempt'])? int($body['try_attempt']) + 1: 1
if ($body['try_attempt'] >= $maximumRetry ){
$message->delivery_info['channel']->basic_nack($message->delivery_info['delivery_tag']);
return
}
$msg = new AMQPMessage(json_encode($message));
$channel->basic_publish($msg, '', 'test-retry');
}
};
We gonna need 3 queues for retying.
queue.example
bindings:
exchange: queue.exchange
routing: queue.example
features:
x-dead-letter-exchange: queue.exchange
x-dead-letter-routing-key: queue.example-dlq
queue.example-dlq
bindings:
exchange: queue.exchange
routing: queue.example-dlq
queue.example-retry
bindings:
exchange: queue.exchange
routing: queue.example-retry
features:
x-dead-letter-exchange: queue.exchange
x-dead-letter-routing-key: queue.example-added
x-message-ttl: 10000
------------- Update -------------
Quorum queues are providing ability out of the box so in the consumer, you can understand how many times each message was retried and you can also define a dead-letter queue for it easily, for more information you can read more about quorom queues and poison message handling
The solution turned out to be easier than I thought, it turns out the task was not specifically about RabbitMQ, but about the scope of variables. If anyone is interested in a solution, here:
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$connection = new AMQPStreamConnection($AMQP);
$channel = $connection->channel();
$channel->queue_declare('test', false, false, false, false);
$callback = function($msg) {
global $channel;
$condition = json_decode($msg->body);
if (!$condition) {
$msg = new AMQPMessage(json_encode(array(
'condition' => false
)));
$channel->basic_publish($msg, '', 'test');
}
};
$channel->basic_consume('test', '', false, true, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
?>
This may be a bit late, but this is how you should do it with this version of php-amqplib "php-amqplib/php-amqplib": "^3.1" You need to set the no_ack parameter of basic consume method to false(which is the default) then explicitly specify it in the callback using the method nack on the AMQPMessage object passed to the callback
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
echo ' [*] Waiting for messages. To exit press CTRL+C', "\n";
$connection = new AMQPStreamConnection($AMQP);
$channel = $connection->channel();
$channel->queue_declare('test', false, false, false, false);
$callback = function($msg) {
$condition = json_decode($msg->body);
if (!$condition) {
// message will be added back to the queue
$msg->nack(true);
}
};
$channel->basic_consume('test', '', false, false, false, false, $callback);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
?>
I create RabbitMQ listener:
$connection = new AMQPConnection(
$AMQP_config['server'],
$AMQP_config['port'],
$AMQP_config['user'],
$AMQP_config['password'],
$AMQP_config['virtual_host']
);
$channel = $connection->channel();
$channel->basic_qos(
null,
1,
null
);
$channel->basic_consume(
$AMQP_config['queue'],
'',
false,
false,
false,
false,
array($this, 'CallbackResponse')
);
while(count($channel->callbacks)) {
$channel->wait();
}
$channel->close();
$connection->close();
public function CallbackResponse(AMQPMessage $msg)
{
$response = json_decode($msg->body)->acopMessage;
if ($response->reqMRef == $this->_request_reference) {
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
}
}
I have 5 messages on my RabbitMQ server. But I receive only one callback, only one entering into CallbackResponse().
I want to check all messages from the queue, find the one I've sent, read it, so there will be 4 messages left.
What I doing wrong, why I receive only first message?
Using this: https://github.com/videlalvaro/php-amqplib
Your QoS is set to 1. So RabbitMQ will only send one message at a time.
As you only ack the message you are expecting for, the first message you received which does not match you condition remains unack. Thus, RabbitMQ will not send new message.
So messages that do not match must be unacked and requeued