I'm trying to figure out how to use Confirms (Publisher Acknowledgements) using the php-amqplib library, which is the library recommended for using RabbitMQ with PHP at http://rabbitmq.com/.
The code itself isn't very well documented and I can't find anything that mirrors the interfaces of the Java and python interfaces that I've found.
The Java example is here.
I've grepped for the various combinations of function names that I could think of in the PHP source and haven't found anything. I'm looking for something similar to:
$channel->confirm_select();
...
$channel->wait();
It looks like https://github.com/videlalvaro/php-amqplib/blob/master/PhpAmqpLib/Helper/Protocol/Protocol091.php has support for the functionality, but I don't see how it is exposed to the user.
I was looking for this today and found that php-amqplib has implemented it.
Check out this demo for details.
Here's a snippet:
$connection = new AMQPStreamConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $connection->channel();
$channel->set_ack_handler(
function (AMQPMessage $message) {
echo "Message acked with content " . $message->body . PHP_EOL;
}
);
$channel->set_nack_handler(
function (AMQPMessage $message) {
echo "Message nacked with content " . $message->body . PHP_EOL;
}
);
/*
* bring the channel into publish confirm mode.
* if you would call $ch->tx_select() befor or after you brought the channel into this mode
* the next call to $ch->wait() would result in an exception as the publish confirm mode and transactions
* are mutually exclusive
*/
$channel->confirm_select();
/*
name: $exchange
type: fanout
passive: false // don't check if an exchange with the same name exists
durable: false // the exchange won't survive server restarts
auto_delete: true //the exchange will be deleted once the channel is closed.
*/
$channel->exchange_declare($exchange, 'fanout', false, false, true);
$i = 1;
$msg = new AMQPMessage($i, array('content_type' => 'text/plain'));
$channel->basic_publish($msg, $exchange);
/*
* watching the amqp debug output you can see that the server will ack the message with delivery tag 1 and the
* multiple flag probably set to false
*/
$channel->wait_for_pending_acks();
Apparently neither the PECL AMQP library nor the pure php-amqplib library implement the Confirms extension. The only real solution at this point is to implement two queues to simulate acknowledgements to the publisher.
There are plans to implement Confirms in both libraries, but I don't see any movement in either.
Related
Summary
I am trying to set up heartbeats with my Consumer so it can detect a dropped connection - if the broker is rebooted for example so it may reconnect to the failover.
I tested the code with the Consumer/Dispatcher running locally and the queue running in AWS and everything worked fine. However, when moving the code to AWS the Consumer sets up heartbeats with the server/broker, but the heartbeat is either never sent by the server/broker or never received by the client/Consumer. As a result the HeartbeatException is thrown as soon as the requested server heartbeat interval has passed.
I based my code on the examples in the stomp-php-examples github
Repo: https://github.com/stomp-php/stomp-php-examples
Heartbeat example: https://github.com/stomp-php/stomp-php-examples/blob/support/version-4/src/heartbeats_server.php
My next best guess to as why this isn't working is something related to the queue config as I'm using the default config provided by AWS (I think). I have googled and searched all over about config settings for heartbeats but haven't got very far as this is a new topic. Any help would be appreciated!
Setup
Amazon MQ (ActiveMQ 5.15.14)
stomp-php 5.0 (latest version as of today)
I'm happy to provide any more details of my setup.
Code
Consumer (stripped down version)
abstract class AmqConsumerAbstract
{
/** #var StatefulStomp */
protected $consumer;
const HEARTBEAT = 5000;
public function listen(): void
{
$observer = new ServerAliveObserver();
$this->client->getConnection()->getObservers()->addObserver($observer);
$this->client->setHeartbeat(0, self::HEARTBEAT); // heartbeats setup here
// Note: the heartbeat above is longer than the read-timeout below
$this->client->getConnection()->setReadTimeout(2, 0);
$this->client->connect();
$this->consumer = new StatefulStomp($this->client);
$this->consumer->subscribe(
$this->getQueueName(),
null,
'client'
);
if (!$observer->isEnabled()) {
// we never get here so I assume everything is working OK
echo 'The Server is not supporting heartbeats.');
exit(1);
} else {
echo sprintf('The Server should send us signals every %d ms.', $observer->getInterval() * 1000);
}
try {
while (true) {
$frame = $this->consumer->read(); // I assumed this line would read the heartbeat?
// there is then some logic that deals with payload and does
// $this->consumer->begin();
// $this->consumer->ack($frame);
// $this->consumer->commit();
if ($observer->isDelayed()) {
$this->echoLog('ServerAliveObserver: Server has been delayed.');
}
}
} catch (HeartbeatException $e) {
echo 'AMQ (STOMP) Error, the server failed to send us heartbeats within the defined interval: ' . $e->getMessage()
$this->reconnect();
} catch (ConnectionException $e) {
echo $e->getMessage();
$this->reconnect();
} catch (Exception $e) {
echo 'AMQ (STOMP) Queue error: ' . $e->getMessage();
exit(1);
}
}
}
I have reproduced on a single-instance broker with version 5.15.14.
I have tried switching on STOMP debugging for the broker, but it appears the uri attribute under <transportConnector> is not allowed. The config will save but pulls out the uri attribute.
<transportConnector> Elements and Their Attributes Permitted in Amazon MQ Configurations
Working with Spring XML configuration files (xsd files)
I’m trying to acknowledge a single message in ActiveMQ 5.15.9 message queue in Symfony 4 application. So far I used 3 stomp clients to achieve this:
enqueue-dev bundle
stomp-php bundle
STOMP php extension (from PECL 2.0.2)
It seems like none of them supports change of ack mode. I’m trying to iterate through all messages from queue and acknowledge only some of them. I've tried many variations even using transactions, but no solution yet. This is possible in PHP 5.3.
php-enqueue/enqueue-dev
While loop stops after first message, it doesn’t move on until I acknowledge current message. With reject method there is an option to requeue message, but ActiveMQ doesn’t support this flag.
use Enqueue\Stomp\StompConnectionFactory;
$stompFactory = new StompConnectionFactory('stomp://127.0.0.1:61613?login=admin&password=admin');
$context = $stompFactory->createContext();
$messageQueue = $context->createQueue('foo.bar');
$consumer = $context->createConsumer($messageQueue);
$consumer->setAckMode('client-individual');
while($msg = $consumer->receive(50)){
$messages[]= $msg;
}
stomp-php
Same situation like in enqueue-dev.
use Stomp\Client;
use Stomp\StatefulStomp;
$client = new Client('tcp://127.0.0.1:61613');
$client->setLogin('admin','admin');
$client->getConnection()->setReadTimeout(1);
$stomp = new StatefulStomp($client);
$stomp->subscribe('foo.bar',null,'client-individual');
while($msg = $stomp->read()) {
$messages[] = $msg;
}
STOMP extension 2.0.2
I can get all messages without acknowledging them, but when I try to acknowledge a single one in the middle, all previous ones are being acknowledged as well. (which means it’s client ack mode). The same code works on php 5.3 and STOMP 1.0.9, ‘client-individual’ is supported and I can acknowledge a single message.
$stomp = new \Stomp('tcp://localhost:61613','admin','admin');
$stomp->subscribe('foo.bar', array('ack' => 'client-individual'));
while ($msg = $stomp->readFrame()) {
$messages[] = $msg;
}
$stomp->ack($messages[1]);
Fairly new to ZeroMQ. I have a simple REQ/REP queue like below. I am using PHP but that doesn't matter as any language binding would be fine for me.
This is client to request a task
$ctx = new ZMQContext();
$req = new ZMQSocket($ctx, ZMQ::SOCKET_REQ);
$req->connect('tcp://localhost:5454');
$req->send("Export Data as Zip");
echo $i . ":" . $req->recv().PHP_EOL;
And this is a worker to actually perform the task.
$ctx = new ZMQContext();
$srvr = new ZMQSocket($ctx, ZMQ::SOCKET_REP);
$srvr->bind("tcp://*:5454");
echo "Server is started at port $port" . PHP_EOL;
while(true)
{
$msg = $srvr->recv();
echo "Message = " . $msg . PHP_EOL;
// Do the work here, takes 10 min, knows the count of lines added and remaining
$srvr->send($msg . " is exported as zip file" . date('H:i:s'));
}
As the task of exporting data takes about 10 min, I want to connect to the server from a different client and get the progress/ percentage of the task done.
I am wondering if that's even a valid approach.
I tried this approach where REQ/REP part works but I get nothing in PUB/SUB part
Server part
$ctx = new ZMQContext();
$srvr = new ZMQSocket($ctx, ZMQ::SOCKET_REP);
$srvr->bind("tcp://*:5454");
// add PUB socket to publish progress
$c = new ZMQContext();
$p = new ZMQSocket($c, ZMQ::SOCKET_PUB);
$p->bind("tcp://*:5460");
echo "Server is started at port 5454" . PHP_EOL;
$prog = 0;
while(true)
{
$p->send($prog++ . '%'); // this part doesn't get to the progress client
$msg = $srvr->recv();
echo "Message = " . $msg . PHP_EOL;
sleep(2);// some long task
$srvr->send($msg . " Done zipping " . date('H:i:s'));
}
Progress client
$ctx = new ZMQContext();
$stat = new ZMQSocket($ctx, ZMQ::SOCKET_SUB);
$stat->connect('tcp://localhost:5460');
while (true){
echo $stat->recv() . PHP_EOL; //nothing shows here
}
Request client
$ctx = new ZMQContext();
$req = new ZMQSocket($ctx, ZMQ::SOCKET_REQ);
$req->connect('tcp://localhost:5454');
for($i=0;$i<100;$i++){
$req->send("$i : Zip the file please");
echo $i . ":" . $req->recv().PHP_EOL; //works and get the output
}
The concept is feasible, some tuning needed:
All PUB counterparties have to setup any non-default subscription, via, at least an empty subscription .setsockopt( ZMQ_SUBSCRIBE, "" ) meaning receive all TOPICs ( none "filter"-ed out ).
Next, both PUB-side and SUB sides ought get .setsockopt( ZMQ_CONFLATE, 1 ) configured, as there is of no value to populate and feed all intermediate values into the en-queue/de-queue pipeline, once the only value is in the "last", most recent message.
Always, the non-blocking mode of the ZeroMQ calls ought be preferred ( .recv( ..., flags = ZMQ_NOBLOCK ) et al ) or the Poller.poll() pre-tests ought be used to sniff first for a (non)-presence of a message, before spending more efforts on reading its context "from" ZeroMQ context-manager. Simply put, there are not many cases, where blocking-mode service calls may serve well in a production-grade system.
Also some further tweaking may help the PUB side, in case a more massive "attack" comes from the un-restricted pool of SUB-side entities and PUB has to create / manage / maintain resources for each of these ( unrestricted ) counterparties.
You need only use PUB/SUB if there is more than one client wanting to receive the same progress updates. Just use PUSH/PULL for a simple, point to point transfer that works over tcp.
Philosophical Discussion
With problems such as this to solve there's two approaches.
Use additional sockets to convey additional message types,
Use just two sockets, but convey more than one message type through them
You're talking about doing 1). It might be worth contemplating 2), though I must emphasise that I know next to nothing of PHP and so don't know if there are language features that encourage one to have separate request and progress clients.
If you do, your original client needs a loop (after it has sent the request) to receive multiple messages, either progress update messages or the final result. Your server, whilst it is doing its 10 minute lookup, will send regular progress update messages, and the final result message at the end. You would probably use PUSH/PULL client to server, and the same again for the progress / result from the server back to the client.
It is architecturally more flexible to follow 2). Once you have a means of sending two or more message types through a single socket and of decoding them at the receiving end, you can send more. For example, you could decide to add a 'cancel' message from the client to the server, or a partial results message from the server back to the client. This is much easier to extend than to keep adding more sockets to your architecture simply because you want to add another message flow between the client and server. Again, I don't know enough about PHP to say that this would definitely be the right way of doing it in that language. It certainly makes a lot of sense in C, C++.
I find things like Google Protocol Buffers (I prefer ASN.1) very useful for this kind of thing. These allow you to define the types of messages you want to send, and (at least with GPB), combine them together inside a single 'oneof' (in ASN.1 one uses tagging to tell different messages apart). GPB and ASN.1 are handy because then you can use different languages, OSes and platforms in your system without really having to worry about what it is being sent. And being binary (not text) they're more efficient across network connections.
I'm working on trace logger of sorts that pushes log message requests onto a Queue on a Service Bus, to later be picked off by a worker role which would insert them into the table store. While running on my machine, this works just fine (since I'm the only one using it), but once I put it up on a server to test, it produced the following error:
HTTP_Request2_MessageException: Malformed response: in D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php on line 1013
0 HTTP_Request2_Response->__construct('', true, Object(Net_URL2)) D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php:1013
1 HTTP_Request2_Adapter_Socket->readResponse() D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php:139
2 HTTP_Request2_Adapter_Socket->sendRequest(Object(HTTP_Request2)) D:\home\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2.php:939
3 HTTP_Request2->send() D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\Http\HttpClient.php:262
4 WindowsAzure\Common\Internal\Http\HttpClient->send(Array, Object(WindowsAzure\Common\Internal\Http\Url)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\RestProxy.php:141
5 WindowsAzure\Common\Internal\RestProxy->sendContext(Object(WindowsAzure\Common\Internal\Http\HttpCallContext)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\Common\Internal\ServiceRestProxy.php:86
6 WindowsAzure\Common\Internal\ServiceRestProxy->sendContext(Object(WindowsAzure\Common\Internal\Http\HttpCallContext)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\ServiceBus\ServiceBusRestProxy.php:139
7 WindowsAzure\ServiceBus\ServiceBusRestProxy->sendMessage('<queuename>/mes…', Object(WindowsAzure\ServiceBus\Models\BrokeredMessage)) D:\home\site\wwwroot\vendor\microsoft\windowsazure\WindowsAzure\ServiceBus\ServiceBusRestProxy.php:155
⋮
I've seen previous posts that describe similar issues; Namely:
Windows Azure PHP Queue REST Proxy Limit (Stack Overflow)
Operations on HTTPS do not work correctly (GitHub)
That imply that this is a known issue regarding the PHP Azure Storage libraries, where there are a limited amount of HTTPS connections allowed. Before requirements were changed, I was accessing the table store directly, and ran into this same issue, and fixed it in the way the first link describes.
The problem is that the Service Bus endpoint in the connection string, unlike Table Store (etc.) connection string endpoints, MUST be 'HTTPS'. Trying to use it with 'HTTP' will return a 400 - Bad Request error.
I was wondering if anyone had any ideas on a potential workaround. Any advice would be greatly appreciated.
Thanks!
EDIT (After Gary Liu's Comment):
Here's the code I use to add items to the queue:
private function logToAzureSB($source, $msg, $severity, $machine)
{
// Gather all relevant information
$msgInfo = array(
"Severity" => $severity,
"Message" => $msg,
"Machine" => $machine,
"Source" => $source
);
// Encode it to a JSON string, and add it to a Brokered message.
$encoded = json_encode($msgInfo);
$message = new BrokeredMessage($encoded);
$message->setContentType("application/json");
// Attempt to push the message onto the Queue
try
{
$this->sbRestProxy->sendQueueMessage($this->azureQueueName, $message);
}
catch(ServiceException $e)
{
throw new \DatabaseException($e->getMessage, $e->getCode, $e->getPrevious);
}
}
Here, $this->sbRestProxy is a Service Bus REST Proxy, set up when the logging class initializes.
On the recieving end of things, here's the code on the Worker role side of this:
public override void Run()
{
// Initiates the message pump and callback is invoked for each message that is received, calling close on the client will stop the pump.
Client.OnMessage((receivedMessage) =>
{
try
{
// Pull the Message from the recieved object.
Stream stream = receivedMessage.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
string message = reader.ReadToEnd();
LoggingMessage mMsg = JsonConvert.DeserializeObject<LoggingMessage>(message);
// Create an entry with the information given.
LogEntry entry = new LogEntry(mMsg);
// Set the Logger to the appropriate table store, and insert the entry into the table.
Logger.InsertIntoLog(entry, mMsg.Service);
}
catch
{
// Handle any message processing specific exceptions here
}
});
CompletedEvent.WaitOne();
}
Where Logging Message is a simple object that basically contains the same fields as the Message Logged in PHP (Used for JSON Deserialization), LogEntry is a TableEntity which contains these fields as well, and Logger is an instance of a Table Store Logger, set up during the worker role's OnStart method.
This was a known issue with the Windows Azure PHP, which hasn't been looked at in a long time, nor has it been fixed. In the time between when I posted this and now, We ended up writing a separate API web service for logging, and had our PHP Code send JSON strings to it over cURL, which works well enough as a temporary work around. We're moving off of PHP now, so this wont be an issue for much longer anyways.
I have next code:
<?php
// callback function for recive the message and canceling consumer
function consumer(\AMQPEnvelope $envelope, \AMQPQueue $queue)
{
$queue->ack($envelope->getDeliveryTag());
$queue->cancel($envelope->getCorrelationId());
echo "Message was recived and consumer will be canceled by consumer tag: {$envelope->getCorrelationId()}\n";
}
// generating uniqie exchange and queue
$correlationId = uniqid(str_replace('.', '', (string)microtime(TRUE)) . '_');
$queueName = "databus_response_{$correlationId}";
$consumerTag = "consumer_tag_{$correlationId}";
// establesh connection
$connection = new \AMQPConnection(array('host'=>'127.0.0.1', 'user'=>'guest', 'password'=>'guest'));
$connection->connect();
$channel = new \AMQPChannel($connection);
// declare exchange
$exchange = new \AMQPExchange($channel);
$exchange->setFlags(AMQP_AUTODELETE);
$exchange->setType(AMQP_EX_TYPE_TOPIC);
$exchange->setName($queueName);
$exchange->declareExchange();
// declare queue
$queue = new \AMQPQueue($channel);
$queue->setFlags(AMQP_EXCLUSIVE);
$queue->setName($queueName);
$queue->declareQueue();
$queue->bind($queueName, '#');
// publish message in exchange
$exchange->publish('Test message', NULL, AMQP_PASSIVE, array('correlation_id' => $consumerTag));
// run consumer for getting this echange and canceling consumer after recive the message
$queue->consume('consumer', AMQP_NOWAIT, $consumerTag);
How you can see, i send one message to queue, and run consumer on this queue. In consumer method, you can see, what i want stop consumer on this queue after reciving first message by "cancel" method, but consumer not stopped. What i do wrong?
AMQP PECL module version 1.2.0 from https://github.com/pdezwart/php-amqp/tree/v1.2.0
PHP 5.4.4-14+deb7u10 (cli)
Linux v270 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux
just return false from consumer callback when you want to stop consuming.
From AMQPQueue.php stub file:
The AMQPQueue::consume() will not return the processing thread back to
the PHP script until the callback function returns FALSE.
P.S.: sad to say that php-amqp extension still lack of good documentation, but you can always read method annotation in stub files or read a bit outdated documentation on official php site in Polish language here - http://www.php.net/manual/pl/book.amqp.php (don't worry, nobody translated it, so all sensitive doc is in English).
Nowadays in the php-amqplib v2.* world you can also call basic_cancel on your AMQPChannel object.
here is a great blog post about controlling the number of consumers and their ttl using this technique.