I want to read all messages from azure service bus (queue).
I have followed instruction from below link
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-php-how-to-use-queues
Currently it fetch one message..
I want to fetch all messages from service bus(queue).
Thanks in Advance..
I don't think it is possible to specify the number of messages you wish to read from the queue (at least with PHP SDK, it is certainly possible with .Net SDK). Essentially receiveQueueMessage is a wrapper over either Peek-Lock Message (Non-Destructive Read) or Receive and Delete Message (Destructive Read) (depending on the configuration) REST API methods and both of them return only a single message.
One way to solve this problem is to run your code in loop till the time you don't receive any message back from the queue. The issue that you could possibly run into with this is you may get back duplicate messages as once the lock acquired by peek-lock method is expired, the message will become visible again.
Reading all the messages in a Queue is possible with Peek operation concept of Service bus.
MessagingFactory messagingFactory = MessagingFactory.CreateFromConnectionString(<Your_Connection_String>);
var queueClient = messagingFactory.CreateQueueClient(<Your_Queue_Name>, ReceiveMode.PeekLock); // Receive mode is by default PeekLock and hence, optional.
BrokeredMessage message = null;
while(true)
{
message = queueClient.Peek(); // You have read one message
if (message == null) // Continue till you receive no message from the Queue
break;
}
This can also return expired or locked messages. Give a quick read about Message browsing if this suits your requirement of reading messages.
Related
I am currently building a small application that uses the message queue built in PHP.
I have 1 "server" process and 1 "client" process. Messages flow from server to client.
They are simple JSON objects, that are serialised, then send.
This code is used
<?php
$send = msg_send($q, MESSAGE_TYPE_EXECUTION, $update, true, false, $error);
if (isset($error) && $error != 0) {
echo 'Execution error: ' . $error . PHP_EOL;
}
// $q is the message queue integer
// MESSAGE_TYPE_EXECUTION is integer 1
// $update is the JSON string
// true is that the JSON string is serialised
// false is that it is blocking (which it is not)
// $error get's filled when an error occurs (see below)
This works without issue, until it does not.
Sometimes after a couple of minutes, sometimes after a couple of hours the following error appears:
PHP Warning: msg_send(): msgsnd failed: Resource temporarily unavailable in
/var/www/server.php on line 57
The value of the $error variable is the integer 11.
All messages that follow this error will have error 11, until I restart the process and all is working again (for a while, until the same error appears again)
I have been searching but cannot find any explanation what error 11 is, how this can be managed and fixed without restarting the process.
Any clue, information, example etc is welcome. I would really like for server.php to be reliable.
-- edit --
client.php is the process that fetches the messages (which are all more or less the same, but with other values)
it uses this fetch the messages from the queue (filled in server.php)
<?php
$update = msg_receive($q, 0, $messagetype, 1024, $message, true, MSG_IPC_NOWAIT && MSG_NOERROR, $error);
if ($update) {
// Do stuff
}
usleep(1000000);
I have not yet checked memory usage, will look into that
Platform used
PHP 7.1.3
Centos 7
So, Solution was found after some information and leads (read the comments on my original question), brought up by #ChrisHaas (Thanks again!). After some tinkering all is running smoothly now, without error 11 for msg_send().
PHP msg_send() call is basically a wrapper of msgsnd
So a lot of information can be found there, also about errors you might encounter (in combination with flags used when reading messages with msg_receive() )
The queue is limited in total size and total messages it can hold (I, however, have not found a way to increase the total size of the queue).
The reason I was getting error 11 was due to a couple of things:
The client I created was too slow fetching messages from the queue, causing it to run into the max limit and crapping out. I did not find a solution for fixing this situation, other than restarting all processes involved. To repeat the same over and over again.
I also increased the size of reading messages in msg_receive() as sometimes the messages where big (most where small). But when you declare a too small size the big messages will remain in queue and clog it up until it craps out. Increasing the max_size helped with fetching the bigger messages too.
Long story short: error 11 is related to a full message queue in my perspective (I still do not have a 100% clear documented answer though).
Pointers to fix the issue:
Be sure you fetch all messages that are big.
Be sure to read out at least as fast as you send the messages in the queue.
Check your queue(s) with the command ipcs -q in the terminal. It allows you to see the queues currently active. Keeping an eye on that allows you to see it slowly filling up on problems.
Wish the documentation on php.net was better in this case...
I am hitting a strange error when I attempt to call $sns->publish (PHP) - it never returns, I am not sure if it dies silently, but I could not catch an exception or get a return code.
I was able to track this down to happen when device for the token (endpoint) appears to be already disabled in the SNS console. It gets disabled on the initial call, I would assume due to the error returned by GCM that token is invalid.
What am I doing wrong and how can I prevent the problem? I do not want to check every endpoint for being enabled since I may be pushing to 10 out of 1000. However I definitely want to continue executing my push loop.
Any thoughts? AWS team forum seems useless, it has been weeks since original reply by AWS team member asking for code with not response since that time.
you can check if the endpoint is disabled before sending push notification as -
$arn_code = ARN_CODE_HERE;
$arn_arr = array("EndpointArn"=>$arn_code);
$endpointAtt = $sns->getEndpointAttributes($arn_arr);
//print_r($endpointAtt);
if($endpointAtt != 'failed' && $endpointAtt['Attributes']['Enabled'] != 'false')
{
....PUBLISH CODE HERE....
}
It will not stop the execution.
Hope it will help you.
When consuming data from a ActiveMQ queue im running into the following problem
With the following code:
$stomp = new Stomp($activeMQURI);
$stomp->subscribe($queue);
while ($stomp->hasFrame()) {
$frame = $stomp->readFrame();
if ($frame) {
$stomp->ack($frame);
}
}
It will only loop through about 1-10 messages before $stomp->hasFrame() returns false. The problem is there are 10k messages still in the queue!
When i put a delay in after the acknowledgment everything works as expected:
$stomp = new Stomp($activeMQURI);
$stomp->subscribe($queue);
while ($stomp->hasFrame()) {
$frame = $stomp->readFrame();
if ($frame) {
$stomp->ack($frame);
sleep(1);
}
}
I was thinking that this was happening because the ActiveMQ server has not had a chance to process the ack before the consumer (my code) requests another frame. Can anyone explain the real reason why this is happening, and maybe a better fix then SLEEP?
You don't really specify what client you are using so here's a general answer. Most client's provide a blocking receive call either timed or infinite wait which will return when a message arrives, or indicate failure in the timed case. The speed at which the broker is going to dispatch messages to your client depends on a great many factors such as the number of consumers on the destination, the prefetch size set by each consumer, and the speed of the network etc, etc. Your code should not expect immediate turned and be able to deal with the case where there is a lull in message traffic. That's about as good an answer as I can give since I don't know any more about your setup.
I have noticed a few websites such as hypem.com show a "You didnt get served" error message when the site is busy rather than just letting people wait, time out or refresh; aggravating what is probably a server load issue.
We are too loaded to process your request. Please click "back" in your
browser and try what you were doing again.
How is this achieved before the server becomes overloaded? It sounds like a really neat way to manage user expectation if a site happens to get overloaded whilst also giving the site time to recover.
Another options is this:
$load = sys_getloadavg();
if ($load[0] > 80) {
header('HTTP/1.1 503 Too busy, try again later');
die('Server too busy. Please try again later.');
}
I got it from php's site http://php.net/sys_getloadavg, altough I'm not sure what the values represent that the sys_getloadavg returns
You could simply create a 500.html file and have your webserver use that whenever a 50x error is thrown.
I.e. in your apache config:
ErrorDocument 500 /errors/500.html
Or use a php shutdown function to check if the request timeout (which defaults to 30s) has been reached and if so - redirect/render something static (so that rendering the error itself cannot cause problems).
Note that most sites where you'll see a "This site is taking too long to respond" message are effectively generating that message with javascript.
This may be to do with the database connection timing out, but that assumes that your server has a bigger DB load than CPU load when times get tough. If this is the case, you can make your DB connector show the message if no connection happens for 1 second.
You could also use a quick query to the logs table to find out how many hits/second there are and automatically not respond to any more after a certain point in order to preserve QOS for the others. In this case, you would have to set that level manually, based on server logs. An alternative method can be seen here in the Drupal throttle module.
Another alternative would be to use the Apache status page to get information on how many child processes are free and to throttle id there are none left as per #giltotherescue's answer to this question.
You can restrict the maximum connection in apache configuration too...
Refer
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
http://www.howtoforge.com/configuring_apache_for_maximum_performance
This is not a strictly PHP solution, but you could do like Twitter, i.e.:
serve a mostly static HTML and Javascript app from a CDN or another server of yours
the calls to the actual heavy work server-side (PHP in your case) functions/APIs are actually done in AJAX from one of your static JS files
so you can set a timeout on your AJAX calls and return a "Seems like loading tweets may take longer than expected"-like notice.
You can use the php tick function to detect when a server isn't loading for a specified amount of time, then display an error messages. Basic usage:
<?php
$connection = false;
function checkConnection( $connectionWaitingTime = 3 )
{
// check connection & time
global $time,$connection;
if( ($t = (time() - $time)) >= $connectionWaitingTime && !$connection){
echo ("<p> Server not responding for <strong>$t</strong> seconds !! </p>");
die("Connection aborted");
}
}
register_tick_function("checkConnection");
$time = time();
declare (ticks=1)
{
require 'yourapp.php' // load your main app logic
$connection = true ;
}
The while(true) is just to simulate a loaded server.
To implement the script in your site, you need to remove the while statement and add your page logic E.G dispatch event or front controller action etc.
And the $connectionWaitingTime in the checkCOnnection function is set to timeout after 3 seconds, but you can change that to whatever you want
I have a requirement that a PHP web application write messages to a non-blocking queue and other process(es) dequeue them. My current design is PHP app create a ZMQ.PUSH socket, do a connect to the destination address and send the message. While at the destination, a process (Java) create a ZMQ.PULL socket, do a bind on the same address and receive the message. However, when the dequeuer process is down (or not started), the messages that the PHP app sent during that time are lost (not delivered to the dequeuer when the process starts again on the same address). Is this an issue with the PHP application creating a new ZMQ.Context everytime?
As an example, I created another Java process to write messages to the queue.
public static void main(String[] args) {
ZMQ.Context context = ZMQ.context(1);
ZMQ.Socket socket = context.socket(ZMQ.PUSH);
socket.connect("tcp://localhost:5557");
for(int i = 0; i < 10; i++) {
socket.send(("Sending : " + message).getBytes(), 0);
}
}
This has the same problem of lost messages if the dequeuing process does not start by the time the above processes finishes.
But adding a while(true) {} to the end of the above method body does not result in any lost messages - all the messages are delivered when the dequeuer starts. So am I correct in the assumption that the ZMQ.Context object being garbage collected causes the problem here? If yes, then how to solve this problem in a PHP web application? Since I cannot persist the ZMQ.Context object.
You are looking for so-called durable messages.
Look at the zmq_setsockopt in particular the ZMQ_IDENTITY option as a first step in your solution.
Make sure you read the sections in the manual about High-Water Marks in order to not get too far behind if the reader is offline for a while.
Good Luck