I am currently building a small application that uses the message queue built in PHP.
I have 1 "server" process and 1 "client" process. Messages flow from server to client.
They are simple JSON objects, that are serialised, then send.
This code is used
<?php
$send = msg_send($q, MESSAGE_TYPE_EXECUTION, $update, true, false, $error);
if (isset($error) && $error != 0) {
echo 'Execution error: ' . $error . PHP_EOL;
}
// $q is the message queue integer
// MESSAGE_TYPE_EXECUTION is integer 1
// $update is the JSON string
// true is that the JSON string is serialised
// false is that it is blocking (which it is not)
// $error get's filled when an error occurs (see below)
This works without issue, until it does not.
Sometimes after a couple of minutes, sometimes after a couple of hours the following error appears:
PHP Warning: msg_send(): msgsnd failed: Resource temporarily unavailable in
/var/www/server.php on line 57
The value of the $error variable is the integer 11.
All messages that follow this error will have error 11, until I restart the process and all is working again (for a while, until the same error appears again)
I have been searching but cannot find any explanation what error 11 is, how this can be managed and fixed without restarting the process.
Any clue, information, example etc is welcome. I would really like for server.php to be reliable.
-- edit --
client.php is the process that fetches the messages (which are all more or less the same, but with other values)
it uses this fetch the messages from the queue (filled in server.php)
<?php
$update = msg_receive($q, 0, $messagetype, 1024, $message, true, MSG_IPC_NOWAIT && MSG_NOERROR, $error);
if ($update) {
// Do stuff
}
usleep(1000000);
I have not yet checked memory usage, will look into that
Platform used
PHP 7.1.3
Centos 7
So, Solution was found after some information and leads (read the comments on my original question), brought up by #ChrisHaas (Thanks again!). After some tinkering all is running smoothly now, without error 11 for msg_send().
PHP msg_send() call is basically a wrapper of msgsnd
So a lot of information can be found there, also about errors you might encounter (in combination with flags used when reading messages with msg_receive() )
The queue is limited in total size and total messages it can hold (I, however, have not found a way to increase the total size of the queue).
The reason I was getting error 11 was due to a couple of things:
The client I created was too slow fetching messages from the queue, causing it to run into the max limit and crapping out. I did not find a solution for fixing this situation, other than restarting all processes involved. To repeat the same over and over again.
I also increased the size of reading messages in msg_receive() as sometimes the messages where big (most where small). But when you declare a too small size the big messages will remain in queue and clog it up until it craps out. Increasing the max_size helped with fetching the bigger messages too.
Long story short: error 11 is related to a full message queue in my perspective (I still do not have a 100% clear documented answer though).
Pointers to fix the issue:
Be sure you fetch all messages that are big.
Be sure to read out at least as fast as you send the messages in the queue.
Check your queue(s) with the command ipcs -q in the terminal. It allows you to see the queues currently active. Keeping an eye on that allows you to see it slowly filling up on problems.
Wish the documentation on php.net was better in this case...
I have a script in a folder that has no relevant execution time limits (I regularly run hour-long scripts from this directory) with a file in it. This file is called from an html form on a codeingiter view and uploads a csv file, reads it and updates database entries with the information contained within.
If I have a smaller csv file (1-1000 entries), it works fine. Anything over that will process about 1000 entries (including echoing debug text onto the screen) and THEN throw a 404 error.
I've since attempted to circumvent the error by making the post page make an ajax call but it does the exact same thing.
"NetworkError: 404 Not Found - xxxx/xxxx/xxxxx.php"
I have confirmed that the script is running prior to the 404 by confirming changes in the database and seeing echo'd debug code on the screen.
Code that is executing when it throws the 404...
(in short, update id (first column) to have value (second column) from csv file.
if(file_exists($targetdir))
{
$file = fopen($targetdir, 'r');
$count = 0;
while(!feof($file))
{
$x = (fgetcsv($file));
if(is_numeric($x[1]) && $x[1] > 9999 && $x[0] != '-')
{
$count++;
// echo "UPDATE xxxx SET xxxxx ='".$x[0]."' WHERE id=".$x[1];
$db->query("UPDATE xxxx SET xxxxx ='".$x[0]."' WHERE id=".$x[1]);
//echo $x[1].' - '.$x[0]."<br/>";
}
}
fclose($file);
}
Edit
It looks like Jquery is throwing the error? I googled around and saw people say it's a server setting (but not what server setting, so for the heck of it, I set the max execution time to 300s and have the same results).
xxxxx.php 404 xhr jquery-1.12.4.min.js:4 1.6 KB 40.07
Image of 404 rendering after text is echo'd
I was able to find the setting that was tripping.
Rather than use the standard php max execution time, the timeout was getting hit from the FastCGI settings. Either from AJAX posting or from the read operation.
Either way, I increased FcgidIOTimeout in my apache settings on the server and was able to execute the full script.
I have a php application running on IIS 7.5 , php 5.4 running as fastcgi. The application works absolutley fine with the exception that long running php scripts seem to hang; no 500 error they simply seem never complete and return the results to the browser.
I've written a simple test script below to eliminate the possibility of programming error in the main app :
<?php
/* test timeout */
/*set_time_limit(110);*/
echo "Testing time out in seconds\n";
for ($i = 0; $i < 175; $i++) {
echo $i." -- ";
if(sleep(1)!=0)
{
echo "sleep failed script terminating";
break;
}
}
?>
If I run the script beyond 175 seconds it hangs. Below that it will return the results to the browser.
Here are the time out parameters that I've set for php and fastcgi. I've also played around setting these really low in order to get various time out errors and have succeeded which brings me to the conclusion that there's another setting that I'm missing .. perhaps.
Correct me if my presumptions are wrong but would it be fair to say that if a script times out the browser will get an error where as if a script is killed off by IIS before completion then nothing is returned to the browser and you get a what looks like a hang as far as the browser is concerned?
fastcgi
activity timeout=800 Idle Timeout = 900 request Timeout 800
Php
max_execution_time=700
If a long running script doesn't communicate with the browser , after 180 seconds or so most browsers will become unresponsive to the server returning the results. The server script wasn't hanging or being terminated, it was the browsers (ie,ff and chrome) becoming unresponsive.
To check this , I ran my script and watched the status of the request.
IIS manager-> select the server-> select worker processes(central pane)->select the application pool-> select view requests (right hand pane) and watched the status and time elapsed columns. You will have to repeatedly click "show all" to see the values updating.
The state changed from ExecuterequestHandler to Sending response and then the script finished as it should but the browsers still looked like they were waiting for the server to respond.
I updated my test script from the above to this to ensure the browsers were being fed responses regularly:
<?php
#ini_set("output_buffering", "Off");
#ini_set('implicit_flush', 1);
#ini_set('zlib.output_compression', 0);
#ini_set('max_execution_time', 800);
header( 'Content-type: text/html; charset=utf-8' );
echo "Testing time out in seconds\n";
for ($i = 0; $i < 600; $i++) {
echo $i." -- ";
if(sleep(1)!=0)
{
echo "sleep failed script terminating";
break;
}
flush();
ob_flush();
}
?>
The output was't coming back to the browsers bit buy bit as it should and the problem remained.
Next step , I looked at response buffering on the server. The setting was set to a very high number and meant that flushing wouldn't work. So I set ResponseBufferLimit to 0 as per the instructions provided by #Dario in PHP flush stopped flushing in IIS7.5
This solved the problem :) If this solution has helped you please visit the above question and give Dario another +1 from me please, and maybe one to the OP for his question and script.
Thanks
I wrote a script downloading a list of pages from a website. From time to time I receive the following error (the number of seconds is variable):
The bwshare module will refuse your requests for the next 7 seconds.
You have downloaded data too rapidly.
I found when using sleep(2) in the loop, it works much better, however the time delay is too expensive.
What's the best way how to deal with this module? Should I scrape it without any delay and if the response will be similar to the above message simply use sleep for the requested number of seconds?
It all depends on how many pages you can get before the error message.
Try and measure how many pages in average you can get.
4 pages before the bwshare message is the minimum.
If you are getting the error message before reaching 4 page downloads, then il would be faster to sleep(2) after each download.
try this way... it might help u.
$requestTime = 0.1; // s/connection
foreach(/* blah */) {
$start = microtime(true);
// Do your stuff to here.. get_file_content($url) and other processing .........
if($timeTaken = microtime(true)-$start < $requestTime) {
usleep(($requestTime-$timeTaken)*1000000);
}
}
if your problem is solved then try to post your answer so that other people may also be benefited
I have a hefty PHP script.
So much so that I have had to do
ini_set('memory_limit', '3000M');
set_time_limit (0);
It runs fine on one server, but on another I get: Out of memory (allocated 1653342208) (tried to allocate 71 bytes) in /home/writeabo/public_html/propturk/feedgenerator/simple_html_dom.php on line 848
Both are on the same package from the same host, but different servers.
Above Problem solved new problem below for bounty
Update: The script is so big because it rawls a site and parsers data from 252 pages, including over 60,000 images, which it makes two copies of. I have since broken it down into parts.
I have another problem now though. when I am writing the image from outside site to server like this:
try {
$imgcont = file_get_contents($va); // $va is an img src from an array of thousands of srcs
$h = fopen($writeTo,'w');
fwrite($h,$imgcont);
fclose($h);
} catch(Exception $e) {
$error .= (!isset($error)) ? "error with <img src='" . $va . "' />" : "<br/>And <img src='" . $va . "' />";
}
All of a sudden it goes to a 500 internal server error page and I have to do it again, at which point it works, because files are only copied it they don't already exist. Is there anyway I can receive the 500 response code and send it back it to the url to make it go again? As this is to all be an automated process?
If this is memory related, I would personally use copy() rather than file_get_contents(). It supports the file wrappers the same way, and I don't see any advantage in loading the whole file in memory just to write it back on the filesystem.
Otherwise, your error_log might give you more information as of why the 500 happens.
There are three parties involved here:
Remote - The server(s) that contain the images you're after
Server - The computer that is running your php script
Client - Your home computer if you are running the script from a web browser, or the same computer as the server if you are running it from Cron.
Is the 500 error you are seeing being generated by 'Remote' and seen by 'Server' (i.e. the images are temporarily unavailable);
Or is it being generated by 'Server' and seen by 'Client' (i.e. there is a problem with your script).
If it is being generated by 'Remote', then see Ali's answer for how to retry.
If it is being generated by your script on 'Server', then you need to identify exactly what the error is - the php error logs should give you more information. I can think of two likely causes:
Reaching PHP's time limit. PHP will only spend a certain amount of time working before returning a 500 error. You can set this to a higher value, or regularly re-set the timer with a call to set_time_limit(), but that won't work if your server is configured in safe mode.
Reaching PHP's memory limit. You seem to have encoutered this already, but worth making sure you're script still isn't eating lots of memory. Consider outputing debug data (possibly only if you set $config['debug_mode'] = true or something). I'd suggest:
try {
echo 'Getting '.$va.'...';
$imgcont = file_get_contents($va); // $va is an img src from an array of thousands of srcs
$h = fopen($writeTo,'w');
fwrite($h,$imgcont);
fclose($h);
echo 'saved. Memory usage: '.(memory_get_usage() / (1024 * 1024)).' <br />';
unset($imgcont);
} catch(Exception $e) {
$error .= (!isset($error)) ? "error with <img src='" . $va . "' />" : "<br/>And <img src='" . $va . "' />";
}
I've also added a line to remove the image from memory, incase PHP isn't doing this correctly itself (in theory that line shouldn't be necessary).
You can avoid both problems by making your script process fewer images at a time and calling it regularly - either using Cron on the server (the ideal solution, although not all shared webhosts allow this), or some software on your desktop computer. If you do this, make sure you consider what will happen if there are two copies of the script running at the same time - will they both fetch the same image at the same time?
So it sounds like you're running this process via a web browser. I'm guessing that you may be getting the 500 error from Apache timing out somehow after a certain period of time or the process dies or something funky. I would suggest you do one of the following:
A) Move the image downloading to a background process, you can run the crawl script in the browser which will write the urls of the images to be downloaded to the db or something and another script will fire up via cron and fetch all the images. You could also have this script work in batches of 100 or so at a time to keep memory consumption down
B) Call the script directly from the command line (this is really the preferred method for something like this anyway, and you should still probably separate the image fetching to another script)
C) If the command line is not an option for some reason, have your browser loaded script touch a file, and have a cron that runs every minute and looks for the file to exist. Then it fires up your script, you can have the output written to a file for you to check later or send an email when it's completed
Is there anyway I can receive the 500 response code and send it back it to the url to make it go again? As this is to all be an automated process?
Here's the simple version of how I would do it:
function getImage($va, $writeTo, $retries = 3)
{
while ($retries > 0) {
if ($imgcont = file_get_contents($va)) {
file_put_contents($writeTo, $imgcont);
return true;
}
$retries--;
}
return false;
}
This doesn't create the file unless we successfully get our image file, and will retry three times by default. You will of course need to add any require exception handling, error checking, etc.
I would definitely stop using file_get_contents() and write the files in chunks, like this:
$read = fopen($url, 'rb');
$write = fope($local, 'wb');
$chunk = 8096;
while (!feof($read)) {
fwrite($write, fread($read, $chunk));
}
fclose($fp);
This will be nicer to your server, and should hopefully solve your 500 problems. As for "catching" a 500 error, this is simply not possible. It is an irretrievable error thrown by your script and written to the client by the web server.
I'm with Swish, this is not really the kind of task that PHP is intended for - you'de be much better using some sort of server side scripting.
Is there anyway I can receive the 500 response code and send it back it to the url to make it go again?
Have you considered using another library? Fetching files from an external server seems to me more like a job for curl or ftp than file_get_content &etc. If the error is external, and you're using curl, you can detect the 500 return code and handle it appropriately without crashing. If not, then maybe you should split your program into two files - one of which fetches a single file/image, and the other that uses curl to repeatedly call the first one. Unless the 500 error means that all php execution crashes, you would be able to detect the failure and handle it.
Something like this pseudocode:
file1.php:
foreach(list_of_files as filename){
do {
x = call_curl('file2.php', filename);
}
while(x == 500);
}
file2.php:
filename=$_GET['filename'];
results = use_curl_to_get_page(filename);
echo results;
Thanks for all your input. I had seperated everything by the time I wrote this question, so the crawler, fired the image grabber, etc.
I took on board the solution to split the number of images, and that also helped.
I also added a try, catch round the file read.
This was only being called from the browser during testing, but now that it is all up and running it is going to be a cron job.
Thanks Swish and Benubird for your particularly detailed and educational answers. Unfortunately I had no cooperation with the developers on the backend where the images are coming from (long and complicated story).
Anyway, all good now so thanks. (Swish how do you call a script from the command line, my knowledge of this field is severely lacking?)