I am having a script which analyses XML data and fills same arrays with information.
For some (huge) input, the script crashed.
There is a foreach loop which is run around 180 times without problems (memory_get_usage() in iteration 180 around 20 MB, each loop adds around 0.1 MB)
Then it happens that with each new loop, the memory usage just doubles.
With the use of lots of logging I was able to track the problem down to the following line in a foreach.
$fu = $f['unit']
$f has the following structure:
array (
'name' => 'Test',
'value' => '4',
'unit' => 'min-1',
)
But in some (many) cases (but also before the 180th iteration), the key unit was not existing in the array.
I was able to eliminate the problem by replacing the line with:
$fu = (isset($f['unit']) ? $f['unit'] : '');
Then the iteration runs until finished (totally 370 iterations).
Is there any explanation for the phenomena?
PHP version: PHP 5.3.3-1ubuntu9.10 with Suhosin-Patch (old...)
Your problem might come from the PHP error handler and not from your actual loop.
like you said, not every "unit" key is existing and will therefore rise an error (or Exception depending on your error handlers). This might also include a stack trace and further debugging information depending on what extensions (xdebug?) you installed.
Both will consume memory.
It's allways a good practice to check for variables existance before using it. Allways enable E_NOTICE errors in your development system to see any such problems.
Related
I have a relatively high traffic site, that about once per day generates the error message:
PHP Warning: Unknown: Input variables exceeded 1000. To increase the limit change max_input_vars in php.ini. in Unknown on line 0
My guess is that it is caused by some crawler that has found a URL link structure that never ends. However, I couldn't find any thing in the access logs that would be a problem (e.g. a url with a 1000+ get parameters).
It there an easy way to get insight into this Warning? It happens before any php scripts are loaded, so I assume it is impossible to introspect with php. If I had simple details such as the URL, it would be easy to solve.
If it only happens occassionally and potentially also affects end users it's actually rather safe to just raise the limit - it's a limitation imposed for practical reasons to circumvent possible attacks.
Practically, to debug this I'd dive into edge cases. In a real world scenario I'd indeed expect this error to only occur when something is nested indefinitely. I'd insert a small detection script somewhere in code that's always included, for example:
function detectLargeInputs($name, $array)
{
if(count($array) > 500)
mail('mymail#domain.tld', 'Large input in '.$name,
print_r($array, true).print_r($_SERVER, true));
}
detectLargeInputs('GET', $_GET);
detectLargeInputs('POST', $_POST);
detectLargeInputs('COOKIE', $_COOKIE);
This should show the problem within a day, including the $_SERVER info that also has fields like REQUEST_URI and HTTP_REFERER which should help you pinpoint the issue exactly.
Attention! This detection won't alert if the input vars are in an sub array:
Array
(
[foo] => 100
[bars] => Array
(
[0] => 1111
[1] => 1111
...(more than 1000)
)
)
The return value of 'count' will be 2 and everything seems ok but PHP won't handle the more than 1000 values in the sub array.
I have a script that sometimes break because I get an error like:
PHP Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes) in ...
The memory limit is NOT too low and was put at 128M. (This should be more than enough.)
The problem is that as the script goes, it takes more and more memory. Basically, it's a for loop. I've checked every variable but there is no place where the script fills 'something' (for example, an array) that would grow and explain why the script takes more memory.
I confirmed the memory problem using: memory_get_usage(). I've printed out the memory footprint after each iteration and the number is always higher.
Using Xdebug does show not any sign that can help me at this point.
Is there any way I could drilldown and know what is taking more memory everytime I loop ? What would you suggest to debug that situation ?
The short answer is: it's not possible to drilldown.
That said, the way I found what the problem was is that I splitted the code inside the loop in subfunctions. Then, I commented them all to only check what was the footprint of looping without doing anything. Then, I've uncommented function by function until I found the problematic one.
Once I've got the problematic function, I did the same process again: comment everything in it then uncomment until I find the problematic piece of code.
I finally found that I was calling a function that used create_function from PHP. Many people are 'complaining' about 'memory leaks' from this function. The problem is not a memory leak but instead the fact that if you call the create_function in a loop, it will really create as many functions as the number of times you loop. To avoid this, I've found this concept to avoid recreating the function thousands of times.
<?php
global $my_func;
if (!isset($my_func)) {
$my_func = create_function($args, $code);
}
$my_func();
?>
Adapting the code to make sure the function is created only once solved the problem. The whole script now takes only 8MB of memory instead of breaking after busting over 128MB.
I am struggling to understand why PHP is faulting without feedback or error (beyond windows error log, fault in php5 faulting module php) whilst executing the following block of code:
$projects = array();
$pqRes = $connection->getResult("SELECT big query");
//build an array by project id
while($record = sqlsrv_fetch_array($pqRes))
{
if(! array_key_exists($record['ProjectID'],$projects))
{
$projects[$record['ProjectID']] = array();
}
$projects[$record['ProjectID']][] = $record; //this line faults php after about 9100 records
}
The outcome is the same whether objects or arrays are pulled from the sql resource, and the offending line is the array assignment.
The assignment causes a fault in php after around 9100 records.
If this loop was counted out so execution termination is controlled I can see that php has consumed about 25mb of memory, by its configuration it is allowed 256.
The faulting record is not always the same, it can vary by 3 or 4 indexes.
The code is in fact quite pointless but in a round about way groups records of the same productID, but I am very interested to know what it could do that might cause php to die so suddenly.
Thanks for your time.
I am not sure what you mean by a fault. But if it means it makes the script terminate, I suggest enabling error log and see what the final error message says. In any case, it may be a PHP bug, so you are always recommended to report it to http://bugs.php.net/, as that is the right place where PHP developers look at eventual bugs.
I have a web application that runs fine on our Linux servers but when running on Mac OS with the Zend Community Edition Server using PHP 5.3 we get the error:
usort(): Array was modified by the user comparison function
every time a page loads for the first time (it takes about 2 minutes for a page to tick over and load, on the linux servers the page loads in 1 second).
Has anyone else experienced this or has any idea how I can fix the problem, I have tried playing around with PHP and Apache memory settings with no luck.
There is a PHP bug that can cause this warning, even if you don't change the array.
Short version, if any PHP debug functions examine the sort array, they will change the reference count and trick usort() into thinking you've changed the data.
So you will get that warning by doing any of the following in your sort function (or any code called from it):
calling var_dump or print_r on any of the sort data
calling debug_backtrace()
throwing an exception -- any exception -- or even just creating an exception
The bug affects all PHP 5 versions >= 5.2.11 but does not affect PHP >= 7. See the bug report for further details.
As far as I can see, the only workaround is either "don't do that" (which is kind of hard for exceptions), or use the error suppression operator #usort() to ignore all errors.
To resolve this issue we can handle as below
1) use error_reporting
$a = array('id' => 2,'val' => 3, 'ind' => 3);
$errorReporting = error_reporting(0);
usort($a);
error_reporting($errorReporting);
2) use #usort($a);
$a = array('id' => 2,'val' => 3, 'ind' => 3);
#usort($a);
What version of PHP is on the linux box?
Are the error_reporting levels the same on both boxes? Try setting them both to E_ALL.
The warning is almost certainly not lying. It's saying that the comparison function you're passing to usort() is changing the array that you're trying to sort - that could definitely make usort take a long time, possibly forever!
My first step would be to study the comparison function, and figure out why that's happening. It's possible that if the linux boxes are using a pre-5.3 version, there is some difference in the behavior of some language function used in the comparison function.
I found that using PHP5.4 , logging with error_log($message, $message_type, $destination, $extra_headers) causes this error , when I clean log entries my problem solved. Logging may temporarily be suspended by disabling and restoring logging after sort function.
In a PHP program, I sequentially read a bunch of files (with file_get_contents), gzdecode them, json_decode the result, analyze the contents, throw most of it away, and store about 1% in an array.
Unfortunately, with each iteration (I traverse over an array containing the filenames), there seems to be some memory lost (according to memory_get_peak_usage, about 2-10 MB each time). I have double- and triple-checked my code; I am not storing unneeded data in the loop (and the needed data hardly exceeds about 10MB overall), but I am frequently rewriting (actually, strings in an array). Apparently, PHP does not free the memory correctly, thus using more and more RAM until it hits the limit.
Is there any way to do a forced garbage collection? Or, at least, to find out where the memory is used?
it has to do with memory fragmentation.
Consider two strings, concatenated to one string. Each original must remain until the output is created. The output is longer than either input.
Therefore, a new allocation must be made to store the result of such a concatenation. The original strings are freed but they are small blocks of memory.
In a case of 'str1' . 'str2' . 'str3' . 'str4' you have several temps being created at each . -- and none of them fit in the space thats been freed up. The strings are likely not laid out in contiguous memory (that is, each string is, but the various strings are not laid end to end) due to other uses of the memory. So freeing the string creates a problem because the space can't be reused effectively. So you grow with each tmp you create. And you don't re-use anything, ever.
Using the array based implode, you create only 1 output -- exactly the length you require. Performing only 1 additional allocation. So its much more memory efficient and it doesn't suffer from the concatenation fragmentation. Same is true of python. If you need to concatenate strings, more than 1 concatenation should always be array based:
''.join(['str1','str2','str3'])
in python
implode('', array('str1', 'str2', 'str3'))
in PHP
sprintf equivalents are also fine.
The memory reported by memory_get_peak_usage is basically always the "last" bit of memory in the virtual map it had to use. So since its always growing, it reports rapid growth. As each allocation falls "at the end" of the currently used memory block.
In PHP >= 5.3.0, you can call gc_collect_cycles() to force a GC pass.
Note: You need to have zend.enable_gc enabled in your php.ini enabled, or call gc_enable() to activate the circular reference collector.
Found the solution: it was a string concatenation. I was generating the input line by line by concatenating some variables (the output is a CSV file). However, PHP seems not to free the memory used for the old copy of the string, thus effectively clobbering RAM with unused data. Switching to an array-based approach (and imploding it with commas just before fputs-ing it to the outfile) circumvented this behavior.
For some reason - not obvious to me - PHP reported the increased memory usage during json_decode calls, which mislead me to the assumption that the json_decode function was the problem.
There's a way.
I had this problem one day. I was writing from a db query into csv files - always allocated one $row, then reassigned it in the next step. Kept running out of memory. Unsetting $row didn't help; putting an 5MB string into $row first (to avoid fragmentation) didn't help; creating an array of $row-s (loading many rows into it + unsetting the whole thing in every 5000th step) didn't help. But it was not the end, to quote a classic.
When I made a separate function that opened the file, transferred 100.000 lines (just enough not to eat up the whole memory) and closed the file, THEN I made subsequent calls to this function (appending to the existing file), I found that for every function exit, PHP removed the garbage. It was a local-variable-space thing.
TL;DR
When a function exits, it frees all local variables.
If you do the job in smaller portions, like 0 to 1000 in the first function call, then 1001 to 2000 and so on, then every time the function returns, your memory will be regained. Garbage collection is very likely to happen on return from a function. (If it's a relatively slow function eating a lot of memory, we can safely assume it always happens.)
Side note: for reference-passed variables it will obviously not work; a function can only free its inside variables that would be lost anyway on return.
I hope this saves your day as it saved mine!
I've found that PHP's internal memory manager is most-likely to be invoked upon completion of a function. Knowing that, I've refactored code in a loop like so:
while (condition) {
// do
// cool
// stuff
}
to
while (condition) {
do_cool_stuff();
}
function do_cool_stuff() {
// do
// cool
// stuff
}
EDIT
I ran this quick benchmark and did not see an increase in memory usage. This leads me to believe the leak is not in json_decode()
for($x=0;$x<10000000;$x++)
{
do_something_cool();
}
function do_something_cool() {
$json = '{"a":1,"b":2,"c":3,"d":4,"e":5}';
$result = json_decode($json);
echo memory_get_peak_usage() . PHP_EOL;
}
I was going to say that I wouldn't necessarily expect gc_collect_cycles() to solve the problem - since presumably the files are no longer mapped to zvars. But did you check that gc_enable was called before loading any files?
I've noticed that PHP seems to gobble up memory when doing includes - much more than is required for the source and the tokenized file - this may be a similar problem. I'm not saying that this is a bug though.
I believe one workaround would be not to use file_get_contents but rather fopen()....fgets()...fclose() rather than mapping the whole file into memory in one go. But you'd need to try it to confirm.
HTH
C.
Call memory_get_peak_usage() after each statement, and ensure you unset() everything you can. If you are iterating with foreach(), use a referenced variable to avoid making a copy of the original (foreach()).
foreach( $x as &$y)
If PHP is actually leaking memory a forced garbage collection won't make any difference.
There's a good article on PHP memory leaks and their detection at IBM
There recently was a similar issue with System_Daemon. Today I isolated my problem to file_get_contents.
Could you try using fread instead? I think this may solve your problem.
If it does, it's probably time to do a bugreport over at PHP.