There's a memory leak in my script and I couldn't find it after 2 days. I found the loop that is causing the memory leak; each iteration of the loop increases the memory usage. I moved the loop into a function to isolate the variables. At the end of the function, I unsetted every variable created by the function so that get_defined_vars() returns an empty array. Here's what I mean:
function the_loop(){
$var="value";
... // processing, including using a library
unset($var);
print_r(get_defined_vars()); // prints empty array
}
while(true){
the_loop();
echo memory_get_usage()."\n"; // steadily increases until memory limit is reached
}
I'm guessing that some variables defined in the_loop() are still in memory. I tried using XDebug's trace tool, but it didn't help. All it showed was that memory usage increases on average over the long run. I'm looking for a tool that can show me all the values in PHP's memory. I will be able to recognize the variable based on the value. What tool can dump PHP's memory?
As Dragon mentioned unset doesn't instantly free memory.
What's better at freeing memory with PHP: unset() or $var = null
I'd consider re-evaluating the way you're using PHP, it's not designed for long/constant running scripts, the garbage handler simply isn't that great.
If you want to dig further into the executing script I'd suggest checking out some tools like:
https://github.com/jokkedk/webgrind
http://xhprof.io/
http://derickrethans.nl/xdebug-and-tracing-memory-usage.html
Also worth a read: What gc_collect_cycles function is useful for?
Calling unset() does not force garbage collection, so while the reference count should decrease there may be others referencing it. Use xdebug_debug_zval($var) before calling unset to see how many references to its value there actually are.
Related
I have read several times that, in order to invoke the garbage collector and actually clean the RAM used by a variable, you have to assign a new value (e.g. NULL) instead of simply unset() it.
This code, however, shows that the memory allocated for the array $a is not cleaned after the NULL assignment.
function build_array()
{
for ($i=0;$i<10000;$i++){
$a[$i]=$i;
}
$i=null;
return $a;
}
echo '<p>'.memory_get_usage(true);
$a = build_array();
echo '<p>'.memory_get_usage(true);
$a = null;
echo '<p>'.memory_get_usage(true);
The output I get is:
262144
1835008
786432
So part of the memory is cleaned, but not all the memory. How can I completely clean the RAM?
You have no way to definitely clear a variable from the memory with PHP.
Its is up to PHP garbage collector to do that when it sees that it should.
Fortunately, PHP garbage collector may not be perfect but it is one of the best features of PHP. If you do things as per the PHP documentation there's no reasons to have problems.
If you have a realistic scenario where it may be a problem, post the scenario here or report it to PHP core team.
Other unset() is the best way to clear vars.
Point is that memory_get_usage(true) shows the memory allocated to your PHP process, not the amount actually in use. System could free unused part once it is required somewhere.
More details on memory_get_usage could be found there
If you run that with memory_get_usage(false), you will see that array was actually gc'd. example
I'm curious about using of unset() language construct just about everywhere, where I took memory or declare some variables (regardless of structure).
I mean, when somebody declares variable, when should it really be left for GC, or be unset()?
Example 1:
<?php
$buffer = array(/* over 1000 elements */);
// 1) some long code, that uses $buffer
// 2) some long code, that does not use $buffer
?>
Is there any chance, that $buffer might affect performance of point 2?
Am I really need (or should) to do unset($buffer) before entering point 2?
Example 2:
<?php
function someFunc(/* some args */){
$buffer = new VeryLargeObject();
// 1) some actions with $buffer methods and properties
// 2) some actions without usage of $buffer
return $something;
}
?>
Am I really need (or should) to do unset($buffer) within someFunc()s body before entering point 2?
Will GC free all allocated memory (references and objects included) within someFunc()s scope, when function will come to an end or will find return statement?
I'm interested in technical explaination, but code style suggestions are welcome too.
Thanks.
In php, all memory gets cleaned up after script is finished, and most of the time it's enough.
From php.net:
unset() does just what it's name says - unset a variable. It does not
force immediate memory freeing. PHP's garbage collector will do it
when it see fits - by intention as soon, as those CPU cycles aren't
needed anyway, or as late as before the script would run out of
memory, whatever occurs first.
If you are doing $whatever = null; then you are rewriting variable's
data. You might get memory freed / shrunk faster, but it may steal CPU
cycles from the code that truly needs them sooner, resulting in a
longer overall execution time.
In reality you would use unset() for cleaning memory pretty rare, and it's described good in this post:
https://stackoverflow.com/a/2617786/1870446
By doing an unset() on a variable, you mark the variable for being "garbage collected" so the memory isn't immediately available. The variable does not have the data anymore, but the stack remains at the larger size.
In PHP >= 5.3.0, you can call gc_collect_cycles() to force a GC pass. (after doing gc_enable() first).
But you must understand that PHP is script language, it's not Java so you shouldn't consider it like one. If your script is really that heavy to use tons of RAM - you can use unset and when script is close to exceed the memory - GC will trigger and clean up everything useless, including your unset variables. But in most cases you can forget about it.
Also, if you would want to go for unsetting every variable you do not use - don't. It will actually make your script execute longer - by using more CPU cycles - for the sake of getting free memory that would, in most cases, would never be needed.
Some people also say that they use unset to explicitly show that they won't use variable anymore. I find it a bad practice too, for me it just makes code more verbose with all these useless unsets.
I was wondering if anyone could answer me this quick question. I tried searching it but I get similar questions but in the wrong context.
What I am wondering is take this code:
function foo()
{
$test_array = array();
for($i=0; $i<10000000; $i++)
{
$test_array[] = $i;
}
}
What happens to $test_array after the function finishes. I know that it looses scope, I am not new to programming.
What I am wondering is should I call
unset($test_array);
before the function ends or does PHP set it for deletion to the garbage collector as the function ends?
I used the for loop just to show a variable of a fair size to get my point across.
Thanks for reading
Kevin
Once $test_array is no longer in scope (and there are no additional references that point to it), it is flagged for garbage collection.
It ceases to be in scope when the process returns from the function to the calling routine.
So there is no need to unset it.
This would only be different if you had declared $test_array as static.
unset() doesn't free the memory a variable uses, it just marks it for the garbage collector which will decide when to free the memory (when it has free cpu cycles or when it runs out of memory, whichever comes first).
However you have to realize that ALL memory used by a PHP script is freed when the script finishes which, most of the time, is measured in milliseconds, so if you're not doing any lengthy operations that would exceed the "normal" execution time of a PHP script you shouldn't worry about freeing memory.
I have two simple questions. What is better/useful for memory cleanup.
$var = null;
or
unset($var);
I have one function with one cycle. I am getting (after few minutes)
Fatal error: Allowed memory size of 419430400 bytes exhausted
I am setting null and unset()-ing every object (at the end of the cycle) but still without any success :( I cant find out what is consuming memory.
And what about function calls in cycle? Will PHP release all allocations in these functions?(after call)
PHP itself confuses both concepts sometimes but, in general, a variable set to NULL is not the same as a variable that does not exist:
<?php
$foo = 'One';
$bar = 'Two';
$foo = NULL;
unset($bar);
var_dump($foo); // NULL
var_dump($bar); // Notice: Undefined variable: bar
var_dump(get_defined_vars()); // Only foo shows up: ["foo"]=> NULL
?>
unset() does just that, it unsets a variable; but it does not immediate free up memory.
PHP's garbage collector will actually free up memory previously used by variables that are now unset, but only when it runs. This could be sooner, when CPU cycles aren't actively being used for other work, or before the script would otherwise run out of memory... whichever situation occurs first.
And be aware that unset won't necessarily release the memory used by a variable if you have other references to that variable. It will simply delete the reference, and reduce the reference count for the actual stored data by 1.
EDIT
While unset doesn't immediately release the memory used (only garbage collection actually does that) the memory that is no longer used as a result is available for the declaration of new variables
I found problem.
First it was caused by xdebug profilling tools (i have turned on everything :) ) - and it consume lot of memory.
So remember: xdebug (when profilling turned on) consumes some memory in PHP process of your application
Second, I didn't release static members used in called functions.
If you unset the variable it is just marked, so on the next garbage collection it will be removed. If setting to null, the data of the variable gets overwritten.
Maybe see also the comments on the php manual: Unset Manual
At least this behaviour is also what i have encountered so far.
But to fix it, you should first try to find out what is causing the memory to grow. The memory_get_usage function should be helpful for this.
And what about function calls in cycle? Will PHP release all allocations in these functions?(after call)
Once the execution leaves the scope of a function, all non-static variables are removed from memory.
In a PHP program, I sequentially read a bunch of files (with file_get_contents), gzdecode them, json_decode the result, analyze the contents, throw most of it away, and store about 1% in an array.
Unfortunately, with each iteration (I traverse over an array containing the filenames), there seems to be some memory lost (according to memory_get_peak_usage, about 2-10 MB each time). I have double- and triple-checked my code; I am not storing unneeded data in the loop (and the needed data hardly exceeds about 10MB overall), but I am frequently rewriting (actually, strings in an array). Apparently, PHP does not free the memory correctly, thus using more and more RAM until it hits the limit.
Is there any way to do a forced garbage collection? Or, at least, to find out where the memory is used?
it has to do with memory fragmentation.
Consider two strings, concatenated to one string. Each original must remain until the output is created. The output is longer than either input.
Therefore, a new allocation must be made to store the result of such a concatenation. The original strings are freed but they are small blocks of memory.
In a case of 'str1' . 'str2' . 'str3' . 'str4' you have several temps being created at each . -- and none of them fit in the space thats been freed up. The strings are likely not laid out in contiguous memory (that is, each string is, but the various strings are not laid end to end) due to other uses of the memory. So freeing the string creates a problem because the space can't be reused effectively. So you grow with each tmp you create. And you don't re-use anything, ever.
Using the array based implode, you create only 1 output -- exactly the length you require. Performing only 1 additional allocation. So its much more memory efficient and it doesn't suffer from the concatenation fragmentation. Same is true of python. If you need to concatenate strings, more than 1 concatenation should always be array based:
''.join(['str1','str2','str3'])
in python
implode('', array('str1', 'str2', 'str3'))
in PHP
sprintf equivalents are also fine.
The memory reported by memory_get_peak_usage is basically always the "last" bit of memory in the virtual map it had to use. So since its always growing, it reports rapid growth. As each allocation falls "at the end" of the currently used memory block.
In PHP >= 5.3.0, you can call gc_collect_cycles() to force a GC pass.
Note: You need to have zend.enable_gc enabled in your php.ini enabled, or call gc_enable() to activate the circular reference collector.
Found the solution: it was a string concatenation. I was generating the input line by line by concatenating some variables (the output is a CSV file). However, PHP seems not to free the memory used for the old copy of the string, thus effectively clobbering RAM with unused data. Switching to an array-based approach (and imploding it with commas just before fputs-ing it to the outfile) circumvented this behavior.
For some reason - not obvious to me - PHP reported the increased memory usage during json_decode calls, which mislead me to the assumption that the json_decode function was the problem.
There's a way.
I had this problem one day. I was writing from a db query into csv files - always allocated one $row, then reassigned it in the next step. Kept running out of memory. Unsetting $row didn't help; putting an 5MB string into $row first (to avoid fragmentation) didn't help; creating an array of $row-s (loading many rows into it + unsetting the whole thing in every 5000th step) didn't help. But it was not the end, to quote a classic.
When I made a separate function that opened the file, transferred 100.000 lines (just enough not to eat up the whole memory) and closed the file, THEN I made subsequent calls to this function (appending to the existing file), I found that for every function exit, PHP removed the garbage. It was a local-variable-space thing.
TL;DR
When a function exits, it frees all local variables.
If you do the job in smaller portions, like 0 to 1000 in the first function call, then 1001 to 2000 and so on, then every time the function returns, your memory will be regained. Garbage collection is very likely to happen on return from a function. (If it's a relatively slow function eating a lot of memory, we can safely assume it always happens.)
Side note: for reference-passed variables it will obviously not work; a function can only free its inside variables that would be lost anyway on return.
I hope this saves your day as it saved mine!
I've found that PHP's internal memory manager is most-likely to be invoked upon completion of a function. Knowing that, I've refactored code in a loop like so:
while (condition) {
// do
// cool
// stuff
}
to
while (condition) {
do_cool_stuff();
}
function do_cool_stuff() {
// do
// cool
// stuff
}
EDIT
I ran this quick benchmark and did not see an increase in memory usage. This leads me to believe the leak is not in json_decode()
for($x=0;$x<10000000;$x++)
{
do_something_cool();
}
function do_something_cool() {
$json = '{"a":1,"b":2,"c":3,"d":4,"e":5}';
$result = json_decode($json);
echo memory_get_peak_usage() . PHP_EOL;
}
I was going to say that I wouldn't necessarily expect gc_collect_cycles() to solve the problem - since presumably the files are no longer mapped to zvars. But did you check that gc_enable was called before loading any files?
I've noticed that PHP seems to gobble up memory when doing includes - much more than is required for the source and the tokenized file - this may be a similar problem. I'm not saying that this is a bug though.
I believe one workaround would be not to use file_get_contents but rather fopen()....fgets()...fclose() rather than mapping the whole file into memory in one go. But you'd need to try it to confirm.
HTH
C.
Call memory_get_peak_usage() after each statement, and ensure you unset() everything you can. If you are iterating with foreach(), use a referenced variable to avoid making a copy of the original (foreach()).
foreach( $x as &$y)
If PHP is actually leaking memory a forced garbage collection won't make any difference.
There's a good article on PHP memory leaks and their detection at IBM
There recently was a similar issue with System_Daemon. Today I isolated my problem to file_get_contents.
Could you try using fread instead? I think this may solve your problem.
If it does, it's probably time to do a bugreport over at PHP.