When running this simple script I get the output posted below.
It makes me think that there is a memory leak in either my code or the Zend Framework/Magento stack. This issue occurs when iterating any kind of Magento collection.
Is there anything that I am missing or doing wrong?
Script:
$customersCollection = Mage::getModel('customer/customer')->getCollection();
foreach($customersCollection as $customer) {
$customer->load();
$customer = null;
echo memory_get_usage(). "\n";
}
Output:
102389104
102392920
...
110542528
110544744
Your issue is that you are issuing fairly expensive queries with each iteration, when you could load the necessary data via the collection queries:
$collection = Mage::getResourceModel('customer/customer_collection')->addAttributeToSelect('*');
will do the same, but all in one query. The caveat to this approach is that if there are any custom event observers for customer_load_before or customer_load_after events (there are no core observers for these), the observer will need to be run manually for each data model.
Edit: credit to osonodoar for spotting an incorrect class reference (customer/customer vs customer/customer_collection)
The memory for an object (or other value) can only be freed when there are no references to it anywhere in the PHP process. In your case, the line $customer = null only decreases the number of references to that object by one, but it doesn't make it reach zero.
If you consider a simpler loop, this may become clearer:
$test = array('a' => 'hello');
foreach ( $test as $key => $value )
{
// $value points at the same memory location as $test['a']
// internally, that "zval" has a "refcount" of 2
$value = null;
// $value now points to a new memory location, but $test['a'] is unnaffected
// the refcount drops to 1, but no memory is freed
}
Because you are using objects, there is an added twist - you can modify the object inside the loop without creating a new copy of it:
$test = array('a' => new __stdClass);
// $test['a'] is an empty object
foreach ( $test as $key => $value )
{
// $value points at the same object as $test['a']
// internally, that object has a "refcount" of 2
$value->foo = "Some data that wasn't there before";
// $value is still the same object as $test['a'], but that object now has extra data
// This requires additional memory to store that object
$value = null;
// $value now points to a new memory location, but $test['a'] is unnaffected
// the refcount drops to 1, but no memory is freed
}
// $test['a']->foo now contains the string assigned in the loop, consuming extra memory
In your case, the ->load() method is presumably expanding the amount of data in each of the members of $customersCollection in turn, requiring more memory for each. Inspecting $customersCollection before and after the loop would probably confirm this.
First off, when unsetting variables use unset($variable) instead of $variable=null. It does essentially the same thing, but is much clearer as to your intent.
Second, PHP is meant to die - memory leaks aren't a huge issue, as a PHP request lasts maybe a few seconds, and then the process dies and all memory it was using is freed up for the next request. Unless you are running into scaling issues, it's nothing to worry about.
Edit: which isn't to say don't worry about the quality of your code, but for something like this, its most likely not worth the effort of trying to prevent it from happening unless it is causing problems.
Other way out to handle memory leak is that call exec within loop and let that exec function do the job part which results in memory leak.
So once it completes its part and terminates all memory leak within that exec will be released.
So with huge iterations this memory loss which keeps adding otherwise will be taken care.
#benmarks response would be the right approach here, as calling load() within a loop is a very very expensive call.
Calling $customer->load() would allocate memory incrementally that would be referenced by $customersCollection, that memory won't be released until the end of the loop.
However, if load() needs to be called for any reason, the code below won't leak memory, as the GC releases all the memory allocated by the model in each iteration.
$customersCollection = Mage::getModel('customer/customer')->getCollection();
foreach($customersCollection as $customer) {
$customerCopy = Mage::getModel('customer/customer')->load($customer->getId());
//Call to $customerCopy methods
echo memory_get_usage(). "\n";
}
Related
I see many code like this:
function load_items(&$items_arr) {
// ... some code
}
load_items($item_arr);
$v = &$items_arr[$id];
compared to code as follow:
function load_items() {
// ... some code
return $items_arr;
}
$item_arr = load_items();
$v = $items_arr[$id];
Did the second code will copy items_arr and $item_arr[$id]?
Will the first code import performance?
No, it will not copy the value right away. Copy-on-write
is one of the memory management methods used in PHP. It ensures that memory isn’t wasted when you copy values between variables.
What that means is that when you assign:
$v = $items_arr[$id];
PHP will simply update the symbol table to indicate that $v points to the same memory address of $item_arr[$id], just if you change $item_arr or $v afterwards then PHP allocates more memory and then performs the copy.
By delaying this extra memory allocation and the copying PHP saves time and memory in some cases.
There's are nice article about memory management in PHP: http://hengrui-li.blogspot.no/2011/08/php-copy-on-write-how-php-manages.html
There's a memory leak in my script and I couldn't find it after 2 days. I found the loop that is causing the memory leak; each iteration of the loop increases the memory usage. I moved the loop into a function to isolate the variables. At the end of the function, I unsetted every variable created by the function so that get_defined_vars() returns an empty array. Here's what I mean:
function the_loop(){
$var="value";
... // processing, including using a library
unset($var);
print_r(get_defined_vars()); // prints empty array
}
while(true){
the_loop();
echo memory_get_usage()."\n"; // steadily increases until memory limit is reached
}
I'm guessing that some variables defined in the_loop() are still in memory. I tried using XDebug's trace tool, but it didn't help. All it showed was that memory usage increases on average over the long run. I'm looking for a tool that can show me all the values in PHP's memory. I will be able to recognize the variable based on the value. What tool can dump PHP's memory?
As Dragon mentioned unset doesn't instantly free memory.
What's better at freeing memory with PHP: unset() or $var = null
I'd consider re-evaluating the way you're using PHP, it's not designed for long/constant running scripts, the garbage handler simply isn't that great.
If you want to dig further into the executing script I'd suggest checking out some tools like:
https://github.com/jokkedk/webgrind
http://xhprof.io/
http://derickrethans.nl/xdebug-and-tracing-memory-usage.html
Also worth a read: What gc_collect_cycles function is useful for?
Calling unset() does not force garbage collection, so while the reference count should decrease there may be others referencing it. Use xdebug_debug_zval($var) before calling unset to see how many references to its value there actually are.
So, I am running a long-running script that is dealing with memory sensitive data (large amounts of it). I (think) I am doing a good job of properly destroying large objects throughout the long running process, to save memory.
I have a log that continuously outputs current memory usage (using memory_get_usage()), and I do not notice rises and drops (significant ones) in memory usage. Which tells me I am probably doing the right thing with memory management.
However, if I log on to the server and run a top command, I notice that the apache process that is dealing with this script never deallocates memory (at least visibly though the top command). It simply remains at the highest memory usage, even if the current memory usage reported by php is much, much lower.
So, my question is: are my attempts to save memory futile if the memory isnt really being freed back to the server? Or am I missing something here.
Thank you.
ps. using php 5.4 on linux
pps. For those who want code, this is a basic representation:
function bigData()
{
$obj = new BigDataObj();
$obj->loadALotOfData();
$varA = $obj->getALotOfData();
//all done
$obj = NULL;
$varA = NULL;
unset($obj,$varA);
}
update: as hek2mgl recommended, I ran debug_zval_dump(), and the output, to me, seems correct.
function bigData()
{
$obj = new BigDataObj();
$obj->loadALotOfData();
//all done
$obj = NULL;
debug_zval_dump($obj);
unset($obj);
debug_zval_dump($obj);
}
Output:
NULL refcount(2)
NULL refcount(1)
PHP has a garbage collector. It will free memory for variable containers which reference count is set to 0, meaning that no userland reference exists anymore.
I guess there are still references to variables which you might think to have cleaned. Need to see your code to show you what is the problem.
I was wondering if anyone could answer me this quick question. I tried searching it but I get similar questions but in the wrong context.
What I am wondering is take this code:
function foo()
{
$test_array = array();
for($i=0; $i<10000000; $i++)
{
$test_array[] = $i;
}
}
What happens to $test_array after the function finishes. I know that it looses scope, I am not new to programming.
What I am wondering is should I call
unset($test_array);
before the function ends or does PHP set it for deletion to the garbage collector as the function ends?
I used the for loop just to show a variable of a fair size to get my point across.
Thanks for reading
Kevin
Once $test_array is no longer in scope (and there are no additional references that point to it), it is flagged for garbage collection.
It ceases to be in scope when the process returns from the function to the calling routine.
So there is no need to unset it.
This would only be different if you had declared $test_array as static.
unset() doesn't free the memory a variable uses, it just marks it for the garbage collector which will decide when to free the memory (when it has free cpu cycles or when it runs out of memory, whichever comes first).
However you have to realize that ALL memory used by a PHP script is freed when the script finishes which, most of the time, is measured in milliseconds, so if you're not doing any lengthy operations that would exceed the "normal" execution time of a PHP script you shouldn't worry about freeing memory.
public function foo($file1, $file2){
$obj = new Obj();
$data = array();
$data[] = $obj->importAFile($file1);
$data[] = $obj->importAFile($file2);
return $data;
}
Does the memory allocated for $obj get freed after the return?
If not how can I free it?
By using unset() on a variable, you've marked it for 'garbage collection' literally, as PHP doesn't really have one, so the memory isn't immediately available. The variable no longer houses the data, but the stack remains at the current size even after calling unset(). Setting the variable to NULL drops the data and shrinks the stack memory almost immediately.
This has worked for me on several occasions where memory exhausted warnings were thrown before the tuning, then calling unset() after nullifying the variable. Calling unset after nullifying mightn't be necessary but I used it nonetheless after the nullification.
PHP uses garbace collector. It frees all variables to which there are no references left. Assuming that $obj->importAFile() does not return reference to $obj, the memory will be freed. However, there is no guarantee when the memory will be freed. If $obj contains reference to itself, in older versions of PHP the memory won't be freed as well. You can read more in PHP documentation
It manages memory for you. You might have a problem only when there are some circular references between your objects