This recursive function uses over 1.4 MB of RAM, and it's not freed. All it returns is a single int. How can I free up much more memory?
function bottomUpTree($item, $depth)
{
if ($depth)
{
--$depth;
$newItem = $item << 1;
return array(
bottomUpTree($newItem - 1, $depth),
bottomUpTree($newItem, $depth),
$item
);
}
unset($depth);
unset($newItem);
return array(NULL, NULL, $item);
}
bottomUpTree(0, 7);
Recursive functions will always suck up memory. Each call will use up a bit more until you reach the bottom and start returning. This is unavoidable. Doing unset() on the function's parameters won't help you... they've already taken up space on the call stack and can't be removed until the function returns.
One option would be to switch over to an iterative function, but that's harder to do with a tree-structure.
But most of all, what does this function actually accomplish? You're returning arrays, but not assigning them anywhere in the calling level, so you're creating a ton of arrays only to throw them away again immediately.
A few tricks to getting the php compiler to release memory are to do the following:
Extract the memory intensive pieces of your recursive function to it's own function/method. PHP won't release memory until the function finishes/exits/returns.
Before returning in your extracted function/method set variables to NULL.
Call gc_collect_cycles() after you call the memory intensive function.
Related
There's a memory leak in my script and I couldn't find it after 2 days. I found the loop that is causing the memory leak; each iteration of the loop increases the memory usage. I moved the loop into a function to isolate the variables. At the end of the function, I unsetted every variable created by the function so that get_defined_vars() returns an empty array. Here's what I mean:
function the_loop(){
$var="value";
... // processing, including using a library
unset($var);
print_r(get_defined_vars()); // prints empty array
}
while(true){
the_loop();
echo memory_get_usage()."\n"; // steadily increases until memory limit is reached
}
I'm guessing that some variables defined in the_loop() are still in memory. I tried using XDebug's trace tool, but it didn't help. All it showed was that memory usage increases on average over the long run. I'm looking for a tool that can show me all the values in PHP's memory. I will be able to recognize the variable based on the value. What tool can dump PHP's memory?
As Dragon mentioned unset doesn't instantly free memory.
What's better at freeing memory with PHP: unset() or $var = null
I'd consider re-evaluating the way you're using PHP, it's not designed for long/constant running scripts, the garbage handler simply isn't that great.
If you want to dig further into the executing script I'd suggest checking out some tools like:
https://github.com/jokkedk/webgrind
http://xhprof.io/
http://derickrethans.nl/xdebug-and-tracing-memory-usage.html
Also worth a read: What gc_collect_cycles function is useful for?
Calling unset() does not force garbage collection, so while the reference count should decrease there may be others referencing it. Use xdebug_debug_zval($var) before calling unset to see how many references to its value there actually are.
I'm developing a PHP-extension, where an object method needs to return an array zval.
The method looks like:
ZEND_METHOD(myObject, myMethod)
{
zval **myArrayProperty;
if (zend_hash_find(Z_OBJPROP_P(getThis()), "myArrayProperty", sizeof("myArrayProperty"), (void **) &myArrayProperty) == FAILURE) {
RETURN_FALSE;
}
RETURN_ZVAL(*myArrayProperty, 1, 0);
}
The code works fine and does the expected thing - it returns object's myArrayProperty. However, I'd like to optimize the process.
myArrayProperty stores an array, which can be quite big. And the RETURN_ZVAL() macro duplicates that array in order to return the value. The duplication process takes good amount of time to acquire memory and copy all the array values. At the same time, the returned array is usually used for read-only operations. So a nice optimization would be to use PHP's mechanism with reference counting and do not duplicate myArrayProperty contents. Rather I'd increase refcount of myArrayProperty and just return pointer to it. This is the same tactic as usually used, when working with variables in a PHP extension.
However, seems, there is no way to do it - you have to duplicate value in order to return it from a PHP extension function. Changing function signature to return value by reference, is not an option, because it links the property and returned value - i.e. changing returned value later, changes the property as well. That is not an acceptable behavior.
The inability to engage reference counting looks strange, because same code in PHP:
function myMethod() {
{
return $this->myArrayProperty;
}
is optimized by the reference counting mechanism. That's why I'm asking this question at StackOverflow in case I missed something.
So, is there a way to return an array from a function in PHP extension, without copying the array in memory?
If your function returns by-value this is only possible as of PHP 5.6 (current master) using the RETURN_ZVAL_FAST macro:
RETURN_ZVAL_FAST(*myArrayProperty);
If your function returns by-reference (return_reference=1 in the arginfo) you can return using the following code:
zval_ptr_dtor(&return_value);
SEPARATE_ZVAL_TO_MAKE_IS_REF(myArrayProperty);
Z_ADDREF_PP(myArrayProperty);
*return_value_ptr = *myArrayProperty;
If your function returns by-value and you're on PHP 5.5 or older you can still optimize the refcount=1 case:
if (Z_REFCOUNT_PP(myArrayProperty) == 1) {
RETVAL_ZVAL(*myArrayProperty, 0, 1);
Z_ADDREF_P(return_value);
*myArrayProperty = return_value;
} else {
RETVAL_ZVAL(*myArrayProperty, 1, 0);
}
I don't have access to PHP < 5.6 but I think the problem is that the value is copied only. To be absolutely sure you should search the code for the defines in question.
That means you might be able to try:
zval *arr;
MAKE_STD_ZVAL(arr);
array_init(arr);
// Do things to the array.
RETVAL_ZVAL(arr, 0, 0);
efree(arr);
This is dangerous if used unwisely. If used with your own temporary containers I don't know of any problems.
You can also probably work on return value directly which might be a better approach. You would likely initialise it and pass it around as a pointer at the start.
You can wrap your return result like this. You can also experiment with references.
It's been awhile, since I coded something like this…
So, what I do in code below: 1). explicitly increasing refcounter 2). returning zval without copying it
ZEND_METHOD(myObject, myMethod)
{
zval **myArrayProperty;
if (zend_hash_find(Z_OBJPROP_P(getThis()), "myArrayProperty", sizeof("myArrayProperty"), (void **) &myArrayProperty) == FAILURE) {
RETURN_FALSE;
}
Z_ADDREF_PP(myArrayProperty);
RETURN_ZVAL(*myArrayProperty, 0, 0);
}
This has been bothering me, lets say I have a function:
public function addItem($item) {
$this->items[] = $item;
return $this;
}
and then I have another function like:
public function addItems($items) {
foreach($items as $item) {
$this->addItem($item); //IS THIS FASTER OR
$this->items[] = $item;//THIS?
}
return $this;
}
basically I always do the option 1 $this->addItem($item) inside loop but on the other hand it seems to me redundant I could just append item to array essentially it does samething. Just want to know which one is faster if there is any difference in performance.
This
$this->items[] = $item;
Will always be (minutely) faster than
$this->addItem($item);
Because the second version incurs the overhead of calling a function, whereas the first does not. It is the same reason you should use the first version instead of array_push() for one element. Quoting the manual:
Note: If you use array_push() to add one element to the array it's
better to use $array[] = because in that way there is no overhead of
calling a function.
Function calls require some processor time. So, less function calls -> more performance.
HOWEVER
The main point is that your code remains clear and secure. Such a code is much more important than any performance.
Remember once and forever:
Clear code, clear variable and function names in
your code will save you HUGE amounts of time and health.
Your case: addItem() function provides a really good abstraction. Just keep it.
In function addItem() you could do some validation and debug things. Also, it can return true or false indicating success.
Function jump is in principle slower than direct access (i.e., 1 is faster).
That being said:
Don't worry about micro optimizations. Prefer clear, clean, concise and/or flexible code to fast code.
If you use some PHP opcode optimizer, chances are those accesses are inlined. In that case, there would be no difference.
The most important thing I ever learned about benchmarking was
Never believe what anybody tells you about what is faster or slower until you prove it yourself
I have since wrote dozens of benchmarks and proved public opinion wrong many times.
When running this simple script I get the output posted below.
It makes me think that there is a memory leak in either my code or the Zend Framework/Magento stack. This issue occurs when iterating any kind of Magento collection.
Is there anything that I am missing or doing wrong?
Script:
$customersCollection = Mage::getModel('customer/customer')->getCollection();
foreach($customersCollection as $customer) {
$customer->load();
$customer = null;
echo memory_get_usage(). "\n";
}
Output:
102389104
102392920
...
110542528
110544744
Your issue is that you are issuing fairly expensive queries with each iteration, when you could load the necessary data via the collection queries:
$collection = Mage::getResourceModel('customer/customer_collection')->addAttributeToSelect('*');
will do the same, but all in one query. The caveat to this approach is that if there are any custom event observers for customer_load_before or customer_load_after events (there are no core observers for these), the observer will need to be run manually for each data model.
Edit: credit to osonodoar for spotting an incorrect class reference (customer/customer vs customer/customer_collection)
The memory for an object (or other value) can only be freed when there are no references to it anywhere in the PHP process. In your case, the line $customer = null only decreases the number of references to that object by one, but it doesn't make it reach zero.
If you consider a simpler loop, this may become clearer:
$test = array('a' => 'hello');
foreach ( $test as $key => $value )
{
// $value points at the same memory location as $test['a']
// internally, that "zval" has a "refcount" of 2
$value = null;
// $value now points to a new memory location, but $test['a'] is unnaffected
// the refcount drops to 1, but no memory is freed
}
Because you are using objects, there is an added twist - you can modify the object inside the loop without creating a new copy of it:
$test = array('a' => new __stdClass);
// $test['a'] is an empty object
foreach ( $test as $key => $value )
{
// $value points at the same object as $test['a']
// internally, that object has a "refcount" of 2
$value->foo = "Some data that wasn't there before";
// $value is still the same object as $test['a'], but that object now has extra data
// This requires additional memory to store that object
$value = null;
// $value now points to a new memory location, but $test['a'] is unnaffected
// the refcount drops to 1, but no memory is freed
}
// $test['a']->foo now contains the string assigned in the loop, consuming extra memory
In your case, the ->load() method is presumably expanding the amount of data in each of the members of $customersCollection in turn, requiring more memory for each. Inspecting $customersCollection before and after the loop would probably confirm this.
First off, when unsetting variables use unset($variable) instead of $variable=null. It does essentially the same thing, but is much clearer as to your intent.
Second, PHP is meant to die - memory leaks aren't a huge issue, as a PHP request lasts maybe a few seconds, and then the process dies and all memory it was using is freed up for the next request. Unless you are running into scaling issues, it's nothing to worry about.
Edit: which isn't to say don't worry about the quality of your code, but for something like this, its most likely not worth the effort of trying to prevent it from happening unless it is causing problems.
Other way out to handle memory leak is that call exec within loop and let that exec function do the job part which results in memory leak.
So once it completes its part and terminates all memory leak within that exec will be released.
So with huge iterations this memory loss which keeps adding otherwise will be taken care.
#benmarks response would be the right approach here, as calling load() within a loop is a very very expensive call.
Calling $customer->load() would allocate memory incrementally that would be referenced by $customersCollection, that memory won't be released until the end of the loop.
However, if load() needs to be called for any reason, the code below won't leak memory, as the GC releases all the memory allocated by the model in each iteration.
$customersCollection = Mage::getModel('customer/customer')->getCollection();
foreach($customersCollection as $customer) {
$customerCopy = Mage::getModel('customer/customer')->load($customer->getId());
//Call to $customerCopy methods
echo memory_get_usage(). "\n";
}
I was wondering if anyone could answer me this quick question. I tried searching it but I get similar questions but in the wrong context.
What I am wondering is take this code:
function foo()
{
$test_array = array();
for($i=0; $i<10000000; $i++)
{
$test_array[] = $i;
}
}
What happens to $test_array after the function finishes. I know that it looses scope, I am not new to programming.
What I am wondering is should I call
unset($test_array);
before the function ends or does PHP set it for deletion to the garbage collector as the function ends?
I used the for loop just to show a variable of a fair size to get my point across.
Thanks for reading
Kevin
Once $test_array is no longer in scope (and there are no additional references that point to it), it is flagged for garbage collection.
It ceases to be in scope when the process returns from the function to the calling routine.
So there is no need to unset it.
This would only be different if you had declared $test_array as static.
unset() doesn't free the memory a variable uses, it just marks it for the garbage collector which will decide when to free the memory (when it has free cpu cycles or when it runs out of memory, whichever comes first).
However you have to realize that ALL memory used by a PHP script is freed when the script finishes which, most of the time, is measured in milliseconds, so if you're not doing any lengthy operations that would exceed the "normal" execution time of a PHP script you shouldn't worry about freeing memory.