I am currently debugging a script that constantly runs into OutOfMemory exceptions. It is run as a cronjob and usually runs fine, but when the cronjob wasn't run for a while (for whatever reason) the script has to handle to many elements that queued up and will run into a OutOfMemory exception.
From examining the code I was not able to spot the problem. I believe one of the iterative function calls might leak memory, but I am not sure which one and where.
Is there an option to get PHP to dump the heap, when an OutOfMemory exception occurs? I might be able to spot the problem from there (most likely).
While I was not able to find a "dump heap on Exception" option, I did find get_defined_vars() which is basically a heap dump if called from a global scope. Using this I was able to see that there where hundreds (actually thousands) of still referenced database rows hanging around in my memory. This was due to a not freed mysql result resource somewhere in the infamous function that caused the leak. I found it and fixed it. It runs well now.
Well, easiest approach would be to use a try-catch block around that part of your script where the error possibly occurs and you will have to dump the stack in the catch part. The problem might be that the machine won't be able to react cause the memory is full and it terminates. I do not know if it helps to discard some variables to free up some memory to output some data.
EDIT: For this purpose use the php function debug-backtrace. This will give you a stack trace. So finding the error will be much likely in case the machine is still up.
Just do not load all objects together to memory, but read-as-you-process-them?
I've had lots of problems with simpleXML and memory leaks. They are a pain in the are to track down... took me days to figure out that simpleXML was causing then and then fix them.
As far as i know you cand programatically set a handled for OOM:)
Also, PHP's functions for displaying memory info fails to detect the memory leaks, i had scripts eating up ~1Gb of ram, but PHP's functions reported only 100Mb used:)
This is as good of a 'heap dump' as I'm able to quickly write in PHP. I take the defined variables and functions, then sort by their serialized length. Serialized length isn't a 100% reliable method for getting a variable's size, but it's pretty good, and generally useful for determining which objects are your memory hogs:
$memmap = array_map(function($var) { return strlen(serialize($var)); },
array_merge(get_defined_functions(), get_defined_vars()));
arsort($memmap);
var_dump($memmap);
You may want to tweak the callback function a bit if you'd like your results to be more verbose, or to recurse through the defined variables.
I've never seen PHP provide a native facility for this but a few other things might exist:
Try: https://github.com/mcfunley/php-heap/blob/master/php-heap.py
It could also be possible to write an extension to achieve the same.
Related
I'm going through the journey of bringing our dating web app up to date past PHP 7+
I have been updating the depreciated code and squishing errors as they've popped up, thanks to the help of this site, and has a question in regards to our depreciated __destruct() functions.
I searched my error I was getting thrown, and found this link on github that solved my issue: https://github.com/sajari/simple-linkedin-php/issues/4
So supposedly the destruct functions break after PHP 7, and commenting them out DID fix my issue, but it left me worried.
Edit: Destruct still works, its just how my dated code was trying to
use it
I have one specific instances of the destruct being called in one of our class files that builds a job request page based on what job a user queries from our database.
/** free up memory
$tmp->__destruct();
unset($tmp);
**/
From what I can tell, everything is working fine, but I wonder what the consequences of this are. Should I replace it with some newer memory cleanup method? This app is so dated, I wonder if this clean-up is even needed anymore, or intentionally put in to stop a memory leak. My boss doesn't seem to know either.
I'm new to PHP in a professional sense, and would like to make sure this update still allows the same level of speed and usability for our clients.
Another stackoverflow question asked here: Deletion of a PHP object; also unset() and __destruct
A user answered by saying that the values are already cleared by the complete execution of a script, BUT could be explicitly cleared to stop any unintentional references.
This just lead to more questions. When that user means script, do they mean the specific function call?
To wrap this up, lets say that destruct and unset call was to stop a memory leak, what is an equivalent replacement for it?
Thanks!
I want to personally mark this question answered for my own sake. It
was extremely helpful!
$tmp->__destruct();
unset($tmp);
You should never be calling destructors directly like this. Calling unset($tmp) by itself will indirectly cause the destructor to fire when the object is removed. So, this code actually causes the destructor to be called twice, which is almost certainly not what you want.
When that user means script, do they mean the specific function call?
No, they mean that one particular PHP execution, from the first line of the first file to the last line of the last file. At the end of execution, everything is automatically freed. In PHP, this execution run usually refers to one single short web request, so explicit calls to unset() for the sake of "freeing memory" are virtually never needed because an implicit unset() is going to happen as soon as the request is done. You'd only ever have to worry about doing this manually if, for example, the $tmp object uses a massive amount of memory at the start of a request, doesn't need it for the rest of the request, and runs for a long time. If you're just calling unset() at the end of the request, it's useless.
It's most likely safe to remove both these lines. The first is redundant and the second is almost never needed.
Note, it's much more likely to use unset() to remove a variable from scope than to free memory, so take care to scan if there's some code later on that needs $tmp to not exist.
First of all - there seem to be many questions about the fundamentals of tick functionality, so I want to add the top user comment at php.net/declare to the pile for anyone looking for further information. I found it while digging around as I tried to figure out the following.
So, I'm working on writing a simple debug helper. I want to add function tracing and benchmarking - basically what tick functionality is perfect for.
Thing is, I want to enable and disable benchmarking depending on arbitrary conditions that occur during script processing. I'm not really looking for fixed debugging à la scoped declare() { ... }.
What I'm looking to do is to put declare(); at the top of my script, and then register and unregister my debugging/benchmarking (tick) function as appropriate. Un/registration won't happen (too) often, so is efficient and reasonable.
But then I got curious: when I don't have a tick function registered... does the fact that I've run declare(ticks=1); have any effect on execution efficiency? Does it cause any extra processing to become permanently enabled anyway?
Analysis of PHP(7)'s source code shows that the answer is technically yes, but I'm not yet sure how.
The answer seems to be in zend_compile.c:8200: it appears this function defers compilation processing to the appropriate routines, then if ticks are enabled it additionally emits a ZEND_TICKS opcode into the opline via zend_emit_tick() in :2167. The opcode reference page for TICKS seems consistent with this conclusion; it shows an example disassembled opcode listing which has TICKS opcodes scattered throughout it, and I was wondering how they got in there until I discovered the above.
The ZEND_TICKS handler (in zend_vm_def.h:6859) seems to call zend_ticks_function(). This is mapped to ticks_function() in zend.c:754, which is in turn mapped to php_run_ticks() in main.2013. This is finally defined in php_ticks.c, where it's all of:
void php_run_ticks(int count)
{
zend_llist_apply_with_argument(
&PG(tick_functions),
(llist_apply_with_arg_func_t) php_tick_iterator,
&count
);
}
Huh. Not bad.
But here's the thing. If I declare(ticks=1);, the above dispatch is being run for literally every statement executed. That's... ouch. For long-running scripts containing high-iteration-count, tight processing loops, I'm wondering how badly that'll add up.
Problem is, I'm not even sure how to benchmark this. The only way I could envisage to do so would be to synthesize some PHP bytecode and then figure out a way to inject that directly into the PHP bytecode interpreter.
And that leads to my question: how much of a performance impact does this additional dispatch have, in practice? How can I quantify it?
Obviously the above investigation was performed on the canonical PHP.net interpreter. I haven't looked into how HHVM does this at all (yet), but I wouldn't at all mind learning how it handles it.
I have a PHP script that is failing with a fatal out-of-memory error. It is a script that processes all records in the DB - it works fine up to about 10k records and then hits the memory error.
However, I can't find out what is using up my application's memory.
I've checked the $_GLOBALS array and that accounts for maybe 1 MB or so.
I've checked the call stack at various points and have seen nothing unexpected.
The base memory requirement for PHP plus all relevant class files, etc. is about 7MB.
My feeling is that there is probably somewhere in the code that is resulting in variable references persisting - either deliberately (e.g. via a static cache in some class or other) or by mistake (e.g. resource handles not being freed).
Obviously functions like memory_get_usage() can tell me how much memory is used at any given point in the script, and tracking this is a slow but effective way of debugging. However, is there any way of getting details about what is actually using that memory?
Happy to accept answers that use an external tool (e.g. XDebug) providing they give useful output (i.e. output that identifies the class/variable names rather than using PHPs internal IDs). The output I expect would be something like you get from var_dump/print_r.
[Note that this question is not about how to debug out-of-memory issues in general, but specifically about whether there is a way to expose the details of memory use when debugging.]
Paul Crovella provided a link, but couldn't be bothered to post it as an answer, so I'm doing it for him.
The php-memprof extension (https://github.com/arnaud-lb/php-memory-profiler) provides a number of tools that can be used to provide exactly the information that the question asks for.
From the readme file:
php-memprof profiles memory usage of PHP scripts, and especially can tell which function has allocated every single byte of memory currently allocated.
Memprof can be enabled during script execution by calling memprof_enable().
Then the memory usage can be dumped by calling one of the memprof_dump_ functions. Both tell which functions allocated all the currently allocated memory.
The original question asked for something like print_r(), and this is provided by the memprof_dump_array() function. However, there are a number of other ways of accessing the memory profile which may be more useful depending on what you are trying to achieve, including dumping the entire memory map in callgrind format, for offline analysis.
As it is a PHP extension, it will require access to php.ini in order to install it, so it may not be suitable for debugging issues on live sites (but nobody does that, right?).
Exceptions are worse than goto statements, because they are going to each of the higher tires separately, to see if something would eventually catch them up; And they will left the function/method unfinished without freeing up their already allocated resources.
That's the main reason I have seen in many articles that are recommending against the use of Exceptions, mostly for C++ and Java developers.
I'm wondering if the same reason goes for PHP as well? The point regarding the PHP is that the script is gonna be finished/terminated usually within seconds from initialization. And I assume -- well, I hope; That PHP core is gonna free up all the resources that have been allocated to that particular script.
That being said, the reason that Exceptions left resources occupied wouldn't be a real concern here, because in any case all of those resources will be free soon by the time script finishes.
Is that correct or am I missing something here? If it's correct, then does that mean I can use Exceptions in PHP without being worried about Zombie Resources or they will stuck there for a while after the script terminates? Is PHP's GC mature enough to take care of those wasted resources?
PHP will free allocated memory, close resources and such when the associated variables go out of scope. That's PHP's primary mechanism for deallocation: scope. Since the scope of a function is typically very quickly exited when an exception is thrown, everything in it will be deallocated the same way as if you'd return from the function regularly. The same will also happen once the scripts ends, as you said.
I was wondering if it is possible to store function variables in memcached. I wrote basic templating system which compiles xml templates into functions. The templates can get very big and I think I could get a performance boost if I could cache it. Would something like this work, or would I just be caching a reference to the function?
$populate_template = function($values){
//compiled template
};
$memcached_object->set("some_key",$populated_template);
EDIT: I do realize that there are php accelerators that do exactly what I want, however it would be a lot easier to be able to use memcached because I won't have to go through the process of getting another technology approved.
I don't think so. Naturally, I looked at serializing it. But:
PHP Warning: Uncaught exception 'Exception' with message 'Serialization of 'Closure' is not allowed' in php shell code:1
With memcached, all you can probably do is store the code as a string, then call create_function after you pull it out.
I don't think there would be a performance boost with "serializing" the function source code as a string and recreating it via create_function. It will rather be hundred times slower than defining the function in PHP, because of the input/output with external medium + doing what would PHP need to do in either case.
You need to go with an opcode cache. But if PHP performance is the only lasting bottleneck in your application, you're a happy man.