I think I've read all topics on stackoverflow.com about memory_get_usage() and memory_peak_get_usage() functions, but it appears that the explanations won't fit my case.
My script showed the following result:
memory peak usage (true): 2
memory peak usage (false): 0.44921112060547
memory get usage: 0.40422821044922
memory peak usage (true): 2
memory peak usage (false): 6.5214996337891
memory get usage: 1.2414016723633
memory peak usage (true): 2
memory peak usage (false): 6.5214996337891
memory get usage: 1.2413940429688
memory peak usage (true): 2
memory peak usage (false): 6.5214996337891
memory get usage: 1.2425689697266
memory peak usage (true): 2
memory peak usage (false): 6.5214996337891
memory get usage: 1.2467498779297
memory peak usage (true): 2
memory peak usage (false): 18.253067016602
memory get usage: 2.5322036743164
memory peak usage (true): 2
memory peak usage (false): 18.253067016602
memory get usage: 2.5322113037109
memory peak usage (true): 2
memory peak usage (false): 18.253067016602
memory get usage: 2.406005859375
The logs are generated by following commands, repeated sometimes through the code:
echo 'memory peak usage (true): '.(memory_get_peak_usage(true)/1024/1024).'<br>';
echo 'memory peak usage (false): '.(memory_get_peak_usage(false)/1024/1024).'<br>';
echo 'memory get usage: '.(memory_get_usage()/1024/1024).'<br><br>';
Info: For the last log i had set some variables to null and i think they are freed by garbage collector and because that the memory get usage lowered to 2.40 from 2.53.
My question is:
WTH is happening? Why memory_get_usage() is larger than memory_get_peak_usage(true)? And why memory_get_peak_usage(false) is larger than memory_get_peak_usage(true)?
Its totally different from this thread:
memory_get_peak_usage() with "real usage"
First of all, there's this page http://jpauli.github.io/2014/07/02/php-memory.html, which explains in detail how PHP allocates memory, and how ZendMM works. I highly recommend you check it out if you're willing to dwell in low-level PHP programming.
Now, moving on to your question, let me tell you: Nowhere in the Internet did i find a satisfactory answer. That being said, what follows is a result of fair assumptions based on documentation and other sources of information.
as stated in PHP's documentation, memory_get_usage (which defaults the $real_usage to false) returns the amount of memory currently being allocated to your script. It also only tracks memory allocated by using emalloc().
Let's give out some definitions before moving further:
memory_get_usage (false) - Returns the sum of bytes requested by calls to emalloc + header bytes + memory alignment. It's not perfect as it doesn't take in account the amount of blocks wasted by not being able to fit space in the remaining segments of the allocated blocks.
memory_get_peak_usage(false) - Returns the peak of memory that has been allocated to your script. Same as the case above, it might give lower results than the actual amount of bytes used.
memory_get_peak_usage(true) - Returns the actual peak of bytes allocated by the script.
"Why memory_get_usage() is larger than memory_get_peak_usage(true)?"
It's most likely happening because they're using different ways of calculating the memory usage.
- Peak usage (true) is trying to tell you the amount of actual bytes your script reached, while get_usage(false) only tells you the amount the sum of all emalloc calls. It might be higher simply because it's not programmed to realize your program is currently freeing memory, and that said 2.5 bytes of data may have actually fit in 2 bytes of your actual memory the entire time, due to constant freeing and allocating.
That's based on the commentary made in the page i linked above:
memory_get_usage() give an average information, often accurate, but not 100% accurate on a byte-basis
Meaning it attempts to tell you the average of amount it believes' being allocated by calculating emallocs, but it can't know how many memory blocks your script is spending as long as it's set to false.
Why memory_get_peak_usage(false) is larger than Memory_get_peak_usage(true)?"
Here's the part where it gets weird. I've honestly never seen that before, and do believe that's most likely a misunderstanding. This may be a result of your script allocating memory in a way the (false) implementation believes memory's still allocated to that resource, when in fact, (true) assures the most your script's used were 2 bytes.
I have no hardcore experience with PHP, and tried to answer this by googling a lot. I hope it gives you some insight on what's going on.
In any case, try not to bash your keyboard in anger even though it doesn't make perfect sense. The "false" implementation of these methods are highly erratic, and shouldn't be used to compare with their "true" counterparts. Worst case scenario, it's just poor implementation on PHP's part.
Good luck on your PHP adventures!
Update
This didn't help me at all, but i did find the source codes for both functions at:
Here goes:
ZEND_API size_t zend_memory_usage(int real_usage)
{
#if ZEND_MM_STAT
if (real_usage) {
return AG(mm_heap)->real_size;
} else {
size_t usage = AG(mm_heap)->size;
return usage;
}
#endif
return 0;
}
ZEND_API size_t zend_memory_peak_usage(int real_usage)
{
#if ZEND_MM_STAT
if (real_usage) {
return AG(mm_heap)->real_peak;
} else {
return AG(mm_heap)->peak;
}
#endif
return 0;
}
References:
memory_get_peak_usage documentation
memory_get_usage documentation
Perfect script for calculating memory usage
Related
Its small code for test:
$strings = array('<big string here (2 Mb)');
$arr = array();
//--> memory usage here is 17.1Mb (checked by pmap)
echo memory_get_usage();//0.5Mb
//(i know, that other 16.6Mb of memory used by process are php libraries)
for($i = 0; $i < 20; ++$i)
{
$strings_local = array_merge($strings, array($i));
$arr[$i] = $strings_local;
unset($strings_local);
}
//--> memory usage here is 20.3Mb (checked by pmap)
echo memory_get_usage();//3.7Mb
//so, here its all ok, 17.1+3.2 = 20.3Mb
for($i = 0; $i < 20; ++$i)
{
unset($arr[$i]);
}
//--> memory usage here is 20.3Mb (checked by pmap)
//BUT?? i UNSET this variables...
echo memory_get_usage();//0.5Mb
So, seems like php is not free memory, even if you unset() your variable. How can i free memory after unset?
PHP has garbage collector which takes care of the memory management for you, which affects to memory usage (of the process) in several different ways.
First, when inspecting memory usage of a process outside of the process, even if PHP sees some memory to be freed, it may not be released back to the OS for optimization purposes related to memory allocation. This is reduce overhead from continuous frees and allocs, that happen more easily with GC’d languages, as allocation procedure is not visible to the actual program.
For that reason, even if one calls gc_collect_cycles() by hand, the memory may not be freed to the OS at all, but rather reused for future allocations. This causes PHP to see smaller memory usage than the process in reality uses, due to some early big reservation which never gets to freed back to the OS.
Second, due to nature of garbage collection, the memory may not be immediately freed after marked unused by the program. Calling gc_collect_cycles() will make the memory freed immediately, but it should be seen unnecessary, and does not work if you have logical (or something in PHP leaks) memory leak in your script.
For knowing what is going on, doing line by line inspection (for example with Xdebug’s function trace) would give you better insight about how PHP (or rather, your program) sees the memory usage.
Combining that to line-by-line inspection from outside of the process (for example your pmap commands) would tell if the PHP actually is freeing any memory at any point after reserving it.
Supposing a multidimensional associative array that, when printed as text with print_r(), creates a 470 KiB file. Is it reasonable to assume that the variable in question takes up half a MiB of server memory per instance if it is different for each user? Therefore if 1000 users hit the server at the same time almost half a GiB of memory will be consumed?
Thanks.
There is an excellent article on this topic at IBM:
http://www.ibm.com/developerworks/opensource/library/os-php-v521/
UPDATE
The original page was taken down, for now the JP version is still there https://www.ibm.com/developerworks/jp/opensource/library/os-php-v521/
Basic takeaways form it are that you can use memory_get_usage() to check how much memory your script currently occupies:
// This is only an example, the numbers below will differ depending on your system
echo memory_get_usage () "\ n";. // 36640
$ A = str_repeat ( "Hello", 4242);
echo memory_get_usage () "\ n";. // 57960
unset ($ a);
echo memory_get_usage () "\ n";. // 36744
Also, you can check the peak memory usage of your script with memory_get_peak_usage().
As an answer to your questions: print_r() is a representation of data which is bloated with text and formatting. The occupied memory itself will be less than the number of characters of print_r(). How much depends on the data. You should check it like in the example above.
Whatever result you get, it will be for each user executing the script, so yes - if 1000 users are requesting it at the same time, you will need that memory.
I came across the PHP's memory_get_usage() and memory_get_peak_usage().
The problem is that I found that these two functions do not provide the real memory used by the current script.
My test script is:
<?php
echo memory_get_usage();
echo '<br />';
$a = str_repeat('hello', 100000);
echo '<br />';
echo memory_get_usage();
echo '<br />';
echo memory_get_peak_usage();
?>
Which returns:
355120
5355216
5356008
What do you understand from this?
The first value is before executing the str_repeat() so it has to be the value of 0.
The second is after the process and it's OK to have a value greater than 0 but not that big value.
The third is the "peak" value and it's slightly greater than the second as I think it should be the biggest value in a processing microsecond.
So do you think that the real value of the current script's memory consumption should be like this:
memory_usage = the second memory usage - the first memory usage
peak_memory_usage = the third (peak_usage) - the first memory usage
which gives:
1) 5355216 - 355120 = 5000096 bytes
2) 5356008 - 355120 = 5000888 bytes
If this is how it works, I assume that the first 355120 bytes are the whole system allocated memory used by apache and other modules, as the first value never changes when you increase or decrease the number of repeats in the str_repeat(), only the two values after the process increase or decrease but never gets smaller that the first value.
According to the php manual, memory_get_usage returns the amount of memory allocated to php, not necessarily the amount being used.
Ok, your first assertion that the first memory_get_usage() should be 0 is wrong. According to PHP's documentation:
Returns the amount of memory, in
bytes, that's currently being
allocated to your PHP script.
Your script is running, therefore it must have some memory allocated to it. The first call informs you of how much that is.
Your second assertion that str_repeat() should not use that much memory is not looking at the whole picture.
You have the string "hello" (which uses 5 bytes) repeated 100,000 times, for a total of 500,000 bytes...minimum. The question is, how did PHP perform this action? Did they use code such as this? (pseudocode):
s = ""
for(i=0; i<100000; i++)
s += "hello"
This code would require that you reallocate a new string for each iteration of the for loop. Now I can't pretend to say that I know how PHP implements str_repeat(), but you have to be extremely careful with how you use memory to keep memory usage down. From the appearance of things, they did not manage memory in that function as well as they could have.
Third, the difference between the peak memory usage and current memory usage likely comes from the stack that was necessary to make the function call to str_repeat(), as well as any local variables necessary within that function. The memory was probably reclaimed when the function returned.
Finally, Apache runs in a different process and we are dealing with virtual memory. Nothing that Apache does will affect the result of memory_get_usage() as processes do not "share" virtual memory.
In my case (PHP 5.3.3 on Mac OS X 10.5) your script prints:
323964
824176
824980
Now, the difference between the second measurement and the first gives 500212, which is very close to the length of "hello" (5) times 100,000. So I would say no surprises here. The peak is a bit greater because of some temporary allocations when evaluating these statements.
(Your other questions are answered already)
I'm trying to track the memory usage of a script that processes URLs. The basic idea is to check that there's a reasonable buffer before adding another URL to a cURL multi handler. I'm using a 'rolling cURL' concept that processes a URLs data as the multi handler is running. This means I can keep N connections active by adding a new URL from a pool each time an existing URL processes and is removed.
I've used memory_get_usage() with some positive results. Adding the real_usage flag helped (not really clear on the difference between 'system' memory and 'emalloc' memory, but system shows larger numbers). memory_get_usage() does ramp up as URLs are added then down as the URL set is depleted. However, I just exceeded the 32M limit with my last memory check being ~18M.
I poll the memory usage each time cURL multi signals a request has returned. Since multiple requests may return at the same time, there's a chance a bunch of URLs returned data at the same time and actually jumped the memory usage that 14M. However, if memory_get_usage() is accurate, I guess that's what's happening.
[Update: Should have run more tests before asking I guess, increased php's memory limit (but left the 'safe' amount the same in the script) and the memory usage as reported did jump from below my self imposed limit of 25M to over 32M. Then, as expected slowly ramped down as URLs where not added. But I'll leave the question up: Is this the right way to do this?]
Can I trust memory_get_usage() in this way? Are there better alternative methods for getting memory usage (I've seen some scripts parse the output of shell commands)?
real_usage works this way:
Zend's memory manager does not use system malloc for every block it needs. Instead, it allocates a big block of system memory (in increments of 256K, can be changed by setting environment variable ZEND_MM_SEG_SIZE) and manages it internally. So, there are two kinds of memory usage:
How much memory the engine took from the OS ("real usage")
How much of this memory was actually used by the application ("internal usage")
Either one of these can be returned by memory_get_usage(). Which one is more useful for you depends on what you are looking into. If you're looking into optimizing your code in specific parts, "internal" might be more useful for you. If you're tracking memory usage globally, "real" would be of more use. memory_limit limits the "real" number, so as soon as all blocks that are permitted by the limit are taken from the system and the memory manager can't allocate a requested block, there the allocation fails. Note that "internal" usage in this case might be less than the limit, but the allocation still could fail because of fragmentation.
Also, if you are using some external memory tracking tool, you can set this
environment variable USE_ZEND_ALLOC=0 which would disable the above mechanism and make the engine always use malloc(). This would have much worse performance but allows you to use malloc-tracking tools.
See also an article about this memory manager, it has some code examples too.
I also assume memory_get_usage() is safe but I guess you can compare both methods and decide for yourself, here is a function that parses the system calls:
function Memory_Usage($decimals = 2)
{
$result = 0;
if (function_exists('memory_get_usage'))
{
$result = memory_get_usage() / 1024;
}
else
{
if (function_exists('exec'))
{
$output = array();
if (substr(strtoupper(PHP_OS), 0, 3) == 'WIN')
{
exec('tasklist /FI "PID eq ' . getmypid() . '" /FO LIST', $output);
$result = preg_replace('/[\D]/', '', $output[5]);
}
else
{
exec('ps -eo%mem,rss,pid | grep ' . getmypid(), $output);
$output = explode(' ', $output[0]);
$result = $output[1];
}
}
}
return number_format(intval($result) / 1024, $decimals, '.', '');
}
Use xdebug, as it was recently (January of 29th) updated to now include memory profiling information. It keeps track of the function calls and how much memory they consume. This allows you to get very insightful view into your code and at the very least sets you in a direction of being aware of the problems.
The documentation is helpful, but essentially you, install it enable the profiling xdebug.profiler_enable = 1 and give the output xdebug.profiler_output_dir=/some/path to a tool such as qcachegrind to do the heavy lifting, letting visually see it.
Well I have never really had a memory problem with my PHP scripts so I do not think I could be of much help finding the cause of the problem but what I can recomend is that you get a PHP accelerator, you will notice a serious performance increase and memory usage with decline. Here is a list of accelerators and an article comparing a few of them (3x better performance with any of them)
Wikipedia List
Benchmark
The benchmarks are 2 years old but you get the idea of the performance increases.
If you have to you can also increase you memory limit in PHP if you are still having problems even with the accelerator. Open up your php.ini and find:
memory_limit = 32M;
and just increase it a little.
Here is my code, which creates 2d array filled with zeros, array dimensions are (795,6942):
function zeros($rowCount, $colCount){
$matrix = array();
for ($rowIndx=0; $rowIndx<$rowCount; $rowIndx++){
$matrix[] = array();
for($colIndx=0; $colIndx<$colCount; $colIndx++){
$matrix[$rowIndx][$colIndx]=0;
}
}
return $matrix;
}
$matrix = zeros(795,6942);
And here is the error that I receive:
Allowed memory size of 134217728 bytes exhausted (tried to allocate 35 bytes)
Any ideas how to solve this?
As a quick calculation, you are trying to create an array that contains :
795*6942 = 5,518,890
integers.
If we consider that one integer is stored on 4 bytes (i.e. 32 bits ; using PHP, it not be less), it means :
5518890*4 = 22,075,560
bytes.
OK, quick calculation... result is "it should be OK".
But things are not that easy, unfortunatly :-(
I suppose it's related to the fact that data is stored by PHP using an internal data-structure that's much more complicated than a plain 32 bits integer
Now, just to be curious, let's modify your function so it outputs how much memory is used at the end of each one of the outer for-loop :
function zeros($rowCount, $colCount){
$matrix = array();
for ($rowIndx=0; $rowIndx<$rowCount; $rowIndx++){
$matrix[] = array();
for($colIndx=0; $colIndx<$colCount; $colIndx++){
$matrix[$rowIndx][$colIndx]=0;
}
var_dump(memory_get_usage());
}
return $matrix;
}
With this, I'm getting this kind of output (PHP 5.3.2-dev on a 64bits system ; memory_limit is set to 128MB -- which is already a lot !) :
int 1631968
int 2641888
int 3651808
...
...
int 132924168
int 133934088
Fatal error: Allowed memory size of 134217728 bytes exhausted
Which means each iteration of the outer for-loop requires something like 1.5 MB of memory -- and I only get to 131 iterations before the script runs out of memory ; and not 765 like you wanted.
Considering you set your memory_limit to 128M, you'd have to set it to something really much higher -- like
128*(765/131) = 747 MB
Well, even with
ini_set('memory_limit', '750M');
it's still not enough... with 800MB, it seems enough ;-)
But I would definitly not recommend setting memory_limit to such a high value !
(If you have 2GB of RAM, your server will not be able to handle more than 2 concurrent users ^^ ;; I wouldn't actually test this if my computer had 2GB of RAM, to be honest)
The only solution I see here is for you to re-think your design : there has to be something else you can do than use this portion of code :-)
(BTW : maybe "re-think your design" means using another language PHP : PHP is great when it comes to developping web-sites, but is not suited to every kind of problem)
The default PHP array implementation is very memory-intensive. If you are just storing integers (and lots of them), you probably want to look at SplFixedArray. It uses a regular contiguous memory block to store the data, as opposed to the traditional hash structure.
You should try increasing the amount of memory available to PHP:
ini_set('memory_limit', '32M');
in your PHP file.