Can any body give me a a introduction of how to program efficiently minimizing memory usage in PHP program correctly and generate my program results using minimum memory ?
Based on how I read your question, I think you may be barking up the wrong tree with PHP. It was never designed for a low memory overhead.
If you just want to be as efficient as possible, then look at the other answers. Remember that every single variable costs a fair bit of memory, so use only what you have to, and let the garbage collector work. Make sure that you only declare variables in a local scope so they can get GC'd when the program leaves that scope. Objects will be more expensive than scalar variables. But the biggest common abuse I see are multiple copies of data. If you have a large array, operate directly on it rather than copying it (It may be less CPU efficient, but it should be more memory efficient).
If you are looking to run it in a low memory environment, I'd suggest finding a different language to use. PHP is nice because it manages everything for you (with respect to variables). But that type coersion and flexibility comes at a price (speed and memory usage). Each variable requires a lot of meta-data stored with it. So an 8 byte int (32 bit) would take 8 bytes to store in C, it will likely take more than 64 bytes in PHP (because of all of the "tracking" information associated with it such as type, name, scoping information, etc). That overhead is normally seen as ok since PHP was not designed for large memory loads. So it's a trade-off. More memory used for easier programming. But if you have tight memory constraints, I'd suggest moving to a different language...
It's difficult to give advice with so little information on what you're trying to do and why memory utilization is a problem. In the common scenarios (web servers that serve many requests), memory is not a limiting factory and it's preferable to serve the requests as fast as possible, even if this means sacrificing memory for speed.
However, the following general guidelines apply:
unset your variables as soon as you don't need them. In a program that's well written, this, however, won't have a big impact, as variables going out of scope have the same effect.
In long running scripts, with lot's of variables with circular references, and if using PHP 5.3, trey calling the garbage collector explicitly in certain points.
First of all: Don't try to optimize memory usage by using references. PHP is smart enough not to copy the contents of a variable if you do something like this:
$array = array(1,2,3,4,5,);
$var = $array;
PHP will only copy the contents of the variable when you write to it. Using references all the time because you think they will save you copying the variable content can often fire backwards ;)
But, I think your question is hard to answer, as long as you are more precise.
For example if you are working with files it can be recommendable not always to file_get_contents() the whole file, but use the f(open|...) functions to load only small parts of the file at once or even skip whole chunks.
Or if you are working with strings make use of functions which return a string offset instead of the rest of a string (e.g. strcspn instead of strpbrk) when possible.
Related
Can php script get allowed memory size and also how much memory can be allocated? I know that it is possible to clean memory using unset. But I'll like to understand how to create php scripts that consume less memory as possibile.
The basic mechanism which PHP uses is garbage collection
How it works in short is something like:
Say you have a certain memory location M allocated to store variable $m e.g.:
$m = [ 0,1,2,3,4,5 ]; //M refers to the memory which is storing this array
As long as $m keeps pointing to M then PHP is not allowed to destroy M. However if you do something like:
$m = null;
This makes $m point to nothing and therefore M no longer is referenced by anything. PHP at this point is allowed to clear that memory, but may not do so immediately. The point is if you ensure that you stop referencing something when you don't need it anymore you're giving PHP the opportunity to run as memory optimized as possible.
However, garbage collection for large complex applications is expensive so keep in mind that PHP may opt to delay garbage collection if it can.
unset will free the memory.
Though major memory consumption will be on the resources from io, db etc.
And for these tasks, it is very important to release the memory or free up the sources for further utilisation.
Also to note that while processing, the processed data will also have near similar usage. Hence freeing that after usage will also improve the use case.
To go more and more memory less, make functions pure as much as possible, where after execution of function, there is only output and no side effect. With this there will be less things in global memory space.
I have a php script to scrap a website (text files only). After running for few hours I noticed the script to stop for reaching the memory limit. I know I can increase the limit, but since the files the script loads are onlty HTML files I explain the reaching of the limit only with the inability of the script to empty the memory after each loop. Could I optimize my script's memory management by flush()ing its memory regularly?
In general, you shouldn't need to manually manage memory in PHP, as it has a high-level Memory Manager built in to the Zend Engine which takes care of this for you. However, it is useful to know a bit about how this works in order to better understand why your code is running out of memory.
As a very basic overview, PHP frees memory based on a "refcount" of how many variables are referencing a particular piece of data. So if you say $a = 'hello'; $b = $a;, a single piece of memory containing the string 'hello' will have a refcount of 2. If you call unset() on either variable, or they fall out of scope (e.g. at the end of the function they were defined in), the refcount will decrease. Once the refcount reaches zero, the data will be deleted and the memory freed. Note that "freed" in this case means freed for use by other parts of that PHP script, not necessarily freed back to the Operating System for use by other processes.
There are a few differences between PHP versions worth knowing:
The reference counting mechanism described above doesn't work if you have circular references (e.g. $obj1->foo = $obj2; $obj2->bar = $obj1;) because the reference count never reaches zero. In PHP 5.2 and earlier, this meant that such circular references led to memory leaks, and had to be manually handled by the programmer. In PHP 5.3, a "Garbage Collector" was added specifically to handle this case. It does not replace the normal refcount mechanism, but if circular references are common in your code, it may be worth reading up on.
PHP 5.4 included a large number of optimizations to the way PHP allocates and uses memory. AFAIK, none of these change the fundamental recommendations of how to write efficient code, they are just a good reason to upgrade your PHP version if you can.
Other than that, there are a few common tips for writing PHP code that makes good use of memory:
Make sure unused variables are discarded when no longer needed. In a well-structured program, this is often a non-issue, because most variables will be local to a particular function; when the function exits, they will go out of scope, and be freed. But if you are creating large intermediate variables, or dynamically creating large numbers of variables, manually calling unset() may be a good idea. And if your code is very linear, or uses large numbers of global and static variables, just refactoring it into a more modular structure may improve its memory performance as well as its readability, maintainability, etc.
Assigning or passing a variable by reference ($foo = &$bar) may cause PHP to use more memory than a straight assignment ($foo = $bar). This is because PHP uses a "Copy On Write" mechanism to to store variables with the same content in one location of memory, but reference assignment conflicts with this mechanism, so PHP has to copy the variable early.
Objects are more memory-hungry than scalar values (int, boolean, string) or arrays. This is one of the things that has been much improved in PHP 5.4, but is still worth thinking about - although obviously not to the exclusion of writing well-structured code!
You can unset variables as you no longer need them (e.g. unset($var) or $var = null). If you're on PHP 5.3 or later, you can also explicitly call the garbage collector: see gc_collect_cycles() and gc_enable().
Some functions seem to be worse than others. I recently found that array_merge_recursive() did horrible things to my code's memory footprint.
If you want to be able to analyse where the memory's going, you can use tools like Xdebug or XHProf/XHGui to help. e.g. Xdebug and tracing memory usage and Profiling with XHProf
See also:
Force freeing memory in PHP
php garbage collection while script running
Before starting, I'm not asking about standard coding practice or "etiquette." My question is more from curiosity with the internals of PHP. My research so far mostly seems to find people confused about scope in PHP.
Does re-using variables come with any benefit/detriment in PHP, either in memory or speed? For science.
Say you are sequentially accessing multiple files throughout a script.
Scenario A: your handles each have a variable, $file1, $file2, $file3, etc.
Scenario B: your handles all reuse the variable $fp
Will this theoretical scenario require respectively resource intensive scripts to matter? Will B allow garbage collection to get rid of the old handles while A won't? Will optimization through Zend make this a non-issue either way?
There is not a cut & dry answer to this question. Optimization and performance will depend heavily on your particular codebase, the platform it runs on, what else is running on the server, and more.
To start with your scenarios are too vague to provide an adequate answer. However, to touch on some of the prior comments/concerns...
PHP does not have very well defined rules for garbage collection. In THEORY scenario A will release the memory when a function exits thanks to garbage collection. In reality this rarely happens. There are a number of triggers that will cause garbage collection to release that memory, but behind the scenes the actual low-level free() and mallocs() are not cut & dry. If you watch your memory stack closely you will find that after a function exit the memory space for $file1, $file2, $file3 will remain. Sometimes until the entire application exits.
Your application construction will also determine which is faster, creating a new entry in the symbol table for $file1, $file2, $file3 or re-using $fp over & over. Re-using $fp, again IN THEORY, would typically mean the memory space does not need to be re-allocated and a new symbol table entry and corresponding management object does not need to be re-created. However this is not always the case. Sometimes re-using $fp actually can be slower because a destroy needs to be called first, then re-creating the object. In some corner cases it may be faster to just create a new $file1, $file2, $file3 on the iterative process and let garbage collection happen all-at-once.
So, the bottom line of all this....
You need to analyze and test your own apps in their native environment to learn how things behave in YOUR playground. It is rarely an "always do this" or "never do that" scenario.
Not confident on my answer, but I do found that reuse vars saves more memory, especially when re-using vars for query results as often time those vars will fill with a lot of other unwanted stuff in there.
You can use
echo memory_get_usage() at different stage of the code execution to see the difference and compare.
But it could get confusing as your code grows and makes it harder for people to read.
Also PHP runs garbage collection when the script is done. so how you name your vars probably won't have anything to do with it, rather it effects how much memory it uses during execution.
Memory management is not something that most PHP developers ever need to think about. I'm running into an issue where my command line script is running out of memory. It performs multiple iterations over a large array of objects, making multiple database requests per iteration. I'm sure that increasing the memory ceiling may be a short term fix, but I don't think it's an appropriate long-term solution. What should I be doing to make sure that my script is not using too much memory, and using memory efficiently?
The golden rule
The number one thing to do when you encounter (or expect to encounter) memory pressure is: do not read massive amounts of data in memory at once if you intend to process them sequentially.
Examples:
Do not fetch a large result set in memory as an array; instead, fetch each row in turn and process it before fetching the next
Do not read large text files in memory (e.g. with file); instead, read one line at a time
This is not always the most convenient thing in PHP (arrays don't cut it, and there is a lot of code that only works on arrays), but in recent versions and especially after the introduction of generators it's easier than ever to stream your data instead of chunking it.
Following this practice religiously will "automatically" take care of other things for you as well:
There is no longer any need to clean up resources with a big memory footprint by closing them and losing all references to them on purpose, because there will be no such resources to begin with
There is no longer a need to unset large variables after you are done with them, because there will be no such variables as well
Other things to do
Be careful of creating closures inside loops; this should be easy to do, as creating such inside loops is a bad code smell. You can always lift the closure upwards and give it more parameters.
When expecting massive input, design your program and pick algorithms accordingly. For example, you can mergesort any amount of text files of any size using a constant amount of memory.
You could try profiling it puting some calls to memory_get_usage(), to look for the place where it's peaking.
Of course, knowing what the code really does you'll have more information to reduce its memory usage.
When you compute your large array of objects, try to not compute it all at once. Walk in steps and process elements as you walk then free memory and take next elements.
It will take more time, but you can manage the amount of memory you use.
I've seen in a few places lately people saying that PHP has a poor capacity for recursion. Recently I wrote a recursive php function for graph traversal and found it to be very slow compared to java. I don't know whether this is because of php's capacity for recursion or because php is slower than java in general.
Some googling revealed this (http://bugs.php.net/bug.php?id=1901)
[7 Aug 1999 12:25pm UTC] zeev at cvs dot php dot net
PHP 4.0 (Zend) uses the stack for
intensive data, rather than using the
heap. That means that its tolerance
recursive functions is significantly
lower than that of other languages.
It's relatively easy to tell Zend not
to use the stack for this data, and
use the heap instead - which would
greatly increase the number of
recursive functions possible - in the
price of reduced speed. If you're
interested in such a setting, let me
know, we may add a compile-time
switch.
What does it mean to say that php uses the stack for intensive data? Does php not set up a run-time stack? Also, is it true in general that recursion in php is much slower than other languages? And by how much?
Thanks!
Okay, I'll take a stab at it.
First: "The stack" is the area used for function call tracking in standard C/C++ type programs. It's a place the operating system and programming language conventions define in memory, and it's treated like a stack (the data structure). When you call a C function fibbonaci(int i) then it places the variable i, and the return address of the function that was already busy calling it, on the stack. That takes some memory. When it's done with that function call, the memory is available again. The stack is of finite size. If you are storing very massive variables on it and make many many recursive calls, then you may run out of room. Right?
So.....
Apparently Zend has two ways to allocate data: on the heap (a more-general area to request memory from) and on the stack, and the stack is a more efficient place to have them because of the way things are programmed. (I don't know why, but I can guess. There may be very-low-level caching concerns - I expect the stack is likelier to be in L1 or L2 cache than arbitrary memory in the heap would be, because the CPU is very likely to be using that area very frequently - every time you call a function, in fact. There may also be allocation overhead for heap data access.)
"Intensive" data in this context, I believe, refers to data which is very likely to be used very soon or very often. It would make sense to use the speedier stack-based allocation for these variables. What sort of variables are you certain to be using very quickly? Well, how about parameters to a function? You're very likely to use those: otherwise why would you be bothering to pass them around? They're also probably likelier to be small data items (references to massive data structures rather than massive data structures themselves - because that gives you copying overhead, among other things). So the stack probably makes sense for storing PHP function parameters for most PHP programmers... but it fails sooner in recursion.
Hopefully that answers at least "what does this mean?". For your recursion performance question: Go benchmark it yourself; it probably depends on what sort of recursion you're trying to do.
At a guess, I'd say that your problem lies elsewhere than the recursion itself. For many things, Java is a lot faster than PHP. There are, sort of, ways to improve PHP's performance.
However, the PHP recursion limitation results in PHP running out of stack and crashing, with the dreaded 'stack overflow' message (pun sort of intended). At this point, your program ceases to execute.
If PHP is using a dynamic stack, you could see some (mild) slowdown due to the time it takes to realloc the stack to a larger block of memory.
Anyway, I'd need to know a bit more about what you're doing to pinpoint your performance problem, which is something I do for a living...