I have PHP program that requires me to instantiate 1800 objects, and each object is associated with 7-10 arrays filled with historical data (about 500 records per array).This program is run by cron every 5 minutes, and not by users.
Anyways, the designer of the program says instantiating 1800 objects at once is required, and is not something we can change. My question is whether or not instantiating this many objects alone is a "code smell", and if having this much data in memory (arrays consisting of a total of 9,000,000 records), is something that would be hard for PHP to handle (assuming adequate memory is available on the host).
Thanks
Classes and objects are mostly a conceptual tool used to organise code in a logical fashion that more or less applies to "things" in the real world. There's no significant difference for the computer when executing code written procedurally vs. object oriented code. The OO code may add a little bit of overhead compared to code written in the most optimal procedural way, but you will hardly ever notice this difference. 1800 objects can be created and destroyed within milliseconds, repeatedly. They by themselves are not a problem.
The question is: does writing it this way in OO significantly help code organisation? If done properly, likely yes. Is there any other realistic way to write the same algorithm in a procedural way which is significantly faster in execution? Would this other way be as logically structured, understandable and maintainable? Would the difference in code level quality be worth the difference in performance? Is it really too slow with its 1800 objects? Are the objects the bottleneck (likely: no) or is the overall algorithm and approach the bottleneck?
In other words: there's no reason to worry about 1800 objects unless you have a clear indication that they are a bottleneck, which they likely are not in and of themselves. Storing the same data in memory without an object wrapper will not typically significantly reduce any resource usage.
It would be slow as an application to initialize all those objects for your system to run. Now I know why you would do it as I have done it before - i would load a lookup object to avoid tapping into the DB if I'm going to do a look up.
However 1800 objects with a 500 record per array - that's pretty heavy and defeats the purpose of touching a database. Im aware that memory shall be available but considering that this is a load up without the crunch - I'm unsure that the 5mins cron will finish.
I suggest to benchmark this with a profiler to see what is the memory used and the time elapse before running this.
There isn't a (practically relevant) limit for the amount of objects itself. As long as there is enough physical RAM, you can always increase the memory llimit. However, from an architectural standpoint it might be very unwise to keep all of this in RAM for the whole execution time, when it is actually not needed. Because of their high dynamics PHP arrays are quite expansive, so it could be a massive performance hit. However, without any details or profiling it is not possible to give you a definitive answer.
But admittedly, it seems quite odd that so many objects are needed. A DBMS might be an alternative for handling this amount of data.
I'm creating a web application that does some very heavy floating point arithmetic calculations, and lots of them! I've been reading a lot and have read you can make C(and C++) functions and call them from within PHP, I was wondering if I'd notice a speed increase by doing so?
I would like to do it this way even if it's only a second difference, unless it's actually slower.
It all depends on the actual number of calculations you are doing. If you have thousands of calculations to do then certainly it will be worthwhile to write an extension to handle it for you. In particular, if you have a lot of data this is where PHP really fails: it's memory manager can't handle a lot of objects, or large arrays (based on experience working with such data).
If the algorithm isn't too difficult you may wish to write it in PHP first anyway. This gives you a good reference speed but more importantly it'll help define exactly what API you need to implement in a module.
Update to "75-100 calculations with 6 numbers".
If you are doing this only once per page load I'd suspect it won't be a significant part of the overall load time (depends what else you do of course). If you are calling this function many times then yes, even 75 ops might be slow -- however since you use only 6 variables perhaps their optimizer will do a good job (whereas with 100 variables it's pretty much guaranteed not to).
Check SWIG.
Swig is a way to make php (and other languages) modules from your C sources rather easily.
Memory management is not something that most PHP developers ever need to think about. I'm running into an issue where my command line script is running out of memory. It performs multiple iterations over a large array of objects, making multiple database requests per iteration. I'm sure that increasing the memory ceiling may be a short term fix, but I don't think it's an appropriate long-term solution. What should I be doing to make sure that my script is not using too much memory, and using memory efficiently?
The golden rule
The number one thing to do when you encounter (or expect to encounter) memory pressure is: do not read massive amounts of data in memory at once if you intend to process them sequentially.
Examples:
Do not fetch a large result set in memory as an array; instead, fetch each row in turn and process it before fetching the next
Do not read large text files in memory (e.g. with file); instead, read one line at a time
This is not always the most convenient thing in PHP (arrays don't cut it, and there is a lot of code that only works on arrays), but in recent versions and especially after the introduction of generators it's easier than ever to stream your data instead of chunking it.
Following this practice religiously will "automatically" take care of other things for you as well:
There is no longer any need to clean up resources with a big memory footprint by closing them and losing all references to them on purpose, because there will be no such resources to begin with
There is no longer a need to unset large variables after you are done with them, because there will be no such variables as well
Other things to do
Be careful of creating closures inside loops; this should be easy to do, as creating such inside loops is a bad code smell. You can always lift the closure upwards and give it more parameters.
When expecting massive input, design your program and pick algorithms accordingly. For example, you can mergesort any amount of text files of any size using a constant amount of memory.
You could try profiling it puting some calls to memory_get_usage(), to look for the place where it's peaking.
Of course, knowing what the code really does you'll have more information to reduce its memory usage.
When you compute your large array of objects, try to not compute it all at once. Walk in steps and process elements as you walk then free memory and take next elements.
It will take more time, but you can manage the amount of memory you use.
I am a PHP developer and I have always thought that micro-optimizations are not worth the time. If you really need that extra performance, you would either write your software so that it's architecturally faster, or you write a C++ extension to handle slow tasks (or better yet, compile the code using HipHop). However, today a work mate told me that there is a big difference in
is_array($array)
and
$array === (array) $array
and I was like "eh, that's a pointless comparison really", but he wouldn't agree with me.. and he is the best developer in our company and is taking charge of a website that does about 50 million SQL queries per day -- for instance. So, I am wondering here: could he be wrong or is micro-optimization really worth the time and when?
Well, for a trivially small array, $array === (array) $array is significantly faster than is_array($array). On the order of over 7 times faster. But each call is only on the order of 1.0 x 10 ^ -6 seconds (0.000001 seconds). So unless you're calling it literally thousands of times, it's not going to be worth it. And if you are calling it thousands of times, I'd suggest you're doing something wrong...
The difference comes when you're dealing with a large array. Since $array === (array) $array requires a new variable to be copied requires the array to be iterated over internally for the comparison, it'll likely be SIGNIFICANTLY slower for a large array. For example, on an array with 100 integer elements, is_array($array) is within a margin of error (< 2%) of is_array() with a small array (coming in at 0.0909 seconds for 10,000 iterations). But $array = (array) $array is extremely slow. For only 100 elements, it's already over twice as slow as is_array() (coming in at 0.203 seconds). For 1000 elements, is_array stayed the same, yet the cast comparison increased to 2.0699 seconds...
The reason it's faster for small arrays is that is_array() has the overhead of being a function call, where the cast operation is a simple language construct... And iterating over a small variable (in C code) will typically be cheaper than the function call overhead. But, for larger variables, the difference grows...
It's a tradeoff. If the array is small enough, iteration will be more efficient. But as the size of the array grows, it will becomes increasingly slower (and hence the function call will become faster).
Another Way To Look At It
Another way to look at it would be to examine the algorithmic complexity of each cast.
Let's take a look at is_array() first. It's source code basically shows it's an O(1) operation. Meaning it's a constant time operation. But we also need to look at the function call. In PHP, function calls with a single array parameter are either O(1) or O(n) depending on if copy-on-write needs to be triggered. If you call is_array($array) when $array is a variable reference, copy-on-write will be triggered and a full copy of the variable will occur.
So therefore is_array() is a best case O(1) and worst-case O(n). But as long as you're not using references, it's always O(1)...
The cast version, on the other hand, does two operations. It does a cast, then it does an equality check. So let's look at each separately. The cast operator handler first forces a copy of the input variable. No matter if it's a reference or not. So simply using the (array) casting operator forces an O(n) iteration over the array to cast it (via the copy_ctor call).
Then, it converts the new copy to an array. This is O(1) for arrays and primitives, but O(n) for objects.
Then, the identical operator executes. The handler is just a proxy to the is_identical_function(). Now, is_identical will short-circuit if $array is not an array. Therefore, it has a best case of O(1). But if $array is an array, it can short-circuit again if the hash tables are identical (meaning both variables are copy-on-write copies of each other). So that case is O(1) as well. But remember that we forced a copy above, so we can't do that if it's an array. So it's O(n) thanks to zend_hash_compare...
So the end result is this table of worst-case runtime:
+----------+-------+-----------+-----------+---------------+
| | array | array+ref | non-array | non-array+ref |
+----------+-------+-----------+-----------+---------------+
| is_array | O(1) | O(n) | O(1) | O(n) |
+----------+-------+-----------+-----------+---------------+
| (array) | O(n) | O(n) | O(n) | O(n) |
+----------+-------+-----------+-----------+---------------+
Note that it looks like they scale the same for references. They don't. They both scale linearly for referenced variables. But the constant factor changes. For example, in a referenced array of size 5, is_array will perform 5 memory allocations, and 5 memory copies, followed by 1 type check. The cast version, on the other hand, will perform 5 memory allocations, 5 memory copies, followed by 2 type checks, followed by 5 type checks and 5 equality checks (memcmp() or the like). So n=5 yields 11 ops for is_array, yet 22 ops for ===(array)...
Now, is_array() does have the O(1) overhead of a stack push (due to the function call), but that'll only dominate runtime for extremely small values of n (we saw in the benchmark above just 10 array elements was enough to completely eliminate all difference).
The Bottom Line
I'd suggest going for readability though. I find is_array($array) to be far more readable than $array === (array) $array. So you get the best of both worlds.
The script I used for the benchmark:
$elements = 1000;
$iterations = 10000;
$array = array();
for ($i = 0; $i < $elements; $i++) $array[] = $i;
$s = microtime(true);
for ($i = 0; $i < $iterations; $i++) is_array($array);
$e = microtime(true);
echo "is_array completed in " . ($e - $s) ." Seconds\n";
$s = microtime(true);
for ($i = 0; $i < $iterations; $i++) $array === (array) $array;
$e = microtime(true);
echo "Cast completed in " . ($e - $s) ." Seconds\n";
Edit: For the record, these results were with 5.3.2 on Linux...
Edit2: Fixed the reason the array is slower (it's due to the iterated comparison instead of memory reasons). See compare_function for the iteration code...
Micro-optimisation is worth it when you have evidence that you're optimising a bottleneck.
Usually it's not worth it - write the most readable code you can, and use realistic benchmarks to check the performance. If and when you find you've got a bottleneck, micro-optimise just that bit of code (measuring as you go). Sometimes a small amount of micro-optimisation can make a huge difference.
But don't micro-optimise all your code... it will end up being far harder to maintain, and you'll quite possibly find you've either missed the real bottleneck, or that your micro-optimisations are harming performance instead of helping.
Is micro-optimization worth the time?
No, unless it is.
In other words, a-priori, the answer is "no", but after you know a specific line of code consumes a healthy percent of clock time, then and only then is it worth optimizing.
In other words, profile first, because otherwise you don't have that knowledge. This is the method I rely on, regardless of language or OS.
Added: When many programmers discuss performance, from experts on down, they tend to talk about "where" the program spends its time. There is a sneaky ambiguity in that "where" that leads them away from the things that could save the most time, namely, function call sites. After all, the "call Main" at the top of an app is a "place" that the program is almost never "at", but is responsible for 100% of the time. Now you're not going to get rid of "call Main", but there are nearly always other calls that you can get rid of.
While the program is opening or closing a file, or formatting some data into a line of text, or waiting for a socket connection, or "new"-ing a chunk of memory, or passing a notification throughout a large data structure, it is spending great amounts of time in calls to functions, but is that "where" it is? Anyway, those calls are quickly found with stack samples.
As the cliche goes, micro-optimization is generally worth the time only in the smallest, most performance-critical hotspots of your code, only after you've proven that's where the bottleneck is. However, I'd like to flesh this out a little, to point out some exceptions and areas of misunderstanding.
This doesn't mean that performance should not be considered at all upfront. I define micro-optimization as optimizations based on low-level details of the compiler/interpreter, the hardware, etc. By definition, a micro-optimization does not affect big-O complexity. Macro-optimizations should be considered upfront, especially when they have a major impact on high-level design. For example, it's pretty safe to say that if you have a large, frequently accessed data structure, an O(N) linear search isn't going to cut it. Even things that are only constant terms but have a large and obvious overhead might be worth considering upfront. Two big examples are excessive memory allocation/data copying and computing the same thing twice when you could be computing it once and storing/reusing the result.
If you're doing something that's been done before in a slightly different context, there may be some bottlenecks that are so well-known that it's reasonable to consider them upfront. For example, I was recently working on an implementation of the FFT (fast Fourier Transform) algorithm for the D standard library. Since so many FFTs have been written in other languages before, it's very well-known that the biggest bottleneck is cache performance, so I went into the project immediately thinking about how to optimize this.
In general you should not write any optimisation which makes your code more ugly or harder to understand; in my book this definitely falls into this category.
It is much harder to go back and change old code than write new code, because you have to do regression testing. So in general, no code already in production should be changed for frivolous reasons.
PHP is such an incredibly inefficient language that if you have performance problems, you should probably look to refactor hot spots so they execute less PHP code anyway.
So I'd say in general no, and in this case no, and in cases where you absolutely need it AND have measured that it makes a provable difference AND is the quickest win (low-hanging fruit), yes.
Certainly scattering micro-optimisations like this throughout your existing, working, tested code is a terrible thing to do, it will definitely introduce regressions and almost certainly make no noticable difference.
Well, I'm going to assume that is_array($array) is the preferred way, and $array === (array) $array is the allegedly faster way (which bring up the question why isn't is_array implemented using that comparison, but I digress).
I will hardly ever go back into my code and insert a micro-optimization*, but I will often put them into the code as I write it, provided:
it doesn't slow my typing down.
the intent of the code is still clear.
That particular optimization fails on both counts.
* OK, actually I do, but that has more to do with me having a touch of OCD rather than good development practices.
We had one place where the optimisation was really helpful.
Here some comparison:
is_array($v) : 10 sec
$v === (array)$v : 3,3 sec
($v.'') === 'Array' : 2,6 sec
The last one does cast to string, an Array is always casted to a string with value 'Array'. This check will be wrong, if the $v is a string with value 'Array' (never happens in our case).
Well, there's more things than speed to take into consideration. When you read that 'faster' alternative, do you instantly think "Oh, this is checking to see if the variable is an array", or do you think "...wtf"?
Because really - when considering this method, how often is it called? What is the exact speed benefit? Does this stack up when the array is larger or smaller? One cannot do optimizations without benchmarks.
Also, one shouldn't do optimizations if they reduce the readability of the code. In fact, reducing that amount of queries by a few hundred thousand (and this is often easier than one would think) or optimizing them if applicable would be much, much more beneficial to performance than this micro-optimization.
Also, don't be intimidated by the guy's experience, as others have said, and think for yourself.
Micro-optimization is not worth it. Code readability is much more important than micro-optimization.
Great article about useless micro-optimization by Fabien Potencier (creator of the Symfony framework):
print vs echo, which one is faster?
Print uses one more opcode because it actually returns something. We
can conclude that echo is faster than print. But one opcode costs
nothing, really nothing. Even if a script have hundreds of calls to
print. I have tried on a fresh WordPress installation. The script
halts before it ends with a "Bus Error" on my laptop, but the number
of opcodes was already at more than 2.3 millions. Enough said.
IMHO, micro-optimizations are actually even more relevant than algorithmic optimizations today if you are working in a performance-critical field. This might be a big if because many people don't actually work in performance-critical areas even for performance-critical software since they might just be making high-level calls into a third party library which does the actual performance-critical work. For example, many people these days trying to write an image or video software might write non-performance-critical code expressing they want at the image level, not having to manually loop through several million pixels themselves at 100+ frames per second. The library does that for them.
When I say that micro-optimizations are more relevant than algorithmic ones today, I don't mean that, say, parallelized SIMD code that minimizes cache misses applying a bubble sort will beat an introsort or radix sort. What I mean is that professionals don't bubble sort large input sizes.
If you take any reasonably high-level language today, of which I include C++, you already have your share of reasonably efficient general-purpose data structures and algorithms at your fingertips. There's no excuse unless you're a beginning computer science student just getting your feet wet and reinventing the most primitive of wheels to be applying quadratic complexity sorts to massive input sizes or linear-time searches which can be accomplished in constant-time with the appropriate data structures.
So once you get past this beginner level, performance-critical applications still have wildly varying performance characteristics. Why? Why would one video processing software have three times the frame rate and more interactive video previews than the other when the developers aren't doing anything extremely dumb algorithmically? Why would one server doing a very similar thing be able to handle ten times the queries with the same hardware? Why would this software load a scene in 5 seconds while the other takes 5 minutes loading the same data? Why would this beautiful game have silky smooth and consistent frame rates while the other one is uglier, more primitive-looking with its graphics and lighting, and stutters here and there while taking twice the memory?
And that boils down to micro-optimizations, not algorithmic differences. Furthermore our memory hierarchy today is so skewed in performance, making previous algorithms that were thought to be good a couple of decades ago no longer as good if they exhibit poor locality of reference.
So if you want to write competitively-efficient software today, far more often than not, that will boil down to things like multithreading, SIMD, GPU, GPGPU, improving locality of reference with better memory access patterns (loop tiling, SoA, hot/cold field splitting, etc.), maybe even optimizing for branch prediction in extreme cases, and so forth, not so much algorithmic breakthroughs unless you're tackling an extremely unexplored territory where no programmers have ventured before.
There are still occasionally algorithmic breakthroughs that are potential game changers, like voxel-cone tracing recently. But those are exceptions and the people who come up with these often invest their lives to R&D (they're generally not people writing and maintaining large scale codebases), and it still boils down to micro-optimizations whether voxel-cone tracing can be applied to real-time environments like games or not. If you aren't good at micro-optimizations, you simply won't get the adequate framerates even using these algorithmic breakthroughs.
I've seen in a few places lately people saying that PHP has a poor capacity for recursion. Recently I wrote a recursive php function for graph traversal and found it to be very slow compared to java. I don't know whether this is because of php's capacity for recursion or because php is slower than java in general.
Some googling revealed this (http://bugs.php.net/bug.php?id=1901)
[7 Aug 1999 12:25pm UTC] zeev at cvs dot php dot net
PHP 4.0 (Zend) uses the stack for
intensive data, rather than using the
heap. That means that its tolerance
recursive functions is significantly
lower than that of other languages.
It's relatively easy to tell Zend not
to use the stack for this data, and
use the heap instead - which would
greatly increase the number of
recursive functions possible - in the
price of reduced speed. If you're
interested in such a setting, let me
know, we may add a compile-time
switch.
What does it mean to say that php uses the stack for intensive data? Does php not set up a run-time stack? Also, is it true in general that recursion in php is much slower than other languages? And by how much?
Thanks!
Okay, I'll take a stab at it.
First: "The stack" is the area used for function call tracking in standard C/C++ type programs. It's a place the operating system and programming language conventions define in memory, and it's treated like a stack (the data structure). When you call a C function fibbonaci(int i) then it places the variable i, and the return address of the function that was already busy calling it, on the stack. That takes some memory. When it's done with that function call, the memory is available again. The stack is of finite size. If you are storing very massive variables on it and make many many recursive calls, then you may run out of room. Right?
So.....
Apparently Zend has two ways to allocate data: on the heap (a more-general area to request memory from) and on the stack, and the stack is a more efficient place to have them because of the way things are programmed. (I don't know why, but I can guess. There may be very-low-level caching concerns - I expect the stack is likelier to be in L1 or L2 cache than arbitrary memory in the heap would be, because the CPU is very likely to be using that area very frequently - every time you call a function, in fact. There may also be allocation overhead for heap data access.)
"Intensive" data in this context, I believe, refers to data which is very likely to be used very soon or very often. It would make sense to use the speedier stack-based allocation for these variables. What sort of variables are you certain to be using very quickly? Well, how about parameters to a function? You're very likely to use those: otherwise why would you be bothering to pass them around? They're also probably likelier to be small data items (references to massive data structures rather than massive data structures themselves - because that gives you copying overhead, among other things). So the stack probably makes sense for storing PHP function parameters for most PHP programmers... but it fails sooner in recursion.
Hopefully that answers at least "what does this mean?". For your recursion performance question: Go benchmark it yourself; it probably depends on what sort of recursion you're trying to do.
At a guess, I'd say that your problem lies elsewhere than the recursion itself. For many things, Java is a lot faster than PHP. There are, sort of, ways to improve PHP's performance.
However, the PHP recursion limitation results in PHP running out of stack and crashing, with the dreaded 'stack overflow' message (pun sort of intended). At this point, your program ceases to execute.
If PHP is using a dynamic stack, you could see some (mild) slowdown due to the time it takes to realloc the stack to a larger block of memory.
Anyway, I'd need to know a bit more about what you're doing to pinpoint your performance problem, which is something I do for a living...