I see comparisons between mysqli_fetch_array() and mysqli_fetch_all() that say that with mysqli_fetch_all() it will take more memory and I will have to iterate over the array.
mysqli_fetch_all() is one call to mysqli but mysqli_fetch_array() is one call per row, for example 100 calls.
I don't know how the mysqli processing works: is calling mysqli_fetch_array() really more efficient when you also take the number of calls into account?
(I already understand that the returned data can be associative arrays or not)
It has nothing to do with whatever efficiency. It's all about usability only.
fetch_all() is a thing that is called a "syntax sugar" - a shorthand to automate a frequently performed operation. It can be easily implemented as a userland function:
function mysqli_fetch_all ($resouce, $mode = MYSQLI_BOTH)
{
$ret = [];
while ($row = $resource->fetch_array($mode))
{
$ret[] = $row;
}
return $ret;
}
Thus you can tell use cases for these functions:
fetch_all() have to be used if you need an array, consists of all the returned rows, that will be used elsewhere.
fetch_assoc() in a loop have to be used if you're going to process all the rows one by one right in place.
As simple as that.
These functions bear different purpose and thus there is no point in comparing them.
Note that PDO is tenfold more sweet in terms of syntax sugar, as it's fetchAll() function can return data in dozens different formats
From PHP's page on mysql_fetch_all():
I tested using fetch_all versus while / fetch_array and:
fetch_all uses less memory (but not for so much).
In my case (test1 and test2): 147008,262848 bytes (fetch_all) versus 147112,262888 bytes (fetch_array & while).
So, about the memory, in both cases are the same.
However, about the performance:
My test takes :350ms (worst case) using fetch_all, while it takes 464ms (worst case) using fetch_array, or about 35% worst using fetch_array and a while cycle.
So, using fetch_all, for a normal code that returns a moderate amount of information is:
a. cleaner (a single line of code)
b. uses less memory (about 0.01% less)
c. faster.
php 5.6 32bits, windows 8.1 64bits
Related
I have an array of thousands of rows and I want to know what is the best way or the best prectices to count the number of rows in PHP that have a coincidence on it.
In the example you can see that I can find the number of records that match with a range.
I´m thinking in this 2 options:
Option 1:
$rowCount = 0;
foreach ($this->data as $row) {
if ($row['score'] >= $rangeStart && $row['score'] <= $rangeEnd) {
$rowCount++;
}
}
return $rowCount;
Option 2:
$countScoreRange = array_filter($this->data, function($result) {
return $result['score'] >= $this->rangeStart && $result['score'] <= $this->rangeEnd;
});
return count($countScoreRange);
Thanks in advance.
it depends on what you mean when are you speaking about best practices?
if your idea about best practice is about performance, it can say that there is one tradeoff that you must care about"
**speed <===> memory**
if you need performance about memory :
in this way : when you think about performance in iterating an iterable object in PHP, you can Use YIELD to create a generator function , from PHP doc :
what is generator function ?
A generator function looks just like a normal function, except that instead of returning a value, a generator yields as many values as it needs to. Any function containing yield is a generator function.
why we must use generator function ?
A generator allows you to write code that uses foreach to iterate over a set of data without needing to build an array in memory, which may cause you to exceed a memory limit, or require a considerable amount of processing time to generate.
so it's better to dedicate a bit of memory instead of reserving an array of thousands
Unfortunately:
Unfortunately, PHP also does not allow you to use the array traversal functions on generators, including array_filter, array_map, etc.
so till here you know if :
you are iterating an array
of thousands element
especially if you use it in a function and the function runs many where.
and you care about performance specially in memory usage.
it's highly recommended to use generator functions instead.
** but if you need performance about speed :**
the comparison will be about 4 things :
for
foreach
array_* functions
array functions (just like nexT() , reset() and etc.)
The 'foreach' is slow in comparison to the 'for' loop. The foreach copies the array over which the iteration needs to be performed.
but you can do some tricks if you want to use it :
For improved performance, the concept of references needs to be used. In addition to this, ‘foreach’ is easy to use.
what about foreach and array-filter ?
as the previous answer said, also based on this article , also this : :
it is incorrect when we thought that array_* functions are faster!
Of course, if you work on the critical system you should consider this advice. Array functions are a little bit slower than basic loops but I think that there is no significant impact on performance.
Using foreach is much faster, and incrementing your count is also faster than using count().
I once tested both performance and foreach was about 3x faster than array_filter.
I'll go with the first option.
As far as I know second and third expressions are executed every time in a for loop.
I always took for granted performance wise second option is recommended, can anyone confirm this?
1) for($i=0;$i<=dosomething();$i++) [...]
2)
$max = dosomething();
for($i=0;$i<=$max;$i++) [...]
You shouldn't call a function inside of a loop definition because that function will be executed every iteration. When you only have a small loop the effect is negligible, however if you have a loop of hundreds or thousands of iterations you'll definitely notice.
But even if you only have a small loop, it's just bad practice. So in a word: don't.
Unless your dosomething() function returns different values and it can be done in a single shot, it's better to use the second method.
$options = array(1,2,3,4,5);
$element_count = count($options);
Functions like count() that returns same value in multiple calls can be saved in a one variable and use it in your for loop.
If you are very strict for performance, use ++$i instead of $i++
The second method is always going to preform better, especially if there is substantial work to be done in doSomething(). If you are only doing tens of loops, and doSomething() is just returning a local variable, then it won't make a noticeable difference.
Yes I confirm you can search benchmarks if you want.
Though I don't know if it's true if it's only a getter to an object
I want to write a condition where I want to know if my function needs a return value or it can be executed as a procedure. Basically it should look like:
foo($x) {
$x++;
echo $x;
if(is_return_needed()) {
return $x;
}
}
where is_return_needed() is the condition if a return value is needed.
And how it should work:
echo foo(50); // should print AND return 51
bar(foo(50)); // should print AND return 51 to bar() function
foo(50); // should only print the value, because the returned value will not be used
And please don't tell me there's no reason to do this. I know I can send an additional boolean argument to the function which will be the condition, but is there a better way to achieve this?
Returning an object (or a large string), PHP will not make a copy of that object/string. That means returning alone, will not slow down your application. I found an article that explains it pretty well.
If the function can avoid building this large object at all, it will become faster. If you change the result of the function outside, or change it just before returning, it will become slower (is has to do the copy then). Building a long string with always adding a small part is very expensive, because every time it has to allocate a new big block of memory.
That said, we would have to see your code, to understand the slowdown you described. Returning a result only sometimes, is a very bad advice, every developer using your code will have a hard time to understand this behaviour. Sooner or later your application will become unstable if you use this often. Actually i find it even dangerous to return mixed typed values, as PHP often does itself.
In a PHP program, I sequentially read a bunch of files (with file_get_contents), gzdecode them, json_decode the result, analyze the contents, throw most of it away, and store about 1% in an array.
Unfortunately, with each iteration (I traverse over an array containing the filenames), there seems to be some memory lost (according to memory_get_peak_usage, about 2-10 MB each time). I have double- and triple-checked my code; I am not storing unneeded data in the loop (and the needed data hardly exceeds about 10MB overall), but I am frequently rewriting (actually, strings in an array). Apparently, PHP does not free the memory correctly, thus using more and more RAM until it hits the limit.
Is there any way to do a forced garbage collection? Or, at least, to find out where the memory is used?
it has to do with memory fragmentation.
Consider two strings, concatenated to one string. Each original must remain until the output is created. The output is longer than either input.
Therefore, a new allocation must be made to store the result of such a concatenation. The original strings are freed but they are small blocks of memory.
In a case of 'str1' . 'str2' . 'str3' . 'str4' you have several temps being created at each . -- and none of them fit in the space thats been freed up. The strings are likely not laid out in contiguous memory (that is, each string is, but the various strings are not laid end to end) due to other uses of the memory. So freeing the string creates a problem because the space can't be reused effectively. So you grow with each tmp you create. And you don't re-use anything, ever.
Using the array based implode, you create only 1 output -- exactly the length you require. Performing only 1 additional allocation. So its much more memory efficient and it doesn't suffer from the concatenation fragmentation. Same is true of python. If you need to concatenate strings, more than 1 concatenation should always be array based:
''.join(['str1','str2','str3'])
in python
implode('', array('str1', 'str2', 'str3'))
in PHP
sprintf equivalents are also fine.
The memory reported by memory_get_peak_usage is basically always the "last" bit of memory in the virtual map it had to use. So since its always growing, it reports rapid growth. As each allocation falls "at the end" of the currently used memory block.
In PHP >= 5.3.0, you can call gc_collect_cycles() to force a GC pass.
Note: You need to have zend.enable_gc enabled in your php.ini enabled, or call gc_enable() to activate the circular reference collector.
Found the solution: it was a string concatenation. I was generating the input line by line by concatenating some variables (the output is a CSV file). However, PHP seems not to free the memory used for the old copy of the string, thus effectively clobbering RAM with unused data. Switching to an array-based approach (and imploding it with commas just before fputs-ing it to the outfile) circumvented this behavior.
For some reason - not obvious to me - PHP reported the increased memory usage during json_decode calls, which mislead me to the assumption that the json_decode function was the problem.
There's a way.
I had this problem one day. I was writing from a db query into csv files - always allocated one $row, then reassigned it in the next step. Kept running out of memory. Unsetting $row didn't help; putting an 5MB string into $row first (to avoid fragmentation) didn't help; creating an array of $row-s (loading many rows into it + unsetting the whole thing in every 5000th step) didn't help. But it was not the end, to quote a classic.
When I made a separate function that opened the file, transferred 100.000 lines (just enough not to eat up the whole memory) and closed the file, THEN I made subsequent calls to this function (appending to the existing file), I found that for every function exit, PHP removed the garbage. It was a local-variable-space thing.
TL;DR
When a function exits, it frees all local variables.
If you do the job in smaller portions, like 0 to 1000 in the first function call, then 1001 to 2000 and so on, then every time the function returns, your memory will be regained. Garbage collection is very likely to happen on return from a function. (If it's a relatively slow function eating a lot of memory, we can safely assume it always happens.)
Side note: for reference-passed variables it will obviously not work; a function can only free its inside variables that would be lost anyway on return.
I hope this saves your day as it saved mine!
I've found that PHP's internal memory manager is most-likely to be invoked upon completion of a function. Knowing that, I've refactored code in a loop like so:
while (condition) {
// do
// cool
// stuff
}
to
while (condition) {
do_cool_stuff();
}
function do_cool_stuff() {
// do
// cool
// stuff
}
EDIT
I ran this quick benchmark and did not see an increase in memory usage. This leads me to believe the leak is not in json_decode()
for($x=0;$x<10000000;$x++)
{
do_something_cool();
}
function do_something_cool() {
$json = '{"a":1,"b":2,"c":3,"d":4,"e":5}';
$result = json_decode($json);
echo memory_get_peak_usage() . PHP_EOL;
}
I was going to say that I wouldn't necessarily expect gc_collect_cycles() to solve the problem - since presumably the files are no longer mapped to zvars. But did you check that gc_enable was called before loading any files?
I've noticed that PHP seems to gobble up memory when doing includes - much more than is required for the source and the tokenized file - this may be a similar problem. I'm not saying that this is a bug though.
I believe one workaround would be not to use file_get_contents but rather fopen()....fgets()...fclose() rather than mapping the whole file into memory in one go. But you'd need to try it to confirm.
HTH
C.
Call memory_get_peak_usage() after each statement, and ensure you unset() everything you can. If you are iterating with foreach(), use a referenced variable to avoid making a copy of the original (foreach()).
foreach( $x as &$y)
If PHP is actually leaking memory a forced garbage collection won't make any difference.
There's a good article on PHP memory leaks and their detection at IBM
There recently was a similar issue with System_Daemon. Today I isolated my problem to file_get_contents.
Could you try using fread instead? I think this may solve your problem.
If it does, it's probably time to do a bugreport over at PHP.
I am aware of that All associated result memory is automatically freed at the end of the script's execution. But would you recommend using it, if I am using a quite of lot of somewhat similar actions as below?
$sql = "select * from products";
$result = mysql_query($sql);
if($result && mysql_num_rows($result) > 0) {
while($data = mysql_fetch_assoc($result)) {
$sql2 = "insert into another_table set product_id = '".$data['product_id']."'
, product_name = '".$data['product_name']."'
";
$result2 = mysql_query($sql2);
**mysql_free_result($result2);**
}
}
Thanks.
Quoting the documentation of mysql_free_result :
mysql_free_result() only needs to be
called if you are concerned about how
much memory is being used for queries
that return large result sets. All
associated result memory is
automatically freed at the end of the
script's execution.
So, if the documentation says it's generally not necessary to call that function, I would say it's not really necessary, nor good practice, to call it ;-)
And, just to say : I almost never call that function myself ; memory is freed at the end of the script, and each script should not eat too much memory.
An exception could be long-running batches that have to deal with large amounts of data, though...
Yes, it is good practice to use mysql_free_result($result). The quoted documentation in the accepted answer is inaccurate. That is what the documentation says, but that doesn't make any sense. Here is what it says:
mysql_free_result() only needs to be called if you are concerned about how much memory is being used for queries that return large result sets. All associated result memory is automatically freed at the end of the script's execution.
The first part of the first sentence is correct. It is true that you don't need to use it for reasons other than memory concerns. Memory concerns are the only reason to use it. However, the second part of the first sentence doesn't make any sense. The claim is that you would only be concerned about memory for queries that return large result sets. This is very misleading as there are other common scenarios where memory is a concern and calling mysql_free_result() is very good practice. Any time queries may be run an unknown number of times, more and more memory will be used up if you don't call mysql_free_result(). So if you run your query in a loop, or from a function or method, it is usually a good idea to call mysql_free_result(). You just have to be careful not to free the result until after it will not be used anymore. You can shield yourself from having to think about when and how to use it by making your own select() and ex() functions so you are not working directly with result sets. (None of the code here is exactly the way I would actually write it, it is more illustrative. You may want to put these in a class or special namespace, and throw a different Exception type, or take additional parameters like $class_name, etc.)
// call this for select queries that do not modify anything
function select($sql) {
$array= array();
$rs= query($sql);
while($o= mysql_fetch_object($rs))
$array[]= $o;
mysql_free_result($rs);
return $array;
}
// call this for queries that modify data
function ex($sql) {
query($sql);
return mysql_affected_rows();
}
function query($sql) {
$rs= mysql_query($sql);
if($rs === false) {
throw new Exception("MySQL query error - SQL: \"$sql\" - Error Number: "
.mysql_errno()." - Error Message: ".mysql_error());
}
return $rs;
}
Now if you only call select() and ex(), you are just dealing with normal object variables and only normal memory concerns instead of manual memory management. You still have to think about normal memory concerns like how much memory is in use by the array of objects. After the variable goes out of scope, or you manually set it to null, it become available for garbage collection so PHP takes care of that for you. You may still want to set it to null before it goes out of scope if your code does not use it anymore and there are operations following it that use up an unknown amount of memory such as loops and other function calls. I don't know how result sets and functions operating on them are implemented under the hood (and even if I did, this could change with different/future versions of PHP and MySQL), so there is the possibility that the select() function approximately doubles the amount of memory used just before mysql_free_result($rs) is called. However using select() still eliminates what us usually the primary concern of more and more memory being used during loops and various function calls. If you are concerned about this potential for double memory usage, and you are only working with one row at a time over a single iteration, you can make an each() function that will not double your memory usage, and will still shield you from thinking about mysql_free_result():
each($sql,$fun) {
$rs= query($sql);
while($o= mysql_fetch_object($rs))
$fun($o);
mysql_free_result($rs);
}
You can use it like this:
each("SELECT * FROM users", function($user) {
echo $user->username."<BR>";
});
Another advantage of using each() is that it does not return anything, so you don't have to think about whether or not to set the return value to null later.
The answer is of course YES in mysqli.
Take a look at PHP mysqli_free_result documentation:
You should always free your result with mysqli_free_result(), when your result object is not needed anymore.
I used to test it with memory_get_usage function:
echo '<br>before mysqli free result: '.memory_get_usage();
mysqli_free_result($query[1]);
echo '<br>after mysqli free result'.memory_get_usage();
And it is the result:
before mysqli free result:2110088
after mysqli free result:1958744
And here, we are talking about 151,344 bytes of memory in only 1000 rows of mysql table.
How about a million rows and how is it to think about large projects?
mysqli_free_result() is not only for large amount of data, it is also a good practice for small projects.
It depends on how large your queries are or how many queries you run.
PHP frees the memory at the end of the script(s) automatically, but not during the run. So if you have a large amount of data comming from a query, better free the result manually.
I would say: YES, it is good practice because you care about memory during the development or your scripts and that is what makes a good developer :-)