INCLUDE mysqli object
--
$sel = $mysqli->query("select * from `items`");
while($res = $sel->fetch_assoc()) {
$items[] = $res;
}
$sel->free_result();
$sel = $mysqli->query("select * from `sets`");
while($res = $sel->fetch_assoc()) {
$sets[] = $res;
}
$sel->free_result();
$sel = $mysqli->query("select * from `parts`");
while($res = $sel->fetch_assoc()) {
$parts[] = $res;
}
$sel->free_result();
--
DO OTHER STUFF
Are the first two times of executing $sel->free_result(); really needed?
I think they are unnecessary when I reuse the variable sel.
Do you agree with me?
According to the PHP documentation on the matter:
You should always free your result with mysqli_free_result(), when
your result object is not needed anymore.
Looking into the comments section of that page gives a reason why:
If you are getting this error: Internal SQL Bug: 2014, Commands out
of sync; you can't run this command now
Then you never called mysqli_result::free(),
mysqli_result::free_result(), mysqli_result::close(), or
mysqli_free_result() in your script, and must call it before executing
another stored procedure.
Basically, you don't need to do it, but there could be odd instances where you're unable to execute other procedures, and commands could be out of sync.
Another issue which could come from not doing it is that the result could be taking up a lot of memory so if you put new code in between the queries which needs memory, then you could hit an issue where the script crashes due to being unable to allocate more memory. By freeing the result you're helping to reduce the chances of this happening should you need to make changes between query executions.
At an extreme level, if you've got a bug in a different part of your system which can read other memory addresses and one of those queries has sensitive information stored in it, by not freeing the result, you're potentially giving an attacker a chance to read that sensitive information from memory.
There's bound to be other cases where there are issues, but those two sprung to mind.
In a nutshell, it's better to free the result and not have the information in memory when it's not needed.
Related
In the first code example on this page, this is the second last function called.
The comment says it frees the result set, but I am not able to understand what exactly is meant by freeing a result set? The result set is an Object so what is meant by freeing it?
/* free result set */
$result->free();
The creation of a result set is a form of dynamic memory allocation, almost certainly done when an SQL query was executed, something like:
$conn = some_connection_to_database();
$rows = $conn->query("select name from etc_passwd where group = 'admin'")
Since the intent is for you to then use that result set, it cannot be freed as part of the query itself.
So, once the query is run and the result set is returned, you are responsible for freeing that memory when you're done with it.
That is where free() will be used. The basic idea (in English) is:
Get result set from somewhere.
Use it.
Return it to its source (this is the free bit).
I have a PHP script which processes a "large" dataset (about 100K records) from a PDO query into a single collection of objects, in a typical loop:
while ($record = $query->fetch()) {
$obj = new Thing($record);
/* do some processing */
$list[] = $obj;
$count++;
}
error_log('Processed '.$count.' records');
This loop processes about 50% of the dataset and then inexplicably breaks.
Things I have tried:
Memory profiling: memory_get_peak_usage() consistently outputs about 63MB before the loop dies. The memory limit is 512MB, set through php.ini.
Using set_time_limit() to increase script execution time to 1 hour (3600 seconds). The loop breaks long before that and I don't see the usual error in the log for this one.
Setting PDO::MYSQL_ATTR_USE_BUFFERED_QUERY to false to avoid buffering the entire dataset
Logging out $query->errorInfo() immediately after the loop break. This was no help as the error code was "00000".
Checking the MySQL error log. Nothing of note in there before, after, or while this script runs.
Batching the processing into 20K-record chunks. No difference. Loop broke in the same spot. However, by "cleaning up" the PDO statement object at the end of each batch, I was able to get the processed total to 54%.
Other weird behavior:
When I set the memory limit using ini_set('memory_limit', '1024MB'), the loop actually dies earlier than with a smaller memory limit, at about 20% progress.
During this loop, the PHP process uses 100% CPU, but once it breaks, usage drops back down to 2%, despite immediate processing in another loop immediately afterwards. Likely, the connection with the MySQL server in the first loop is very resource-intensive.
I am doing this all locally using MAMP PRO if that makes any difference.
Is there something else that could be consistently breaking this loop that I haven't checked? Is this simply not a viable strategy for processing this many records?
UPDATE
After using a batching strategy (20K increments), I have started to see a MySQL error consistently around the third batch: MySQL server has gone away; possibly a symptom of a long-running unbuffered query.
If You really need to process 100K records on the fly, You should do the processing in SQL, and fetch the result as You need it - it should save a lot of time.
But You probably cant do that for some reason. You always process all the rows from statement, so use fetchAll once - and let MySQL alone after that, like that:
$records = $query->fetchAll()
foreach ($records as record)
{
$obj = new Thing($record);
/* do some processing */
$list[] = $obj;
$count++;
}
error_log('Processed '.$count.' records');
Also, select only rows that You will use.
If this does not help, You can try with this: Setting a connect timeout with PDO .
I usually try to minimize calls to MySQL where possible, but I've finally encountered the case where I have to do multiple calls to MySQL in a single script.
It looks like you can't use mysql_fetch_assoc twice in a single script(!).
It seems that mysql_data_seek is the solution, but I cannot seem to get it to work.
To be clear, I have two queries I need to make. The second query depends on results from the first... Here's the structure:
$result = mysql_query($query1);
while($row = mysql_fetch_assoc($result)){
$pos = $row['position'];
}
mysql_free_result($result); // result freed per comment below.
$query2 = ' '; //... dependent on $pos - in mysql shell this returns results!
$result2 = mysql_query($query2)
while($row = mysql_fetch_assoc($result2)){
echo $row['id'];
}
What happens above is that the second while loop returns no results even though the query should have nontrivial rows.
Just to be sure:
Is this how you clear the pointer from the previous result to be able to use mysql_fetch_assoc again?
mysql_data_seek($result,mysql_num_rows($result) - 1);
I'm not sure what to use as the second argument. Admittedly, I am not that clear on pointers, but it seems I should clear the pointer to 0. But I get this error:
Offset 0 is invalid for MySQL result index 8 (or the query data is unbuffered
Check your connection with mysql_error() and see if you're getting the "commands out of sync" error. If so, you need to call mysql_free_result() after completing the first query and before starting the second. You could also just call mysql_close() to close your database connection and then reopen a new one.
For more details, see this question.
OP changed the question, so see the edit
*Deleted the posted codes here**
EDIT
After your edited your question and made clear you have actually 2 resources it looks like there is something else wrong. You don't have to worry about pointer when you use two different resources to supply mysql_fetch_assoc(). The thing with mysql_fetch_assoc() is that it takes your param ($result) by reference.
Now to answer your question:
I usually try to minimize calls to MySQL where possible, but I've finally encountered the case where I have to do multiple calls to MySQL in a single script.
Nothing wrong with multiple SQL calls in one script. Although in general you should try to minimize the SQL calls (because they may hurt performance).
It looks like you can't use mysql_fetch_assoc twice in a single script(!).
Plain wrong. Ofc you can do it. As long as you note the above. However when you have two result sets this wouldn't be your problem.
It seems that mysql_data_seek is the solution, but I cannot seem to get it to work.
Again: this has nothing to do with it when you use two (different) result sets.
To be clear, I have two queries I need to make. The second query depends on results from the first.
This shouldn't be any problem at all. It looks like is something else wrong. Have you verified that the second query really is what you think it is? Are you sure there are records? Are you sure there aren't any (MySQL) errors. Do you have error reporting enabled? Have you tried printing out mysql_error()? To better be able to help you can you please provide your real code and not etc etc stuff? Maybe something else is going on.
Or maybe you are simply trying to run the second query inside the first loop. Which would be bad in so many ways.
I am aware of that All associated result memory is automatically freed at the end of the script's execution. But would you recommend using it, if I am using a quite of lot of somewhat similar actions as below?
$sql = "select * from products";
$result = mysql_query($sql);
if($result && mysql_num_rows($result) > 0) {
while($data = mysql_fetch_assoc($result)) {
$sql2 = "insert into another_table set product_id = '".$data['product_id']."'
, product_name = '".$data['product_name']."'
";
$result2 = mysql_query($sql2);
**mysql_free_result($result2);**
}
}
Thanks.
Quoting the documentation of mysql_free_result :
mysql_free_result() only needs to be
called if you are concerned about how
much memory is being used for queries
that return large result sets. All
associated result memory is
automatically freed at the end of the
script's execution.
So, if the documentation says it's generally not necessary to call that function, I would say it's not really necessary, nor good practice, to call it ;-)
And, just to say : I almost never call that function myself ; memory is freed at the end of the script, and each script should not eat too much memory.
An exception could be long-running batches that have to deal with large amounts of data, though...
Yes, it is good practice to use mysql_free_result($result). The quoted documentation in the accepted answer is inaccurate. That is what the documentation says, but that doesn't make any sense. Here is what it says:
mysql_free_result() only needs to be called if you are concerned about how much memory is being used for queries that return large result sets. All associated result memory is automatically freed at the end of the script's execution.
The first part of the first sentence is correct. It is true that you don't need to use it for reasons other than memory concerns. Memory concerns are the only reason to use it. However, the second part of the first sentence doesn't make any sense. The claim is that you would only be concerned about memory for queries that return large result sets. This is very misleading as there are other common scenarios where memory is a concern and calling mysql_free_result() is very good practice. Any time queries may be run an unknown number of times, more and more memory will be used up if you don't call mysql_free_result(). So if you run your query in a loop, or from a function or method, it is usually a good idea to call mysql_free_result(). You just have to be careful not to free the result until after it will not be used anymore. You can shield yourself from having to think about when and how to use it by making your own select() and ex() functions so you are not working directly with result sets. (None of the code here is exactly the way I would actually write it, it is more illustrative. You may want to put these in a class or special namespace, and throw a different Exception type, or take additional parameters like $class_name, etc.)
// call this for select queries that do not modify anything
function select($sql) {
$array= array();
$rs= query($sql);
while($o= mysql_fetch_object($rs))
$array[]= $o;
mysql_free_result($rs);
return $array;
}
// call this for queries that modify data
function ex($sql) {
query($sql);
return mysql_affected_rows();
}
function query($sql) {
$rs= mysql_query($sql);
if($rs === false) {
throw new Exception("MySQL query error - SQL: \"$sql\" - Error Number: "
.mysql_errno()." - Error Message: ".mysql_error());
}
return $rs;
}
Now if you only call select() and ex(), you are just dealing with normal object variables and only normal memory concerns instead of manual memory management. You still have to think about normal memory concerns like how much memory is in use by the array of objects. After the variable goes out of scope, or you manually set it to null, it become available for garbage collection so PHP takes care of that for you. You may still want to set it to null before it goes out of scope if your code does not use it anymore and there are operations following it that use up an unknown amount of memory such as loops and other function calls. I don't know how result sets and functions operating on them are implemented under the hood (and even if I did, this could change with different/future versions of PHP and MySQL), so there is the possibility that the select() function approximately doubles the amount of memory used just before mysql_free_result($rs) is called. However using select() still eliminates what us usually the primary concern of more and more memory being used during loops and various function calls. If you are concerned about this potential for double memory usage, and you are only working with one row at a time over a single iteration, you can make an each() function that will not double your memory usage, and will still shield you from thinking about mysql_free_result():
each($sql,$fun) {
$rs= query($sql);
while($o= mysql_fetch_object($rs))
$fun($o);
mysql_free_result($rs);
}
You can use it like this:
each("SELECT * FROM users", function($user) {
echo $user->username."<BR>";
});
Another advantage of using each() is that it does not return anything, so you don't have to think about whether or not to set the return value to null later.
The answer is of course YES in mysqli.
Take a look at PHP mysqli_free_result documentation:
You should always free your result with mysqli_free_result(), when your result object is not needed anymore.
I used to test it with memory_get_usage function:
echo '<br>before mysqli free result: '.memory_get_usage();
mysqli_free_result($query[1]);
echo '<br>after mysqli free result'.memory_get_usage();
And it is the result:
before mysqli free result:2110088
after mysqli free result:1958744
And here, we are talking about 151,344 bytes of memory in only 1000 rows of mysql table.
How about a million rows and how is it to think about large projects?
mysqli_free_result() is not only for large amount of data, it is also a good practice for small projects.
It depends on how large your queries are or how many queries you run.
PHP frees the memory at the end of the script(s) automatically, but not during the run. So if you have a large amount of data comming from a query, better free the result manually.
I would say: YES, it is good practice because you care about memory during the development or your scripts and that is what makes a good developer :-)
I recently ran across some code which the person did the first one. Would like some thoughts on if the top one is better or why a person would write it that way? Any positive reasons over the bottom way.
$result = mysql_query($query)or die("Obtaining location data failed!");
for ($i = mysql_num_rows($result) - 1; $i >=0; $i--)
{
if (!mysql_data_seek($result, $i))
{
echo "Cannot seek to row $i\n";
continue;
}
if(!($row = mysql_fetch_object($result)))
continue;
echo $row->locationname;
}
mysql_free_result($result);
vs
$result = mysql_query($query) or die("Obtaining location data failed!");
while($row = mysql_fetch_object($result)){
echo $row->locationname;
unset($row);
}
mysql_free_result($result);
It looks like the top code is iterating through the mysql result backwards, where the first one is going through it forwards.
The second code example looks cleaner, and there is probably a way to adjust the query to get the results in reverse order in the first place, instead of the somewhat convoluted way the top loop was performed.
Those two are not equivalent since only the first processes the result set in reverse order.
I'd do that with an ORDER BY x DESC clause if only to keep the code simple. When using mysql_query() the complete result set is transferred from the MySQL server to the php process before the function returns and mysql_data_seek() is only moving some pointer within the process' memory, so performace-wise it shouldn't matter much. But if you at some point decide to use an unbuffered query instead it might very well affect the performance.
Definitely the second one :
less code = less code to maintain =~ maybe less bugs !!
The top one has definite advantages when it comes to job security and 'lines of code' performance metrics. Apart from that there is no good reason to do what they did.