We noticed that a call to our API endpoint usually have multiple duplicated MySQL queries for SELECT statements (sometimes hundreds). So we decided to create a hashmap to store the
mysqli_result and check if a given query has already been done and return the result saved in our map.
My question is: do mysqli_result maintain a reference to our connection? Would this actually help us to avoid overloading our database with queries that have already been requested unnecessarily? If the mysqli_result is too big, could this overflow the memory for our fpm process?
What I want to know basically is what approach is better:
For query A, reuse mysqli_result object everytime I need it;
Redo query A everytime I need it.
mysqli_result() requires neither a connection to MySQL nor a mysqli object once the results are fetched from MySQL.
Caveat: The above only applies to the buffered queries. If you are using unbuffered resultsets then mysqli_result requires an open mysqli connection to fetch the data otherwise you will receive an error saying:
PHP Warning: mysqli_result::fetch_all(): Error while reading a row in ...
However, this error message can only happen when you close the connection when using unbuffered queries. e.g.
$res = $mysqli->query('SELECT id FROM Users LIMIT 1', MYSQLI_USE_RESULT);
$mysqli->close();
$res->fetch_all();
Even though mysqli_result requires a valid connection when creating an instance of it, it only needs the connection to fetch the data. The second parameter in the mysqli_result::__construct() is used to decide whether the resultset should be buffered in PHP or stored on MySQL server and fetched row by row. When you create an instance of mysqli_result you need to pass an instance of mysqli as the first parameter.
// Execute query on MySQL server
$res = $mysqli->real_query('SELECT id FROM Users LIMIT 1');
// Create the result object.
// The second argument can be MYSQLI_STORE_RESULT for buffered result or MYSQLI_USE_RESULT for unbuffered.
$res = new mysqli_result($mysqli, MYSQLI_STORE_RESULT);
// If MYSQLI_STORE_RESULT was used then you can close mysqli here.
unset($mysqli); // or $mysqli->close(); or both
$data = $res->fetch_all();
// If MYSQLI_USE_RESULT was used then you can't close mysqli yet.
// unset($mysqli);
$data = $res->fetch_all();
Buffering of SQL queries is a rather complex task and it would be better to avoid duplicate calls rather than implementing caching. Try to refactor your code so that it does not call the same SELECT query multiple times during the execution of your script.
There is also a pre-made solution for this, although not very popular. Mysqlnd query result cache plugin
And as always remember that:
premature optimization is the root of all evil
Related
When I execute a prepared statement to select rows in a db table, does pdo "fetch" the records and caches it?
i.e. If I perform a fetch after executing a "select" statement, does pdo perform multiple db calls for each record I want fetched or does it simply fetch each record from its cache? (assuming it has cache)
tnx.
When you ask does pdo perform multiple db calls for each record if you mean a database connection or query by saying calls, then no. the way it works pdo opens the connection, query once then if you use fetchAll() it gets all the values at once while fetch() will get one value at the the time.
The caching you are referring when using prepare() a and execute() for statements that will be issued multiple times with different parameter values optimizes the performance of your application by allowing the driver to negotiate client and/or server side caching of the query plan and meta information.
Unless you are fetching huge amount of records (into the memory) you should not be concern with the cost of using fetch.
Sooo.. to answer your question PDO will not cache fetch , it will be stored into memory.
I want to understand how many client sever calls are made for a typical mysqli query?
Step(1) $result = mysqli_query($link, $query);
Depending on the type of query, we use other mysqli function after this like
mysqli_fetch_fields, mysqli_affected_rows, mysqli_insert_id, mysqli_fetch_row
etc. then we close the result object.
Now, is all data retrieved and stored in php memory after step (1)?
Or mysqli_fetch_fields, mysqli_insert_id etc makes another call to mysql server?
Reason for asking: Trying to understand how mysqli calls work. But can not find such explanation anywhere for beginners like me.
PHP MySQLi API is built on MySQL C API. So it would be better if you have knowlegdes of it.
Basically, SELECT query could generate large ResultSet and this ResultSet is transfered from Server to Client when you call PHP's mysqli_store_result() (In C API, mysql_store_result()).
C API mysql_fetch_row() just returns a pointer to MYSQL_RES* (which is already stored in PHP right after mysql_store_result(). But 'mysqli_fetch_row()` would require some memories to make PHP's array.
mysqli_insert_id() (which is last_insert_id() of C API) just returns insert id of MYSQL connection data stucture which means there is no extra memory for insert id.
If you want to know how MySQLi works, I would recommand to learn MySQL C API and see PHP source codes.
mysqli_query runs the query on the server and returns false is the query failed, true is the query was successful but did not return anything (UPDATE query for example) or a mysqli_result otherwise. That mysqli_result is a class that extends Traversable interface, so yes, it's in memory. All other functions mysqli_fetch_fields, mysqli_affected_rows etc. are just methods in that class so those just read what's already in memory.
For more details, read this: php documentation
The documentation tells you everything you need to know about mysqli.
mysqli_query execute the query and returns a query object, this object has some methods and amongst them there are:
mysqli_fetch_fields:
Returns an array of objects representing the fields in a result set
mysqli_affected_rows:
Returns the number of rows affected by the last INSERT, UPDATE, REPLACE or DELETE query.
mysqli_insert_id:
Returns the auto generated id used in the last query
mysqli_fetch_row:
Get a result row as an enumerated array
Being all method of an object they don't execute sql requests, they simply access object values and gives you back different results depending on the method.
Strictly from MySQL's point of view (database performance, not PHP performance) what's the difference between a Mysqli fetch_assoc() loop vs. Mysqli fetch_all() when retrieving query results?
Let's say for $result = $qdb->query("SELECT name, id FROM cats");
In other words, does each additional fetch_assoc() or fetch_array(MYSQLI_NUM) iteration result in more MySQL communication or is the entire query result already pulled from MySQL at one time?
In other words, can Mysqli fetch_all() make life easier for MySQL?
To emphasize, I'm only concerned with what MySQL hears and responds with, if there's any difference. This is not a question about PHP performance, why one way is better than the other, etc. Also, this is not a PDO question http://php.net/manual/en/mysqli-result.fetch-all.php
From reading the code, mysqli_fetch_assoc() fetches one row.
Whereas mysqli_fetch_all() calls mysqlnd_fetch_all(), which use a loop to fetch one row at a time until all rows have been fetched, then it breaks out of the loop.
Here's the relevant function in mysqlnd, edited for length:
MYSQLND_METHOD(mysqlnd_res, fetch_all)(MYSQLND_RES * result, unsigned int flags, zval *return_value TSRMLS_DC ZEND_FILE_LINE_DC)
{
...
do {
MAKE_STD_ZVAL(row);
mysqlnd_fetch_into(result, flags, row, MYSQLND_MYSQLI);
if (Z_TYPE_P(row) != IS_ARRAY) {
zval_ptr_dtor(&row);
break;
}
add_index_zval(return_value, i++, row);
} while (1);
...
}
So the answer is: from the point of view of the MySQL server, there is no such thing as "fetch all." The PHP extensions either fetch one row, or else fetch one row at a time until all the rows in the result set have been fetched.
Strictly from MySQL's point of view (database performance, not PHP performance) neither Mysqli fetch_assoc() nor Mysqli fetch_all() has any significance.
Strictly from general performance point of view, there is not a slightest difference. You can use anything that suits you more from application design, sensibility and readability point of view.
Ok, so if I set into a variable an sql_query like so:
$query = mysql_query("..");
and call multiple mysql_result's like so:
mysql_result($query, 0);
mysql_result($query, 1);
mysql_result($query, 2);
How many queries will the page call to the server?
Only once? or three times?
When you execute mysql_query, It executes the sql and it keeps the result in an internal result structure. Next time when you call mysql_result, it fetches from that internal result.
For buffered query the reusult will be copying from MySQL server to PHP as soon as mysql_query is executed. For un-buffered query it'll be copied lazily.
So in both case Query executes only one time. But for un-buffered query it'll be fetched from MySQL server. every time you call mysql_result or mysq_fetch_*.Buffered and Unbuffered queries
You have given the answer yourself. If you are calling "query" function once, then it will only
execute once no matter how many times you parse its return value
$query = mysql_query("..");
When you run the above code then query gets executed and the returned resource is in $query variable. Then you can fetch data from it multiple times but query wont run again.
Here will be one query with database that's means Server. mysql_result() should not be mixed with calls to other functions that deal with the result set.
For more information you can visit : http://php.net/manual/en/function.mysql-result.php
Only *mysql_query* is executing against the mysql server. Only one query
mysql_result only fetches data from a mysql resource, it doesn't matter if you have previously gotten it from a query in code or managed to get it from another source.
From PHP 5.5.0 onwards this function is deprecated; the new way to perform this action would be to create a mysqli object and use over the functions mysqli::query and mysqli::fetch_field
Well, the php code below successfull adds up all the rows in the url field.
$sql = "SELECT * FROM table WHERE url <> ''";
$result = mysql_query($sql,$con);
$sql_num = mysql_num_rows($result);
while($sql_row=mysql_fetch_array($result))
{
urls[] = $sql_row["url"];
}
The problem is that if the list of url are in millions, then it takes a lot of time (especially in localhost). So, I'd like to know anothe way of getting the sql query result directly into an array without using a loop. Is it possible?
The problem is not the loop, it's that you are transferring millions of pieces of data (possibly large) from your database into memory. Whichever way you're doing that, it'll take time. And somebody needs to loop somewhere anyway.
In short: reduce the amount of data you get from the database.
You should consider using mysqli for that purpose. The fetch_all() method would allow to do that.
http://php.net/manual/en/mysqli-result.fetch-all.php
UPDATE
As per comments, I tried both methods. I tried using mysql_fetch_array in a loop, and using mysqli::fetch_all() method, on a large table we have in production. mysqli::fetch_all() did use less memory and ran faster than the mysql_fetch_array loop.
The table has about 500000 rows. mysqli::fetch_all() finished loading the data in an array in 2.50 seconds, and didn't hit the 1G memory limit set in the script. mysql_fetch_array() failed from memory exhaustion after 3.45 seconds.
mysql is deprecated, and the functionality you want is found in mysqli and PDO. It's the perfect excuse to switch to the newer MySQL extensions. For both mysqli and PDO, the method is fetchAll (note that the mysqli::fetch_all requires the mysqlnd driver to run).
there is no option for it in mysql. though you can use pdo's fetchall()
http://php.net/pdostatement.fetchall