in what is efficient to execute multiple queries:
this with nextRowset() function to move over the queries
$stmt = $db->query("SELECT 1; SELECT 2;");
$info1 = $stmt->fetchAll();
$stmt->nextRowset();
$info2 = $stmt->fetchAll();
or multiple executions plan which is a lot easier to manage?
$info1 = $db->query("SELECT 1;")->fetchAll();
$info2 = $db->query("SELECT 2;")->fetchAll();
Performance of the code is likely to be similar.
The code at the bottom, to me, is more efficient for your software design because:
it is more readable
it can be changed with less chance of error since each of them addresses 1 query only
individual query and its interaction can be moved to a different function easily and can be tested individually
That's why I feel that overall efficiency (not just how fast data comes back from DB to PHP to the user, but also maintainability/refactoring of code) will be better with the code at the bottom.
"SQL injection" by a hacker is easier when you issue multiple statements at once. So, don't do it.
If you do need it regularly, write a Stored Procedure to perform all the steps via one CALL statement. That will return multiple "rowsets", so similar code will be needed.
Related
I have an SQL query which can return quite a lot results (something like 10k rows) but I cannot use the SQL LIMIT parameter, as I don't know the exact amount of needed rows (there's a special grouping done in PHP). So the plan was to stop fetching rows once I have enough.
Since PDO normally operates in buffered mode, which fetches the whole result set and passes it to PHP, I switched PDO to unbuffered mode with
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
Now I expected that executing the query should take about the same time no matter what LIMIT I pass. So basically
$result = $pdo->query($query);
$count = 0;
while ($row = $result->fetch()) {
++$count;
if ($count > 10) break;
}
should execute in about the same time for
$query = 'SELECT * FROM myTable';
and
$query = 'SELECT * FROM myTable LIMIT 10';
However the first one takes 8 seconds whereas the second one executes instantly. So it seems like the unbuffered query also waits until the whole result set is fetched - which shouldn't be the case according to the documentation.
Is there any way to get the query result instantly in PHP with PDO and stop the query once I have enough results?
Database applications like "Sequel Pro SQL" can do this (I can hit cancel after 1 second and get the results that were already queried until that time) so it can't be a general problem with MySQL servers.
I can workaround the problem by choosing a very high LIMIT which always has enough valid results after my grouping. But since performance is an issue, I'd like to query only as many entries as really needed. Please don't suggest anything that involves grouping in MySQL, the terrible performance of that is the reason we have to change the behaviour.
Now I expected that executing the query should take about the same time no matter what LIMIT I pass. So basically
This might not be completely true. While you won't get the overhead of receiving all your results, they are all queried (without a limit)! You do get the advantage of keeping most of the results serverside until you need them, but your server actually does perform the whole query first as far as I know. I'm not sure how complicated your query is, but this could be the issue?
Say for instance you have a very slow join (not indexed), but only want the first 10 by id, your query will get 10 based on the index, and then only do the join for those 10. This'll be quick
But if you don't actually limit, but ask for the result in batches, your complete join will have to be done (slow!) and then your resultsset is released in parts.
A quicker method might be to repeat your limited query untill you have your result. I know, this will increase overhead, but it might be way quicker. Only way to know is to test.
as response to your comment: this is from the manual
Unbuffered MySQL queries execute the query and then return a resource while the data is still waiting on the MySQL server for being fetched.
So it executes the query. The complete query. So as I tried to explain above, it will not be as quick as the same query with a LIMIT 10, as it doesn't perform a partial query! The fact that a different DB engine does this does not mean MySQL can...
Have you tried using prepare/execute instead of query, and putting a $stmt->closeCursor(); call after the break?
$stmt = $dbh->prepare($query);
$stmt->execute();
$count = 0;
while ($row = $stmt->fetch()) {
++$count;
if ($count > 10) break;
}
$stmt->closeCursor();
I'm dealing with big volumes of data. It's a huge table where I'm performing unions through a SQL statement, from my php, and sending over to my own localhost db. I've got the thing sorted out but I want to optimize this. It had to stay overnight merging around 83.000 rows.
$con = new PDO("pgsql:host, port,dbname, user, password");
$con->beginTransaction(); // cursors require a transaction.
$stmt = $con->prepare($query);
$stmt->execute();
$innerStatement = $con->prepare("FETCH 1 FROM cursor1");
while($innerStatement->execute() && $row = $innerStatement->fetch(PDO::FETCH_ASSOC)) {
insertDataToDB($row);
}
Question: will changing the line to "FETCH 1000 FROM cursor1" make it so that I'm fetching 1000 rows each time instead of one? Will that help performance?
I'm hoping this larger operation was a one time thing. But in future I will have to move smaller amounts of data... still the query is rather heavy since it relies in comparisons with timestamps, otherwise how would I know if my DB is updated or not?
Thank you.
I have a PHP page that is interacting with a MYSQL database using PDO. I have a function that updates numerous fields in the database as requested. Since I do not know how many fields will be updated in advance, it is tricky to write a single query. Which of these methods is preferable (or is there another better way I do not know about)?
Query Building
$query = "UPDATE users SET ";
foreach ($changes as $field => $new_value)
{
$valid_field = validate_field($field);
$query .= "${valid_field} = :${valid_field} ";
}
$query = "WHERE id = :id ;";
// Prepare statement, bind values, execute, check for errors, etc
From what I have heard, this is not preferable. I tend to agree; this looks kinda ugly.
Transaction
$pdo_object->beginTransaction();
foreach ($changes as $field => $new_value)
{
$valid_field = validate_field($field);
$query = "UPDATE users SET ${valid_field} = :${valid_field} WHERE id = :id";
// Prepare statement, bind values, execute, check for errors, etc
}
$pdo_object->commit();
This seems safer to me, but the way it is written it looks like it searches the table for the row with that ID many times rather than just once like the other query.
Is the Query Building method faster than the Transaction method? Should the Transaction method be used despite slower speed for safety/security reasons?
Actually, transactions (which, in MySQL, requires use of "InnoDB" tables ...) are quite efficient. The database engine will (probably ...) lock all of the storage pages that are covered by the query, and might delay actually writing the pages back to the store until after the transaction COMMITs.
My suggestion is simply: (a) don't be afraid of transactions when they seem to be called for, and (b), "just focus on simplicity and clarity." Write code that is obvious, easy to read, and easy to maintain. Then, presume that the SQL engine knows how to do its job. :-)
This is an optimisation question RE: 1st principles.. Imagine I am doing a big heavy lifting comparison.. 30k files vs 30k database entries.. is it most process efficient to do one big MySQL into an array then loop through physical files checking vs the array or is it better to loop through the files and then one at a time do one line MySQL calls..
Here is some pseudo code to help explain:
//is this faster?
foreach($recursiveFileList as $fullpath){
$Record = $db->queryrow("SELECT * FROM files WHERE fullpath='".$fullpath."'");
//do some $Record logic
}
//or is this faster
$BigList = array();
$db->query("SELECT * FROM files");
while($Record = $db->rows()){
$BigList[$Record['fullpath']] = $Record;
}
foreach($recursiveFileList as $fullpath){
if (isset($BigList[$fullpath])){
$Record = $BigList[$fullpath];
//do some $Record logic
}
}
Update: if you always know that your $recursiveFileList is 100% of the table, then doing one query per row would be needless overhead. In that case, just use SELECT * FROM files.
I wouldn't use either of the two styles you show.
The first style runs one separate SQL query for each individual fullpath. This causes some overhead of SQL parsing, optimization, etc. Keep in mind that MySQL does not have the capability of remembering the query optimization from one invocation of a similar query to the next; it analysis and performs query optimization every time. The overhead is relatively small, but it adds up.
The second style shows fetching all rows from the table, and sorting it out in the application layer. This has a lot of overhead, because typically your $recursiveFileList might match only 1% or 0.1% or an even smaller portion of the rows in the table. I have seen cases where transferring excessive amounts of data over the network literally exhausted a 1Gbps network switch, and this put a ceiling on the requests per second for the application.
Use query conditions and indexes wisely to let the RDBMS examine and return only the matching rows.
The two styles you show are not the only options. What I would suggest is to use a range query to match multiple file fullpath values in a single query.
$sql = "SELECT * FROM files WHERE fullpath IN ("
. array_fill(0, count($recursiveFileList), "?") . ")";
$stmt = $pdo->prepare($sql);
$stmt->execute($recursiveFileList);
while ($row = $stmt->fetch()) {
//do some $Record logic
}
Note I also use a prepared query with ? parameter placeholders, and then pass the array of fullpath values separately when I call execute(). PDO is nice for this, because you can just pass an array, and the array elements get matched up to the parameter placeholders.
This also solves the risk of SQL injection in this case.
An example of my scenario is a large setup page for an application, the method I use is for example:
//query 1
$stmt = $dbh->prepare("...");
$stmt->execute();
//query 2
$stmt = $dbh->prepare("...");
$stmt->execute();
Would this be an accepted method to write more queries? I have no clue how it's supposed to be done (or who does what, rather), I assume writing the second $stmt is the most acceptable way, as there is no need to create other variables, am I right?
I really wish to know how people do this sort of thing.. I don't want to release 'ugly' code if I have to.
Yes, that is perfectly acceptable way to execute queries. No need to create new $stmt objects.
Also, if you ever get the error Lost connection to MySQL server during query when performing multiple queries within a single page, always issue this with the query: This will tell the MySQL driver to use the buffered versions of the MySQL API.
PDO::setAttribute("PDO::MYSQL_ATTR_USE_BUFFERED_QUERY", true);
So that your query looks like:
$db->prepare('select * from tablename', array(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => true));
$db->execute();