I was browsing around Stack Overflow attempting to find how to limit an SQL query with a while loop and I came across this code.
$count = 0;
while ($count < 4 && $info = mysql_fetch_assoc($result)) {
//stuff
$count++;
}
Q 1: What is the difference between this code and using the SQL LIMIT clause?
Q 2: For what reason would somebody want to use this code, rather than using LIMIT?
With this code, the MySQL server will send all the results to the client, but the client ignores everything after the 4th row. So the server has to do more work, and more bandwidth will be used between the client and server.
They might want to use mysql_num_rows() to find out how many total rows were selected, even though they only want to display the first 4. However, MySQL provides a way to do that with LIMIT -- you can put the SQL_CALC_FOUND_ROWS option in the SELECT clause, and then use SELECT FOUND_ROWS() to get the total number of rows. So there's no good reason, except they don't know about this feature.
Everyting #Barmar said is right on. Following with code like that will cause lots of problems as your result sets start to grow. Let a database do what its good at doing, let it supply the limit of results you want/need. Just think of what happens when you do a SELECT with no LIMIT clause in the command line client where there are thousands of rows...it just goes on and on.
One more thing, I wouldn't recommend using mysql_num_rows() as its a deprecated function. Might as well go along with mysqli or PDO.
Related
I have an SQL query which can return quite a lot results (something like 10k rows) but I cannot use the SQL LIMIT parameter, as I don't know the exact amount of needed rows (there's a special grouping done in PHP). So the plan was to stop fetching rows once I have enough.
Since PDO normally operates in buffered mode, which fetches the whole result set and passes it to PHP, I switched PDO to unbuffered mode with
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
Now I expected that executing the query should take about the same time no matter what LIMIT I pass. So basically
$result = $pdo->query($query);
$count = 0;
while ($row = $result->fetch()) {
++$count;
if ($count > 10) break;
}
should execute in about the same time for
$query = 'SELECT * FROM myTable';
and
$query = 'SELECT * FROM myTable LIMIT 10';
However the first one takes 8 seconds whereas the second one executes instantly. So it seems like the unbuffered query also waits until the whole result set is fetched - which shouldn't be the case according to the documentation.
Is there any way to get the query result instantly in PHP with PDO and stop the query once I have enough results?
Database applications like "Sequel Pro SQL" can do this (I can hit cancel after 1 second and get the results that were already queried until that time) so it can't be a general problem with MySQL servers.
I can workaround the problem by choosing a very high LIMIT which always has enough valid results after my grouping. But since performance is an issue, I'd like to query only as many entries as really needed. Please don't suggest anything that involves grouping in MySQL, the terrible performance of that is the reason we have to change the behaviour.
Now I expected that executing the query should take about the same time no matter what LIMIT I pass. So basically
This might not be completely true. While you won't get the overhead of receiving all your results, they are all queried (without a limit)! You do get the advantage of keeping most of the results serverside until you need them, but your server actually does perform the whole query first as far as I know. I'm not sure how complicated your query is, but this could be the issue?
Say for instance you have a very slow join (not indexed), but only want the first 10 by id, your query will get 10 based on the index, and then only do the join for those 10. This'll be quick
But if you don't actually limit, but ask for the result in batches, your complete join will have to be done (slow!) and then your resultsset is released in parts.
A quicker method might be to repeat your limited query untill you have your result. I know, this will increase overhead, but it might be way quicker. Only way to know is to test.
as response to your comment: this is from the manual
Unbuffered MySQL queries execute the query and then return a resource while the data is still waiting on the MySQL server for being fetched.
So it executes the query. The complete query. So as I tried to explain above, it will not be as quick as the same query with a LIMIT 10, as it doesn't perform a partial query! The fact that a different DB engine does this does not mean MySQL can...
Have you tried using prepare/execute instead of query, and putting a $stmt->closeCursor(); call after the break?
$stmt = $dbh->prepare($query);
$stmt->execute();
$count = 0;
while ($row = $stmt->fetch()) {
++$count;
if ($count > 10) break;
}
$stmt->closeCursor();
Is it possible to SELECT specific rows and DELETE the selected result in ONE request?
$res = $Connection->query("SELECT * FROM tasks");
if($res->num_rows > 0){
while($row = $res->fetch_assoc()){ ...
The problem is that I have limited Number of queries to my SQL database and I want to minimize it as mush as possible.
You can't do that using the regular mysql-api in PHP. Just execute two queries. The second one will be so fast that it won't matter. This is a typical example of micro optimization. Don't worry about it. (So timing doesn't matter much)
For the record, since you are worried about the number of queries, it can be done using mysqli and the mysqli_multi_query-function.
P.S. - I haven't tried this, but since this mysql_multi_query is there in the documentation, it might help... :)
Both SELECT and DELETE need one request individually. And there is no SQL can do this work in the official documents.
I'm using a PHP webservice where I have performed a simple SELECT query, and stored it
$result = run_query($get_query);
I now need to perform further querying on the data based on different parameters, which I know is possible via MySQL in the form:
SELECT *
FROM (SELECT *
FROM customers
WHERE CompanyName > 'g')
WHERE ContactName < 'g'
I do know that this performs two Select queries on the table. However, what I would like to know is if I can simply use my previously saved query in the FROM section of the second section, such as this, and if my belief that it helps performance by not querying the entire database again is true:
SELECT *
FROM ($result)
WHERE ContactName < 'g'
You can make a temp table to put the initial results and then use it to select the data and in the second query. This will work faster only if your 1-st query is slow.
PHP and SQL are different languages and very different platforms. They often don't even run in the same computer. Your PHP variables won't interact at all with the MySQL server. You use PHP to create a string that happens to contain SQL code but that's all. In the end, the only thing that counts is the SQL code you sent to the server—how you manage to generate it is irrelevant.
Additionally, you can't really say how MySQL will run a query unless you obtain an explain plan:
EXPLAIN EXTENDED
SELECT *
FROM (SELECT *
FROM customers
WHERE CompanyName > 'g')
WHERE ContactName < 'g'
... but I doubt it'll read the table twice for your query. Memory is much faster than disk.
Thanks for the responses, everyone. Turns out what I was looking for was a "query of query", which isn't supported directly by PHP but I found a function over here which provides the functionality: http://www.tom-muck.com/blog/index.cfm?newsid=37
That was found from this other SO question: Can php query the results from a previous query?
I still need to do comparisons to determine whether it improves speed.
If I understand your question correctly you want to know whether saving the "from" part of your SQL query in a php variable improves the performance of you querying your SQL server, then the answer is NO. Simply because the variable keeping the value is inserted into the query.
Whether performance is gained in PHP, the answer is most probable yes; but depends on the length of the variable value (and how often you repeat using the variable instead of building a new complete query) whether the performance will be notable.
Why not just get this data in a single query like this?
SELECT *
FROM customers
WHERE CompanyName > 'g'
AND ContactName < 'g'
The is one php file that governs rotating ads that is causing serious server performance issues and causing "too many connections" sql errors on the site. Here is the php script. Can anyone give me some insight into how to correct this as I am an novice at php.
<?
require("../../admin/lib/config.php");
// connect to database
mysql_pconnect(DB_HOST,DB_USER,DB_PASS);
mysql_select_db(DB_NAME);
$i = 1;
function grab()
{
$getBanner = mysql_query("SELECT * FROM sponsor WHERE active='Y' AND ID != 999 AND bannerRotation = '0' ORDER BY RAND() LIMIT 1");
$banner = mysql_fetch_array($getBanner);
if ($banner['ID'] == ''){
mysql_query("UPDATE sponsor SET bannerRotation = '0'");
}
if (file_exists(AD_PATH . $banner['ID'] . ".jpg")){
$hasAd = 1;
}
if (file_exists(BANNER_PATH . $banner['ID'] . ".jpg")){
return "$banner[ID],$hasAd";
} else {
return 0;
}
}
while ($i <= 3){
$banner = grab();
if ($banner != 0){
$banner = explode(",",$banner);
mysql_query("UPDATE sponsor SET bannerView = bannerView + 1 WHERE ID='$banner[0]'");
mysql_query("UPDATE sponsor SET bannerRotation = '1' WHERE ID = '$banner[0]'");
echo "banner$i=$banner[0]&hasAd$i=$banner[1]&";
$i++;
}
}
?>
I see not mysqli
The problem is that mysql_pconnect() opens a persistent connection to the database and is not closed at end of execution, and as you are not calling mysql_close() anywhere the connection never gets closed.
Its all in the manual: http://php.net/manual/en/function.mysql-pconnect.php
Well, the good news for your client is that the previous developer abandoned the project. He could only have done more damage if he had continued working on it.
This script is using ext/mysql, not ext/mysqli. It would be better to use mysqli or PDO_mysql, since ext/mysql is deprecated.
It's recommended to use the full PHP open tag syntax (<?php), not the short-tags syntax (<?). The reason is that not every PHP environment enables the use of short tags, and if you deploy code into such an environment, your code will be viewable by anyone browsing to the page.
This script does no error checking. You should always check for errors after attempting to connect to a database or submitting a query.
The method of using ORDER BY RAND() LIMIT 1 to choose a random row from a database is well known to be inefficient, and it cannot be optimized. As the table grows to have more than a trivial number of rows, this query is likely to be your bottleneck. See some of my past answers about optimizing ORDER BY RAND queries, or a great blog by Jan Kneschke on selecting random rows.
Even if you are stuck using ORDER BY RAND(), there's no need to call it three times to get three distinct random sponsors. Just use ORDER BY RAND() LIMIT 3. Then you don't need the complex and error-prone update against bannerRotation to ensure that you get sponsors that haven't been chosen before.
Using SELECT * fetches all the columns, even though they aren't needed for this function.
If a sponsor isn't eligible for random selection, i.e. if it has active!='Y' or if its ID=999, then I would move it to a different table. This will simplify your queries, and make the table of sponsors smaller and quicker to query.
The UPDATE in the grab() function has no WHERE clause, so it applies to all rows in the sponsor table. I don't believe this is intentional. I assume it should apply only to the single row WHERE ID=$banner['ID'].
This code has two consecutive UPDATE statements against the same row of the same table. Combine these into a single UPDATE statement that modifies two columns.
The grab() function appends values together separated by commas, and then explodes that string into an array as soon as it returns. As if the programmer doesn't know that a function can return an array.
Putting the $i++ inside a conditional block makes it possible for this code to run in an infinite loop. That means this script can run forever. Once a few dozen of these are running concurrently, you'll run out of connections.
This code uses no caching. Any script that serves ad banners must be quick, and doing multiple updates to the database is not going to be quick enough. You need to use some in-memory caching for reads and writes. A popular choice is memcached.
Why is this client coding their own ad-banner server so inexpertly? Just use Google DFP Small Business.
Yikes!
grab() is being called from within a loop, but is not parameterized. Nor does there seem to be any rationale for repeatedly calling it.
A 200% speedup is easily realizable.
Until recently I've been using mysql_real_escape_string() to fix most of my variables before making SQL queries to my database. A friend said that I should be using PDO's prepared statements instead, so after reading a bit about them I'm now switching over to them.
I've only encountered one problem so far in switching over, and that's counting the rows to returned by a SELECT statement. On occasion in my code, I'd run an SQL query and then count the number of rows returned from the SELECT statement. Depending on whether a result set returned, I would take different actions. Sometimes I do need to use the result set from it. MySQL let me go straight to mysql_fetch_assoc() after mysql_num_rows() with no problem. However, PDO doesn't seem to have anything like mysql_num_rows().
I've been reading some responses on SO that gave me a solution, to either use COUNT() in the SQL statement or to use the PHP function count() on the result set. COUNT() would work fine in the SQL statement if I didn't need the result set in some places, however, several people have mentioned that using count() on the result set is fairly inefficient.
So my question is, how should I be doing this if I need to count the number of rows selected (if any), then run a script with the result set? Is using count() on the result set the only way in this case, or is there a more efficient way to do things?
Below is a short example of something similar to my previous SQL code:
$query=mysql_query('SELECT ID FROM Table WHERE Name='Paul' LIMIT 1);
if(mysql_num_rows($query)>0)
{
print_r(mysql_fetch_assoc($query));
}
else
{
//Other code.
}
Thanks.
EDIT
I do know that you use fetchAll() on the statement before counting the result set (which gives me what I need), but I'm just trying to figure out the most efficient way to do things.
$stmt->rowCount();
http://php.net/manual/en/pdostatement.rowcount.php
the rows must be fetched(buffered into memory, or iterated) for it to work. It's not uncommon for your pdo driver to be configured to do this automatically.
You will have to use Count(). You can run two queries like
SELECT COUNT(ID) FROM Table WHERE Name='Paul'
one you have get the count, then run the query with select clause
SELECT ID FROM Table WHERE Name='Paul' LIMIT 1
Count() function is not inefficient at all if you are using it like COUNT(ID), because most probably id is primary key and have an index. MYSQL wont even have to access the table.