I'm just wondering which method is the most effective if I'm literally just wanting to get the number of rows in a table.
$res = mysql_query("SELECT count(*) as `number` FROM `table1`");
$count = mysql_fetch_result($res,0,'number');
or
$res = mysql_query("SELECT `ID` FROM `table1`");
$count = mysql_num_rows($res);
Anyone done any decent testing on this?
mysql_query() transfers all result records from the MySQL into the php pcrocess before it returns (unlike mysql_unbufferd_query()). That alone would make the mysql_num_rows() version slower.
Furthermore for some engines (like MyISAM) MySQL can serve a Count(*) request from the index of the table without hitting the actual data. A SELECT * FROM foo on the other hand results in a full table scan and MySQL has to read every single dataset.
Test in database with more then 2300000 rows, type:InnoDB, size near 1 GiB, using xhprof
test1:
....SELECT COUNT(id) as cnt FROM $table_name....;
row= mysqli_fetch_assoc($res2);
echo $row['cnt'];
//result1:
1,144,106
1,230,576
1,173,449
1,163,163
1,218,992
test2:
....SELECT COUNT(*) as cnt FROM $table_name....;
row= mysqli_fetch_assoc($res2);
echo $row['cnt'];
//result2:
1,120,253
1,118,243
1,118,852
1,092,419
1,081,316
test3:
....SELECT * FROM $table_name....;
echo mysqli_num_rows($res2);
//result3:
7,212,476
6,530,615
7,014,546
7,169,629
7,295,878
test4:
....SELECT * FROM $table_name....;
echo mysqli_num_rows($res2);
//result4:
1,441,228
1,671,616
1,483,050
1,446,315
1,647,019
conclusion:
The fastest method is in the test2 :
....SELECT COUNT(*) as cnt FROM $table_name....;
row= mysqli_fetch_assoc($res2);
echo $row['cnt'];
Definitely the first. MySQL can usually do this by looking at an index rather than the whole table, and if you use MyISAM (the default), the row count for the table is stored in the table metadata and will be returned instantly.
Your second method will not only read the entire table into memory but also send it to the client through the network before the client counts the rows. Extremely wasteful!
I don't really think any testing is needed.
Doing the COUNT in the SQL query
1) Sends only one row of data back the
to client (instead of every row)
2) Lets SQL do the count
for you which is likely always going
to be faster than PHP.
I guess count(1) will be even faster:
$res = mysql_query("SELECT count(1) as `number` FROM `table1`");
$count = mysql_fetch_result($res,0,'number');
Although haven't tried the proposed methods, the first makes database fetch all the records and count them in the database, the second makes database fetch a separate field for all the records and count the number of results on the server.
As a rule of thumb the less data you fetch for a particular record the less time it will take therefore I'd vote for updated first method (fetching constant for every record and counting the number of constants fetched).
Using Count with index and inodb makes it too much slow, but when use it with mysqli_num_rows it returns without any delay. you can check mysqli_num_rows result at http://ssajalandhar.org/generalinstruction-0-1-0.html it wouldn't take fraction of second to load. For me mysqli works awesome.
Related
I am counting the entries in my SQL database:
$sql = "SELECT * FROM files WHERE id = ?";
$q = $pdo->prepare($sql);
$q->execute([$id]);
$rowCount =$q->rowCount();
The result of $rowCount is 500000.
But to output this single number takes 5 seconds! Is it possible to get this result faster?
Use the COUNT() function https://dev.mysql.com/doc/refman/8.0/en/counting-rows.html:
$sql = "SELECT COUNT(*) FROM files WHERE id = ?";
Also ensure that 'id' is an indexed column:
https://dev.mysql.com/doc/refman/8.0/en/mysql-indexes.html
Replace * with a field(use auto-increment id) - This will reduce the time a bit.
Index that field. - If you use indexed field the query performance will increase.
SELECT * ..., then counting in PHP, requires shoveling all columns of all rows back to PHP. That's a lot of effort for very little gain.
SELECT COUNT(col) ... does the counting in by MySQL, but it must check for whether col is NULL. And it needs to get at the value of col for every row.
SELECT COUNT(*) ... counts the rows by whatever way is most efficient. This involves looking for the 'smallest' index (or the whole table, if no secondary indexes), and counting through it.
You must learn about INDEXes to get anywhere in databases! This is only one minor use for them.
Okay so I have a table which currently has 40000 rows and I need to SELECT them all. I use a index for id and url column and if I have to SELECT a value by id or url it's instant but if I have to SELECT * it's very slow. What I'm trying to do is searching my database and output the matches and I did this with a
while($arr = mysqli_fetch_array($query))
{ #code... echo $arr['whatever_i_need']."<br>"; }
$query = mysqli_query($link,"SELECT * FROM table");
In the future I will have hundreds of millions of rows in the database so I would like to return the search results fast in 1 sec or something. If you can give me solutions I would really appreciate! Thanks!
EDIT:
I don't want to display all of the data but I want to loop through it quickly to find all the matches
If you want speed then you definitely don't want the query to return every row from the table, and then "loop through" every row returned by the query to identify the ones you are interested in returning. That approach might give acceptable performance with small tables, but it definitely doesn't scale.
For performance, you want the database to locate just the rows you want to return, filter out the ones you don't want, and return just the subset.
And that comes down to writing an appropriate SQL query; executing an appropriate SELECT statement.
SELECT t.col1
, t.col2
, t.col3
FROM mytable t
WHERE t.col3 LIKE '%foo%'
AND t.col2 >= '2016-03-15'
AND t.col2 < '2016-06-15'
ORDER BY t.col2 DESC, t.col1 DESC
LIMIT 200
Performance is about making sure appropriate indexes are available and that the query execution is making effective use of the available indexes.
It has long been known that PDO does not support COUNT(*) and a query like below would fail as it doesn't return any affected rows,
$q = $dbc -> prepare("SELECT COUNT(*) FROM table WHERE id = ?");
$q -> execute(array($id));
echo $q -> rowCount();
Doing some research I found that you can also get the row count using other methods of count and not using count at all, for example the following query is supposed be the same as above but will return correct for PDO,
$q = $dbc -> prepare("SELECT 1 FROM table WHERE id = ?");
$q -> execute(array($id));
echo $q -> rowCount();
There are various sources on the internet claiming that;
"SELECT COUNT(*)
"SELECT COUNT(col)
"SELECT 1
Are all the same as each other (with a few differences) so how come using mysql which PDO cannot properly return a true count, does
"SELECT 1
work?
Methods of count discussion
Why is Select 1 faster than Select count(*)?
PDO does not support COUNT(*)
WTF? Of course PDO supports COUNT(*), you are using it the wrong way.
$q = $dbc->prepare("SELECT COUNT(id) as records FROM table WHERE id = ?");
$q->execute(array($id));
$records = (int) $q->fetch(PDO::FETCH_OBJ)->records;
If you are using a driver other than MySQL, you might have to test rowCount first, like this.
$records = (int) ($q->rowCount()) ? $q->fetch(PDO::FETCH_OBJ)->records : 0;
Oh. You are confusing everything.
PDO do not interfere with SQL queries. It support EVERYTHING supported by SQL.
When doing COUNT(*) you shouldn't use rowcount at all, as it just makes no sense. You have to retreive the query result instead.
Dunno what "various sources" you are talking about but COUNT(*) and COUNT(col) (and even COUNT(1)) are the same and the only proper way to get count of records when you need no records themselves.
COUNT is an aggregate function, it counts rows for you. So, it returns the result already, no more counting required. Ad it returns just a scalar value in the single row. Thus, using rowcount on this single row makes no sense
SELECT 1 is not the same as above, as it selects just literal 1 for the every row found in the table. So, it will return a thousand 1s if there is a thousands rows in your database. So, rowcount will give you the result but it is going to be an extreme waste of the server resources.
there is a simple rule to follow:
Always request the only data you need.
If you need the count of rows - request count of rows. Not a thousand of 1s to count them later.
Sounds sensible?
Best way I think to test if a line exist in your database is to perform.
SELECT 1 FROM table WHERE condition LIMIT 1
If it find a row it will stop and tell you there is a line. If it don't and you have an index on your where clause column it will also goes very fast to see there are none available.
I'm currently displaying a random row from all the entries and that works fine.
SELECT * FROM $db_table where live = 1 order by rand() limit 1
now, i'd like to limit it to the last 100 entries in the db.
every row in the db has an ID and a timestamp.
it's a small database, so overhead-minimization is not a priority.
thanks!
EDIT:
Still can't get it running.. I get a mysql_fetch_array error:
"Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result resource
Here's all of my code:
<?php $sql = "SELECT * FROM
(SELECT * FROM $db_table ORDER BY $datetime DESC LIMIT 100)
ORDER BY rand() LIMIT 1";
$query = mysql_query($sql);
while($row = mysql_fetch_array($query)) {
echo "".$row['familyname']."";
} ?>
Thanks again!
This is what I came up with off the top of my head. I've tested it and it works in SQLite, so you shouldn't have much trouble with MySQL. The only change was that SQLite's random function is random() not rand():
SELECT * FROM
(SELECT * FROM $db_table ORDER BY $timestamp DESC LIMIT 100)
ORDER BY rand() LIMIT 1
This page has a pretty detailed writeup on how to optimize an ORDER BY RAND()-type query. It's actually too involved for me to explain adequately on SO (also, I don't fully understand some of the SQL commands used, though the general concept makes sense), but the final optimized query makes use of several optimizations:
First, ORDER BY RAND(), which uses a filesort algorithm on the entire table, is dropped. Instead, a query is constructed to simply generate a single random id.
At this stage, an index scan is being used, which is even less efficient than a filesort in many cases, so this is optimized away with a subquery.
The WHERE clause is replaced with a JOIN to reduce the number of rows fetched by the outer SELECT, and the number of times the subquery is executed, to just 1.
In order to account for holes in the ids (from deletions) and to ensure an equal distribution, a mapping table is created to map row numbers to ids.
Triggers are used to automatically update & maintain the mapping table.
Lastly, stored procedures are created to allow multiple rows to be selected at once. (Here, ORDER BY is reintroduced, but used only on the result rows.)
Here are the performance figures:
Q1. ORDER BY RAND()
Q2. RAND() * MAX(ID)
Q3. RAND() * MAX(ID) + ORDER BY ID
100 1.000 10.000 100.000 1.000.000
Q1 0:00.718s 0:02.092s 0:18.684s 2:59.081s 58:20.000s
Q2 0:00.519s 0:00.607s 0:00.614s 0:00.628s 0:00.637s
Q3 0:00.570s 0:00.607s 0:00.614s 0:00.628s 0:00.637s
Whats the most efficient way of selecting total number of records from a large table? Currently, Im simply doing
$result = mysql_query("SELECT id FROM table");
$total = mysql_num_rows($result)
I was told this was not very efficient or fast, if you have a lot of records in the table.
You were told correctly. mysql can do this count for you which is much more efficient.
$result = mysql_query( "select count(id) as num_rows from table" );
$row = mysql_fetch_object( $result );
$total = $row->num_rows;
You should use SQL's built in COUNT function:
$result = mysql_query("SELECT COUNT(id) FROM table");
MyISAM tables already store the row count
SELECT COUNT(*) FROM table
on a MyISAM table simply reads that value. It doesn't scan the table or the index(es). So, it's just as fast or faster than reading the value from a different table.
According to the MySQL documentation this is most efficient if you're using a MyISAM table (which is the most usual type of tables used):
$result = mysql_query("SELECT COUNT(*) FROM table");
Otherwise you should do as Wayne stated and be sure that the counted column is indexed.
Can I just add, that the most "efficient" way of getting the total number of records, particularly in a large table, is to save the total amount as a number in another table.
That way, you don't have to query the entire table everytime you want to get the total.
You will however, have to set up some code or Triggers in the database to increase or decrease that number when a row is added/deleted.
So its not the easiest way, but if your website grows, you should definitely consider doing that.
Even though I agree to use the built-in functions, I don't really see any performance difference between mysql_num_rows and count(id). For 25000 results, same performance (can say exact.) Just for the record.
What about something like this:
$result = mysql_query("SELECT COUNT(id) AS total_things from table");
$row = mysql_fetch_array($result,MYSQL_ASSOC);
$num_results = $row["total_things"];
I had a large table (>50 million rows) and it took a long time to count the primary key, so I use the following:
SELECT TABLE_NAME, TABLE_ROWS
FROM information_schema.tables
WHERE TABLE_SCHEMA = "database";
Replace database with the name of your schema.
Just wanted to note that SHOW TABLE STATUS returns a Rows column, though I can't speak to its efficiency. Some light Googling turns up reports of slowness in MySQL 4 over two years ago. Might make for interesting time trials.
Also note the InnoDB caveat regarding inaccurate counts.
Use aggregate function. Try the below SQL Command
$num= mysql_query("SELECT COUNT(id) FROM $table");
mysqli_query() is deprecated. Better use this:
$result = $dbh->query("SELECT id FROM {table_name}");
$total = $result->num_rows;
Using PDO:
$result = $dbh->query("SELECT id FROM {table_name}");
$total = $result->rowCount();
(where '$dbh' = handle of the db connected to)