php pagination slow on large or last pages - php

Question

I have over 6000 results with come to about more than 235 pages using pagination. When I click first page, it loads really fast ~ 300ms up until around 40th page. after that it really goes down hill with about 30 ~ 40+ seconds of page load time. I am using indexed database. I tried to us mysql catch query, but did not like it. Can someone help me out.
php:
$sql = mysql_query("SELECT * FROM data WHERE (car = '$cars') AND (color = '$color' AND price BETWEEN '".$min."' AND '".$max."'
ORDER BY price LIMIT {$startpoint} , {$limit}");
Index:
data 0 PRIMARY 1 id A 106199 NULL NULL BTREE
data 1 car_index 1 car A 1799 NULL NULL BTREE
data 1 car_index 2 color A 2870 NULL NULL BTREE
data 1 car_index 3 price A 6247 NULL NULL BTREE
data 1 car_index 4 location A 106199 NULL NULL BTREE

Answer 1

This is a common issue with MySQL (and other database systems). Using LIMIT + OFFSET (which is what you are using implicitely with LIMIT x, y) works great at first but slows down exponentially as the number of fetched rows grows.
Adding an index is definitely a good first step, as you should always query data based on an index, to avoid full table scans.
Only having an index on price won't be enough as you have other WHERE attributes. Basically, this is what MySQL is doing:
Assuming that $limit = 25 and $startPoint = 0, MySQL will start reading the table from the beginning and stop after it finds 25 matching rows and will return them. Let's assume that it read 500 rows for this first iteration. Next iteration and because it does not have an index on car + color + price, it does not know how to jump directly to the 25th matching row (the 500th row in the table), so it will start reading from the beginning again, skip the first 25 matching rows and return the 25 next matching rows. Let's assume that this iteration also required 500 extra rows to be read.
Now you see what's going wrong. For every iteration, MySQL will have to read the all the rows from the beginning, exponentially increasing the time it takes to return row.
In my example, to fetch 100 (25 * 4 iterations) rows, MySQL will have to read 500 + 1000 + 1500 + 2000 = 5000 rows while you could expect it to only read 500 * 4 = 2,000 rows. To fetch 1000 (25 * 40 iterations) rows, MySQL will have to read 500 + 1000 + 1500 + ... 20000 = 410,000 rows!! That's way more than the 500 * 40 = 20,000 rows you could expect.
To optimize your query, first only select the data you need (no SELECT *). Then the trick is to remember the last fetched id.
$lastFetchedId = 0;
do {
$sql = mysql_query("SELECT * FROM data WHERE id > $lastFetchedId AND (car = '$cars' AND color = '$color' AND price BETWEEN '".$min."' AND '".$max."')
ORDER BY price LIMIT {$limit}");
$hasFoundRows = false;
while ($row = mysql_fetch_assoc($sql)) {
$hasFoundRows = true;
$lastFetchedId = $row['id'];
// do something with the row
}
} while ($hasFoundRows === false);
Having MySQL taking care of the ordering works well only if you have an index on all the columns you are using in the WHERE clause. Think about it this way: if the data is not sorted, how would MySQL know which rows will match and where the matching rows are. To be able to sort the results and only return a subset, MySQL needs to build a sorted list of ALL the rows that actually match. This means going through the entire table to first get all the matching rows, then sort them and finally return only a few of them.
Hope that helps you understand better what you can do better here :)

Answer 2

It would be a good idea to post here the table structure to see what indexes you have.
Please add an index on the column price, it should improve the query performance.
Cheers