There are more than 10m records in a table. Assume I display 15 items per page, this query is the simplest one (offset = 1,500,000)
select x,y,z
from table_name
order by x
limit 15
offset 1500000
this query takes about 5 seconds. Experts in database suggest the following approach:
I changed the query to this
select x,y,z
from table_name
where id > 3649168
order by x
limit 15
where 3649168 is the id of last record in page 1,499,999 and it takes about 250ms which is a super fast approach.
To get 3649168, I set a fields last_record_id in each page i and in page i+1 I use this to fetch information. This approach works perfect for the next page but I don't know how to get the previous page.
Basically I have a rating website where users rate other people from different schools. Right now I'm trying to do the leaderboard script but I cannot properly get the rating to show. My query is only returning one result. ($helpMe) Essentially I'm declaring the average total since my db doesn't store that, it only stores total number and total votes and divides accordingly.
https://pastebin.com/fVf4B8En here is the main code
I suspect that the culprit is here but not sure how
$helpMe = mysql_query("SELECT `rating_number`, FORMAT( (`total_points` / `rating_number`), 1 ) AS `average_rating` FROM `view_rating` WHERE `status` = 1 ORDER BY `average_rating` DESC");
while ($ratingRow = mysql_fetch_assoc($helpMe)) {
I am trying to paginate the records fetched into pages of 10 (using PHP AJAX on MSSQL Server 2005). (I am passing SQL as PHP strings to MSSQL server)
I wish to display button for each page (for example if there are 45 records then I wish to display 5 buttons with each button fetching 10 records).
I have came across this code-
SELECT *
FROM ( SELECT ROW_NUMBER() OVER ( ORDER BY OrderDate ) AS RowNum, *
FROM Orders
WHERE OrderDate >= '1980-01-01'
) AS RowConstrainedResult
WHERE RowNum >= 1
AND RowNum <= 10
ORDER BY RowNum
But this is highly ineffective because firstly whole database is fetched then from it 10 rows is fetched every time.
Could anyone suggest some technique (that works with MSSQL server 2005) wihich fetches only 10 rows at a time and not the whole 100 records at a time.
The above method also includes an extra column (it requires a primary key). I wish to get rid of that column too!
Also, since I have to display the page numbers (pagination tabs), I wish to have a count of total number of records beforehand.
Thanks!
Learn to use LINQ. Very efficient.
Edit
Well then.
SELECT *
FROM etc
WHERE Row_Number() BETWEEN '".$startLimit."' AND '".$endLimit."'
Might work for you. Remember to escape the variables so no dangerous code can bass by.
I am building a php site using jquery and the DataTables plugin. My page is laid out just as it needs to be with pagination working but in dealing with large datasets, I have noticed the server is pulling ALL returned rows as opposed to the 10 row (can be more) limit stated within each 'page'.
Is it possible to limit the results of a query and yet keep say the ID numbers of those results in memory so when page 2 is hit (or the result number is changed) only new data is sought after?
Does it even make sense to do it this way?
I just dont want to query a DB with 2000 rows returned then have a 'front-end-plugin' make it look like the other results are hidden when they are truthfully on the page from the start.
The LIMIT clause in SQL has two parts -- the limit and the offset.
To get the first 10 rows:
SELECT ... LIMIT 0,10
To get the next 10 rows:
SELECT ... LIMIT 10,10
To get the next 10 rows:
SELECT ... LIMIT 20,10
As long as you ORDER the result set the same each time, you absolutely don't have to (and don't want to) first ask the database to send you all 2000 rows.
To display paging in conjunction with this, you still need to know how many total rows match your query. There are two ways to handle that --
1) Ask for a row count with a separate query
SELECT COUNT(*) FROM table WHERE ...
2) Use the SQL_CALC_FOUND_ROWS hint in your query, which will tell MySQL to calculate how many rows match the query before returning only the 10 you asked for. You then issue a SELECT FOUND_ROWS() query to get that result.
SELECT SQL_CALC_FOUND_ROWS column1, column2 ... LIMIT 0,10
2 is preferable since it does not add an extra query to each page load.
Ok here is the situation (using PHP/MySQL) you are getting results from a large mysql table,
lets say your mysql query returns 10,000 matching results and you have a paging script to show 20 results per page, your query might look like this
So page 1 query
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse=something else
LIMIT 0,20
So page 2 query
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse=something else
LIMIT 20,40
Now you are grabbing 20 results at a time but there are a total of 10,000 rows that match your search criteria,
How can you only return 3,000 of the 10,000 results and still do your paging of 20 per page with a LIMIT 20 in your query?
I thought this was impossible but myspace does it on there browse page somehow, I know they aren't using php/mysql but how can it be achieved?
UPDATE
I see some people have replied with a couple of methods, it seems none of these would actually improve the performance by limiting the number to 3000?
Program your PHP so that when it finds itself ready to issue a query that ends with LIMIT 3000, 20 or higher, it would just stop and don't issue the query.
Or I am missing something?
Update:
MySQL treats LIMIT clause nicely.
Unless you have SQL_CALC_FOUND_ROWS in your query, MySQL just stops parsing results, sorting etc. as soon as it finds enough records to satisfy your query.
When you have something like that:
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse='something else'
LIMIT 0, 20
, MySQL will fetch first 20 records that satisfy the criteria and stop.
Doesn't matter how many records match the criteria: 50 or 1,000,000, performance will be the same.
If you add an ORDER BY to your query and don't have an index, then MySQL will of course need to browse all the records to find out the first 20.
However, even in this case it will not sort all 10,000: it will have a "running window" of top 20 records and sort only within this window as soon as it finds a record with value large (or small) enough to get into the window.
This is much faster than sorting the whole myriad.
MySQL, however, is not good in pipelining recorsets. This means that this query:
SELECT column
FROM (
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse='something else'
LIMIT 3000
)
LIMIT 0, 20
is worse performance-wise than the first one.
MySQL will fetch 3,000 records, cache them in a temporary table (or in memory) and apply the outer LIMIT only after that.
Firstly, the LIMIT paramters are Offset and number of records, so the second parameter should always be 20 - you don't need to increment this.
Surely if you know the upper limit of rows you want to retrieve, you can just put this into your logic which runs the query, i.e. check that Offset + Limit <= 3000
As Sohnee said, or (depending on your requirements) you can get all the 3000 records by SQL and then use array_slice in php to get chunks of the array.
You could achieve this with a subquery...
SELECT name FROM (
SELECT name FROM tblname LIMIT 0, 3000
) `Results` LIMIT 20, 40
Or with a temporary table, whereby you select all 3000 rows into a temp table then page by the temporary row id, which will be sequential.
You can specify the limit as a function of the page number (20*p, 20*p+2) in your php code, and limit the value of the page number to 150.
Or you could get the 3000 records and them using jquery tabs, split the records on 20 per page.