I am trying to paginate the records fetched into pages of 10 (using PHP AJAX on MSSQL Server 2005). (I am passing SQL as PHP strings to MSSQL server)
I wish to display button for each page (for example if there are 45 records then I wish to display 5 buttons with each button fetching 10 records).
I have came across this code-
SELECT *
FROM ( SELECT ROW_NUMBER() OVER ( ORDER BY OrderDate ) AS RowNum, *
FROM Orders
WHERE OrderDate >= '1980-01-01'
) AS RowConstrainedResult
WHERE RowNum >= 1
AND RowNum <= 10
ORDER BY RowNum
But this is highly ineffective because firstly whole database is fetched then from it 10 rows is fetched every time.
Could anyone suggest some technique (that works with MSSQL server 2005) wihich fetches only 10 rows at a time and not the whole 100 records at a time.
The above method also includes an extra column (it requires a primary key). I wish to get rid of that column too!
Also, since I have to display the page numbers (pagination tabs), I wish to have a count of total number of records beforehand.
Thanks!
Learn to use LINQ. Very efficient.
Edit
Well then.
SELECT *
FROM etc
WHERE Row_Number() BETWEEN '".$startLimit."' AND '".$endLimit."'
Might work for you. Remember to escape the variables so no dangerous code can bass by.
Related
I have been working on try to create php web-based paging for our tables
which have over a million rows.
Based on what I have read, I have 3 options
retrieve all rows in resultset - not possiblefor me coz of the size
retrieve 1000 rows, store in temp table and create an iterator for
it and page through it - too many queries - too many inserts!!
run a query each time if someone opts page forward or backwards
Right now I am trying to get option 3 working.
I have the first page showing up as
"select * from accout order by acct fetch first 10 rows only"
Page next
"select * from account where acct>(last record) order by acct fetch
first 10 only"
page last record
"select * from account where acct=(select max(acct) from account)"
The problem is showing the previous page and i really would appreciate
help in this.
SELECT *
FROM (
SELECT
*,
ROW_NUMBER() OVER (ORDER BY acct) AS RowNum
FROM
account
) AS Data
WHERE
RowNum BETWEEN 100 AND 110;
First, you should get rid of the SELECT *. Select only the fields you need.
Place an index on acct, this will help the ROW_NUMBER() OVER (ORDER BY acct) construct.
Use a SELECT COUNT(*) FROM account to determine how many pages you will have.
Also read Fastest most/efficient way to do pagination with SQL searching DB2
The LIMIT..OFFSET solution is supported in DB2 10+.. For older versions, you have to enable the MySQL compatibility with:
$ db2set DB2_COMPATIBILITY_VECTOR=MYS
$ db2stop
$ db2start
in db2cmd in order to use that syntax.
SELECT * FROM foo LIMIT 10, 1;
try the limit and offset in mysql..
this mostly use in creating pagination
I'm trying to get 4 random results from a table that holds approx 7 million records. Additionally, I also want to get 4 random records from the same table that are filtered by category.
Now, as you would imagine doing random sorting on a table this large causes the queries to take a few seconds, which is not ideal.
One other method I thought of for the non-filtered result set would be to just get PHP to select some random numbers between 1 - 7,000,000 or so and then do an IN(...) with the query to only grab those rows - and yes, I know that this method has a caveat in that you may get less than 4 if a record with that id no longer exists.
However, the above method obviously will not work with the category filtering as PHP doesn't know which record numbers belong to which category and hence cannot select the record numbers to select from.
Are there any better ways I can do this? Only way I can think of would be to store the record id's for each category in another table and then select random results from that and then select only those record ID's from the main table in a secondary query; but I'm sure there is a better way!?
You could of course use the RAND() function on a query using a LIMIT and WHERE (for the category). That however as you pointed out, entails a scan of the database which takes time, especially in your case due to the volume of data.
Your other alternative, again as you pointed out, to store id/category_id in another table might prove a bit faster but again there has to be a LIMIT and WHERE on that table which will also contain the same amount of records as the master table.
A different approach (if applicable) would be to have a table per category and store in that the IDs. If your categories are fixed or do not change that often, then you should be able to use that approach. In that case you will effectively remove the WHERE from the clause and getting a RAND() with a LIMIT on each category table would be faster since each category table will contain a subset of records from your main table.
Some other alternatives would be to use a key/value pair database just for that operation. MongoDb or Google AppEngine can help with that and are really fast.
You could also go towards the approach of a Master/Slave in your MySQL. The slave replicates content in real time but when you need to perform the expensive query you query the slave instead of the master, thus passing the load to a different machine.
Finally you could go with Sphinx which is a lot easier to install and maintain. You can then treat each of those category queries as a document search and let Sphinx randomize the results. This way you offset this expensive operation to a different layer and let MySQL continue with other operations.
Just some issues to consider.
Working off your random number approach
Get the max id in the database.
Create a temp table to store your matches.
Loop n times doing the following
Generate a random number between 1 and maxId
Get the first record with a record Id greater than the random number and insert it into your temp table
Your temp table now contains your random results.
Or you could dynamically generate sql with a union to do the query in one step.
SELECT * FROM myTable WHERE ID > RAND() AND Category = zzz LIMIT 1
UNION
SELECT * FROM myTable WHERE ID > RAND() AND Category = zzz LIMIT 1
UNION
SELECT * FROM myTable WHERE ID > RAND() AND Category = zzz LIMIT 1
UNION
SELECT * FROM myTable WHERE ID > RAND() AND Category = zzz LIMIT 1
Note: my sql may not be valid, as I'm not a mySql guy, but the theory should be sound
First you need to get number of rows ... something like this
select count(1) from tbl where category = ?
then select a random number
$offset = rand(1,$rowsNum);
and select a row with offset
select * FROM tbl LIMIT $offset, 1
in this way you avoid missing ids. The only problem is you need to run second query several times. Union may help in this case.
For MySQl you can use
RAND()
SELECT column FROM table
ORDER BY RAND()
LIMIT 4
I have a mysql database. How do I select the most recent 10 records? Im not storing timestamps. But the most the recent records are the ones at the bottom rite? Also. How so I get the next ten , the next ten and so on on clicking a button. Kinda like a bunch of forum posts. The recent ones show up first.
I believe you have an auto increment column as a primary key you can use this column and to order by desc
select * from table order by id desc limit 10
otherwise you have a very poor database design
If you have an AUTO_INCREMENT column you can order by that in descending order then limit by 10.
But I suggest you store timestamps and order by that instead so you know you're sorting your records according to date, and not some other value that coincides with date of insertion.
In addition to what #BoltClock mentioned, prequerying the maximum ID might help the engine with what other records are retrieved... ie: if you have a million records, and most recent 10, I don't know if it will still try to query out the million, order them, and THEN dump it.. I would try something like
select STRAIGHT_JOIN
YT.*
from
( select max( IDColumn ) as MaxOnFile
from YourTable ) PreQuery,
YourTable YT
where
YT.IDColumn >= PreQuery.MaxOnFile -10
order by
YT.IDColumn DESC
limit 10
However, if for some reason, records are allowed to be deleted, you may opt to subtract a little farther back than the -10... but at least this way, the system won't even TRY to process all the other records...
I am building a php site using jquery and the DataTables plugin. My page is laid out just as it needs to be with pagination working but in dealing with large datasets, I have noticed the server is pulling ALL returned rows as opposed to the 10 row (can be more) limit stated within each 'page'.
Is it possible to limit the results of a query and yet keep say the ID numbers of those results in memory so when page 2 is hit (or the result number is changed) only new data is sought after?
Does it even make sense to do it this way?
I just dont want to query a DB with 2000 rows returned then have a 'front-end-plugin' make it look like the other results are hidden when they are truthfully on the page from the start.
The LIMIT clause in SQL has two parts -- the limit and the offset.
To get the first 10 rows:
SELECT ... LIMIT 0,10
To get the next 10 rows:
SELECT ... LIMIT 10,10
To get the next 10 rows:
SELECT ... LIMIT 20,10
As long as you ORDER the result set the same each time, you absolutely don't have to (and don't want to) first ask the database to send you all 2000 rows.
To display paging in conjunction with this, you still need to know how many total rows match your query. There are two ways to handle that --
1) Ask for a row count with a separate query
SELECT COUNT(*) FROM table WHERE ...
2) Use the SQL_CALC_FOUND_ROWS hint in your query, which will tell MySQL to calculate how many rows match the query before returning only the 10 you asked for. You then issue a SELECT FOUND_ROWS() query to get that result.
SELECT SQL_CALC_FOUND_ROWS column1, column2 ... LIMIT 0,10
2 is preferable since it does not add an extra query to each page load.
Ok here is the situation (using PHP/MySQL) you are getting results from a large mysql table,
lets say your mysql query returns 10,000 matching results and you have a paging script to show 20 results per page, your query might look like this
So page 1 query
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse=something else
LIMIT 0,20
So page 2 query
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse=something else
LIMIT 20,40
Now you are grabbing 20 results at a time but there are a total of 10,000 rows that match your search criteria,
How can you only return 3,000 of the 10,000 results and still do your paging of 20 per page with a LIMIT 20 in your query?
I thought this was impossible but myspace does it on there browse page somehow, I know they aren't using php/mysql but how can it be achieved?
UPDATE
I see some people have replied with a couple of methods, it seems none of these would actually improve the performance by limiting the number to 3000?
Program your PHP so that when it finds itself ready to issue a query that ends with LIMIT 3000, 20 or higher, it would just stop and don't issue the query.
Or I am missing something?
Update:
MySQL treats LIMIT clause nicely.
Unless you have SQL_CALC_FOUND_ROWS in your query, MySQL just stops parsing results, sorting etc. as soon as it finds enough records to satisfy your query.
When you have something like that:
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse='something else'
LIMIT 0, 20
, MySQL will fetch first 20 records that satisfy the criteria and stop.
Doesn't matter how many records match the criteria: 50 or 1,000,000, performance will be the same.
If you add an ORDER BY to your query and don't have an index, then MySQL will of course need to browse all the records to find out the first 20.
However, even in this case it will not sort all 10,000: it will have a "running window" of top 20 records and sort only within this window as soon as it finds a record with value large (or small) enough to get into the window.
This is much faster than sorting the whole myriad.
MySQL, however, is not good in pipelining recorsets. This means that this query:
SELECT column
FROM (
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse='something else'
LIMIT 3000
)
LIMIT 0, 20
is worse performance-wise than the first one.
MySQL will fetch 3,000 records, cache them in a temporary table (or in memory) and apply the outer LIMIT only after that.
Firstly, the LIMIT paramters are Offset and number of records, so the second parameter should always be 20 - you don't need to increment this.
Surely if you know the upper limit of rows you want to retrieve, you can just put this into your logic which runs the query, i.e. check that Offset + Limit <= 3000
As Sohnee said, or (depending on your requirements) you can get all the 3000 records by SQL and then use array_slice in php to get chunks of the array.
You could achieve this with a subquery...
SELECT name FROM (
SELECT name FROM tblname LIMIT 0, 3000
) `Results` LIMIT 20, 40
Or with a temporary table, whereby you select all 3000 rows into a temp table then page by the temporary row id, which will be sequential.
You can specify the limit as a function of the page number (20*p, 20*p+2) in your php code, and limit the value of the page number to 150.
Or you could get the 3000 records and them using jquery tabs, split the records on 20 per page.