There are more than 10m records in a table. Assume I display 15 items per page, this query is the simplest one (offset = 1,500,000)
select x,y,z
from table_name
order by x
limit 15
offset 1500000
this query takes about 5 seconds. Experts in database suggest the following approach:
I changed the query to this
select x,y,z
from table_name
where id > 3649168
order by x
limit 15
where 3649168 is the id of last record in page 1,499,999 and it takes about 250ms which is a super fast approach.
To get 3649168, I set a fields last_record_id in each page i and in page i+1 I use this to fetch information. This approach works perfect for the next page but I don't know how to get the previous page.
Related
I have a database with more than 600 rows but I can only retrieve/display 100 every hour. So I use
select * from table ORDER BY id DESC LIMIT 100
to retrieve the first 100. How do I write a script that will retrieve the data in batches of 100 every 1hr so that I can use it in a cron job?
Possible solution.
Add a field for to mark the record was already shown.
ALTER TABLE tablename
ADD COLUMN shown TINYINT NULL DEFAULT NULL;
NULL will mean that the record was not selected, 1 - that record is marked for selection, 0 - that record was already selected.
When you need to select up to 100 records you
2.1. Mark records to be shown
UPDATE tablename
SET shown = 1
WHERE shown = 1
OR shown IS NULL
ORDER BY shown = 1 DESC, id ASC
LIMIT 100;
shown = 1 condition in WHERE considered the fact that some records were marked but were not selected due to some error. shown = 1 DESC re-marks such records before non-marked.
If there is 100 or less records which were not selected all of them will be marked, else only 100 records with lower id (most ancient) will be marked.
2.2. Select marked records.
SELECT *
FROM tablename
WHERE shown = 1
ORDER BY id
LIMIT 100;
2.3. Mark selected records.
UPDATE tablename
SET shown = 0
WHERE shown = 1
ORDER BY id
LIMIT 100;
This is applicable when only one client selects the records.
If a lot of clients may work in parallel, and only one cliens must select a record, then use some cliens number (unique over all clients) for to mark a record for selection instead of 1.
Of course if there is only one client, and you guarantee that selection will not fail, you may simply store last shown ID somewhere (on the client side, or in some service table on the MySQL side) and simply select "next 100" starting from this stored ID:
SELECT *
FROM tablename
WHERE id > #stored_id
ORDER BY id
LIMIT 100;
and
SELECT MAX(id)
FROM tablename
WHERE id > #stored_id
ORDER BY id
LIMIT 100;
for to store instead of previous #stored_id.
Thank you #Akina and #Vivek_23 for your contributions. I was able to figure out an easier way to go about it.
Add a new field to table, eg shownstatus
Create a cronjob to display 100 (LIMIT 100) records with their shownstatus not marked as shown from table every hour and then update each record's shownstatus to shown NB. If I create a cronjob to run every hour for the whole day, I can get all records displayed and their shownstatus updated to shown by close of day.
Create a second cronjob to update all record's shownstatus to notshown
The downside to this is that, you can only display a total of 2,400 records a day. ie. 100 records every hour times 24hrs. So if your record grows to about 10,000. You will need to set your cronjob to run for atleast 5 days to display all records.
Still open to a better approach if there's any, but till then, I will have to just stick to this for now.
Let's say you made a cron that hits a URL something like
http://yourdomain.com/fetch-rows
or a script for instance, like
your_project_folder/fetch-rows.php
Let's say you have a DB table in place that looks something like this:
| id | offset | created_at |
|----|--------|---------------------|
| 1 | 100 | 2019-01-08 03:15:00 |
| 2 | 200 | 2019-01-08 04:15:00 |
Your script:
<?php
define('FETCH_LIMIT',100);
$conn = mysqli_connect(....); // connect to DB
$result = mysqli_query($conn,"select * from cron_hit_table where id = (select max(id) from cron_hit_table)")); // select the last record to get the latest offset
$offset = 0; // initial default offset
if(mysqli_num_rows($result) > 0){
$offset = intval(mysqli_fetch_assoc($result)['offset']);
}
// Now, hit your query with $offset included
$result = mysqli_query($conn,"select * from table ORDER BY id DESC LIMIT $offset,100");
while($row = mysqli_fetch_assoc($result)){
// your data processing
}
// insert new row to store next offset for next cron hit
$offset += FETCH_LIMIT; // increment current offset
mysqli_query($conn,"insert into cron_hit_table(offset) values($offset)"); // because ID would be auto increment and created_at would have default value as current_timestamp
mysqli_close($conn);
Whenever cron hits, you fetch last row from your hit table to get the offset. Hit the query with that offset and store the next offset for next hit in your table.
Update:
As pointed out by #Dharman in the comments, you can use PDO for more abstracted way of dealing with different types of database(but make sure you have appropriate driver for it, see checklist of drivers PDO supports to be sure) along with minor checks of query syntaxes.
I have a question, how do you display a million results in front-end HTML/PHP/MYSQL in a fastest way? I have 2 tables: Table 2 a million per month and Table 3 consists a thousand of data's per month. Now, how am I going to display all results in a fastest way of retrieving a data in mysql? Here's my sample query when retrieving a data:
SELECT `r`.`ID`, `r`.`CurrentMonth`, GROUP_CONCAT(DISTINCT `s`.`NameEmployee` SEPARATOR '; ') AS `Name`, `s`.`BA` as `Area`, `r`.`PresRdgDate` as `ReadingDate`, `r`.`PresRemarks`,
time_format(concat(MIN(`r`.`ReadingTime`),'00'),'%H:%i:%s') AS `StartReading`,
time_format(concat(MAX(`r`.`ReadingTime`),'00'),'%H:%i:%s') AS `EndReading`,
TIMEDIFF(time_format(concat(MAX(`r`.`ReadingTime`),'00'),'%H:%i:%s'), time_format(concat(MIN(`r`.`ReadingTime`),'00'),'%H:%i:%s')) AS `TotalRdgTime`,
COUNT(`r`.`AccountNum`) AS `TotalRdgPerMRU`
FROM `table2` as `r`
LEFT JOIN `table3` as `s` ON `s`.`ID` = `r`.`ID`
WHERE (`r`.`CurrentMonth` >= '01' AND `r`.`CurrentMonth` <= '02')
AND `r`.`ID` LIKE '%%'
GROUP BY `r`.`ID`;
I am using an ajax to send the request from a user and pass to php, I also using a pagination to separate the display results by 10, so it won't load all data's in one page. When I am generating a search , it gives me a result within 5 mins, and when I click the next page, another 5 mins of waiting. Please suggest what is the better/best way to retrieve a data in MYSql and display the results not more than 5 mins. Plus I need to add a sort per column in front-end, also it will cause a slow display of the data in front-end.
You said you are using pagination by displaying 10 results per page. But, you did not use any limit and offset value in your query.
Use "LIMIT 10 OFFSET 0" at the end of your query for 1st page, "LIMIT 10 OFFSET 10" for 2nd page, "LIMIT 10 OFFSET 20" for 3rd page and so on....
I'm thinking of implement a view history for my wordpress blog, where users can view their previously viewed articles as a list in their account page.
I would like to limit this to 24 unique page history per user at any point of time, meaning, if the number of articles exceeds 24, the oldest article row would be deleted, and the new article added to the table.
I'm using PHP and MySQL.
Here's my current thoughts on implementation:
Create a table with user_id and post_id columns
When user views an article, insert new row into the table
Select the rows with the current user_id, and if number of rows is more than 24,
Delete the oldest row
I'm not sure if this is the best method, since it's 3 additional database queries per user page view which is pretty heavy.
Is there a better way to do this?
The ideea is good. Improve it by updating the oldest row instead of deleting it and then adding a new one. ;)
Also make a single read query and a single write query.
make a query that is like this one
SELECT (SELECT COUNT(*) FROM recentArticles WHERE userID = 23)
AS NoOfArticles, articleID, timestamp
FROM recentArticles
WHERE userID = 23
ORDER BY timestamp ASC
LIMIT 1;
If NoOfArticles < 24 then execute an insert query, else execute an update query to articleID
You could always implement this using cookies to key on which pages have been visited and keeping a running list on the users PC. This will reduce the traffic to the site, put the processing scripts on the user-side, and could be easier to implement.
That being said, I agree with the other answer about updating the oldest entry if the table is full. This eliminates the need to delete and add a row. Key off a time-date stamp to sort the entries when you're displaying them and to figure out which page was the oldest (and needs to be updated if >= 24 pages.
You can merge two queries in one. In this way, You will save execution time of your script. So, basically, DELETE row if the post count is more than 24. You can modify this query according to your exact need. But yes, you can think on this way.
DELETE FROM `table_name`
WHERE id=(SELECT (CASE WHEN COUNT(id)>24 then id END)
AS last_id
FROM `table_name`
WHERE user_id='XX'
ORDER BY id DESC LIMIT 1);
I am building a php site using jquery and the DataTables plugin. My page is laid out just as it needs to be with pagination working but in dealing with large datasets, I have noticed the server is pulling ALL returned rows as opposed to the 10 row (can be more) limit stated within each 'page'.
Is it possible to limit the results of a query and yet keep say the ID numbers of those results in memory so when page 2 is hit (or the result number is changed) only new data is sought after?
Does it even make sense to do it this way?
I just dont want to query a DB with 2000 rows returned then have a 'front-end-plugin' make it look like the other results are hidden when they are truthfully on the page from the start.
The LIMIT clause in SQL has two parts -- the limit and the offset.
To get the first 10 rows:
SELECT ... LIMIT 0,10
To get the next 10 rows:
SELECT ... LIMIT 10,10
To get the next 10 rows:
SELECT ... LIMIT 20,10
As long as you ORDER the result set the same each time, you absolutely don't have to (and don't want to) first ask the database to send you all 2000 rows.
To display paging in conjunction with this, you still need to know how many total rows match your query. There are two ways to handle that --
1) Ask for a row count with a separate query
SELECT COUNT(*) FROM table WHERE ...
2) Use the SQL_CALC_FOUND_ROWS hint in your query, which will tell MySQL to calculate how many rows match the query before returning only the 10 you asked for. You then issue a SELECT FOUND_ROWS() query to get that result.
SELECT SQL_CALC_FOUND_ROWS column1, column2 ... LIMIT 0,10
2 is preferable since it does not add an extra query to each page load.
Ok here is the situation (using PHP/MySQL) you are getting results from a large mysql table,
lets say your mysql query returns 10,000 matching results and you have a paging script to show 20 results per page, your query might look like this
So page 1 query
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse=something else
LIMIT 0,20
So page 2 query
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse=something else
LIMIT 20,40
Now you are grabbing 20 results at a time but there are a total of 10,000 rows that match your search criteria,
How can you only return 3,000 of the 10,000 results and still do your paging of 20 per page with a LIMIT 20 in your query?
I thought this was impossible but myspace does it on there browse page somehow, I know they aren't using php/mysql but how can it be achieved?
UPDATE
I see some people have replied with a couple of methods, it seems none of these would actually improve the performance by limiting the number to 3000?
Program your PHP so that when it finds itself ready to issue a query that ends with LIMIT 3000, 20 or higher, it would just stop and don't issue the query.
Or I am missing something?
Update:
MySQL treats LIMIT clause nicely.
Unless you have SQL_CALC_FOUND_ROWS in your query, MySQL just stops parsing results, sorting etc. as soon as it finds enough records to satisfy your query.
When you have something like that:
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse='something else'
LIMIT 0, 20
, MySQL will fetch first 20 records that satisfy the criteria and stop.
Doesn't matter how many records match the criteria: 50 or 1,000,000, performance will be the same.
If you add an ORDER BY to your query and don't have an index, then MySQL will of course need to browse all the records to find out the first 20.
However, even in this case it will not sort all 10,000: it will have a "running window" of top 20 records and sort only within this window as soon as it finds a record with value large (or small) enough to get into the window.
This is much faster than sorting the whole myriad.
MySQL, however, is not good in pipelining recorsets. This means that this query:
SELECT column
FROM (
SELECT column
FROM table_name
WHERE userId=1
AND somethingelse='something else'
LIMIT 3000
)
LIMIT 0, 20
is worse performance-wise than the first one.
MySQL will fetch 3,000 records, cache them in a temporary table (or in memory) and apply the outer LIMIT only after that.
Firstly, the LIMIT paramters are Offset and number of records, so the second parameter should always be 20 - you don't need to increment this.
Surely if you know the upper limit of rows you want to retrieve, you can just put this into your logic which runs the query, i.e. check that Offset + Limit <= 3000
As Sohnee said, or (depending on your requirements) you can get all the 3000 records by SQL and then use array_slice in php to get chunks of the array.
You could achieve this with a subquery...
SELECT name FROM (
SELECT name FROM tblname LIMIT 0, 3000
) `Results` LIMIT 20, 40
Or with a temporary table, whereby you select all 3000 rows into a temp table then page by the temporary row id, which will be sequential.
You can specify the limit as a function of the page number (20*p, 20*p+2) in your php code, and limit the value of the page number to 150.
Or you could get the 3000 records and them using jquery tabs, split the records on 20 per page.