I have this MariaDB table:
id, name
The id column has these attributes: Primary, auto_increment, unique.
The table has 40,000 rows.
I'm using this PHP & MariaDB to load rows from this table.
This is the PHP code:
$get_rows = $conn->prepare("SELECT * FROM my_table where id> 0 ORDER BY id ASC LIMIT 30 OFFSET ?");
$get_rows->bind_param('i', $offset);
//etc.
The query returned everything correctly at the first time, but in the next query (made through AJAX), I received the next 30 rows with a gap of one row between the current result and the next one. And this goes on and on.
In the table, the row #1 had been deleted. So, I restored it, and now the query works. However, I will definitely have to delete more rows in the future. (I don't have the option of soft-deleting).
Is there any way I can keep deleting rows, and have these queries return correct results (without skipping any row)?
EDIT
Here's an example of the range of the ids in the first 2 queries:
Query 1:
247--276
Query 2:
278--307
(277 is missing)
NB I asked ChatGPT, but it couldn't help. :')
LIMIT and OFFSET query rows by position, not by value. So if you deleted a row in the first "page," then the position of all subsequent rows moves down by one.
One solution to ensure you don't miss a row is to define pages by the greatest id value on the preceding page, instead of by the offset.
$get_rows = $conn->prepare("
SELECT * FROM my_table where id> ?
ORDER BY id ASC LIMIT 30");
$get_rows->bind_param('i', $lastId);
This only works if your previous query viewed the preceding page, so you can save the value of the last id in that page.
Related
I have a query that fetches a set of records as shown:
select id
from wallactions
where status=1 and created > DATE_ADD(NOW(),INTERVAL -1 DAY)
order by id desc LIMIT $priorRecordsCnt
The LIMIT variable $priorRecordsCnt changes often and can grow very large. It's necessary because i need to know the last id in that set of records based on the $priorRecordsCnt count value.
The problem is, I need to only access the last id in this set, as shown:
$last=array_values(array_slice($result,-1))[0]['id'];
I feel like this is pretty expensive to just get the id of the last record in the set.
Is there a way to optimize this query so it uses the count variable $priorRecordsCnt but i don't need to fetch all the records just to obtain the last value?
The limit clause takes two arguments - an optional offset and the row count. Instead of using $priorRecordsCnt as the record count, you should use it as the offset, and limit the record count to 1:
select id
from wallactions
where status=1 and created > DATE_ADD(NOW(),INTERVAL -1 DAY)
order by id desc LIMIT $priorRecordsCnt, 1
-- Here ---------------------------------^
I have a database with more than 600 rows but I can only retrieve/display 100 every hour. So I use
select * from table ORDER BY id DESC LIMIT 100
to retrieve the first 100. How do I write a script that will retrieve the data in batches of 100 every 1hr so that I can use it in a cron job?
Possible solution.
Add a field for to mark the record was already shown.
ALTER TABLE tablename
ADD COLUMN shown TINYINT NULL DEFAULT NULL;
NULL will mean that the record was not selected, 1 - that record is marked for selection, 0 - that record was already selected.
When you need to select up to 100 records you
2.1. Mark records to be shown
UPDATE tablename
SET shown = 1
WHERE shown = 1
OR shown IS NULL
ORDER BY shown = 1 DESC, id ASC
LIMIT 100;
shown = 1 condition in WHERE considered the fact that some records were marked but were not selected due to some error. shown = 1 DESC re-marks such records before non-marked.
If there is 100 or less records which were not selected all of them will be marked, else only 100 records with lower id (most ancient) will be marked.
2.2. Select marked records.
SELECT *
FROM tablename
WHERE shown = 1
ORDER BY id
LIMIT 100;
2.3. Mark selected records.
UPDATE tablename
SET shown = 0
WHERE shown = 1
ORDER BY id
LIMIT 100;
This is applicable when only one client selects the records.
If a lot of clients may work in parallel, and only one cliens must select a record, then use some cliens number (unique over all clients) for to mark a record for selection instead of 1.
Of course if there is only one client, and you guarantee that selection will not fail, you may simply store last shown ID somewhere (on the client side, or in some service table on the MySQL side) and simply select "next 100" starting from this stored ID:
SELECT *
FROM tablename
WHERE id > #stored_id
ORDER BY id
LIMIT 100;
and
SELECT MAX(id)
FROM tablename
WHERE id > #stored_id
ORDER BY id
LIMIT 100;
for to store instead of previous #stored_id.
Thank you #Akina and #Vivek_23 for your contributions. I was able to figure out an easier way to go about it.
Add a new field to table, eg shownstatus
Create a cronjob to display 100 (LIMIT 100) records with their shownstatus not marked as shown from table every hour and then update each record's shownstatus to shown NB. If I create a cronjob to run every hour for the whole day, I can get all records displayed and their shownstatus updated to shown by close of day.
Create a second cronjob to update all record's shownstatus to notshown
The downside to this is that, you can only display a total of 2,400 records a day. ie. 100 records every hour times 24hrs. So if your record grows to about 10,000. You will need to set your cronjob to run for atleast 5 days to display all records.
Still open to a better approach if there's any, but till then, I will have to just stick to this for now.
Let's say you made a cron that hits a URL something like
http://yourdomain.com/fetch-rows
or a script for instance, like
your_project_folder/fetch-rows.php
Let's say you have a DB table in place that looks something like this:
| id | offset | created_at |
|----|--------|---------------------|
| 1 | 100 | 2019-01-08 03:15:00 |
| 2 | 200 | 2019-01-08 04:15:00 |
Your script:
<?php
define('FETCH_LIMIT',100);
$conn = mysqli_connect(....); // connect to DB
$result = mysqli_query($conn,"select * from cron_hit_table where id = (select max(id) from cron_hit_table)")); // select the last record to get the latest offset
$offset = 0; // initial default offset
if(mysqli_num_rows($result) > 0){
$offset = intval(mysqli_fetch_assoc($result)['offset']);
}
// Now, hit your query with $offset included
$result = mysqli_query($conn,"select * from table ORDER BY id DESC LIMIT $offset,100");
while($row = mysqli_fetch_assoc($result)){
// your data processing
}
// insert new row to store next offset for next cron hit
$offset += FETCH_LIMIT; // increment current offset
mysqli_query($conn,"insert into cron_hit_table(offset) values($offset)"); // because ID would be auto increment and created_at would have default value as current_timestamp
mysqli_close($conn);
Whenever cron hits, you fetch last row from your hit table to get the offset. Hit the query with that offset and store the next offset for next hit in your table.
Update:
As pointed out by #Dharman in the comments, you can use PDO for more abstracted way of dealing with different types of database(but make sure you have appropriate driver for it, see checklist of drivers PDO supports to be sure) along with minor checks of query syntaxes.
I want to retrieve the last 100 values, I've used this query:
SELECT * FROM values WHERE ID BETWEEN max(ID)-100 and max(ID);
but I receive this message:
ERROR 1111 (HY000): Invalid use of group function
Order by the ID in descending order and take only the first 100 records of the result
SELECT * FROM values
order by id desc
limit 100
This is the more reliable version since there can be gaps in the ID sequence which would make your query inaccurate (besides of it being wrong syntactically).
Your question is not very clear
What is last 100 values? Last 100 ids inserted? Or last 100 rows updated?
Assuming that you are looking for the last 100 roes inserted, you approach has issues. First know that the IDs are not sequentially committed to DB.
For, example ID of a row can be 5 at sometime and ID of a row inserted later can be 4. How this happens is beyond the scope, but just know that it is possible.
Coming to solution
Just do
SELECT TOP 100 * from VALUES ORDER BY ID DESC
I'm trying to make an image database which does not keep a consistent record of ID's. For example it might go 1,2,6,7,12, but as you can see that is only 5 rows.
Inside the table I have fileid and filename. I created a PHP script to show me the image when I give the fileid. But if I give it the ID 5 which does not exist I get an error. That's fine as I want an error for that, but not for users who will browse through these images using forward and back buttons. The forward and back buttons would need to retrieve the true fileid which comes after the given ID. Hopefully that makes sense.
This is how I imagine the code to look like:
SELECT offset( WHERE fileid=4 )
That would give me the offset of the row where fileid is equal to 4. I think this is easy enough to understand. The reasons I need this are for creating the forward and back button. So I planned to add 1 or take 1 from the offset which gives me the new ID, and the new filename. That way when users browse it will skip the dead ID values automatically, but it will give an error when giving a false ID.
Going up:
SELECT * FROM table WHERE id > 'your_current_id' ORDER BY id LIMIT 1;
Going down:
SELECT * FROM table WHERE id < 'your_current_id' ORDER BY id DESC LIMIT 1;
ps: it is better to make LIMIT 2, so that you can see that you are at the first or at the last records in the database when only one record is returned.
If your results are ordered by x, ascending, the following will give you your current offeset in the table:
SELECT COUNT(*) FROM tablename WHERE x < x_of_your_current_item;
If you just want to SELECT the next or previous row, you can skip having to do two queries by just directly selecting one row:
SELECT * FROM tablename WHERE x > x_of_your_current_item ORDER BY x LIMIT 1;
will give you the next item (and similarly < and adding DESC to the order-by would give you previous).
You can use offset. Initially set offset as zero.
First time your query will be
SELECT * FROM TABLE order by table_id LIMIT 0,1
and next
SELECT * FROM TABLE order by table_id LIMIT 1,1
..
and so on
This way one will get records from the beginning till the end.
Now about back and forward buttons.
Back Button: First time or whenever offset is zero disable back button
Forward Button:
when you query for a current record you check for the next record too
i.e. after this query
SELECT * FROM TABLE order by table_id LIMIT 0,1 fire a query like this
SELECT * FROM TABLE order by table_id LIMIT current_offset+1,1 and check if the query produces any results if it produces a result then set a boolean say next = TRUE else next = FALSE;
Using this boolean enable or disable Forward button.
One more thing on click of back button send the offset as current_offset - 1 and for forward button current_offset + 1
I hope this helps. I just came across this and thought of this solution.
SELECT * FROM table WHERE ... ORDER BY id DESC LIMIT 50,10;
SELECT * FROM table WHERE ... ORDER BY id DESC LIMIT 60,10;
SELECT * FROM table WHERE ... ORDER BY id DESC LIMIT 70,10;
Simply put, how can I count how many rows there are before a certain row. I'm using incremental ID's, but rows are deleted a random, so just checking to see what the ID is won't work.
If I have, say, 30 rows, and I've selected one based on a name (or anything really), how many rows are there before that one? It could be 16, 1, 12, or anything.
I'm using MySQL and CodeIgniter.
I assume your primary key column has datatype integer
SELECT COUNT(*) FROM `table`
WHERE id < (SELECT id FROM `table` WHERE `conditions are met for specific row`)
Assuming it's an auto_increment column, deleted rows won't be filled in again so this should do the job.
SELECT COUNT(*) FROM table WHERE id_column < your_selected_row_id;
On your model:
$id = $this->db->get('users')->where("name", "John")->id;
$rows = $this->db->get('users')->where("id < ", $id)->num_rows();
return $rows;
Notice how I'm using "chained methods" and for that you need PHP5 which is the default for CI 2.
You first need to get the ID of the record you need to start counting "backwards" which is the first line, considering a table called users and that the column you are filtering is "name" and the row you want to find has the name value of John.
The second line will give you the number of rows that the query "where id < number" returned where number is the ID you got from the first query. Maybe you can even chain both lines.