SQL query execution time dramatically increasing - php

Question

My problem is explained below.
This is my PHP code running on my server right now :
$limit = 10000;
$annee = '2017';
//Counting the lines I need to delete
$sql = " SELECT COUNT(*) FROM historisation.cdr_".$annee." a
INNER JOIN transatel.cdr_transatel_v2 b ON a.id_cdr = b.id_cdr ";
$t = $db_transatel->selectAll($sql);
//The number of lines I have to delete
$i = $t[0][0];
do {
if ($i < $limit) {
$limit = $i;
}
//The problem is comming from that delete
$selectFromHistoryAndDelete = " DELETE FROM transatel.cdr_transatel_v2
WHERE id_cdr IN (
SELECT a.id_cdr FROM historisation.cdr_".$annee." a
INNER JOIN (SELECT id_cdr FROM historisation.cdr_transatel_v2) b ON a.id_cdr = b.id_cdr
)
LIMIT " . $limit;
$delete = $db_transatel->exec($selectFromHistoryAndDelete, $params);
$i = $i - $limit;
} while ($i > 0);
The execution of the query.
As you can see on the picture, in the first 195 loops the execution time was between 13 and 17 seconds.
It increased to 73 seconds on the 195th loop and to 1305 seconds on the 196th loop.
Now the query is running for 2000 seconds.
The query is deleting rows in a test table that no one is using right.
I'm deleting row 10,000 by 10,000 for the query to be quick and not overload the server.
I am wondering why is the execution time increasing like that, I though it will be quicker at the end because I though the inner join would be much quicker as they are less rows in the table.
Does anyone has an idea ?
Edit : The tables engine is MyISAM.

Answer 1

Based on your latest comment the inner join is redundant, since you're deleting from the table that contains the values you're joining on. In essence you're having to process b.id_cdr = a.id_cdr twice, since the number of values compared on cdr_2017 are not changed by the inner join, just the number of values queried to be deleted.
As for the cause of the incremental slowness, it is because you are manually performing the same function as SELECT cdr_id FROM cdr_2017 LIMIT 10000 OFFSET x.
That is to say, your query has to perform a full-table scan on cdr_2017 to determine the id values to delete. As you delete the values, the SQL optimizer has to move further through the cdr_2017 table to retrieve the values.
Resulting in
DELETE FROM IN(1,2,3,...10000)
DELETE FROM IN(1,2,3,...20000)
...
DELETE FROM IN(1,2,3,...1000000)
Assuming cdr_id is the incremental primary key, to resolve the issue you could use the last index retrieved from cdr_2017 to filter the selected values.
This will be much faster, as a full-table scan is no longer required to validate the joined records, since you're now utilizing an indexed value on both sides of the query.
$sql = " SELECT COUNT(a.cdr_id) FROM historisation.cdr_".$annee." a
INNER JOIN transatel.cdr_transatel_v2 b ON a.id_cdr = b.id_cdr ";
$t = $db_transatel->selectAll($sql);
//The number of lines I have to delete
$i = $t[0][0];
//set starting index
$previous = 0;
do {
if ($i < $limit) {
$limit = $i;
}
$selectFromHistoryAndDelete = 'DELETE d
FROM transatel.cdr_transatel_v2 AS d
JOIN (
SELECT #previous := cdr_id AS cdr_id
FROM historisation.cdr_2017
WHERE cdr_id > ' . $previous . '
ORDER BY cdr_id
LIMIT 10000
) AS a
ON a.cdr_id = d.cdr_id';
$db_transatel->exec($selectFromHistoryAndDelete, $params);
//retrieve last id selected in cdr_2017 to use in next iteration
$v = $db_transatel->selectAll('SELECT #previous'); //prefer fetchColumn
$previous = $v[0][0];
$i = $i - $limit;
} while ($i > 0);
//optionally reclaim table-space
$db_transatel->exec('OPTIMIZE TABLE transatel.cdr_transatel_v2', $params);
You could also refactor to use cdr_id > $previous AND cdr_id < $last to remove the order by limit clauses, which should also improve performance.
Though I would like to note, that a table lock on cdr_transatel_v2 is performed during this operation by the MyISAM database engine. Due to the way MySQL handles concurrent sessions and queries, there is not much gain from a batch delete in this manner, and is really only applicable to InnoDB and transactions. Especially when using PHP with FastCGI, as opposed to Apache mod_php. Since other queries not on cdr_transatel_v2 will still be executed and write operations on cdr_transatel_v2 will still be queued. If using mod_php I would reduce the limit to 1,000 records to reduce queue times.
For more information see https://dev.mysql.com/doc/refman/5.7/en/internal-locking.html#internal-table-level-locking
Alternative approach.
Considering the large number of records that need to be deleted, when the records deleted exceed those that are kept, it would be more beneficial to invert the operation by using INSERT instead of DELETE.
#ensure the storage table doesn't exist already
DROP TABLE IF EXISTS cdr_transatel_temp;
#duplicate the structure of the original table
CREATE TABLE transatel.cdr_transatel_temp
LIKE transatel.cdr_transatel_v2;
#copy the records that are not to be deleted from the original table
INSERT transatel.cdr_transatel_temp
SELECT *
FROM transatel.cdr_transatel_v2 AS d
LEFT JOIN historisation.cdr_2017 AS b
ON b.cdr_id = d.cdr_id
WHERE b.cdr_id IS NULL;
#replace the original table with the storage table
RENAME TABLE transatel.cdr_transatel_v2 to transatel.backup,
transatel.cdr_transatel_temp to cdr_transatel_v2;
#remove the original table
DROP TABLE transatel.backup;