Select, edit and then insert data into new database - php

Question

What is the best way to SELECT all from MYSQL DATABASE_1 almost every item edit (split, merge, validate email format, validate URL format, etc.) - by using php and then INSERT INTO new DATABASE 2.
The old table has almost 3 million rows.
If I do foreach for every row (which is not good idea :) ) it takes more than 1 hour for 2 thousand rows.
If I insert multiple rows in one query:
INSERT INTO tbl (col1, col2, ...) VALUES (item1, item2, ...), (item3, item4, ...)
it is almost the same
Any ideas how to do it professionally and fast?

Answer 1

You can do it paging the query and using several php processes. For example, a process one retrieve and edit 100000 elements the insert into new DB.
To implement it you can create a php parent process that distributes the workload into child processes, something like this:
$total_rows = //Get from your mysql table
$offsets = []
$limit = 10000;
//Create offsets to process in bulk
for ($i=0 ; $i<$total_rows ; $i++) {
$offsets[] = ($i * limit);
}
//Process rows
while ($processed_rows < $total_rows)
$pids = [];
//Create 10 childs processes
for ($i=0;$i<10;$i++) {
if ($pid) {
$pids[] = $pid;
} else {
//Get next offset to process
$offset = array_shift ($offsets);
process_and_insert_row($offset, $limit); //this function gets, edits and inserts items
}
}
//Wait for childs to finish
foreach ($pids as $pid) {
pcntl_waitpid($pid, $status);
$processed_rows += $limit;
unset($pids[$pid]);
}
}
Warning: The code is not tested, but I think it demonstrates the idea behind the explanation...