Inserting thousands of rows in one query MySQL using PHP - php

Question

I have tried whole day (again) but with error trying to insert in total of over billion of rows while inserting about 10k within one query. I want to use prepared statements to do this with MySQL, using innoDB tables. Hardware resources are not a problem, problem has all the time been indexing and one row / insert.
Do not say that I should use straight from list methods since I need to do some calculations before inserting. Input comes from file.
So I have a big while loop, looping text file lines and that's working fine.
I currently do it with inserting one per query.
$stmt = $conn->prepare("INSERT INTO $tableName(row1, row2, row3) VALUES (?, ?, ?)");
$stmt->bind_param("sss", $str1, $str2, $str3);
$stmt->execute();
I have looked at examples doing it like so:
$stmt = $conn->prepare("INSERT INTO $tableName(row1, row2, row3) VALUES (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?) ..... ");
But I need to construct this to for like 5-10k rows for single query.
If someone has done something like this before, please give some reference to work with.
Thanks in advance!

Answer 1

The whole idea of a prepared query is that it reduces the overhead of setting up the query each time it's run – prepare once, execute repeatedly. That means there's not going to be a huge amount of difference (from the POV of the database) between running a prepared statement in a loop, or passing one giant query.
You don't include your whole code here, but an error that many people make is to include the statement preparation in the loop. Don't make that mistake! Something like this will be most efficient:
// these need a value before being used in bind_param
$str1 = $str2 = $str3 = "";
$stmt = $conn->prepare("INSERT INTO $tableName(row1, row2, row3) VALUES (?, ?, ?)");
$stmt->bind_param("sss", $str1, $str2, $str3);
while ($we_have_data) {
$str1 = "foo";
$str2 = "bar";
$str3 = "baz";
$stmt->execute();
}
Though I always recommend using PDO prepared statements, as you get to avoid the potential mess of binding parameters:
// assuming $conn is a PDO object now
$stmt = $conn->prepare("INSERT INTO $tableName(row1, row2, row3) VALUES (?, ?, ?)");
while ($we_have_data) {
$str1 = "foo";
$str2 = "bar";
$str3 = "baz";
$stmt->execute([$str1, $str2, $str3]);
}

Answer 2

Maybe because limit executing time from server. Try to use insert batch.
Add to array
$i = 0;
.. looping
$i++;
$mark[] = '(?,?,?)';
$val[] = $str1.",".$str2.",".$str3;
// check when 1k record
if($i == 1000){
$stmt = $conn->prepare("INSERT INTO ? (row1, row2, row3) VALUES ".implode(',', $mark);
$stmt->bind_param("sss", implode(',', $val));
$stmt->execute();
$mark = []; $val = []; $i = 0; // reset
}