How to insert 100000 records in a schema - php

Question

I'm using Php 5.4, mySql 5.6
I have a schema that I need to fill with 100000s entries in a single query (for testing purpose). I have some problems because the server takes too long to elaborate, so it reaches the time limit for the elaboration. Or it directly fails the call to the database, sending a message of general fatal error without any further explanation.
My question is simple: why? Are there some php parameters that have to be changed?
Moreover, is better to run a single insert more that 1000000 times or a single multi-insert query with 1000000 records?
P.S.: if I cycle 13 times a single instance which inserts 8000 records everything it's ok and it doesn't take so long...

Answer 1

You could do this, for example:
<?php
$x=1;
$col1 = 1;
$col2 = 1;
$col3 = 1;
$quantity = 1000000-1;
$query = "INSERT INTO `table` (`col1`, `col2`, `col3`) VALUES ('$col1', '$col2', '$col3') ";
while ($x <= $quantity)
{
$col1++;
$col2++;
$col3++;
$x++;
$query .= ", ('$col1', '$col2', '$col3') ";
}
echo $query;
?>
Changing quantity to 100-1 generates this:
INSERT INTO `table` (`col1`, `col2`, `col3`) VALUES ('1', '1', '1') ,
('2', '2', '2') , ('3', '3', '3') , ('4', '4', '4') , ('5', '5', '5')
, ('6', '6', '6') , ('7', '7', '7') , ('8', '8', '8') , ('9', '9',
'9') , ('10', '10', '10') , ('11', '11', '11') , ('12', '12', '12') ,
('13', '13', '13') , ('14', '14', '14') , ('15', '15', '15') , ('16',
'16', '16') , ('17', '17', '17') , ('18', '18', '18') , ('19', '19',
'19') , ('20', '20', '20') , ('21', '21', '21') , ('22', '22', '22') ,
('23', '23', '23') , ('24', '24', '24') , ('25', '25', '25') , ('26',
'26', '26') , ('27', '27', '27') , ('28', '28', '28') , ('29', '29',
'29') , ('30', '30', '30') , ('31', '31', '31') , ('32', '32', '32') ,
('33', '33', '33') , ('34', '34', '34') , ('35', '35', '35') , ('36',
'36', '36') , ('37', '37', '37') , ('38', '38', '38') , ('39', '39',
'39') , ('40', '40', '40') , ('41', '41', '41') , ('42', '42', '42') ,
('43', '43', '43') , ('44', '44', '44') , ('45', '45', '45') , ('46',
'46', '46') , ('47', '47', '47') , ('48', '48', '48') , ('49', '49',
'49') , ('50', '50', '50') , ('51', '51', '51') , ('52', '52', '52') ,
('53', '53', '53') , ('54', '54', '54') , ('55', '55', '55') , ('56',
'56', '56') , ('57', '57', '57') , ('58', '58', '58') , ('59', '59',
'59') , ('60', '60', '60') , ('61', '61', '61') , ('62', '62', '62') ,
('63', '63', '63') , ('64', '64', '64') , ('65', '65', '65') , ('66',
'66', '66') , ('67', '67', '67') , ('68', '68', '68') , ('69', '69',
'69') , ('70', '70', '70') , ('71', '71', '71') , ('72', '72', '72') ,
('73', '73', '73') , ('74', '74', '74') , ('75', '75', '75') , ('76',
'76', '76') , ('77', '77', '77') , ('78', '78', '78') , ('79', '79',
'79') , ('80', '80', '80') , ('81', '81', '81') , ('82', '82', '82') ,
('83', '83', '83') , ('84', '84', '84') , ('85', '85', '85') , ('86',
'86', '86') , ('87', '87', '87') , ('88', '88', '88') , ('89', '89',
'89') , ('90', '90', '90') , ('91', '91', '91') , ('92', '92', '92') ,
('93', '93', '93') , ('94', '94', '94') , ('95', '95', '95') , ('96',
'96', '96') , ('97', '97', '97') , ('98', '98', '98') , ('99', '99',
'99') , ('100', '100', '100')
It's not adviced to do so. It's going to take a while to run and I'm not sure if it will work, you can increase the timeout for the page to make sure it runs.
But since it's for testing purposes..
You can change the values of the col variables if you want, but you can't use $var++ on strings.

Answer 2

Is the problem that the test takes longer than the 30 seconds or so that PHP allows before killing the process?
If so, you can set max_execution_time in php.ini.
See
http://php.net/manual/en/info.configuration.php#ini.max-execution-time
http://php.net/manual/en/function.set-time-limit.php

Answer 3

You could try to minimise the traffic between the application server and the SQL engine:
Start by inserting just one record, and then in a PHP loop, double the number of records in each iteration:
First record:
INSERT INTO TABLE mytable (txt) VALUES ('Test');
In a loop:
INSERT INTO TABLE mytable (txt)
SELECT txt FROM mytable;
If you execute this loop 20 times, you'll end up with 220 records, i.e. ~ 1000000
If you have an AUTO_INCREMENT id field, it will of course be filled with unique values. You can also increment another numerical field explicitly, like this:
INSERT INTO TABLE mytable (num) VALUES (1);
In a loop (PHP):
$count = 1;
foreach (range(1, 20) as $i) {
$sql = "INSERT INTO TABLE mytable (num)
SELECT num + $count FROM mytable";
$mysqli->query($sql);
$count += $count;
}

Answer 4

You should insert this one by one for best perfomance and for more clear statement. You should also remove any index or constrains as well as use MyISAM instead of innodb for fastest sql insert. Best is to use .sql file and console input.