PHP MySQL query stops after 1273459 loops - php

Question

I am using the script:
$file=fopen('part1.csv', 'r');
mysql_connect('localhost', '~~~', '~~~') or die(mysql_error());
mysql_select_db('Stubby') or die(mysql_error());
while (($buffer = fgets($file, 4096)) !== false) {
//echo $buffer;
$q = mysql_query('INSERT INTO allCombos (combo) VALUES (\'' . $buffer . '\')') or die(mysql_error());
}
fclose($file);
To load the very long contents of a CSV into a database. The CSV has around 3.5M lines. The querys stop at 1273459 lines. Why?

Answer 1

PHP generally sets its default script load timelimit to 30 seconds; you're probably hitting that limit. You can manually override it.
set_time_limit(0); //sets the time limit to infinity
set_time_limit(600); //sets the time limit to 10 minutes
Another possibility is that your script has run out of memory. You can raise it by doing something like:
ini_set('memory_limit', '32M'); //raises limit to 32 megabytes

Answer 2

Just stops, or any error is shown? On the other hand, you seem just copy file to database, why not to use LOAD DATA INFILE command? So you don't need a loop, and maybe even php application, that command can load csv file into the table (the fastest way to do it).
In your case you can execute the followig command to export:
LOAD DATA INFILE 'part1.csv' INTO TABLE allCombos
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
(just copied from MySQL LOAD DATA description page and set your parameters). Btw, are there any other fields, or csv file has everything what should be in that table?

Answer 3

I'm guessing you're running out of memory or exceeding the processing time allowed. Why not chunk it into smaller pieces and do it in groups of 500,000 or something? Or adjust the timeout.
http://php.net/manual/en/function.set-time-limit.php
Increasing memory you would have to do through the php.ini file.
http://php.net/manual/en/ini.core.php

Answer 4

build one big insert and hit database ones:
q = 'INSERT INTO allCombos (combo) VALUES ';
while (($buffer = fgets($file, 4096)) !== false) {
//echo $buffer;
$q .=' (\'' . $buffer . '\'), ';
}
q = substr ( q, 0, -2 ); // remove last comma
mysql_query (q);
playing with time limits also will help, however it will be more effective using resources...
most effective is using LOAD DATA INFILE form Maxym