More effecienent way to process data feed of 100K entries? - php

Question

I have a csv file that has roughly 100K entries I need to process and insert into a data base.
Previously it was very slow because it makes an SQL call for every entry. I do this though because if I try to build 1 single query to do this I will run out of memory.
I migrated to a new server and now I get an error every time I run it:
SQL Error : 2006 MySQL server has gone away
I am not sure but think this is just happening because how inefficient my code is.
What can I do to make it perform better and not get the error?
Here is the code:
//empty table before saving new feed
$model->query('TRUNCATE TABLE diamonds');
$fp = fopen($this->file,'r');
while (!feof($fp))
{
$diamond = fgetcsv($fp);
//skip the first line
if(!empty($firstline))
{
$firstline = true;
continue;
}
if(empty($diamond[17]))
{
//no price -- skip it
continue;
}
$data = array(
'seller' => $diamond[0],
'rapnet_seller_code' => $diamond[1],
'shape' => $diamond[2],
'carat' => $diamond[3],
'color' => $diamond[4],
'fancy_color' => $diamond[5],
'fancy_intensity' => $diamond[6],
'clarity' => empty($diamond[8]) ? 'I1' : $diamond[8],
'cut' => empty($diamond[9]) ? 'Fair' : $diamond[9],
'stock_num' => $diamond[16],
'rapnet_price' => $diamond[17],
'rapnet_discount' => empty($diamond[18]) ? 0 : $diamond[18],
'cert' => $diamond[14],
'city' => $diamond[26],
'state' => $diamond[27],
'cert_image' => $diamond[30],
'rapnet_lot' => $diamond[31]
);
$measurements = $diamond[13];
$measurements = strtolower($measurements);
$measurements = str_replace('x','-',$measurements);
$mm = explode('-',$measurements);
$data['mm_width'] = empty($mm[0]) ? 0 : $mm[0];
$data['mm_length'] = empty($mm[1]) ? 0 : $mm[1];
$data['mm_depth'] = empty($mm[2]) ? 0 : $mm[2];
//create a new entry and save the data to it.
$model->create();
$model->save($data);
}
fclose($fp);

Answer 1

You're probably exceeding MySQL's max_allowed_packet setting, which sets a hard limit (in bytes) on how long a query string can be. There's nothing wrong with doing multi-value inserts, but 100k of them is definitely pushing things.
Instead of doing all 100k at once, try doing 1000 in a loop. You're still reducing total query count (down from 100k to just 1000), so it's still a net gain.