How to make a CSV upload perform faster? - php

Question

I was wondering if any of you guys have tried something about CSV migration in a live form(multipart/form-data) mine is working the only thing I hate about is that it consumes so much of time and it's reaching the maximum execution timeout. The quick fix I made is by setting the maximum execution time in my php.ini(or set_time_limit()) but it's really annoying me to wait for half an hour just to import the whole data though it's not more than 100kb. Am I just overreacting or something?
This is the code:
function upload($id, $old_eid)
{
$filename = $_FILES['event_file']['tmp_name'];
$handle = fopen($filename, "r");
while(($data = fgetcsv($handle, 1000, ",")) !== FALSE){
$id = $id;
$id2 = $data[2];
$ckr = $this->Manager_model->check_if_record_exists($id, $id2);
if(count($ckr) > 0):
$this->session->set_flashdata('err', '<div class="error">Duplicated record</div>');
redirect("manager/csver/$id");
else:
$data['col1'] = $data[0];
$data['col2'] = $id;
$data['col3'] = $data[3].' '.$data[4];
$data['col4'] = $data[2];
$data['col5'] = $data[6];
$data['col6'] = $data[1];
$data['col7'] = $data[7];
$data['col8'] = mt_rand(11111, 99999);
$data['col9'] = $old_eid;
$this->Manager_model->add_csv($data);
$this->Manager_model->add_csv_to_photo($data);
endif;
}
fclose($handle);
$this->session->set_flashdata('success', '<div class="success">CSV successfully uploaded</div>');
redirect("manager/records/$id");
//$this->session->set_flashdata('msg', '<div class="success">Records successfully uploaded</div>');
}
My Manager_model:
function add_csv($data)
{
$src = array(
'col1'=> $data['col1'],
'col2' => $data['col2'],
'col3' => $data['col3'],
'col4' => $data['col4'],
'col5' => $data['col5'],
'col6' => $data['col6'],
'col7' => $data['col7'],
'col8' => $data['col8'],
);
$this->db->insert('e_records2', $src);
if($this->db->affected_rows() == '1'):
return TRUE;
endif;
return FALSE;
}
function add_csv_to_photo($data) {
$src = array(
'col1'=> $data['col1'],
'col2' => $data['col2'],
'col3' => $data['col3'],
'col4' => $data['col4'],
'col5'=> $data['col5'],
'col6'=> $data['col6'],
);
$this->db->insert('e_records', $src);
if($this->db->affected_rows() == '1'):
return TRUE;
endif;
return FALSE;
} function check_if_record_exists($id, $id2)
{
$eid = $id;
$id2 = $id2;
$query = $this->db->query("select * from races_results where eid = $eid AND id2 = $id2");
return $query->result();
}
P.S.
I'm not talking about PhpMyAdmin here cos I know how import csv file works there. And plus it would create a lot of trivial tasks to have a file to migrate using the bone.

Answer 1

Why not run the profiler to optimize your code? Codeigniter includes this useful piece for problems like this http://codeigniter.com/user_guide/general/profiling.html
It will give you a breakdown of your SQL queries and what is taking long, and where.
$this->output->enable_profiler(TRUE);

Answer 2

The problem seems to me that you are querying the DB once (or twice ?) per line in your CSV file.
Of course you're going to get horrible performance.
You can do the whole query in one go and have the DB make the CSV for you in no time.
SELECT DISTINCT f1,f2,f3,... FROM tablex WHERE .. INTO OUTFILE 'c:/dir/ca.csv'
FIELDS ESCAPED BY '"' FIELDS TERMINATED BY ';' LINES TERMINATED BY '\n';
//note the use of forward slashes even on Windows.
See: http://dev.mysql.com/doc/refman/5.0/en/select-into.html
The speed of the select itself is the limiting factor here.
Make sure you have write permissions on the directory and note that MySQL will never overwrite files.
This command is very fast on MySQL.

Answer 3

$id = $id;
really?
$ckr = $this->Manager_model->check_if_record_exists($id, $id2);
One obvious way to make it go faster would be to have a unique index on eid and id2 and ignore duplicate row errors on the INSERT.
But really, f you want it to go much faster, just tell mysql to parse and load the data.