php : csv import to multiple table is taking too long - php

Question

i am using csv importer library to load bulk of products into mysql database with codeigniter. csv file has some column names from one table like name, price and some column names from another table like categories. it inserts name and price to first table and categories will be inserted after we got id from first table, so category will go with id of the product. but this is taking too long to process the csv file (1000 entries = 2 minutes). the code i have put is
$csv_file_data = $this->csvimport->get_array($_FILES['csv']['tmp_name']);
foreach ($csv_file_data as $csv_data) {
$required = array(
'name' => $csv_data['name'],
'price' => $csv_data['price']
);
foreach ($required as $key => $value) {
$this->db->set($key, $value);
}
$categories= array();
foreach ($csv_data as $key => $value) {
if(strstr($key, 'categories')) {
$category = explode('|', $value);
for($i = 0; $i < sizeof($category); $i++) {
array_push($categories, $category[$i]);
}
}
}
$this->db->insert('first_table');
$id = $this->db->insert_id();
foreach ($categories as $metatag) {
$this->db->set('productid', $id);
$this->db->set('category', $metatag);
$this->db->insert('second_table');
}
}
but it is taking too long to process only 1000 entries, how to optimize this to process 5000 entries in less than 20-30 seconds ?

Answer 1

I believe this is what you're after ...
$csv_file_data = $this->csvimport->get_array($_FILES['csv']['tmp_name']);
foreach ($csv_file_data as $csv_data) {
$required = array(
'name' => $csv_data['name'],
'price' => $csv_data['price']
);
$this->db->set($required);
$categories = array();
foreach ($csv_data as $key => $value) {
if (strstr($key, 'categories')) {
$category = explode('|', $value);
for ($i = 0; $i < sizeof($category); $i++) {
$categories[] = $category[$i];
}
}
}
$this->db->insert('first_table');
$bulk = array();
$id = $this->db->insert_id();
foreach ($categories as $metatag) {
$bulk[] = array(
'category' => $metatag,
'productid' => $id,
);
}
$this->db->insert_batch('table2', $bulk);
}
Note the use of insert_batch which will dramatically improve your speed and also you can give set() an array.

Answer 2

Can you try a version like this and tell me if this is working and return me the time for 100 rows ?
$bulk = [];
$csv_file_data = $this->csvimport->get_array($_FILES['csv']['tmp_name']);
$required = [];
foreach ($csv_file_data as $csv_data) {
$bulk = [];
$id = $this->db->insert_id();
$required[] = [
'name' => $csv_data['name'],
'price' => $csv_data['price'],
];
foreach ($csv_data as $key => $value) {
if (false !== strpos($key, 'categories')) {
$category = explode('|', $value);
foreach ($category as $metatag) {
$bulk[] = [
'category' => $metatag,
'productid' => $id,
];
}
}
}
}
$this->db->insert_batch('first_table', $required);
$this->db->insert_batch('second_table', $bulk);