How to inspect empty rows in uploaded csv - php

Here i have a csv file which have contains numerous contacts rows, and these rows ae saved to my database table. Now i just want to inspect empty rows from uploaded csv by customer.
Here is an example csv with some empty rows
In this above csv, have 3rd and 6th rows are empty. so want to inspect these empty row number and discard csv with error.
Here is my csv code
$filename = $_FILES["csv_file"]["tmp_name"];
if ($_FILES["csv_file"]["size"] > 0) {
$file = fopen($filename, "r");
$importdata = fgetcsv($file, 10000, ",");
$counter = 1;
while (!feof($file)) {
if ($counter > 1) {
$alldata[] = fgetcsv($file);
}
$counter++;
}
fclose($file);
$csvfieldcounter = 1;
foreach ($alldata as $importdata) {
$userdata = $this->session->userdata();
$userId = $userdata['id'];
$status = 'Y';
if ($importdata[4] == 'Disable' || $importdata[4] == 'disable')
$status = 'N';
else if ($importdata[4] == 'Enable' || $importdata[4] == 'enable')
$status = 'Y';
$data = array(
'customer_name' => $importdata[0],
'customer_email' => $importdata[1],
'customer_mobile' => $importdata[2],
'birth_date' => $importdata[3],
'status' => $status,
'user_id' => $userId,
'cat_type' => $file_cat
);
if ($importdata[2]) {
$run = $this->db->insert('customer', $data);
$csvfieldcounter++;
$id = $this->db->insert_id();
}
}
$this->session->set_flashdata('csv_imported','Your CSV have been successfully imported.');
redirect('/customer', $csvfieldcounter);
}
I just want a little help for get that. your kind efforts would be appreciated Thanks :)

I suggest you to use below mentioned library. This will help you address above issue also it will decrease lines of code in your controller.
https://github.com/parsecsv/parsecsv-for-php
This will help you.
If you worried about how to use than check below mentioned code.
$this->load->library('Parsecsv');
$csv = new Parsecsv($file);
$users = $csv->data;
$noOfUsers = count($users);
foreach($users as $user):
if(!empty($user)):
// Write your code
endif;
endforeach;

Related

When import CSV skip header or first row

I know this is duplicate question. But I have tried all the answers which I have found on the https://stackoverflow.com/ .
I think in my case I am inserting and updating data if same date match than record will update if not than it will insert.
Below is my file code:
<?php
if ( isset( $_POST['submit'] ) && $_POST['upload-csv'] == 'upload' ) {
$error = array();
$success = array();
$filename = $_FILES['file']['name'];
$filetype = wp_check_filetype( $filename );
if ( $filetype['ext'] == 'csv' && $filetype['type'] == 'text/csv' ) {
$handle = fopen( $_FILES['file']['tmp_name'], "r" );
$row = 0;
$skip_row_number = array("1");
while ( ($data = fgetcsv( $handle, 1000, "," )) !== FALSE ) {
$data = array_map("utf8_encode", $data);
if ($row > 0)
{
$table_name = $wpdb->prefix . 'prayer';
$ipquery = $wpdb->get_results("SELECT * FROM `$table_name` WHERE `date` = '".$data[0]."'");
$query_res = $wpdb->num_rows;
// Check if same date data
if($query_res >=1){
$updateQuery = "UPDATE `$table_name` SET
`date` = '".$data[0]."',
`first_start` = '".$data[1]."',
`first_end` = '".$data[2]."',
`second_start` = '".$data[3]."',
`second_end` = '".$data[4]."',
`third_start` = '".$data[5]."',
`third_end` = '".$data[6]."',
`forth_start` = '".$data[7]."',
`forth_end` = '".$data[8]."',
`five_start` = '".$data[9]."',
`five_end` = '".$data[10]."',
`done` = '".$data[10]."'
WHERE `$table_name`.`date` = '".$data[0]."';";
$up_res = $wpdb->query($updateQuery);
}else{
$query = "INSERT INTO $table_name (date, first_start, first_end, second_start,
second_end, third_start, third_end, forth_start, forth_end, five_start, five_end, done)
VALUES ('".$data[0]."','".$data[1]."','".$data[2]."','".$data[3]."','".$data[4]."','".$data[5]."','".$data[6]."','".$data[7]."','".$data[8]."','".$data[9]."','".$data[10]."','".$data[11]."')";
$insert_res = $wpdb->query($query);
}
}
$row++;
}
fclose( $handle );
$success[] = 'Import done.';
} else {
$error[] = 'Please upload CSV file only';
}
}
?>
I have tried the below answer for skip the header:
Skip the first line of a CSV file
Import CSV, exclude first row
skip first line of fgetcsv method in php
Help me sort out this issue.
ParseCSV is latest and easy way to get data from CSV and you can get control of data from CSV easily.
in which you have to add library file as per proper path
e.g. require_once 'parsecsv.lib.php';
After then it return title and data seperately.
$filename = 'abc.csv'; // path of CSV file or after upload pass temp path of file
$csv = new parseCSV();
$csv->auto($filename);
foreach ($csv->titles as $value):
$getcsvtitle[] = // get header in variable
endforeach;
foreach ($csv->data as $key => $row):
//$getdataasperrow = // get data row wise.
endforeach;
ParseCSV return data in array format after just adding library, so you can easily separate header and other data, and compatible to new line and special character as well as i have been always using it as well.
Please refer below link for further detail as well.
ParseCSV GitHub

How to upload a Large CSV file really fast in Laravel

This question has been asked so many times , I have tried couple of way also but this time I am stuck since my requirement is bit specific . None of the generic methods worked for me .
Details
File Size = 75MB
Total Rows = 300000
PHP Code
protected $chunkSize = 500;
public function handle()
{
try {
set_time_limit(0);
$file = Flag::where('imported','=','0')
->orderBy('created_at', 'DESC')
->first();
$file_path = Config::get('filesystems.disks.local.root') . '/exceluploads/' .$file->file_name;
// let's first count the total number of rows
Excel::load($file_path, function($reader) use($file) {
$objWorksheet = $reader->getActiveSheet();
$file->total_rows = $objWorksheet->getHighestRow() - 1; //exclude the heading
$file->save();
});
$chunkid=0;
//now let's import the rows, one by one while keeping track of the progress
Excel::filter('chunk')
->selectSheetsByIndex(0)
->load($file_path)
->chunk($this->chunkSize, function($results) use ($file,$chunkid) {
//let's do more processing (change values in cells) here as needed
$counter = 0;
$chunkid++;
$output = new ConsoleOutput();
$data =array();
foreach ($results->toArray() as $row)
{
$data[] = array(
'data'=> json_encode($row),
'created_at'=>date('Y-m-d H:i:s'),
'updated_at'=> date('Y-m-d H:i:s')
);
//$x->save();
$counter++;
}
DB::table('price_results')->insert($data);
$file = $file->fresh(); //reload from the database
$file->rows_imported = $file->rows_imported + $counter;
$file->save();
$countx = $file->rows_imported + $counter;
echo "Rows Executed".$countx.PHP_EOL;
},
false
);
$file->imported =1;
$file->save();
echo "end of execution";
}
catch(\Exception $e)
{
dd($e->getMessage());
}
}
So the above Code runs really fast for the 10,000 rows CSV File.
But when I upload a larger CSV its not working .
My Only restriction here is that I have to use following logic to transform each row of the CSV to KeyPair value json data
foreach ($results->toArray() as $row)
{
$data[] = array(
'data'=> json_encode($row),
'created_at'=>date('Y-m-d H:i:s'),
'updated_at'=> date('Y-m-d H:i:s')
);
//$x->save();
$counter++;
}
Any suggestions would be appreciated , Its been more than and Hour now and still only 100,000 rows have been inserted
I find this is really slow
Database : POSTGRES

how to handle duplicate records in SQL?

I am uploading a file to mysql database. Now the file contains records of login and logout of users. Below is its structure.
I am using PHP codeigniter to upload that file and insert its data into SQL.
Here date_data is Defined as UNIQUE So that i dont get any duplicate records for same day as it contains information of daily user's login and logout data.
Now in a case where i have uploaded a data of 1st Nov to 10th December. and then again i upload data from 1st December to 1St January i will get an error because it will give error for the duplicate data from 1st Dec and other consecutive days. Is it possible that i can skip the duplicate data and insert the remaining unique data??
As for my current code it stops the execution when it finds duplicate data. i want to insert only that data which is unique.
Below is my code to insert into SQL table:
Controller
public function upload()
{
$file = rand(1000, 100000) . "-" . $_FILES['file']['name'];
$file_loc = $_FILES['file']['tmp_name'];
$file_size = $_FILES['file']['size'];
$file_type = $_FILES['file']['type'];
$folder = "uploads/";
$location = $_FILES['file'];
$new_size = $file_size / 1024; // new file size in KB
$new_file_name = strtolower($file);
$final_file = str_replace(' ', '-', $new_file_name); // make file name in lower case
if (move_uploaded_file($file_loc, $folder . $final_file))
{
$handle = fopen($folder.$final_file, "r") or die("file cannot open");
if ($handle) {
while (($line = fgets($handle)) !== false)
{
$lineArr = explode("\t", "$line");
$result = $this->attendance_m->insert_file_content($lineArr) ;
}
if (fclose($handle)) {
$this->alert('successfully uploaded', 'admin/attendance.php?success');
redirect('admin/attendance');
}
}
else{
echo "file cannot open";
}
}
}
The Model:
public function insert_file_content($lineArr)
{
$data = array(
'emp_id' => $lineArr[0],
'date_data' => $lineArr[1],
'abc' => $lineArr[2],
'def' => $lineArr[3],
'entry' => $lineArr[4],
'ghi' => $lineArr[5],
);
$this->db->insert('daily_data2', $data);
}
To ignore a duplicate record on insert
$data = array(
'emp_id' => $lineArr[0],
'date_data' => $lineArr[1],
'abc' => $lineArr[2],
'def' => $lineArr[3],
'entry' => $lineArr[4],
'ghi' => $lineArr[5],
);
$sql = "INSERT IGNORE INTO `daily_data2`
(`emp_id`,`date_data`,`abc`,`def`,`entry`,`ghi`)
VALUES
(?,?,?,?,?,?)";
$this->db->query($sql, $data);
You could also try out this way
$result = $this->db->get_where('daily_data2', array('date_data' => $data['date_data'));
if(count( $result->result_array() ) < 1)
{
//insert a new record
}

What's the best way to compare / insert / update products in a MySQL db from a .CSV file

At our company we pull a .CSV file from the suppliers FTP server and update our product data (price, stock,..) each morning.
We wrote a cron for this task as it should run automatically.
The current script is working in most cases. However, sometimes we recieve an error: 'Allowed memory size of 134217728 bytes exhausted (tried to allocate 75 bytes)'.
We use CodeIgniter with DataMapper ORM. A possible design error might be the fact that the script is working with objects instead of array's...
Each time 49000 rows are checked.
Can anyone help us find another way of doing this?
The following script is the function that runs after the files are copied.
// Include auth connection params
$udb = $this->_completeParams($db);
// Check if an update was downloaded
$supplier = new Supplier(NULL,$udb);
$supplier->where(array('alias'=>'XX','name'=>'xxxxxxxxx'))->get(1);
$cronStart = date('Y-m-d H:i:s');
$cronStartDate = date('Y-m-d');
//mail($this->adminMail, 'CRON', 'Gestart:' .$cronStart, $this->headerMail);
//$message .= '1: '.memory_get_usage()."\r\n";
if($supplier->import_found) {
//if(true) {
$rows = 0;
$updated = 0;
$new = 0;
//$aAvailable = array();
$message .= '<h3>Start: '.$cronStart.'</h3>' . "\r\n";
$object = new Supplier_product(NULL,$udb);
$cat = new Supplier_category(NULL, $udb);
$manu = new Supplier_manufacturer(NULL, $udb);
$auvibel = new Supplier_auvibel(NULL, $udb);
$bebat = new Supplier_bebat(NULL, $udb);
$recupel = new Supplier_recupel(NULL, $udb);
$reprobel = new Supplier_reprobel(NULL, $udb);
$files = glob($this->tempDir.'XXXXX/prices/*');
foreach($files as $file) {
$ext = pathinfo($file, PATHINFO_EXTENSION);
$data = ($ext == 'txt')?$this->_csvToArray($file, ';'):false;
// If the CSV data is in $data
if($data !== false) {
$totalCount = count($data);
for($i = 0; $i <= $totalCount; $i++) {
//$aAvailable[] = $data[$i]['ArtID'];
$rows++;
//$message .= 'loop start: '.memory_get_usage()."\r\n";
$object->where(array('art_id'=>$data[$i]['ArtID'],'supplier_id'=>$supplier->id))->get(1);
$auvibel->select('value')->where(array('art_id'=>$data[$i]['ArtID'], 'supplier_id'=>$supplier->id))->get(1);
$auvibel->value = ($auvibel->exists())?$auvibel->value:0;
$bebat->select('value')->where(array('art_id'=>$data[$i]['ArtID'], 'supplier_id'=>$supplier->id))->get(1);
$bebat->value = ($bebat->exists())?$bebat->value:0;
$recupel->select('value')->where(array('art_id'=>$data[$i]['ArtID'], 'supplier_id'=>$supplier->id))->get(1);
$recupel->value = ($recupel->exists())?$recupel->value:0;
$reprobel->select('value')->where(array('art_id'=>$data[$i]['ArtID'], 'supplier_id'=>$supplier->id))->get(1);
$reprobel->value = ($reprobel->exists())?$reprobel->value:0;
$intrastat = 0;
$data[$i]['LP_Eur'] = ($data[$i]['LP_Eur'] != '')?str_replace(',', '.', $data[$i]['LP_Eur']):0;
$data[$i]['DE_Eur'] = ($data[$i]['DE_Eur'] != '')?str_replace(',', '.', $data[$i]['DE_Eur']):0;
$data[$i]['D1_Eur'] = ($data[$i]['D1_Eur'] != '')?str_replace(',', '.', $data[$i]['D1_Eur']):0;
$data[$i]['D1_Eur'] = ($data[$i]['D2_Eur'] != '')?str_replace(',', '.', $data[$i]['D2_Eur']):0;
$data[$i]['PricePersonal_Eur'] = ($data[$i]['PricePersonal_Eur'] != '')?str_replace(',', '.', $data[$i]['PricePersonal_Eur']):0;
$data[$i]['BackorderDate'] = ($data[$i]['BackorderDate'] != '')?date('Y-m-d', strtotime($data[$i]['BackorderDate'])):NULL;
$data[$i]['ModifDate'] = ($data[$i]['ModifDate'] != '')?date('Y-m-d', strtotime($data[$i]['ModifDate'])):NULL;
if($object->exists()) {
if($object->allow_cron_update) { //if($data[$i]['ModifDate'] != $object->modified) {
// Check if category group exists
$cat->select('id')->where(array(
'supplier_id' => $supplier->id,
'name_a' => $data[$i]['Class1'],
'name_b' => $data[$i]['Class2'],
'name_c' => $data[$i]['Class3'],
))->get(1);
if(!$cat->exists()) {
// Category should be added
$cat->supplier_id = $supplier->id;
$cat->name_a = $data[$i]['Class1'];
$cat->name_b = $data[$i]['Class2'];
$cat->name_c = $data[$i]['Class3'];
$cat->save();
// Log as notification: New supplier categorie
$this->_notify('Niewe categorie',array(
'body' => $supplier->name.' heeft "'.$cat->name_a.' - '.$cat->name_b.' - '.$cat->name_c.'" als nieuwe categorie toegevoegd.',
'controller' => 'leveranciers',
'trigger' => 'new_supplier_category',
'url' => base_url().'leveranciers/item/'.$supplier->id.'/categorien',
'icon' => 'icon-truck',
'udb' => $udb,
));
}
// Check if manufacturer exists
$manu->select('id')->where(array(
'name' => $data[$i]['PublisherName']
))->get(1);
if(!$manu->exists()) {
// Manufacturer should be added
$manu->name = $data[$i]['PublisherName'];
$manu->save($supplier);
}
// Add the product to the database
$object->art_id = $data[$i]['ArtID'];
$object->supplier_id = $supplier->id;
$object->supplier_category_id = $cat->id;
$object->supplier_manufacturer_id = $manu->id;
$object->part_id = $data[$i]['PartID'];
$object->ean_code = $data[$i]['EanCode'];
$object->name = $data[$i]['Description'];
$object->description = NULL;
$object->version = $data[$i]['Version'];
$object->language = $data[$i]['Language'];
$object->media = $data[$i]['Media'];
$object->trend = $data[$i]['Trend'];
$object->price_group = $data[$i]['PriceGroup'];
$object->price_code = $data[$i]['PriceCode'];
$object->eur_lp = $data[$i]['LP_Eur'];
$object->eur_de = $data[$i]['DE_Eur'];
$object->eur_d1 = $data[$i]['D1_Eur'];
$object->eur_d2 = $data[$i]['D2_Eur'];
$object->eur_personal = $data[$i]['PricePersonal_Eur'];
$object->stock = $data[$i]['Stock'];
$object->backorder = ($data[$i]['BackorderDate'] != '' && !empty($data[$i]['BackorderDate']))?$data[$i]['BackorderDate']:NULL;
$object->modified = ($data[$i]['ModifDate'] != '' && !empty($data[$i]['ModifDate']))?$data[$i]['ModifDate']:NULL;
$object->flag = 'MODIFIED';
$object->auvibel = $auvibel->value;
$object->bebat = $bebat->value;
$object->intrastat = $intrastat;
$object->recupel = $recupel->value;
$object->reprobel = $reprobel->value;
$object->save();
$updated++;
}
elseif(($object->auvibel != $auvibel) || ($object->bebat != $bebat) || ($object->recupel != $recupel) || ($object->reprobel != $reprobel)) {
$object->auvibel = $auvibel->value;
$object->bebat = $bebat->value;
$object->intrastat = $intrastat;
$object->recupel = $recupel->value;
$object->reprobel = $reprobel->value;
$object->save();
}
}
else {
// Check if category group exists
$cat->select('id')->where(array(
'supplier_id' => $supplier->id,
'name_a' => $data[$i]['Class1'],
'name_b' => $data[$i]['Class2'],
'name_c' => $data[$i]['Class3'],
))->get(1);
if(!$cat->exists()) {
// Category should be added
$cat->supplier_id = $supplier->id;
$cat->name_a = $data[$i]['Class1'];
$cat->name_b = $data[$i]['Class2'];
$cat->name_c = $data[$i]['Class3'];
$cat->save();
// Log as notification: New supplier categorie
$this->_notify('Niewe categorie',array(
'body' => $supplier->name.' heeft "'.$cat->name_a.' - '.$cat->name_b.' - '.$cat->name_c.'" als nieuwe categorie toegevoegd.',
'controller' => 'leveranciers',
'trigger' => 'new_supplier_category',
'url' => '[hidden-url]'.$supplier->id.'/categorien',
'icon' => 'icon-truck',
'udb' => $udb,
));
}
// Check if manufacturer exists
$manu->select('id')->where(array(
'name' => $data[$i]['PublisherName']
))->get(1);
if(!$manu->exists()) {
// Manufacturer should be added
$manu->name = $data[$i]['PublisherName'];
$manu->save($supplier);
}
// Add the product to the database
$object->art_id = $data[$i]['ArtID'];
$object->supplier_id = $supplier->id;
$object->supplier_category_id = $cat->id;
$object->supplier_manufacturer_id = $manu->id;
$object->part_id = $data[$i]['PartID'];
$object->ean_code = $data[$i]['EanCode'];
$object->name = $data[$i]['Description'];
$object->description = NULL;
$object->version = (($data[$i]['Version'] != '')?$data[$i]['Version']:NULL);
$object->language = (($data[$i]['Language'] != '')?$data[$i]['Language']:NULL);
$object->media = (($data[$i]['Media'] != '')?$data[$i]['Media']:NULL);
$object->trend = (($data[$i]['Trend'] != '')?$data[$i]['Trend']:NULL);
$object->price_group = (($data[$i]['PriceGroup'] != '')?$data[$i]['PriceGroup']:NULL);
$object->price_code = (($data[$i]['PriceCode'] != '')?$data[$i]['PriceCode']:NULL);
$object->eur_lp = (($data[$i]['LP_Eur'] != '')?$data[$i]['LP_Eur']:NULL);
$object->eur_de = (($data[$i]['DE_Eur'] != '')?$data[$i]['DE_Eur']:NULL);
$object->eur_d1 = (($data[$i]['D1_Eur'] != '')?$data[$i]['D1_Eur']:NULL);
$object->eur_d2 = (($data[$i]['D2_Eur'] != '')?$data[$i]['D2_Eur']:NULL);
$object->eur_personal = $data[$i]['PricePersonal_Eur'];
$object->stock = $data[$i]['Stock'];
$object->backorder = ($data[$i]['BackorderDate'] != '' && !empty($data[$i]['BackorderDate']))?$data[$i]['BackorderDate']:NULL;
$object->modified = ($data[$i]['ModifDate'] != '' && !empty($data[$i]['ModifDate']))?$data[$i]['ModifDate']:NULL;
$object->flag = NULL;
$object->auvibel = $auvibel->value;
$object->bebat = $bebat->value;
$object->intrastat = $intrastat;
$object->recupel = $recupel->value;
$object->reprobel = $reprobel->value;
$object->save();
//$object->clear_cache();
$new++;
}
//$message .= 'loop end A: '.memory_get_usage().' - '.$i."\r\n";
$object->clear();
$cat->clear();
$manu->clear();
$auvibel->clear();
$bebat->clear();
$recupel->clear();
$reprobel->clear();
unset($data[$i]);
//$message .= 'loop end B: '.memory_get_usage()."\r\n";
}
}
unset($manu);
unset($auvibel);
unset($bebat);
unset($recupel);
unset($reprobel);
if(is_file($file)) {
unlink($file);
}
$object->clear();
//$message .= 'BEFORE MARK EOL: '.memory_get_usage()."\r\n";
/**
* Mark products as EOL when not found in file
*/
$eolCount = 0;
$eol = $object
->group_start()
->where('flag IS NULL')
->or_where('flag !=', 'EOL')
->group_end()
->where('supplier_id', $supplier->id)
->group_start()
->group_start()->where('updated IS NOT NULL')->where('updated <',$cronStart)->group_end()
->or_group_start()->where('updated IS NULL')->where('created <',$cronStart)->group_end()
->group_end()
->get_iterated();
$p = new Product(NULL,$udb);
//unset($aAvailable);
foreach($eol as $i => $product) {
$product->flag = "EOL";
$product->save();
if($product->art_id != NULL) {
// The 'copied' products should be marked eol in the webshop!
$p->where('art_code',$product->art_id)->where('supplier_product_id', $product->id)->get();
if($p->exists()) {
$p->eol = date('Y-m-d H:i:s');
$p->save();
}
$p->clear();
}
$product->clear();
$eolCount++;
//unset($eol[$i]);
//$message .= 'INSIDE MARK EOL: '.memory_get_usage()."\r\n";
}
unset($product);
$object->clear();
//$message .= 'AFTER MARK EOL: '.memory_get_usage()."\r\n";
if($eolCount > 0) {
// Log as notification: supplier products marked EOL
$this->_notify('EOL melding',array(
'body' => "Er ".(($eolCount == 1)?'is een product':'zijn '.$eolCount.' producten')." gemarkeerd als EOL",
'controller' => 'leveranciers',
'trigger' => 'eol_supplier_product',
'url' => '[hidden-url]'.$supplier->id.'/artikels',
'icon' => 'icon-truck',
'udb' => $udb,
));
}
}
// After looping files build e-mail.
$message .= 'Totaal: '.$rows. "\r\n";
$message .= 'new: '.$new. "\r\n";
$message .= 'updated: '.$updated. "\r\n";
$message .= 'EOL: '.$eolCount. "\r\n";
$subject = 'Import XXXXX Update';
}
// No updates found
else {
$subject = 'Import XXXXX No Update Found';
$message .= "\r\n";
}
$message .= '<h3>Einde: '.date('Y-m-d H:i:s').'</h3>' . "\r\n";
mail($this->adminMail, $subject, $message, $this->headerMail);
// Remove import_found marker for supplier
$supplier->import_found = false;
$supplier->save();
We had a similar situation. After a lot of attempts at making the script better, we decided that we needed another approach to make our import work and not take ~10 hours.
What we did was dump all the PHP code, and instead use mysqlimport to load the contents of the CSV file directly into a table. That table now contains everything we need, but not in a form that's useful for us (no structure, some fields need some processing, etc.)
However, because everything is now in the database, we can do everything we want with a query.
For example, deleting all data that is no longer in the import file, thats just DELETE FROM structured_table AS st LEFT JOIN unstructured_table AS ut ON st.someField = ut.someField WHERE ut.someField IS NULL;, updating existing records is just UPDATE structured_table AS st INNER JOIN unstructured_table AS ut ON st.someField = ut.someField SET st.anotherField = CONCAT(ut.aField, ' ', ut.yetAnotherField);.
Obviously, for a complex import script, your queries will be more complex and you'll need more of them. You might even need to throw some stored procedures in to do processing on individual fields. But if you can take this kind of approach you'll end up with a process that can handle a lot of data and is very scalable.
I have a similar situation... Compare around 20M records every day to update a few records with changes and add / remove the delta. Data source is CSV as well. I use perl, while I think php also work.
Each record must have a linking key, SKU of product? Or something like that. May already be the primary key /unique key in your DB table.
You know the lst of fields that you want to compare and update.
Step 1: read ALL records from DB, store in an array using the linking key as named index.
1.1: value is concat of all fields need to compare, or md5() of the concat result to save memory.
Step 2: loop through the CSV file, extract the linking key and new values per row.
2.1: if linking key is NOT in the array, INSERT action to DB.
2.2: isset() return true so compare the values (or md5() of the value concat), if different, UPDATE action to DB.
2.3: delete this entry from the array.
Step 3: by the end of reading the CSV, the entries remains in the array were records to DELETE.
In my case, it use less than 2GB RAM for the process and runs around 3 minutes, which should be feasible and acceptable.

Only the first IF statement out of 3 is executed within a PHP loop

The following code uploads multiple images no problem. However, I'm trying to get it to update a field in a table based on what iteration the loop is in. PROBLEM: The IF Statement seems to not work when looped. I.e. it only adds the first file_name to the database.
Anyone see what I'm doing wrong here? Much appreciated if so!!!
for ($i = 1; $i < 4; $i++)
{
/* Handle the file upload */
$upload = $this->upload->do_upload('image' . $i);
/* File failed to upload - continue */
if ($upload === FALSE)
continue;
/* Get the data about the file */
$data = $this->upload->data();
$uploadedFiles[$i] = $data;
if ($i == 1)
{
$filenames1 = array(
'product_image_front' => $data['file_name'],
);
$this->db->where('id', $this->db->insert_id());
$this->db->update('products', $filenames1);
}
if ($i == 2)
{
$filenames2 = array(
'product_image_back' => $data['file_name'],
);
$this->db->where('id', $this->db->insert_id());
$this->db->update('products', $filenames2);
}
if ($i == 3)
{
$filenames3 = array(
'product_image_back' => $data['file_name'],
);
$this->db->where('id', $this->db->insert_id());
$this->db->update('products', $filenames3);
}
}
insert_id - Get the ID generated in the last query.
Store it in a variable before the loop.

Categories