Importing large CSV files - php

I'm looking at importing a CSV file, but this file is rather large.
What I'm looking to do, is 2 things:
1) Scan the CSV to validate values in particular fields
2) Once the file is valid, import
The CSV data can ONLY be inserted if the file is valid (All or nothing)
The problem is, I'm looping through twice, 1st to check the CSV fields are valid and then another for loop to save.
The problem is memory. I run out of memory (file is 100,000 rows with 45 fields)
Is there an easier way to do this and reduce memory? I'm using an AR implementation, would using PDO be quicker?
Thanks
EDIT:
$data = array();
// open the file and loop through
if( ($handle = fopen('details.csv', "r")) !== FALSE) {
$rowCounter = 0;
while (($rowData = fgetcsv($handle, 0, ",")) !== FALSE) {
if( 0 === $rowCounter) {
$headerRecord = $rowData;
} else {
foreach( $rowData as $key => $value) {
$data[ $rowCounter - 1][$headerRecord[ $key] ] = $value;
}
}
$rowCounter++;
}
fclose($handle);
}
$errors = array();
// loop to check through the fields for validation
for($i=0;$i<count($data);$i++) {
$row++;
if(!valid_email($data[$i]['EMAIL']))) {
$errors[] = 'Invalid Email Address';
break;
}
}
if(empty($errors)) {
for($j=0;$j<count($assocData);$j++) {
$row++;
$details = new Details();
// set the fields here
$details->email = $data[$j]['EMAIL'];
$details->save();
unset($details);
}
}

You're already looping through the data in your first foreach. Why don't you validate the fields inside that loop, if validation passes add to an array to save and only save (in a single transaction) if the loop completes.

Related

Value's not visible when importing CSV after values are added by fputcsv with php

I'm trying to make a translation module for my web project and i hit a wall. I written two functions, one for importing the CSV data and the other for changing, deleting and adding new data to the CSV file.
Everything works like a charm except the adding part. It adds the data to the CSV file but when i want to import it in the website via PHP it doesn't display the added values. (It see that their should be values but it gives empty results in return) the function i use for reading the csv file is:
// Load in CSV (File name, delimiter, Fixed array[key,value] (optional))
function load__CSV($filename='', $delimiter=';', $fixed = null){
if(!file_exists($filename) || !is_readable($filename)) {
return FALSE;
}
$header = NULL;
$data = array();
if (($handle = fopen($filename, 'r')) !== FALSE) {
while (($row = fgetcsv($handle, 100000, $delimiter)) !== FALSE) {
if (!$header) {
$header = $row;
} else {
$data[] = array_combine($header, $row);
}
}
fclose($handle);
}
if($fixed != null) {
foreach($data as $entry){
$set_csv[$entry[''.$fixed[0].'']] = $entry[''.$fixed[1].''];
}
} else {
$set_csv = $data;
}
return $set_csv;
}
The function i use to add, edit or remove CSV content is:
// Change csv
function update__csv($csv = 'csv/language.csv',$key = array(2,''),$values = array([3,''],[4,'']),$status = 'change') {
$input = fopen(BASE_URL.$csv, 'r');
$output = fopen(BASE_URL.'csv/temporary.csv', 'w');
while (false !== ($data = fgetcsv($input, 10000, ";"))) {
if ($data[$key[0]] == $key[1]) {
foreach ($values as $value) {
$data[$value[0]] = $value[1];
}
if ($status == 'change' || $status == 'new') {
fputcsv($output, $data, ";");
}
} else {
fputcsv($output, $data, ";");
}
}
if($status == 'new'){
fputcsv($output, $values, ";");
}
fclose( $input );
fclose( $output );
unlink(BASE_URL . $csv);
rename(BASE_URL . 'csv/temporary.csv', BASE_URL . $csv);
}
If i add new values to the CSV file and then open the CSV on my PC and safe it again (without changing a thing) to CSV with UTF-8 encoding then it works as expected and loads the correct data. If i open the CSV in Notepad++ i see this difference between manual added items and php added items:
I tried other encoding methods but that is kinda new for me so i think i did something wrong. Thank you in advance for helping me!
I did find the answer after some more debugging. One value of the array didn't contain a string but a true or false statement. This meant that it didn't add the data the right way into the CSV file.
I fixed it by adding strval() to every variable before putting it into an array.

detect error insert object in massive insert doctrine symfony

I am using Doctrine for an application that the main task should import data from an CSV file into database. there are at least 50.000 entries.
The code work perfect except one thing, detecting the row not inserted in the database.
if (($handle = fopen($rootPath, 'r')) !== FALSE) {
set_time_limit(0);
$tempObjets=array();
while (($row = fgetcsv($handle, 4096, ';')) !== FALSE) {
$iteration++;
$contactt = new Contact();
$option = array();
$optiondata = array();
$datacon = new Datacontact();
//precessing row data (read row and put the data into contact
$datacon->setContact($contactt);
$em->persist($datacon);
$tempObjets[] = $datacon;
if ($iteration % 1000 == 0) {
try{
$em->flush();
$em->clear();
foreach ($tempObjets as $tempObject) {
$em->detach($tempObject);
}
$tempObjets = null;
gc_enable();
gc_collect_cycles();
}catch (\Doctrine\ORM\ORMException $e )
{
$this->get('session')->getFlashBag()->add('error', 'error');
$this->get('logger')->error($e->getMessage());
}
suppose that I the file csv containt 50000 entries, but just 45000 entries have been inserted, and 5000 entries contains error and I should put them in an array variables called for example errorsdata() that contain all error object.
How I can detect if a row has been inserted or not and how I add it to errorsdata variable given that script flush the data every 1000 iteration.
thanks for help

PHP script for adding prefix to sku when duplicate in .csv

At the moment I have a script that will remove the row in my csv when it has already seen the sku before.
Here is the script:
<?php
// array to hold all unique lines
$lines = array();
// array to hold all unique SKU codes
$skus = array();
// index of the `sku` column
$skuIndex = -1;
// open the "save-file"
if (($saveHandle = fopen("unique.csv", "w")) !== false) {
// open the csv file
if (($readHandle = fopen("original.csv", "r")) !== false) {
// read each line into an array
while (($data = fgetcsv($readHandle, 8192, ",")) !== false) {
if ($skuIndex == -1) {
// we need to determine what column the "sku" is; this will identify
// the "unique" rows
foreach ($data as $index => $column) {
if ($column == 'sku') {
$skuIndex = $index;
break;
}
}
if ($skuIndex == -1) {
echo "Couldn't determine the SKU-column.";
die();
}
// write this line to the file
fputcsv($saveHandle, $data);
}
// if the sku has been seen, skip it
if (isset($skus[$data[$skuIndex]])) continue;
$skus[$data[$skuIndex]] = true;
// write this line to the file
fputcsv($saveHandle, $data);
}
fclose($readHandle);
}
fclose($saveHandle);
}
?>
This works fine but I am starting to need the content that is deleted.
What i need now, is to modify the code to add the same prefix to all duplicate sku's as there will only be 2 of the same sku.
I do not know where to start.
Adding a prefix to duplicates
This will add a prefix to any duplicate SKU and will then store it into the unique CSV output, e.g. XYZ123 becomes duplicate-XYZ123.
Change:
if (isset($skus[$data[$skuIndex]])) continue;
to:
if (isset($skus[$data[$skuIndex]])) $data[$skuIndex] = 'duplicate-' . $data[$skuIndex];
Fixing the duplicate header row
Add continue; after fputcsv($saveHandle, $data); Inside if ($skuIndex == -1) {. Because fputcsv(...) appears twice in the loop, it will be run twice on the first iteration of the loop.

How do I replace 1 value within a row in a CSV file using php?

So this is my very simple and basic account system (Just a school project), I would like the users to be able to change their password. But I am unsure on how to just replace the Password value within a row keeping all the other values the same.
CSV File:
ID,Username,Email,DateJoined,Password,UserScore,profilePics
1,Test,Test#Test.com,03/12/2014,Test,10,uploads/profilePics/Test.jpg
2,Alfie,Alfie#test.com,05/12/2014,1234,136,uploads/profilePics/Alfie.png
PHP:
("cNewPassword" = confirm new password)
<?php
session_start();
if(empty($_POST['oldPassword']) || empty($_POST['newPassword']) || empty($_POST['cNewPassword'])) {
die("ERROR|Please fill out all of the fields.");
} else {
if($_POST['newPassword'] == $_POST['cNewPassword']) {
if ($_POST['oldPassword'] == $_SESSION['password']) {
$file = "Database/Users.csv";
$fh = fopen($file, "w");
while(! feof($file)) {
$rows = fgetcsv($file);
if ($rows[4] == $_POST['oldPassword'] && $rows[1] == $_SESSION['username']) {
//Replace line here
echo("SUCCESS|Password changed!");
}
}
fclose($file);
}
die("ERROR|Your current password is not correct!");
}
die("ERROR|New passwords do not match!");
}
?>
You'll have to open file in read mode, open a temporary one in write mode, write there modified data, and then delete/rename files. I'd suggest trying to set up a real DB and work using it but if you're going for the csv, the code should look like more or less like this:
$input = fopen('Database/Users.csv', 'r'); //open for reading
$output = fopen('Database/temporary.csv', 'w'); //open for writing
while( false !== ( $data = fgetcsv($input) ) ){ //read each line as an array
//modify data here
if ($data[4] == $_POST['oldPassword'] && $data[1] == $_SESSION['username']) {
//Replace line here
$data[4] = $_POST['newPassword'];
echo("SUCCESS|Password changed!");
}
//write modified data to new file
fputcsv( $output, $data);
}
//close both files
fclose( $input );
fclose( $output );
//clean up
unlink('Database/Users.csv');// Delete obsolete BD
rename('Database/temporary.csv', 'Database/Users.csv'); //Rename temporary to new
Hope it helps.
My suggestion is a little function of mine which will turn your database data into an array which you can modify and then return to original state:
With this set of function, you simply have to precise how each row/row data are separated.
function dbToArray($db, $row_separator = "\n", $data_separator = ",") {
// Let's seperator each row of data.
$separate = explode($row_separator, $db);
// First line is always the table column name:
$table_columns =
$table_rows = array();
foreach ($separate as $key => $row) {
// Now let's get each column data out.
$data = explode($data_separator, $row);
// I always assume the first $row of data contains the column names.
if ($key == 0)
$table_columns = $data;
else if ($key > 0 && count($table_columns) == count($data)) // Let's just make sure column amount matches.
$table_rows[] = array_combine($table_columns, $data);
}
// Return an array with columns, and rows; each row data is bound with it's equivalent column name.
return array(
'columns' => $table_columns,
'rows' => $table_rows,
);
}
function arrayToDb(array $db, $row_separator = "\n", $data_separator = ",") {
// A valid db array must contain a columns and rows keys.
if (isset($db['columns']) && isset($db['rows'])) {
// Let's now make sure it contains an array. (This might too exagerated of me to check that)
$db['columns'] = (array) $db['columns'];
$db['rows'] = (array) $db['rows'];
// Now let's rewrite the db string imploding columns:
$returnDB = implode($data_separator, $db['columns']).$row_separator;
foreach ($db['rows'] as $row) {
// And imploding each row data.
$returnDB .= implode($data_separator, $row).$row_separator;
}
// Retunr the data.
return $returnDB;
}
// Faaaaaaaaaaaail !
return FALSE;
}
Let's just point out I tried these with your db example, and it works even when tested on it's own results such as : dbToArray(arrayToDb(dbToArray())) multiple times.
Hope that help. If I can be clearer don't hesitate. :)
Cheers,
You need a 3 step process to do this (create 3 loops, could be optimized to 1 or 2 loops):
Load the relevant data to memory
Update the desired data
Save the data to the file
Good luck! :)
PS. Also your passwords should never been stored in clear text, wether in memory(session) or on disk(csv), use a hasing function!

Importing csv into mysql (List of Codes)

So I have a list of codes (100,000) to be exact and I got them delivered in a csv file.
I want to put theses codes in a database so I can later get them using my php script.
However my question is how do I get my codes from the file in the right table?
This is what my database looks like at the moment.
The codes from the file need to be inserted in table code.
Hope someone can help me out with this one.
You probably will find this helpful (pls adjust the table and others names):
LOAD DATA INFILE 'codes.csv'
INTO TABLE codes
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
(#id,code,#active)
SET id = null;
SET active = '1'
More details:
http://www.mysqltutorial.org/import-csv-file-mysql-table/
Try this:
you can upload your csv file and post it. then use this function.
this is the PHP code:
function get_csv_file($filename){
if (($handle = fopen($filename, "r")) !== false) {
$filesize = filesize($filename);
$firstRow = true;
$aData = array();
while (($data = fgetcsv($handle, $filesize, ";")) !== false) {
if($firstRow) {
$aData = $data;
$firstRow = false;
} else {
for($i = 0;$i < count($data); $i++) { //only for csv file
if($data[$i] != '')
$aData[] = $data[$i];
}
}
}
//print_r($aData);
fclose($handle);
$finalscrap = array_unique($aData);
// echo "<pre>";
// print_r($finalscrap);
// echo "<pre/>";
return $finalscrap; //this is your record as array format
}
}

Categories