I am using Doctrine for an application that the main task should import data from an CSV file into database. there are at least 50.000 entries.
The code work perfect except one thing, detecting the row not inserted in the database.
if (($handle = fopen($rootPath, 'r')) !== FALSE) {
set_time_limit(0);
$tempObjets=array();
while (($row = fgetcsv($handle, 4096, ';')) !== FALSE) {
$iteration++;
$contactt = new Contact();
$option = array();
$optiondata = array();
$datacon = new Datacontact();
//precessing row data (read row and put the data into contact
$datacon->setContact($contactt);
$em->persist($datacon);
$tempObjets[] = $datacon;
if ($iteration % 1000 == 0) {
try{
$em->flush();
$em->clear();
foreach ($tempObjets as $tempObject) {
$em->detach($tempObject);
}
$tempObjets = null;
gc_enable();
gc_collect_cycles();
}catch (\Doctrine\ORM\ORMException $e )
{
$this->get('session')->getFlashBag()->add('error', 'error');
$this->get('logger')->error($e->getMessage());
}
suppose that I the file csv containt 50000 entries, but just 45000 entries have been inserted, and 5000 entries contains error and I should put them in an array variables called for example errorsdata() that contain all error object.
How I can detect if a row has been inserted or not and how I add it to errorsdata variable given that script flush the data every 1000 iteration.
thanks for help
Related
I am working on a college project that must use a CSV file to create the values for a table. It must have a class and it must have specific class variables (one of them I have already created for testing). Only the variables are specified as having to be part of the class but I think it would be better to pack the function inside the class (unless someone has a different opinion).
Basically, I want to use the class function to create the values for the class variables. Here is my script (refer to the comments for explanation):
<?php
class productRow {
function make_table_row($i) {
$row = 1;
$tablerows = [];
if (($input = fopen("input.csv", "r")) !== FALSE) {
while (($tabledata = fgetcsv($input, 1000, ",")) !== FALSE) { //cycles through the rows of data creating arrays
if ($row == 1) {
$row++;
continue; //skips the first row because it's a header row I don't want on my input
}
$tablerows[] = $tabledata; //uses the roles to populate a multidimensional array
$row++;
}
fclose($input);
return $tablerows[$i]; //uses the $i argument to return one of the row arrays
}
}
var $itemNumber = this->make_table_row($i)[0]; //Line 118: calls make_table_row function to get the first item from the row returned
}
$pr1 = new productRow;
echo $pr1->make_table_row(1); //calls the function from within the class
?>
I get this error: Fatal error: Constant expression contains invalid operations in C:\xampp\htdocs\Tylersite\index.php on line 118
I know the return works because I tested it with print_r, including adding an array number as I did with that variable value to get a specific array value. So I must not be calling that function instance correctly. I've tried various things including removing the $this keyword as I wasn't sure I needed it, but the truth is I don't really know how to do it and I'm having trouble finding documentation on the correct syntax. Does anyone know what to do?
Example using constructor function
class productRow {
var $itemNumber = []; // default value doesn't matter as it will change in __construct
public function __construct($i) {
$this->itemNumber = $this->make_table_row($i)[0];
}
function make_table_row($i) {
$row = 1;
$tablerows = [];
if (($input = fopen("input.csv", "r")) === FALSE) {
// return early - reduces indentation
return;
}
while (($tabledata = fgetcsv($input, 1000, ",")) !== FALSE) { //cycles through the rows of data creating arrays
if ($row == 1) {
$row++;
continue; //skips the first row because it's a header row I don't want on my input
}
$tablerows[] = $tabledata; //uses the roles to populate a multidimensional array
$row++;
}
fclose($input);
return $tablerows[$i]; //uses the $i argument to return one of the row arrays
}
}
$pr1 = new productRow(1);
var_dump($pr1->make_table_row(1));
At the moment I have a script that will remove the row in my csv when it has already seen the sku before.
Here is the script:
<?php
// array to hold all unique lines
$lines = array();
// array to hold all unique SKU codes
$skus = array();
// index of the `sku` column
$skuIndex = -1;
// open the "save-file"
if (($saveHandle = fopen("unique.csv", "w")) !== false) {
// open the csv file
if (($readHandle = fopen("original.csv", "r")) !== false) {
// read each line into an array
while (($data = fgetcsv($readHandle, 8192, ",")) !== false) {
if ($skuIndex == -1) {
// we need to determine what column the "sku" is; this will identify
// the "unique" rows
foreach ($data as $index => $column) {
if ($column == 'sku') {
$skuIndex = $index;
break;
}
}
if ($skuIndex == -1) {
echo "Couldn't determine the SKU-column.";
die();
}
// write this line to the file
fputcsv($saveHandle, $data);
}
// if the sku has been seen, skip it
if (isset($skus[$data[$skuIndex]])) continue;
$skus[$data[$skuIndex]] = true;
// write this line to the file
fputcsv($saveHandle, $data);
}
fclose($readHandle);
}
fclose($saveHandle);
}
?>
This works fine but I am starting to need the content that is deleted.
What i need now, is to modify the code to add the same prefix to all duplicate sku's as there will only be 2 of the same sku.
I do not know where to start.
Adding a prefix to duplicates
This will add a prefix to any duplicate SKU and will then store it into the unique CSV output, e.g. XYZ123 becomes duplicate-XYZ123.
Change:
if (isset($skus[$data[$skuIndex]])) continue;
to:
if (isset($skus[$data[$skuIndex]])) $data[$skuIndex] = 'duplicate-' . $data[$skuIndex];
Fixing the duplicate header row
Add continue; after fputcsv($saveHandle, $data); Inside if ($skuIndex == -1) {. Because fputcsv(...) appears twice in the loop, it will be run twice on the first iteration of the loop.
So this is my very simple and basic account system (Just a school project), I would like the users to be able to change their password. But I am unsure on how to just replace the Password value within a row keeping all the other values the same.
CSV File:
ID,Username,Email,DateJoined,Password,UserScore,profilePics
1,Test,Test#Test.com,03/12/2014,Test,10,uploads/profilePics/Test.jpg
2,Alfie,Alfie#test.com,05/12/2014,1234,136,uploads/profilePics/Alfie.png
PHP:
("cNewPassword" = confirm new password)
<?php
session_start();
if(empty($_POST['oldPassword']) || empty($_POST['newPassword']) || empty($_POST['cNewPassword'])) {
die("ERROR|Please fill out all of the fields.");
} else {
if($_POST['newPassword'] == $_POST['cNewPassword']) {
if ($_POST['oldPassword'] == $_SESSION['password']) {
$file = "Database/Users.csv";
$fh = fopen($file, "w");
while(! feof($file)) {
$rows = fgetcsv($file);
if ($rows[4] == $_POST['oldPassword'] && $rows[1] == $_SESSION['username']) {
//Replace line here
echo("SUCCESS|Password changed!");
}
}
fclose($file);
}
die("ERROR|Your current password is not correct!");
}
die("ERROR|New passwords do not match!");
}
?>
You'll have to open file in read mode, open a temporary one in write mode, write there modified data, and then delete/rename files. I'd suggest trying to set up a real DB and work using it but if you're going for the csv, the code should look like more or less like this:
$input = fopen('Database/Users.csv', 'r'); //open for reading
$output = fopen('Database/temporary.csv', 'w'); //open for writing
while( false !== ( $data = fgetcsv($input) ) ){ //read each line as an array
//modify data here
if ($data[4] == $_POST['oldPassword'] && $data[1] == $_SESSION['username']) {
//Replace line here
$data[4] = $_POST['newPassword'];
echo("SUCCESS|Password changed!");
}
//write modified data to new file
fputcsv( $output, $data);
}
//close both files
fclose( $input );
fclose( $output );
//clean up
unlink('Database/Users.csv');// Delete obsolete BD
rename('Database/temporary.csv', 'Database/Users.csv'); //Rename temporary to new
Hope it helps.
My suggestion is a little function of mine which will turn your database data into an array which you can modify and then return to original state:
With this set of function, you simply have to precise how each row/row data are separated.
function dbToArray($db, $row_separator = "\n", $data_separator = ",") {
// Let's seperator each row of data.
$separate = explode($row_separator, $db);
// First line is always the table column name:
$table_columns =
$table_rows = array();
foreach ($separate as $key => $row) {
// Now let's get each column data out.
$data = explode($data_separator, $row);
// I always assume the first $row of data contains the column names.
if ($key == 0)
$table_columns = $data;
else if ($key > 0 && count($table_columns) == count($data)) // Let's just make sure column amount matches.
$table_rows[] = array_combine($table_columns, $data);
}
// Return an array with columns, and rows; each row data is bound with it's equivalent column name.
return array(
'columns' => $table_columns,
'rows' => $table_rows,
);
}
function arrayToDb(array $db, $row_separator = "\n", $data_separator = ",") {
// A valid db array must contain a columns and rows keys.
if (isset($db['columns']) && isset($db['rows'])) {
// Let's now make sure it contains an array. (This might too exagerated of me to check that)
$db['columns'] = (array) $db['columns'];
$db['rows'] = (array) $db['rows'];
// Now let's rewrite the db string imploding columns:
$returnDB = implode($data_separator, $db['columns']).$row_separator;
foreach ($db['rows'] as $row) {
// And imploding each row data.
$returnDB .= implode($data_separator, $row).$row_separator;
}
// Retunr the data.
return $returnDB;
}
// Faaaaaaaaaaaail !
return FALSE;
}
Let's just point out I tried these with your db example, and it works even when tested on it's own results such as : dbToArray(arrayToDb(dbToArray())) multiple times.
Hope that help. If I can be clearer don't hesitate. :)
Cheers,
You need a 3 step process to do this (create 3 loops, could be optimized to 1 or 2 loops):
Load the relevant data to memory
Update the desired data
Save the data to the file
Good luck! :)
PS. Also your passwords should never been stored in clear text, wether in memory(session) or on disk(csv), use a hasing function!
I'm looking at importing a CSV file, but this file is rather large.
What I'm looking to do, is 2 things:
1) Scan the CSV to validate values in particular fields
2) Once the file is valid, import
The CSV data can ONLY be inserted if the file is valid (All or nothing)
The problem is, I'm looping through twice, 1st to check the CSV fields are valid and then another for loop to save.
The problem is memory. I run out of memory (file is 100,000 rows with 45 fields)
Is there an easier way to do this and reduce memory? I'm using an AR implementation, would using PDO be quicker?
Thanks
EDIT:
$data = array();
// open the file and loop through
if( ($handle = fopen('details.csv', "r")) !== FALSE) {
$rowCounter = 0;
while (($rowData = fgetcsv($handle, 0, ",")) !== FALSE) {
if( 0 === $rowCounter) {
$headerRecord = $rowData;
} else {
foreach( $rowData as $key => $value) {
$data[ $rowCounter - 1][$headerRecord[ $key] ] = $value;
}
}
$rowCounter++;
}
fclose($handle);
}
$errors = array();
// loop to check through the fields for validation
for($i=0;$i<count($data);$i++) {
$row++;
if(!valid_email($data[$i]['EMAIL']))) {
$errors[] = 'Invalid Email Address';
break;
}
}
if(empty($errors)) {
for($j=0;$j<count($assocData);$j++) {
$row++;
$details = new Details();
// set the fields here
$details->email = $data[$j]['EMAIL'];
$details->save();
unset($details);
}
}
You're already looping through the data in your first foreach. Why don't you validate the fields inside that loop, if validation passes add to an array to save and only save (in a single transaction) if the loop completes.
i'm with some troubles figuring out how to read the CSV file that I upload, probably i'm missing something in my Controller code.
public function actionImport() {
$model = new Produtos;
$this->render('import', array('model' => $model) );
if( isset($_FILES['csv_file']) ) {
$handle = fopen($_FILES['csv_file']['tmp_name'], 'r');
if ($handle) {
while( ($line = fgetcsv($handle, 1000, ";")) != FALSE) {
$model->codigo = $line[0];
$model->nome = $line[1];
$model->descricao = $line[2];
$model->stock = $line[3];
$model->data_reposicao = $line[4];
$model->save();
}
}
fclose($handle);
}
}
This is only saving me the last line in the CSV... please some help!
Any help will be really appreciated.
Thank you
I think You missed the name of file, try $_FILES['csv_file']['tmp_name']
http://php.net/manual/en/features.file-upload.post-method.php for reference.
Also Yii does provide file handling, check http://www.yiiframework.com/doc/api/1.1/CUploadedFile for reference
Don't forget to check whether your data is successfully validated or not.
Insert this code between yours:
$model->data_reposicao = $line[4];
if (!$model->validate())
throw new Exception("Validation failed.");
$model->save();
So you can see what's going wrong.
$_FILES is an array, which contains ['element_name']-array. In your case $_FILES is an array of $_FILES['csv_file']['name'], $_FILES['csv_file']['type'], $_FILES['csv_file']['error'], $_FILES['csv_file']['size'] and $_FILES['csv_file']['tmp_name'].
So for short; $_FILES['csv_file'] is an array.
you are saving the same model instance again and again... that is why only the last line gets saved... you will have to create a new model for every line, i.e. in your while loop, add $model = new Produtos;
You need to initialise your model object every time when you need to insert your row into table. The code should be like:
public function actionImport() {
$model = new Produtos;
$this->render('import', array('model' => $model) );
if( isset($_FILES['csv_file']) ) {
$handle = fopen($_FILES['csv_file']['tmp_name'], 'r');
if ($handle) {
while( ($line = fgetcsv($handle, 1000, ";")) != FALSE) {
$modeln = new Produtos;
$modeln->codigo = $line[0];
$modeln->nome = $line[1];
$modeln->descricao = $line[2];
$modeln->stock = $line[3];
$modeln->data_reposicao = $line[4];
$modeln->save();
}
}
fclose($handle);
}
}
I know this is an old post, but I just came across this and just wanted to help someone who might be having the same issue (b/w issue is not with CSV or file upload)
Issue is with how Yii handles saving
You need to set isNewRecord attribute to true and primary key to NULL of model object before saving to save a new record everytime.
$model->PRIMARYKEYCOLUMN = NULL; //Replace PRIMARYKEYCOLUMN with the name of column
$model->isNewRecord = true;
$model->save();
You need to do above step whenever you are saving rows in loop.
$handle = fopen($_FILES['Userimportcsv']['tmp_name']['csv_file'], 'r');
if($handle) {
$row = 1;
while( ($line = fgetcsv($handle, 1000, ",")) != FALSE) {
if($row>1) {
$newModel = new Countries;
$newModel->countryName = $line[0];
$newModel->status = $line[1];
$newModel->save();
}
$row++;
}
}
fclose($handle);