I am developing an application which has to read large CSV file and process data. It will be definitely not possible to make it in one request because processing the data also takes time, it is not just about reading.
So what I tried so far and what has been working well so far is the following:
// Open file
$handle = fopen($file, 'r');
// Move pointer to a place where it stopped last time
fseek($handle, $offset);
// Read limited line and process
for ($i = 0; $i < $limit; $i++) {
// Get length of line for offset purposes
$newlength = strlen(fgets($handle));
// Move pointer back. fgets moves pointer so we move it back for fgetcsv to get that line again
fseek($handle, $offset);
$line = fgetcsv($handle, 0, $csv_delimiter);
// Process data here
// Save offset
$offset += $newlength;
}
So the problem is here on this line:
$newlength = strlen(fgets($handle));
It fails when csv column has line breaks.
I also tried $newlength = strlen(implode(';', fgetcsv($handle, 0, $csv_delimiter))); but this does not always work. It usually fails for few characters. Probably quotations and end of line is not handled properly here.
All I need is to get length of csv line, not just single line, but csv line which might have line breaks within quotes.
Anybody has better solution?
do one thing, create one mysql temporary table named "my_csv_data", and add one field in that table with all fields which are in csv file and extra add one "is_processed" with enum(0,1) default value '0'.
now import your all csv data in that sql table. it will never take more time for single insert.
now cerate one function/file which access my_csv_data table 10 or 100 records where is_processed='0' and process it and if process done successfully then update "is_processed" field to '1'.
now create one cronjob which hit that file/function. periodically.
using this way data will going to silently insert in your table without disturb/suffer any admin/front end user.
i have codeignitor code where i uploading the csv file data and insert it into mysql database. hope this will help you
if($_FILES["file"]["size"] > 0)
{
$file = fopen($filename, "r");
while (($emapData = fgetcsv($file, 10000, ",")) !== FALSE)
{
$data = array(
'reedumption_code' => $emapData[0],
'jb_note_id' =>$jbmoney_id,
'jbmoney' =>$jbamount,
'add_date'=>time(),
'modify_date'=>time(),
'user_id'=>0,
'status'=>1,
'assign_date'=>0,
'del_status'=>1,
'store_status'=>1
);
$this->load->model('currency_model');
$insertId = $this->currency_model->insertCSV($data);
}
fclose($file);
redirect('currency/add_currency?msg=Data Imported Successfully');
}
Related
I need help processing files holding about 46k lines or more than 30MB of data.
My original idea was to open the file and turn each line into an array element. This worked the first time as the array held about 32k values total.
The second time, the process was repeated, the array only held 1011 elements, and finally, the third time it could only hold 100.
I'm confused and don't know much about the backend array processes. Can someone explain what is happening and fix the code?
function file_to_array($cvsFile){
$handle = fopen($cvsFile, "r");
$path = fread($handle, filesize($cvsFile));
fclose($handle);
//Turn the file into an array and separate lines to elements
$csv = explode(",", $path);
//Remove common double spaces
foreach ($csv as $key => $line){
$csv[$key] = str_replace(' ', '', str_getcsv($line));
}
array_filter($csv);
//get the row count for the file and array
$rows = count($csv);
$filerows = count(file($cvsFile)); //this no longer works
echo "File has $filerows and array has $rows";
return $csv;
}
The approach here can be split in 2.
Optimized file reading and processing
Proper storage solution
Optimized file processing can be done like so:
$handle = fopen($cvsFile, "r");
$rowsSucceed = 0;
$rowsFailed = 0;
if ($handle) {
while (($line = fgets($handle)) !== false) { // Reading file by line
// Process CSV line and check if it was parsed correctly
// And count as you go
if (!empty($parsedLine)) {
$csv[$key] = ... ;
$rowsSucceed++;
} else {
$rowsFailed++;
}
}
fclose($handle);
} else {
// Error handling
}
$totalLines = $rowsSucceed + $rowsFailed;
Also you can avoid array_filter() simply by not adding processed line if its empty.
It will allow to optimize memory usage during script execution.
Proper storage
Proper storage here is needed for performing operations on certain amount of data. File reading are ineffective and expensive. Using simple file based database like sqlite can help you a lot and increase overall performance of your script.
For this purpose you probably should process your CSV directly to database and than perform count operation on parsed data avoiding excessive file line counts etc.
Also it gives you further advantage on working with data not keeping it all in memory.
Your question says you want to "turn each line into an array element" but that is definitely not what you are doing. The code is quite clear; it reads the entire file into $path and then uses explode() to make one massive flat array of every element on every line. Then later you're trying to run str_getcsv() on each item, which of course isn't going to work; you've already exploded all the commas away.
Looping over the file using fgetcsv() makes more sense:
function file_to_array($cvsFile) {
$filerows = 0;
$handle = fopen($cvsFile, "r");
while ($line = fgetcsv($handle)) {
$filerows++;
// skip empty lines
if ($line[0] === null) {
continue;
}
//Remove common double spaces
$csv[] = str_replace(' ', '', $line);
}
//get the row count for the file and array
$rows = count($csv);
echo "File has $filerows and array has $rows";
fclose($handle);
return $csv;
}
Is there an effective way to update/delete specific row in CSV file? Every other method included reading contents of entire file, creating temporary file and then replacing old file with it, etc...
But let's say, I have big CSV with 10000 records, so this kind of solution would be rather resource-heavy.
Let's say, I am unable to use database, so writing to file is the only way of storing data.
So, the question is, what would be the most effective way to do it?
Thank you in advance!
You're going to have to read the entire file. Sorry, no way around that. A CSV is a single, flat, text file with randomly sized fields and rows.
You definitely shouldn't be working directly with a CSV for database operations. You ought to pull the data into a database to work with it, then output it back to CSV when you're done.
You don't mention why you can't use a database, so I'm going to guess it's a resource issue, and you also don't say why you don't want to rewrite the file, so I'm going to guess it's due to performance. You could cache a number of operations and perform them all at once, but you're not going to get away from rewriting all or at least some portion of the file.
Consider reading the csv line by line into a multi-dimensional array, and at a certain row make your changes. Then, export array data out to csv. Below example modifies the 100th row assuming a 6-column comma delimited csv file (0-5).
Now, if you want to delete the row, then exclude it from $newdata array by conditionally skipping to next loop iteration with continue. Alternatively, if you want to update, simple set current inner array $newdata[$i] to new values:
$i = 0;
$newdata = [];
$handle = fopen("OldFile.csv", "r");
// READ CSV
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
// UPDATE 100TH ROW DATA (TO EXCLUDE, KEEP ONLY $i++ AND continue)
if ($i == 99) {
$newdata[$i][] = somenewvalue;
$newdata[$i][] = somenewvalue;
$newdata[$i][] = somenewvalue;
$newdata[$i][] = somenewvalue;
$newdata[$i][] = somenewvalue;
$newdata[$i][] = somenewvalue;
$i++;
continue;
}
$newdata[$i][] = $data[0];
$newdata[$i][] = $data[1];
$newdata[$i][] = $data[2];
$newdata[$i][] = $data[3];
$newdata[$i][] = $data[4];
$newdata[$i][] = $data[5];
$i++;
}
// EXPORT CSV
$fp = fopen('NewFile.csv', 'w');
foreach ($newdata as $rows) {
fputcsv($fp, $rows);
}
fclose($fp);
Break the CSV into multiple files all in one directory.
That way you still have to rewrite files, but you don't have to rewrite nearly as much.
Bit late but for people who may search same thing, you could put your csv into an sqlite what addionaly gives you the ability to search in the dataset. There is some sample code: Import CSV File into a SQLite Database via PHP
I have a CSV that is downloaded from the wholesaler everynight with updated prices.
What I need to do is edit the price column (2nd column) and multiply the current value by 1.3 (30%).
My code to read the provided CSV and take just the columns I need is below, however I can't seem to figure out how to edit the price column.
<?php
// open the csv file in write mode
$fp = fopen('var/import/tb_prices.csv', 'w');
// read csv file
if (($handle = fopen("var/import/Cbl_4036_2408.csv", "r")) !== FALSE) {
$targetColumns = array(1, 2, 3); // get data from the 1st, 4th and 15th column
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$targetData = array(); // array that hold target data
foreach($targetColumns as $column){ // loop throught the targeted columns array
if($column[2]){
$data[$column] = $data[0] * 1.3;
}
$targetData[] = $data[$column]; // get the data from the column
}
# Populate the multidimensional array.
$csvarray[$nn] = $targetData; // add target data to csvarray
// write csv file
fputcsv($fp, $targetData);
}
fclose($handle);
fclose($fp);
echo "CSV File Written Successfully!";
}
?>
Could somebody point me in the right direction please, explaining how you've worked out the function too so I can learn at the same time.
You are multiplying your price column always as - $data[0] * 1.3.
It may be wrong here.
Other views:
If you are doing it once in a lifetime of this data(csv) handling, try to solve it using mysql itself only. Create the table similar to the database, import the .csv data into that mysql table. And then, SQL operate as you want.
No loops; no coding, no file read/write, and precise control over what you want to do with UPDATE. You just need to be aware of the delimiters (line separators eg. \r\n, column separators (eg. comma or tab or semicolon) and data encoding in double/single-quotes or not)
Once you modify your data, you can export it back to csv again.
If you want to handle the .csv file itself, open it in one connection (read only mode), and write to another file - saving the original data.
you say that the column that contains the price is the second but then use that index with zero. anyway the whole thing can be easier
$handle = fopen("test.csv", "r");
if ( $handle !== FALSE) {
$out = "";
while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
$data[1] = ((float)$data[1] * 1.3);
$out .= implode(";",$data) . "\n";
}
fclose($handle);
file_put_contents("test2.csv", $out);
}
this code open a csv file with comma as separator.
than read every line and for every line it's multiplies the second coloumn (index 1) for 1.3
this line
$out .= implode(";",$data) . "\n";
generate a line for new csb file. see implode on the officile documentation ...
after I close the connection to the file. and 'useless to have a connection with two files when you can do the writing of the second file in one fell swoop. the thing is true for small files
Simply trying to use the url query string as a lookup code in an csv table. This works 9/10 times, but on occasion this will return the wrong line (usually a few lines below what should be the correct line).
csv looks something like this
taskcode1, info1, info1, info1
taskcode2, info2, info2, info2
taskcode3, info3, info3, info3
The problem is that sometimes (around 1/10 times so far), a given url query of taskcode1 will actually return line info3.
This csv file is being read concurrently by more than one user. Could the problem be stemming from simultaneously reading? I know there can be issues for writing, and a flock on the file may be necessary. Here's the actual code in my php script. Thank you for any advice.
Notice that as soon as the task code is found, $this_taskcode == $taskcode, I break the while loop.
//get query from request and look up task configuration
$csv_file_path = "tasks.csv";
$taskcode = $_SERVER['QUERY_STRING'];
//open csv file and find taskcode
$fid = fopen($csv_file_path, 'r');
//loop through each line of csv until taskcode is found, then save the whole line as $hit
while (($line = fgetcsv($fid)) !== FALSE){
$this_taskcode = $line[0];
if ($this_taskcode == $taskcode){
$hit = $line;
break;
};
}
fclose($fid);
I am attempting to insert the data from an uploaded file into a single dimension array.
The file is as such that there is one student number to a line like so:
392232,392231,etc
this is the most common way I've found online:
while (($line = fgetcsv($file, 25, ',')) !== FALSE) {
//$line is an array of the csv elements
print_r($line);
}
However form what I understand this will create an array ($line) for each row. Which is not what I want.
that aside I tried this to see if it is working and my code is not printing out the array after using ftgetcsv(). The file is successfully uploading.
here is my code:
if(isset($_FILES['csv_file']) && is_uploaded_file($_FILES['csv_file']['tmp_name'])){
//create file name
$file_path = "csv_files/" . $_FILES['csv_file']['name'];
//move uploaded file to upload dir
if (!move_uploaded_file($_FILES['csv_file']['tmp_name'], $file_path)) {
//error moving upload file
echo "Error moving uploaded file";
}
print_r($_FILES['csv_file']);
$file = fopen('$file_path', 'r');
while (($line = fgetcsv($file, 25, ',')) !== FALSE) {
//$line is an array of the csv elements
print_r($line);
}
//delete csv file
unlink($file_path);
}
First off, can anyone obviously see why it wouldnt work to at least print them as seperate arrays of data (each row).
Second, is it possible to set it so that it creates a 1d array of all rows in the file?
Many thanks,
Question 1 is because of
print_r($_FILES['csv_file']);
$file = fopen('$file_path', 'r');
should be:
$file = fopen($file_path, 'r');
and for Question 2, check out the array_push
1st Question:
This line will actually try to open a file called '$file_path' because you're using single quotes (so it doesn't expand to the value of the variable). You can just remove the quotes.
$file = fopen('$file_path', 'r');
$file is null after this.
2nd Question:
If all you want to do is convert a file into an array by lines you can use one of these instead:
file() - get whole file into a 1D array of lines of the file (closest to what you want)
fgets() - get a string per line per call; keep calling this until it returns false to get each line one at a time
file_get_contents() - get the whole file into a string and process as you like
According to PHP.net $line has to return as array.
"returns an array containing the fields read."
But if you are sure it's contains only one student number you can use $line[0] to get the first line value (Ignoring the ",")
Here are some general comments on your code:
You are passing the file path into the fopen() function incorrectly. The variable should not be surrounded with single quotes.
Since you are deleting the CSV file after processing it, moving it is unnecessary. Simply use $_FILES['csv_file']['tmp_name'] as the path to the file.
Since there is only one entry per row in your CSV file, simply access the first element of the array that is returned from fgetcsv(): $numbers[] = $line[0];