still struggling with PHP and CSV file manipulation. I will try to ask this properly so I can get some help.
I have a CSV file with about 4000 lines/rows, and I wrote this code to create an array of the entire CSV, and pull the the LAST line of the CSV file out to use in my script. The code works to to all this wit success.
// CREATE ASSOCIATIVE ARRAY FROM LAST ROW OF CSV FILE
$csv = array();
if (FALSE !== $handle = fopen("Alabama-TEST.csv", "r"))
{
while (FALSE !== $row = fgetcsv($handle))
{
$csv[] = $row;
}
}
$new_csv = array();
foreach ($csv as $row)
{
$new_row = array();
for ($i = 0, $n = count($csv[0]); $i < $n; ++$i)
{
$new_row[$csv[0][$i]] = $row[$i];
}
$new_csv[] = $new_row;
}
The variable $new_row is the last row in the CSV and I am able to use the data fine. But when the script is finished running I want to delete this last row called $new_row.
Here is an example of my CSV file at the top view;
CITY ADDRESS STATE NAME
Birmingham 123 Park St. Alabama Franky
So I just want to remove the last row and keep the header at the top, for the next time the script runs. I've been trying for 3 days solid trying to figure this out, so I'm hoping some kind person who KNOWS WHAT THEY ARE DOING can help.
Here is the code I have tried to remove the last row;
$inp = file('Alabama-TEST.csv');
$out = fopen('Alabama-TEST.csv','w');
or ($i=0;$i<count($inp)-1;$i++)
fwrite($out,$inp[$i]);
fclose($out);
Since I'm using a large file at around 4000 rows, is there a way to do this without using too much memory?
You need to use array_pop on your array ($new_csv) and fputcsv to save the file:
array_pop($new_csv); // Removes the last element in the array
$out = fopen('Alabama-TEST-new.csv','w');
foreach ($new_csv as $row) { // sorry for writing it badly, try now
fputcsv($out, $row);
}
fclose($out);
I don't have an environment to test this, sorry.
Related
I need help processing files holding about 46k lines or more than 30MB of data.
My original idea was to open the file and turn each line into an array element. This worked the first time as the array held about 32k values total.
The second time, the process was repeated, the array only held 1011 elements, and finally, the third time it could only hold 100.
I'm confused and don't know much about the backend array processes. Can someone explain what is happening and fix the code?
function file_to_array($cvsFile){
$handle = fopen($cvsFile, "r");
$path = fread($handle, filesize($cvsFile));
fclose($handle);
//Turn the file into an array and separate lines to elements
$csv = explode(",", $path);
//Remove common double spaces
foreach ($csv as $key => $line){
$csv[$key] = str_replace(' ', '', str_getcsv($line));
}
array_filter($csv);
//get the row count for the file and array
$rows = count($csv);
$filerows = count(file($cvsFile)); //this no longer works
echo "File has $filerows and array has $rows";
return $csv;
}
The approach here can be split in 2.
Optimized file reading and processing
Proper storage solution
Optimized file processing can be done like so:
$handle = fopen($cvsFile, "r");
$rowsSucceed = 0;
$rowsFailed = 0;
if ($handle) {
while (($line = fgets($handle)) !== false) { // Reading file by line
// Process CSV line and check if it was parsed correctly
// And count as you go
if (!empty($parsedLine)) {
$csv[$key] = ... ;
$rowsSucceed++;
} else {
$rowsFailed++;
}
}
fclose($handle);
} else {
// Error handling
}
$totalLines = $rowsSucceed + $rowsFailed;
Also you can avoid array_filter() simply by not adding processed line if its empty.
It will allow to optimize memory usage during script execution.
Proper storage
Proper storage here is needed for performing operations on certain amount of data. File reading are ineffective and expensive. Using simple file based database like sqlite can help you a lot and increase overall performance of your script.
For this purpose you probably should process your CSV directly to database and than perform count operation on parsed data avoiding excessive file line counts etc.
Also it gives you further advantage on working with data not keeping it all in memory.
Your question says you want to "turn each line into an array element" but that is definitely not what you are doing. The code is quite clear; it reads the entire file into $path and then uses explode() to make one massive flat array of every element on every line. Then later you're trying to run str_getcsv() on each item, which of course isn't going to work; you've already exploded all the commas away.
Looping over the file using fgetcsv() makes more sense:
function file_to_array($cvsFile) {
$filerows = 0;
$handle = fopen($cvsFile, "r");
while ($line = fgetcsv($handle)) {
$filerows++;
// skip empty lines
if ($line[0] === null) {
continue;
}
//Remove common double spaces
$csv[] = str_replace(' ', '', $line);
}
//get the row count for the file and array
$rows = count($csv);
echo "File has $filerows and array has $rows";
fclose($handle);
return $csv;
}
What I need to do is to be able to move the first row from a testdata.csv every time I run the .php to another .csv with the name testdata_new.csv(appending data).
This is an example of data that includes Name, Age, Job
Example data testdata.csv:
John,32,Scientist
Mary,25,Employer
Nick,36,Designer
Miky,46,Sales
Alex,29,Logistics
This is what the .php will do running it:
Cut the first row from testdata.csv(john,32,scientist) and paste it to the new testdata_new.csv under the first row(header) that will always be "Name Age Job".
Save testdata_new.csv and testdata.csv with the remaining rows.
I did some tests but I'm still far away from the solution.
<?php
$file = "testdata.csv";
$f = fopen($file, "r");
$i = 0;
$file2 = str_replace(".csv", "_new.csv", $file);
$f2 = fopen($file2,"a");
while ($i<2) {
$record = fgetcsv($f);
foreach($record as $field) {
echo $field . "<br>";
}
$i++;
}
fwrite($f2,fread($f, filesize($file)));
fclose($f);
fclose($f2);
?>
Executing the script will display the first row of the template.csv file
and will produce another file with the name template_new.csv with the following rows:
Mary,25,Employer
Nick,36,Designer
Miky,46,Sales
Alex,29,Logistics
What I really need to have in the template_new.csv file is only the first row displayed:
John,32,Scientist
And save again the template.csv without the first row as the idea is to cut and paste the rows, as following:
Mary,25,Employer
Nick,36,Designer
Miky,46,Sales
Alex,29,Logistics
Thank you all in advance for your help!
As easy as this ;-)
$old_file = 'testdata.csv';
$new_file = 'testdata_new.csv';
$file_to_read = file_get_contents($old_file); // Reading entire file
$lines_to_read = explode("\n", $file_to_read); // Creating array of lines
if ( $lines_to_read == '' ) die('EOF'); // No data
$line_to_append = array_shift( $lines_to_read ); // Extracting first line
$file_to_append = file_get_contents($new_file); // Reading entire file
if ( substr($file_to_append, -1, 1) != "\n" ) $file_to_append.="\n"; // If new file doesn't ends in new line I add it
// Writing files
file_put_contents($new_file, $file_to_append . $line_to_append . "\n");
file_put_contents($old_file, implode("\n", $lines_to_read));
I'm trying to display only the rows that contain a specific word in a specific column. Basically I would like to show only the rows that have "yes" in the Display column.
First_Name, Last_Name, Display
Kevin, Smith, yes
Jack, White, yes
Joe, Schmo, no
I've been trying various things with fgetcsv & str_getcsv from other answers and from php.net but nothing is working so far.
It doesn't do anything but this is my current code:
$csv = fopen('file.csv', 'r');
$array = fgetcsv($csv);
foreach ($array as $result) {
if ($array[2] == "yes") {
print ($result);
}
}
Let's have a look at the documentation for fgetcsv():
Gets line from file pointer and parse for CSV fields
fgetcsv reads a single line, not the whole file. You can keep reading lines until you reach the end of the file by putting it in a while loop, e.g.
<?php
$csv = fopen('file.csv', 'r');
// Keep looping as long as we get a new $row
while ($row = fgetcsv($csv)) {
if ($row[2] == "yes") {
// We can't just echo $row because it's an array
//
// Instead, let's join the fields with a comma
echo implode(',', $row);
echo "\n";
}
}
// Don't forget to close the file!
fclose($csv);
You should use data tables.
https://datatables.net/examples/basic_init/zero_configuration.html
That's how I deal with my textfiles. But be carefull, with a large amount of Data (> 10000 rows) you should have a loog at the deferRender option.
https://datatables.net/reference/option/deferRender <-- JSON DATA required.
Been hunting high and low for this one. I have a script that creates an array from the 2nd row/line of a CSV file, leaving the headers alone, then does it's thing. Great!
BUT when it's done I want to remove/delete that 2nd line from the CSV file so the next time my script runs it's using the next row coming up.
I've tried PHP code ideas like this with no luck so far.
$cnt = 0;
if (($handle = fopen("Alabama.csv", "r")) !== FALSE) {
while (($csvadata = fgetcsv($handle, 0, ",")) !== FALSE) {
$data[$cnt++] = $csvadata;
}
fclose($handle);
unset($data[2]);
}
$fp = fopen('Alabama.csv', 'w');
foreach ($data as $fields) {
fputcsv($fp, $fields);
}
fclose($fp);
Here is a sample of the input file. I've included the header row and the first row of data. I need to remove the 1st line of data and keep the header intact. I've show here up to column E for example but the file actually goes to column T. Also each file will have AT MOST 4000 lines/rows.
STATE FDICCERT ADDRESS ASSETS CITY
Alabama 35 100 North Gay Street 768256 Auburn
try prepending
array_shift($data);
right before
foreach ($data as $fields) {
That way you will skip the first $data item when writing your file.
I am using the current code to read a csv file and add it to an array:
echo "starting CSV import<br>";
$current_row = 1;
$handle = fopen($csv, "r");
while ( ($data = fgetcsv($handle, 10000, ",") ) !== FALSE )
{
$number_of_fields = count($data);
if ($current_row == 1) {
//Header line
for ($c=0; $c < $number_of_fields; $c++)
{
$header_array[$c] = $data[$c];
}
} else {
//Data line
for ($c=0; $c < $number_of_fields; $c++)
{
$data_array[$header_array[$c]] = $data[$c];
}
array_push($products, $data_array);
}
$current_row++;
}
fclose($handle);
echo "finished CSV import <br>";
However when using a very large CSV this times out on the server, or has a memory limit error.
I'd like a way to do it in stages, so after the first say 100 lines it will refresh the page, starting at line 101.
I will probably be doing this with a meta refresh and a URL parameter.
I just need to know how to adapt that code above to start at the line I tell it to.
I have looked into fseek() but I'm not sure how to implement this here.
Can you please help?
The timout can be circumvented using
ignore_user_abort(true);
set_time_limit(0);
When experiencing problems with the memory limit, it may be wise to take a step back and look at what you're actually doing with the data you're processing. Are you pushing the data into a database? calculate something off the data but don't need to store the actual data, …
Do you really need to push (array_push($products, $data_array);) the rows into an array (for later processing)? can you instead write to the database directly? or calculate directly? or build an html <table> directly? or whatever the hell you're doing right then an there, within the while() loop, without pushing everything into an array first?
If you're able to chunk the processing, I guess you don't need that array at all. Otherwise you'd have to restore the array for every chunk - not solving the memory issue one bit.
If you can manage to change your processing algorithm to waste less memory / time, you should seriously consider that over any chunked processing requiring a round-trip to the browser (for so many performance and security reasons…).
Anyways, you can, at any time, identify the current stream offset with ftell() and re-set to that position using fseek(). You'd only need to pass that integer to your next iteration.
Also there is no need for your inner for() loops. This should produce the same results:
<?php
$products = array();
$cols = null;
$first = true;
$handle = fopen($csv, "r");
while (($data = fgetcsv($handle, 10000, ",")) !== false) {
if ($first) {
$cols = $data;
$first = false;
} else {
$products[] = array_combine($cols, $data);
}
}
fclose($handle);
echo "finished CSV import <br>";