I'm coding a plugin that runs everyday at 5am. It combines multiple csv files (That have a txt extension).
Currently, it is working... HOWEVER, the output format is incorrect.
The input will look like this:
"","","","","email#gmail.com","PARK PLACE 109 AVE","SOME RANDOM DATA","","","",""
And so on. this is only a partial row.
The ouput of this code does not retun the same format. It produces something like this without the " in columns without data
,,,,email#gmail.com,"PARK PLACE 109 AVE","SOME RANDOM DATA",,,,
Here is the part of the function that combines everything:
function combine_and_email_csv_files() {
// Get the current time and date
$now = new DateTime();
$date_string = $now->format('Y-m-d_H-i-s');
// Get the specified directories
$source_directory = get_option('csv_file_combiner_source_directory');
$destination_directory = get_option('csv_file_combiner_destination_directory');
// Load the CSV files from the source directory
$csv_files = glob("$source_directory/*.txt");
// Create an empty array to store the combined CSV data
$combined_csv_data = array();
// Loop through the CSV files
foreach ($csv_files as $file) {
// Load the CSV data from the file
$csv_data = array_map('str_getcsv', file($file));
// Add the CSV data to the combined CSV data array
$combined_csv_data = array_merge($combined_csv_data, $csv_data);
}
// Create the combined CSV file
$combined_csv_file = fopen("$destination_directory/$date_string.txt", 'w');
// Write the combined CSV data to the file
foreach ($combined_csv_data as $line) {
fputcsv($combined_csv_file, $line);
}
// Close the combined CSV file
fclose($combined_csv_file);
}
No matter, what I've tried... it's not working. I'm missing something simple I know.
Thank you Nigel!
So this thread, Forcing fputcsv to Use Enclosure For *all* Fields helped me get there....
Using fputs instead of fputscsv and force "" on null values is the short answer for me. Works beautifully... code is below:
function combine_and_email_csv_files() {
// Get the current time and date
$now = new DateTime();
$date_string = $now->format('Y-m-d_H-i-s');
// Get the specified directories
$source_directory = get_option('csv_file_combiner_source_directory');
$destination_directory = get_option('csv_file_combiner_destination_directory');
// Load the CSV files from the source directory
$csv_files = glob("$source_directory/*.txt");
// Create an empty array to store the combined CSV data
$combined_csv_data = array();
// Loop through the CSV files
foreach ($csv_files as $file) {
// Load the CSV data from the file
$csv_data = array_map('str_getcsv', file($file));
// Add the CSV data to the combined CSV data array
$combined_csv_data = array_merge($combined_csv_data, $csv_data);
}
// Create the combined CSV file
$combined_csv_file = fopen("$destination_directory/$date_string.txt", 'w');
// Write the combined CSV data to the file
foreach ($combined_csv_data as $line) {
// Enclose each value in double quotes
$line = array_map(function($val) {
if (empty($val)) {
return "\"\"";
}
return "\"$val\"";
}, $line);
// Convert the line array to a CSV formatted string
$line_string = implode(',', $line) . "\n";
// Write the string to the file
fputs($combined_csv_file, $line_string);
}
Thank you Sammitch
After much haggling with this problem... Sammitch pointed out why not just concat the files... Simplicity is the ultimate sophistication... right?
*Note: this will only work for my specific circumstance. All I'm doing now is concating the files and checking each file ends with a new line and just plain skipping the csv manipulation.
Code below:
function combine_and_email_csv_files() {
// Get the current time and date
$now = new DateTime();
$date_string = $now->format('Y-m-d_H-i-s');
// Get the specified directories
$source_directory = get_option('csv_file_combiner_source_directory');
$destination_directory = get_option('csv_file_combiner_destination_directory');
// Load the files from the source directory
$files = glob("$source_directory/*.txt");
// Create the combined file
$combined_file = fopen("$destination_directory/$date_string.txt", 'w');
// Loop through the files
foreach ($files as $file) {
// Read the contents of the file
$contents = file_get_contents($file);
// Ensure that the file ends with a newline character
if (substr($contents, -1) != "\n") {
$contents .= "\n";
}
// Write the contents of the file to the combined file
fwrite($combined_file, $contents);
}
// Close the combined file
fclose($combined_file);
Related
The target is how to read a range of rows/lines from large CSV file into a JSON array in order to handle large files and read the data in pagination method, each page fetches a range of lines ( e.x. page number 1 fetch from line 1 to 10, page number 2 fetch from line 11 to line 20, and so and ).
the below PHP script read from the being CSV file to the desired line ($desired_line), My question is how we can determine the starting line to read from a specific line ($starting_line)
<?php
// php function to convert csv to json format
function csvToJson($fname, $starting_line, $desired_line) {
// open csv file
if (!($fp = fopen($fname, 'r'))) {
die("Can't open file...");
}
//read csv headers
$key = fgetcsv($fp,"1024","\t");
$line_counter = 0;
// parse csv rows into array
$json = array();
while (($row = fgetcsv($fp,"1024","\t")) && ($line_counter < $desired_line)) {
$json[] = array_combine($key, $row);
$line_counter++;
}
// release file handle
fclose($fp);
// encode array to json
return json_encode($json);
}
// Define the path to CSV file
$csv = 'file.csv';
print_r(csvToJson($csv, 20, 30));
?>
You should use functions like:
fgets() to read the file line by line
fseek() to move to the position of the last fgets() of the chunk
ftell() to read the position for fseek()
Something like this (it's only a schema):
<?php
...
$line_counter = 0;
$last_pos = ...
$fseek($fp,$last_pos);
while($line = fgets($fp)){ // read a line of the file
$line_counter++;
(...) // parse line of csv here
if($line_counter == 100){
$lastpos = ftell($fp);
(...) // save the $lastpos for next reading cycle
break;
}
}
...
?>
You can also skip the fseek() and ftell() part and just count the lines every time from the beginning, but that will generally have to go through the whole file from the beginning till the desired lines.
I have a .csv file which I can use with Google maps API to successfully create map data.
What I'm looking to do is merge 2 (or more) .csv files and display the TOTAL data on the Google map in the same way. They are all in the same format.
I have the paths to the 2 csv files and if need be, a blank .csv file in the same directory where the files could be merged to...
Unfortuantely, the .csv files all have an initial 'header row' which would be awesome to omit when merging...
If anyone can point me in the right direction, I'd be very happy. Thanks
edit: I've tried:
$data1 = file_get_contents('google_map_data.csv');
$data2 = file_get_contents('google_map_data2.csv');
$TOTALdata = "google_map_dataALL.csv";
function joinFiles(array $files, $result)
{
if(!is_array($files)) {
throw new Exception('`$files` must be an array');
}
$wH = fopen($result, "w+");
foreach($files as $file)
{
$fh = fopen($file, "r");
while(!feof($fh))
{
fwrite($wH, fgets($fh));
}
fclose($fh);
unset($fh);
fwrite($wH, "\n"); //usually last line doesn't have a newline
}
fclose($wH);
unset($wH);
joinFiles(array($data1, $data2), $TOTALdata);
I'm assuming both files are small, so loading them all in one go should be OK.
The code loads both files then removes the first line off the second one. It also removes any end of line from the first file, but adds it's own to ensure it always has a new line...
$data1 = file_get_contents('google_map_data.csv');
$data2 = file_get_contents('google_map_data2.csv');
$TOTALdata = "google_map_dataALL.csv";
$data2 = ltrim(strstr($data2, PHP_EOL));
file_put_contents($TOTALdata, rtrim($data1).PHP_EOL.$data2);
I need to import csv file with php getcsv function but the file is not well formated and cannot import it. After uploading the file, I'd like to delete all " inside the file to have a proper file with ; for each field.
This is an example of my file:
"DENO;NAME;SURNAME;""BIRTH"";""ZIP"";CITY;E-MAIL;TELEPHONE"
"M;DAVID;BON;""1959-02-12 00:00:00"";75009;PARIS;email#gmail.com;010000000"
"M;DOE;JHON;""1947-02-02 00:00:00"";75008;PARIS;email#gmail.com;060000000"
"M;DAVE;Philippe;""1950-01-01 00:00:00"";75002;""PARIS"";email#gmail.com;070000000"
I think I would need to read each line of the file,and maybe use str_replace but I don't know how to write the new file...
Assuming you can get your code into a text string, you can just replace/remove the double-quotes. Then parse the text and get it into a format that will support fputcsv().
When writing the folder, be sure to check that your user has permissions to write to this folder (for example if the parent folder has permissions drwxr-xr-x, you can fix it with chmod g+w folder_name).
<?php
// sample text to parse
$some_text = '"DENO;NAME;SURNAME;""BIRTH"";""ZIP"";CITY;E-MAIL;TELEPHONE"
"M;DAVID;BON;""1959-02-12 00:00:00"";75009;PARIS;email#gmail.com;010000000"
"M;DOE;JHON;""1947-02-02 00:00:00"";75008;PARIS;email#gmail.com;060000000"
"M;DAVE;Philippe;""1950-01-01 00:00:00"";75002;""PARIS"";email#gmail.com;070000000"';
// remove quotes
$final_text = str_replace('"', '', $some_text);
// create array of rows by searching for new lines
$data = str_getcsv($final_text, "\n");
// create an empty array to save our final csv data
$final_csv = array();
// loop thru the array and save to the final csv data
foreach ($data as $value) {
// before saving to final csv data, split row into individual column items
$value_array = explode(";", $value);
$final_csv[] = $value_array;
}
// helper debugger to show data before writing it
echo "<pre>";
print_r($final_csv);
echo "</pre>";
// create a new file for writing or open and truncate to 0
$fp = fopen('file.csv', 'w');
// write to file from final csv data
foreach ($final_csv as $fields) {
fputcsv($fp, $fields);
}
// close file
fclose($fp);
?>
I am having a lot of problem while writing a file within foreach loop. It either writes the line which is at the end in the array or is it at the start of the array.
For Example:
A file contains such elements,
page.php?id=1
page.php?id=3
page.php?id=4
investor.php?id=1&la=1
page.php?id=15
page.php?id=13
page.php?id=14
The code will open this file and then separate each array using explode by using = delimiter. And will return such elements
page.php?id
page.php?id
page.php?id
investor.php?id
page.php?id
page.php?id
page.php?id
then it will choose unique elements using array_unique function & then save it in a file. I have this code. Please Help me
$lines = file($fopen2);
foreach($lines as $line)
{
$rfi_links = explode("=",$line);
echo $array = $rfi_links[0];
$save1 = $rfi.$file.$txt;
$fp=fopen("$save1","w+");
fwrite($fp,$array);
fclose($fp);
}
$links_duplicate_removed = array_unique($array);
print_r($links_duplicate_removed);
"w+" would create a new file on each open, wiping out the old content.
"a+" solves the problem, but it's better to open the file for writing before the loop, and closing after it.
What kind of does not make sense, is the fact that you're writing the current url always to that file while overwriting its previous content. In every step of the foreach-loop, you reopen that file, erase its content and write one url to that file. In the next step, you reopen exactly the same file and do that again. That's why you end up with only the last url in that file.
You will need to collect all urls in an array, throw out duplicates and then write the unique ones to the disc:
$lines = file($fopen2);
$urls = array(); // <-- create empty array for the urls
foreach ($lines as $line) {
$rfi_links = explode('=', $line, 2); // <-- you need only two parts, rights?
$urls[] = $rfi_links[0]; // <-- push new URL to the array
}
// Remove duplicates from the array
$links_duplicate_removed = array_unique($urls);
// Write unique urls to the file:
file_put_contents($rfi.$file.$ext, implode(PHP_EOL, $links_duplicate_removed));
Another solution (much more inspired by your former method) is to open the file once, before starting to iterate over the lines:
$lines = file($fopen2);
$urls = array();
// Open file
$fp = fopen($rfi.$file.$ext, 'w');
foreach ($lines as $line) {
$rfi_url = explode('=', $line, 2);
// check if that url is new
if (!in_array($rfi_url[0], $urls)) {
// it is new, so add it to the array (=mark it as "already occured")
$urls[] = $rfi_url[0];
// Write new url to the file
fputs($fp, $rfi_url[0] . PHP_EOL);
}
}
// Close the file
fclose($fp);
So I was able to parse the txt file into a csv file
$data = array();
while ($line= fgets ($fh)) {
$stack = array($LAUS,$FIPS,$CountyName,$Date,$_CLF,$_EMP,$_UNEMP,$RATE);
array_push($data, $stack);
}
$file = fopen('file.csv','w');
foreach ($data as $fields) {
fputcsv($file, $fields,',','"');
}
fclose($file);
My question is, what is the best way to create multiple csv files that are seperated by Month (also the year, like Jan01.csv, Jan02.csv).
I'm taking a bit of a guess at the formatting of your date
while ($line = fgets ($fh)) {
// Not sure where you're getting these values, but I'm assuming it's correct
$stack = array($LAUS,$FIPS,$CountyName,$Date,$_CLF,$_EMP,$_UNEMP,$RATE);
// Assuming $Date looks like this '2011-10-04 15:00:00'
$filename = date('My', strtotime($Date)) . '.csv';
$file = fopen($filename,'a+');
fputcsv($file, $stack,',','"');
fclose($file);
}
This will be a little slow since you're opening and closing files constantly, but since I don't know the size of your original data set I don't want to use up all the memory caching the result before I write it.
Be aware that running this multiple times will end up with duplicate data being inserted into your CSV files. You may want to add some code to remove/clear out any currently existing CSV files before you run this bit of code.