How Merging multiple CSV files without headers being repeated (using PHP)? - php

I'm very rookie in PHP and when I combine several csv files with this code, several headers are shown. How could I leave a single header?
I am using a ubuntu 18.04 server with PHP 7.1
<?php
$csvdir = "./csv";
$csvcontent = '';
if (is_dir($csvdir)) {
if ($handle = opendir($csvdir)) {
while (($file = readdir($handle)) !== false) {
if (substr($file, -4) === ".csv") {
$csvcontent .= file_get_contents($csvdir . $file);
}
}
closedir($handle);
}
}
$result = fopen('./todos.csv', 'w');
fwrite($result, $csvcontent);
fclose($result);
?>

According to this answer with example code to skip first lines after getting file contents:
$content = file_get_contents($filename);
$lines = explode("\n", $content);
$skipped_content = implode("\n", array_slice($lines, 2));
You might alter your code like this (although, personally, i might simply alter the production of the csv files to stop producing headers...as exploding the content could slow down your processes).
<?php
$got_headers = 0;
$csvdir = "./csv";
$csvcontent = '';
if (is_dir($csvdir)) {
if ($handle = opendir($csvdir)) {
while (($file = readdir($handle)) !== false) {
if (substr($file, -4) === ".csv") {
if (empty($got_headers)) {
// get the headers the first time only
$csvcontent .= file_get_contents($csvdir . $file);
$got_headers = 1;
} else {
// now, pull the lines after the first
$content = file_get_contents($csvdir . $file);
$lines = explode("\n", $content);
$csvcontent .= implode("\n", array_slice($lines, 1));
}
}
}
closedir($handle);
}
}
$result = fopen('./todos.csv', 'w');
fwrite($result, $csvcontent);
fclose($result);

This has the added benefit of not requiring you to hold the entirety of the resulting file in memory:
$files = []; // use your existing code to build a list of files
$first = true;
$out = fopen('todos.csv', 'wb');
foreach( $files as $file ) {
$in = fopen($file, 'rb');
$line = fread($in); // get the header line
// write only the first file's header
if( $first ) {
fwrite($out, $line);
$first = false;
}
// transcribe the rest of the file.
while( $line = fread($in) ) {
fwrite($out, $line);
}
fclose($in);
}
fclose($out);

Related

Need to run twice in-order for it to fully work

$find = '.5010.';
$directory_with_files = './'.date('m-d-Y');
$dh = opendir($directory_with_files);
$files = array();
while (false !== ($filename = readdir($dh)))
{
if(in_array($filename, array('.', '..')) || is_dir($filename))
continue;
$files[] = $filename;}
foreach($files as $file){
//find only 5010 files
if(stripos($file, $find) !== false){
// open the 5010 file
$handle = fopen(date('m-d-Y').'/'.$file, "r");
$file_content = file_get_contents(date('m-d-Y').'/'.$file);
$handle2 = fopen(date('m-d-Y').'/'.$file, "r");
$file_content2 = file_get_contents(date('m-d-Y').'/'.$file);
if ($handle) {
$header = '';
$name = '';
$footer = '';
$payor_blocks = array();
// determine if file has more than one payor
$payor_count = substr_count($file_content, 'N1*PR*');
//if the file has more than one payor
if($payor_count > 1) {
//read the file line by line
$header_end = false;
$block_start = false;
$count = 1;
if($handle2){
$line_number = 0;
$line_stop= array();
while (($line1 = fgets($handle2)) !== false) {
$line_number++;
if(strpos($line1, 'CAS') !==false){
$line_stop[] = $line_number;}}
$footer_line = count($line_stop)-2;
$footer_line = $line_stop[$footer_line];
$line_number = 0; }
//look for occurances of CAS and what line each on is on
while (($line = fgets($handle)) !== false) {
$line_number++;
//look for the first payor block
if(strpos($line, 'N1*PR*') !== false || $block_start) {
$header_end = true; $block_start = true;
if(strpos($line, 'N1*PR*') !== false) {
$count++;
}
//see if the block finished
if($line_number == $footer_line) {
$block_start = false;
$payor_blocks[$count] .= $line;
$count++; }
$payor_blocks[$count] .= $line;}
else {
if($header_end) {
$footer .= $line."\n"; }
else {
$header .= $line."\n";}}
$refid = 'REF*2U*';
if(stripos($line, $refid) !== false)
{
$refnumber = str_replace(array($refid, '~'), array('', ''), $line);
$refnumber = trim($refnumber);
if($refnumber != '')
{
$refnumber = '_'.$refnumber.'_';
$filerenamed = str_replace($find, $refnumber,$file);
copy('./'.date('m-d-Y').'/'.$file, './'.date('m-d-Y').'/'. $filerenamed);
}
echo $refnumber . "\n";
}
}
//get payor blocks and create a file foreach payor
$new_files = array();
foreach($payor_blocks as $block) {
$filename = date('m-d-Y').'/'.$file . "_" . $count;
$count++;
$new_files[] = array(
'name' => $filename,
'content' => $header."\n".$block."\n".$footer
);
}
foreach($new_files as $new_file) {
$myfile = fopen($new_file['name'], "w");
fwrite($myfile, $new_file['content']);
fclose($myfile);
}
}
else{
while (($line = fgets($handle)) !== false)
{
$refid = 'REF*2U*';
if(stripos($line, $refid) !== false)
{
$refnumber = str_replace(array($refid, '~'), array('', ''), $line);
$refnumber = trim($refnumber);
if($refnumber != '')
{
$refnumber = '_'.$refnumber.'_';
$filerenamed = str_replace($find, $refnumber,$file);
copy('./'.date('m-d-Y').'/'.$file, './'.date('m-d-Y').'/'. $filerenamed);
}
echo $refnumber . "\n";
}
}
}
}
}
// DONE - close the file
fclose($handle);
}
foreach($files as $fiftyfile){
if(stripos($fiftyfile, $find) !== false){
$handle3 = fopen(date('m-d-Y').'/'.$fiftyfile, "r");
$file_content3 = file_get_contents(date('m-d-Y').'/'.$fiftyfile);
if ($handle3) {
if(unlink('./'.date('m-d-Y').'/'.$fiftyfile))
{
echo "file named $fiftyfile has been deleted successfully";
}
else
{
echo "file is not deleted";
}
}
}
}
I have a few files in my directory with filenames that contain "3256.5010.548674.23a" In this code it opens the file and searches if there is more than one "N1*PR*" and if there is to split them into separate files. Lastly to change ".5010." to the REF number which is something like "8743" . Then it deletes all the files with ".5010." And combines the rest in one document. It works fine however, when I first run it it splits and renames, but only deletes the first files not all the ".5010." (not the ones that were split), which then when I run it again after that, it deletes everything but renames the old ones, since it goes through the "else statement" that also does the renaming. How could I solve the issue with the delete?

How to quickly search through a very large list of strings / records on a text file?

I have some text file size(500mb). I need a script that search given string from this text file as first as possible. i was trying
$query="demo";
$FileName = "search.txt";
$fh = fopen($FileName, 'r') or die("Can't open file");
$data = fread($fh, filesize($FileName));
$items = explode(" ", $data);
$Pos = array_search ($query, $items);
if($Pos !== FALSE)
{
echo "Found";
}
else
{
echo "Not Found";
}
Is there any script or algorithm to search more faster?
Thanks in advance.
Do not read the entire file.
Read line by line and check if the $query is contained withing the line (assuming the result can't be in multiple lines):
$query = 'demo';
$fileName = 'search.txt';
$file = fopen($fileName, 'r') or die("Can't open file");
while ($line = fgets($file)) {
if (strpos($query, $line) !== false ) {
$found = true;
break;
}
}
fclose($fh);

PHP function to download and overwrite the previews downloaded file

I am using the following code to download an archived csv file and uncompress it:
$url="http://www.some.zip";
$target = 'data-' . md5(microtime()) . '.zip';
function download($src, $dst) {
$f = fopen($src, 'rb');
$o = fopen($dst, 'wb');
while (!feof($f)) {
if (fwrite($o, fread($f, 2048)) === FALSE) {
return 1;
}
}
fclose($f);
fclose($o);
return 0;
}
download($url,$target);
if ( file_exists($target) ){
echo "Download Successuful <br />";
$arc = new ZipArchive;
if (true !== $arc->open($target)) {
echo "Unzipping Failed <br />";
}else {
file_put_contents($out, $arc->getFromIndex(0));
echo "Unzipping Successuful <br />";
fclose($handle);
}
}else {
echo "Download Failed <br />";
}
However, on a second run, it does't do anything and I would like to overwrite the initial file with the newer file. (the CSV File)
How should I do that? The solution should take about the same time as the first download!
The easiest solution would be to first check if the file exists, then remove it.
function download($src, $dst) {
if(file_exists($dst)) unlink($dst);
$f = fopen($src, 'rb');
$o = fopen($dst, 'wb');
while (!feof($f)) {
if (fwrite($o, fread($f, 2048)) === FALSE) {
return 1;
}
}
fclose($f);
fclose($o);
return 0;
}

Read Directories From files, Loop on it and check existence PHP Didn't Work

I am trying to check the existence of directories list on file as below:
<?php
$file = "L:/tmp/file1.txt";
$f = fopen($file, "r");
while ($line = fgets($f,500)) {
$line = str_replace("\\","/",$line);
$found=is_dir($strTest);
if($found) {
echo "<br>the dir $strTest was found";
} else {
echo "<br>the dir $strTest was not found";
}
}
?>
the File I read from like this:
L:\tmp\Folder1
L:\tmp\Folder2
L:\tmp\Folder3
L:\tmp\Folder4
The result is All Folders Not found except the last one .... but I am sure that all the list are exist
The problem is that in first folder names
L:\tmp\Folder1
L:\tmp\Folder2
L:\tmp\Folder3
when you use fgets it takes \n as well. So in these names you have next line symbol. In the last one L:\tmp\Folder4 there is no \n, so thats why the only found is the last one.
<?php
$file = "file.txt";
$f = fopen($file, "r");
while ($line = fgets($f, 500)) {
$line = str_replace("\\", "/", $line);
$line = preg_replace("/
/", "", $line);
if (is_dir($line)) {
echo "<br />the dir $line was found";
} else {
echo "<br />the dir $line was not found";
}
}
?>
try with this code (replace your parameters)
$handle = opendir('/path/to/directory')
if ($handle) {
while (false !== ($file = readdir($handle))) {
print "$file<br />\n";
}
closedir($handle);
}

Remove Line From CSV File

I have .csv file with 4 columns. What's the easiest way to remove a line identical with the id of the first column? Here's where I got stuck:
if($_GET['id']) {
$id = $_GET['id'];
$file_handle = fopen("testimonials.csv", "rw");
while (!feof($file_handle) ) {
$line_of_text = fgetcsv($file_handle, 1024);
if ($id == $line_of_text[0]) {
// remove row
}
}
fclose($file_handle);
}
Unfortunately, databases were not a choice.
$table = fopen('table.csv','r');
$temp_table = fopen('table_temp.csv','w');
$id = 'something' // the name of the column you're looking for
while (($data = fgetcsv($table, 1000)) !== FALSE){
if(reset($data) == $id){ // this is if you need the first column in a row
continue;
}
fputcsv($temp_table,$data);
}
fclose($table);
fclose($temp_table);
rename('table_temp.csv','table.csv');
I recently did a similar thing in for a newsletter unsubscription, heres my code:
$signupsFile = 'newsletters/signups.csv';
$signupsTempFile = 'newsletters/signups_temp.csv';
$GLOBALS["signupsFile"] = $signupsFile;
$GLOBALS["signupsTempFile"] = $signupsTempFile;
function removeEmail($email){
$removed = false;
$fptemp = fopen($GLOBALS["signupsTempFile"], "a+");
if (($handle = fopen($GLOBALS["signupsFile"], "r")) !== FALSE) {
while (($data = fgetcsv($handle)) !== FALSE) {
if ($email != $data[0] ){
$list = array($data);
fputcsv($fptemp, $list);
$removed = true;
}
}
fclose($handle);
fclose($fptemp);
unlink($GLOBALS["signupsFile"]);
rename($GLOBALS["signupsTempFile"], $GLOBALS["signupsFile"]);
return $removed;
}
this uses the temp file method of writing out the csv line by line to avoid memory errors. Then once the new file has been created, it deletes the original and renames the temp file.
You can modify this code so that it looks for an ID instead of an email address eg:
$id = $_GET['id'];
$fptemp = fopen('testimonials-temp.csv', "a+");
if (($handle = fopen('testimonials.csv', "r")) !== FALSE) {
while (($id= fgetcsv($handle)) !== FALSE) {
if ($id != $data[0] ){
$list = array($data);
fputcsv($fptemp, $list);
}
}
fclose($handle);
fclose($fptemp);
unlink('testimonials.csv');
rename('testimonials-temp.csv','testimonials.csv');
$id = $_GET['id'];
if($id) {
$file_handle = fopen("testimonials.csv", "w+");
$myCsv = array();
while (!feof($file_handle) ) {
$line_of_text = fgetcsv($file_handle, 1024);
if ($id != $line_of_text[0]) {
fputcsv($file_handle, $line_of_text);
}
}
fclose($file_handle);
}
You can do:
$new = '';
while (!feof($file_handle))
{
$line_of_text = fgetcsv($file_handle, 1024);
if ($id != $line_of_text[0])
{
$new .= implode(',',$line_of_text) . PHP_EOL;
}
}
basically you running threw each line and check if the id does NOT match the id sent in the get parameter, if it does not then it writes the line to the new container / variable.
And then rewrite the $new value to the file, this should work ok:
How big is the file
Do you have a CSV Header on line 0?
I have found a solution, that does not need to copy the file.
$file = 'testimonials.csv'
// open two handles on the same file
$input = fopen($file ,'r'); // read mode
$output = fopen($file, 'c'); // write mode
if($input !== FALSE && $output !== FALSE) { // check for error
while (($data = fgetcsv($input, $CSVLIMIT, $sep)) !== FALSE) {
if(reset(data) == $id) {
continue;
}
fputcsv($output, $data, $sep);
}
// close read handle
fclose($input);
// shorten file to remove overhead created by this procedure
ftruncate($output, ftell($output));
fclose($output);
}
Note: only one of those fopen commands could fail, leaking the handle for the second one. It would be good, to check both handles independetly and close them on a error.

Categories