How would I go about deleting a specific number of rows from a large csv file? ie, I am looking to delete the first 100 rows from a file containing over 200,000 rows. Just curious to see if there is another way to handle things other than completely re-writing the file to a temp file and back to overwrite the original file.
Thanks in advance for any help/suggestions.
Does it need to be in php?
You can use the unix command tail to print every line in a file starting from the nth line:
tail -n +101 file.csv > trimmed.csv
Or using PHP:
$lines = file('file.csv'); // read file as array of lines
$trimmed = array_slice($lines, 100); // slice array at index
file_put_contents('trimmed.csv', implode($trimmed)); // implode and write to file
Related
What is the best way to overwrite a specific line in a file? I basically want to search a file for the string '#parsethis' and overwrite the rest of that line with something else.
If the file is really big (log files or something like this) and you are willing to sacrifice speed for memory consumption you could open two files and essentially do the trick Jeremy Ruten proposed by using files instead of system memory.
$source='in.txt';
$target='out.txt';
// copy operation
$sh=fopen($source, 'r');
$th=fopen($target, 'w');
while (!feof($sh)) {
$line=fgets($sh);
if (strpos($line, '#parsethis')!==false) {
$line='new line to be inserted' . PHP_EOL;
}
fwrite($th, $line);
}
fclose($sh);
fclose($th);
// delete old source file
unlink($source);
// rename target file to source file
rename($target, $source);
If the file isn't too big, the best way would probably be to read the file into an array of lines with file(), search through the array of lines for your string and edit that line, then implode() the array back together and fwrite() it back to the file.
Your main problem is the fact that the new line may not be the same length as the old line. If you need to change the length of the line, there is no way out of rewriting at least all of the file after the changed line. The easiest way is to create a new, modified file and then move it over the original. This way there is a complete file available at all times for readers. Use locking to make sure that only one script is modifying the file at once, and since you are going to replace the file, do the locking on a different file. Check out flock().
If you are certain that the new line will be the same length as the old line, you can open the file in read/write mode (use r+ as the second argument to fopen()) and call ftell() to save the position the line starts at each time before you call fgets() to read a line. Once you find the line that you want to overwrite, you can use fseek() to go back to the beginning of the line and fwrite() the new data. One way to force the line to always be the same length is to space pad it out to the maximum possible length.
This is a solution that works for rewriting only one line of a file in place with sed from PHP. My file contains only style vars and is formatted:
$styleVarName: styleVarProperty;\n
For this I first add the ":" to the ends of myStyleVarName, and sed replaces the rest of that line with the new property and adds a semicolon.
Make sure characters are properly escaped in myStyleVarProp.
$command = "pathToShellScript folder1Name folder2Name myStyleVarName myStyleVarProp";
shell_exec($command);
/* shellScript */
#!/bin/bash
file=/var/www/vhosts/mydomain.com/$1/$2/scss/_variables.scss
str=$3"$4"
sed -i "s/^$3.*/$str;/" $file
or if your file isn't too big:
$sample = file_get_contents('sample');
$parsed =preg_replace('##parsethis.*#', 'REPLACE TO END OF LINE', $sample);
You'll have to choose delimiters '#' that aren't present in the file though.
If you want to completely replace the contents of one file with the contents of another file you can use this:
rename("./some_path/data.txt", "./some_path/data_backup.txt");
rename("./some_path/new_data.txt", "./some_path/data.txt");
So in the first line you backup the file and in the second line you replace the file with the contents of a new file.
As far as I can tell the rename returns a boolean. True if the rename is successful and false if it fails. One could, therefore, only run the second step if the first step is successful to prevent overwriting the file unless a backup has been made successfully. Check out:
https://www.php.net/manual/en/function.rename.php
Hope that is useful to someone.
Cheers
Adrian
I'd most likely do what Jeremy suggested, but just for an alternate way to do it here is another solution. This has not been tested or used and is for *nix systems.
$cmd = "grep '#parsethis' " . $filename;
$output = system($cmd, $result);
$lines = explode("\n", $result);
// Read the entire file as a string
// Do a str_repalce for each item in $lines with ""
I want to delete current iterating row from CSV file,
Suppose I have 5 rows (1,2,3,4,5) into my CSV file, I opened my file while iterating the rows one by one using foreach, when it comes to 3rd row then 3rd row should be deleted from CSV file and other 1,2,4,5 rows will be same while same iterration. I don't want to use like Skip the 3rd iterration and save it into another file, not like rename file name. So, Please anyone can help me how to do this in PHP?
Suppose, like SQL command to delete current row, is there anything in PHP?
Thanks.
You will need to have permissions to write the file. If you remove a row, you will have to save the file. As you do not want to have any empty rows as well, you need to read the whole file.
I suggest to get the file content, and split it into a array by lines. With the explode function you can split the content by the line break, most likely "\n". So an array wich will contain each line of the csv file will be existing. Now you can simply remove the line from the array and create a string out of it, before saveing the changed content back to the csv file.
// get csv content into string variable
$csvContent = file_get_contents(__DIR__ . "/yourfile.csv");
// create array out of csv, limited by PHP_EOL wich determines the systems line break
// this means, every line in the csv file will be represended by an array key and its value
$csvArray = explode(PHP_EOL, $csvContent);
// unset the 3th csv line.
unset($csvArray[2]); // keep in mind array starts at 0, so third line is array key 2
// from here on you can manipulate the array $csvArray as you like.
// add lines, remove lines or just use the given content for future operations.
// you can also iterate over the array with: foreach ($csvArray as $line => $value) {}
// create string out of modified csv array
$csvContent = implode(PHP_EOL, $csvArray);
// save result back into the csv file
file_put_contents( __DIR__ . "/yourfile.csv", $csvContent);
Check implode/explode php function docs:
http://php.net/manual/en/function.implode.php
http://php.net/manual/en/function.explode.php
I'm using the Sebastian Bergmann PHPUnit selenium webdriver.
Current I have:
$csv = file_get_contents('functions.csv', NULL,NULL,1);
var_dump($csv);
// select 1 random line here
This will load my CSV file and give me all possible data from the file.
It has multiple rows for example:
Xenoloog-FUNC/8_4980
Xylofonist-FUNC/8_4981
IJscoman-FUNC/8_4982
Question: How can I get that data randomly?
I just want to use 1 ( random) line of data.
Would it be easier to just grab 1 (random) line of the file instead of everything?
Split the string into an array, then grab a random index from that array:
$lines = explode("\n", $csv);
$item = $lines[array_rand($lines)];
You could use the offset and maxlen parameters to grab part of the file using file_get_contents. You could also use fseek after fopen to select part of a file. These functions both take numbers of bytes as arguments. This post has a little more information:
Get part of a file by byte number in php
It may require some hacking to translate a particular row-index of a CSV file to a byte offset. You might need to generate and load a small meta-data file which contains a list of bytes-occupancies for each row of of CSV data. That would probably help.
Need to process large csv files with php.
Working with fgetcsv and performance seems pretty good. For even better/ faster processing I would like the csv files to have column names on the first row, which are missing right now.
Now I would like to add a top row with columns names to the csv file with PHP before processing the files with fgetcsv.
Is there a way to add a line top the top of the file with php easily? Or would this mean I have to take a approach like below?
List item
List item
create a temp file
add the column names to this temp file
read original csv file contents
put original csv contents into the temp file
delete original file
rename the temp file
Any feedback on how to do this in the most effective way is highly appreciated. Thanks for your time :)
Please try this .. much simpler and straight to the point.
$file = 'addline.csv';
$header = "Name, IP, Host, RAM, Task, NS \r\n";
$data = file_get_contents($file);
file_put_contents($file, $header.$data);
You are done. Hope this helps...
Read the CSV file using fgetcsv.
Close the file.
Open the file using "w" (write) mode.
Write the headers.
Close the file.
Open the file using "a" (append) mode.
Write the CSV file using fputcsv.
Voila!
Just do it the way you provided. there is no easy way to extend a file on the beginning instead of the end. it's like writing on paper. it's easy to add new lines below your text but you can't write above you text.
Don't use fopen($file, "r+") or fopen($file, "c") as this will remove existing lines at the beginning or even all lines of your file.
Why not to define header in your code manually, especially if every file has the same header?
If you really need to insert header into CSV files before you process them maybe you should consider using sed like this: sed -i 1i'"header 1","header 2"' file.csv
I want to merge two large CSV files with PHP. This files are too big to even put into memory all at once. In pseudocode, I can think of something like this:
for i in file1
file3.write(file1.line(i) + ',' + file2.line(i))
end
But when I'm looping through a file using fgetcsv, it's not really clear how I would grab line n from a certain file without loading the whole thing into memory first.
Any ideas?
Edit: I forgot to mention that each of the two files has the same number of lines and they have a one-to-one relationship. That is, line 62,324 in file1 goes with line 62,324 in file2.
Not sure what operating system you're on, but if you're using Linux, using the paste command is probably a lot easier than trying to do this in PHP.
If this is a viable solution and you don't absolutely need to do it in PHP, you could try the following:
paste -d ',' file1 file2 > combined_file
Take a look at the fgets function. You could read a single line of each file, process them, and write them to your new file, then move on to the next line until you've reached the end of your file.
PHP: fgets
Specifically look at the example titled Example #1 Reading a file line by line in the PHP manual. It's also important to note the return value of the the fgets functions.
Returns a string of up to length - 1
bytes read from the file pointed to by
handle. If there is no more data to
read in the file pointer, then FALSE
is returned.
So, if it doesn't return FALSE you know you still have more lines to process.
You can use fgets().
$file1 = fopen('file1.txt', 'r');
$file2 = fopen('file2.txt', 'r');
$merged = fopen('merged.txt', 'w');
while (
($line1 = fgets($file1)) !== false
&& ($line2 = fgets($file2)) !== false) {
fwrite($merged, $line1 . ',' . $line2);
}
fgets() reads one line from a file. As you can see, this code uses it on both files at the same time, writing the merged lines to a third file. The manual here:
http://php.net/fgets
http://php.net/fopen
http://php.net/fwrite
Try using fgets() to read one line from each file at a time.
I think the solution for this is to map first line begins for each line ( and some kind of key if you need ) and then make a new csv using fread and fwrite ( we know beginning and ending of each line now , so we need just seek and read )
Another way is to put it into MySQL ( if it is possible ) and then back to new CSV