I have a PHP script that allow users to upload their data. The first line of the csv file are the headers (fname, lname, age, address, email).
My plan is - after the users uploaded their csv, my script will run a function to check the spelling of the headers. If there are misspelled header, my script will correct it. I am using the code below to correct the headers:
if (($file = fopen($csvFile , "r")) != FALSE) {
$ctr = 0;
$record = fgetcsv($file, 1024)) != FALSE) {
if ($ctr == 0) {
correctHeader($record);
# write to new csv.
} else {
# write to new csv.
}
}
}
After correcting, the value of the header and the succeeding lines will be appended on the new csv file. I think this step can be optimized, if I could just edit the first line of the csv (header) and skip the # write to new csv step.
One of the ways I can think of is the following:
Use fgets() to get the first line of the file (instead of fgetcsv()).
Save the length of the line in bytes.
Parse the line with str_getcsv().
Correct headers as needed.
Save headers into a new CSV file.
fopen() the original CSV file for reading.
fseek() the original CSV file handle to length of first line (saved in step 2) + 1.
fopen() the new CSV file for writing (appending actually).
fread() the original CSV file in the loop until EOF and fwrite() chunks into the new CSV file.
Fix bugs.
Have a pint. :)
Here's the code (minus the loop for reading):
<?php
$from = 'd.csv';
$to = 'd.good.csv';
$old = fopen($from, 'r');
if (!is_resource($old)) {
die("Failed to read from source file: $from");
}
$headerLine = fgets($old);
$headerLine = fixHeaders($headerLine);
$new = fopen($to, 'w');
if (!is_resource($new)) {
die("Failed to write to destination file: $new");
}
// Save the fixed header into the new file
fputs($new, $headerLine);
// Read the rest of old and save to new.
// Old file is already open and we are the second line.
// For large files, reading should probably be done in the loop with chunks.
fwrite($new, fread($old, filesize($from)));
// Close files
fclose($old);
fclose($new);
// Just an example
function fixHeaders($line) {
return strtoupper($line);
}
Related
This question already has answers here:
Reading very large files in PHP
(8 answers)
Closed 1 year ago.
I have a file with around 100 records for now.
The file has users in json format per line.
Eg
{"user_id" : 1,"user_name": "Alex"}
{"user_id" : 2,"user_name": "Bob"}
{"user_id" : 3,"user_name": "Mark"}
Note : This is a just very simple example, I have more complex json values per line in the file.
I am reading the file line by line and store that in an array which obviously will be big if there are a lot of items in the file.
public function read(string $file) : array
{
//Open the file in "reading only" mode.
$fileHandle = fopen($file, "r");
//If we failed to get a file handle, throw an Exception.
if ($fileHandle === false) {
throw new Exception('Could not get file handle for: ' . $file);
}
$lines = [];
//While we haven't reach the end of the file.
while (!feof($fileHandle)) {
//Read the current line in.
$lines[] = json_decode(fgets($fileHandle));
}
//Finally, close the file handle.
fclose($fileHandle);
return $lines;
}
Next, Ill process this array and only take the parameters I need (some parameters might be further processed) and then Ill export this array to csv.
public function processInput($users){
$data = [];
foreach ($users as $key => $user)
{
$data[$key]['user_id'] = $user->user_id;
$data[$key]['user_name'] = strtoupper($user->user_name);
}
// Call export to csv $data.
}
What should be the best way to read the file (incase we have a big file)?
I know file_get_contents is not optimized way and instead fgets is a better approach.
Is there a much better way considering big file read and then put it to csv.
You need to modify your reader to make it more "lazy" in some sense. For example consider this:
public function read(string $file, callable $rowProcessor) : void
{
//Open the file in "reading only" mode.
$fileHandle = fopen($file, "r");
//If we failed to get a file handle, throw an Exception.
if ($fileHandle === false) {
throw new Exception('Could not get file handle for: ' . $file);
}
//While we haven't reach the end of the file.
while (!feof($fileHandle)) {
//Read the current line in.
$line = json_decode(fgets($fileHandle));
$rowProcessor($line);
}
//Finally, close the file handle.
fclose($fileHandle);
return $lines;
}
Then your will need different code that works with this:
function processAndWriteJson($filename) { //Names are hard
$writer = fopen('output.csv', 'w');
read($filename, function ($row) use ($writer) {
// Do processing of the single row here
fputcsv($writer, $processedRow);
});
}
If you want to get the same result as before with your read method you can do:
$lines = [];
read($filename, function ($row) use ($writer) {
$lines[] = $row;
});
It does provide some more flexibility. Unfortunately it does mean you can only process one line at a time and scanning up and down the file is harder
Hello i have this code:
file1 = file_get_contents("read.txt");
$path2 = "write.txt";
$file2 = file_get_contents($path2);
if ($file1 !== $file2){
file_put_contents($path2, $file1);
echo "working";
}
how can i get the first 10000 lines or more from the read.txt file and write them in write.txt ?
There are many ways you could do this reading the entire file, but it would be better to use streams and only read the data you need.
<?php
$source="file.txt";
$destination="file2.txt";
$requiredLines=10000;
//compare the modification times, if source is newer than destination - then we do our work
if(filemtime($source)>filemtime($destination)){
//work out maximum length of file, as one line may be the whole file.
$filesize = filesize($source);
//open file for reading - this doesnt actually read the file it allows us to "stream" it
$sourceHandle = fopen($source, "r");
//open file for writing
$destinationHandle = fopen($destination, "w");
$linecount=0;
//loop through file until we reach the end of the file (feof) or we reach the desired number of lines
while (!feof($sourceHandle) && $linecount++<$requiredLines) {
//read one line
$line = stream_get_line($sourceHandle, $filesize, "\n");
//write the line
fwrite($destinationHandle,$line);
}
//close both files
fclose($sourceHandle);
fclose($destinationHandle);
}
You can find more info on streams here: Understanding PHP Streams
I need to login to bunch of servers tail a log and write the output to a file (output needs to be appended to the file so that the file contains entry from each server). Here is what I have so far:
<?php
chdir('c:/php');
include('Net/SSH2.php');
$servers=array('server1','server2','server3','server4');
foreach ($servers as &$value1) {
$server=$value1.".example.com";
$ssh = new Net_SSH2($server);
if (!$ssh->login('user', 'passwd')) {
echo 'unable to login '.$value1;
continue;
}
$file_name="/apps/logs/was/gws1/perf.log";
$line= $ssh->exec("tail -50 $file_name");
$rawLines = explode("\n", $line);
$lines = array();
$fp = fopen('C:/perf_log.csv', 'w');
echo $rawLines;
fputcsv($fp, $rawLines);
fclose($fp);
}
exit
?>
this is creating one line in the file. I need each line in the $rawFiles to be new line in the perf_log.csv. Any ideas how I could do this in php?
You're using w when opening the file, which is truncating any previously existing lines:
Open for writing only; place the file pointer at the beginning of the file and truncate the file to zero length. If the file does not exist, attempt to create it.
What you actually want is a:
Open for reading and writing; place the file pointer at the end of the file. If the file does not exist, attempt to create it.
I have a big CSV file. I want to separate this file into separate files based on the value in one of the fields.
This is what I have done. Using fgetcsv I convert the CSV into an array, and using in_array, I check the content and display if it contains the string within the array.
I will be getting the comparison string from another text file iteratively to check whether it is contained in the csv. In this case I have specified it as "Testing".
Below is the code:
if (($handle = fopen("test.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
if(in_array("Testing", $data))
{
var_dump($data);
}
}
fclose($handle);
}
This is working, but now I am stuck. How do I write $data into another CSV file? Or is there a better way to do this?
It's actually pretty simple and if the string just has to be on the line, you don't even need fgetcsv. Just
$srcFile = new SplFileObject('test.csv');
$destFile = new SplFileObject('new.csv', 'w+');
foreach ($srcFile as $line) {
if (strpos($line, 'testing') !== FALSE) {
$destFile->fwrite($line);
}
}
This will create two file objects. The first one holding the content of your source file. The second one creating an all new file for the lines containing your search string. We then just iterate over each line and check if the search string exists. If so, we write it to destination file.
The source file will not be touched this way. If you want to have one file with the search string and one file without, just create a third SplFileObject and add an else block to the if writing the line to that one then. In the end, delete the source csv file.
You have to do some tricky thing I am providing some basic idea for doing so, here is the code:
//opening file
if ($fp = fopen('log.csv', 'r')) {
$line_number = 0;
//loop for Reading file as line by line csv file
while ($line = fgetcsv($fp, 0, ';')) {
if ($line_number++ == 0) {
continue;
}
//array data string to make possible to provide file name
//according to column name required
$date = explode(' ', $line[0]);
//Change the column name according to your needs
$file = $date[0] .'.log';
file_put_contents(
//change the folder name according to your needs
'monthly/'. $file,
//printing data in appended file
implode(';', $line) ."\n",
FILE_APPEND
);
}
//closing file
fclose($fp);
}
It reads CSV file line by line, extracts date part from the first column and creates new file and appends data to it.
Note:
folder "monthly" must be writable
I have:
<?php
$file=fopen(date("Y-m-d").".txt","r+") or exit("Unable to open file!");
if ($_POST["lastname"] <> "")
{
fwrite($file,$_POST["lastname"]."\n");
}
fclose($file);
?>
but it overwrites the beginning of the file. How do I make it insert?
I'm not entirely sure of your question - do you want to write data and not have it over-write the beginning of an existing file, or write new data to the start of an existing file, keeping the existing content after it?
To insert text without over-writing the beginning of the file, you'll have to open it for appending (a+ rather than r+)
$file=fopen(date("Y-m-d").".txt","a+") or exit("Unable to open file!");
if ($_POST["lastname"] <> "")
{
fwrite($file,$_POST["lastname"]."\n");
}
fclose($file);
If you're trying to write to the start of the file, you'll have to read in the file contents (see file_get_contents) first, then write your new string followed by file contents to the output file.
$old_content = file_get_contents($file);
fwrite($file, $new_content."\n".$old_content);
The above approach will work with small files, but you may run into memory limits trying to read a large file in using file_get_conents. In this case, consider using rewind($file), which sets the file position indicator for handle to the beginning of the file stream.
Note when using rewind(), not to open the file with the a (or a+) options, as:
If you have opened the file in append ("a" or "a+") mode, any data you write to the file will always be appended, regardless of the file position.
A working example for inserting in the middle of a file stream without overwriting, and without having to load the whole thing into a variable/memory:
function finsert($handle, $string, $bufferSize = 16384) {
$insertionPoint = ftell($handle);
// Create a temp file to stream into
$tempPath = tempnam(sys_get_temp_dir(), "file-chainer");
$lastPartHandle = fopen($tempPath, "w+");
// Read in everything from the insertion point and forward
while (!feof($handle)) {
fwrite($lastPartHandle, fread($handle, $bufferSize), $bufferSize);
}
// Rewind to the insertion point
fseek($handle, $insertionPoint);
// Rewind the temporary stream
rewind($lastPartHandle);
// Write back everything starting with the string to insert
fwrite($handle, $string);
while (!feof($lastPartHandle)) {
fwrite($handle, fread($lastPartHandle, $bufferSize), $bufferSize);
}
// Close the last part handle and delete it
fclose($lastPartHandle);
unlink($tempPath);
// Re-set pointer
fseek($handle, $insertionPoint + strlen($string));
}
$handle = fopen("file.txt", "w+");
fwrite($handle, "foobar");
rewind($handle);
finsert($handle, "baz");
// File stream is now: bazfoobar
Composer lib for it can be found here
You get the same opening the file for appending
<?php
$file=fopen(date("Y-m-d").".txt","a+") or exit("Unable to open file!");
if ($_POST["lastname"] <> "")
{
fwrite($file,$_POST["lastname"]."\n");
}
fclose($file);
?>
If you want to put your text at the beginning of the file, you'd have to read the file contents first like:
<?php
$file=fopen(date("Y-m-d").".txt","r+") or exit("Unable to open file!");
if ($_POST["lastname"] <> "")
{
$existingText = file_get_contents($file);
fwrite($file, $existingText . $_POST["lastname"]."\n");
}
fclose($file);
?>