Do you have any idea how to improve the performance by reading CSV file from a zip file?
Firstly it open the zip file, then put the data into a memory and then read it by fgetcsv
$zip = new ZipArchive();
if ($zip->open($fileName)) {
$info = $zip->statIndex(0);
$fp = $zip->getStream($info['name']);
if(!$fp) exit("failed\n");
while (!feof($fp)) {
$contents .= fread($fp, 2);
}
fclose($fp);
$zip->close();
}
$temp = fopen("php://memory", "rw");
fwrite($temp, $contents);
fseek($temp, 0);
while (($data = fgetcsv($temp, 0)) !== false) {
....
}
Quick check with php manual showed that this should work:
<?php
$fp = fopen('zip://test.zip#test', 'r'); // test name of file in archive
if (!$fp) {
exit("cannot open\n");
}
while (($data = fgetcsv($fp, 0)) !== false) {
...
}
fclose($fp);
Related
I want to read a file line by line, but without completely loading it in memory.
My file is too large to open in memory, and if try to do so I always get out of memory errors.
The file size is 1 GB.
You can use the fgets() function to read the file line by line:
$handle = fopen("inputfile.txt", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line read.
}
fclose($handle);
}
if ($file = fopen("file.txt", "r")) {
while(!feof($file)) {
$line = fgets($file);
# do same stuff with the $line
}
fclose($file);
}
You can use an object oriented interface class for a file - SplFileObject http://php.net/manual/en/splfileobject.fgets.php (PHP 5 >= 5.1.0)
<?php
$file = new SplFileObject("file.txt");
// Loop until we reach the end of the file.
while (!$file->eof()) {
// Echo one line from the file.
echo $file->fgets();
}
// Unset the file to call __destruct(), closing the file handle.
$file = null;
If you want to use foreach instead of while when opening a big file, you probably want to encapsulate the while loop inside a Generator to avoid loading the whole file into memory:
/**
* #return Generator
*/
$fileData = function() {
$file = fopen(__DIR__ . '/file.txt', 'r');
if (!$file) {
return; // die() is a bad practice, better to use return
}
while (($line = fgets($file)) !== false) {
yield $line;
}
fclose($file);
};
Use it like this:
foreach ($fileData() as $line) {
// $line contains current line
}
This way you can process individual file lines inside the foreach().
Note: Generators require >= PHP 5.5
There is a file() function that returns an array of the lines contained in the file.
foreach(file('myfile.txt') as $line) {
echo $line. "\n";
}
The obvious answer wasn't there in all the responses.
PHP has a neat streaming delimiter parser available made for exactly that purpose.
$fp = fopen("/path/to/the/file", "r");
while (($line = stream_get_line($fp, 1024 * 1024, "\n")) !== false) {
echo $line;
}
fclose($fp);
Use buffering techniques to read the file.
$filename = "test.txt";
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer = fread($source_file, 4096); // use a buffer of 4KB
$buffer = str_replace($old,$new,$buffer);
///
}
foreach (new SplFileObject(__FILE__) as $line) {
echo $line;
}
One of the popular solutions to this question will have issues with the new line character. It can be fixed pretty easy with a simple str_replace.
$handle = fopen("some_file.txt", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
$line = str_replace("\n", "", $line);
}
fclose($handle);
}
This how I manage with very big file (tested with up to 100G). And it's faster than fgets()
$block =1024*1024;//1MB or counld be any higher than HDD block_size*2
if ($fh = fopen("file.txt", "r")) {
$left='';
while (!feof($fh)) {// read the file
$temp = fread($fh, $block);
$fgetslines = explode("\n",$temp);
$fgetslines[0]=$left.$fgetslines[0];
if(!feof($fh) )$left = array_pop($lines);
foreach ($fgetslines as $k => $line) {
//do smth with $line
}
}
}
fclose($fh);
Be careful with the 'while(!feof ... fgets()' stuff, fgets can get an error (returnfing false) and loop forever without reaching the end of file. codaddict was closest to being correct but when your 'while fgets' loop ends, check feof; if not true, then you had an error.
SplFileObject is useful when it comes to dealing with large files.
function parse_file($filename)
{
try {
$file = new SplFileObject($filename);
} catch (LogicException $exception) {
die('SplFileObject : '.$exception->getMessage());
}
while ($file->valid()) {
$line = $file->fgets();
//do something with $line
}
//don't forget to free the file handle.
$file = null;
}
<?php
echo '<meta charset="utf-8">';
$k= 1;
$f= 1;
$fp = fopen("texttranslate.txt", "r");
while(!feof($fp)) {
$contents = '';
for($i=1;$i<=1500;$i++){
echo $k.' -- '. fgets($fp) .'<br>';$k++;
$contents .= fgets($fp);
}
echo '<hr>';
file_put_contents('Split/new_file_'.$f.'.txt', $contents);$f++;
}
?>
Function to Read with array return
function read_file($filename = ''){
$buffer = array();
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer[] = fread($source_file, 4096); // use a buffer of 4KB
}
return $buffer;
}
PHP script who open and search data from .txt is:
function explodeRows($data) {
$rowsArr = explode("\n", $data);
return $rowsArr;
}
function explodeTabs($singleLine) {
$tabsArr = explode("\t", $singleLine);
return $tabsArr;
}
$filename = "/txt/name.txt";
$handle = fopen($filename, 'r');
$data = fread($handle, filesize($filename));
$rowsArr = explodeRows($data);
for($i=0;$i<count($rowsArr);$i++) {
$lineDetails = explode("|",$rowsArr[$i]);
if ($kodas == $lineDetails[2]) {
$link3=$lineDetails[4];
echo "";
} }
fclose($handle);
It's works well, but now I transfer name.txt to another folder (folder name txt). How to make, first open this folder and search open name.txt
$filename = "txt/name.txt";
$handle = fopen($filename, 'r');
$data = fread($handle, filesize($filename));
$rowsArr = explodeRows($data);
Code:
$filename = 'Master_List_DeDuped.csv';
$fp = fopen($filename, "r");
while (false !== ($line = fgets($fp))) {
echo $line;
die(); // For Debugging only
}
fclose($fp);
The resulting error:
Warning: fgets(): 3 is not a valid stream resource in /home3/public_html/index.php on line 288
Line 288 is the while statement. The same commands work fine with a smaller file. My file is about 1.1 gigs. Is it just a file size limitation?
Edit: I've tried adding the length parameter to fgets, but the same error shows. http://us2.php.net/fgets
Changed the code a bit based on the example at php.net http://us2.php.net/fgets . The code that works is:
$filename = 'Master_List_DeDuped.csv';
$fp = #fopen($filename, "r");
if ($fp) {
while (($line = fgets($fp, 4096)) !== false) {
echo $line;
die(); // For Debugging only
}
}
fclose($fp);
I want to read a file line by line, but without completely loading it in memory.
My file is too large to open in memory, and if try to do so I always get out of memory errors.
The file size is 1 GB.
You can use the fgets() function to read the file line by line:
$handle = fopen("inputfile.txt", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line read.
}
fclose($handle);
}
if ($file = fopen("file.txt", "r")) {
while(!feof($file)) {
$line = fgets($file);
# do same stuff with the $line
}
fclose($file);
}
You can use an object oriented interface class for a file - SplFileObject http://php.net/manual/en/splfileobject.fgets.php (PHP 5 >= 5.1.0)
<?php
$file = new SplFileObject("file.txt");
// Loop until we reach the end of the file.
while (!$file->eof()) {
// Echo one line from the file.
echo $file->fgets();
}
// Unset the file to call __destruct(), closing the file handle.
$file = null;
If you want to use foreach instead of while when opening a big file, you probably want to encapsulate the while loop inside a Generator to avoid loading the whole file into memory:
/**
* #return Generator
*/
$fileData = function() {
$file = fopen(__DIR__ . '/file.txt', 'r');
if (!$file) {
return; // die() is a bad practice, better to use return
}
while (($line = fgets($file)) !== false) {
yield $line;
}
fclose($file);
};
Use it like this:
foreach ($fileData() as $line) {
// $line contains current line
}
This way you can process individual file lines inside the foreach().
Note: Generators require >= PHP 5.5
There is a file() function that returns an array of the lines contained in the file.
foreach(file('myfile.txt') as $line) {
echo $line. "\n";
}
The obvious answer wasn't there in all the responses.
PHP has a neat streaming delimiter parser available made for exactly that purpose.
$fp = fopen("/path/to/the/file", "r");
while (($line = stream_get_line($fp, 1024 * 1024, "\n")) !== false) {
echo $line;
}
fclose($fp);
Use buffering techniques to read the file.
$filename = "test.txt";
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer = fread($source_file, 4096); // use a buffer of 4KB
$buffer = str_replace($old,$new,$buffer);
///
}
foreach (new SplFileObject(__FILE__) as $line) {
echo $line;
}
One of the popular solutions to this question will have issues with the new line character. It can be fixed pretty easy with a simple str_replace.
$handle = fopen("some_file.txt", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
$line = str_replace("\n", "", $line);
}
fclose($handle);
}
This how I manage with very big file (tested with up to 100G). And it's faster than fgets()
$block =1024*1024;//1MB or counld be any higher than HDD block_size*2
if ($fh = fopen("file.txt", "r")) {
$left='';
while (!feof($fh)) {// read the file
$temp = fread($fh, $block);
$fgetslines = explode("\n",$temp);
$fgetslines[0]=$left.$fgetslines[0];
if(!feof($fh) )$left = array_pop($lines);
foreach ($fgetslines as $k => $line) {
//do smth with $line
}
}
}
fclose($fh);
Be careful with the 'while(!feof ... fgets()' stuff, fgets can get an error (returnfing false) and loop forever without reaching the end of file. codaddict was closest to being correct but when your 'while fgets' loop ends, check feof; if not true, then you had an error.
SplFileObject is useful when it comes to dealing with large files.
function parse_file($filename)
{
try {
$file = new SplFileObject($filename);
} catch (LogicException $exception) {
die('SplFileObject : '.$exception->getMessage());
}
while ($file->valid()) {
$line = $file->fgets();
//do something with $line
}
//don't forget to free the file handle.
$file = null;
}
<?php
echo '<meta charset="utf-8">';
$k= 1;
$f= 1;
$fp = fopen("texttranslate.txt", "r");
while(!feof($fp)) {
$contents = '';
for($i=1;$i<=1500;$i++){
echo $k.' -- '. fgets($fp) .'<br>';$k++;
$contents .= fgets($fp);
}
echo '<hr>';
file_put_contents('Split/new_file_'.$f.'.txt', $contents);$f++;
}
?>
Function to Read with array return
function read_file($filename = ''){
$buffer = array();
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer[] = fread($source_file, 4096); // use a buffer of 4KB
}
return $buffer;
}
My file contains this content (1.txt):
user.php?id=XXXXXX
user.php?id=XXXXXX
user.php?id=XXXXXX
user.php?id=XXXXXX
user.php?id=XXXXXX
text/numbers
user.php?id=XXXXXX
user.php?id=XXXXXX
and so on ...
(XXXXXX = numbers)
It's possible to take only user.php?id=XXXXXX and copy them to another file, without coping unnecessary text because file contains about 50 000 lines ?
<?php
$file = fopen('source.txt', 'rb');
$newfile = fopen('target.txt', 'wb');
while(($line = fgets($file)) !== false) {
if(strpos($line, 'user.php') !== false) {
fputs($newfile, $line);
}
}
fclose($newfile);
fclose($file);
?>
This code hasn't been tested, but I think it will work properly.
<?php
//Get all the matches from the file
$fileContents = file_get_contents('1.txt');
preg_match_all('/user.php\?id=[0-9]{6}/', $fileContents, $matches);
//Output to new file
$fh = fopen('output.txt', 'w+');
foreach ($matches['0'] as $match) {
fputs($fh, $match."\r\n");
}
fclose($fh);
?>
<?php
error_reporting(0);
$fs=fopen("1.txt", "r");
$ft=fopen("2.txt", "w");
if ($fs==NULL)
{
echo "Can't Open Source File ...";
exit(0);
}
if ($ft==NULL)
{
echo "Can't Open Destination File ...";
fclose ($fs);
exit(1);
}
else
{
while ($ch=fgets($fs))
fputs($ft, $ch);
fclose ($fs);
fclose ($ft);
}
//echo "File Handling successfully ...";
?>
suppose you want to copy the contents of "abc.php" file to "working_file.php"
place the following code in working_file.php , it will simply copy all the content.
<div>
<?php
$file = fopen("abc.php", "r") or die("Unable to open file!");
echo fread($myfile,filesize("abc.php"));
fclose($file);
?>
</div>