In Java I use Scanner to read text file line by line, so memory usage will be very low, because only one line is in memory once. Is there similar method in PHP?
use fseek, fgets
$handle = fopen("/tmp/uploadfile.txt", "r") or die("Couldn't get handle");
if ($handle) {
while (!feof($handle)) {
$line = fgets($handle);
// Process line here..
}
fclose($handle);
}
Reading very large files in PHP
fgets($fileHandle) is what you're looking for. You get the file handle using fopen("filename.txt", "r"), and close it with fclose($fileHandle).
Related
I am trying to read large .gz file line by line.
Here is what i got so far:
$sfp = gzopen($filename, "rb");
while (!gzeof($sfp))
{
$line = gzread($sfp, 4096);
}
and thats where the problem comes in, gzread reading the length specified in variable (in this case 4096) and ignoring new lines .
I checked "fget" function and it works properly, so its reading line limiting the size by hitting new line or the size which ever comes first.
How I can do the same with gzread or any work around ?
Use fgets() just like reading an ordinary file.
You shouldn't use b mode if you're reading lines, since line-by-line implies it's a text file.
$sfp = gzopen($filename, "r");
while ($line = fgets($sfp)) {
echo $line;
}
I have a 197gb text file that I want to read and push the contents into MySql database. I know, I can't put that big file in PHP buffer and read it as whole, So I want to read few hundred lines as a time and keep on reading next and next to read the whole file.
I am trying it with this but the page returns nothing
<?php
$i = 0;
$handle = fopen("./data/200gbfile.txt", "r") or die("Couldn't get handle");
if ($handle) {
while (($line = fgets($handle)) !== false) {
echo $line . "<br />";
if ($i > 100) {
exit;
}
$i++;
}
fclose($handle);
} else {
echo "Error Opeing File!";
}
?>
Is there a limit of the max file size to be handled in php setting?
EDIT: for the 197gb file in question, fopen is failing to return anything and
the output page is just going blank.
You can read the file in chunks to save memory:
For example:
$fd = #fopen("./data/200gbfile.txt", "r");
while (!feof($fd)) {
$data = fread($fd, 1024); // read the file in 1024kb chunks
// handle current data (read line by line for example)
}
fclose($fd);
But no idea if that works with a file with 100Gbytes+.
Edit: # with fopen is required as suggested by Roman.
you can use ini_set('memory_limit','16M'); to set size accordingly but i don't wether it will handle such huge file. never tested that..
I'm iterating through a handful of csv files (none that exceed more than 320MB) to clean up and format the files how I need them using the follow script:
while($row = mysqli_fetch_array($rsSelect)){
$fileName = $row["file_name"];
if(strpos($fileName,'DATA') == true){
$file = $dir.$row["manifest_files"]."";
echobr($file);
$file1 = file_get_contents($file, null);
unset($file);
$file2 = str_replace('","',' ', $file1);
unset($file1);
$file3 = str_replace('"','', $file2);
file_put_contents($dir.$row["file_name"].".txt",$file3,FILE_USE_INCLUDE_PATH);
unset($file2);
unset($file3);
}
I receive the following error no matter how high I set my memory limits within php.ini or unsetting variabeles where I can to free up memory. I've tried setting the limit inside the script as well and still no go. My machine has no shortage of RAM, enough to easily store when manipulating the files as I need them.
ERROR: ( ! ) Fatal error: Out of memory (allocated 683147264) (tried to allocate 364322204 bytes)
Using file_get_contents() requires the entire file to be read into memory in one big chunk. You're probably better off using fopen() with a loop around fgets(). This will read just one line at a time:
$fp = fopen('file.csv', 'r');
while (($line = fgets($fp)) !== false) {
// $line is the whole line
}
Alternatively, you might be able to use fgetcsv() and process individual fields as needed:
$fp = fopen('file.csv', 'r');
while (($row = fgetcsv($fp)) !== false) {
// $row is an array of values from the line
}
I am new to php.
I am trying to count the lines on a txt document, but this always returns me 1 (despite the fact that there are a lot more lines in the file):
<?php
$file = "example.txt";
$lines = count(file($file));
print "There are $lines lines in $file";
?>
Why do you think this is?
As a side note, I am using Mac OSx.
Thanks
Try this:
$file = "example.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
$line = fgets($handle);
$linecount++;
}
fclose($handle);
echo $linecount;
From the PHP manual ( http://www.php.net/manual/en/function.file.php ):
Note: If PHP is not properly recognizing the line endings when reading files
either on or created by a Macintosh computer, enabling the auto_detect_line_endings
run-time configuration option may help resolve the problem.
That may be the cause of it. Hard to say without more information.
This will use less memory, since it doesn't load the whole file into memory:
$file="largefile.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
$line = fgets($handle);
$linecount++;
}
fclose($handle);
echo $linecount;
fgets loads a single line into memory (if the second argument $length is omitted it will keep reading from the stream until it reaches the end of the line, which is what we want). This is still unlikely to be as quick as using something other than PHP, if you care about wall time as well as memory usage.
The only danger with this is if any lines are particularly long (what if you encounter a 2GB file without line breaks?). In which case you're better off doing slurping it in in chunks, and counting end-of-line characters:
$file="largefile.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
$line = fgets($handle, 4096);
$linecount = $linecount + substr_count($line, PHP_EOL);
}
fclose($handle);
echo $linecount;
I prefer the second code if I want to know only lines in the particular file
As I start the process of writing my site in PHP and MySQL, one of the first PHP scripts I've written is a script to initialize my database. Drop/create the database. Drop/create each of the tables. Then load the tables from literals in the script.
That's all working fine! Whoohoo :-)
But I would prefer to read the data from files rather than hard-code them in the PHP script.
I have a couple of books on PHP, but they're all oriented toward web development using MySQL. I can't find anything about reading and writing to ordinary files.
Yes, I know there's a gazillion questions here on stackoverflow about reading TXT files, but when I look at each one, they're for C or C# or VB or Perl. I'm beginning to think that PHP just can't read files :-(
All I need is a brief PHP example of how to open a TXT file on the server, read it sequentially, display the data on the screen, and close the file, as in this pseudo-code:
program readfile;
handle = open('myfile.txt');
data = read (handle);
while (not eof (handle)) begin
display data;
data = read (handle);
end;
close (handle);
end;
I will also need to write files on the server when I get to the part of my site where people upload avatars, and save them as JPG or GIF files. But that's for later.
Thanks!
From the PHP manual for fread():
<?php
// get contents of a file into a string
$filename = "/usr/local/something.txt";
$handle = fopen($filename, "r");
$contents = fread($handle, filesize($filename));
fclose($handle);
?>
EDIT
per the comment, you can read a file line by line with fgets()
<?php
$handle = #fopen("/tmp/inputfile.txt", "r");
if ($handle) {
while (($buffer = fgets($handle, 4096)) !== false) {
echo $buffer;
}
if (!feof($handle)) {
echo "Error: unexpected fgets() fail\n";
}
fclose($handle);
}
?>
All I need is a brief PHP example of how to open a TXT file on the server, read it sequentially, display the data on the screen, and close the file, as in this pseudo-code:
echo file_get_contents('/path/to/file.txt');
Yes that brief, see file_get_contents, you normally don't need a loop:
$file = new SPLFileObject('/path/to/file.txt');
foreach($file as $line) {
echo $line;
}
Well, since you're asking about resources on the subject, there's a whole book on it in the PHP.net docs.
A basic example:
<?php
// get contents of a file into a string
$filename = "/usr/local/something.txt";
$handle = fopen($filename, "r");
$contents = fread($handle, filesize($filename));
fclose($handle);
?>
Why you not read php documentation about fopen
$file = fopen("source/file.txt","r");
if(!file)
{
echo("ERROR:cant open file");
}
else
{
$buff = fread ($file,filesize("source/file.txt"));
print $buff;
}
file_get_contents does all that for you and returns the text file in a string :)
You want to read line by line? Use fgets.
$handle = #fopen("myfile.txt", "r");
if ($handle) {
while (($content = fgets($handle, 4096)) !== false) {
//echo $content;
}
if (!feof($handle)) {
echo "Error: unexpected fgets() fail\n";
}
fclose($handle);
}