I have a 197gb text file that I want to read and push the contents into MySql database. I know, I can't put that big file in PHP buffer and read it as whole, So I want to read few hundred lines as a time and keep on reading next and next to read the whole file.
I am trying it with this but the page returns nothing
<?php
$i = 0;
$handle = fopen("./data/200gbfile.txt", "r") or die("Couldn't get handle");
if ($handle) {
while (($line = fgets($handle)) !== false) {
echo $line . "<br />";
if ($i > 100) {
exit;
}
$i++;
}
fclose($handle);
} else {
echo "Error Opeing File!";
}
?>
Is there a limit of the max file size to be handled in php setting?
EDIT: for the 197gb file in question, fopen is failing to return anything and
the output page is just going blank.
You can read the file in chunks to save memory:
For example:
$fd = #fopen("./data/200gbfile.txt", "r");
while (!feof($fd)) {
$data = fread($fd, 1024); // read the file in 1024kb chunks
// handle current data (read line by line for example)
}
fclose($fd);
But no idea if that works with a file with 100Gbytes+.
Edit: # with fopen is required as suggested by Roman.
you can use ini_set('memory_limit','16M'); to set size accordingly but i don't wether it will handle such huge file. never tested that..
Hello i have this code:
file1 = file_get_contents("read.txt");
$path2 = "write.txt";
$file2 = file_get_contents($path2);
if ($file1 !== $file2){
file_put_contents($path2, $file1);
echo "working";
}
how can i get the first 10000 lines or more from the read.txt file and write them in write.txt ?
There are many ways you could do this reading the entire file, but it would be better to use streams and only read the data you need.
<?php
$source="file.txt";
$destination="file2.txt";
$requiredLines=10000;
//compare the modification times, if source is newer than destination - then we do our work
if(filemtime($source)>filemtime($destination)){
//work out maximum length of file, as one line may be the whole file.
$filesize = filesize($source);
//open file for reading - this doesnt actually read the file it allows us to "stream" it
$sourceHandle = fopen($source, "r");
//open file for writing
$destinationHandle = fopen($destination, "w");
$linecount=0;
//loop through file until we reach the end of the file (feof) or we reach the desired number of lines
while (!feof($sourceHandle) && $linecount++<$requiredLines) {
//read one line
$line = stream_get_line($sourceHandle, $filesize, "\n");
//write the line
fwrite($destinationHandle,$line);
}
//close both files
fclose($sourceHandle);
fclose($destinationHandle);
}
You can find more info on streams here: Understanding PHP Streams
I am new to php.
I am trying to count the lines on a txt document, but this always returns me 1 (despite the fact that there are a lot more lines in the file):
<?php
$file = "example.txt";
$lines = count(file($file));
print "There are $lines lines in $file";
?>
Why do you think this is?
As a side note, I am using Mac OSx.
Thanks
Try this:
$file = "example.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
$line = fgets($handle);
$linecount++;
}
fclose($handle);
echo $linecount;
From the PHP manual ( http://www.php.net/manual/en/function.file.php ):
Note: If PHP is not properly recognizing the line endings when reading files
either on or created by a Macintosh computer, enabling the auto_detect_line_endings
run-time configuration option may help resolve the problem.
That may be the cause of it. Hard to say without more information.
This will use less memory, since it doesn't load the whole file into memory:
$file="largefile.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
$line = fgets($handle);
$linecount++;
}
fclose($handle);
echo $linecount;
fgets loads a single line into memory (if the second argument $length is omitted it will keep reading from the stream until it reaches the end of the line, which is what we want). This is still unlikely to be as quick as using something other than PHP, if you care about wall time as well as memory usage.
The only danger with this is if any lines are particularly long (what if you encounter a 2GB file without line breaks?). In which case you're better off doing slurping it in in chunks, and counting end-of-line characters:
$file="largefile.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
$line = fgets($handle, 4096);
$linecount = $linecount + substr_count($line, PHP_EOL);
}
fclose($handle);
echo $linecount;
I prefer the second code if I want to know only lines in the particular file
In Java I use Scanner to read text file line by line, so memory usage will be very low, because only one line is in memory once. Is there similar method in PHP?
use fseek, fgets
$handle = fopen("/tmp/uploadfile.txt", "r") or die("Couldn't get handle");
if ($handle) {
while (!feof($handle)) {
$line = fgets($handle);
// Process line here..
}
fclose($handle);
}
Reading very large files in PHP
fgets($fileHandle) is what you're looking for. You get the file handle using fopen("filename.txt", "r"), and close it with fclose($fileHandle).
I'm trying to read a specific line from a text file using php.
Here's the text file:
foo
foo2
How would I get the content of the second line using php?
This returns the first line:
<?php
$myFile = "4-24-11.txt";
$fh = fopen($myFile, 'r');
$theData = fgets($fh);
fclose($fh);
echo $theData;
?>
..but I need the second.
Any help would be greatly appreciated
$myFile = "4-24-11.txt";
$lines = file($myFile);//file in to an array
echo $lines[1]; //line 2
file — Reads entire file into an array
omg I'm lacking 7 rep to make comments. This is #Raptor's & #Tomm's comment, since this question still shows up way high in google serps.
He's exactly right. For small files file($file); is perfectly fine. For large files it's total overkill b/c php arrays eat memory like crazy.
I just ran a tiny test with a *.csv with a file size of ~67mb (1,000,000 lines):
$t = -microtime(1);
$file = '../data/1000k.csv';
$lines = file($file);
echo $lines[999999]
."\n".(memory_get_peak_usage(1)/1024/1024)
."\n".($t+microtime(1));
//227.5
//0.22701287269592
//Process finished with exit code 0
And since noone mentioned it yet, I gave the SplFileObject a try, which I actually just recently discovered for myself.
$t = -microtime(1);
$file = '../data/1000k.csv';
$spl = new SplFileObject($file);
$spl->seek(999999);
echo $spl->current()
."\n".(memory_get_peak_usage(1)/1024/1024)
."\n".($t+microtime(1));
//0.5
//0.11500692367554
//Process finished with exit code 0
This was on my Win7 desktop so it's not representative for production environment, but still ... quite the difference.
If you wanted to do it that way...
$line = 0;
while (($buffer = fgets($fh)) !== FALSE) {
if ($line == 1) {
// This is the second line.
break;
}
$line++;
}
Alternatively, open it with file() and subscript the line with [1].
I would use the SplFileObject class...
$file = new SplFileObject("filename");
if (!$file->eof()) {
$file->seek($lineNumber);
$contents = $file->current(); // $contents would hold the data from line x
}
you can use the following to get all the lines in the file
$handle = #fopen('test.txt', "r");
if ($handle) {
while (!feof($handle)) {
$lines[] = fgets($handle, 4096);
}
fclose($handle);
}
print_r($lines);
and $lines[1] for your second line
$myFile = "4-21-11.txt";
$fh = fopen($myFile, 'r');
while(!feof($fh))
{
$data[] = fgets($fh);
//Do whatever you want with the data in here
//This feeds the file into an array line by line
}
fclose($fh);
This question is quite old by now, but for anyone dealing with very large files, here is a solution that does not involve reading every preceding line. This was also the only solution that worked in my case for a file with ~160 million lines.
<?php
function rand_line($fileName) {
do{
$fileSize=filesize($fileName);
$fp = fopen($fileName, 'r');
fseek($fp, rand(0, $fileSize));
$data = fread($fp, 4096); // assumes lines are < 4096 characters
fclose($fp);
$a = explode("\n",$data);
}while(count($a)<2);
return $a[1];
}
echo rand_line("file.txt"); // change file name
?>
It works by opening the file without reading anything, then moving the pointer instantly to a random position, reading up to 4096 characters from that point, then grabbing the first complete line from that data.
If you use PHP on Linux, you may try the following to read text for example between 74th and 159th lines:
$text = shell_exec("sed -n '74,159p' path/to/file.log");
This solution is good if your file is large.
You have to loop the file till end of file.
while(!feof($file))
{
echo fgets($file). "<br />";
}
fclose($file);
Use stream_get_line: stream_get_line — Gets line from stream resource up to a given delimiter
Source: http://php.net/manual/en/function.stream-get-line.php
You could try looping until the line you want, not the EOF, and resetting the variable to the line each time (not adding to it). In your case, the 2nd line is the EOF. (A for loop is probably more appropriate in my code below).
This way the entire file is not in the memory; the drawback is it takes time to go through the file up to the point you want.
<?php
$myFile = "4-24-11.txt";
$fh = fopen($myFile, 'r');
$i = 0;
while ($i < 2)
{
$theData = fgets($fh);
$i++
}
fclose($fh);
echo $theData;
?>
I like daggett answer but there is another solution you can get try if your file is not big enough.
$file = __FILE__; // Let's take the current file just as an example.
$start_line = __LINE__ -1; // The same with the line what we look for. Take the line number where $line variable is declared as the start.
$lines_to_display = 5; // The number of lines to display. Displays only the $start_line if set to 1. If $lines_to_display argument is omitted displays all lines starting from the $start_line.
echo implode('', array_slice(file($file), $start_line, lines_to_display));
I searched for a one line solution to read specific line from a file.
Here my solution:
echo file('dayInt.txt')[1]