I have a 197gb text file that I want to read and push the contents into MySql database. I know, I can't put that big file in PHP buffer and read it as whole, So I want to read few hundred lines as a time and keep on reading next and next to read the whole file.
I am trying it with this but the page returns nothing
<?php
$i = 0;
$handle = fopen("./data/200gbfile.txt", "r") or die("Couldn't get handle");
if ($handle) {
while (($line = fgets($handle)) !== false) {
echo $line . "<br />";
if ($i > 100) {
exit;
}
$i++;
}
fclose($handle);
} else {
echo "Error Opeing File!";
}
?>
Is there a limit of the max file size to be handled in php setting?
EDIT: for the 197gb file in question, fopen is failing to return anything and
the output page is just going blank.
You can read the file in chunks to save memory:
For example:
$fd = #fopen("./data/200gbfile.txt", "r");
while (!feof($fd)) {
$data = fread($fd, 1024); // read the file in 1024kb chunks
// handle current data (read line by line for example)
}
fclose($fd);
But no idea if that works with a file with 100Gbytes+.
Edit: # with fopen is required as suggested by Roman.
you can use ini_set('memory_limit','16M'); to set size accordingly but i don't wether it will handle such huge file. never tested that..
Related
When I run this function on multiple scripts one script generated warning:
fread(): Length parameter must be greater than 0
function test($n){
echo "<h4>$n at ".time()."</h4>";
for ($i = 0; $i<50; $i++ ){
$fp = fopen("$n.txt", "r");
$s = fread($fp, filesize("$n.txt") );
fclose($fp);
$fp = fopen("$n.txt", "w");
$s = $_SERVER['HTTP_USER_AGENT'].' '.time();
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
fwrite($fp, $s);
// fflush($fp);// flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
}
}
I try to write reading of the file for multiple users, but only one user can write the file. I know that when I use fwrite with flock - LOC_EX, next scripts must wait till the write is finished. But here it seems like filesize doesn't wait till the write operation is finished. My opinion is that it tries to reach the file when the file size is 0, and as a result this produces the problem: 0 bytes will be read from the file, when it is written by original script.
Is it possible to fix this for fread function?
Purpose of this script is to test fread with some limit and to check the data which I read later, if the data are really written when I did not used fflush.
function test($n){
echo "<h4>$n at ".time()."</h4>";
for ($i = 0; $i<50; $i++ ){
$start = microtime(true);
$fp = fopen("$n.txt", "r");
if(filesize($n.txt) > 0)
{
$s = fread($fp, filesize($n.txt) );
fclose($fp);
$fp = fopen("$n.txt", "w");
$s = $_SERVER['HTTP_USER_AGENT'].' '.time();
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
fwrite($fp, $s);
// fflush($fp);// flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
}
else
{
echo "Filesize must be greater than 0";
}
}
}
please change $s variables name its use same things two time
$fp = fopen("$n.txt", "r");
$s = fread($fp, filesize("$n.txt") );
fclose($fp);
The error occurs in the middle line of the above three lines.
Firstly, these three lines could be rewritten into a single line as follows:
$s = file_get_contents("$n.txt");
However, these isn't necessary, as these three lines are entirely redundant in your code. They don't do anything useful.
What they do is open a file, store its contents to $s and then close it.
But you are then immediately setting $s to a different value, thus throwing away the previous value, and making it pointless to have read it from the file in the first place.
If you need to keep the original contents of the file, then use file_get_contents() and make sure you don't overwrite the contents of the variable.
If you don't need the original contents of the file, then just delete those three lines from your code.
Incidentally, this error highlights a couple of good coding practices that you should take on board: Firstly, never re-use a variable for two different things, and secondly always give your variables (and functions) good names. $s is not a good name; $previousFileContents would be a better name; it would have made the error much more obvious.
Hello i have this code:
file1 = file_get_contents("read.txt");
$path2 = "write.txt";
$file2 = file_get_contents($path2);
if ($file1 !== $file2){
file_put_contents($path2, $file1);
echo "working";
}
how can i get the first 10000 lines or more from the read.txt file and write them in write.txt ?
There are many ways you could do this reading the entire file, but it would be better to use streams and only read the data you need.
<?php
$source="file.txt";
$destination="file2.txt";
$requiredLines=10000;
//compare the modification times, if source is newer than destination - then we do our work
if(filemtime($source)>filemtime($destination)){
//work out maximum length of file, as one line may be the whole file.
$filesize = filesize($source);
//open file for reading - this doesnt actually read the file it allows us to "stream" it
$sourceHandle = fopen($source, "r");
//open file for writing
$destinationHandle = fopen($destination, "w");
$linecount=0;
//loop through file until we reach the end of the file (feof) or we reach the desired number of lines
while (!feof($sourceHandle) && $linecount++<$requiredLines) {
//read one line
$line = stream_get_line($sourceHandle, $filesize, "\n");
//write the line
fwrite($destinationHandle,$line);
}
//close both files
fclose($sourceHandle);
fclose($destinationHandle);
}
You can find more info on streams here: Understanding PHP Streams
I need a php script that will open an external php file (from the same server folder), go through it line by line, and then normally display the page in the browser, as it would by just opening the external php page directly.
I need to open the external file line by line, so I can do some processing on the content of the file before showing it.
My current code is:
<?php
$handle = fopen("test.php", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line here, and change if needed
echo "$line\n";
}
fclose($handle);
}
else {
// error opening the file.
}
?>
This works, and the page is displayed, but any php code in the original external file is not honored - it is written out as text, and not rendered by the browser.
I need the external file to fully display, just as it would if I opened the file (in this case "test.php") by itself.
Other questions I have seen on SO deal with opening or displaying a full file at once, but I need to loop through my file and do some processing on the contents first, so need to evaluate it line by line.
Any ideas would be appreciated.
Thanks
I would save the changes to a temporary file, and then include it.
<?php
$handle = fopen("test.php", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line here, and change if needed
$newCode .= "$line\n";
}
fclose($handle);
}
else {
// error opening the file.
}
// temporary file name
$temp_file = tempnam(sys_get_temp_dir(), 'myfile').".php";
// save modified code
file_put_contents($temp_file, $newCode);
// include modified code
include $temp_file;
// delete file
unlink($temp_file);
?>
Retrieve the content, process it, keep it in memory then eval() it:
<?php
$newCode = "";
$handle = fopen("test.php", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line here, and change if needed
//$line = myLineProcess($line);
$newCode .= "$line\n";
}
fclose($handle);
}
else {
// error opening the file.
}
//run the code
eval('?>'.$newCode.'<?php;');
?>
As I start the process of writing my site in PHP and MySQL, one of the first PHP scripts I've written is a script to initialize my database. Drop/create the database. Drop/create each of the tables. Then load the tables from literals in the script.
That's all working fine! Whoohoo :-)
But I would prefer to read the data from files rather than hard-code them in the PHP script.
I have a couple of books on PHP, but they're all oriented toward web development using MySQL. I can't find anything about reading and writing to ordinary files.
Yes, I know there's a gazillion questions here on stackoverflow about reading TXT files, but when I look at each one, they're for C or C# or VB or Perl. I'm beginning to think that PHP just can't read files :-(
All I need is a brief PHP example of how to open a TXT file on the server, read it sequentially, display the data on the screen, and close the file, as in this pseudo-code:
program readfile;
handle = open('myfile.txt');
data = read (handle);
while (not eof (handle)) begin
display data;
data = read (handle);
end;
close (handle);
end;
I will also need to write files on the server when I get to the part of my site where people upload avatars, and save them as JPG or GIF files. But that's for later.
Thanks!
From the PHP manual for fread():
<?php
// get contents of a file into a string
$filename = "/usr/local/something.txt";
$handle = fopen($filename, "r");
$contents = fread($handle, filesize($filename));
fclose($handle);
?>
EDIT
per the comment, you can read a file line by line with fgets()
<?php
$handle = #fopen("/tmp/inputfile.txt", "r");
if ($handle) {
while (($buffer = fgets($handle, 4096)) !== false) {
echo $buffer;
}
if (!feof($handle)) {
echo "Error: unexpected fgets() fail\n";
}
fclose($handle);
}
?>
All I need is a brief PHP example of how to open a TXT file on the server, read it sequentially, display the data on the screen, and close the file, as in this pseudo-code:
echo file_get_contents('/path/to/file.txt');
Yes that brief, see file_get_contents, you normally don't need a loop:
$file = new SPLFileObject('/path/to/file.txt');
foreach($file as $line) {
echo $line;
}
Well, since you're asking about resources on the subject, there's a whole book on it in the PHP.net docs.
A basic example:
<?php
// get contents of a file into a string
$filename = "/usr/local/something.txt";
$handle = fopen($filename, "r");
$contents = fread($handle, filesize($filename));
fclose($handle);
?>
Why you not read php documentation about fopen
$file = fopen("source/file.txt","r");
if(!file)
{
echo("ERROR:cant open file");
}
else
{
$buff = fread ($file,filesize("source/file.txt"));
print $buff;
}
file_get_contents does all that for you and returns the text file in a string :)
You want to read line by line? Use fgets.
$handle = #fopen("myfile.txt", "r");
if ($handle) {
while (($content = fgets($handle, 4096)) !== false) {
//echo $content;
}
if (!feof($handle)) {
echo "Error: unexpected fgets() fail\n";
}
fclose($handle);
}
I'm trying to read a specific line from a text file using php.
Here's the text file:
foo
foo2
How would I get the content of the second line using php?
This returns the first line:
<?php
$myFile = "4-24-11.txt";
$fh = fopen($myFile, 'r');
$theData = fgets($fh);
fclose($fh);
echo $theData;
?>
..but I need the second.
Any help would be greatly appreciated
$myFile = "4-24-11.txt";
$lines = file($myFile);//file in to an array
echo $lines[1]; //line 2
file — Reads entire file into an array
omg I'm lacking 7 rep to make comments. This is #Raptor's & #Tomm's comment, since this question still shows up way high in google serps.
He's exactly right. For small files file($file); is perfectly fine. For large files it's total overkill b/c php arrays eat memory like crazy.
I just ran a tiny test with a *.csv with a file size of ~67mb (1,000,000 lines):
$t = -microtime(1);
$file = '../data/1000k.csv';
$lines = file($file);
echo $lines[999999]
."\n".(memory_get_peak_usage(1)/1024/1024)
."\n".($t+microtime(1));
//227.5
//0.22701287269592
//Process finished with exit code 0
And since noone mentioned it yet, I gave the SplFileObject a try, which I actually just recently discovered for myself.
$t = -microtime(1);
$file = '../data/1000k.csv';
$spl = new SplFileObject($file);
$spl->seek(999999);
echo $spl->current()
."\n".(memory_get_peak_usage(1)/1024/1024)
."\n".($t+microtime(1));
//0.5
//0.11500692367554
//Process finished with exit code 0
This was on my Win7 desktop so it's not representative for production environment, but still ... quite the difference.
If you wanted to do it that way...
$line = 0;
while (($buffer = fgets($fh)) !== FALSE) {
if ($line == 1) {
// This is the second line.
break;
}
$line++;
}
Alternatively, open it with file() and subscript the line with [1].
I would use the SplFileObject class...
$file = new SplFileObject("filename");
if (!$file->eof()) {
$file->seek($lineNumber);
$contents = $file->current(); // $contents would hold the data from line x
}
you can use the following to get all the lines in the file
$handle = #fopen('test.txt', "r");
if ($handle) {
while (!feof($handle)) {
$lines[] = fgets($handle, 4096);
}
fclose($handle);
}
print_r($lines);
and $lines[1] for your second line
$myFile = "4-21-11.txt";
$fh = fopen($myFile, 'r');
while(!feof($fh))
{
$data[] = fgets($fh);
//Do whatever you want with the data in here
//This feeds the file into an array line by line
}
fclose($fh);
This question is quite old by now, but for anyone dealing with very large files, here is a solution that does not involve reading every preceding line. This was also the only solution that worked in my case for a file with ~160 million lines.
<?php
function rand_line($fileName) {
do{
$fileSize=filesize($fileName);
$fp = fopen($fileName, 'r');
fseek($fp, rand(0, $fileSize));
$data = fread($fp, 4096); // assumes lines are < 4096 characters
fclose($fp);
$a = explode("\n",$data);
}while(count($a)<2);
return $a[1];
}
echo rand_line("file.txt"); // change file name
?>
It works by opening the file without reading anything, then moving the pointer instantly to a random position, reading up to 4096 characters from that point, then grabbing the first complete line from that data.
If you use PHP on Linux, you may try the following to read text for example between 74th and 159th lines:
$text = shell_exec("sed -n '74,159p' path/to/file.log");
This solution is good if your file is large.
You have to loop the file till end of file.
while(!feof($file))
{
echo fgets($file). "<br />";
}
fclose($file);
Use stream_get_line: stream_get_line — Gets line from stream resource up to a given delimiter
Source: http://php.net/manual/en/function.stream-get-line.php
You could try looping until the line you want, not the EOF, and resetting the variable to the line each time (not adding to it). In your case, the 2nd line is the EOF. (A for loop is probably more appropriate in my code below).
This way the entire file is not in the memory; the drawback is it takes time to go through the file up to the point you want.
<?php
$myFile = "4-24-11.txt";
$fh = fopen($myFile, 'r');
$i = 0;
while ($i < 2)
{
$theData = fgets($fh);
$i++
}
fclose($fh);
echo $theData;
?>
I like daggett answer but there is another solution you can get try if your file is not big enough.
$file = __FILE__; // Let's take the current file just as an example.
$start_line = __LINE__ -1; // The same with the line what we look for. Take the line number where $line variable is declared as the start.
$lines_to_display = 5; // The number of lines to display. Displays only the $start_line if set to 1. If $lines_to_display argument is omitted displays all lines starting from the $start_line.
echo implode('', array_slice(file($file), $start_line, lines_to_display));
I searched for a one line solution to read specific line from a file.
Here my solution:
echo file('dayInt.txt')[1]