I'm looking for the most efficient way to write the contents of the PHP input stream to disk, without using much of the memory that is granted to the PHP script. For example, if the max file size that can be uploaded is 1 GB but PHP only has 32 MB of memory.
define('MAX_FILE_LEN', 1073741824); // 1 GB in bytes
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR.'/'.$MyTempName.'.tmp', 'w');
fwrite($hDest, fread($hSource, MAX_FILE_LEN));
fclose($hDest);
fclose($hSource);
Does fread inside an fwrite like the above code shows mean that the entire file will be loaded into memory?
For doing the opposite (writing a file to the output stream), PHP offers a function called fpassthru which I believe does not hold the contents of the file in the PHP script's memory.
I'm looking for something similar but in reverse (writing from input stream to file). Thank you for any assistance you can give.
Yep - fread used in that way would read up to 1 GB into a string first, and then write that back out via fwrite. PHP just isn't smart enough to create a memory-efficient pipe for you.
I would try something akin to the following:
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR . '/' . $MyTempName . '.tmp', 'w');
while (!feof($hSource)) {
/*
* I'm going to read in 1K chunks. You could make this
* larger, but as a rule of thumb I'd keep it to 1/4 of
* your php memory_limit.
*/
$chunk = fread($hSource, 1024);
fwrite($hDest, $chunk);
}
fclose($hSource);
fclose($hDest);
If you wanted to be really picky, you could also unset($chunk); within the loop after fwrite to absolutely ensure that PHP frees up the memory - but that shouldn't be necessary, as the next loop will overwrite whatever memory is being used by $chunk at that time.
Related
My PHP script is receiving large data (100 - 500 MB) from a client. I want my PHP script run fast, without using too much memory.
To save traffic, I don't use Base64 or form data. I send binary data directly in a POST request.
The data consists of two parts: 2000 Bytes header, and the rest, that has to be stored as a file on the server.
$fle = file_get_contents("php://input",FALSE,NULL,2000);
file_put_contents("file.bin", $fle);
The problem is, that file_get_contents ignores the offset parameter, and reads the data from byte 0. Is there any better way to do it?
** I don't want to read the whole data and slice off the last N-2000 bytes, as I am afraid it would use too much memory.
Use the lower-level file IO functions and read/write a little bit at a time.
$bufsz = 4096;
$fi = fopen("php://input", "rb");
$fo = fopen("file.bin", "wb");
fseek($fi, 2000);
while( $buf = fread($fi, $bufsz) ) {
fwrite($fo, $buf);
}
fclose($fi);
fclose($fo);
This will read/write in 4kB chunks.
I'd like to know if there is a faster way of concatenating 2 text files in PHP, than the usual way of opening txt1 in a+, reading txt2 line by line and copying each line to txt1.
If you want to use a pure-PHP solution, you could use file_get_contents to read the whole file in a string and then write that out (no error checking, just to show how you could do it):
$fp1 = fopen("txt1", 'a+');
$file2 = file_get_contents("txt2");
fwrite($fp1, $file2);
It's probably much faster to use the cat program in linux if you have command line permissions for PHP
system('cat txt1 txt2 > txt3');
$content = file_get_contents("file1");
file_put_contents("file2", $content, FILE_APPEND);
I have found using *nix cat to be the most effective here, but if for whatever reason you don't have access to it, and you are concatenating large files, then you can use this line by line function. (Error handling stripped for simplicity).
function catFiles($arrayOfFiles, $outputPath) {
$dest = fopen($outputPath,"a");
foreach ($arrayOfFiles as $f) {
$FH = fopen($f,"r");
$line = fgets($FH);
while ($line !== false) {
fputs($dest,$line);
$line = fgets($FH);
}
fclose($FH);
}
fclose($dest);
}
While the fastest way is undobtedly to use OS commands, like cp or cat, this is hardly advisable for compatibility.
The fastest "PHP only" way is using file_get_contents, that reads the whole source file, in one shot but it also has some drawbacks. It will require a lot of memory for large files and for this reason it may fail depending on the memory assigned to PHP.
A universal clean and fast solution is to use fread and fwrite with a large buffer.
If the file is smaller than the buffer, all reading will happen in one burst, so speed is optimal, otherwise reading happens at big chunks (the size of the buffer) so the overhead is minimal and speed is quite good.
Reading line by line with fgets instead, has to test for every charachter, one by one, if it's a newline or line feed.
Also, reading line by line with fgets a file with many short lines will be slower as you will read many little pieces, of different sizes, depending of where newlines are positioned.
fread is faster as it only checks for EOF (which is easy) and reads files using a fixed size chunk you decide, so it can be made optimal for your OS or disk or kind of files (say you have many files <12k you can set the buffer size to 16k so they are all read in one shot).
// Code is untested written on mobile phone inside Stack Overflow, comes from various examples online you can also check.
<?php
$BUFFER_SIZE=1*1024*1024; // 1MB, bigger is faster.. depending on file sizes and count
$dest = fopen($fileToAppendTo "a+");
if (FALSE === $dest) die("Failed to open destination");
$handle = fopen("source.txt", "rb");
if (FALSE === $handle) {
fclose($dest);
die("Failed to open source");
}
$contents = '';
while( !feof($handle) ) {
fwrite($dest, fread($handle, $BUFFER_SIZE) );
}
fclose($handle);
fclose($dest);
?>
This question already has answers here:
PHP what is the best way to write data to middle of file without rewriting file
(3 answers)
Closed 9 years ago.
I have a file that I'm reading with PHP. I want to look for some lines that start with some white space and then some key words I'm looking for (for example, "project_name:") and then change other parts of that line.
Currently, the way I handle this is to read the entire file into a string variable, manipulate that string and then write the whole thing back to the file, fully replacing the entire file (via fopen( filepath, "wb" ) and fwrite()), but this feels inefficient. Is there a better way?
Update: After finishing my function I had time to benchmark it. I've used a 1GB large file for testing but the results where unsatisfying :|
Yes, the memory peak allocation is significantly smaller:
standard solution: 1,86 GB
custom solution: 653 KB (4096 bytes buffersize)
But compared to the following solution there is just a slight performance boost:
ini_set('memory_limit', -1);
file_put_contents(
'test.txt',
str_replace('the', 'teh', file_get_contents('test.txt'))
);
the script above tooks ~16 seconds, the custom solution took ~13 seconds.
Resume: The custome solution is slight faster on large files and consumes much less memory(!!!).
Also if you want to run this in a web server environment the custom solution is better as many concurrent scripts would likely consume the whole available memory of the system.
Original Answer:
The only thing that comes in mind, is to read the file in chunks which fit the file systems block size and write the content or modified content back to a temporary file. After finish processing you use rename() to overwrite the original file.
This would reduce the memory peak and should be significantly faster if the file is really large.
Note: On a linux system you can get the file system block size using:
sudo dumpe2fs /dev/yourdev | grep 'Block size'
I got 4096
Here comes the function:
function freplace($search, $replace, $filename, $buffersize = 4096) {
$fd1 = fopen($filename, 'r');
if(!is_resource($fd1)) {
die('error opening file');
}
// the tempfile can be anywhere but on the same partition as the original
$tmpfile = tempnam('.', uniqid());
$fd2 = fopen($tmpfile, 'w+');
// we store len(search) -1 chars from the end of the buffer on each loop
// this is the maximum chars of the search string that can be on the
// border between two buffers
$tmp = '';
while(!feof($fd1)) {
$buffer = fread($fd1, $buffersize);
// prepend the rest from last one
$buffer = $tmp . $buffer;
// replace
$buffer = str_replace($search, $replace, $buffer);
// store len(search) - 1 chars from the end of the buffer
$tmp = substr($buffer, -1 * (strlen($search)) + 1);
// write processed buffer (minus rest)
fwrite($fd2, $buffer, strlen($buffer) - strlen($tmp));
};
if(!empty($tmp)) {
fwrite($fd2, $tmp);
}
fclose($fd1);
fclose($fd2);
rename($tmpfile, $filename);
}
Call it like this:
freplace('foo', 'bar', 'test.txt');
i would like to understand how to use the buffer of a read file.
Assuming we have a big file with a list of emails line by line ( delimiter is a classic \n )
now, we want compare each line with each record of a table in our database in a kind of check like line_of_file == table_row.
this is a simple task if you have a normal file, otherwise, if you have a huge file the server usually stop the operation after few minute.
so what's the best way of doing this kind of stuff with the file buffer?
what i have so far is something like this:
$buffer = file_get_contents('file.txt');
while($row = mysql_fetch_array($result)) {
if ( preg_match('/'.$email.'/im',$buffer)) {
echo $row_val;
}
}
$buffer = file_get_contents('file.txt');
$lines = preg_split('/\n/',$buffer);
//or $lines = explode('\n',$buffer);
while($row = mysql_fetch_array($result)) {
if ( in_array($email,$lines)) {
echo $row_val;
}
}
Like already suggested in my closevotes to your question (hence CW):
You can use SplFileObject which implements Iterator to iterate over a file line by line to save memory. See my answers to
Least memory intensive way to read a file in PHP and
How to save memory when reading a file in Php?
for examples.
Don't use file_get_contents for large files. This pulls the entire file into memory all at once. You have to read it in pieces
$fp = fopen('file.txt', 'r');
while(!feof($fp)){
//get onle line
$buffer = fgets($fp);
//do your stuff
}
fclose($fp);
Open the file with fopen() and read it incrementally. Probably one line at a time with fgets().
file_get_contents reads the whole file into memory, which is undesirable if the file is larger than a few megabytes
Depending on how long this takes, you may need to worry about the PHP execution time limit, or the browser timing out if it doesn't receive any output for 2 minutes.
Things you might try:
set_time_limit(0) to avoid running up against the PHP time limit
Make sure to output some data every 30 seconds or so so the browser doesn't time out; make sure to flush(); and possibly ob_flush(); so your output is actually sent over the network (this is a kludge)
start a separate process (e.g. via exec()) to run this in the background. Honestly, anything that takes more than a second or two is best run in the background
I wrote a program to generate large .SQL files for quickly populating very large databases. I scripted it in PHP. When I started coding I was using fopen() and fwrite(). When files got too large the program would return control to the shell and the file would be incomplete.
Unfortunately I'm not sure exactly how large is 'too large'. I think it may have been around 4GB.
To solve this problem I had the file echo to stdout. I redirected it when I called the program like so:
[root#localhost]$ php generatesql.php > myfile.sql
Which worked like a charm. My output file ended up being about 10GB.
My question, then, is: Are fopen() and fwrite() limited by the file system in terms of how large a file they are capable of generating? If so; is this a limitation of PHP? Does this happen in other languages as well?
What's probably occuring is the underlying PHP build is 32bit and can't handle file pointers >4GB - see this related question.
Your underlying OS is obviously capable of storing large files, which why you're able to redirect stdout to a large file.
Incidentally, an SQL file is likely to be highly compressible, so you might like to consider using the gzip fopen wrapper to compress the file as you write it.
$file = 'compress.zlib:///path/to/my/file.sql.gz';
$f = fopen($file, 'wb');
//just write as normal...
fwrite($f, 'CREATE TABLE foo (....)');
fclose($f);
Your dump will be a fraction of the original size, and you can restore it simply piping the output from zcat into an SQL client, e.g. for mysql
zcat /path/to/my/file.sql.gz | mysql mydatabase
Yes and no. Its not fopen() or fwrite() which are limited directly, its the files, which cannot exceed some dimensions depending on the filesystem. Have a look at Comparison of filesystems on Wikipedia.
Is it possible that your script is taking too long to execute and has timed out?
Alternatively, is it possible that you're reaching your memory limits within the script?
You can write more than 2GB of data in a stream, not in a file (as the fseek internal pointer of the file is exceeding PHP limits, and streams are not usually seekable)
<?
$target = fopen('test.tar', 'w'); //this is a file, limited by php to 2GB
$body = str_repeat("===", 1024 * 1024);
while(true)
fwrite($target, $test);
<?
$target = popen('cat > test.tar', 'w'); //this is a stream, no limitation here
$body = str_repeat("===", 1024 * 1024);
while(true)
fwrite($target, $test);