I'm using the following code to download a large file (>100mb). The code is executed in a shell.
$fileHandle = fopen($url, 'rb');
$bytes = 100000;
while ($read = #fread($fileHandle, $bytes)) {
debug(strlen($read));
if (!file_put_contents($filePath, $read, FILE_APPEND)) {
return false;
}
}
Where I would expect that debug(strlen($read)) would output 100000, this is the actual output:
10627
8192
8192
8192
...
Why doesn't fread read more than 8192 bytes after the first time, and why does it read 10627 bytes on the first iteration?
This makes downloading the file very slow, is there a better way to do this?
The answer to your question is (quoting from the PHP docs for fread()):
if the stream is read buffered and it does not represent a plain file, at most one read of up to a number of bytes equal to the chunk size (usually 8192) is made; depending on the previously buffered data, the size of the returned data may be larger than the chunk size
The solution to your performance problem is to using stream_copy_to_stream() which should be faster than block reading using fread(), and more memory efficient as well
I checked the manual, and found this: http://php.net/manual/en/function.fread.php
"If the stream is read buffered and it does not represent a plain file, at most one read of up to a number of bytes equal to the chunk size (usually 8192) is made;"
Since you're opening a URL this is probably the case.
It doesn't explain the 10627 though...
Besides that, why do you expect 100000 byte reads to be faster than 8192?
I doubt that's your bottle neck. My guess is that either the download speed from the URL or the writing speed of the HD is the problem.
Related
I have many file chunks and I need to merge them using PHP fopen function.However, I'm worring about the memory usage.
For example,I got about 100 files listed in split_hash.txt,each is about 100mb. And here I combine them together:
<?php
$hash = file_get_contents("split_hash.txt");
$list = explode("\r\n",$hash);
$fp = fopen("hadoop2.zip","ab");
foreach($list as $value){
if(!empty($value)) {
$handle = fopen($value,"rb");
fwrite($fp,fread($handle,filesize($value)));
fclose($handle);
unset($handle);
}
}
fclose($fp);
echo "ok";
Will it cost a lot of my memory?
Will it cost a lot of my memory?
it will if you use fread($handle,filesize($value)) to get the whole length of the file for your fread, use fread in smaller chunks per file.
I would change:
fwrite($fp,fread($handle,filesize($value)));
to:
while (!feof($handle)) {
fwrite($fp,fread($handle,1048576));
}
so that you are only dealing with 10 megabytes at a time
It will cost you the maximum of your 100 file size in memory peak.
you are reading whole file and write it to another file. In this case it doesn't matter how many files do you have, but it matters how big is file.
You are saying it's ~100 mb, with default php settings it's 128 mb memory limit, which is acceptable for your case
I have the following PHP code:
$fp=fopen("http://myurl.com/very_big_file.txt",'r');
$feed=fread($fp,16);
It works correctly but I have some doubts. Does it really reads only first 16 bytes without downloading the whole file?
string fread ( resource $handle , int $length )
fread() reads up to length bytes from the file pointer referenced by handle. Reading stops as soon as one of the following conditions is met:
length bytes have been read
EOF (end of file) is reached
a packet becomes available or the socket timeout occurs (for network streams) <-- this line says it.
if the stream is read buffered and it does not represent a plain file, at most one read of up to a number of bytes equal to the chunk size (usually 8192) is made; depending on the previously buffered data, the size of the returned data may be larger than the chunk size.
I'm looking for the most efficient way to write the contents of the PHP input stream to disk, without using much of the memory that is granted to the PHP script. For example, if the max file size that can be uploaded is 1 GB but PHP only has 32 MB of memory.
define('MAX_FILE_LEN', 1073741824); // 1 GB in bytes
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR.'/'.$MyTempName.'.tmp', 'w');
fwrite($hDest, fread($hSource, MAX_FILE_LEN));
fclose($hDest);
fclose($hSource);
Does fread inside an fwrite like the above code shows mean that the entire file will be loaded into memory?
For doing the opposite (writing a file to the output stream), PHP offers a function called fpassthru which I believe does not hold the contents of the file in the PHP script's memory.
I'm looking for something similar but in reverse (writing from input stream to file). Thank you for any assistance you can give.
Yep - fread used in that way would read up to 1 GB into a string first, and then write that back out via fwrite. PHP just isn't smart enough to create a memory-efficient pipe for you.
I would try something akin to the following:
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR . '/' . $MyTempName . '.tmp', 'w');
while (!feof($hSource)) {
/*
* I'm going to read in 1K chunks. You could make this
* larger, but as a rule of thumb I'd keep it to 1/4 of
* your php memory_limit.
*/
$chunk = fread($hSource, 1024);
fwrite($hDest, $chunk);
}
fclose($hSource);
fclose($hDest);
If you wanted to be really picky, you could also unset($chunk); within the loop after fwrite to absolutely ensure that PHP frees up the memory - but that shouldn't be necessary, as the next loop will overwrite whatever memory is being used by $chunk at that time.
Say I read a number of bytes like this:
$data = fread($fp, 4096);
Since fread will stop reading if it reaches the end of the file, how can I know exactly how much was read? Would strlen($data) work? Or could that be potentially wrong?
What I'm trying to accomplish, is to read a number of bytes, and then go back to where I was before I read. And I'm trying to avoid using arithmetic (ftell, fread, ftell, subract, fseek), since a file could potentially be larger than PHP_INT_MAX and potentially mess that up. What I would want is to just do fseek($fp, -$bytes_read, SEEK_CUR), but for that I need to know how many bytes I just read...
After fread use ftell($fp) to get the current file position.
Check this (untested):
mb_strlen($data, '8bit')
The second argument '8bit' forces the function to return the number of bytes.
Found in comments at php manual for mb_strlen.
I understand the differences between fgets() and fgetss() but I don't get the difference between fgets() and fread(), can someone please clarify this subject? Which one is faster? Thanks!
fgets reads a line -- i.e. it will stop at a newline.
fread reads raw data -- it will stop after a specified (or default) number of bytes, independently of any newline that might or might not be present.
Speed is not a reason to use one over the other, as those two functions just don't do the same thing :
If you want to read a line, from a text file, then use fgets
If you want to read some data (not necessarily a line) from a file, then use fread.
fread() for binary data and fread has a limit on how many chars you can read
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
$buffer = fread($source_file, 5);
var_dump($buffer); //return string with length 5 chars!
}
Number 5 is length bytes have been read .
The function fgets reads a single line from a text file. It is reading so long until the end of the current line (or the end of the file) is reached. Therefore, if you would like to read one line from a text file, you should use fgets.
The function fread not only reads until the end of the line but to the end of the file [e.g. fread($handle)] or as many bytes as specified as a parameter [e.g. fread($handle, 1024)]. So, if you would like to read a complete file, no matter whether it is a text file with all containing lines or arbitrary raw data from a file, you should use fread.
Both the functions are used to read data from files
fgets($filename, $bytes)
fgets usually reads $bytes-1 amount of data and stops at a newline or an EOF(end-of-file) whichever comes first. If the bytes are not specified, then the default value is 1024 bytes.
fread($filename, $bytes)
fread reads exactly $bytes amount of data and stops only at EOF.
The accepted answer is correct, but there is one more case for fread to stop reading. fread has a chunk limit of 8192 bytes. I discovered this when I was getting different results from fread($stream, 8300) and fget($stream, 8300).
From fread docs:
if the stream is read buffered and it does not represent a plain file, at most one read of up to a number of bytes equal to the chunk size (usually 8192) is made; depending on the previously buffered data, the size of the returned data may be larger than the chunk size.