I am using following code to convert a binary file into an array.
$handle = fopen($file, "r");
$contents = fread($handle,filesize($file));
$array = unpack("s*", $contents);
I want to be able to read it in chunks and send multiple separate requests to process it in parallel.
For example, I want to grab first 16000 bytes, then next 16000 etc.
So I would end up with multiple sets of data to process in parallel
$content1 = first 16000 bytes
$content2 = bytes from 16000 to 32000
$content3 = bytes from 32000 to 48000
I think this is pretty simple I am just not sure how it can be done.
A simple way would be to use substr() to split out chunks until it runs out of something to process...
$start = 0;
$size = 16000;
$contents = file_get_contents($file);
while ($chunk = substr($contents, $start, $size)) {
// Process
echo ">".$chunk."<".PHP_EOL;
$start +=$size;
}
Another way would be to convert it to array to split the string into chunks, you can use str_split()
$contents = file_get_contents($file);
$chunks = str_split($contents, 16000);
file_get_contents() does all the open file/read/close in one go, the str_split() then splits it up into an array of the size chunk you want it (16000 in this case).
Not sure how much performance gain you will get by this, but that is something you will have to test for yourself.
(Also check the notes on the manual page in case you are using multi-byte encoded files).
you should use the multi thread in php
see
http://php.net/manual/en/intro.pthreads.php
and
Does PHP have threading?
Given that the OP has accepted Nigel's answer, then the question was actually how to read arbitrary chunks from a file. That can be done with a slight variation of the original code. Instead of reading the complete file contents:
fread($handle, filesize($file));
^^^^^^^^^^^^^^^
… you pass your chunk size as second argument:
$contents = fread($handle, 16000);
Prior to that, you move to the desired location:
// E.g. Read 4th chunk:
fseek($handle, 3 * 16000);
Full stuff:
$handle = fopen($file, "r");
fseek($handle, 3 * 16000);
$contents = fread($handle, 16000);
Add some error checking and you're done. These are really old functions very close to the C implementation so they should be pretty fast and require very little memory.
Related
I've read a part of file and now want to make sure the part is the right size. How can I do it in php?
$part = fread($file, 1024);
return some_function($part) == 1024;
I've read the examples, but a I doubt to use strlen in cause of Null-terminated string, that might be inside the binary data in $part. In this way strlen returns size from start of part and until first null-byte is presented in data.
As stated in the PHP manual, strlen returns the number of bytes in the string, not the character length.
In PHP, a null byte in a string does NOT count as the end of the string, and any null bytes are included in the length of the string.
So strlen can be used for binary data, no matter if the data is from a file or some other source.
According to PHP documentation of fread() function you may use the construction using filesize() as shown in the first example:
<?php
// get contents of a file into a string
$filename = "/usr/local/something.txt";
$handle = fopen($filename, "r");
$contents = fread($handle, filesize($filename));
fclose($handle);
?>
Update: to find a size of the file can be used function stat() without opening or fstat() on opened file.
I'd like to know if there is a faster way of concatenating 2 text files in PHP, than the usual way of opening txt1 in a+, reading txt2 line by line and copying each line to txt1.
If you want to use a pure-PHP solution, you could use file_get_contents to read the whole file in a string and then write that out (no error checking, just to show how you could do it):
$fp1 = fopen("txt1", 'a+');
$file2 = file_get_contents("txt2");
fwrite($fp1, $file2);
It's probably much faster to use the cat program in linux if you have command line permissions for PHP
system('cat txt1 txt2 > txt3');
$content = file_get_contents("file1");
file_put_contents("file2", $content, FILE_APPEND);
I have found using *nix cat to be the most effective here, but if for whatever reason you don't have access to it, and you are concatenating large files, then you can use this line by line function. (Error handling stripped for simplicity).
function catFiles($arrayOfFiles, $outputPath) {
$dest = fopen($outputPath,"a");
foreach ($arrayOfFiles as $f) {
$FH = fopen($f,"r");
$line = fgets($FH);
while ($line !== false) {
fputs($dest,$line);
$line = fgets($FH);
}
fclose($FH);
}
fclose($dest);
}
While the fastest way is undobtedly to use OS commands, like cp or cat, this is hardly advisable for compatibility.
The fastest "PHP only" way is using file_get_contents, that reads the whole source file, in one shot but it also has some drawbacks. It will require a lot of memory for large files and for this reason it may fail depending on the memory assigned to PHP.
A universal clean and fast solution is to use fread and fwrite with a large buffer.
If the file is smaller than the buffer, all reading will happen in one burst, so speed is optimal, otherwise reading happens at big chunks (the size of the buffer) so the overhead is minimal and speed is quite good.
Reading line by line with fgets instead, has to test for every charachter, one by one, if it's a newline or line feed.
Also, reading line by line with fgets a file with many short lines will be slower as you will read many little pieces, of different sizes, depending of where newlines are positioned.
fread is faster as it only checks for EOF (which is easy) and reads files using a fixed size chunk you decide, so it can be made optimal for your OS or disk or kind of files (say you have many files <12k you can set the buffer size to 16k so they are all read in one shot).
// Code is untested written on mobile phone inside Stack Overflow, comes from various examples online you can also check.
<?php
$BUFFER_SIZE=1*1024*1024; // 1MB, bigger is faster.. depending on file sizes and count
$dest = fopen($fileToAppendTo "a+");
if (FALSE === $dest) die("Failed to open destination");
$handle = fopen("source.txt", "rb");
if (FALSE === $handle) {
fclose($dest);
die("Failed to open source");
}
$contents = '';
while( !feof($handle) ) {
fwrite($dest, fread($handle, $BUFFER_SIZE) );
}
fclose($handle);
fclose($dest);
?>
This question already has answers here:
PHP what is the best way to write data to middle of file without rewriting file
(3 answers)
Closed 9 years ago.
I have a file that I'm reading with PHP. I want to look for some lines that start with some white space and then some key words I'm looking for (for example, "project_name:") and then change other parts of that line.
Currently, the way I handle this is to read the entire file into a string variable, manipulate that string and then write the whole thing back to the file, fully replacing the entire file (via fopen( filepath, "wb" ) and fwrite()), but this feels inefficient. Is there a better way?
Update: After finishing my function I had time to benchmark it. I've used a 1GB large file for testing but the results where unsatisfying :|
Yes, the memory peak allocation is significantly smaller:
standard solution: 1,86 GB
custom solution: 653 KB (4096 bytes buffersize)
But compared to the following solution there is just a slight performance boost:
ini_set('memory_limit', -1);
file_put_contents(
'test.txt',
str_replace('the', 'teh', file_get_contents('test.txt'))
);
the script above tooks ~16 seconds, the custom solution took ~13 seconds.
Resume: The custome solution is slight faster on large files and consumes much less memory(!!!).
Also if you want to run this in a web server environment the custom solution is better as many concurrent scripts would likely consume the whole available memory of the system.
Original Answer:
The only thing that comes in mind, is to read the file in chunks which fit the file systems block size and write the content or modified content back to a temporary file. After finish processing you use rename() to overwrite the original file.
This would reduce the memory peak and should be significantly faster if the file is really large.
Note: On a linux system you can get the file system block size using:
sudo dumpe2fs /dev/yourdev | grep 'Block size'
I got 4096
Here comes the function:
function freplace($search, $replace, $filename, $buffersize = 4096) {
$fd1 = fopen($filename, 'r');
if(!is_resource($fd1)) {
die('error opening file');
}
// the tempfile can be anywhere but on the same partition as the original
$tmpfile = tempnam('.', uniqid());
$fd2 = fopen($tmpfile, 'w+');
// we store len(search) -1 chars from the end of the buffer on each loop
// this is the maximum chars of the search string that can be on the
// border between two buffers
$tmp = '';
while(!feof($fd1)) {
$buffer = fread($fd1, $buffersize);
// prepend the rest from last one
$buffer = $tmp . $buffer;
// replace
$buffer = str_replace($search, $replace, $buffer);
// store len(search) - 1 chars from the end of the buffer
$tmp = substr($buffer, -1 * (strlen($search)) + 1);
// write processed buffer (minus rest)
fwrite($fd2, $buffer, strlen($buffer) - strlen($tmp));
};
if(!empty($tmp)) {
fwrite($fd2, $tmp);
}
fclose($fd1);
fclose($fd2);
rename($tmpfile, $filename);
}
Call it like this:
freplace('foo', 'bar', 'test.txt');
I have read a fixed length Record from the file Using php. I have given the code below.
$f = fopen('data.txt', 'rb');
while (!feof($f)) {
// read one segment of 52 bytes
if ($s = fread($f, 52)) {
// unpack the binary structure into an associative array
print_r(unpack('ieid/ieage/a20name/a20city', $s));
}
}
fclose($f);
Now i need to know How to write the Fixed Length record to the file. I searched in many web sites but still i didn't get the correct answer . How can i write the fixed Length record to the binary / text file ?
I'm looking for the most efficient way to write the contents of the PHP input stream to disk, without using much of the memory that is granted to the PHP script. For example, if the max file size that can be uploaded is 1 GB but PHP only has 32 MB of memory.
define('MAX_FILE_LEN', 1073741824); // 1 GB in bytes
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR.'/'.$MyTempName.'.tmp', 'w');
fwrite($hDest, fread($hSource, MAX_FILE_LEN));
fclose($hDest);
fclose($hSource);
Does fread inside an fwrite like the above code shows mean that the entire file will be loaded into memory?
For doing the opposite (writing a file to the output stream), PHP offers a function called fpassthru which I believe does not hold the contents of the file in the PHP script's memory.
I'm looking for something similar but in reverse (writing from input stream to file). Thank you for any assistance you can give.
Yep - fread used in that way would read up to 1 GB into a string first, and then write that back out via fwrite. PHP just isn't smart enough to create a memory-efficient pipe for you.
I would try something akin to the following:
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR . '/' . $MyTempName . '.tmp', 'w');
while (!feof($hSource)) {
/*
* I'm going to read in 1K chunks. You could make this
* larger, but as a rule of thumb I'd keep it to 1/4 of
* your php memory_limit.
*/
$chunk = fread($hSource, 1024);
fwrite($hDest, $chunk);
}
fclose($hSource);
fclose($hDest);
If you wanted to be really picky, you could also unset($chunk); within the loop after fwrite to absolutely ensure that PHP frees up the memory - but that shouldn't be necessary, as the next loop will overwrite whatever memory is being used by $chunk at that time.