Segfault with PHP fread() on text file over 2GB - php

Question

I have a php script which reads in a text file and does a count of all the lines in the file which match a specified regular expression. The script has worked well up until now as it segfaulted on the fread of a file over 2GB.
Actually before the segfault, I initially received the PHP Fatal Error: PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 2223941409 bytes).
To fix that I added this line to my script: ini_set('memory_limit', '4G');
That fixes the memory size exhausted error but I get the segfault on fread now.
Here's a condensed working version of the script which will exhibit the error:
#!/usr/bin/php
<?php
ini_set('memory_limit', '4G');
$file = $argv[1];
$fh = fopen($file, 'r');
$fsize = filesize($file);
print("SIZE: ".$fsize."\n" );
$myData = fread($fh, $fsize);
print("Got passed fread!\n");
fclose($fh);
preg_match_all( '/Z\t/', $myData, $sArray );
$scount = count($sArray,COUNT_RECURSIVE);
print("COUNT: ".$scount."\n");
?>
Sample output:
$ runtest.php testfile.txt
SIZE: 2223941408
Segmentation fault (core dumped)
Other info:
OS: CentOS release 6.7 (Final) x86_64
PHP 5.3.3 (cli) (built: Jul 9 2015 17:39:00) 64-bit

Answer 1

You're probably using a 32-bit PHP distribution. Under such architecture a PHP process cannot allocate more than 2 GB of RAM. In practice the upper limit is closer to 1GB than 2GB—the interpreter crashes way before getting to the 2 GB limit. Additionally, integer variables cannot be greater than PHP_INT_MAX which, in 32 builds, is as small as 2,147,483,647 (232-1).
This highlights two problems in your code:
$fsize = filesize($file);
... will not work if the file size is greater that PHP_INT_MAX.
Because PHP's integer type is signed and many platforms use 32bit integers, some filesystem functions may return unexpected results for files which are larger than 2GB.
$myData = fread($fh, $fsize);
... will crash for large files because you're loading the complete file contents in memory and then doing additional processing that will probably eat even more memory.
You'd better redesign your algorithm and read the file in small chunks (the task where fread() excels at). Counting the occurrences of a two-character substring should only need a few KB of RAM.
Here's a possible approach that assumes single byte encoding (as your code does):
// Ridiculously small value for illustration purposes, set to something bigger for better performance
define('CHUNK_SIZE', 4);
$fsize = $scount = 0;
$fh = fopen($file, 'r');
$possible_pending_match = false;
while (!feof($fh)) {
$chunk = fread($fh, CHUNK_SIZE);
$fsize += strlen($chunk);
$scount += substr_count($chunk, "Z\t");
if ($possible_pending_match && $chunk[0]==="\t") {
$scount++;
}
$possible_pending_match = substr($chunk, -1)==='Z';
}
print("SIZE: ".$fsize."\n" );
print("COUNT: ".$scount."\n");
print("MEMORY: ".memory_get_peak_usage(true)." bytes\n");
You'd need to add 1 to $scount to get the same result as your code, which counts one extra item for no evident reason—it feels like a bug but I don't know the specs.

Answer 2

Hi 2GB means that there is some internal 32-bit limitation in PHP. Are you running 32-bit PHP?
There is an alternative solution. You can do it with a very small memory overhead using a shell command called by PHP. The memory used would be no more than a couple of MB as grep and wc only load portions of the file into memory.
$lines = shell_exec("grep 'Z\t' $file | wc --lines");
grep: command to search files using regular expression
wc: command that returns the number of words/lines/chars