The title tells everything, can i change the limit of a strings length? I've checked this Page on stackoverflow, and it only tells that there is a limit, but how can i change it?
I've tried changing the memory limit using PHP like this:
ini_set('memory_limit', '-1');
But it didn't work...
From the documentation:
Note: string can be as large as up to 2GB (2147483647 bytes maximum)
and also:
The string in PHP is implemented as an array of bytes and an integer indicating the length of the buffer. It has no information about how those bytes translate to characters, leaving that task to the programmer.
That integer is probably the limit.
Related
I am having a for loop which is iterating through a large array $result. Every array item $row is used to generate a new line of string $line. This string is concatenated into $str. To preserve memory, I am using unset($result[$i]) to clear every array item that has been processed in the original array. This looks like this:
$resultcount = count($result);
for($i=0; $i<$resultcount; ++$i){
$row = $result[$i];
$line = do_something($row);
$str.= '<tr><td>'.$line.'</td></tr>';
unset($result[$i]);
}
return $str;
This (more or less exotic piece of code) works unless the string exceeds a length of approx. 1'000'000 characters. In this case the return value is just empty. This is highly irritating because:
The calculation effort (in this example do_something()) is not a problem. By using echo count($result).' - '.strlen($str)."\n" I can see that the loop finishes properly. There is no web server or php error shown.
The memory_limit is also not a problem. I am working with more data on other parts of the application.
It appears that the problem lies in return $str itself. If I am using return substr($str, 0, 980000) then it just works fine. Further debugging shows that the string gets tainted as soon as it reaches the length of 999'775 bytes.
I can't put longer return value string into another string variable. But I am able to do a strlen() to get a proper result (1310307). So the return value string has a total length of 1'310'307 bytes. But I can't use them properly as a string.
Where does this limitation come from and how may I circumvent it?
You seem to describe return value in contradicting way:
In this case the return value is just empty.
but
But I am able to do a strlen() to get a proper result (1310307). So the return value string has a total length of 1'310'307 bytes.
Something can't be empty and have length at the same time?
I have tested returning a string that long and it works fine for me. Returning larger and larger strings works to a point memory limit is exceeded, which triggers explicit Fatal Error.
On a technical side strings can be much larger than million characters:
Note: As of PHP 7.0.0, there are no particular restrictions regarding the length of a string on 64-bit builds. On 32-bit builds and in earlier versions, a string can be as large as up to 2GB (2147483647 bytes maximum)
http://php.net/manual/en/language.types.string.php
I suspect that the failing operation has to do with how you use the result rather than return from the function.
Well, it took me two and a half year to figure this out ;(
Rarst was right: It was not a problem of the function compiling and delivering the string. Later in the process I was using preg_replace() to do some further replacements. This function is not able to handle long strings. This could be fixed by using ini_set('pcre.backtrack_limit', 99999999999) beforehand as mentioned in How to do preg_replace on a long string
So how big can a $variable in PHP get? I've tried to test this, but I'm not sure that I have enough system memory (~2gb). I figure there has to be some kind of limit. What happens when a string gets too large? Is it concatenated, or does PHP throw an exception?
http://php.net/manual/en/language.types.string.php says:
Note: As of PHP 7.0.0, there are no particular restrictions regarding the length of a string on 64-bit builds. On 32-bit builds and in earlier versions, a string can be as large as up to 2GB (2147483647 bytes maximum)
In PHP 5.x, strings were limited to 231-1 bytes, because internal code recorded the length in a signed 32-bit integer.
You can slurp in the contents of an entire file, for instance using file_get_contents()
However, a PHP script has a limit on the total memory it can allocate for all variables in a given script execution, so this effectively places a limit on the length of a single string variable too.
This limit is the memory_limit directive in the php.ini configuration file. The memory limit defaults to 128MB in PHP 5.2, and 8MB in earlier releases.
If you don't specify a memory limit in your php.ini file, it uses the default, which is compiled into the PHP binary. In theory you can modify the source and rebuild PHP to change this default value.
If you specify -1 as the memory limit in your php.ini file, it stop checking and permits your script to use as much memory as the operating system will allocate. This is still a practical limit, but depends on system resources and architecture.
Re comment from #c2:
Here's a test:
<?php
// limit memory usage to 1MB
ini_set('memory_limit', 1024*1024);
// initially, PHP seems to allocate 768KB for basic operation
printf("memory: %d\n", memory_get_usage(true));
$str = str_repeat('a', 255*1024);
echo "Allocated string of 255KB\n";
// now we have allocated all of the 1MB of memory allowed
printf("memory: %d\n", memory_get_usage(true));
// going over the limit causes a fatal error, so no output follows
$str = str_repeat('a', 256*1024);
echo "Allocated string of 256KB\n";
printf("memory: %d\n", memory_get_usage(true));
String can be as large as 2GB.
Source
PHP's string length is limited by the way strings are represented in PHP; memory does not have anything to do with it.
According to phpinternalsbook.com, strings are stored in struct { char *val; int len; } and since the maximum size of an int in C is 4 bytes, this effectively limits the maximum string size to 2GB.
In a new upcoming php7 among many other features, they added a support for strings bigger than 2^31 bytes:
Support for strings with length >= 2^31 bytes in 64 bit builds.
Sadly they did not specify how much bigger can it be.
The maximum length of a string variable is only 2GiB - (2^(32-1) bits). Variables can be addressed on a character (8 bits/1 byte) basis and the addressing is done by signed integers which is why the limit is what it is. Arrays can contain multiple variables that each follow the previous restriction but can have a total cumulative size up to memory_limit of which a string variable is also subject to.
To properly answer this qustion you need to consider PHP internals or the target that PHP is built for.
To answer this from a typical Linux perspective on x86...
Sizes of types in C:
https://usrmisc.wordpress.com/2012/12/27/integer-sizes-in-c-on-32-bit-and-64-bit-linux/
Types used in PHP for variables:
http://php.net/manual/en/internals2.variables.intro.php
Strings are always 2GB as the length is always 32bits and a bit is wasted because it uses int rather than uint. int is impractical for lengths over 2GB as it requires a cast to avoid breaking arithmetic or "than" comparisons. The extra bit is likely being used for overflow checks.
Strangely, hash keys might internally support 4GB as uint is used although I have never put this to the test. PHP hash keys have a +1 to the length for a trailing null byte which to my knowledge gets ignored so it may need to be unsigned for that edge case rather than to allow longer keys.
A 32bit system may impose more external limits.
I'm currently using the following two methods in my class to get the job done:
function xseek($h,$pos){
rewind($h);
if($pos>0)
fread($h,$pos);
}
function find($str){
return $this->startingindex($this->name,$str);
}
function startingindex($a,$b){
$lim = 1 + filesize($a) - strlen($b)/2;
$h = fopen($a,"rb");
rewind($h);
for($i=0;$i<$lim;$i++){
$this->xseek($h,$i);
if($b==strtoupper(bin2hex(fread($h,strlen($b)/2)))){
fclose($h);
return $i;
}
}
fclose($h);
return -1;
}
I realize this is quite inefficient, especially for PHP, but I'm not allowed any other language on my hosting plan.
I ran a couple tests, and when the hex string is towards the beginning of the file, it runs quickly and returns the offset. When the hex string isn't found, however, the page hangs for a while. This kills me inside because last time I tested with PHP and had hanging pages, my webhost shut my site down for 24 hours due to too much cpu time.
Is there a better way to accomplish this (finding a hex string's offset in a file)? Is there certain aspects of this that could be improved to speed up execution?
I would read the entire contents of the file into one hex string and use strrpos, but I was getting errors about maximum memory being exceeded. Would this be a better method if I chopped the file up and searched large pieces with strrpos?
edit:
To specify, I'm dealing with a settings file for a game. The settings and their values are in a block where there is a 32-bit int before the setting, then the setting, a 32-bit int before the value, and then the value. Both ints represent the lengths of the following strings. For example, if the setting was "test" and the value was "0", it would look like (in hex): 00000004746573740000000130. Now that you mention it, this does seem like a bad way to go about it. What would you recommend?
edit 2:
I tried a file that was below the maximum memory I'm allowed and tried strrpos, but it was very much slower than the way I've been trying.
edit 3: in reply to Charles:
What's unknown is the length of the settings block and where it starts. What I do know is what the first and last settings USUALLY are. I've been using these searching methods to find the location of the first and last setting and determine the length of the settings block. I also know where the parent block starts. The settings block is generally no more than 50 bytes into its parent, so I could start the search for the first setting there and limit how far it will search. The problem is that I also need to find the last setting. The length of the settings block is variable and could be any length. I could read the file the way I assume the game does, by reading the size of the setting, reading the setting, reading the size of the value, reading the value, etc. until I reached a byte with value -1, or FF in hex. Would a combination of limiting the search for the first setting and reading the settings properly make this much more efficient?
You have a lot of garbage code. For example, this code is doing nearly nothing:
function xseek($h,$pos){
rewind($h);
if($pos>0)
fread($h,$pos);
}
because it reads everytime from the begining of the file. Furthemore, why do you need to read something if you are not returning it? May be you looke for fseek()?
If you need to find a hex string in binary file, may be better to use something like this: http://pastebin.com/fpDBdsvV (tell me if there some bugs/problems).
But, if you are parsing game's settings file, I'd advise you to use fseek(), fread() and unpack() to seek to a place of where setting is, read portion of bytes and unpack it to PHP's variable types.
So how big can a $variable in PHP get? I've tried to test this, but I'm not sure that I have enough system memory (~2gb). I figure there has to be some kind of limit. What happens when a string gets too large? Is it concatenated, or does PHP throw an exception?
http://php.net/manual/en/language.types.string.php says:
Note: As of PHP 7.0.0, there are no particular restrictions regarding the length of a string on 64-bit builds. On 32-bit builds and in earlier versions, a string can be as large as up to 2GB (2147483647 bytes maximum)
In PHP 5.x, strings were limited to 231-1 bytes, because internal code recorded the length in a signed 32-bit integer.
You can slurp in the contents of an entire file, for instance using file_get_contents()
However, a PHP script has a limit on the total memory it can allocate for all variables in a given script execution, so this effectively places a limit on the length of a single string variable too.
This limit is the memory_limit directive in the php.ini configuration file. The memory limit defaults to 128MB in PHP 5.2, and 8MB in earlier releases.
If you don't specify a memory limit in your php.ini file, it uses the default, which is compiled into the PHP binary. In theory you can modify the source and rebuild PHP to change this default value.
If you specify -1 as the memory limit in your php.ini file, it stop checking and permits your script to use as much memory as the operating system will allocate. This is still a practical limit, but depends on system resources and architecture.
Re comment from #c2:
Here's a test:
<?php
// limit memory usage to 1MB
ini_set('memory_limit', 1024*1024);
// initially, PHP seems to allocate 768KB for basic operation
printf("memory: %d\n", memory_get_usage(true));
$str = str_repeat('a', 255*1024);
echo "Allocated string of 255KB\n";
// now we have allocated all of the 1MB of memory allowed
printf("memory: %d\n", memory_get_usage(true));
// going over the limit causes a fatal error, so no output follows
$str = str_repeat('a', 256*1024);
echo "Allocated string of 256KB\n";
printf("memory: %d\n", memory_get_usage(true));
String can be as large as 2GB.
Source
PHP's string length is limited by the way strings are represented in PHP; memory does not have anything to do with it.
According to phpinternalsbook.com, strings are stored in struct { char *val; int len; } and since the maximum size of an int in C is 4 bytes, this effectively limits the maximum string size to 2GB.
In a new upcoming php7 among many other features, they added a support for strings bigger than 2^31 bytes:
Support for strings with length >= 2^31 bytes in 64 bit builds.
Sadly they did not specify how much bigger can it be.
The maximum length of a string variable is only 2GiB - (2^(32-1) bits). Variables can be addressed on a character (8 bits/1 byte) basis and the addressing is done by signed integers which is why the limit is what it is. Arrays can contain multiple variables that each follow the previous restriction but can have a total cumulative size up to memory_limit of which a string variable is also subject to.
To properly answer this qustion you need to consider PHP internals or the target that PHP is built for.
To answer this from a typical Linux perspective on x86...
Sizes of types in C:
https://usrmisc.wordpress.com/2012/12/27/integer-sizes-in-c-on-32-bit-and-64-bit-linux/
Types used in PHP for variables:
http://php.net/manual/en/internals2.variables.intro.php
Strings are always 2GB as the length is always 32bits and a bit is wasted because it uses int rather than uint. int is impractical for lengths over 2GB as it requires a cast to avoid breaking arithmetic or "than" comparisons. The extra bit is likely being used for overflow checks.
Strangely, hash keys might internally support 4GB as uint is used although I have never put this to the test. PHP hash keys have a +1 to the length for a trailing null byte which to my knowledge gets ignored so it may need to be unsigned for that edge case rather than to allow longer keys.
A 32bit system may impose more external limits.
Here is my code, which creates 2d array filled with zeros, array dimensions are (795,6942):
function zeros($rowCount, $colCount){
$matrix = array();
for ($rowIndx=0; $rowIndx<$rowCount; $rowIndx++){
$matrix[] = array();
for($colIndx=0; $colIndx<$colCount; $colIndx++){
$matrix[$rowIndx][$colIndx]=0;
}
}
return $matrix;
}
$matrix = zeros(795,6942);
And here is the error that I receive:
Allowed memory size of 134217728 bytes exhausted (tried to allocate 35 bytes)
Any ideas how to solve this?
As a quick calculation, you are trying to create an array that contains :
795*6942 = 5,518,890
integers.
If we consider that one integer is stored on 4 bytes (i.e. 32 bits ; using PHP, it not be less), it means :
5518890*4 = 22,075,560
bytes.
OK, quick calculation... result is "it should be OK".
But things are not that easy, unfortunatly :-(
I suppose it's related to the fact that data is stored by PHP using an internal data-structure that's much more complicated than a plain 32 bits integer
Now, just to be curious, let's modify your function so it outputs how much memory is used at the end of each one of the outer for-loop :
function zeros($rowCount, $colCount){
$matrix = array();
for ($rowIndx=0; $rowIndx<$rowCount; $rowIndx++){
$matrix[] = array();
for($colIndx=0; $colIndx<$colCount; $colIndx++){
$matrix[$rowIndx][$colIndx]=0;
}
var_dump(memory_get_usage());
}
return $matrix;
}
With this, I'm getting this kind of output (PHP 5.3.2-dev on a 64bits system ; memory_limit is set to 128MB -- which is already a lot !) :
int 1631968
int 2641888
int 3651808
...
...
int 132924168
int 133934088
Fatal error: Allowed memory size of 134217728 bytes exhausted
Which means each iteration of the outer for-loop requires something like 1.5 MB of memory -- and I only get to 131 iterations before the script runs out of memory ; and not 765 like you wanted.
Considering you set your memory_limit to 128M, you'd have to set it to something really much higher -- like
128*(765/131) = 747 MB
Well, even with
ini_set('memory_limit', '750M');
it's still not enough... with 800MB, it seems enough ;-)
But I would definitly not recommend setting memory_limit to such a high value !
(If you have 2GB of RAM, your server will not be able to handle more than 2 concurrent users ^^ ;; I wouldn't actually test this if my computer had 2GB of RAM, to be honest)
The only solution I see here is for you to re-think your design : there has to be something else you can do than use this portion of code :-)
(BTW : maybe "re-think your design" means using another language PHP : PHP is great when it comes to developping web-sites, but is not suited to every kind of problem)
The default PHP array implementation is very memory-intensive. If you are just storing integers (and lots of them), you probably want to look at SplFixedArray. It uses a regular contiguous memory block to store the data, as opposed to the traditional hash structure.
You should try increasing the amount of memory available to PHP:
ini_set('memory_limit', '32M');
in your PHP file.