Running PHP Version 7.1.30 under RHEL 7.7.
I'm wanting to bump memory_limit, but wasn't sure if I had the syntax right (i.e. 256M or 256MB). So to start with I put a bad value "Hugo" in as the memory_limit setting. The trouble with this is the result of phpinfo() (run under httpd) literally has the string "Hugo" in place, i.e.:
So this has me somewhat concerned that PHP doesn't actually do any sanity checking for the value(s). (If the value provided was bad I would expect it to revert to a default, e.g.)
Can anyone comment on this - in particular, how do you know whether PHP will be enforcing things (if an arbitary string can be provided).
The confusing thing here is that the setting looks like an integer with some special syntax, but is internally defined as a string. The string is then parsed into a separate global variable whenever the value is changed. Crucially, the result of parsing the string to an integer isn't saved back to the settings table, so when you call phpinfo(), you see the original input, not the parsed value.
You can see this in the source:
The setting is defined with a callback for when it is changed
The callback is fed the raw string value
The callback for that particular setting parses the string to an integer using a function called zend_atol, which handles the special suffixes
It then calls a function which sets a global variable ("AG" means "Allocation Manager Global Variable", the macro is used to manage thread-safety if that's compiled in)
The supported syntax is ultimately defined in zend_atol, which:
parses the string for a numeric value, ignoring any additional text
looks at the last character of the string, and multiplies the preceding value if it is g, G, m, M, k, or K
A value with no digits at the start will be parsed as zero. When setting the global variable, this will set the memory limit to the minimum allowed, based on the constant ZEND_MM_CHUNK_SIZE.
You can see the effect by setting the memory limit, then running a loop that quickly allocates a large amount of memory and seeing what comes out in the error message. For instance:
# Invalid string; sets to compiled minimum
php -r 'ini_set("memory_limit", "HUGO"); while(true) $a[]=$a;'
# -> PHP Fatal error: Allowed memory size of 2097152 bytes exhausted
# Number followed by a string; takes the number
php -r 'ini_set("memory_limit", "4000000 HUGO"); while(true) $a[]=$a;'
# -> PHP Fatal error: Allowed memory size of 4000000 bytes exhausted
# Number followed by a string, but ending in one of the recognised suffixes
# This finds both the number and the suffix, so is equivalent to "4M", i.e. 4MiB
php -r 'ini_set("memory_limit", "4 HUGO M"); while(true) $a[]=$a;'
# -> PHP Fatal error: Allowed memory size of 4194304 bytes exhausted
First thing first,
We first need to understand how PHP.ini work in the way of interpretation workflow.
memory_limit is directives for PHP.
when using with PHP function you have to do something like this ini_set(‘memory_limit’,’256MB’). So, this function will temporarily set your value to the interpreter variable. If you see closer then you can get the two columns One is for the Local and One is for global. That shows the capability of the values to the individual respectively.
But, When you defined for global you need to set as a suffix with K, M, G respectively. If we exceed this value using apache .htaccess it requires the same for the PHP fpm.
I've started using the increment() method of the PHP Memcached client, and with that switched to the binary protocol. Apparently, increment() is only supported on the binary protocol. Occasionally, I'm seeing garbage results come back from incremented keys. For example:
$memcached = new \Memcached();
$memcached->setOption(\Memcached::OPT_BINARY_PROTOCOL, TRUE);
…
$this->cache->increment($key,1,1);
…
$this->cache->get($key);
Output:
"1\u0000ants1 0 1\r\n1\r\n1\r\n25\r"
Given that the key did not exist before it was incremented at first, and an initial value of 1 was given to the increment() call, I'd expect the value returned to be an integer. Instead, the strings returned look like left-over junk, e.g. the ants part of that string has no relevance.
Other (possibly) pertinent info:
I'm seeing this on a range of different keys
Our Memcached server is an AWS Elasticache instance
Other clients using the same cache node are not using the binary protocol.
All clients are running the same OS (CentOS), PHP and Memcached versions.
tl;dr;
This is a bug in the PHP extension code...
I dug into the PHP extension code that wraps libmemcached and the libmemcached API code itself, but I think I've found the possible underlying cause of your problem...
If you take a look at the PHP Memcached::increment() implementation you'll see on line 1858 of php_memcached.c
status = memcached_increment_with_initial(m_obj->memc, key, key_len, (unsigned int)offset, initial, expiry, &value);
The problem here is that offset may or may not be 64 bits wide. The libmemcached API tells us that the memcached_increment_with_initial function signature expects a uint64_t for offset whereas here offset is declared long and then cast to unsigned int.
So if we were do something like this...
$memcached = new memcached;
$memcached->addServer('127.0.0.1','11211');
$memcached->setOption(\Memcached::OPT_BINARY_PROTOCOL, TRUE);
$memcached->delete('foo'); // remove the key if it already exists
$memcached->increment('foo',1,1);
var_dump($memcached->get('foo'));
You'd see something like...
string(22) "8589934592
"
as the output from that script. Note this only works if the key foo does not already exist on that memcached server. Also note the length of that string at 22 characters, when clearly it's not supposed to be anywhere near that.
If you look at the hex representation of that string....
var_dump(bin2hex($memcached->get('foo')));
The result is clear garbage at the end...
string(44) "38353839393334353932000d0a000000000000000000"
The object that was being stored was clearly corrupted between the casts. So you may end up getting the same result as me or you may end up getting completely broken data as you have demonstrated above. It depends on how the cast effected the chunk of memory being stored at the time (which is falling into undefined behavior here). Also the only seemingly root cause for this is using an initial value with increment (using increment subsequently after this does not demonstrate that problem or if the key already exists).
I guess the problem of this stems from the fact that the libmemcached API has two different size requirements for offset parameter between memcached_increment and memcached_increment_with_initial
memcached_increment(memcached_st *ptr, const char *key, size_t key_length, uint32_t offset, uint64_t *value)
The former takes uint32_t whereas the later takes uint64_t and PHP's extension code casts both to unsigned int, which would be equivalent to uint32_t pretty much.
This discrepency in the width of the offset parameter is likely what causes the key to be corrupted somehow between the calling PHP extension code and the API code.
Is there a limit on a value string in php Associative array?
I have a query that returns value of GROUP_CONCAT as CSV for an array element. When I run the query directly, I see all values being returned. When I print the array in PHP, the string is truncated.
Not sure how get around it?
Any ideas?
More info:
I just var_dump the results I got from query. Here it is.
It shows 1024. Looks like there is a limit. What is it?
array(5) {
["FIELD_ID"]=>
string(3) "232"
["answers"]=>
string(1024) "48,50,52,54,56,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023,1024,1025,1026,1027,1028,1029,1030,1031,1032,1033,1034,1035,1036,1037,1038,1039,1040,1041,1042,1043,1044,1045,1046,1047,1048,1049,1050,1051,1052,1053,1054,1055,1056,1057,1058,1059,1060,1061,1062,1063,1064,1065,1066,1067,1068,1069,1070,1071,1072,1073,1074,1075,1076,1077,1078,1079,1080,1081,1082,1083,1084,1085,1086,1087,1088,1089,1090,1091,1092,1093,1094,1095,1096,1097,1098,1099,1100,1101,1102,1103,1104,1105,1106,1107,1108,1109,1110,1111,1112,1113,1114,1115,1116,1117,1118,1119,1120,1121,1122,1123,1124,1125,1126,1127,1128,1129,1130,1131,1132,1133,1134,1135,1136,1137,1138,1139,1140,1141,1142,1143,1144,1145,1146,1147,1148,1149,1150,1151,1152,1153,1154,1155,1156,1157,1158,1159,1160,1161,1162,1163,1164,1165,1166,1167,1168,1169,1170,1171,1172,1173,1174,1175,1176,1177,1178,1179,1180,1181,1182,1183,1184,1185,1186,1187,1188,1189,1190,1191,1192,"
["ANSWER_TYPE"]=>
string(2) "SS"
["FIELD_TYPE"]=>
string(2) "DA"
["main_field_name"]=>
string(0) ""
}
More Info:
The issue still exists even after setting the variable group_concat_max_len to 5000. Any other ideas? Any issues with PDO?
After restarting MySQL, it works. Thanks!
You might need to read up on the length of the max concat here.
Command-Line Format --group_concat_max_len=#
Option-File Format group_concat_max_len
Option Sets Variable Yes, group_concat_max_len
Variable Name group_concat_max_len
Variable Scope Global, Session
Dynamic Variable Yes
-- Permitted Values
Platform Bit Size 32
Type numeric
Default 1024
Range 4 .. 4294967295
-- Permitted Values
Platform Bit Size 64
Type numeric
Default 1024
Range 4 .. 18446744073709547520
Edit: I find it rather amusing that the string that is returned to you is 1024 in length - which just happens to be the default max length in mysql. Coincedence?
When you're using GROUP_CONCAT in mysql with the default length limit of 1024, anything above that in your results will get cut off. This happened to me as well.
To determine if this is the issue, try this:
// echo string length to determine what your variable value length is
echo strlen($variable_here);
Then check what the concat limit is set to in mysql by executing this query:
SHOW VARIABLES LIKE 'group_concat_max_len'
I ran it in phpMyAdmin. If you get the same value from strlen and the SHOW VARIABLES query, you'll need to update your my.conf file or ask your host to do it. In WHM/cPanel, it is located in /etc/my.conf and in that file you'll need to have something like this:
[mysqld] #<-- Under that
group_concat_max_len=5120 #<-- add this
I chose 5x1024 to make it 5120 just to not have to deal with it again. Once you've saved the conf file back to the server, you'll need to restart the mysql service. If you're using WHM, you can do that under Home » Restart Services »
SQL Server (MySQL)
Once I did the above, the strlen did in fact echo out 3519 and the value was no longer cut off.
Your question is somewhat tricky to answer becuase:
1. you give very little information on the actual size of your string.
2. You're asking several questions (array size and string size).
You should check out this question and then supply us with extra information like which version of php you're using and also as #FLuffeh says in his comment please show us some code.
I'm currently using the following two methods in my class to get the job done:
function xseek($h,$pos){
rewind($h);
if($pos>0)
fread($h,$pos);
}
function find($str){
return $this->startingindex($this->name,$str);
}
function startingindex($a,$b){
$lim = 1 + filesize($a) - strlen($b)/2;
$h = fopen($a,"rb");
rewind($h);
for($i=0;$i<$lim;$i++){
$this->xseek($h,$i);
if($b==strtoupper(bin2hex(fread($h,strlen($b)/2)))){
fclose($h);
return $i;
}
}
fclose($h);
return -1;
}
I realize this is quite inefficient, especially for PHP, but I'm not allowed any other language on my hosting plan.
I ran a couple tests, and when the hex string is towards the beginning of the file, it runs quickly and returns the offset. When the hex string isn't found, however, the page hangs for a while. This kills me inside because last time I tested with PHP and had hanging pages, my webhost shut my site down for 24 hours due to too much cpu time.
Is there a better way to accomplish this (finding a hex string's offset in a file)? Is there certain aspects of this that could be improved to speed up execution?
I would read the entire contents of the file into one hex string and use strrpos, but I was getting errors about maximum memory being exceeded. Would this be a better method if I chopped the file up and searched large pieces with strrpos?
edit:
To specify, I'm dealing with a settings file for a game. The settings and their values are in a block where there is a 32-bit int before the setting, then the setting, a 32-bit int before the value, and then the value. Both ints represent the lengths of the following strings. For example, if the setting was "test" and the value was "0", it would look like (in hex): 00000004746573740000000130. Now that you mention it, this does seem like a bad way to go about it. What would you recommend?
edit 2:
I tried a file that was below the maximum memory I'm allowed and tried strrpos, but it was very much slower than the way I've been trying.
edit 3: in reply to Charles:
What's unknown is the length of the settings block and where it starts. What I do know is what the first and last settings USUALLY are. I've been using these searching methods to find the location of the first and last setting and determine the length of the settings block. I also know where the parent block starts. The settings block is generally no more than 50 bytes into its parent, so I could start the search for the first setting there and limit how far it will search. The problem is that I also need to find the last setting. The length of the settings block is variable and could be any length. I could read the file the way I assume the game does, by reading the size of the setting, reading the setting, reading the size of the value, reading the value, etc. until I reached a byte with value -1, or FF in hex. Would a combination of limiting the search for the first setting and reading the settings properly make this much more efficient?
You have a lot of garbage code. For example, this code is doing nearly nothing:
function xseek($h,$pos){
rewind($h);
if($pos>0)
fread($h,$pos);
}
because it reads everytime from the begining of the file. Furthemore, why do you need to read something if you are not returning it? May be you looke for fseek()?
If you need to find a hex string in binary file, may be better to use something like this: http://pastebin.com/fpDBdsvV (tell me if there some bugs/problems).
But, if you are parsing game's settings file, I'd advise you to use fseek(), fread() and unpack() to seek to a place of where setting is, read portion of bytes and unpack it to PHP's variable types.
So how big can a $variable in PHP get? I've tried to test this, but I'm not sure that I have enough system memory (~2gb). I figure there has to be some kind of limit. What happens when a string gets too large? Is it concatenated, or does PHP throw an exception?
http://php.net/manual/en/language.types.string.php says:
Note: As of PHP 7.0.0, there are no particular restrictions regarding the length of a string on 64-bit builds. On 32-bit builds and in earlier versions, a string can be as large as up to 2GB (2147483647 bytes maximum)
In PHP 5.x, strings were limited to 231-1 bytes, because internal code recorded the length in a signed 32-bit integer.
You can slurp in the contents of an entire file, for instance using file_get_contents()
However, a PHP script has a limit on the total memory it can allocate for all variables in a given script execution, so this effectively places a limit on the length of a single string variable too.
This limit is the memory_limit directive in the php.ini configuration file. The memory limit defaults to 128MB in PHP 5.2, and 8MB in earlier releases.
If you don't specify a memory limit in your php.ini file, it uses the default, which is compiled into the PHP binary. In theory you can modify the source and rebuild PHP to change this default value.
If you specify -1 as the memory limit in your php.ini file, it stop checking and permits your script to use as much memory as the operating system will allocate. This is still a practical limit, but depends on system resources and architecture.
Re comment from #c2:
Here's a test:
<?php
// limit memory usage to 1MB
ini_set('memory_limit', 1024*1024);
// initially, PHP seems to allocate 768KB for basic operation
printf("memory: %d\n", memory_get_usage(true));
$str = str_repeat('a', 255*1024);
echo "Allocated string of 255KB\n";
// now we have allocated all of the 1MB of memory allowed
printf("memory: %d\n", memory_get_usage(true));
// going over the limit causes a fatal error, so no output follows
$str = str_repeat('a', 256*1024);
echo "Allocated string of 256KB\n";
printf("memory: %d\n", memory_get_usage(true));
String can be as large as 2GB.
Source
PHP's string length is limited by the way strings are represented in PHP; memory does not have anything to do with it.
According to phpinternalsbook.com, strings are stored in struct { char *val; int len; } and since the maximum size of an int in C is 4 bytes, this effectively limits the maximum string size to 2GB.
In a new upcoming php7 among many other features, they added a support for strings bigger than 2^31 bytes:
Support for strings with length >= 2^31 bytes in 64 bit builds.
Sadly they did not specify how much bigger can it be.
The maximum length of a string variable is only 2GiB - (2^(32-1) bits). Variables can be addressed on a character (8 bits/1 byte) basis and the addressing is done by signed integers which is why the limit is what it is. Arrays can contain multiple variables that each follow the previous restriction but can have a total cumulative size up to memory_limit of which a string variable is also subject to.
To properly answer this qustion you need to consider PHP internals or the target that PHP is built for.
To answer this from a typical Linux perspective on x86...
Sizes of types in C:
https://usrmisc.wordpress.com/2012/12/27/integer-sizes-in-c-on-32-bit-and-64-bit-linux/
Types used in PHP for variables:
http://php.net/manual/en/internals2.variables.intro.php
Strings are always 2GB as the length is always 32bits and a bit is wasted because it uses int rather than uint. int is impractical for lengths over 2GB as it requires a cast to avoid breaking arithmetic or "than" comparisons. The extra bit is likely being used for overflow checks.
Strangely, hash keys might internally support 4GB as uint is used although I have never put this to the test. PHP hash keys have a +1 to the length for a trailing null byte which to my knowledge gets ignored so it may need to be unsigned for that edge case rather than to allow longer keys.
A 32bit system may impose more external limits.