I have a very big html form (containing table with rows, which contain multiple inputs), which i need to submit to PHP script via POST request.
The problem is some values don't come through and are absent in PHP's $_POST superglobal.
I checked (using Firebug extension) that the values are actually sent to server by the browser.
$_POST gets populated, but some values are just missing.
I checked what is raw request using:
$raw_post = file_get_contents('php://input');
and the string returned has the values. They are just not parsed into $_POST array. The strange thing i noticed is, it seems that the php://input values are cut after some length, and rest of the string does not come through to $_POST.
I thought about post_max_size and memory_limit and set them to large values:
memory_limit = 256M
post_max_size = 150M
but according to php documentation $_POST should not contain any values if request made is bigger than post_max_size.
Due to big size of form and request I cannot post it here, but i can post php script i used to debug the problem:
var_dump($file = file_get_contents('php://input'));
var_dump($_POST);
//... then i parsed the $file
Server version: Apache/2.2.9 (Debian)
PHP version: PHP 5.3.2-0.dotdeb.2
Can enyone explain reason of such strange PHP behaviour, and what should i do (change php settings, code?) to use $_POST array while processing form?
EDIT: To be clear: not only the values are missing. $_POST does not contain these keys either.
e.x. fragment of raw post:
t_dodparam%5B198%5D=&t_dodparam2%5B198%5D=&t_kolejnosc%5B198%5D=199&n_indeks=201&n_wartosc=testtesttest
Key 't_dodparam' is in post and it has key 198. The rest of parameters are missing (e.x. t_dodparam2 is in post, but it has no such key as 198, and there is no such key as n_wartosc in $_POST)
PHP modifies fields containing the characters space, dot, open square bracket and others to be compatible with with the deprecated register_globals
you can find a lot of workarounds in the comments here:
PHP: Variables From External Sources
For Exampe (comment by POSTer):
<?php
//Function to fix up PHP's messing up POST input containing dots, etc.
function getRealPOST() {
$pairs = explode("&", file_get_contents("php://input"));
$vars = array();
foreach ($pairs as $pair) {
$nv = explode("=", $pair);
$name = urldecode($nv[0]);
$value = urldecode($nv[1]);
$vars[$name] = $value;
}
return $vars;
}
?>
I just fixed this issue by adding a value to max_input_vars in my PHP configuration file. According to this. it was introduced in 5.3.9, yet after some package upgrades I experienced the issue in 5.3.2.
The default for max_input_vars is 1000, which was too small for my form.
There are many different things that could be causing this. Best to check your error log. Many of the things that cause this symptom will put messages in the error log, but not display errors in your PHP application.
Some of the possible causes:
Suhosin
Suhosin is an extension for PHP designed to protect servers and users from known and unknown flaws in PHP applications and the PHP core. One of the things that it does is limit the size of $_POST, $_GET, $_REQUEST, and $_COOKIE. If your problem is Suhosin you will get an error in your log such as
ALERT - configured POST variable limit exceeded - dropped variable 'foo'
The solution is simple, just increase the maximum number of allowed variables in php.ini. If you don’t have a suhosin section, just create one. Like such:
[suhosin]
suhosin.request.max_vars = 1000 # Default is 200
suhosin.post.max_vars = 1000 # Default is 200
There are other suhosin settings that can cause this, but these are the most likely candidates.
Invalid form field names
Back in the old days PHP had a setting called register_globals (now depricated) that automatically converted GET and POST variables into PHP variables. So if your form had fields 'foo' and 'bar', when that form is submitted the variables $foo and $bar would be automatically created. However there are several characters that are not valid for use in PHP variable names (space, dot, open square bracket and others). Depending on what characters you used, the variable may have the invalid characters stripped, be missing its value, or be unset. Despite the fact that register_globals is no longer used, PHP still strips these characters when building $_POST, $_GET, $_REQUEST, and $_COOKIE. If you are unable to fix the values of the form fields you might try something like:
<?php
/**
* Converts raw POST data into an array.
*
* Does not work for hierarchical POST data.
*
* #return array
*/
function real_post() {
static $post;
if (!isset($post)) {
$pairs = explode("&", file_get_contents("php://input"));
$post = array();
foreach ($pairs as $pair) {
$x = explode("=", $pair);
$post[rawurldecode($x[0])] = rawurldecode($x[1]);
}
}
return $post;
}
?>
What about using "parse_str" to convert the query string into php structures? This funcion is the inverse of http_build_query.
$b = array();
parse_str(file_get_contents("php://input"), $b);
I am posting Jquery's .ajax() function. I was trying to retrieve my data from $_POST but it was incomplete. Then I found this post which put me on the right track. However, the getRealPOST() method described above wasn't working for me - it can't handle multi-dimentional and nested arrays very well. Instead I used PHP's parse_str() method which did the trick and was a bit cleaner:
$rawdata = file_get_contents('php://input');
$urldecoded = urldecode($rawdata);
parse_str($urldecoded, $parsed);
$data = $parsed['data'];
(In my JS I'm posting an object where the payload is in the data property. Yours would be likely be different.)
I also went into my php.ini and cranked up max_memory and max_input_vars, but it didn't seem to solve my issue. I also spent a while chasing a red herring because I was using my error log to print the raw data, and I forgot that error_log has a limit on how many characters it will print unless you remember to increase it.
Anyway, hope this helps to anyone who finds themselves battling this issue.
use this function for getting true data
function get_real_post() {
function set_nested_value(&$arr, &$keys, &$value) {
$key = array_shift($keys);
if (count($keys)) {
// Got deeper to go
if (!array_key_exists($key, $arr)) {
// Make sure we can get deeper if we've not hit this key before
$arr[$key] = array();
} elseif (!is_array($arr[$key])) {
// This should never be relevant for well formed input data
throw new Exception("Setting a value and an array with the same key: $key");
}
set_nested_value($arr[$key], $keys, $value);
} elseif (empty($key)) {
// Setting an Array
$arr[] = $value;
} else {
// Setting an Object
$arr[$key] = $value;
}
}
$input = array();
$parts = array();
$rawdata=file_get_contents("php://input");
$rawdata=urldecode($rawdata);
$pairs = explode("&", $rawdata);
foreach ($pairs as $pair) {
$key_value = explode("=", $pair, 2);
preg_match_all("/([a-zA-Z0-9_]*)(?:\[([^\[\]]*(?:(?R)[^\[\]]*)*)\])?/", urldecode($key_value[0]), $parts);
$keys = array($parts[1][0]);
if (isset($parts[2][0])&&$parts[2][0]!="") {
array_pop($parts[2]); // Remove the blank one on the end
$keys = array_merge($keys, $parts[2]);
}
$value = urldecode($key_value[1]);
if ($value == "true") {
$value = true;
} else if ($value == "false") {
$value = false;
} else if (is_numeric($value)) {
if (strpos($value, ".") !== false) {
$num = floatval($value);
} else {
$num = intval($value);
}
if (strval($num) === $value) {
$value = $num;
}
}
set_nested_value($input, $keys, $value);
}
return $input;
}
I found this answer via a search and feel I should offer an alternative issue. In my case, my post wasn't too large, and yet the value I was submitting in the field was not showing up in the $_POST array. As it turned out, I accidentally had another field further down in my form with the same name. So:
<form>
<input type="text" name="field1">
<input type="text" name="field2">
<input type="text" name="field3">
<input type="text" name="field1">
<input type="submit">
</form>
When the $_POST variable is populate with data from that form, the value in your first field will be overwritten with the value in that last field with the same name. If the first field is required and you fill in a value, but the last field is not required and gets submitted empty, you will see similar symptoms as this question because the value $_POST['field1'] will show the value of the last element in you form which is empty.
TLDR:
Make sure to check for duplicate field names in your form!
For future reference:
On an Ubuntu box, I've been struggling with this issue since a relatively long time and used to adopt a workaround similar as described above to save the day.
I now tracked the issue down in my php.ini and finally found it in line with
max_input_nesting_level = 0
I commented the line above, restarted apache and all's fixed.
I had this same issue and it turned out that I was using AJAX to dynamically modify the input field based on other inputs in the form. The ajax function re-created the input and failed to include an input name.
Came across this (admittedly old) post while trying to find a fix for the bug of Javascript Object keys including square brackets in their names and then PHP getting confused and acting like it's never even heard of nesting. #Dalin has a good basic response, but it doesn't make a nested array, nor does it convert value types to boolean / number - hence my version (get_real_post() is also on GitHub).
Despite the name, this should work identically for _GET (ie, anything that php://input grabs).
/**
* Gets the _POST data with correct handling of nested brackets:
* "path[to][data[nested]]=value"
* "path"
* -> "to"
* -> "data[nested]" = value
* #return array
*/
function get_real_post() {
function set_nested_value(&$arr, &$keys, &$value) {
$key = array_shift($keys);
if (count($keys)) {
// Got deeper to go
if (!array_key_exists($key, $arr)) {
// Make sure we can get deeper if we've not hit this key before
$arr[$key] = array();
} elseif (!is_array($arr[$key])) {
// This should never be relevant for well formed input data
throw new Exception("Setting a value and an array with the same key: $key");
}
set_nested_value($arr[$key], $keys, $value);
} elseif (empty($key)) {
// Setting an Array
$arr[] = $value;
} else {
// Setting an Object
$arr[$key] = $value;
}
}
$input = array();
$parts = array();
$pairs = explode("&", file_get_contents("php://input"));
foreach ($pairs as $pair) {
$key_value = explode("=", $pair, 2);
preg_match_all("/([a-zA-Z0-9]*)(?:\[([^\[\]]*(?:(?R)[^\[\]]*)*)\])?/", urldecode($key_value[0]), $parts);
$keys = array($parts[1][0]);
if (!empty($parts[2][0])) {
array_pop($parts[2]); // Remove the blank one on the end
$keys = array_merge($keys, $parts[2]);
}
$value = urldecode($key_value[1]);
if ($value == "true") {
$value = true;
} else if ($value == "false") {
$value = false;
} else if (is_numeric($value)) {
if (strpos($value, ".") !== false) {
$num = floatval($value);
} else {
$num = intval($value);
}
if (strval($num) === $value) {
$value = $num;
}
}
set_nested_value($input, $keys, $value);
}
return $input;
}
I faced the same problem. My form was creating the input elements in a loop. Upto 1000 input elements were getting posted and print_r($_POST); showed all the posted values.
When the input elements in the form numbered above 1000, print_r($_POST) just vanished. As if the form was not getting posted at all. I reached your page of stackoverflow.com by google search.
Yes increasing post_max_size, memory_limit etc did not solve anything. I have practical experience that increasing memory_limit beyond 128M can be dangerous too. Once our live Apache hosting server got hung up. I was the experimenter :-)
max_input_vars was the only limiting factor.
echo ini_get('max_input_vars');
showed that it was 1000 by default.
Strangely max_input_vars was absent in the php.ini.
Anyway I added
max_input_vars=10000
in my php.ini and restarted Apache.
Now echo ini_get('max_input_vars'); showed that it had increased to 10000 and my large form data could be trapped after posting. Problem solved.
I see that the max_input_vars is PHP_INI_PERDIR type. Which means I cannot change its value for the individual php page where I need a higher value using ini_set('max_input_vars', 10000);
I had a similar issue and modifying the MAX_INPUT_VARS fixed it.
It's worth checking what you send, and what is received. In my case, the AJAX call contained a formatted array of all the values but the VAR_DUMP($this->input->post()); revealed missing values.
So, the obvious answer was as someone here said - max_input_vars.
You need to change the max_input_vars value of the php.ini file
step 1: Locate the php.ini file
step 1(a): Print php info values to find out all the PHP predefined configurations.
echo phpinfo(); //will print the php info values
step 1(b): Make a note of the php.ini file location from the phpinfo values printed in the previous step as shown in the image below
step 2: Edit the php.ini file.
step 2(a): Find the variable max_input_vars and remove the semicolon before it and set the value to 10000 like below
max_input_vars = 10000
step 3: Restart apache
sudo service apache2 restart
I faced same problem and i found a very valuable solution without increasing any limit or size in .ini.
We know that max_input_vars = 1000 by default and this is prefect.
Just do one thing concatenate array values and try to precise your number of variables under 1000.
Each variable name count one not their values.
For example:
<code>
$i = 10; //counts 1
$i = array(1,2,3,4,5); //counts 5
</code>
You can do this:
<code>
$i = 1_2_3_4_5; // counts 1
</code>
Hope understanding.
Related
Code used for processing form $_POST submissions is working well on most forms but suddenly broke on a new set of forms. I can't see any difference in the forms themselves as it's based purely on the posted values and I have it fixed but I am curious why the sudden problem.
There are some cases where specific post values are not to be processed and those, when they are not needed, are in $RemoveFields as a comma-separated list which is converted to an array and on the one set of forms, it doesn't matter if $RemoveFields has any value or not but on the other set it crashes when empty.
By adding a conditional I was able to make it work but can anyone tell me what the problem is on the original code? Both the old and new are below. The first works on only some of the forms while the second seems to work on all.
The original code:
// Remove unneeded fields specified in $RemoveFields variable
if (isset($RemoveFields) && !is_array($RemoveFields)) $RemoveFields = array($RemoveFields);
$filteredarray = array_diff_key($_POST, array_flip($RemoveFields));
The same code but with a conditional for the $filteredarray value:
// Remove unneeded fields specified in $RemoveFields variable
if (isset($RemoveFields) && !is_array($RemoveFields)) $RemoveFields = array($RemoveFields);
$filteredarray = (isset($RemoveFields)) ? array_diff_key($_POST, array_flip($RemoveFields)) : $_POST;
In the original code, you call array_flip($RemoveFields) even when $RemoveFields is not set. This fails because the argument to array_flip() must be an array.
You should use isset() to protect both lines of code:
if (isset($RemoveFields)) {
if (!is_array($RemoveFields)) {
$RemoveFields = array($RemoveFields);
}
$filteredarray = array_diff_key($_POST, array_flip($RemoveFields));
} else {
$filteredarray = $_POST;
}
So i'm having trouble getting a bit of code to work. Essentially what I want to do is:
in a foreach loop, if a given array value is set, compare that existing value to the current loop value, then set the existing value = current value (for the iteration) if the existing value is already greater than current val. Here is the code i'm working with:
if ($usedebayxml->ack == 'Success') {
foreach($usedebayxml->searchResult->item as $key => $value) {
if(isset($newarray[1]['TotalCost'])) {
if($newarray[1]['TotalCost'] > ((integer)$value->shippingInfo->shippingServiceCost + (integer)$value->sellingStatus->currentPrice)) {
$newarray[1]['Title'] = (string)$value->title ;
$newarray[1]['ShippingCost'] = (integer)$value->shippingInfo->shippingServiceCost;
$newarray[1]['Price'] = (integer)$value->sellingStatus->currentPrice;
$newarray[1]['Condition'] = 'New';
$newarray[1]['TotalCost'] = (integer)$value->shippingInfo->shippingServiceCost + (integer)$value->sellingStatus->currentPrice;
}
}
else
$newarray[1]['Title'] = (string)$value->title;
$newarray[1]['ShippingCost'] = (integer)$value->shippingInfo->shippingServiceCost;
$newarray[1]['Price'] = (integer)$value->sellingStatus->currentPrice;
$newarray[1]['Condition'] = 'Used';
$newarray[1]['TotalCost'] = (integer)$value->shippingInfo->shippingServiceCost + (integer)$value->sellingStatus->currentPrice;
}
}
With this code, what is returned is ultimately the values in the LAST key object in the xml file (im using simpleXML if that helps). In other words, i don't think the first if block (if isset) is being entered into, and the values are being set to whatever the values are for the current iteration. Can anyone see any flaw in my logic here? I've been stumped on this one for a while.
I am a supreme idiot. The logic here is fine, i was just missing a { for the opening else block. dur! After adding this, this bit of code works as intended :)
I'm surprised though that i wasn't throwing any errors without having this....I think that was probably throwing me off in determining why it wasn't working originally.
I will be reusing a Drupal db_query result set unpacking function many, many times in my code for a variety of different queries - I am using O-O and as such I want to reuse it and be as 'DRY' as possible.
Therefore I have tried to strip it down to the most generic functions so that as long as the $columns supplied match the columns used in the query and similarly in the $resultset, I can loop and assign values to keys, as is shown, and return a $rows[].
I've not yet come across the issue of trying to use a variable's value as a variable name (the $key), if it's just something I should avoid entirely, please say.
foreach($this->resultSet as $aRecord) {
$c = 0;
while (isset($this->columns[$c])) {
$value = $this->columns[$c];
$rows[$i] = array(
$key[$this->columns[$c]] => $aRecord->$value,
);
$c++;
}
$i++;
}
I've read through the following and am beginning to think this is just knowledge I'm missing in my PHP experience so far.
Can I use a generated variable name in PHP?
PHP use function return value as array
https://wiki.php.net/rfc/functionarraydereferencing
It felt wrong, and someone once told me that if you have to start writing complex functions in PHP you've probably missed an available function PHP already offers.. so true... thanks to (at the time of writing...) 'MrCode' for this suggestion.
$this->sql = "SELECT foo, bar FROM foobar";
$this->result = db_query($this->sql);
if ($this->result->rowCount() > 0) {
while ($row = $this->result->fetchAssoc()) {
$this->resultArray[] = $row;
}
}
OK so I'm making something to do some data mining but I do changes to an array (by overwritting previous array values) in a loop and they show that they've been changed but once I get outside of a greater loop the values change back to their original values.
Probably easier to give an example:
It starts off like this, turning a bunch of the parts of the array into the word "MATCH".
Now if I was to immediately dump the values of the array it would show that some values have changed to "MATCH" (ie, right after changing the value I would echo the array slot and it would show it's value to be "MATCH") However after I get outside the loop the array changes back to it's original contents
Here is a compressed version of the code:
//i've got this big loop for doing the main work
do {
//Set dat ticker
$q = 0;
// Run through entire previous scrape array to check for matches and mark them as unchanged
do {
if ($itemTitle[$i] == $prodURLS[$q]) {
$prodURLS[$q] = "MATCH";
echo "When the value is printing immediately it shows that it's changed: ".$prodURLS[$q]."<br>";
}
$q++;
} while ($q < $urlArraySize);
$i++;
} while ($i < $itemtitleArraySize);
//If I were to try to print the variable down here it would be reverted to like it was before I changed it to "MATCH"
print_r($prodURLS);
From running your code, setting the variables as follow, it works for me:
$prodURLS = array('a','b','c');
$itemTitle = array('a');
$urlArraySize = count($prodURLS);
$itemtitleArraySize = count($itemTitle);
$i = 0;
My only recommendations with only this amount of information, are:
To provide more context information, as madth3 suggests.
To check the scope in which you are setting/checking values. You may need the & operator to pass variables by reference, or the global keyword to use global variables.
To use the foreach loop, it will make your code smaller and easier to read. Also you won't need to count the size of the arrays and will have other advantages, e.g. in the use of associative arrays. Again, be careful about the use of variables by reference. For example:
foreach ($itemTitle as $item) {
foreach ($prodURLS as &$prod) {
if ($item == $prod) {
$prod = 'MATCH';
}
}
}
unset($prod); //Unset variable set by reference if you are going to use it later on!
Also, you may find useful some of the php array functions like array_walk. Check out the PHP Manual on the array functions reference.
Really, there isn't a lot that can be said from just the code you provided.
Good luck.
I was checking my script for vulnerabilities and was shocked the way i used to do in the past which is extremely insecure:
foreach ($_GET as $key => $value){
$$key = $value;
}
or shorter
extract( $_GET );
I altered with firebug some POST/GET variables to match a name i used in my script. they can be overwritten if the name would be guessed correctly.
So i thought i had to do it individually naming like this:
$allowed_vars =
$allowed_vars = array("time","hotfile","netload","megaupload","user","pfda","xyz","sara","amount_needed");
foreach ($_GET as $key => $value)
{
if (in_array($key,$allowed_vars))
{
$$key = $value;
}
}
This way saves some time than naming them individually.
What kind of automation have to be used for this?
I don't use any automatism of the kind.
I see no point in assigning request variables to global variables automatically.
If it's one or two variables, I could deal with them manually.
If there are more, I'd rather keep them as array members for the convenient handling.
Yet I am using some sort of whitelisting approach similar to yours.
but not to create global variables out of POST data but to add that data into SQL query.
Like in this simple helper function to produce SET statement:
function dbSet($fields) {
$set='';
foreach ($fields as $field) {
if (isset($_POST[$field])) {
$set.="`$field`='".mysql_real_escape_string($_POST[$field])."', ";
}
}
return substr($set, 0, -2);
}
$id = intval($_POST['id']);
$fields = explode(" ","name surname lastname address zip fax phone");
$query = "UPDATE $table SET ".dbSet($fields)." stamp=NOW() WHERE id=$id";
You can save even more time by not extrating them at all. Just use them from the $_GET array. The advantages of this are not just avoiding collision with script variables (or worse) but also that you don't have to update that "automatism" when you add request parameters.
When I am working with POST data, as from a form, I often process each explicitly:
$data = array();
$data['field1'] = someSaniFunction($_POST['field1']);
$data['field2'] = someOtherFunction($_POST['field2']);
...
In this way I ensure that each field is properly handled, and only the fields I expect are touched.
In my experience, you shouldn't transform data in the $_REQUEST array into variables using $$ as it gives the possibility of overwriting variables held within the current scope.
Instead, you should consider having a request object or array in which you filter the data and only access named variables that you require. This way, you do not have to keep extending your allowed variable names and still maintain security.
The ZF for example has a request object, and they recommend using a input filter when working on this data:
http://framework.zend.com/manual/en/zend.filter.input.html
You can use the extract function in a more secure way:
extract($_REQUEST, EXTR_SKIP);
This will not overwrite variables that already exist in your code. See here for other parameters you can use