jQuery post request interrupted: Only half of post parameters arrive - php

Question

I have a noticed a strange phenomenon in my LAMP environment.
Over the frontend I execute an AJAX post request with jQuery like this:
$.post('save.php', {data1: d1, data2: d2, [...], dataN: dN})
The variables d1 to dN are collected from the website (e.g. from text inputs, textareas, checkboxes, etc.) with jQuery beforehand.
The file save.php takes the post parameters data1 to dataN and saves them in the database in one query.
The request takes about 500ms and works without problems unless I change pages (e.g. by clicking a link) during the request.
Normally, I would expect the request to be aborted and ignored (which would be fine) but (and this is the strange behaviour) the request seems to be completed but only with part of the data transmitted and thus saved.
That means for example, that the php script saves only data1 to data5 and sets data6 to dataN to empty.
The problem seems to be caused by the AJAX request already (not the php script) since fields $_POST['data6'] to $_POST['dataN'] are not set in php in this scenario.
So my questions:
Why does this happen (is this expected behaviour)?
How can I avoid it?
Update
The problem is neither jQuery nor php solely. jQuery collects the values correctly and tries to post them to php. I just validated it - it works.
The php script on the other hand handles everything it gets as expected - it just does not receive the whole request.
So the problem must be the interrupted request itself. Unlike I'd expect it does not abort or fail, it still transmits all the data until the cut off.
Then php gets this post data and starts handling it - obviously missing some information.
Update 2
I fixed the problem by adding a parameter eof after dataN and checking if it was set in php. This way I can be sure the whole request was transmitted.
Nevertheless this does not fix the source of the problem which I still don't understand.
Any help anyone?

Answer 1

Try the following actions to debug the problem:
Check post_max_size in your php settings and compare it with the data size you are posting.
User HTTP request builder, i.e. Use Fiddler to make an http request and check what it returns.
Use print_r($_POST); on the top of the save.php, to check what you are getting in it.
Use tool like Firebug to check what jQuery has posted.
You should also verify the json object on client side that you are posting. i.e. JSON.stringify(some_object);
Try posting some basic sample data { "data1":1, "data2":2, "data3":3, "data4":4, "data5":5 , "data6":6 }
Most probably you are sending to much data or likely data is invalid!
Edits:
Very Foolish act but lets say you posted count as well. so directly check isset($_POST['data'.$_POST['count']] )

Answer 2

I think we can rule out problems at the server site (unless it's some exotic or self-crafted server daemon), because nobody ever sends "end-of-data"-parameters with a HTTP POST request to make sure all data is really sent. This is handled by HTTP itself (see e.g. Detect end of HTTP request body). Moreover, I don't think that you have to check the Content-Length header when POSTing data to your server, simply because of the fact that nobody does this, ever. At least not in totally common circumstances like you describe them (sending Ajax POST through jQuery).
So I suppose that jQuery sends a syntactically correct POST, but it's cut off. My guess is that if you interrupt this data collecting by navigating to another page, jQuery builds an Ajax request out of the data which it was able to gather and sends a syntactically correct POST to your server, but with cut off data.
Since you're using Firebug, please go to its net tab and activate persist, so traffic data is not lost when navigating to another page. Then trigger your Ajax POST, navigate to another page (and thereby "interrupt" the Ajax call) and check in Firebug's net tab what data has actually been sent to the server by opening ALL the POST requests and checking the Headers tab (and inside this, the Request Headers tab).
My guess is that one of two things might happen:
You will see that the data sent to the server is cut off already in the headers being presented to you in Firebug's net tab and the Content-Length is calculated correctly according to the actual (cut off) length of the POST data. Otherwise, I'm sure the server would reject the request as Bad Request as a whole.
You will see that there are multiple POST requests, some of them (perhaps with the full, non-cut off data) actually interrupted and therefore never reaching the server, but at least one POST request (again, with the cut off data) that ist triggered by some other mechanism in your Javascript, i.e. not the trigger you thought, but by navigating to another page, more and other Ajax requests might be triggered (just a guess since I don't know your source code).
In either case, I think you'll find out that this problem ist client related and the server just processes the (incomplete, but (in terms of HTTP) syntactically valid) data the client sent to it.
From that point on, you could debug your Javascript and implement some mechanism that prevents sending incomplete data to your server. Again, it's hard to tell what to do exactly since I don't know the rest of your source code, but maybe there's some heavy action going on in collecting the data, and you could possibly make sure that the POST only happens if all the data is really collected. Or, perhaps you could prevent navigation until the Ajax request is completed or such things.
What might be interesting, if all of this doesn't make sense, would be to have a look at more of your source code, especially how the Ajax POST is triggered and if there are any other events and such if you navigate to another page. Sample data you're sending could also be interesting.
EDIT: I'd also like to point out that outputting data with console.log() might be misleading, since it's in no way guaranteed that this is the data actually being sent, it's just a logline which evaluates to the given output at the exact time when console.log() is called. That's why I suggested sniffing the network traffic, because then (and only then) you can be sure what is really being sent (and received).
Nonetheless, this is a little tricky if you're not used to it (and impossible if you use encrypted traffic e.g. by using HTTPS), so the Firebug net tab might be a good compromise.

Answer 3

You can verify the value of the Content-Length header being received by the PHP.
This value ought to have been calculated client side when running the POST query. If it does not match, that's your error then and there. And that's all the diagnostics you need - if the Content-Length does not match the POST data, reject the POST as invalid; no need of extra parameters (computing the POST data length might be a hassle, though). Also, you might want to investigate why does PHP, while decoding the POST and therefore being able to verify its length, nonetheless seems to accept a wrong length (maybe the information needed to detect the error is somewhere among the $_SERVER variables?).
If it does match though, and still data isn't arriving (i.e., the Content-Length is smaller, and correctly describes the cut-off POST), then it is proof that the POST was inspected after the cut-off, and therefore either the error is in the browser (or, unlikely, in jQuery) or there is something between the browser and the server (a proxy?) that is receiving an incomplete query (with Content-Length > Actual length) and is incorrectly rewriting it, making it appear "correct" to the server, instead of rejecting it out of hand.
Some testing of both the theory and the workaround
Executive summary: I got the former wrong, but the latter apparently right. See code below for a sample that works on my test system (Linux OpenSuSE 12.3, Apache).
I believed that a request with wrong Content-Length would be refused with a 400 Bad Request. I was wrong. It seems that at least my Apache is much more lenient.
I used this simple PHP code to access the key variables of interest to me
<?php
$f = file_get_contents("php://input");
print $_SERVER['CONTENT_LENGTH'];
print "\nLen: " . strlen($f) . "\n";
?>
and then I prepared a request with a wrong Content-Length sending it out using nc:
POST /p.php HTTP/1.0
Host: localhost
Content-Length: 666
answer=42
...and lo and behold, nc localhost 80 < request yields no 400 error:
HTTP/1.1 200 OK
Date: Fri, 14 Jun 2013 20:56:07 GMT
Server: Apache/2.2.22 (Linux/SUSE)
X-Powered-By: PHP/5.3.17
Vary: Accept-Encoding
Content-Length: 12
Content-Type: text/html
666
Len: 10
It occurred to me then that content length might well be off by one or two in case the request ended with carriage return, and what carriage return - LF? CRLF?. However, when I added simple HTML to be able to POST it from a browser
<form method="post" action="?"><input type="text" name="key" /><input type="submit" value="go" /></form>
I was able to verify that in Firefox (latest), IE8, Chrome (latest), all running on XP Pro SP3, the value of the Content-Length is the same as the strlen of php://input.
Except when the request is cut off, that is.
The only problem is that php://input is not always available even for POST data.
This leaves us still in a quandary:
IF THE ERROR IS AT THE NETWORK LEVEL, i.e., the POST is prepared and supplied with a correct Content-Length, but the interruption makes it so that the whole data is cut off as this comment by Horen seems to indicate:
So only the first couple of post parameters arrived, sometimes the value of one parameter was even interrupted in the middle
then really checking Content-Length will prevent PHP from handling an incomplete request:
<?php
if ('POST' == $_SERVER['SERVER_PROTOCOL'])
{
if (!isset($_SERVER['Content-Length']))
{
header($_SERVER['SERVER_PROTOCOL'] . ' 400 Bad Request', True, 400);
die();
}
if (strlen(file_get_contents('php://input'))!=(int)($_SERVER['Content-Length']))
{
header($_SERVER['SERVER_PROTOCOL'] . ' 400 Bad Request', True, 400);
die();
}
}
// ... go on
?>
ON THE OTHER HAND if the problem is in jQuery, i.e. somehow the interruption prevents jQuery from assembling the full POST, and yet the POST is made up, the Content-Length calculated of the incomplete data, and the packet sent off -- then my workaround can't possibly work, and the "telltale" extra field must be used, or... perhaps the .post function in jQuery might be extended to include a CRC field?

Answer 4

Post data looks just like GET:
Header1: somedata1\r\n
Header2: somedata2\r\n
...
HeaderN: somedataN\r\n
\r\n
data1=1&data2=2&...&dataN=N
When request is aborted, in some cases, last line may be passed only partially. So, here are some possible solutions:
Compare Content-Length and strlen($HTTP_RAW_POST_DATA)
Validate input data
Pass not so much data at one time

Answer 5

I have tried to recreate this problem using triggers, manually, changing server settings, doing my very best to #$%& things up, using different data sizes but I never ever got only half a request in PHP. Simply because apache will not invoke PHP untill the request is completely done. See this question about Reading “chunked” POST data in PHP
So the only thing that can go wrong is that Jquery only gathers part of the data and then makes a POST request. Using just $.post('save.php', data) as you mentioned, that wont happen. Its either working to gather the data or its waiting for a response from the server.
If you switch sites during the gathering, there wont be a request. And if you switch after the request has been made and you move away quickly, before all data has been transmitted, the apache server will see it as half a request and wont invoke PHP.
Some suggestions:
Is it possible that you are using seperate pages for succesfull and partial requests? Because PHP does only add the first 1000 elements to $_POST and perhaps the failed requests have more then data1000=data elements? So there wont be an EOF param.
Is it possible that you are gathering the data in a global var in javascript and have an onbeforeunload method that sends data as well? Because then there might only be half the data in the POST.
Can you share some information on the data you are seding? Are there a lot of small elements (like data1 till data10000) or a few large once?
Is it always the same element that you receive last? Like always data6 as you mention? Because if it is, the chances of a failed attempt always at the exact same dataN field would be very slim.

Answer 6

My problem was there were too many variables in one of my post objects.
PHP has a max_input_vars variable which is set to 1000 by default.
I added this line to my .htaccess file (since I don't have access to the php.ini file):
php_value max_input_vars 5000
Problem solved!

Answer 7

Can you check your host log at
/var/log/messages
Last time i had "missing"post variables at php i found that i was sending null ASCII chars and the server(CentOS) was considering it an attack, then dropping those specific variables... Took me a week to figure it out! This was the server log response:
suhosin[1173]: ALERT - ASCII-NUL chars not allowed within request variables - dropped variable 'data3' (attacker '192.168.0.37', file '/var/www/upload_reader.php')
If that is your problem, tyr to, with js, compress your variables, encode them with base64. Post them with ajax, then receive then at php, decode64 then uncompress! That solved for me ;)

Answer 8

Solved the problem by increasing max_input_vars in my server's php.ini file
Since I had more than 1000 variables in the array, only part of them was received by the server!
Hope this helps someone!

Answer 9

In order to find our more about what is happening, why not properly code you ajax request using jQuery's ajax function. Use all the callback functions to track what happened to your call or what came back? The element type is set to POST and the element data carries whatever object structure { ... } you like.
$.ajax({
url : "save.php",
type : "POST",
data : {
"ajax_call" : "SOME_CUSTOM_AJAX_REQUEST_REFERENCE",
"data1" : data1,
"data2" : data2,
"data2" : data2,
"dataN" : dataN
},
//dataType : "html", contentType: "text/html; charset=utf-8",
dataType : "json", contentType: "application/json; charset=utf-8",
beforeSend: function () {
//alert('before send...');
},
dataFilter: function () {
//alert('data filter...');
},
success: function(data, textStatus, jqXHR) {
//alert('success...');
var response = JSON.parse(jqXHR.responseText, true);
if (undefined != response.data) {
my_error_function();
}
my_response_function(response.data);
},
error: function(jqXHR, textStatus, errorThrown) {
//alert('error...');
},
complete: function (xhr, status) {
//alert('end of call...');
my_continuation_function();
}
});

Answer 10

Before send a request, set "onbeforepageunload" handler for document(to prohibit the transition to another page), and unbind after success.
To example:
$(document).on('unload', function(e){
e.preventDefault();
// Here you can display a message, you need to wait a bit
return false;
});

Answer 11

A guess - the suhosin function on your Ubuntu/Debian server causes the field to be cut off?

Answer 12

I have the same problem. POST size is about 650KB, and gets corrupted if closing browser window or refreshing page in case of ajax, before post is completed. ajax.success() is not fired, but partial data is posted to "save.php" with 200 OK status. $_SERVER's Content-length is of no use as suggested elswhere since it matches the actual content-length of the partial data.
I figured out 2 ways of overcoming this:
as you propose, append a post variable at the end. Seems to work, although it seems a bit risky.
make your "save.php" script save to a temporary db column, and then use the ajax.success() to call another php script, say savefinal.php, without passing any data, which transfers data from the temp column to the final column (or just flags this data as valid). That way if post is interrupted data will only reside on the temp column on the database (or will not be flagged as valid).
The .success() is not called if post is interrupted, so this should work.
I presume this is a jquery bug sending a wrong (very small) content-length to apache, and apache is forced to assume that the post request has completed, but I'm not really sure.

Answer 13

Why does this happen (is this expected behaviour)?
Was also looking for the cause of this strange behaviour when some variables from post were missing and came to this question where the very similar behaviour is explained with the slow connection of the web client sending a POST request with multipart/form-data.
Here goes the mentioned question:
I am facing a problem when a remote web client with slow connection
fails to send complete POST request with multipart/form-data content
but PHP still uses partially received data to populate $_POST array.
As a result one value in $_POST array can be incomplete and more
values can be missing.
See How to check for incomplete POST request in PHP
How can I avoid it?
There you can find the recommended solution, as well. Nevertheless, the same solution was already proposed by you.
You can add field <input type="hidden" name="complete"> (for example)
as the last parameter. in PHP check firstly whether this parameter was
sent from client. if this parameter sent - you can be sure that you
got the entire data.