PHP HTTP POST fails when cURL data > 1024 - php

Question

Note: solution at the end
If I attempt to do a HTTP POST of over 1024 characters, it fails. Why? Here is a minimal example:
recipient.php:
<?php
if (strlen(file_get_contents('php://input')) > 1000
|| strlen($HTTP_RAW_POST_DATA) > 1000) {
echo "This was a triumph.";
}
?>
sender.php:
<?php
function try_to_post($char_count) {
$url = 'http://gpx3quaa.joyent.us/test/recipient.php';
$post_data = str_repeat('x', $char_count);
$c = curl_init();
curl_setopt_array($c,
array( CURLOPT_URL => $url,
CURLOPT_HEADER => false,
CURLOPT_CONNECTTIMEOUT => 999,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => $post_data
)
);
$result = curl_exec($c);
echo "{$result}\n";
curl_close($c);
}
for ($i=1020;$i<1030;$i++) {
echo "Trying {$i} - ";
try_to_post($i);
}
?>
output:
Trying 1020 - This was a triumph.
Trying 1021 - This was a triumph.
Trying 1022 - This was a triumph.
Trying 1023 - This was a triumph.
Trying 1024 - This was a triumph.
Trying 1025 -
Trying 1026 -
Trying 1027 -
Trying 1028 -
Trying 1029 -
configuration:
PHP Version 5.2.6
libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3 libidn/1.8
lighttpd-1.4.19
Solution
Add the following option for cURL:
curl_setopt($ch,CURLOPT_HTTPHEADER,array("Expect:"));
The reason seems to be that any POST over 1024 character causes the "Expect: 100-continue" HTTP header to be sent, and Lighttpd 1.4.* does not support it. I found a ticket for it: http://redmine.lighttpd.net/issues/show/1017
They say it works in 1.5.

Answer 1

You can convince PHP's curl backend to stop doing the 100-continue-thing by setting an explicit request header:
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:'));
This way you can post a request however long you would ever want and curl will not do the dual phase post.
I've blogged about this nearly two years ago.

Answer 2

First thoughts...
The manual page for curl_setopt says of CURLOPT_POSTFIELDS
"The full data to post in a HTTP "POST"
operation. To post a file, prepend a
filename with # and use the full path.
This can either be passed as a
urlencoded string like
'para1=val1&para2=val2&...' or as an
array with the field name as key and
field data as value."
Could it be that your value is being treated as if it were urlencoded, and thus looks like a big long name with no value. Something somewhere is deciding to truncate that name.
Maybe you could alter it to something like
$post_data = "data=".str_repeat('x', $char_count);
Turns out this was too easy, and the problem was a little deeper. So, how to debug?
Find out exactly what CURL sends to the server
Another debugging tactic might be to formulate a curl command line which achieves the same thing, and have it output the HTTP request details as it makes them.
Testing the server by hand
You can eliminate the server from the equation by perform a request by hand, e.g. telnetting to port 80 on your server and sending it a request >1024 chars
POST /test/recipient.php HTTP/1.0
Host: gpx3quaa.joyent.us
Content-Length:1028
xxxxx(I put 1028 chars here, no point copying them all here!)
I got this response
HTTP/1.0 200 OK
Connection: close
Content-type: text/html; charset=UTF-8
Content-Length: 19
Date: Tue, 20 Jan 2009 21:35:16 GMT
Server: lighttpd/1.4.19
This was a triumph.Connection closed by foreign host.
So at least you now know it's all on the client side, possible some CURL option or configuration setting somewhere :(
Final Answer!
The problem intrigued me so I dug deeper
If you use CURLOPT_VERBOSE=>true, you'll see that CURL sends an extra header on the bigger posts:Expect: 100-Continue. Your lighttpd server doesn't like this, it would seem.
You can stop CURL from doing this by forcing it to use HTTP/1.0 with CURLOPT_HTTP_VERSION=>CURL_HTTP_VERSION_1_0 in your curl_setopt options array.

Answer 3

I had a similar problem with a IIS server, using SSL v3.
I kept getting the following cURL error when CURLOPT_POSTFIELDS was longer than 1024 :
52 - SSL read: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number, errno 0
Adding CURLOPT_HTTPHEADER : "Expect:" solved the problem for me.
Thank you so much for this thread!

Answer 4

Check if you have Suhosin patch enabled. By default, it cuts off POST data after certain number of indexes. You can bypass it in Suhosin config tho.