Attempting to submit a form with CURL, both via PHP and the command line. The response from the server consists of null content (the headers posted below).
When the same URL is submitted via a browser, the response consists of a proper webapge.
Have tried submitting the CURL request parameters via POST and GET via each of the following command line curl flags "-d" "-F" and "-G".
If the query string parameters are posted with "-d" flag, resulting header is:
HTTP/1.1 302 Moved Temporarily
Date: Thu, 02 Jun 2011 21:41:54 GMT
Server: Apache
Set-Cookie: JSESSIONID=DC5F435A96A353289F58593D54B89570; Path=/XXXXXXX
P3P: CP="CAO PSA OUR"
Location: http://www.XXXXXXXX.com/
Content-Length: 0
Connection: close
Content-Type: text/html;charset=UTF-8
Set-Cookie: XXXXXXXXXXXXXXXX=1318103232.20480.0000; path=/
If the query string parameters are posted with "-F" flag, the resulting header is:
HTTP/1.1 100 Continue
HTTP/1.1 500 Internal Server Error
Date: Thu, 02 Jun 2011 21:52:54 GMT
Server: Apache
Content-Length: 1677
Connection: close
Content-Type: text/html;charset=utf-8
Set-Cookie: XXXXXXXXXXXXXX=1318103232.20480.0000; path=/
Vary: Accept-Encoding
<html><head><title>Apache Tomcat/5.5.26 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.servlet.ServletException: Servlet execution threw an exception<br>
</pre></p><p><b>root cause</b> <pre>java.lang.NoClassDefFoundError: com/oreilly/servlet/multipart/MultipartParser<br>
com.corsis.tuesday.servlet.mp.MPRequest.<init>(MPRequest.java:27)<br>
com.corsis.tuesday.servlet.mp.MPRequest.<init>(MPRequest.java:21)<br>
com.corsis.tuesday.servlet.TuesdayServlet.doPost(TuesdayServlet.java:494)<br>
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)<br>
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)<br>
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/5.5.26 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/5.5.26</h3></body></html>
Questions:
What might cause a server to respond different depending on the nature of the CURL request.
How to successfully submit request via CURL?
HTTP/1.1 100 Continue
I had problems associated with this header before. Some servers simply do not understand it. Try this option to override Expect header.
curl_setopt( $curl_handle, CURLOPT_HTTPHEADER, array( 'Expect:' ) );
To add to what Richard said, I have seen cases where servers check the User-Agent string and behave differently based on its value.
I have just had an experience with this and what fixed it was surprising. In my situation I was logging into a server so I could upload a file, have the server do work on it, and then download the new file. I did this in Chrome first and used the dev tools to capture over 100 HTTP requests in this simple transaction. Most are simply grabbing resources I don't need if I am trying to do all of this from the command line, so I filtered out only the ones I knew at a minimum I should need.
Initially this boiled down to a GET to set the cookie and log in with a username and password, a POST to upload the file, a POST to execute the work on the file, and a GET to retrieve the new file. I could not get the first POST to actually work though. The response from that POST is supposed to be information containing the upload ID, time uploaded, etc, but instead I was getting empty JSON lists even though the status was 200 OK.
I used CURL to spoof the requests from the browser exactly (copying the User-Agent, overriding Expect, etc) and was still getting nothing. Then I started arbitrarily adding in some of the requests that I captured from Chrome between the first GET and POST, and low and behold after adding in a GET request for the JSON history before the POST the POST actually returned what it was supposed to.
TL;DR Some websites require more requests after the initial log in before you can POST. I would try to capture a successful exchange between the server and browser and look at all of the requests. Some requests might not be as superfluous as the seem.
Related
What are this Weird Characters on my Ajax result 138d on the start and 0 on the end of my ajax result. how to disable this?
138d
{"feeds":[{"pubdate":"Sun, 28 Nov 2021 23:00:00 EST"]}
0
Your HTTP client is buggy. It improperly extracted the body from the HTTP response.
There are three ways to signaling the end of an HTTP response:
Using a Content-Length header.
Using the chunked transfer coding.
Closing the socket.
The first two methods allow more than one request-response exchange to occur over the same connection, which is far more efficient than creating a new connection for each request. (Especially for HTTPS.)
The Content-Length header is quite simple to use, but it requires knowing the size of the message body before starting to send it. The chunked transfer coding doesn't have that limitation. It allows servers to start sending a response before its size becomes known.
An HTTP response that uses the chunked transfer coding might look like this:
HTTP/1.1 200 OK␍␊
Transfer-Encoding: chunked␍␊
Content-Type: application/json␍␊
␍␊
28␍␊
..............................␍␊
28␍␊
..............................␊␍␊
0␍␊
The message body of that response is
............................................................␊
The response that returned your JSON apparently used the chunked transfer coding. And whatever you used to parse the HTTP response incorrectly returned the still-chunked message body. This is a bug. Quote RFC2616,
All HTTP/1.1 applications MUST be able to receive and decode the
"chunked" transfer-coding
I'm trying to submit a (java servlet) form using CURL in PHP, but it seems like there is a problem with the parameters. I cant really understand why its happening since I'm testing the CURL with a identical string parameters that is being used by the browser.
After some research in diverse forums I wasn't able to find a solution to my particular problem.
this is the POSTFIELDS string generated by the browser (and working):
submissionType=pd&__multiselect_PostCodeList=&selectedPostCode=01&selectedPostCode=02&selectedPostCode=03&__multiselect_selectedPostCodes=
and I'm using and identical (for testing) string in the PHP script but it im getting a HTML file as a answers telling "Missing parameters in search query".
I believe that the form
__multiselect_PostCodeList=
&selectedPostCode=01
&selectedPostCode=02
&selectedPostCode=03
&__multiselect_selectedPostCodes=
is quite wired (never see before this) and I'm wondering that it can be the reason of why the post is not working from CURL.
The form seems to be successfully submitted since I'm getting this header
HTTP/1.1 200 OK
Date: Wed, 07 Aug 2013 08:02:56 GMT
Content-length: 1791
Content-type: text/html;charset=UTF-8
X-Powered-By: Servlet/2.4 JSP/2.0
Vary: Accept-Encoding
Content-Encoding: gzip
Connection: Keep-Alive
Note: I tried submitting the same form from Lynx and I'm also getting the same result ("Missing parameters in search query"). So it seems like its only working with browsers like Mozilla or Chrome.
Please some help will be really appreciated, I don't have any more ideas at this point.
Thanks!
Oscar
Short question: Can I rely on the behaviour of header('.', TRUE, 404);, header('.', TRUE, 501);, etc.?
Now, let me elaborate.
Before asking my question, I'll present a few usages of the header() call and the corresponding HTTP response code generated by it.
header('HTTP/1.1 404 Bummer!'); -- HTTP/1.1 404 Bummer!
header('HTTP/1.1 501 I am sick'); -- HTTP/1.1 501 I am sick
header('Bummer!', TRUE, 404); -- HTTP/1.1 404 Not Found
header('I am sick', TRUE, 501); -- HTTP/1.1 501 Method Not Implemented
So, we can see that if we use the third parameter while the first parameter is junk, the first parameter is ignored. However, the documentation at http://php.net/manual/en/function.header.php says:
Note that this parameter only has an effect if the string is not
empty.
So, we still have to put something in the first parameter. This looks a bit ugly to me because the $string is ignored when we specify $http_response_code in the third parameter but we are still required to put some value for $string even though it will never be used.
But I can understand why this turned out this way. Traditionally, header() accepted only parameter and we could set arbitrary response codes like in the first two examples. The second and third parameters were added later as optional paramters. So, we have to specify something for the first parameter if we want to use the second and third. Moreover, sometimes we may need to put valid headers in the first parameter along with a valid response code in the third parameter. I have included one such example in the end.
So, I'm planning to use this function in this manner in my code: header('.', TRUE, 404);, header('.', TRUE, 501);, etc. As per the examples above, it'll produce correct HTTP responses as per the standard. I want to know if I can rely on this behaviour. I ask this question because I can't find the documentation mentioning explicitly that the first parameter ($string) is going to be ignored when we provide the third ($http_response_code).
BTW, I know that the first argument can be useful in situations like this.
header('Foo: Bar', TRUE, 501); leads to:
HTTP/1.1 501 Method Not Implemented
Date: Sun, 09 Oct 2011 19:01:19 GMT
Server: Apache/2.2.20 (Debian)
X-Powered-By: PHP/5.3.8-1
Foo: Bar
Vary: Accept-Encoding
Connection: close
Content-Type: text/html
A more practical example is header('Location: http://example.com/there/', TRUE, 307); which would generate:
HTTP/1.1 307 Temporary Redirect
Date: Sun, 09 Oct 2011 19:09:29 GMT
Server: Apache/2.2.20 (Debian)
X-Powered-By: PHP/5.3.8-1
Location: http://example.com/there/
Vary: Accept-Encoding
Content-Type: text/html
Anyway, back to my question. Can I rely on the behaviour of header('.', TRUE, 404);, header('.', TRUE, 501);, etc.?
Not sure if you're looking for this, but I guess you don't want to build the HTTP status line:
header('.', TRUE, 501);
Makes my Apache return this status line:
HTTP/1.1 500 Internal Server Error
But
header(' ', TRUE, 501);
Makes my Apache return this status line:
HTTP/1.1 501 Not Implemented
Which might be what you're looking for.
Maybe you just want to build the status line?
header('HTTP/1.1 501 No Implementation for you :)');
Which makes my Apache return this status line:
HTTP/1.1 501 No Implementation for you :)
Traditionally, it's done this way:
header('HTTP/1.1 501');
Which get's some default status line message text then:
HTTP/1.1 501 Method Not Implemented
This does not change the protocol version:
header('HTTP/1.0 404');
As it gives:
HTTP/1.1 404 Not Found
Hope this helps.
Update by Susam:
With PHP 5.4, a simple and elegant way to do this is:
http_response_code(404);
or
http_response_code(501);
There's currently no function to only set the numeric HTTP response code. However, I think PHP 5.4 will finally fill this void with http_response_code().
I wouldn't use header(".", TRUE, 501) myself. It might work on some web server software but very unlikely to work on them all.
I'm working on carddav client. As server i use davical v. 0.9.9.6. I don't understand why i'm getting invalid content-type error when http headers contains correct value. I look into source code and found this condition:
if ( isset($_SERVER['CONTENT_LENGTH']) && $_SERVER['CONTENT_LENGTH'] > 7) {...
After little research I found php set $_SERVER['CONTENT_LENGTH'] only with POST method and uploading file. Is there any way to configure php to always set $_SERVER['CONTENT_LENGTH']?
I'm asking generally, not only for this case...
//EDIT
I'm doing HTTP PUT request to davical server (using php curl).
PUT /caldav.php/testuser/contacts/newc.vcf HTTP/1.1
Host: davical
Content-Type: text/vcard;
BEGIN:VCARD
VERSION:3.0
FN:ME
...
On davical side is condition testing CONTENT_LENGTH which is not set. So it's a davical bug?
//EDIT 2
Finally I figure it out!
PUT request with calback readfunc requires set INFILE_SIZE via curl_setopt(...)
There is none auto value and put Content-Length field manualy into header is also wrong.
Example (incorrect):
// PUT REQUEST
curl_setopt($ch,CURLOPT_HTTPHEADER,"Content-Length: $length"); //mistake
curl_setopt($ch,CURLOPT_PUT,true);
curl_setopt($ch,CURLOPT_READFUNCTION,array($this,'readfunc'));
....
--------------------------------------------------------------
// WIRESHARK TCP STREAM DUMP
PUT /caldav.php/testuser/contacts/novy.vcf HTTP/1.1
Authorization: Basic xxxxxxxxxxxxxxx
Host: davical
Accept: */*
Content-Type: text/vcard
Content-Length: xxx
Expect: 100-continue
HTTP/1.1 100 Continue
155
BEGIN:VCARD
VERSION:3.0
...
END:VCARD
0
HTTP/1.1 200 OK
----------------------------------------------------------------
// On server side
isset($_SERVER['CONTENT_LENGTH'])==false
Second (correct) example
// PUT REQUEST
curl_setopt($ch,CURLOPT_INFILESIZE,$length);
curl_setopt($ch,CURLOPT_PUT,true);
curl_setopt($ch,CURLOPT_READFUNCTION,array($this,'readfunc'));
....
--------------------------------------------------------------
// WIRESHARK TCP STREAM DUMP
PUT /caldav.php/testuser/contacts/novy.vcf HTTP/1.1
Authorization: Basic xxxxxxxxxxxxxxx
Host: davical
Accept: */*
Content-Type: text/vcard
Content-Length: xxx
Expect: 100-continue
HTTP/1.1 100 Continue
BEGIN:VCARD
VERSION:3.0
...
END:VCARD
HTTP/1.1 200 OK
----------------------------------------------------------------
// On server side
isset($_SERVER['CONTENT_LENGTH'])==true
Although i have never used CONTENT_LENGHT i can tell you why this is probably happening:
In a request, you don't have to set the Content-Lenght header... IT IS NOT MANDATORY. Except for specific situations. If your POSTed content is of type "multipart/form-data" it becomes necessary to use content-lenght for each part because each part is seperated by a boundary and each part will have its own headers...
For example:
Content-Type: MultiPart/Form-Data
Boundary: #FGJ4823024562DGGRT3455
MyData=1&Username=Blabla&Password=Blue
#FGJ4823024562DGGRT3455==
Content-Type: image/jpef:base64
Content-Lenght: 256
HNSIFRTGNOHVDFNSIAH$5346twSADVni56hntgsIGHFNR$Iasdf==
So here this is a crude example of what a multi part request works, you see that the second part has a content-lenght. This is why sometimes the content-lenght is set and sometimes not, because you need to read X bytes before finding another boundary and extract the correct data.
It doesn't mean your server will never send it in in other cases, but my 2 cents are this is the case right now. Its because you are not in POST, but in some other modes.
Only requests that have a request body have a content length request header (or at least only then it makes sense) and so therefore the $_SERVER variable is set.
If you need it to be always set (which I think is bogus), you can do this yourself on the very beginning of your script:
isset($_SERVER['CONTENT_LENGTH']) && $_SERVER['CONTENT_LENGTH'] = 0;
Assuming that if it is not set, it's of zero length. See as well Improved handling of HTTP requests in PHP.
You could probably set them by yourself. Why do you need that this values are set? And what should they set to?
Maybe you're missing information on $_SERVER['CONTENT_TYPE'] or
$_SERVER['CONTENT_LENGTH'] as I did. On POST-requests these are
available in addition to those listed above.
-> http://www.php.net/manual/en/reserved.variables.server.php#86495
i modified my previous code. you can see my previous post if your intersted setting cookie through curl
But here is a fresh beginning my new code looks linke this
i have a php file using curl like this
<?php
$ch=curl_init();
$url="http://localhost/javascript%20cookies/test_cookies.php";
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_COOKIEFILE,dirname(__FILE__) . "/cookie.txt");
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,1);
curl_exec($ch);
curl_close($ch);
?>
the file test_cookies.php looks like this
<?php
if($_COOKIE['user']==1)
{
header("Set-Cookie:user=1; color=blue");
header("Location:http://localhost/javascript%20cookies/test_cookies.html");
}
?>
the file test_cookies.html has some javascript that checks for cookies and if it finds those cookies then it displays the text accordingly.
the php file with curl code is sending the cookies and the test_cookies.php is setting the cookie and redirecting to the page test_cookies.html but this page is not receiving the cookies and thus it is not not showing the content accordingly.
can somebody tell me whats the problem now?
here are the headers i get displayed in firefox on setting CURLOPT_HEADER to true
HTTP/1.1 302 Found Date: Mon, 16 May 2011 15:03:59 GMT Server: Apache/2.2.14 (Win32) DAV/2 mod_ssl/2.2.14 OpenSSL/0.9.8l mod_autoindex_color PHP/5.3.1 mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.10.1 X-Powered-By: PHP/5.3.1 Set-Cookie: user=1; color=blue Location: http://localhost/javascript%20cookies/test_cookies.html Content-Length: 0 Content-Type: text/html HTTP/1.1 200 OK Date: Mon, 16 May 2011 15:03:59 GMT Server: Apache/2.2.14 (Win32) DAV/2 mod_ssl/2.2.14 OpenSSL/0.9.8l mod_autoindex_color PHP/5.3.1 mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.10.1 Last-Modified: Mon, 16 May 2011 12:13:24 GMT ETag: "11000000013d0c-493-4a363950a70f3" Accept-Ranges: bytes Content-Length: 1171 Content-Type: text/html
you can see that there are two set of headers displayed.is this because i am making 2 calls to header?
It is not clear what you are trying to achieve with the code. You'd get better help if you explain why you are doing this. It is probable that your approach is wrong and you could get the ultimate result with a different/simpler approach.
And now to answer your question, this is what happens:
From your browser, you send a GET request to curl.php (the name I use for your first file),
curl in that file checks cookie.txt and finds user=1, so it sends a GET request to test_cookies.php and sends the cookie with the request
test_cookies.php sees that user=1 is true, so it sends a header to curl.php and asks it to set two cookies: user=1 and color=blue (for the first cookie this is pointless, user=1 is already set, but let's ignore this as no harm is done)
You have not set CURLOPT_COOKIEJAR option, so when curl.php receives the set cookie header it does not give a damn (nothing happens)
Next, test_cookies.php sends a redirect header to curl.php, since you have set CURLOPT_FOLLOWLOCATION,1, curl.php sends another GET request, this time to get test_cookies.html
Content of test_cookies.html is returned to curl.php,
curl_exec($ch); causes the returned content (source of test_cookies.html) be echoed back to your browser,
Your browser parses what it received and the javascript is executed. It checks for a cookie named user and does not find one, so it displays content for when there is no cookie (because there isn't).
Now, you may wonder what happens if you add:
curl_setopt($ch,CURLOPT_COOKIEJAR,dirname(__FILE__) . "/cookie.txt");
What happens is that your cookie.txt will be updated in step 4 and will have two cookies; user=1 and color=blue. But this does NOT give you the result you expect. Because header("Set-Cookie:user=1; color=blue"); is instructing curl to set the cookie, and curl does this by storing the cookies in the file you specified in CURLOPT_COOKIEJAR option. So, even though you added that option, when you reach step 8 javascript will not find that cookie, because the browser you use does not know or care about your cookie.txt, it looks somewhere else to check if a cookie exists. In Google Chrome for instance the format is SQLite and for XP the cookies are stored in %USERPROFILE%\Local Settings\Application Data\Google\Chrome\User Data\Default\Cookies.
i will now explain what i was trying to do.
i had an html page that asked for some verification code and then it sent that verification code to a php script which on verifying the code set a cookie and redirected to the same html page.
the html page had some externally linked javascript which on checking the cookie value displayed the content of the page with some tweaking.
i am now writing the code for this
the html file with content and form
test_cookies.html
//some css,javascript and html and then a form
<form method="post" action="http://localhost/javascript%20cookies/test_cookies.php">
the php file which verifies the code
test_cookies.php
if($_POST['value']=="code")
setcookie("user",1);
if($_POST['value']!="code")
setcookie("user",1,time()-1);
header("Location:http://localhost/javascript%20cookies/test_cookies.html");
and now the php file with curl code
curl_cookies.php
<?php
$ch=curl_init();
$url="http://localhost/javascript%20cookies/test_cookies.php";
$post="value=code"; //here i have hard-coded the pst value for a demo but i could have got this from the user
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,1);
curl_setopt($ch,CURLOPT_POSTFIELDS,$post);
curl_setopt($ch,CURLOPT_HEADER,1);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
$res=curl_exec($ch);
curl_close($ch);
preg_match('/Location: (.*)\s/',$res,$location);
preg_match('/Set-Cookie: (.*)\s/',$res,$cookie);
$cookie=rtrim($cookie[0])."; path=/ "; //path needed to be changed because curl_cookies.php and test_cookies.html are in different directories.
header($cookie);
header($location[0]);
?>
this finally worked and the browser is displaying the tweaked content. this thing taught me a lot about http.
Thanks to all who helped me when i was in dark.