I have a problem between CURL and PHP. I tried to connect to an Exchange 2010 Server via EWS. The connection to the server works fine in command line using CURL but not with PHP Curl extension.
I attached an image with the debug information. On the left you see command line output, on the right PHP verbose output. When the PHP Curl Extension throws an error "Connection closure while negotiation auth (HTTP 1.0?)" the command line continues with a third HTTP request with results in an HTTP/1.1 302 Found:
Some additional information:
I use this library for CURL-Requests: https://github.com/jamesiarmes/php-ntlm/blob/master/src/SoapClient.php
we have about 80 exchange servers where there is no problem. Only with this server there is a problem. Our customer told about a software called "Sophos" used as a proxy for the webserver
CURLOPT_HTTPAUTH is CURLAUTH_NTLM
PHP Version 7.3.1 / 7.3.9 also tested
cURL Information 7.63.0 / 7.52.1 also tested
Does anybody know why the PHP Curl Extension closes the connection before the third request? Is this a bug of the extension, can I use a constant of PHP Curl to avoid that or is there another solution?
The connection is closed because the server says so. See your screenshot, one line above where the "What is the problem here?" points.
HTTP/1.1 401 Unauthorized
[...]
Connection: close
Content-Type: application/x-asmx
And probably the server closes the connection afterwards.
So that is not an action, but a result. The message is emitted in Curl_http_readwrite_headers:
#if defined(USE_NTLM)
if(conn->bits.close &&
(((data->req.httpcode == 401) &&
(conn->http_ntlm_state == NTLMSTATE_TYPE2)) ||
((data->req.httpcode == 407) &&
(conn->proxy_ntlm_state == NTLMSTATE_TYPE2)))) {
infof(data, "Connection closure while negotiating auth (HTTP 1.0?)\n");
data->state.authproblem = TRUE;
}
#endif
#if defined(USE_SPNEGO)
if(conn->bits.close &&
(((data->req.httpcode == 401) &&
(conn->http_negotiate_state == GSS_AUTHRECV)) ||
((data->req.httpcode == 407) &&
(conn->proxy_negotiate_state == GSS_AUTHRECV)))) {
infof(data, "Connection closure while negotiating auth (HTTP 1.0?)\n");
data->state.authproblem = TRUE;
}
Presumably from the first block (NTLM), but these are the two occurrences, and they are next to each other anyway.
Fun fact: the same function only a lot later checks for the presence of a Connection: close header, so having the mystical flag conn->bits.close set probably means that the server dropped the connection already and it was detected on socket-level.
Side remark: the two sides of the comparison show very dissimilar interactions. On the left there is a practically empty GET request (Host, Authorization, User-Agent and Accept headers are provided), while on the right side there is a lot more complicated POST request (same headers plus Method, SOAPAction, an empty content with Expect for continuation).
I have faced such like this problem by using SoapClient
and Microsoft Exchange 2010 Server, and the trick was by changing the header array option 'Expect: 100-continue'
to 'Expect: 200-ok'
protected function buildHeaders($action)
{
return array(
'Method: POST',
'Connection: Keep-Alive',
'User-Agent: PHP-SOAP-CURL',
'Content-Type: text/xml; charset=utf-8',
"SOAPAction: \"$action\"",
'Expect: 200-ok',
);
}
The 100 (Continue) status code indicates that the initial part of a request has been received and has not yet been rejected by the server.
The 200 (OK) status code indicats that the request has succeeded. The meaning of a success varies depending on the HTTP method.
You can also check this out HTTP status codes
I wish this could help
Related
I'm currently developing a REST API in which I need to return a 102 HTTP status code (processing) while I'm generating an export.
Workflow :
POST /exports
return 201 with data
GET /exports/id
return 102 with data if the export is processing
return 200 with data if the export is completed
When I try to retrieve export data while it's processing, there are no response headers: response headers are missing with 102 HTTP status code. If I change the status code with 2xx for instance, it's working fine. I can't figure out. Is there anything specific with the 102 HTTP status code? When I say response headers are missing I mean: Chrome > Developer tools > Network Tab > Click on request > Headers tab > Only showing "General" and "Request Headers" (same with FF & Postman).
Used Technologies :
Ubuntu 18.04 LTS
PHP 7.2 (latest release)
laravel/lumen 5.6.21
Apache 2.4.29
Controller Code :
/**
* Return export by id
*
* #param int $id
* #return \Illuminate\Http\JsonResponse
*
* #throws AuthorizationException
* #throws ModelNotFoundException
*/
public function getItem(int $id)
{
if($export = Export::find($id))
{
$this->authorize(__FUNCTION__, $export);
if($export->status != Export::STATUS_COMPLETED)
{
return response()->json($export, 102);
}
return response()->json($export);
}
throw new ModelNotFoundException();
}
Expected Request Headers :
Access-Control-Allow-Origin
Cache-Control
Connection
Content-Length
Content-Type
Date
Proxy-Connection
Server
Vary
EDIT
I should have mentioned that it worked on my previous config :
Ubuntu 17.10 LTS
PHP 7.1 (latest release)
laravel/lumen 5.6.16
Apache 2.4.27
I haven't found in any release notes what could have impacted the request answer.
Use HTTP 202 Accepted instead.
See: https://softwareengineering.stackexchange.com/questions/316208/http-status-code-for-still-processing
Explained:
RFC 2518 says "The server MUST send a final response after the request has been completed", and this is interpreted to mean that your server needs to send an final response code in addition to the initial HTTP 102. Not doing so creates timeout problems for clients waiting for a final response but not getting one. Firefox will choke, Chrome will time out and convert it into HTTP 200 OK. cURL will inform that there is unread content.
So use HTTP 102 Processing only as a hint to clients that, "Okay, but this might take a minute...", after which you follow up with a final code and response body.
If it is a long running process that you want to periodically poll, use HTTP 202 Accepted and close the response.
It's also worth noting that http_response_code() does not handle HTTP 102.
Bad Example:
<?php header('HTTP/1.1 102 Processing'); exit; ?>
Good Example:
<?php
header('HTTP/1.1 102 Processing'); // let client know it might take a while
sleep(2); // do stuff that takes a while
header('HTTP/1.1 200 OK'); // counterintuitive, but works
I am submitting a CURL Post request to an API which, upon success, returns status "201 Created" with the URL of the resource in the LOCATION part of the header. What I'd like is to automatically retrieve the newly created resource but so far haven't been able to do so. I've tried setting curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); but to no avail. it's worth noting that the resource does require a GET request.
I'm not sure if it's the status code being a 201, or if the request method needs to change from POST to GET, but for some reason it's not following the LOCATION header. Fetching curl_getinfo($ch,CURLINFO_EFFECTIVE_URL); seems to confirm this as the result is the same as the original URL, not the new LOCATION.
As a last resort I have considered simply parsing the headers and creating a new CURL request, but this would not be optimal and I'm guessing that I'm just missing something simple to make this work as desired.
How can I get CURL to automatically follow and submit a GET request to the LOCATION returned with a 201 response?
You can't.
With curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); curl follow the response LOCATION only for 3xx status codes.
AFAIK and stating as the documentation there is no way to force curl to follow-location with a 201 response.
You have to parse the header, fetch the LOCATION and then issue a second curl request.
Following a location with a status different than 3xx would be an anomaly. Also from the curl command line tool and C library documentation: -L, --location -- (HTTP) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place
Giving a quick look at the curl source code you find on /lib/http.c the function Curl_http_readwrite_headers.
Location follow is handled inside with this conditional:
else if((k->httpcode >= 300 && k->httpcode < 400) &&
checkprefix("Location:", k->p) &&
!data->req.location) {
/* this is the URL that the server advises us to use instead */
char *location = Curl_copy_header_value(k->p);
if(!location)
return CURLE_OUT_OF_MEMORY;
if(!*location)
/* ignore empty data */
free(location);
else {
data->req.location = location;
if(data->set.http_follow_location) {
DEBUGASSERT(!data->req.newurl);
data->req.newurl = strdup(data->req.location); /* clone */
if(!data->req.newurl)
return CURLE_OUT_OF_MEMORY;
/* some cases of POST and PUT etc needs to rewind the data
stream at this point */
result = http_perhapsrewind(conn);
if(result)
return result;
}
}
}
The location is followed with a status code between 300 and 399
I understand this doesn't answer your question in the pure sense of it -- my solution isn't even in PHP, but here's a way (using jq) to follow a 201 with cURL:
shrinkImageFunction() {
echo
echo Compressing $1 . . .
curl --user api: https://tinypng.com/developers \
--data-binary #$1 \
-o .$1-json \
-s \
https://api.tinify.com/shrink
cat .$1-json | jq -r '.output.url' > .$1-url
url=$(cat .$1-url)
if [ -z "$url" ]; then
echo Something went wrong
return
fi
echo Downloading $url . . .
echo
curl -o compressed-$1 $url
echo
rm .$1-json
rm .$1-url
}
# alias for using tinypng
alias shrinkImage=shrinkImageFunction
I threw this together today while doing similar research and figured I'd share having come across this question while looking for options.
It could easily be tweaked to parse the http 201 header response using sed or similar instead of jq where I'm parsing the http body json response provided by tinypng (used merely for reference -- I'm not affiliated with them).
I have a PHP script that connects to an URL through cURL and then does something, depending on the returned HTTP status code:
$ch = curl_init();
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_URL => $url,
CURLOPT_USERAGENT => "What?!?"
);
curl_setopt_array($ch, $options);
$out = curl_exec($ch);
$code = curl_getinfo($ch)["http_code"];
curl_close($ch);
if ($code == "200") {
echo "200";
} else {
echo "not 200";
}
Some webservers are slow to reply, and although the page is loaded in my browser after a few seconds my script, when it tries to connect to that server, tells me that it did not receive a positive ("200") reply. So, apparently, the connection initiated by cURL timed out.
But why? I don't set a timeout in my script, and according to other answers on this site the default timeout for cURL is definitely longer than the three or four seconds it takes for the page to load in my browser.
So why does the connecion time out, and how can I get it to last longer, if, apparently, it is already set to infinite?
Notes:
The same URL doesn't always time out. So sometimes cURL can connect.
It is not one specific URL that sometimes times out, but different URLs at different times.
I'm on a shared server, so I don't have root access to any files.
I tried to look at curl_getinfo($ch) and curl_error($ch) – as per #drew010's suggestion in the comments – but both were empty whenever the problem happened.
The whole script runs for a little more than one minute. In this time it connects to 300+ URLs successfully. Even when one of the URLs fails, the other connections are successfully made. So the script does not time out.
cURL does not time out either, because when I try to connect to an URL with a script sleeping for 59 seconds, cURL successfully connects. So apparently the slowness of the failing URL is not a problem in itself for cURL.
Update
Following #Karlos' suggestion in his answer, I used:
CURLOPT_VERBOSE => 1,
CURLOPT_STDERR => $curl_log
(using code from this answer) and found the following in $curl_log when an URL failed (URL and IP changed):
* About to connect() to www.somesite.com port 80 (#0)
* Trying 104.16.37.249... * connected
* Connected to www.somesite.com (104.16.37.249) port 80 (#0)
GET /wp_german/?feed=rss2 HTTP/1.1
User-Agent: myURL
Host: www.somesite.com
Accept: */*
* Recv failure: Connection reset by peer
* Closing connection #0
So, I have found the why – thank you #Karlos! – and apparently #Axalix was right and it is a network problem. I'll now follow suggestions given on this site for that kind of failure. Thanks to everyone for their help!
My experience working with curl showed me that sometimes when using the option:
CURLOPT_RETURNTRANSFER => true
the server might not give a successful reply or, at least, a successful reply within the timeframe that curl has to receive the response and cache it, so the results are returned by the curl into the variable you assign. In your code:
$out = curl_exec($ch);
In this stackoverflow question CURLOPT_RETURNTRANSFER set to true doesnt work on hosting server, you can see that that the option CURLOPT_RETURNTRANSFER is directly affected by the requested host web server implementation.
As you are using explicitly the response body, and your code relies on the response headers, a good way to solve this might be to:
CURLOPT_RETURNTRANSFER => false
and execute the curl code to work on the response headers.
Once you have the header with the code you are interested, you could run a php script that echoes the curl response and parse it by yourself:
<?php
$url=isset($_GET['url']) ? $_GET['url'] : 'http://www.example.com';
$ch= curl_init();
$options = array(
CURLOPT_RETURNTRANSFER => false,
CURLOPT_URL => $url,
CURLOPT_USERAGENT => "myURL"
);
curl_setopt_array($ch, $options);
curl_exec($ch);
curl_close($ch);
?>
In any case the reply to your question why your request does not get an error, I guess that the use of the option CURLOPT_NOSIGNAL and the different timeout options explained in the set_opt php manual might get you closer to it.
In order to dig further, the option CURLOPT_VERBOSE might help you to have extra information about the request behavior through the STDERR.
The reason may be your hosting provider is imposing some limits on outgoing connections.
Here is what can be done to secure your script:
Create a queue in DB with all the URLs that need to be fetched.
Run cron every minute or 5 minutes, take a few URLs from DB - mark them as in progress.
Try to fetch those URLs. Mark every fetched URL as success in DB.
Increment failure count for unsuccessful ones.
Continue going through queue until its empty.
If you implement such a solution you will be able to process every single URL under any unfavourable conditions.
I am trying to monitor URL's which are in my database. I an using CURL to determine if the url is still alive or not. I gave a simple condition in if clause as if($httpcode>=200 && $httpcode<300)
{
return 1;
} but when i pass http://apps.facebook.com/chkouabeaddeb to CURL it returns 0. but if I put the same URL in browser it redirects to the application. what can i do to make CURL send me correct response ?
curl -I http://apps.facebook.com/chkouabeaddeb
HTTP/1.1 302 Found
Location: /chkouabeaddeb/
302 (a redirect) is greater than 300, so your code and Facebook are both working exactly as programmed.
You're probably getting a 300-class redirection status from curl, which you're assuming means "dead". It doesn't, it means you need to handle it as a redirection.
Note: solution at the end
If I attempt to do a HTTP POST of over 1024 characters, it fails. Why? Here is a minimal example:
recipient.php:
<?php
if (strlen(file_get_contents('php://input')) > 1000
|| strlen($HTTP_RAW_POST_DATA) > 1000) {
echo "This was a triumph.";
}
?>
sender.php:
<?php
function try_to_post($char_count) {
$url = 'http://gpx3quaa.joyent.us/test/recipient.php';
$post_data = str_repeat('x', $char_count);
$c = curl_init();
curl_setopt_array($c,
array( CURLOPT_URL => $url,
CURLOPT_HEADER => false,
CURLOPT_CONNECTTIMEOUT => 999,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => $post_data
)
);
$result = curl_exec($c);
echo "{$result}\n";
curl_close($c);
}
for ($i=1020;$i<1030;$i++) {
echo "Trying {$i} - ";
try_to_post($i);
}
?>
output:
Trying 1020 - This was a triumph.
Trying 1021 - This was a triumph.
Trying 1022 - This was a triumph.
Trying 1023 - This was a triumph.
Trying 1024 - This was a triumph.
Trying 1025 -
Trying 1026 -
Trying 1027 -
Trying 1028 -
Trying 1029 -
configuration:
PHP Version 5.2.6
libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3 libidn/1.8
lighttpd-1.4.19
Solution
Add the following option for cURL:
curl_setopt($ch,CURLOPT_HTTPHEADER,array("Expect:"));
The reason seems to be that any POST over 1024 character causes the "Expect: 100-continue" HTTP header to be sent, and Lighttpd 1.4.* does not support it. I found a ticket for it: http://redmine.lighttpd.net/issues/show/1017
They say it works in 1.5.
You can convince PHP's curl backend to stop doing the 100-continue-thing by setting an explicit request header:
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:'));
This way you can post a request however long you would ever want and curl will not do the dual phase post.
I've blogged about this nearly two years ago.
First thoughts...
The manual page for curl_setopt says of CURLOPT_POSTFIELDS
"The full data to post in a HTTP "POST"
operation. To post a file, prepend a
filename with # and use the full path.
This can either be passed as a
urlencoded string like
'para1=val1¶2=val2&...' or as an
array with the field name as key and
field data as value."
Could it be that your value is being treated as if it were urlencoded, and thus looks like a big long name with no value. Something somewhere is deciding to truncate that name.
Maybe you could alter it to something like
$post_data = "data=".str_repeat('x', $char_count);
Turns out this was too easy, and the problem was a little deeper. So, how to debug?
Find out exactly what CURL sends to the server
Another debugging tactic might be to formulate a curl command line which achieves the same thing, and have it output the HTTP request details as it makes them.
Testing the server by hand
You can eliminate the server from the equation by perform a request by hand, e.g. telnetting to port 80 on your server and sending it a request >1024 chars
POST /test/recipient.php HTTP/1.0
Host: gpx3quaa.joyent.us
Content-Length:1028
xxxxx(I put 1028 chars here, no point copying them all here!)
I got this response
HTTP/1.0 200 OK
Connection: close
Content-type: text/html; charset=UTF-8
Content-Length: 19
Date: Tue, 20 Jan 2009 21:35:16 GMT
Server: lighttpd/1.4.19
This was a triumph.Connection closed by foreign host.
So at least you now know it's all on the client side, possible some CURL option or configuration setting somewhere :(
Final Answer!
The problem intrigued me so I dug deeper
If you use CURLOPT_VERBOSE=>true, you'll see that CURL sends an extra header on the bigger posts:Expect: 100-Continue. Your lighttpd server doesn't like this, it would seem.
You can stop CURL from doing this by forcing it to use HTTP/1.0 with CURLOPT_HTTP_VERSION=>CURL_HTTP_VERSION_1_0 in your curl_setopt options array.
I had a similar problem with a IIS server, using SSL v3.
I kept getting the following cURL error when CURLOPT_POSTFIELDS was longer than 1024 :
52 - SSL read: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number, errno 0
Adding CURLOPT_HTTPHEADER : "Expect:" solved the problem for me.
Thank you so much for this thread!
Check if you have Suhosin patch enabled. By default, it cuts off POST data after certain number of indexes. You can bypass it in Suhosin config tho.