I am working on downloading ZIP from URL, and I have a problem with this. First step of my algorithm is to check what is the Content-Type and Content-Length of given url:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.dropbox.com/s/0hvgw7nvbdnh13d/ColaClassic.zip");
curl_setopt($ch, CURLOPT_HEADER, 1); //I
curl_setopt($ch, CURLOPT_NOBODY, 1); //without body
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); //L
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_exec($ch);
$content_type = curl_getinfo($ch, CURLINFO_CONTENT_TYPE);
However, value of variable $content-type is text/html; charset=utf-8
Then I checked Content-Type from command line like this:
curl -IL https://www.dropbox.com/s/0hvgw7nvbdnh13d/ColaClassic.zip
and I got correct result (application/zip).
So, what is the difference between these two codes, and how do I get correct Content-Type in my php script?
Edit:
curl_setopt($ch, CURLOPT_URL, 'https://www.dropbox.com/s/0hvgw7nvbdnh13d/ColaClassic.zip');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'HEAD');
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_STDERR, $verbose);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
Verbose output from php curl:
* Hostname was found in DNS cache
* Hostname in DNS cache was stale, zapped
* Trying 162.125.69.1...
* Connected to www.dropbox.com (162.125.69.1) port 443 (#14)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
* subject: businessCategory=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=Delaware; serialNumber=4348296; C=US; ST=California; L=San Francisco; O=Dropbox, Inc; CN=www.dropbox.com
* start date: 2017-11-14 00:00:00 GMT
* expire date: 2020-02-11 12:00:00 GMT
* subjectAltName: www.dropbox.com matched
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
* SSL certificate verify ok.
> HEAD /s/0hvgw7nvbdnh13d/ColaClassic.zip HTTP/1.1
Host: www.dropbox.com
Accept: */*
Verbose output from cmdline curl:
* Trying 162.125.69.1...
* TCP_NODELAY set
* Connected to www.dropbox.com (162.125.69.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: businessCategory=Private Organization; jurisdictionCountryName=US; jurisdictionStateOrProvinceName=Delaware; serialNumber=4348296; C=US; ST=California; L=San Francisco; O=Dropbox, Inc; CN=www.dropbox.com
* start date: Nov 14 00:00:00 2017 GMT
* expire date: Feb 11 12:00:00 2020 GMT
* subjectAltName: host "www.dropbox.com" matched cert's "www.dropbox.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd8c4007a00)
> HEAD /s/0hvgw7nvbdnh13d/ColaClassic.zip HTTP/2
> Host: www.dropbox.com
> User-Agent: curl/7.54.0
> Accept: */*
Seems dropbox is issuing a different response code depending on user agent — or rather lack thereof. Your command line operation sends something like curl/7.47.0 (or your version) while the php script sends an empty user agent. Adding the user agent to your php request will get dropbox to respond appropriately with a HTTP/1.1 301 Moved Permanently response and then your script will follow the location on as expected:
$ch = curl_init();
// emulates user agent from command line.
$user_agent = 'curl/' . curl_version()['version'];
curl_setopt($ch, CURLOPT_URL, "https://www.dropbox.com/s/0hvgw7nvbdnh13d/ColaClassic.zip");
curl_setopt($ch, CURLOPT_HEADER, 1); //I
curl_setopt($ch, CURLOPT_NOBODY, 1); //without body
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); //L
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_exec($ch);
$content_type = curl_getinfo($ch, CURLINFO_CONTENT_TYPE);
echo $content_type;
UPDATE: Oddly, I just tried a few other things, like emulating various browser useragent strings and it seems dropbox only seems to issue a redirect when presented with the curl/X.X.X useragent. ¯\_(ツ)_/¯
Related
I tried replicating a working curl command from CLI to a corresponding command via PHP and expected the same result. Instead I got a 400 bad request when doing the request via PHP.
I'm making a request for a Oath2 token via an API that requires you to use a certificate on all requests. The request works when I perform it with curl via CLI but when I try via PHP I get a 400 bad request response from the API.
This request with curl via CLI works.
curl --request POST \
--url https://sysorgoauth2.test.XXX.TLD/oauth2/v1/sysorg/token \
--cert /app/keys/eid.crt.pem \
--key /app/keys/eid.key.pem \
--header "Content-Type: application/x-www-form-urlencoded" \
--verbose \
-d "grant_type=client_credentials&client_id=<ID>&client_secret=<SECRET>&scope=<SCOPE>"
This request with curl via PHP does not work.
$headers = [
'User-Agent: curl/7.74.0 via PHP',
'Content-Type: application/x-www-form-urlencoded',
'Accept: */*',
];
$url = 'https://sysorgoauth2.test.XXX.TLD/oauth2/v1/sysorg/token';
$data = [
'grant_type' => 'client_credentials',
'client_id' => '<ID>',
'client_secret' => '<SECRET>',
'scope' => '<SCOPE>',
];
$body = http_build_query($data);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_SSLCERT, '/app/keys/eid.crt.pem');
curl_setopt($ch, CURLOPT_SSLKEY, '/app/keys/eid.key.pem');
curl_setopt($ch, CURLOPT_POSTFIELDS, $body);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
// For debug.
curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
$stream = fopen('/tmp/curl_debug.txt', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $stream);
$response = curl_exec($ch);
curl_close($ch);
This is the output from curl via PHP:
* Trying (IP redacted):443...
* Connected to sysoauth2.test.XXX.TLD (IP redacted) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=X; ST=X; L=X; O=X; CN=sysoauth2.test.XXX.TLD
* start date: May 20 00:00:00 2022 GMT
* expire date: Jun 20 23:59:59 2023 GMT
* subjectAltName: host "sysoauth2.test.XXX.TLD" matched cert's "sysoauth2.test.XXX.TLD"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
> POST /oauth2/v1/sys/token HTTP/1.1
Host: sysoauth2.test.XXX.TLD
User-Agent: curl/7.74.0
Content-Type: application/x-www-form-urlencoded
Accept: */*
Content-Length: 187
* upload completely sent off: 187 out of 187 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Length: 172
< Content-Type: application/json; charset=UTF-8
< Cache-Control: no-store
< Pragma: no-cache
< Connection: Close
<
* Closing connection 0
This is the output from curl via CLI:
* Trying (IP redacted):443...
* Connected to sysorgoauth2.test.XXX.TLD (IP redacted) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=X; ST=X; L=X; O=X; CN=sysorgoauth2.test.XXX.TLD
* start date: May 20 00:00:00 2022 GMT
* expire date: Jun 20 23:59:59 2023 GMT
* subjectAltName: host "sysorgoauth2.test.XXX.TLD" matched cert's "sysorgoauth2.test.XXX.TLD"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
> POST /oauth2/v1/sysorg/token HTTP/1.1
> Host: sysorgoauth2.test.XXX.TLD
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 187
>
* upload completely sent off: 187 out of 187 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 161
< Content-Type: application/json; charset=UTF-8
< Cache-Control: no-store
< Pragma: no-cache
< Connection: Close
<
{
"access_token":"<ACCESS_TOKEN>",
"expires_in":3600,
"token_type":"Bearer",
"scope":"<SCOPE>"
}
* Closing connection 0
The part that differs between the two are:
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
I have tried all kinds of curl options in PHP to get this working. Perhaps CURLOPT_SSLCERT and CURLOPT_SSLKEY does not correspond to --cert and --key?
Any help would be much appreciated.
UPDATE 1
I captured the requests for both CLI and PHP with Wireshark and there is a difference in the end of the sequence, TCP Retransmission. Not sure what that means. Will investigate.
Wireshark CLI capture
Wireshark PHP capture
Wireshark PHP capture 2
UPDATE 2
Running the PHP code outside of my Docker environment does not produce the error. So this is somehow related to my dev environment.
I apologize if the title is inappropriate, but I kind of could not think of a better definition for it.
I am going nuts over this problem. I have been working on collecting feeds and data via cURL for the past 5+ years and have never encountered this kind of situation. I have a large json to collect over the GET method from a remote server via HTTPS from address that looks something like this
https://private.example.com/thisDotNetEndPoint?token=bla-bla-trutj&someParam=1
someParam is changeable, and for some values with lower amount of data everything works fine, almost identical speeds to browser, but in several cases cURL always goes to tiomeout set, while in browser and from console everything works fine
PHP
My cURL is as follows:
$ch = curl_init();
$url = 'https://private.example.com/thisDotNetEndPoint?token=bla-bla-trutj&someParam=1';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
// I've added this user agent as it is the same as the one Chrome uses
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36');
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
// I have tried removing the SSL part below, but no difference
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "HIGH:!SSLv3s");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // tried this with true, but no difference
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT, 1200); // what ever the timeout I set the cURL always goes to timeout
curl_setopt($ch, CURLOPT_VERBOSE, true);
$response = curl_exec($ch);
if (curl_errno($ch)) {
print("cURL error: " . curl_error($ch));
print_r(curl_getinfo($ch));
} else {
print_r(json_decode($response));
}
curl_close($ch);
This is the verbose output:
* Hostname was NOT found in DNS cache
* Trying 12.34.567.89...
* Connected to private.example.com (12.34.567.89) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSL connection using ECDHE-RSA-AES256-SHA384
* Server certificate:
* subject: OU=Domain Control Validated; CN=*.example.com
* start date: 2016-03-03 09:41:38 GMT
* expire date: 2018-03-04 09:52:18 GMT
* subjectAltName: private.example.com matched
* issuer: C=US; ST=Arizona; L=Scottsdale; O=Starfield Technologies, Inc.; OU=http://certs.starfieldtech.com/repository/; CN=Starfield Secure Certificate Authority - G2
* SSL certificate verify ok.
> GET /thisDotNetEndPoint?token=bla-bla-trutj&someParam=1 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
Host: private.example.com
Accept: */* */
* Operation timed out after 1200001 milliseconds with 0 bytes received
* Closing connection 0
It always goes to timout whatever the timout I set, tried even setting it to 2 hours.
I've even tried adding these but no difference:
curl_setopt($ch, CURLOPT_NOSIGNAL, 1);
curl_setopt($ch, CURLOPT_LOW_SPEED_LIMIT, 1);
curl_setopt($ch, CURLOPT_LOW_SPEED_TIME, 1200);
Browser
When I enter the same url in browser the response comes back in 6-9 minutes
cURL from console
I have used the simplest command and it works in same time as browser:
$ curl -X GET -v 'https://private.example.com/thisDotNetEndPoint?token=bla-bla-trutj&someParam=1'
Verbose output:
* Hostname was NOT found in DNS cache
* Trying 12.34.567.89...
* Connected to private.example.com (12.34.567.89) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES256-SHA384
* Server certificate:
* subject: OU=Domain Control Validated; CN=*.example.com
* start date: 2016-03-03 09:41:38 GMT
* expire date: 2018-03-04 09:52:18 GMT
* subjectAltName: private.example.com matched
* issuer: C=US; ST=Arizona; L=Scottsdale; O=Starfield Technologies, Inc.; OU=http://certs.starfieldtech.com/repository/; CN=Starfield Secure Certificate Authority - G2
* SSL certificate verify ok.
> GET /thisDotNetEndPoint?token=bla-bla-trutj&someParam=1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: private.example.com
> Accept: */* */
>
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: application/json; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-StackifyID: V1|b8b10c35-2649-4f67-ba6a-b5ad15ef553b|C56050|CD18|
< Set-Cookie: .ASPXANONYMOUS=looI88UVBp6Cg5tLkzVejO4CNRilhyKjMY4hFqhuO48vdVT19U8h5oisC9khFv1rOmH6Ii_lEec-9XhipEvh1UkewhufqfmlTGFsyQCaML06NVa-5-Vr_OikZb07R6pdHCeRtn9liBVJfamJmXiElA2; expires=Thu, 02-Feb-2017 20:54:18 GMT; path=/; HttpOnly
< X-AspNetMvc-Version: 5.2
< Rx-CID: ae9907d6fc394b24b6599e74ab5a668f
< Rx_RequestId: f3fff82c4de04bba90b2bbc5704ac787
< X-Powered-By: ASP.NET
< Strict-Transport-Security: max-age=31536000
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: rx-cid
< Date: Fri, 25 Nov 2016 10:25:00 GMT
< Content-Length: 2231472
<
[and the response is printed here]
Any ideas?
Thanks in advance.
Did you notice the difference between the console and your php verbose output? The useragent is missing in your php code. curl commandline by default adds this useragent, whereas the php-curl doesn't.
User-Agent: curl/7.35.0
Use the option CURLOPT_USERAGENT.
curl_setopt($ch, CURLOPT_USERAGENT, "Opera 11.0");
I read some SO questions and answers but I can't solve my problem.
I've a cURL request:
function dmd_check_key($arg){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://***.de/dmd-pages-pro/dmd_key_generator.php?key='.$arg.'&website='.$_SERVER['HTTP_HOST'],
CURLOPT_USERAGENT => 'Codular Sample cURL Request',
CURLOPT_VERBOSE => 1
));
curl_setopt($curl, CURLOPT_STDERR, fopen("curl_debug.txt", "w+"));
$resp = curl_exec($curl);
curl_close($curl);
if(!get_option('dmd-pages-key-status')){add_option('dmd-pages-key-status', $resp);}else{update_option('dmd-pages-key-status', $resp);}
if(!get_option('dmd-pages-key')){add_option('dmd-pages-key', $arg);}else{update_option('dmd-pages-key', $arg);}
return $resp;
}
I got this errormessage:
Hostname was NOT found in DNS cache
Trying 00.13.133.000...
Connected to dimadirekt.de (00.13.133.000) port 443 (#0)
successfully set certificate verify locations:
CAfile: none CApath: /etc/ssl/certs
SSL certificate problem: unable to get local issuer certificate
Closing connection 0
Can someone explain me the errormessage and how I can resolve it?
I think the problem is the certificate. But on which server? I have two.
One server with the cURL file (nonssl) and one server which handles the data (ssl).
I try this settings:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
But it is still not working.
EDIT:
With the settings line above:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
The errormessage is gone.
Now the problem is that I don't get any response.
Now I get this response:
Hostname was NOT found in DNS cache
Trying 00.13.133.212...
Connected to ***.de (00.13.000.212) port 443 (#0)
successfully set certificate verify locations:
CAfile: none CApath: /etc/ssl/certs
SSL connection using ECDHE-RSA-AES128-GCM-SHA256
Server certificate:
subject: OU=Domain Control Validated; CN=www.***.de
start date: 2016-03-03 00:00:00 GMT
expire date: 2019-06-01 23:59:59 GMT
subjectAltName: ***.de matched
issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Domain Validation Secure Server CA
SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
GET /dmd-pages-pro/dmd_key_generator.php?key=27dda19e85378bb8df73fa3f2806a30c&website=ak.dimadirekt.com
HTTP/1.1 User-Agent: Codular Sample cURL Request Host: ***.de
Accept: /
< HTTP/1.1 200 OK < Date: Tue, 31 May 2016 11:07:49 GMT
* Server Apache is not blacklisted < Server: Apache < Vary: Accept-Encoding < Transfer-Encoding: chunked < Content-Type: text/html
<
* Connection #0 to host dimadirekt.de left intact
I am running PHP 5.6 on a Windows Server 2008R2 server. I am trying to get TLS v1.2 communications working between this server and another internal server (running Windows Server 2003x64). We use internal Windows AD Root and Issuing Cert Authorities so our certificate chain is Root CA - Issuing CA - Server Cert. When I try and connect, I get the following error:
* Trying 192.168.1.10...
* Connected to targetserver.example.com (192.168.1.10) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: e:\php\cacert.pem
CApath: none
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
I have converted both the root CA cert and the issuing CA cert to PEM format (using openssl) and added them to the end of the cacert.pem file.
<?php
function nxs_cURLTest($url, $msg, $testText){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_CAINFO, "e:\\php\\cacert.pem");
$verbose = fopen('php://temp', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);
$response = curl_exec($ch);
$errmsg = curl_error($ch);
$cInfo = curl_getinfo($ch);
curl_close($ch);
echo "<br />Testing ... ".$url." - ".$cInfo['url']."<br />";
if (stripos($response, $testText)!==false)
echo "....<b style='color:green;'>".$msg." - OK</b><br />";
else
{
echo "....<b style='color:red;'>".$msg." - Problem</b><br /><pre>";
print_r($errmsg);
print_r($cInfo);
print_r(strlen($response) . " bytes received.");
print_r(htmlentities($response));
rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "<br />Verbose output:</br />";
echo "<pre>", htmlspecialchars($verboseLog), "</pre>";
echo "</pre>There is a problem with cURL. You need to contact your server admin or hosting provider.<br />";
}
}
nxs_cURLTest("https://targetserver.example.com/curl/", "HTTPS to TargetServer", "Document contents");
?>
If I try to just use cURL from the command line, then I get a different error (EDIT: forced tls 1.0 as Win Serv 2003 doesn't support anything higher):
E:\openssl>curl -Iv --tlsv1.0 "https://targetserver.example.com"
* Rebuilt URL to: https://targetserver.example.com/
* Trying 192.168.1.10...
* Connected to targetserver.example.com (192.168.1.10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: E:\openssl\curl-ca-bundle.crt
CApath: none
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
* TLSv1.0 (OUT), TLS alert, Client hello (1):
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
(note: ca-bundle.crt is a renamed copy of cacert.pem from above)
If I connect to a different server that runs Server 2008R2 but that also has its certificate issued from our internal CA I get the following:
E:\openssl>curl -Iv --tlsv1.0 "https://int-winsrv2008.example.com"
* Rebuilt URL to: https://int-winsrv2008.example.com/
* Trying 192.168.1.6...
* Connected to int-winsrv2008.example.com (192.168.1.6) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: E:\openssl\curl-ca-bundle.crt
CApath: none
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (IN), TLS handshake, Server key exchange (12):
* TLSv1.0 (IN), TLS handshake, Server finished (14):
* TLSv1.0 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.0 (OUT), TLS change cipher, Client hello (1):
* TLSv1.0 (OUT), TLS handshake, Finished (20):
* TLSv1.0 (IN), TLS change cipher, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / ECDHE-RSA-AES256-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=KY; ST=Grand Cayman; L=George Town; O=Port Authority of the Cayman Islands; OU=IT Department; CN=int-winsrv2008.example.com
* start date: Aug 10 15:26:14 2015 GMT
* expire date: Aug 9 15:26:14 2017 GMT
* common name: int-winsrv2008.example.com (matched)
* issuer: DC=com; DC=example; CN=example-Issuing-CA
* SSL certificate verify ok.
> HEAD / HTTP/1.1
> Host: int-winsrv2008.example.com
> User-Agent: curl/7.47.1
> Accept: */*
>
* TLSv1.0 (IN), TLS handshake, Hello request (0):
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (IN), TLS handshake, Server key exchange (12):
* TLSv1.0 (IN), TLS handshake, Request CERT (13):
* TLSv1.0 (IN), TLS handshake, Server finished (14):
* TLSv1.0 (OUT), TLS handshake, Certificate (11):
* TLSv1.0 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.0 (OUT), TLS change cipher, Client hello (1):
* TLSv1.0 (OUT), TLS handshake, Finished (20):
* TLSv1.0 (IN), TLS change cipher, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Finished (20):
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 689
Content-Length: 689
< Content-Type: text/html
Content-Type: text/html
< Last-Modified: Thu, 18 Jul 2013 15:46:23 GMT
Last-Modified: Thu, 18 Jul 2013 15:46:23 GMT
< Accept-Ranges: bytes
Accept-Ranges: bytes
< ETag: "f12a46f4cd83ce1:0"
ETag: "f12a46f4cd83ce1:0"
< Server: Microsoft-IIS/7.5
Server: Microsoft-IIS/7.5
< X-Powered-By: ASP.NET
X-Powered-By: ASP.NET
< Date: Tue, 09 Feb 2016 15:18:51 GMT
Date: Tue, 09 Feb 2016 15:18:51 GMT
<
* Connection #0 to host int-winsrv2008.example.com left intact
So it seems it's something to do with Win Server 2003. What am I missing?
For completeness sake, I thought I would answer this with what I found out.
There was a few different errors occurring here. First, I was trying to use TLS1.2 which is not supported by Windows Server 2003. Second, I am using host headers which are not evaluated at the point of the certificate handshake (unless you use SNI which is only supported by IIS 8/Win Serv 2012) so I was not getting our internal certificate but our default wildcard cert issued by Comodo. Finally, there seems to be an issue with the Comodo certificate chain on our server (seems to be missing one of the root or intermediate certs).
Currently, I am trying to connect PHP website with MS Dynamics 2013. I found the following example (https://github.com/rocketeer007/php-dynamics-crm-2011) to connect to but it is not working.
I provided the configuration but is getting following error i.e.
string(576) "http://www.w3.org/2005/08/addressing/soap/faults:Sendera:InvalidSecurity
An error occurred when verifying security for the message."
I couldn't find more information about the soap response.
Here is the more information that I am sending in my request i.e.
**NOTE: For example purposes below I have changed the URLs**
//Parameters values used in the below CURL
$soapUrl = https://example.crmserver.com/adfs/services/trust/13/usernamemixed
$headers = 'POST /adfs/services/trust/13/usernamemixed HTTP/1.1
Host: adfs.crmserver.com
Connection: Keep-Alive
Content-type: application/soap+xml; charset=UTF-8
Content-length: 1481'
$content = '<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<s:Header>
<a:Action s:mustUnderstand="1">http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue</a:Action>
<a:ReplyTo>
<a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
</a:ReplyTo>
<a:To s:mustUnderstand="1">https://adfs.crmserver.co.nz/adfs/services/trust/13/usernamemixed</a:To>
<o:Security xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" s:mustUnderstand="1">
<u:Timestamp u:Id="_0">
<u:Created>2015-07-16T04:09:52.00Z</u:Created>
<u:Expires>2015-07-16T04:10:52.00Z</u:Expires>
</u:Timestamp>
<o:UsernameToken u:Id="user">
<o:Username>desktop\abc_service.mrl</o:Username>
<o:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">password</o:Password>
</o:UsernameToken>
</o:Security>
</s:Header>
<s:Body>
<trust:RequestSecurityToken xmlns:trust="http://docs.oasis-open.org/ws-sx/ws-trust/200512">
<wsp:AppliesTo xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy">
<a:EndpointReference>
<a:Address>https://dev2013.crm.crmserver.co.nz/XRMServices/2011/Discovery.svc</a:Address>
</a:EndpointReference>
</wsp:AppliesTo>
<trust:RequestType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</trust:RequestType>
</trust:RequestSecurityToken>
</s:Body>
// Request
$cURLHandle = curl_init();
curl_setopt($cURLHandle, CURLOPT_URL, $soapUrl);
curl_setopt($cURLHandle, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($cURLHandle, CURLOPT_TIMEOUT, self::$connectorTimeout);
curl_setopt($cURLHandle, CURLOPT_SSL_VERIFYPEER, TRUE);
curl_setopt($cURLHandle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_NONE);
curl_setopt($cURLHandle, CURLOPT_HTTPHEADER, $headers);
curl_setopt($cURLHandle, CURLOPT_POST, 1);
curl_setopt($cURLHandle, CURLOPT_POSTFIELDS, $content);
curl_setopt($cURLHandle, CURLOPT_HEADER, false);
curl_setopt($cURLHandle, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
$responseXML = curl_exec($cURLHandle);
if(curl_exec($cURLHandle) === false)
{
echo 'Curl error: ' . curl_error($ch);
}
else
{
echo "no error <br />";
var_dump($responseXML);
}
I tried to see the error but get following i.e.
no error
I am stuck and researched over the internet and couldn't find anything on it therefore, is posting over here...
More Information:
When I tried this from command line i.e.
curl -H "Authorization: Bearer" https://dev2013.crm.example.com/XRMServices/2011/Discovery.svc -v
Then, I got the following output i.e.
About to connect() to dev2013.crm.appserv.co.nz port 443 (#0)
* Trying x.x.x.x... connected
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES128-SHA
* Server certificate:
* subject: C=AU; ST=Sydney; L=Sydney; O=Example Ltd; CN=*.crm.example.com
* start date: 2015-06-24 00:00:00 GMT
* expire date: 2017-08-22 23:59:59 GMT
* subjectAltName: dev2013.crm.example.com matched
* issuer: C=US; O=thawte, Inc.; CN=thawte SSL CA - G2
* SSL certificate verify ok.
> GET /XRMServices/2011/Discovery.svc HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: dev2013.crm.example.com
> Accept: */*
> Authorization: Bearer
>
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Set-Cookie: ISAWPLB{11FDD9E3-24F0-455D-8590-436AF3F4D26D}={B0F3D377-6B32-45C6-A517-DA83209E8EA4}; HttpOnly; Path=/
< Content-Length: 3127
< Date: Thu, 16 Jul 2015 03:26:39 GMT
< Content-Type: text/html; charset=UTF-8
< Server: Microsoft-IIS/8.5
< Cache-Control: private
< X-AspNet-Version: 4.0.30319
< REQ_ID: 5b87d8d2-8b5e-4979-b090-7df14a3a1603
< Set-Cookie: ReqClientId=751fb9c2-ac10-478a-b933-2117420c660c; expires=Thu, 16-Jul-2065 03:26:39 GMT; path=/; secure; HttpOnly
< X-Powered-By: ASP.NET
It means that there is no issue with connecting to CRM using the above discovery server.
Looking for instant help on the error and the best recommendation to get authentication from the CRM using PHP SOAP webservices.
Thanks in advance.
Cheers,
Jason Lattimer has an updated example using IFD.
http://jlattimer.blogspot.com.au/2015/02/soap-only-authentication-using-php.html
At a glance there seems to be a few discrepancies.
Firstly you don't have a message ID (Should be just under the action line)
<a:MessageID>urn:uuid:{A RANDOM GUID GOES HERE}</a:MessageID>
Secondly you have the following line
<o:UsernameToken u:Id="user">
Which I walso believe should be a GUID
<o:UsernameToken u:Id="{GUID GOES HERE}">
Finally you look like you're pointing to the Discovery Service where I think you should be pointing to the organisation service.
Organization.svc
I haven't done any PHP-IFD stuff myself so I'd suggest going to Jason's blog and comparing it with yours.
Campey, thanks for your time. Both DEV and LIVE Servers are in the same timezone the only difference that I have found the DEV server timezone is 10 minutes behind from the LIVE server.
I corrected the time on DEV server and then the authentication problem is solved.
Thanks everyone for investing time to read the issue.