Scraping with client certificate request with php and curl - php

I'm trying to download a file that needs to be authenticated through a client digital certificate, I already have the certificate but I do not know how to configure it in curl.
$useragent = '...';
$post = array( ... );
$certPass = '123456';
$certPath = _DIR_PATH.'cert/';
$certPfx = $certPath.'certificate.pfx';
$cert = $certPath.'certificate.pem';
$url = 'https://www.url.com/path/to/access';
$ch = curl_init( $url );
$options = array(
CURLOPT_FAILONERROR => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_AUTOREFERER => true,
CURLOPT_HEADER => true,
CURLOPT_NOBODY => true,
CURLOPT_CAINFO => $cert,
CURLOPT_CAPATH => $certPath,
CURLOPT_SSH_PRIVATE_KEYFILE => $certPfx,
CURLOPT_SSLCERT => $cert,
CURLOPT_SSLCERTPASSWD => $certPass,
CURLOPT_SSL_VERIFYHOST => 2,
CURLOPT_SSL_VERIFYPEER => true,
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $post,
CURLOPT_USERAGENT => $useragent,
CURLOPT_COOKIE => 'ASP.NET_SessionId='.$cookie
);
curl_setopt_array( $ch, $options );
$resp = curl_exec($ch);
$ch_errno = curl_errno($ch);
$ch_erro = curl_error($ch);
curl_close($ch);
I am always getting the message: SSL certificate problem: unable to get local issuer certificate.
Can someone help me?

PHP cURL: Fixing the “SSL certificate problem: unable to get local issuer certificate” error.
If you are using PHP’s cURL functions to connect to a HTTPS URL, you might come across the following error:
SSL certificate problem: unable to get local issuer certificate. (cURL error code 60)
This is a common error that occurs whenever you attempt to use PHP’s cURL functions to connect to a HTTPS website. Essentially, your cURL client has not been configured to connect to SSL-enabled websites.
The Quick Fix.
CURLOPT_SSL_VERIFYHOST: This option tells cURL that it must verify the host name in the server cert.
CURLOPT_SSL_VERIFYPEER: This option tells cURL to verify the authenticity of the SSL cert on the server.
For example.
$url = 'https://google.com';
//Initiate cURL.
$ch = curl_init($url);
//Disable CURLOPT_SSL_VERIFYHOST and CURLOPT_SSL_VERIFYPEER by
//setting them to false.
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
//Execute the request.
curl_exec($ch);
//Check for errors.
if(curl_errno($ch)){
throw new Exception(curl_error($ch));
}

You need to configure your php.ini porperly with current (valid) certificates.
curl.cainfo = "/etc/php7.2/cacert.pem"
openssl.cafile = "/etc/php7.2/cacert.pem"
Look at https://curl.haxx.se/docs/caextract.html to download the current one. After that restart your webserver.
Do not something like
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
which will work, but disables any verification and security.

Related

cURL Error #:SSL_write() error: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry - PHP

When I try to send a cURL request, I get the following error:
* SSL certificate verify ok.
> * SSL_write() error: error:1409F07F:SSL routines:ssl3_write_pending:bad write retry
* Closing connection 0
this is the request I am sending:
$url = 'https://example.com/services/service-name';
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => $url,
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $postData,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_SSL_VERIFYPEER => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1_2,
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp,
));
$response = curl_exec($curl);
Sometimes it works, but other times it gives me the SSL error "Curl Error Number 55".
Curl version: 7.84.0
SSL version: OpenSSL/1.1.1p
On localhost, it works all the time, but on production server "Linux" it keeps giving me the SSL error. Any help is greatly appreciated
After trying everything I know, the problem was related to DNS using IPv6 instead of IPv4. I forced the cURL request to use IPv4 and haven't received any error since
curl_setopt($curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);

unable to use client certificate(no key found or wrong pass phrase?)

I am trying to make a SOAP call to a server using CURL as belows.
The Requirement is
We need to pass the ssl certificate and pass the Username and Password
$ssl = "ssl_file_relative_address.pem";
$pub_ssl_password = 'mynameiskhan';
//Get the data
$data = the_data_xml.xml;
//Get the WSDL Address
$wsdl = "address/to/wsdl?parameter=value";
$soapUser = "Username"; // username
$soapPassword = "password"; // password
$options = [
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_SSL_VERIFYHOST => FALSE,
CURLOPT_SSL_VERIFYPEER => FALSE,
CURLOPT_URL => $wsdl,
CURLOPT_SSLCERT => $ssl,
//CURLOPT_SSLCERTPASSWD => $pub_ssl_password,
CURLOPT_USERPWD => $soapUser.":".$soapPassword,
CURLOPT_HTTPAUTH => CURLAUTH_BASIC
];
$ch = curl_init();
curl_setopt_array($ch , $options);
$response = curl_exec($ch);
//curl_close($ch);
if (curl_errno($ch)) {
print curl_error($ch);
}
I'm getting the following Error from CURL : unable to use client certificate (no key found or wrong pass phrase?)
What is it that I'm doing Wrong...
Found the Solution. It required an intermediate CA Certificate.
The Solution is
$options = [
CURLOPT_HTTPHEADER => ['Content-type: application/json'],
CURLOPT_URL => 'https://address/to/service?param=value',
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_CAINFO => getcwd()."\cacert.pem",
URLOPT_SSLCERT => getcwd().'\cert.pem',
CURLOPT_SSLCERTPASSWD => 'ssl_password',
CURLOPT_HTTPAUTH => CURLAUTH_BASIC,
CURLOPT_USERPWD => $soapUser.":".$soapPassword,
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => $data
];
$ch = curl_init();
curl_setopt_array($ch , $options);
$response = curl_exec($ch);
if (curl_errno($ch)) {
print curl_error($ch);
}else{
print_r($response);
}
curl_close($ch);
Do not forget to mention the CURLOPT_HTTPHEADER to its content type, it is important.
Also download the intermediate certificate from https://curl.haxx.se/ca/cacert.pem. It contains all the valid CA's.
Thanks #drew010 for help.
When you specify a client authentication certificate using CURLOPT_SSLCERT, the PEM file should contain a -----BEGIN CERTIFICATE----- line followed by the certificate.
You also need to supply cURL with the corresponding private key to the certificate using CURLOPT_SSLKEY which is a file beginning with -----BEGIN PRIVATE KEY-----.
If the private key is in ssl_file_relative_address.pem, then try copying the private key to a separate file.
If the private key is encrypted, you can specify the password using CURLOPT_SSLKEYPASSWD.

SSL Certificate - 3rd party

I am working with Rightmove and their realtime data feed, we received a (self-signed?) certificate (.p12 file) that we had to insert into our browser/PC to connect to the testserver, this is all working great.
Now we are writing our own script, and connecting to the test server via CURL, however we are loading this script from our server and it does not have access to the test server (handshake failed - authentication, after research looks like it is expecting certificate which makes sense), but how do we get access? They are not of great help and I am wondering if we need to add this certificate to our domain/server as well to gain authentication?
Hope you can help!
You would use the CURLOPT_SSLCERT & CURLOPT_SSLCERTPASSWD options with your CURL command. e.g
$url = "https://www.example.com";
$cert_file = 'certificate_file.pem';
$cert_password = 'password';
$ch = curl_init();
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_USERAGENT => 'Useragent',
CURLOPT_URL => $url ,
CURLOPT_SSLCERT => $cert_file ,
CURLOPT_SSLCERTPASSWD => $cert_password ,
);
curl_setopt_array($ch , $options);
$output = curl_exec($ch);

PHP - https stream through http proxy

I'm trying to get the content of a stream over HTTPS, but I have to go over an HTTP proxy.
I'd like not to use cURL but rather use fopen with a context argument.
The thing is, I can't make it work over HTTPS (HTTP is working fine though).
This DOES NOT work :
$stream = stream_context_create(Array("http" => Array("method" => "GET",
"timeout" => 20,
"proxy" => "tcp://my-proxy:3128",
'request_fulluri' => True
)));
echo file_get_contents('https://my-stream', false, $context);
This DOES work (cURL) :
$url = 'https://my-stream';
$proxy = 'my-proxy:3128';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;
Does somebody know what is wrong with the first piece of code ? if it works with cURL there has to be a way to make it work with a context.
I tried to change the context options to a bunch of different values woth no luck.
Any help would be greatly appreciated !
Thanks.
You did not specifiy the exact error message, try adding ignore_errors => true. But if you are getting a 400 Bad Request from Apache, the problem you are probably hitting, is a Server Name Indication & host header mismatch. There is also a PHP bug related to this: https://bugs.php.net/bug.php?id=63519
Try the following fix until this bug is resolved:
$stream = stream_context_create(array(
'http' => array(
'timeout' => 20,
'proxy' => 'tcp://my-proxy:3128',
'request_fulluri' => true
),
'ssl' => array(
'SNI_enabled' => false // Disable SNI for https over http proxies
)
));
echo file_get_contents('https://my-stream', false, $context);

Php SoapClient stream_context option

I want to use a third party's web service. To use the web service I need to connect with HTTPS. My problem is that for the development process I have a test api with an invalid certificate. I would like to set SoapClient no to verify the server's certificate. Here is the way I tried:
$opts = array(
'ssl' => array(
'verify_peer' => false
),
'https' => array(
'curl_verify_ssl_peer' => false,
'curl_verify_ssl_host' => false
)
);
$streamContext = stream_context_create($opts);
$client = new SoapClient("https://urlToSoapWs",
array(
'login' => 'user',
'password' => 'pwd',
'authentication' => SOAP_AUTHENTICATION_BASIC,
'local_cert' => file_get_contents('C:/somelocation/1.pem'),
'passphrase' => 'passphrase',
'stream_context' => $streamContext
));
I also tried with CURL and worked! But I want to use SoapClient. You can find the code with CURL below:
// create a new cURL resource
$ch = curl_init("https://urlToSoapWs");
// setting the request type to POST:
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: text/xml"));
// setting the authorization method to BASIC:
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
// supplying your credentials:
curl_setopt($ch, CURLOPT_USERPWD, "user:pwd");
$body = "<SOAP-ENV:Envelope>somexmlhere</SOAP-ENV:Envelope>";
// filling the request body with your SOAP message:
curl_setopt($ch, CURLOPT_POSTFIELDS, $body);
// configuring cURL not to verify the server certificate:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSLCERT, "pathToTheCertificatePemFile");
curl_setopt($ch, CURLOPT_SSLCERTPASSWD, "pwd");
//curl_setopt($ch, CURLOPT_SSLCERTTYPE, "PEM");
curl_setopt($ch, CURLOPT_SSLKEY, "pathTotheKeyFile");
curl_setopt($ch, CURLOPT_SSLKEYPASSWD, "pwd");
// telling cURL to return the HTTP response body as operation result
// value when calling curl_exec:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// calling cURL and saving the SOAP response message in a variable which
// contains a string like "<SOAP-ENV:Envelope ...>...</SOAP-ENV:Envelope>":
$result = curl_exec($ch);
// closing cURL:
curl_close($ch);
If you have found the bug in the code that I provided using the SoapClient please post it.
Thanks.
Maybe not the invalid Certificate is a Problem, more the SSLv2/3 Handshake; can you try manually specifing a Cipher like this:
$stream_opts = array(
// 'ssl'=>array('ciphers'=>"3DES" // also working
// further ciphers on http://www.openssl.org/docs/apps/ciphers.html
'ssl'=>array('ciphers'=>"SHA1"
)
);
$myStreamContext = stream_context_create($stream_opts);
$soapOptions['stream_context'] = $stream_opts;
$soapClient = new SoapAuthClient("https://...", $soapOptions);
Good luck!
It looks like you have hit this authentication plus SSL bug in SoapClient. You can either recompile php with the patch included in that link, or wait until they integrate it in the official build.

Categories