Php Curl doesn't work on just one website - php

i've an error very very strange.
I need to access this website: http://sub.example.com/ with cURL. From localhost i've no problem as well as if i use cURL from command line or browser to access the website.
I'm using this code on my Siteground server:
<?php
$url="http://sub.example.com/";
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL,$url);
$result=curl_exec($ch);
var_dump($result);
?>
and i get $result = false.
If i use Try / Catch i get: Curl failed with error #6: Could not resolve host
I've also tried to substitute $url with another website's url and it always works (http and https, domain and subdomain, on my server and external)!
I spent already a day trying to solve this, please help me!
Thank you in advance, Luca

according to googledns, sub.example.com does not exist.
$ nslookup sub.example.com 8.8.8.8
*** google-public-dns-a.google.com can't find sub.example.com: Non-existent domain
Server: google-public-dns-a.google.com
Address: 8.8.8.8
the fact that it works for you on localhost means you have a weird DNS, or perhaps an entry for it in your hosts file , while your Siteground server does not.
anyway, Curl failed with error #6: Could not resolve host means the server's dns could not find sub.example.com, but luckily, if you know what ip you want to connect to, you can bypass the DNS entirely, by instead writing:
curl_setopt($ch, CURLOPT_URL,'http://93.184.216.34/');
curl_setopt($ch,CURLOPT_HTTPHEADER,array('Host: sub.example.com'));
(there is also a more correct-ish curlopt to tell curl which ip goes with wich domain, but the the name escapes me at the moment, and iirc, php has not added official support for it, so you'd have to use the magic value for it, rather than the defined name, but manually setting the Host: header is pretty easy anyway)

Related

Curl / php Empty response from PayPal IPN url

In the past couple of days, our PayPal IPN has stopped working and receiving a empty response from PayPal. Nothing has changed on the server or in our code.
Attempting to cUrl to the PayPal IPN url simply returns an empty response.
$url = "https://www.paypal.com/cgi-bin/webscr?cmd=_notify-validate";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
$data = curl_exec($ch);
curl_close($ch);
print $data;
The above should return "INVALID" and works as expected on my localhost and from other servers.
Changing the url to any other domain other then PayPal also works without issues.
So it seems it only received the empty response from paypal.com
I wonder if any one has run into the issue or could give pointers where to look in the hope to- resolve it?
Kind Regards
Musaffar
It looks like Paypal made some security changes at the weekend. Some oscommerce users have been getting similar symptoms since Sunday with the curl error turning out to be:
SSL certificate problem: unable to get local issuer certificate
In their cases it was due to the local certificate copy only having the Paypal certificate and not the root or intermediate certificates, with the solution to update it.
The working certificate can be found in:
OSCOM Phoenix github
I would perhaps start checking if curl gave you any errors. PHP gives you in this case curl_errno($ch) and curl_error($ch) which will return a message or respectively the error code to it.
In case there is nothing suspicious to it. I would perhaps try to validate if you're able to run the above from your CLI on the server. (requires SSH access to it)
curl -H "User-Agent: ..." "https://www.paypal.com/cgi-bin/webscr?cmd=_notify-validate"
I hope this gives you a good starting point. Please let me know if this helps or if you need more ideas.
Edit#2: After comments where added from the author:
cUrl does indeed return an error : error 35 and the error message is as follows:
error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
In this case I would guess that your system is outdated and does not support the SSL ciphers from paypal. You could try to force a specific TLS version. As of today paypal webservers support only TLS1.2 or TLS1.3. This is how you can force it:
curl_setopt($ch, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_2);
or
curl_setopt($ch, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_3);
Additionaly #Preston PHX mentioned in a comment a good refernces for this issue coming from paypal:
https://github.com/paypal/TLS-update#php
https://developer.paypal.com/docs/api/info-security-guidelines/

Browser opens a link successfully, but not curl and file_get_contents

I'm trying to use Instagram API. when I open following link in browser, it's completely fine an you can click on it and see the json response:
https://www.instagram.com/nasa/?__a=1
When I tried to open the same url via file_get_contents() I faced 403 Forbidden Error.
So I tried to use curl. here is my code :
$url = "https://www.instagram.com/nasa/?__a=1";
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL,$url);
$result=curl_exec($ch);
curl_close($ch);
var_dump($result);
The problem is $result is an empty string. When I try to get contents using file_get_contents, I face 403 Forbidden Error, and when I try to get contents using curl it return an empty string.
Can Some body help? Tnx.
Edit
I dont get 403 Forbidden in my browser because I'm logged in.
you need to enable cookie support (eg CURLOPT_COOKIEFILE) AND log in before you can access https://www.instagram.com/nasa/?__a=1 , and your curl code never attempts to log in.
here you can see how to log in to Instagram with PHP: https://stackoverflow.com/a/41684531/1067003

PHP cURL how to add the User Agent value OR overcome the Servers blocking cURL requests?

I am transferring an Object Array. I have a cURL client (submitter) on own Server and listening script on other's Server, which one is not under my control. Then i think there, they are blocking the incoming cURL requests because when i test with the normal HTML <form>, it is working. But not via cURL anyway.
So i think they have done some restriction to cURL.
Then my questions here are:
Can a Server restrict/block the cURL incoming requests?
If so, can i trick/change the HTTP Header (User Agent) in my initiating cURL script?
Or is there any other possible stories?
Thanks!
IF you are still facing the problem then do the following.
1.
$config['useragent'] = 'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0';
curl_setopt($curl, CURLOPT_USERAGENT, $config['useragent']);
curl_setopt($curl, CURLOPT_REFERER, 'https://www.domain.com/');
2.
$dir = dirname(__FILE__);
$config['cookie_file'] = $dir . '/cookies/' . md5($_SERVER['REMOTE_ADDR']) . '.txt';
curl_setopt($curl, CURLOPT_COOKIEFILE, $config['cookie_file']);
curl_setopt($curl, CURLOPT_COOKIEJAR, $config['cookie_file']);
NOTE: You need a COOKIES folder in directory.
3.
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
If doing these don't solve the problem then Give the Sample Input/Output/Error/etc.
So, that more precise solution can be provided.
$agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)';
$curl=curl_init();
curl_setopt($curl, CURLOPT_USERAGENT, $agent);
In the server side, we can block some requests by recognize the header fields(including refer, cookie, user-agent and so on) in http request, the ip address, access frequency. And in most case, requests generated by machine usually has something different than human requests,for example, no refer & cookie, or with higher access frequency, we can write some rules to deny these requests.
According to 1, you can try your best to simulate real requests by filling the header fields, using random and slower frequency, using more ip addresses. (sounds like attack)
Generally, using lower frequency and do not make heavy load for their server, follow their access rules, they will seldom block your requests.
Server cannot block only cURL requests because they are just HTTP requests. So changing User Agent of your cURL can solve your problem, as server will think you are connecting through browser presented in UA.
Example of curl GET call in php.
ftp file in a variable.
The solution was on Stackoverflow... where ?!?
not mine.
BTW, you need to be able to execute php code from within html
modify your /etc/apache2/mods-enabled' edit '#mime.conf
if you want to do so...
Go to end of file and add the following line:
"AddType application/x-httpd-php .html .htm"
BEFORE tag '< /ifModules >'
verified and tested with 'apache 2.4.23' and 'php 5.6.17-1' under 'debian'
I choose to execute php in html file because faster development.
example code begin :
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title></title>
</head>
<body>
<?php
$host = "https://tgftp.nws.noaa.gov/data/observations/metar/decoded/CYHU.TXT";
$agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $host);
curl_setopt($curl, CURLOPT_USERAGENT, $agent);
curl_setopt ($curl, CURLOPT_RETURNTRANSFER, 1) ;
curl_exec($curl);
$ftp_result=curl_exec($curl);
print_r($ftp_result);
//and the big work commencing,
//extracting text ...
$zelocation="";
$zedatetime="";
$zewinddirection="";
$zewindspeed="";
$zeskyconditions="";
$zetemp="";
$zehumidity="";
?>
</body>
</html>
I've faced the same issue when I was trying login to a website using CURL, the server was rejecting my request until I've sent the user-agent header and the cookies returned when entering the login page, however, you can use this curl library if you don't familiar with curl.
$curl = new Curl();
$curl->setHeaders('user-agent', 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0');
// Disable SSL verification
$curl->setOpt(CURLOPT_SSL_VERIFYPEER, '0');
$curl->post($url, $data);
$response = $curl->getRawResponse();

get_file_content not working for https

I am using php and google's new CSE in my website . The URL for request is like
https://www.googleapis.com/customsearch/v1?key=INSERT-YOUR-KEY&cx=017576662512468239146:omuauf_lfve&q=lectures
The connection is https and file_get_content() is getting failed. How can i get it working?
Since i need to host the site on some external web-hosting servers, I am in need of solution which don't alter php configuration file or work with default options found on most of web-hosting sites.
You can use curl:
## HTTPS url that you are targeting.
$url = "https://www.googleapis.com/customsearch/v1?key=INSERT-YOUR-KEY&cx=017576662512468239146:omuauf_lfve&q=lectures";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Opera/9.23 (Windows NT 5.1; U; en)');
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
## Below two option will enable the HTTPS option.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
$result = curl_exec($ch);
echo $result;
If you can't modify the config yourself, talk to the hosting provider. It most probably has to do with the lacking of openssl in the php core or startup.
If you are doing this for someone else and you have no control or contact with the host (the "some external host"), explain the situation and help them find a suitable host with the appropriate features enabled.

Curl request not working on live server

Can anybody tell me why this CURL code only works on my local server and not on live server?
Tried on 3 different hosting and nothing works.
Checked everything on live
1) Curl enabled
2) PHP version is OK
3) Curl executes without any error but no result
Its been 3 days and I am not able to find any solution
please help.
error_reporting(1);
set_time_limit(1500);
$fname=time().'_myfile.flv';
header('Content-type: video/x-flv');
header("Content-Type: application/octet-stream");
header("Content-Disposition: attachment; filename=\"$fname\"");
define('USERAGENT', "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.2; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)");
$url='http://v3.lscache5.c.youtube.com/videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=914010%2C907605&algorithm=throttle-factor&itag=34&ip=112.0.0.0&burst=40&sver=3&signature=D51A660BDF83B54B3584425DBE8930D5D0F805E1.B3FB21D0CAF625D36A17B558A0A653F20788B49F&expire=1313503200&key=yt1&ipbits=8&factor=1.25&id=1cacd26a9913e4ec';
$ch = curl_init() or die("Error");
curl_setopt($ch, CURLOPT_USERAGENT, USERAGENT);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
if(curl_exec($ch) === FALSE)
{
die("Curl failed: " . curl_error($ch)); // Never goes here
}
curl_close($ch);
?>
I had this issue for several days, was not able to find any errors in the PHP and the cURL response seemed to come back completely null. Finally found suggested code to put into the cURL request
if (curl_exec($curl) === FALSE) {
die("Curl Failed: " . curl_error($curl));
} else {
return curl_exec($curl);
}
Adding this finally gave me an error in the PHP which was:
Curl Failed: SSL certificate problem, verify that the CA cert is OK.
Details: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Searching that error on SO gave me this: HTTPS and SSL3_GET_SERVER_CERTIFICATE:certificate verify failed, CA is OK
Which lead me to add code to my cURL request that essentially disables the SSL verification.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
I'm not working w/ my server admin to find a better solution because I don't know if this workaround is so good, but for now this works.
Do a curl_getinfo($ch) after exec to see the response code returned by the server.
Test it with
error_reporting(E_ALL);
Some things to check:
Your value for error_reporting() is pretty weird: 1 equals E_ERROR and implies that you are ignoring almost everything, including warnings. I would not recommend that even in production, not to mention development.
You define $header_list and never use it.
Make sure you are inspecting the real ouput, not the output as rendered by a browser.
I'm sure it'll help you (I've been working through this problem alone for a few days)
add an agent to your cURL:
(option when we have crul_opt in the array)
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100508 SeaMonkey/2.0.4',
(option when setting curl individually)
curl_setopt($curl,CURLOPT_USERAGENT,'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:7.0.1) Gecko/20100101 Firefox/7.0.1');
I noticed that it works because postman gave me the code that I pasted mindlessly into the files, although it did not include an agent (for localhost and in postman it works) and live hosting did not support it. (the postman himself adds his agent)

Categories