I have problem with scraping data over proxy which use authentication. Proxy is online, its private and it work very fast from browser.
However proxy requires authentication and i just cant think of why this simple script doesnt work?!
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://automation.whatismyip.com/n09230945.asp");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_PROXY, "proxyIP:proxyPORT");
curl_setopt($ch, CURLOPT_PROXYUSERPWD,"username:password");
$result = curl_exec($ch);
curl_close($ch);
return $result;
?>
I have tried to set CURLOPT_HTTPAUTH with any valid value and it is same.. Any suggestion?
Try setting:
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
You need to use CURLOPT_PROXYTYPE to tell curl which type of proxy you're using, as it will otherwise assume default type which is HTTP...
Try this :
curl_setopt($ch, CURLOPT_PROXY, 'proxyIP');
curl_setopt($ch, CURLOPT_PROXYPORT, 'proxyPORT');
instead of this :
curl_setopt($ch, CURLOPT_PROXY, "proxyIP:proxyPORT");
Related
I am trying to use curl through a proxy, when I dont use a proxy it works fine but if I use a proxy it returns null, no error messages or anything.
I have already tried other opened questions but their answers are inconclusive.
Here is my code:
<?php
$url = 'http://dynupdate.no-ip.com/ip.php';
$proxy = '122.117.225.161:8080';
//$proxyauth = 'user:pass';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
//curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
var_dump($curl_scraped_page);
?>
The proxy there is a free working proxy (I manually checked), I have also tried it with a private proxy (I used the user:pass authentication method) but still returns null.
Any information will be helpful.
Thanks!
Modern browsers support HTTPS proxies that can be connected to via a PAC file (see https://www.igvita.com/2012/06/25/spdy-and-secure-proxy-support-in-google-chrome/ if you're not familiar).
I'm trying to replicate the same thing and connect to such a proxy via php CURL, but I'm simply getting a blank response, no headers or content.
My code is as follows:
$url = "http://checkip.dyndns.com";
$proxy = "proxy.domain.com:443";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL , 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'test:test');
curl_setopt($ch, CURLOPT_PROXYTYPE, "CURLPROXY_HTTP");
$response = curl_exec($ch);
curl_close($ch);
echo $response;
Any ideas?
If anyone is interested, the solution to this is to use spdycat instead of curl: https://github.com/tatsuhiro-t/spdylay
There's no support for connecting to a proxy with HTTPS with curl yet. There is a work-in-progress branch in git though: https://github.com/bagder/curl/tree/HTTPS-proxy
We will appreciate help to get that in shape to merge.
I am using curl to extract some data of a website.First time it works perfectly , but now it not parse any data from the website.I think they are blocked my server ip.So I can use public proxy ip.But still it not working.My code is ,
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $urls);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_PROXYPORT, '3127');
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_PROXY, '204.93.54.15');
$original_file = curl_exec($ch);
curl_close($ch);
I can't get user name and password of public proxies.Is it important to set public proxy's username and password?It don't show any errors.Any one please help me.
Here is a good answer to use proxy in curl
How to use CURL via a proxy?
Please check this. If you have any queries feel free to ask.
Thanks
Close your connection when you are done with it. That should do the trick.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $urls);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXYPORT, '3127');
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_PROXY, '204.93.54.15');
$original_file = curl_exec($ch);
print_r($original_file);
curl_close($ch);
I am using CURL to invoke an API with self signed public key in .pfx extension
function execute_curl($string,$URL)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $URL);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml','SOAPAction:"http://xxxxxxxxxxxxxxxxxx/SendDoc"'));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $string);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE);
$ch_result = curl_exec($ch);
return $ch_result;
}
I cannot set the CURLOPT_SSL_VERIFYPEER to FALSE for security purpose, anyone can show me how I can properly add the certificate along with its password into curl? EDIT I am not even sure if that is the solution for it, so far I am only getting no response from it. Any additional solution or ways to make it work is very much appreciated.
Also, I am trying to make it work on 2 environment, my dev environment which is in windows and also my test server environment which is in fedora, should the certificate be in different format?
Normal proxies (ex: 72.41.132.22:3128) work well with cURL, however when I use SOCKS 5 proxies with username/pass, It just gives me "[1" on the page.
Is there a way to use SOCKY 5 proxies with cURL ?
$proxy = "cagsan:jw22wdw#108.61.25.223:34792";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
You need to tell cURL the proxy is a SOCKS5 proxy, otherwise cURL assumes it's an HTTP proxy:
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
From the docs:
CURLOPT_PROXYTYPE
Either CURLPROXY_HTTP (default) or CURLPROXY_SOCKS5.
since cURL 7.21.7, you can use CURLOPT_PROXY and specify the SOCKS protocol:
curl_setopt($ch, CURLOPT_PROXY, 'socks5://bob:marley#localhost:12345');
More informations in the libcurl documentation.
For those looking to connect via a hostname(localhost?), and it's not working with CURLPROXY_SOCKS5, you can try CURLPROXY_SOCKS5_HOSTNAME.
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5_HOSTNAME);
In some earlier PHP versions, you will have to do:
curl_setopt($ch, CURLOPT_PROXYTYPE, 7);