I am using curl to extract some data of a website.First time it works perfectly , but now it not parse any data from the website.I think they are blocked my server ip.So I can use public proxy ip.But still it not working.My code is ,
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $urls);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_PROXYPORT, '3127');
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_PROXY, '204.93.54.15');
$original_file = curl_exec($ch);
curl_close($ch);
I can't get user name and password of public proxies.Is it important to set public proxy's username and password?It don't show any errors.Any one please help me.
Here is a good answer to use proxy in curl
How to use CURL via a proxy?
Please check this. If you have any queries feel free to ask.
Thanks
Close your connection when you are done with it. That should do the trick.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $urls);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXYPORT, '3127');
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_PROXY, '204.93.54.15');
$original_file = curl_exec($ch);
print_r($original_file);
curl_close($ch);
Related
I am trying to use curl through a proxy, when I dont use a proxy it works fine but if I use a proxy it returns null, no error messages or anything.
I have already tried other opened questions but their answers are inconclusive.
Here is my code:
<?php
$url = 'http://dynupdate.no-ip.com/ip.php';
$proxy = '122.117.225.161:8080';
//$proxyauth = 'user:pass';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
//curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
var_dump($curl_scraped_page);
?>
The proxy there is a free working proxy (I manually checked), I have also tried it with a private proxy (I used the user:pass authentication method) but still returns null.
Any information will be helpful.
Thanks!
Modern browsers support HTTPS proxies that can be connected to via a PAC file (see https://www.igvita.com/2012/06/25/spdy-and-secure-proxy-support-in-google-chrome/ if you're not familiar).
I'm trying to replicate the same thing and connect to such a proxy via php CURL, but I'm simply getting a blank response, no headers or content.
My code is as follows:
$url = "http://checkip.dyndns.com";
$proxy = "proxy.domain.com:443";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL , 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'test:test');
curl_setopt($ch, CURLOPT_PROXYTYPE, "CURLPROXY_HTTP");
$response = curl_exec($ch);
curl_close($ch);
echo $response;
Any ideas?
If anyone is interested, the solution to this is to use spdycat instead of curl: https://github.com/tatsuhiro-t/spdylay
There's no support for connecting to a proxy with HTTPS with curl yet. There is a work-in-progress branch in git though: https://github.com/bagder/curl/tree/HTTPS-proxy
We will appreciate help to get that in shape to merge.
$url = 'http://dynupdate.no-ip.com/ip.php';
$proxy = '127.0.0.1:8888';
//$proxyauth = 'user:password';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
//curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;
How would I make this continuously connect to this server?
I'm working on a bot for this site to improve the technology. Also, how would I authenticate cookies on the server, for example: I send a request to the server with this script using the certain cookies I would like to use.
There is an option called CURLOPT_COOKIEJAR where you can set the file that contains the cookie:
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
Once you get the cookie you have to use the same $ch variable to keep the connection.
You can found a complete example of cURL usage here: http://blog.binacube.com/2014/08/complete-curl-example-login-proxy-and.html
I am working on core php.
I am requesting a URL by curl. This is working on my development server but same thing are not working on live server.
Below is my Code:
$url = "http://www.streamatemodels.com/login.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 15);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'submitted=1&g=&sausr='.urlencode('username').'&sapwd='.urlencode('password').'');
$result = curl_exec($ch);
$error = curl_errno($ch);
$code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
print_r($result);
Please help me if anyone have solution.
Update
I checked this with new username and password but same things happen. Its working on my development server but not working on live and local host.
The following line:
curl_setopt($ch, CURLOPT_POSTFIELDS, 'submitted=1&g=&sausr='.urlencode('username').'&sapwd='.urlencode('password').'');
should be changed to:
curl_setopt($ch, CURLOPT_POSTFIELDS, 'submitted=1&g=&sausr='.urlencode($username).'&sapwd='.urlencode($password).'');
If it still doesn't work try to create the query-string outside of curl_setopt:
$params = 'submitted=1&g=&sausr='.urlencode($username).'&sapwd='.urlencode($password).'';
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
Another advise: it's better practice to use http_build_query($params)
UPDATE
I had the time to check the code, and it works fine on my local host (meaning: check your username/password!):
I have problem with scraping data over proxy which use authentication. Proxy is online, its private and it work very fast from browser.
However proxy requires authentication and i just cant think of why this simple script doesnt work?!
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://automation.whatismyip.com/n09230945.asp");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_PROXY, "proxyIP:proxyPORT");
curl_setopt($ch, CURLOPT_PROXYUSERPWD,"username:password");
$result = curl_exec($ch);
curl_close($ch);
return $result;
?>
I have tried to set CURLOPT_HTTPAUTH with any valid value and it is same.. Any suggestion?
Try setting:
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
You need to use CURLOPT_PROXYTYPE to tell curl which type of proxy you're using, as it will otherwise assume default type which is HTTP...
Try this :
curl_setopt($ch, CURLOPT_PROXY, 'proxyIP');
curl_setopt($ch, CURLOPT_PROXYPORT, 'proxyPORT');
instead of this :
curl_setopt($ch, CURLOPT_PROXY, "proxyIP:proxyPORT");