I've had my cURL implementation running successfully for the last few months without hiccups; however, last week I suddenly started to have a problem with one specific website (www.viewmag.com). I can visit the site (and have it resolve) perfectly in a browser, but cURL returns the following:
* About to connect() to www.viewmag.com port 80 (#0)
* Trying 205.178.145.65... * Timeout
* connect() timed out!
* Closing connection #0
For sanity, I tried to ping the website with two different boxes, but each ping timed out.
Box 1 (Linux):
ping www.viewmag.com
PING www.viewmag.com (205.178.145.65) 56(84) bytes of data.
Box 2 (Windows):
ping www.viewmag.com
Pinging www.viewmag.com [205.178.145.65] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
My cURL is as follows:
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, 'http://www.viewmag.com');
curl_setopt ($ch, CURLOPT_USERAGENT, 'cURL crawler');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt ($ch, CURLOPT_AUTOREFERER, true);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 2);
$html = curl_exec($ch);
Does anyone have any thoughts as to why cURL is failing and why I would be able to visit this site in a browser, but not be able to ping/cURL it? Thanks in advance
Maybe your server IP is banned on that site?
Maybe try to set longer timeout? I visited that site and it works so slow, that you may need more than 5 seconds.
Added later:
Looks like your server IP is banned.
I tried this (its copy of your code, changes are in comments):
<?php
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, 'http://www.viewmag.com');
// I changed UA here
curl_setopt ($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt ($ch, CURLOPT_AUTOREFERER, true);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 2);
$html = curl_exec($ch);
// I added this
echo $html;
?>
and it works on my test server (data center in Germany).
In all probability they have increased security in their server. Some setting in server has changed to stop you from cUrling it. Try masquerading as a known user agent. Pinging might not work because they have just taken down the ping server so that attacks such as Distributed Denial of Service (DDOS) can be thwarted. Sadly at this point it cannot be determined what exact combination can or will make it work. You will need to employ trial and error.
Related
I am trying to access an API URL using curl, which has worked perfectly on my computer. But once I moved over my website to a VPS it suddenly stopped working, due to that the API-Host is using cloudflare, and since my VPS-IP eventually looks "suspect", it forces me to pass this 5 second firewall before I visit the site.
I don't know how I can bypass this, or somehow get trough this layer using curl.
This is my code.
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_ENCODING, '');
curl_setopt ($ch, CURLOPT_MAXREDIRS, 10);
curl_setopt ($ch, CURLOPT_TIMEOUT, 180);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt ($ch, CURLOPT_FAILONERROR, true);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
$result = curl_exec($ch);
curl_close ($ch);
I have tried to set a specific browser for the curl request, and see if that works, but no, the API won't give me anything back except nullpointers.
Any help appreciated.
I'm trying to get HLS stream in other site, but they always return 403 Forbidden :( This is my function, it works well on localhost, but not in my server.
function getPage($url, $referer, $header){
$timeout=30;
$curl = curl_init();
if(strstr($referer,"://")){
curl_setopt ($curl, CURLOPT_REFERER, $referer);
}
curl_setopt ($curl, CURLOPT_URL, $url);
curl_setopt ($curl, CURLOPT_TIMEOUT, $timeout);
curl_setopt ($curl, CURLOPT_USERAGENT, sprintf("Mozilla/%d.0",rand(4,5)));
curl_setopt ($curl, CURLOPT_HEADER, $header);
curl_setopt ($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($curl, CURLOPT_SSL_VERIFYPEER, 0);
$html = curl_exec ($curl);
curl_close ($curl);
return $html;
}
echo getPage("http://www.wezatv.com/dooball/assp1.php", "http://www.wezatv.com/", 1);
Anyone can help me?
Thanks,
make sure that your server IP is not a IP
use curl proxy to test ,
your link http://www.wezatv.com/dooball/assp1.php works normally on my server
Your code is 100% correct, I think the problem is your server blocked that request or the target server blocked your server's IP
Make sure your server doesn't block the out connection on port 80
If no firewall, could you try to execute the command "wget http://www.wezatv.com/dooball/assp1.php" to see can it download the content? If it also cannot get the content with error code is 403, it mean the server wezatv.com blocked every request from your server.
Some server temporary block every request if it found too many request in a short moment. So if you want to claw it's data, you should reduce the number of request, such as implement a delay
I am trying to login here http://studio.smartshoot.com/filmmaker/home using CURL, but can't seem to get it to work.The result is it says "JavaScript is not enabled in your browser. Please enable JavaScript in your browser settings before continuing to use this site."
So far, I've tried the following:
<?
$username="username";
$password="password";
$url="http://ABC.COM?q=login&destination=filmmaker%2Fhome";
$cookie="cookie.txt";
$postdata = "name=".$username."&pass=".$password."&edit-user-login=user_login&edit-name=".$username."&remember_me=1";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt ($ch, CURLOPT_REFERER, $url);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt ($ch, CURLOPT_POST, 1);
$result = curl_exec ($ch);
echo $result;
curl_close($ch);
?>
the answer's in the question. cURL does not execute JavaScript, so even if you're logged in you'll get whatever message the authors intended for a noscript user-agent.
This is simple like this - a page tests, if you have JavaScript enabled, by using the JavaScript to execute a command, that will remove things like "you don't have JavaScript enabled".
Therefore, if you do NOT have JS enabled, the error won't go away.
Optionally (and even more likely) the JavaScript will REDIRECT your browser to the actual page, if JS is not enabled - again, you will stay at the first page, which says you don't have JS enabled...
PHP curl is NOT a full browser. It is just a library that is used for communicating with servers, using HTTP, FTP, et cetera. It does not do neither rendering nor parsing.
For this functionality, you would need a JavaScript engine... to my knowledge there is not one in PHP that is fully functional.
[Problem]
There is a website which works for US-citizens only (shows info "A" for US-citizens, info "B" for non-US citizens). I need to constantly monitor this webpage for changes ("A" info) - an email should be sent when something is changed! How do I do it? The problem is that I live in Europe!
[Already accomplished]
I have a linux server, daemon and curl PHP script which accomplishes the following task! It works great for all "non-US-only" websites.
[Question]
One way to solve the problem might be to rent a US server but that's not acceptable at all and it is going to cost a lot! I believe that another way to solve the problem might be - to use a US VPN on my server, but for some reasons I won't do that. Is there a way to run curl through proxy maybe? Any ideas?
Current code is the following:
function getrequest($url_site/*,$post_data*/) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url_site);
curl_setopt ($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3');
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE_FILE); // Cookie management.
curl_setopt($ch, CURLOPT_COOKIEFILE, COOKIE_FILE);
$result = curl_exec($ch); // run the whole process
curl_close($ch);
return $result;
}
and
$sleep_time = 1;
$login_wp_url = "http://www.mysite.com";
set_time_limit(60*10);
$result = getrequest($login_wp_url);
How do I grab contents from US-only website?
P.S. to get the idea of what I mean - try visiting the Hulu from Europe countries.
P.P.S. that's not a Hulu, not a homework.
Many cloud service providers, e.g. Heroku and Amazon, offer their smallest instances for free. You could simply set up one of these for free, make sure that you are provisioned on an US-located server and run your script there.
Another possibility would be to use a (free) proxy for these requests. Here is a list of free proxie servers: http://www.xroxy.com/proxy-country-US.htm.
curl_setopt($ch, CURLOPT_PROXY, "http://160.76.xxx.xxx:8080");
curl_setopt($ch, CURLOPT_PROXYPORT, 8080);
curl_setopt ($ch, CURLOPT_PROXYUSERPWD, "xxx:xxx");
i'm trying to make post to an external url using curl, the externa page use https, here is the desc of the server i'm using
Server Apache/2.2.11 (Win32) mod_ssl/2.2.11 OpenSSL/0.9.8k PHP/5.3.0
the external url make a redirect to another url that i send in the post, but everytime i try i get this error
curl_errno=35 (Unknown SSL protocol error in connection to [secure site]:443)
so i check the firebug for the response and it say
Failed to load source for: http://localhost/3Party/PHP_VPC_3Party_Auth_Capture_Order_DO.php
Here is the code I'm using
ob_start();
// initialise Client URL object
$ch = curl_init();
// set the URL of the VPC
curl_setopt ($ch, CURLOPT_URL, $vpcURL);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $this->postData);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_exec ($ch);
if (curl_error($ch)) {
$this->errorMessage =
"curl_errno=". curl_errno($ch) . " (" . curl_error($ch) . ")";
}
curl_close ($ch);
I think the problem is the fact that you are trying to access an "http" URL (instead of "https") on port 443.
You can also try setting the SSL version manually:
curl_setopt($ch, CURLOPT_SSLVERSION, 3);
Replace 3 with whatever SSL version the remote server is using.
After a few weeks dealing with this issue, i was able to at least establish the connection, i don't know if it is the real answer but it works for me, i just added to the example above, the options to use proxy, just like this
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_PROXYAUTH, CURLAUTH_NTLM );
curl_setopt($ch, CURLOPT_PROXY, 'my.proxy');
curl_setopt($ch, CURLOPT_PROXYPORT, 'my.port');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'domain\user:password');
hope this can help
It might also be tls/ssl version preference by the server. In this case, you have to try specify different version constants from here: https://curl.haxx.se/libcurl/c/CURLOPT_SSLVERSION.html
E.g. what worked for me was:
curl_setopt($ch, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_1);