I am trying to access an API URL using curl, which has worked perfectly on my computer. But once I moved over my website to a VPS it suddenly stopped working, due to that the API-Host is using cloudflare, and since my VPS-IP eventually looks "suspect", it forces me to pass this 5 second firewall before I visit the site.
I don't know how I can bypass this, or somehow get trough this layer using curl.
This is my code.
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_ENCODING, '');
curl_setopt ($ch, CURLOPT_MAXREDIRS, 10);
curl_setopt ($ch, CURLOPT_TIMEOUT, 180);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt ($ch, CURLOPT_FAILONERROR, true);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
$result = curl_exec($ch);
curl_close ($ch);
I have tried to set a specific browser for the curl request, and see if that works, but no, the API won't give me anything back except nullpointers.
Any help appreciated.
Related
I'm trying to get HLS stream in other site, but they always return 403 Forbidden :( This is my function, it works well on localhost, but not in my server.
function getPage($url, $referer, $header){
$timeout=30;
$curl = curl_init();
if(strstr($referer,"://")){
curl_setopt ($curl, CURLOPT_REFERER, $referer);
}
curl_setopt ($curl, CURLOPT_URL, $url);
curl_setopt ($curl, CURLOPT_TIMEOUT, $timeout);
curl_setopt ($curl, CURLOPT_USERAGENT, sprintf("Mozilla/%d.0",rand(4,5)));
curl_setopt ($curl, CURLOPT_HEADER, $header);
curl_setopt ($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($curl, CURLOPT_SSL_VERIFYPEER, 0);
$html = curl_exec ($curl);
curl_close ($curl);
return $html;
}
echo getPage("http://www.wezatv.com/dooball/assp1.php", "http://www.wezatv.com/", 1);
Anyone can help me?
Thanks,
make sure that your server IP is not a IP
use curl proxy to test ,
your link http://www.wezatv.com/dooball/assp1.php works normally on my server
Your code is 100% correct, I think the problem is your server blocked that request or the target server blocked your server's IP
Make sure your server doesn't block the out connection on port 80
If no firewall, could you try to execute the command "wget http://www.wezatv.com/dooball/assp1.php" to see can it download the content? If it also cannot get the content with error code is 403, it mean the server wezatv.com blocked every request from your server.
Some server temporary block every request if it found too many request in a short moment. So if you want to claw it's data, you should reduce the number of request, such as implement a delay
I am trying to login here http://studio.smartshoot.com/filmmaker/home using CURL, but can't seem to get it to work.The result is it says "JavaScript is not enabled in your browser. Please enable JavaScript in your browser settings before continuing to use this site."
So far, I've tried the following:
<?
$username="username";
$password="password";
$url="http://ABC.COM?q=login&destination=filmmaker%2Fhome";
$cookie="cookie.txt";
$postdata = "name=".$username."&pass=".$password."&edit-user-login=user_login&edit-name=".$username."&remember_me=1";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt ($ch, CURLOPT_REFERER, $url);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt ($ch, CURLOPT_POST, 1);
$result = curl_exec ($ch);
echo $result;
curl_close($ch);
?>
the answer's in the question. cURL does not execute JavaScript, so even if you're logged in you'll get whatever message the authors intended for a noscript user-agent.
This is simple like this - a page tests, if you have JavaScript enabled, by using the JavaScript to execute a command, that will remove things like "you don't have JavaScript enabled".
Therefore, if you do NOT have JS enabled, the error won't go away.
Optionally (and even more likely) the JavaScript will REDIRECT your browser to the actual page, if JS is not enabled - again, you will stay at the first page, which says you don't have JS enabled...
PHP curl is NOT a full browser. It is just a library that is used for communicating with servers, using HTTP, FTP, et cetera. It does not do neither rendering nor parsing.
For this functionality, you would need a JavaScript engine... to my knowledge there is not one in PHP that is fully functional.
I've had my cURL implementation running successfully for the last few months without hiccups; however, last week I suddenly started to have a problem with one specific website (www.viewmag.com). I can visit the site (and have it resolve) perfectly in a browser, but cURL returns the following:
* About to connect() to www.viewmag.com port 80 (#0)
* Trying 205.178.145.65... * Timeout
* connect() timed out!
* Closing connection #0
For sanity, I tried to ping the website with two different boxes, but each ping timed out.
Box 1 (Linux):
ping www.viewmag.com
PING www.viewmag.com (205.178.145.65) 56(84) bytes of data.
Box 2 (Windows):
ping www.viewmag.com
Pinging www.viewmag.com [205.178.145.65] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
My cURL is as follows:
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, 'http://www.viewmag.com');
curl_setopt ($ch, CURLOPT_USERAGENT, 'cURL crawler');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt ($ch, CURLOPT_AUTOREFERER, true);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 2);
$html = curl_exec($ch);
Does anyone have any thoughts as to why cURL is failing and why I would be able to visit this site in a browser, but not be able to ping/cURL it? Thanks in advance
Maybe your server IP is banned on that site?
Maybe try to set longer timeout? I visited that site and it works so slow, that you may need more than 5 seconds.
Added later:
Looks like your server IP is banned.
I tried this (its copy of your code, changes are in comments):
<?php
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, 'http://www.viewmag.com');
// I changed UA here
curl_setopt ($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt ($ch, CURLOPT_AUTOREFERER, true);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 2);
$html = curl_exec($ch);
// I added this
echo $html;
?>
and it works on my test server (data center in Germany).
In all probability they have increased security in their server. Some setting in server has changed to stop you from cUrling it. Try masquerading as a known user agent. Pinging might not work because they have just taken down the ping server so that attacks such as Distributed Denial of Service (DDOS) can be thwarted. Sadly at this point it cannot be determined what exact combination can or will make it work. You will need to employ trial and error.
I have a PHP script that primes my website cache.
I'm trying to bypass the Nginx frontend and grab the headers from the Apache backend.
The following example works (via command line) :
curl -I -H "Host: example.com" 127.0.0.1
However, when I try to do the same in PHP - it does not.
$headers = array("Host: example.com");
$url = "127.0.0.1";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 60);
curl_setopt ($ch, CURLOPT_HEADER, true);
curl_setopt ($ch, CURLOPT_NOBODY, true);
curl_setopt ($ch, CURLOPT_HTTPHEADER, $headers);
$ret = curl_exec ($ch);
curl_close ($ch);
echo "$ret";
This always returns the first website listed in my apache virtualhost files - not the website listed in the 'host' http headers.
Any idea why it works just fine via command line - but not in the php script?
I don't know if it is possible at all, however what I would try is to pass the URL at init:
$ch = curl_init($url);
And then not with curl_setopt. It's just a suggestion to test, no guarantee this helps. So more a lengthy comment.
You can also enable verbose mode and see what specifically curl sends for request headers, I have outlined this in an answer here: Bad Request. Connecting to sites via curl on host and system. This should show what is going on behind the curtain, very useful if you do not yet sniff the network traffic.
I think here is the answer.
"Do this instead:
$host = "www.example.com";
$url = "http://$host/";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
"
I use this code to login to remote server. Evrything working fine when I don't use proxy. But with proxy it doesn't.
Here is code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_TIMEOUT, 20);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, $ref_url);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy.':'.$port);
ob_start();
return curl_exec ($ch); // execute the curl command
ob_end_clean();
curl_close ($ch);
unset($ch);
I try to disable CURLOPT_HTTPPROXYTUNNEL but it didn't help.
So, without proxy I can login fine.
With proxy not.
Proxy is good ad working.
What's the proxy error you get using the code you posted? Does the proxy need authentication?
I was having a same problem , the code works fine without the proxy but with it its returning nothing
, I have done some debugging and found out that actually on most of the servers only the port 80 is allowed , and many of the proxies we use connect through another port
You can unblock the port no for which you have maximum proxies
You can filter the list with only port 80 proxies
Note : Also only use port http, https and socks4 proxies , socks5 proxies runs on udp not on tcp/ip