Suddenly access denied cUrl in PHP - php

I used the following function to get access to an API (live working example)
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.halteverbotszonen.com/api/numbers');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$output = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
Since a few days (can't tell exactly when) it gives me a 403 error when executing the curl call. Accessing https://www.halteverbotszonen.com/api/numbers directly is possible. I have not changed anything on any of the two servers, what could possibly cause this and where could I see that (any logs for this?)
I have a second api where the same happens (accessible directly works, but not via curl call).
It's the same hoster, could they have changed something that does not allow incoming curl calls?
Any hint appreciated

- Maybe due to https/http ?
- Maybe a different conf inside your apache/php ?
- Maybe the distant server banned your IP
:o
In most of cases, when something works and PAF another day didn't work anymore, it's a software update problem (like conf file) or network problem (like IP) or distant problem (like the server). I guess :D

Related

PHP Curl return different results from URL in browser

I am using PHP Curl with this code:
curl_setopt($ch, CURLOPT_URL, 'https://www.segundamano.mx/anuncios/ciudad-de-mexico/alvaro-obregon/florida/renta-inmuebles/departamentos?precio=0-10000');
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
//curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20100101 Firefox/22.0");
$uagent = 'Mozilla/5.0 (Windows NT 6.1; rv:22.0) Firefox/22.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/36.0.1985.125 Chrome/36.0.1985.125 Safari/537.36';
curl_setopt($ch, CURLOPT_USERAGENT, $uagent);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com');
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
My question is.. why PHP Curl gives different result Than Searching URL in BROWSER?
PHP Curls gives a big BODY CONTENT... with this LINE...
In Spanish... "No encontramos resultados para tu búsqueda..."
In English.....There are no results for your search...
What happen with this URL?
How Can I CURL and read by code this URL and get the REAL RESULTS AS BROWSER?
Help me please!
Thanks!!!
The link you have mentioned is a single-page web application or web site that interacts with the user by dynamically rewriting the current page rather than loading entire new pages from a server.
Also, this website is using vue js.
Please find the below links for more details.
https://en.wikipedia.org/wiki/Single-page_application
https://vuejs.org/
Because JavaScript is the root of all evil. the website gets the search results you want with AJAX after you've succesfully loaded the page. Just open the "network" tab of your browser inspection tool and see the requests flying around.
Fun part: the website does have a (seemingly authorized) API that it can talk too, maybe you can try that? https://webapi.segundamano.mx/nga/api/v1.1/public

file_get_contents (and wget) very slow

I'm using the google text to speech api, but for some reason it's being really slow when I connect to it via php or command line.
I'm doing this:
$this->mp3data = file_get_contents("http://translate.google.com/translate_tts?tl=en&q={$text}");
Where $text is just a urlencoded string.
I've also tried doing it via wget on the command line:
wget http://translate.google.com/translate_tts?tl=en&q=test
Either way takes about 20 seconds or more. Via php it does eventually get the contents and add them to a new file on my server as I want it to. Via wget it times the connection out.
However, if I just go to that url in the browser, it's pretty much instant.
Could anyone shed any light on why this might be occuring?
Thanks.
It's due to how Google parses robots. You need to spoof the User-Agent headers to pretend to be a computer.
Some info on how to go about this would be here:
https://duckduckgo.com/?q=php%20curl%20spoof%20user%20agent
Managed to sort this out now, this is what I ended up doing and now it's only taking a few seconds:
$header=array("Content-Type: audio/mpeg");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $uri);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
$this->mp3data = curl_exec($ch);
curl_close($ch);

checking US-only website status using curl

[Problem]
There is a website which works for US-citizens only (shows info "A" for US-citizens, info "B" for non-US citizens). I need to constantly monitor this webpage for changes ("A" info) - an email should be sent when something is changed! How do I do it? The problem is that I live in Europe!
[Already accomplished]
I have a linux server, daemon and curl PHP script which accomplishes the following task! It works great for all "non-US-only" websites.
[Question]
One way to solve the problem might be to rent a US server but that's not acceptable at all and it is going to cost a lot! I believe that another way to solve the problem might be - to use a US VPN on my server, but for some reasons I won't do that. Is there a way to run curl through proxy maybe? Any ideas?
Current code is the following:
function getrequest($url_site/*,$post_data*/) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url_site);
curl_setopt ($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3');
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE_FILE); // Cookie management.
curl_setopt($ch, CURLOPT_COOKIEFILE, COOKIE_FILE);
$result = curl_exec($ch); // run the whole process
curl_close($ch);
return $result;
}
and
$sleep_time = 1;
$login_wp_url = "http://www.mysite.com";
set_time_limit(60*10);
$result = getrequest($login_wp_url);
How do I grab contents from US-only website?
P.S. to get the idea of what I mean - try visiting the Hulu from Europe countries.
P.P.S. that's not a Hulu, not a homework.
Many cloud service providers, e.g. Heroku and Amazon, offer their smallest instances for free. You could simply set up one of these for free, make sure that you are provisioned on an US-located server and run your script there.
Another possibility would be to use a (free) proxy for these requests. Here is a list of free proxie servers: http://www.xroxy.com/proxy-country-US.htm.
curl_setopt($ch, CURLOPT_PROXY, "http://160.76.xxx.xxx:8080");
curl_setopt($ch, CURLOPT_PROXYPORT, 8080);
curl_setopt ($ch, CURLOPT_PROXYUSERPWD, "xxx:xxx");

There is something wrong with my CURL script

I'm using a CURL script to basically recreate the process a user having to hit send on a form. I would like everything to run in the background but it never sends when this script executes.
$ch = curl_init();
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS,
"&shopId=".$ID."&encodedMessage=".urlencode($encodedMessage)."&signature=".$signature);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$result = curl_exec($ch);
Is it perhaps not executing?
Tested, working great.
Did you set the variable $url right?
curl is not a good option to emulate fork.
As soon as your current script stops, the curl request will be killed and probably the target url will also stop executing its own script.
you could use a cron or similar external process to run the curl requests and it would run with no interruption. It would also be helpful to queue up parallel requests.

How do I download files of big sizes from somewhere on the web to the web server with PHP?

How do I download files of big sizes from somewhere on the web to the web server with PHP? Also, what should be allowed on the server in order to make this happen? Thanks.
Could this do a good job?
<?php
ini_set(max_execution_time, 0);
$the_link = $_GET['url'];
$ch = curl_init($the_link);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322;)");
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/cookies.txt');
$the_file = curl_exec($ch);
curl_close($ch);
$hdl = fopen("file", 'w');
fwrite($hdl, $the_file);
fclose($hdl);
?>
Use curl for that. PHP must have curl support and you need, obviously, a writeable filesystem.

Categories