Curl Location.reload(true) Result - php

When I try to get markafoni.com's html data with CURL, it returns;
<script>
document.cookie = 'NSId=2;expires=Sun, 17-Jan-2038 01:00:00 GMT';
location.reload(true);
</script><noscript>%90'a varan indirim Markafoni'de</noscript>
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER,true);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_COOKIESESSION,true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt($ch, CURLOPT_REFERER, 'http://www.markafoni.com/');
curl_setopt($ch, CURLOPT_URL, 'https://www.markafoni.com/');
How can I solve this problem?

The problem is the server's technique for setting cookies, which is rather idiosyncratic. It may in fact be intended to prevent screen-scraping like you're doing, but there may be other reasons.
The server clearly has two different responses.
If there is no NSId cookie set, send some Javascript to set one and send nothing else.
If there is an NSId cookie set, send the page content.
cURL obviously can receive and set cookies from the server, as you do with the following code:
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
This, however, presumes that the server is setting cookies in the normal way, i.e. using the Cookie HTTP header. Since it's doing it with Javascript (highly idiosyncratic!) cURL doesn't understand it.
You'll have to set the cookie yourself using the CURLOPT_COOKIE option:
curl_setopt($ch, CURLOPT_COOKIE, 'NSId=2');
The CURLOPT_COOKIE option sets the cookie string to be sent by cURL.

Related

Failed to call the browser based request using PHP Curl

If we call the same URL in the browser then it auto-download the CSV file. So, I want the same feature using PHP curl to save the CSV file under the same folder. But it gives me an empty result every time. Can you please guide me on what's is wrong in the code below?
$url="https://www.centrano.com/catalog_download.php?email=info#sporttema.com&password=dHB3L1FpTEg1c2pLZ29SUkdnUWcwWTFqN2RIamQx";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
$agent= "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36";
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 100000); //time out of 15 seconds
$output = curl_exec($ch);
print_r($output);
curl_close($ch);
Add the following two CURL options to make it work:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/cookies.txt');
The page redirects several times to the same host and requires for the session cookies to remain present (thus, storing them in a cookie jar.
Also: change your centrano.com account password IMMEDIATELY! Even though it helped solving this, it's generally not a good idea to make it public.

PHP Curl return different results from URL in browser

I am using PHP Curl with this code:
curl_setopt($ch, CURLOPT_URL, 'https://www.segundamano.mx/anuncios/ciudad-de-mexico/alvaro-obregon/florida/renta-inmuebles/departamentos?precio=0-10000');
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
//curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20100101 Firefox/22.0");
$uagent = 'Mozilla/5.0 (Windows NT 6.1; rv:22.0) Firefox/22.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/36.0.1985.125 Chrome/36.0.1985.125 Safari/537.36';
curl_setopt($ch, CURLOPT_USERAGENT, $uagent);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com');
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
My question is.. why PHP Curl gives different result Than Searching URL in BROWSER?
PHP Curls gives a big BODY CONTENT... with this LINE...
In Spanish... "No encontramos resultados para tu búsqueda..."
In English.....There are no results for your search...
What happen with this URL?
How Can I CURL and read by code this URL and get the REAL RESULTS AS BROWSER?
Help me please!
Thanks!!!
The link you have mentioned is a single-page web application or web site that interacts with the user by dynamically rewriting the current page rather than loading entire new pages from a server.
Also, this website is using vue js.
Please find the below links for more details.
https://en.wikipedia.org/wiki/Single-page_application
https://vuejs.org/
Because JavaScript is the root of all evil. the website gets the search results you want with AJAX after you've succesfully loaded the page. Just open the "network" tab of your browser inspection tool and see the requests flying around.
Fun part: the website does have a (seemingly authorized) API that it can talk too, maybe you can try that? https://webapi.segundamano.mx/nga/api/v1.1/public

How to use User's current IP address as the proxy in PHP cURL?

Is it possible to use user's current IP address as the proxy address with PHP cURL? Currently, I am trying following code but giving request time out error. I test it with hma proxy it is working fine but when I'm trying to uses user's IP address it is giving request time error in the server's log.
function get_page($url){
$proxy=$_SERVER['REMOTE_ADDR'].":80";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_HEADER, 0); // return headers 0 no 1 yes
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // return page 1:yes
curl_setopt($ch, CURLOPT_TIMEOUT, 40); // http request timeout 20 seconds
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // Follow redirects, need this if the url changes
curl_setopt($ch, CURLOPT_MAXREDIRS, 2); //if http server gives redirection responce
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); // cookies storage / here the changes have been made
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // false for https
curl_setopt($ch, CURLOPT_ENCODING, "gzip"); // the page encoding
$data = curl_exec($ch); // execute the http request
curl_close($ch); // close the connection
return $data;
}
There are a lots of live examples which is doing same. For example these two tools from Google
google.com/gwt/x
developers.google.com/speed/pagespeed/insights
Not possible. While you COULD make your server issue a SYN packet using the user's IP as the "source", any reply packets from the target machine will go to the USER's machine, not your server.
It is NOT possibl to spoof a complete TCP connection unless you control the network infrastructure upstream of the target/victim.

Login to remote url using Curl -PHP

I am trying to login here http://studio.smartshoot.com/filmmaker/home using CURL, but can't seem to get it to work.The result is it says "JavaScript is not enabled in your browser. Please enable JavaScript in your browser settings before continuing to use this site."
So far, I've tried the following:
<?
$username="username";
$password="password";
$url="http://ABC.COM?q=login&destination=filmmaker%2Fhome";
$cookie="cookie.txt";
$postdata = "name=".$username."&pass=".$password."&edit-user-login=user_login&edit-name=".$username."&remember_me=1";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt ($ch, CURLOPT_REFERER, $url);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt ($ch, CURLOPT_POST, 1);
$result = curl_exec ($ch);
echo $result;
curl_close($ch);
?>
the answer's in the question. cURL does not execute JavaScript, so even if you're logged in you'll get whatever message the authors intended for a noscript user-agent.
This is simple like this - a page tests, if you have JavaScript enabled, by using the JavaScript to execute a command, that will remove things like "you don't have JavaScript enabled".
Therefore, if you do NOT have JS enabled, the error won't go away.
Optionally (and even more likely) the JavaScript will REDIRECT your browser to the actual page, if JS is not enabled - again, you will stay at the first page, which says you don't have JS enabled...
PHP curl is NOT a full browser. It is just a library that is used for communicating with servers, using HTTP, FTP, et cetera. It does not do neither rendering nor parsing.
For this functionality, you would need a JavaScript engine... to my knowledge there is not one in PHP that is fully functional.

There is something wrong with my CURL script

I'm using a CURL script to basically recreate the process a user having to hit send on a form. I would like everything to run in the background but it never sends when this script executes.
$ch = curl_init();
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS,
"&shopId=".$ID."&encodedMessage=".urlencode($encodedMessage)."&signature=".$signature);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$result = curl_exec($ch);
Is it perhaps not executing?
Tested, working great.
Did you set the variable $url right?
curl is not a good option to emulate fork.
As soon as your current script stops, the curl request will be killed and probably the target url will also stop executing its own script.
you could use a cron or similar external process to run the curl requests and it would run with no interruption. It would also be helpful to queue up parallel requests.

Categories