$ckfile = tempnam ("/tmp", "CURLCOOKIE");
$useragent="Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16";
$ch = curl_init ("website.com");
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec ($ch);
The website sets a cookie and then redirects. Would this code suffice? Because it seems to be not getting the cookie properly. How can I check to see if it's set? Better yet, if I know what cookies I want can I just make it or something?
Edit: So my CURL script visits the website right? The website sets cookies for validation, and I want to see if my cURL script is receiving those cookies properly. I want to know if there's a test for that, and/or I want to know if I can just create a cookie to validate for the website.
I recently had a project where I needed to pass cookies between servers, and I found that setting both CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR to the same file did the trick.
If you're just wanting to see if the cookies from the remote site are being set, you should be able to look at the CURLOPT_COOKIEJAR file in a text editor.
If this were the command line curl you'd throw in the -v parameter. Pay attention to the lines starting with > Cookie:.
The php version of this is:
curl_setopt($ch,CURLOPT_VERBOSE,TRUE);//and set CURLOPT_STDERR to STDOUT
And if this were a browser I would debug with one of these: you can use Fiddler on IE, TamperData or Firebug (net) on Firefox, the inspector on Chrome and on Safari, or a proxy with any browser to watch how the form is submitted by a "normal" browser.
The proxy approach might work if you set php's curl to use it.
curl_setopt($ch,CURLOPT_HTTPPROXYTUNNEL,TRUE);//& CURLOPT_PROXYPORT CURLOPT_PROXY
If you want to just add cookies to one request you could write a line like:
$ch->headers[] = 'Cookie: recent=543..; _session_id=6185..; __utma=572.1.1.1.1; __utmc=572..; __utmz=572.1.1.1.1.utmccn=(referral)|utmcsr=domain.com|utmcct=/request/path|utmcmd=referral';
but I'm not certain about that one, these might be more along the same line:
curl_setopt($ch,CURLOPT_COOKIE,"recent=543..; _session_id=618..");
// Possibly, but I think this might overwrite other headers.
curl_setopt($ch,CURLOPT_HTTPHEADER, "Cookie: recent=543..; _session_id=6185..");
Related
I'm a novice programmer in PHP. Last week I read about cURL that capture my attention to study it. first, I copy and paste codes posted on different blogs and it goes run good like my code below.
<?php
$handle=curl_init('http://www.google.co.kr/');
curl_setopt($handle, CURLOPT_VERBOSE, true);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($handle, CURLOPT_SSL_VERIFYPEER, false);
$content = curl_exec($handle);
echo $content;
?>
BUT WHY I CAN'T cURL the website
http://www.todayhumor.co.kr/
since that, i am using same code above it outputs
Looking for your positive response guys. thank you in advance.
After calling curl_exec($handle) you should close the session with curl_close($handle). Maybe you tried so many times and now it doesn't work anymore, because you have so many open sessions on your local server. I would add that line to your code, restart xampp and try again.
Edit:
The server rejects requests without a valid user-agent. Add a user-agent to your request: curl_setopt($handle, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36'); this worked for me
I've got a really odd problem and no idea how to debug it. Maybe some experienced developer can help me. I've the following code:
$url = 'https://home.mobile.de/home/ses.html?customerId=471445&json=true&_='.time();
echo $url;
$agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36';
// Initiate curl
$ch = curl_init();
// Activate debugging
curl_setopt($ch, CURLOPT_VERBOSE, true);
// Disable SSL verification
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
// Will return the response, if false it print the response
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Set browser user agent
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
// Set the url
curl_setopt($ch, CURLOPT_URL,$url);
// Execute
$result=curl_exec($ch);
// Closing
curl_close($ch);
$php_object = json_decode($result);
var_dump($php_object);
I've put this code into a php file called playground.php. If I open playground.php with Chrome (I am using MAMP as local server) then everything works as expected. Also if I run on the osx command line "php playground.php" it works as expected, but for any reason it does not work if I run it inside the Phpstorm cli as shown below.
Any idea what could be wrong and how I can debug this issue?
Many thanks in advance.
Thanks to LazyOne I was able to find out that a firewall rule was blocking the outgoing request. Many thanks!
Seem to be in a bit of a predicament. As far as I am aware, there have been no changed to PHP or Apache, however a code that has worked for almost 6 months just stoped working today at 2pm.
The code is:
function ls_record($prospectid,$campid){
$api_post = "method=NewProspect&prospect_id=".$prospectid."&campaign_id=".$campid;
$ch = curl_init();
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $api_post);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_URL, "http://XXXXX/XXXXXX/store.php");
$x = print_r(curl_exec($ch), TRUE);
return $x;
}
It returns NULL, I tried usingfile_get_contents()which also returnsNULL`. I checking the Apache error logs and see nothing...I need some help on this one.
Do you have access to the command line of the server? It could be that the destination has blocked you somehow.
If you have command line access, try this
wget http://XXXXX/XXXXXX/store.php
That should at least return something (if not headers)
use curl_getinfo to check your curl execution status, it maybe that the server you try to extract content from need your curl to set user-agent, some site check user-agent to block unwanted curl access.
below are the user agent I used to disguise my curl as desktop chrome browser.
curl_setopt($ch,CURLOPT_USERAGENT,' Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36');
I face same problem on my server , because of the low internet speed. Internet speed is go down for some time and curl take so many time to execute , so it return a timeout error . After a few minute it is working fine without any changes on server.
I'm using cUrl with cookies and user agents to log into several websites and fetch information. It's very convenient but it doesn't work for some websites. And I'm very curious about why. For example, I can't use it to log in to my Facebook account or a Wordpress site. Surprisingly it works fine with twitter and most other websites. Do some websites have extra security check, and if so , what are they and how can I pass them? Thanks!
Following is the php code I used:
$webPage = "https://twitter.com";
$myCookieFile = "myCookies.txt"; //cookies are prepared here
$myUserAgent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US;
rv:1.9.1.2)Gecko/20090729 Firefox/3.5.2 GTB5';
$ch = curl_init ($webPage);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $myCookieFile);//cookieFile used to read
curl_setopt($ch, CURLOPT_USERAGENT, $myUserAgent);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec ($ch);
There are a number of issues with this.
One is that a lot of sites DO and SHOULD use SSL for logins. In that case, you'll need the CURL opt:
CURLOPT_SSL_VERIFYPEER
Secondly, some sites require you load a page of their site before you can use their submit form.
Third, some sites may use 301 redirects. I have seen some that even use META redirects after login. For this use:
CURLOPT_FOLLOWLOCATION
Finally, I recommend using this for CURL:
http://semlabs.co.uk/journal/object-oriented-curl-class-with-multi-threading
I'm writing a cURL script, but how can I check if it's working and passing properly when it's visiting the website?
$ckfile = '/tmp/cookies.txt';
$useragent= "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7A400";
$ch = curl_init ("http://website.com");
curl_setopt($ch, CURLOPT_AUTOREFERER , true);
=> true
curl_setopt($ch, CURLOPT_USERAGENT, $useragent); // set user agent
curl_setopt ($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$output = curl_exec ($ch);
curl_close($ch);
just make a php page like this on your server and try your script on your own url
var_dump($_SERVER);
and check the HTTP_USER_AGENT string.
You can also achieve the same things by looking at the Apache logs.
But I am pretty sure curl is setting the User-Agent string like it should ;-)
You'll find the FF extension LiveHTTPHEaders will help you see exactly what happens to the headers when using a normal browsing session.
http://livehttpheaders.mozdev.org/
This will increase your understanding of how your target server responds, and even shows if it redirects your request internally.