Seem to be in a bit of a predicament. As far as I am aware, there have been no changed to PHP or Apache, however a code that has worked for almost 6 months just stoped working today at 2pm.
The code is:
function ls_record($prospectid,$campid){
$api_post = "method=NewProspect&prospect_id=".$prospectid."&campaign_id=".$campid;
$ch = curl_init();
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $api_post);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_URL, "http://XXXXX/XXXXXX/store.php");
$x = print_r(curl_exec($ch), TRUE);
return $x;
}
It returns NULL, I tried usingfile_get_contents()which also returnsNULL`. I checking the Apache error logs and see nothing...I need some help on this one.
Do you have access to the command line of the server? It could be that the destination has blocked you somehow.
If you have command line access, try this
wget http://XXXXX/XXXXXX/store.php
That should at least return something (if not headers)
use curl_getinfo to check your curl execution status, it maybe that the server you try to extract content from need your curl to set user-agent, some site check user-agent to block unwanted curl access.
below are the user agent I used to disguise my curl as desktop chrome browser.
curl_setopt($ch,CURLOPT_USERAGENT,' Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36');
I face same problem on my server , because of the low internet speed. Internet speed is go down for some time and curl take so many time to execute , so it return a timeout error . After a few minute it is working fine without any changes on server.
Related
I am trying to echo site data & for 95% of sites file_get_content, curl works just fine but for few sites, it never works whatever I tried. I tried to define proper user agent, changes SSL verify to false but nothing worked.
test site where it fails with forbidden https://norskbymiriams.dk/
wget is unable to copy ssl sites however wget is compiled with ssl support. checked with wget -V
i tried these codes.none worked for this particular site
file_get_contents
$list_url = "https://norskbymiriams.dk/";
$html = file_get_contents($list_url);
echo $html;
curl
$handle=curl_init('https://norskbymiriams.dk');
curl_setopt($handle, CURLOPT_HEADER, true);
curl_setopt($handle, CURLOPT_VERBOSE, true);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($handle, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($handle, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($handle, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36");
curl_setopt($handle, CURLOPT_FOLLOWLOCATION, true);
$content = curl_exec($handle);
echo $content;
any help will be great
Some websites analyse a request extremely good. If there is a single thing that makes that web server think you are a crawling bot, it might return 403.
I would try this:
make a request from browser, see all request headers, and place them in my curl request (simulate a real browser).
my curl request would look like this:
curl 'https://norskbymiriams.dk/'
-H 'Upgrade-Insecure-Requests: 1'
-H
'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100
Safari/537.36'
--compressed
Please try it. it works.
You can make a request in Chrome for example, and use Network tab from Developer tools to inspect a page request. If you right click on it, you will see Copy as cURL
Therefore test each header separately in your actual cURL request, see which is the missing link, then add it and continue your crawling.
I'm a novice programmer in PHP. Last week I read about cURL that capture my attention to study it. first, I copy and paste codes posted on different blogs and it goes run good like my code below.
<?php
$handle=curl_init('http://www.google.co.kr/');
curl_setopt($handle, CURLOPT_VERBOSE, true);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($handle, CURLOPT_SSL_VERIFYPEER, false);
$content = curl_exec($handle);
echo $content;
?>
BUT WHY I CAN'T cURL the website
http://www.todayhumor.co.kr/
since that, i am using same code above it outputs
Looking for your positive response guys. thank you in advance.
After calling curl_exec($handle) you should close the session with curl_close($handle). Maybe you tried so many times and now it doesn't work anymore, because you have so many open sessions on your local server. I would add that line to your code, restart xampp and try again.
Edit:
The server rejects requests without a valid user-agent. Add a user-agent to your request: curl_setopt($handle, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36'); this worked for me
I've got a really odd problem and no idea how to debug it. Maybe some experienced developer can help me. I've the following code:
$url = 'https://home.mobile.de/home/ses.html?customerId=471445&json=true&_='.time();
echo $url;
$agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36';
// Initiate curl
$ch = curl_init();
// Activate debugging
curl_setopt($ch, CURLOPT_VERBOSE, true);
// Disable SSL verification
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
// Will return the response, if false it print the response
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Set browser user agent
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
// Set the url
curl_setopt($ch, CURLOPT_URL,$url);
// Execute
$result=curl_exec($ch);
// Closing
curl_close($ch);
$php_object = json_decode($result);
var_dump($php_object);
I've put this code into a php file called playground.php. If I open playground.php with Chrome (I am using MAMP as local server) then everything works as expected. Also if I run on the osx command line "php playground.php" it works as expected, but for any reason it does not work if I run it inside the Phpstorm cli as shown below.
Any idea what could be wrong and how I can debug this issue?
Many thanks in advance.
Thanks to LazyOne I was able to find out that a firewall rule was blocking the outgoing request. Many thanks!
I'm writing a cURL script, but how can I check if it's working and passing properly when it's visiting the website?
$ckfile = '/tmp/cookies.txt';
$useragent= "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0_1 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7A400";
$ch = curl_init ("http://website.com");
curl_setopt($ch, CURLOPT_AUTOREFERER , true);
=> true
curl_setopt($ch, CURLOPT_USERAGENT, $useragent); // set user agent
curl_setopt ($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$output = curl_exec ($ch);
curl_close($ch);
just make a php page like this on your server and try your script on your own url
var_dump($_SERVER);
and check the HTTP_USER_AGENT string.
You can also achieve the same things by looking at the Apache logs.
But I am pretty sure curl is setting the User-Agent string like it should ;-)
You'll find the FF extension LiveHTTPHEaders will help you see exactly what happens to the headers when using a normal browsing session.
http://livehttpheaders.mozdev.org/
This will increase your understanding of how your target server responds, and even shows if it redirects your request internally.
$ckfile = tempnam ("/tmp", "CURLCOOKIE");
$useragent="Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16";
$ch = curl_init ("website.com");
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec ($ch);
The website sets a cookie and then redirects. Would this code suffice? Because it seems to be not getting the cookie properly. How can I check to see if it's set? Better yet, if I know what cookies I want can I just make it or something?
Edit: So my CURL script visits the website right? The website sets cookies for validation, and I want to see if my cURL script is receiving those cookies properly. I want to know if there's a test for that, and/or I want to know if I can just create a cookie to validate for the website.
I recently had a project where I needed to pass cookies between servers, and I found that setting both CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR to the same file did the trick.
If you're just wanting to see if the cookies from the remote site are being set, you should be able to look at the CURLOPT_COOKIEJAR file in a text editor.
If this were the command line curl you'd throw in the -v parameter. Pay attention to the lines starting with > Cookie:.
The php version of this is:
curl_setopt($ch,CURLOPT_VERBOSE,TRUE);//and set CURLOPT_STDERR to STDOUT
And if this were a browser I would debug with one of these: you can use Fiddler on IE, TamperData or Firebug (net) on Firefox, the inspector on Chrome and on Safari, or a proxy with any browser to watch how the form is submitted by a "normal" browser.
The proxy approach might work if you set php's curl to use it.
curl_setopt($ch,CURLOPT_HTTPPROXYTUNNEL,TRUE);//& CURLOPT_PROXYPORT CURLOPT_PROXY
If you want to just add cookies to one request you could write a line like:
$ch->headers[] = 'Cookie: recent=543..; _session_id=6185..; __utma=572.1.1.1.1; __utmc=572..; __utmz=572.1.1.1.1.utmccn=(referral)|utmcsr=domain.com|utmcct=/request/path|utmcmd=referral';
but I'm not certain about that one, these might be more along the same line:
curl_setopt($ch,CURLOPT_COOKIE,"recent=543..; _session_id=618..");
// Possibly, but I think this might overwrite other headers.
curl_setopt($ch,CURLOPT_HTTPHEADER, "Cookie: recent=543..; _session_id=6185..");