I'm sure this is fairly simple. I'm using the function below to retrieve sites raw html
in order to parse it. during my testing, I decided to run my code on stackoverflow.com
Instead of getting the html response the Chrome is printing out the actual site rather then assigning the html to its veritable. What am I missing?
function get_site_html($site_url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 4);
curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_URL, $site_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
global $base_url;
$base_url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
$http_response_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close ($ch);
return $response;
}
The site raw html should be assigned to $response, and then return it.
Your code works. Try echo htmlentities($response); You'll get the raw html for the site you're curling.
Related
I have written the following function, which was code I used somewhere else and modified slightly to work as a function (using $url in function parameters):
function curl2str($url) {
$cURL = curl_init($url);
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, true);
curl_setopt($cURL, CURLOPT_CONNECTTIMEOUT, 5);
$data = curl_exec($cURL);
curl_close($cURL);
return $data;
}
I simply want a function to return a URL into a string, for a quick and easy API. The URL I am passing it is valid and works fine when I put it into a browser. I am calling it like so:
<?=curl2str("**valid URL here**");?>
For some reason it is just returning false. What am I doing wrong?
update
When I put this questions URL into as $url, I get a response. But when I use my custom URL, which works fine in the browser, and simply display's a list of files in the directory, I get bool:false.
update 2
It would seem that any domain works fine, apart from the one that I am trying to access. It just so happens that this is a root domain on the same server, I am running this script from a subdomain, but because of basedir_restrictions I cannot access a folder from the subdomain. So I wrote a little php to get the contents of the folder, and output it to the browser as a serialized array (JSON is not installed). But I cannot get a response from this root domain at all. It works fine in the browser, just not in cURL. And everything else works fine in cURL.
:(
Try this code into your function:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
if(curl_errno($ch))
echo 'Curl error: '.curl_error($ch);
curl_close ($ch);
Note: curl_errno($ch); return error number>0 if any error occurs from cURL and use curl_error($ch); to see what is the error from cURL.
I use this function:
function curl($url, $cookie = false, $post = false, $header = false, $follow_location = false, $referer=false,$proxy=false)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_HEADER, $header);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $follow_location);
if ($cookie) {
curl_setopt ($ch, CURLOPT_COOKIE, $cookie);
}
if ($post) {
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
}
$response = curl_exec ($ch);
curl_close($ch);
return $response;
}
I'm using the below code to pull the html text from a site to publish on my own.
How can I manipulate the curl handle before echo?
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$returned = curl_exec($ch);
curl_close ($ch);
echo $returned;
You can manipulate your returned data:
$returned = curl_exec($ch);
curl_close ($ch);
$returned is the response so you can manipulate it
Did you mean something like this
Why am I not getting the HTML code when I use PHP curl? This is my code:
// $content = file_get_contents('http://www.datadiary.com/Company/311734/dimsinstituteofhotelmanagement');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.datadiary.com/Company/311734/dimsinstituteofhotelmanagement');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_COOKIESESSION, false);
$content = curl_exec($ch);
curl_close($ch);
echo $content;
By default curl_exec sends the response to the output (usually the browser). Set the CURLOPT_RETURNTRANSFER option if you want curl_exec to return the result instead:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
Relevant manual entries:
http://php.net/curl-exec
http://php.net/curl-setopt
You need to add the option to return the transfer as a string:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
I am using the following code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_TIMEOUT, 12);
$result = curl_exec($ch);
curl_close ($ch);
However it's printing the results straight away. Is it possible to put the JSON result into a variable so I can print it out when I want to?
Set CURLOPT_RETURNTRANSFER option:
// ...
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
Per the docs:
CURLOPT_RETURNTRANSFER - TRUE to return the transfer as a string of the return value of curl_exec() instead of outputting it out directly.
Have you tried?
curl_setopt($ch, CURLOPT_VERBOSE, 0);
This worked for me!
after php 5.1 curl will display always result you can view in documentation. for avoid it simply use
echo "< span style='display:none'>";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_TIMEOUT, 12);
$result = curl_exec($ch);
curl_close ($ch);
echo"< /span>";
I am using PHP curl method to get a string type response. To create the request I use:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, $data);
$response = curl_exec($ch);
$err = curl_error($ch);
curl_close($ch);
if($response === false)
throw new Exception(__CLASS__."::".__FUNCTION__."_".$err);
return $response;
Why I always receive a bool(true) response instead of the string I echo from the other side?
Thanks
Since you already have
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
in your code. curl_exec should already returns the content of the page instead of a BOOL.
This is a snippet of a library I use. As pointed out this might not be needed but it helped me out once...
//The content - if true, will not download the contents
curl_setopt($ch, CURLOPT_NOBODY, false);
Also it seems to have some bugs related to CURLOPT_NOBODY (which might explain why you have this issue):
http://osdir.com/ml/web.curl.general/2005-07/msg00073.html
http://curl.haxx.se/mail/curlphp-2008-03/0072.html