i am not able to fetch the web page content of Google search page, but when i am using curl to get info from an API, it is working fine. i want to get Google page content so later on i can manually extract links from that page. i have tried many times, but unable to get web page content from any URL. Presently, i am working with XAMPP software to run PHP codes.
i am using curl_exec() command. http://www.php.net/manual/en/curl.examples-basic.php
is the source link for the code i tried.
Please also tell me how to use https:// in xampp software using curl command in php
Try like this
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.google.com");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // For HTTPS
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); // For HTTPS
$response=curl_exec($ch);
echo $response; // Google's HTML source will be printed
curl_close($ch);
The reason why you were not able to get the source is you were not using the CURLOPT_URL parameter instead it was CURLOPT_FILE.
Related
My php Curl request(also curl command in terminal) unfortunately shows a different code than opening the URL manually in the browser.
Here is my problem: I want to display the currently available films from https://reservierung.kinolambach.at/filmlist/?location_id=1 (german cinema)
on my website. When you check your network tab, the website sends an ajax call to a url with the specific date to the server and gets a response. Try it out by calling https://reservierung.kinolambach.at/filmlist/?location_id=1&date=4.1.2021&film_table=true in your browser(due to Covid there is a test film available on 4. Jan 2021)
My website does not run on the same domain.
Using an ajax call did not work because of CORS-Policy, so my second approach is to load the data via curl from php.
Here is my php code, which does not work. For every request I make I get the result "No film available on this day". It seems as if "No film available on this day" (german: 'Für diesen Tag ist kein Programm verfügbar.') is the "default" response.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://reservierung.kinolambach.at/filmlist/?location_id=1&date=4.1.2021&film_table=true');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HTTPGET, 1);
curl_setopt($ch, CURLOPT_COOKIESESSION, 0);
$result = curl_exec($ch);
echo($result);
curl_close($ch);
There are several curl_setopt which I tried out, none did work.
Here is my question
How is it possible, that curl shows a different result than calling the url in the browser? You can even try it in your terminal with curl https://reservierung.kinolambach.at/filmlist/?location_id=1&date=1.1.2021&film_table=true
How can I change this? I guess it has to do with the header.
Thanks in advance and all the best!
curl_setopt($ch, CURLOPT_URL, 'https://reservierung.kinolambach.at/filmlist/?
location_id=1&date=4.1.2021&film_table=true');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'C:/tmp/cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'C:/tmp/cookies.txt');
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false);
Works form me with cookies enabled. On second request as my comment before.
Currently I have page say page1.php. Now in a certain event, I just want to run another link say http://example.com without actually refreshing this page. The link is a kind of script which updates my database. I tried using shell_exec('php '.$url); where $url='http://example.com' however it showed me an error that could not open file so I suppose shell_exec works only for internal files present on the server. Is there a way to do this directly or I have to go with AJAX? Thanks in advance
Try using curl to send the request to the server with php.
$url = 'http://example.com';
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_NOBODY, TRUE);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_exec($ch);
curl_close($ch);
Alternatively you could try file_get_contents
file_get_contents('http://example.com');
I would do this front-end and I would use JSONP: much clean and safer IMHO.
I'm trying to parse a shopping site webpage using CURL in PHP.
the url is: http://computers.pricegrabber.com/printers/HP-Officejet-Pro-8600-Plus-All-One-Wireless-Inkjet-Printer/m916995235.html/zip_code=97045/sort_type=bottomline
Here's the code I use.
function getWebsiteCURL($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
echo getWebsiteCURL("http://computers.pricegrabber.com/printers/HP-Officejet-Pro-8600-Plus-All-One-Wireless-Inkjet-Printer/m916995235.html/zip_code=97045/sort_type=bottomline");
It works, but I couldn't get the full HTML code.
Anybody have any idea why?
TIA.
This is often caused by connection timeout.
Try setting the opt:
CURLOPT_TIMEOUT => 120
curl cannot interpret Javascript, you can see what curl will see if you disable javascript in a browser and navigate to the page. If you need to interpret Javascript then I would use a headless browser like phantomjs. In PHP you could use PHP PhantomJS.
I try to program a webboot using PHP/CURL, but I face a problem in handling a specific page that it's loading some contents dynamically !! .. to explain more :
when I try to download the page using PHP/CURL, I do not get some contents ! then I discovered that this contents are loaded after page is loaded. and this is why CURL does not handle these missed contents.
can any one help me !
my sample code is :
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $reffer);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $redirect);
curl_setopt($ch, CURLOPT_COOKIEFILE, ABSOLUTE_PATH."Cookies/cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, ABSOLUTE_PATH."Cookies/cookies.txt");
$result = curl_exec($ch);
What URL are you trying to load? It could be that the page you're requesting has one or more AJAX requests that load content in after the fact. I don't think that cURL can accomodate runtime-loaded information via AJAX or other XHR request.
You might want to look at something like PhantomJS, which is a headless WebKit browser which will execute the page fully and return the dynamically assembled DOM.
Because the page uses javascript to load the content, you are not going to be able to do this via cURL. Check out this page for more information on the problem: http://googlewebmastercentral.blogspot.com/2007/11/spiders-view-of-web-20.html
I have a site that uses cURL to access some pages, stores the returned results in variables, and then uses these variables within its own page. The script works well except where the target cURL page has a header('Location: ...') command inside it. It seems to just ignore this header command.
The cURL command is as follows...
//Load result page into variable so portions can be allocated to correct variables
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url); # URL to post to
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1 ); # return into a variable
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
$loaded_result = curl_exec( $ch ); # run!
curl_close($ch);
I've tried changing the CURLOPT_HEADER to 1 but it doesn't do anything.
So how can I allow script redirection within the target urls using cURL to grab the results? By the way, the pages work fine if accessed other than via cURL but iFrames are not an option in this instance.
If you want cURL to follow redirections add this:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE); // Follow redirects
You'll want the options CURLOPT_FOLLOWLOCATION and CURLOPT_MAXREDIRS. See the manual.
try
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);