I am trying to make a simple web crawler with PHP and I am having issues getting the HTML source of a given URL. I am currently using cURL to get the source.
My code:
$url = "http://www.nytimes.com/";
function url_get_contents($Url) {
if (!function_exists('curl_init')) {
die('CURL is not installed!');
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
if ($output === false) { die(curl_error($ch)); }
curl_close($ch);
return $output;
}
echo url_get_contents($url);
?>
Right now nothing gets echoed and there aren't any errors, so it is a bit of a mystery. Any suggestions or fixes will be appreciated
Edit: I added
if ($output === false) { die(curl_error($ch)); }
to the middle of the function and it ended up giving me an error (finally!):
Could not resolve host: www.nytimes.com
I still do not really know what the problem is. Any ideas?
Thanks
Turns out that it was not a cURL problem
My host server (Ubuntu VM) was working off of a "host-only" network adapter which blocked access to all other IPs or domains outside of it's host machine making it impossible for cURL to connect to URLs.
Once it was changed to "bridged" network adapter I had access to the outside world.
Hope this helps.
Variable case mismatch ($url vs. $Url). Change:
function url_get_contents($Url) {
to
function url_get_contents($url) {
Related
[![enter image description here][1]][1]I am trying to get the some tag value but it's showing some error.
Below is the code, please suggest some solution.
This is the method i used for httpGet request.
function httpGet($result15)
{
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$result15);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
$output=curl_exec($ch);
curl_close($ch);
return $output;
}
$result15= httpGet("https://www.googleapis.com/customsearch/v1?key=API_KEY&cx=003255er&q=cancer&num=1&alt=atom");//new cse
echo $result15;
$xml = new DOMDocument();
$xml->loadXML($result15);
foreach( $xml->entry as $entry )
{
echo "URL=".(string)$entry->id.PHP_EOL;
echo "Summary=".(string)$entry->summary.PHP_EOL;
}
You might find the curl request is failing. You need to do a couple of things...
function httpGet($result15)
{
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$result15);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // Add this
$output=curl_exec($ch);
// If this fails, output error.
if ($output === FALSE) {
echo curl_error($ch);
// Not sure what you want to do, but 'exit' will work for now
exit();
}
curl_close($ch);
return $output;
}
This will display an error if the curl request fails. You will need to decide how your going to cope with this. You could return false, and then in your code further down, check this before trying to load it as XML. The code above just stops on errors.
Your next piece of code seems to mix SimpleXML and DOMDocument, you can use SimpleXML if the document structure is fairly straight forward...
$xml = simplexml_load_string($result15);
foreach( $xml->entry as $entry )
{
I have a web-service, which is deployed on a server. The web-service is working perfectly. Now I want to do is to deploy the same web-service on another server. And then at my client site I want to check that If any of the server is running than the call is made.
I want to do something like that
$Ip1= "192.168.1.1/GetSomeData";
$Ip2= "202.47.22.1/GetSomeDate";
Now I want to check the Ip1 whether it is running or not
if(Ip1=="running")
{
//call the web-service
}//if the Ip1 is not working
else if (Ip2=="running")
{
//call the web-service
}
else
{
//do nothing
}
How can i achieve that in Yii2?
Any help would be highly appreciated.
If web-service is under your control you could make an echo method ans simply check if it echoes back an answer to you using normal web-service call.
On the other hand you could use curl to check for an existing file or service on your web-server as in this post:
function isRunning($url=NULL)
{
if($url == NULL) return false;
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
return ($httpcode>=200 && $httpcode<300) ? true : false;
}
Of course then your code looks something like this:
if(isRunning(ipAddress1))
{
//call the web-service
}//if the Ip1 is not working
else if (isRunning(ipAddress2))
{
//call the web-service
}
else
{
//do nothing
}
Where ipAddress is a file or path on that server. There are also a lot of other ways. You could use fsockopen if you have an open port, or use shell_exec to fetch ping result....
Hope my answer helped.
I've got the OneNote API PHP Sample (thanks jamescro!) working with all the POST examples, but there's no GET example and I haven't managed to put together code of my own that works. Here's what I've tried without success:
// Use page ID returned by POST
$pageID = '/0-1bf269c43a694dd3aaa7229631469712!93-240BD74C83900C17!600';
$initUrl = URL . $pageID;
$cookieValues = parseQueryString(#$_COOKIE['wl_auth']);
$encodedAccessToken = rawurlencode(#$cookieValues['access_token']);
$ch = curl_init($initUrl);
curl_setopt($ch, CURLOPT_URL, $initUrl); // Set URL to download
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
if (! $response === false) {
curl_close($ch);
echo '<i>Response</i>: '. htmlspecialchars($response);
}
else {
$info = curl_getinfo($ch);
curl_close($ch);
echo '<i>Error</i>: ';
echo var_export($info);
}
It just returns 'Error' with an info dump. What am I doing wrong?
without information on the specific error I'm not sure what issue you are hitting. Try looking at the PHP Wordpress plugin here: https://github.com/wp-plugins/onenote-publisher/blob/master/api-proxy.php
look at what is sent to wp_remote_get - there are necessary headers that are needed.
Also make sure you have the scope "office.onenote" when you request the access token.
If you need more help, please add information about the specific URL you are attempting to call, as well as the contents of your headers. If you have any errors, please include the output.
Solved:
As Jay Ongg pointed out, "there are necessary headers that are needed".
After adding more detailed error checking and getting a 401 response code, I added:
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type:text/html\r\n".
"Authorization: Bearer ".$encodedAccessToken));
... and could access the requested page.
I want to show contents of a remote file (a file on another server) on my website.
I used the following code, readfile() function is working fine on the current server
<?php
echo readfile("editor.php");
But when I tried to get a remote file
<?php
echo readfile("http://example.com/php_editor.php");
It showed the following error :
301 moved
The document has moved here 224
I am getting this error remote files only, local files are showing with no problem.
Is there anyway to fix this?
Thanks!
Option 1 - Curl
Use CURL and set the CURLOPT_FOLLOWLOCATION-option to true:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http//example.com");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
if(curl_exec($ch) === FALSE) {
echo "Error: " . curl_error($ch);
} else {
echo curl_exec($ch);
}
curl_close($ch);
?>
Option 2 - file_get_contents
According to the PHP Documentation file_get_contents() will follow up to 20 redirects as default. Therefore you could use that function. On failure, file_get_contents() will return FALSE and otherwise it will return the entire file.
<?php
$string = file_get_contents("http://www.example.com");
if($string === FALSE) {
echo "Could not read the file.";
} else {
echo $string;
}
?>
I have this piece of code below which works fine on my remote hosted server, but isnt for some reason working on my local linux machine. Ive tried using file_get_contents as well to get the restful service but it also returns false.
Does anyone know Why this is happening?
thanks :)
$xml_data = simplexml_load_file("****");
if ($xml == FALSE)
{
echo "Failed loading XML\n";
foreach (libxml_get_errors() as $error)
{
echo "\t", $error->message;
}
}
You are getting this error because remote file access has been disabled on your server. An alternative to this is using CURL.
Use my code below to use CURL:
function produce_XML_object_tree($raw_XML) {
libxml_use_internal_errors(true);
try {
$xmlTree = new SimpleXMLElement($raw_XML);
} catch (Exception $e) {
// Something went wrong.
$error_message = 'SimpleXMLElement threw an exception.';
foreach(libxml_get_errors() as $error_line) {
$error_message .= "\t" . $error_line->message;
}
trigger_error($error_message);
return false;
}
return $xmlTree;
}
$xml_feed_url = '******';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $xml_feed_url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$xml = curl_exec($ch);
curl_close($ch);
$cont = produce_XML_object_tree($xml);
Now use $cont as an object to access different nodes in the xml.
Make sure you have allow_url_fopen turned on in your php.ini
http://php.net/manual/filesystem.configuration.php
Well I had same issue and though I would post this to assist anyone who may have not tried this solution yet.
I had a PHP script which worked fine locally, but when using it on a client server running plesk it would not work and failed when trying to grab the external xml file.
I was trying to reference an external xml file from a php script. The server I was using was running plesk. Before considering changing host, All I simply did was update the settings for PHP on the server to run as an Apache Module instead of FastCGI.
error message which I was receiving (example):
Warning: simplexml_load_file(url) [function.simplexml-load-file]: failed to open stream: Permission denied
This resolved the issue in my case.
I used following reports settings in the PHP script:
assert_options(ASSERT_ACTIVE, 1);
assert_options(ASSERT_BAIL, 1);
assert_options(ASSERT_QUIET_EVAL, 1);
error_reporting(E_ALL);
ini_set('display_errors', 1);
use like this
$xml = simplexml_load_file('http://localhost/test/123.xml');
foreach ($xml->children() as $child) {
$remoteCount[$child->getName()] = $child;
}
var_dump($remoteCount);
Change: if ($xml == FALSE) to if ($xml === FALSE) (source).
I had the same problem it's just a stupid undeclared point in the simplexml
the xml file format should have a container tag, so, you just have to put a parent tag containing all your data like this:
<?xml version="1.0">
<data>
...all your file content here...
</data>
In my case, it's missing the XML php library, reinstall it and works fine
https://wpml.org/forums/topic/fatal-error-uncaught-error-call-to-undefined-function-simplexml_load_file-3/