How do I get the content of an xml file through a command line php script? If I access this link through the IE browser, I am able to get the XML file: http://alerts.weather.gov/cap/ma.php?x=0. If I try to get the file through command line with c:\path\php.exe get_advisory_upd.php, the script shows an error host did not respond in allowed time. This seems to be a security issue. I must have a scheduled task to get that xml at specified intervals. How do I do that? file_get_contents() returned that same error, simplexml_load_file() might not have shown any errors, but did not get the xml file.
PHP Script get_advisory_upd.php:
<?php
ini_set('display_errors', 1);
error_reporting(E_ALL);
$file = 'atom-advisory-MAZ015_UPD.txt';
$newfile = 'atom-advisory-MAZ015.txt.bak';
if (!copy($file, $newfile)) {
echo "Failed to copy $file...\n";
}
/* $contents = file_get_contents('http://alerts.weather.gov/cap/ma.php?x=0'); */
// Use cURL to get the RSS feed into a PHP string variable.
$ref_url = "http://192.x.x.x/weather/get_advisory_upd.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://alerts.weather.gov/cap/ma.php?x=0');
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_REFERER, $ref_url);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 1000);
$contents = curl_exec($ch);
echo 'Curl error: '. curl_error($ch);
curl_close($ch);
/* $contents = simplexml_load_file('http://alerts.weather.gov/cap/ma.php?x=0');
echo "contents \n".$contents; */
file_put_contents($file, $contents);
?>
UPDATE
I am running this script from an intranet. As y_a_v_a suggested I specified the CURLOPT_REFERER option to tell the remote host my url. I do that with
$ref_url = "http://192.x.x.x/weather/get_advisory_upd.php";
curl_setopt($ch, CURLOPT_REFERER, $ref_url);
Is there a different way to specify the URL?
Set a sane CURLOPT_REFERER and set CURLOPT_CONNECTTIMEOUT to zero and run your script. Verify what happens by adding
$curl_error = curl_error($ch);
and dump $curl_error after you closed the curl action.
Related
SO, I have been fighting with a piece of code that I want to use to get a remote page's source code using curl.
The code executes successfully, both in the browser and on command line. However, I get the of the main file only. When parameters are added, they are not considered whatsoever in the output.
The Code:
STACK : Ubuntu, Nginx, PHP-FPM 7.2
$urlcontent = 'https://XXX.YYY.COM/file/?var1=value1' ;
// Create a new cURL resource
$curl = curl_init();
if (!$curl) {
die("Couldn't initialize a cURL handle");
}
// Set the file URL to fetch through cURL
curl_setopt($curl, CURLOPT_URL, $urlcontent);
// Set a different user agent string (Googlebot)
curl_setopt($curl, CURLOPT_USERAGENT, 'CodiBot/2.1');
// Follow redirects, if any
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
// Fail the cURL request if response code = 400 (like 404 errors)
curl_setopt($curl, CURLOPT_FAILONERROR, true);
// Return the actual result of the curl result instead of success code
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
// Wait for 10 seconds to connect, set 0 to wait indefinitely
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
// Execute the cURL request for a maximum of 50 seconds
curl_setopt($curl, CURLOPT_TIMEOUT, 50);
// Do not check the SSL certificates
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
// Fetch the URL and save the content in $html variable
$html = curl_exec($curl);
// Check if any error has occurred
if (curl_errno($curl))
{
echo 'cURL error: ' . curl_error($curl);
}
else
{
// cURL executed successfully
print_r(curl_getinfo($curl));
print_r($html);
}
curl_close($curl);
PROBLEM
I get the content for https://XXX.YYY.COM/file but not the corresponding ?var1=value1 part. IN other words, as I feed info to be retrieved to DB I get only the html of the main file.
I tried :
curl_setopt($ch, CURLOPT_POSTFIELDS, 'foo=1&bar=2&baz=3');
I know the remote server may have CORS enabled, but I tried the same url using a remote curl retriever and it succeeded. SO, it may not be the remote server
I am doing a system where one of my sites goes to the other to get documents.
On the first site I am using Curl to make a request to get the file wanted:
I am using the solution from Download file from URL using CURL :
function collect_file($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_AUTOREFERER, false);
curl_setopt($ch, CURLOPT_REFERER, "http://example.com");
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
return($result);
}
function write_to_file($text,$new_filename){
$fp = fopen($new_filename, 'w');
fwrite($fp, $text);
fclose($fp);
}
$curlUrl = 'http://site2.com/file-depository/14R4NP8JkoIHwIyjnexSUmyJibdpHs5ZpFs3NLFCxcs54kNhHj';
$new_file_name = "testfile-new.png";
$temp_file_contents = collect_file($curlUrl);
write_to_file($temp_file_contents,$new_file_name);
I am testing downloading an image. If i use a direct URL into $curlUrl , for instance http://site2.com/file-depository/image.png it works perfect.
What I am doing is that the URL http://site2.com/file-depository/14R4NP8JkoIHwIyjnexSUmyJibdpHs5ZpFs3NLFCxcs54kNhHj is then parsed and checked against a database to match the document requested, once there is a document matched I need to provide this document to the Curl response.
I have tried many ways to read the file but everytime i am getting a file on the other end but it is only 1kb in size (45 expected) and when trying to open it i get an error unkown file type etc.
On the second site, once the URL is validated here is what I have:
$file = readfile('some-image.png');
echo $file;
I am guessing there is part of the information which belongs to the file missing but can't figure it out, any pointers appreciated!
I have replaced
function write_to_file($text,$new_filename){
$fp = fopen($new_filename, 'w');
fwrite($fp, $text);
fclose($fp);
}
by file_put_contents($new_file_name,trim($temp_file_contents));
Please note the trim(), the issue was that I was apparently collecting some empty space in front of the file content.
I am trying to display content from a txt file on a remote server using a cURL if the function file_get_contents is not turned on on their server. Right now, I have it set up as:
$updatelog ='http://www.linktoremotsite.com/updatelog.txt';
if(function_exists('file_get_contents')){
$log = file_get_contents($updatelog);
}else{
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $updatelog);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$log = curl_exec($ch);
curl_close($ch);
}
echo '<div>' . $log . '</div>';
and the file_get_contents works fine, but for those that don't have that enabled on their site, I like them to be able to use a curl to see the contents of the .txt file. But I can't seem to get the contents of the .txt file to display.
The .txt file does have some HTML in it for the file_get_contents to display neatly, but not much and I like to keep that with the curl if possible.
I am using a local server on my computer and i am trying to make 2 php scripts to send an xml file and receive it.
To send the xml file i use this code :
<?php
/*
* XML Sender/Client.
*/
// Get our XML. You can declare it here or even load a file.
$file = 'http://localhost/iPM/books.xml';
if(!$xml_builder = simplexml_load_file($file))
exit('Failed to open '.$file);
// We send XML via CURL using POST with a http header of text/xml.
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://localhost/iPM/receiver.php");
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $xml_builder);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_REFERER, 'http://localhost/iPM/receiver.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$ch_result = curl_exec($ch);
curl_close($ch);
// Print CURL result.
echo $ch_result;
?>
To receive the xml file i use this code :
<?php
/*
* XML Server.
*/
// We use php://input to get the raw $_POST results.
$xml_post = file_get_contents('php://input');
// If we receive data, save it.
if ($xml_post) {
$xml_file = 'received_xml_' . date('Y_m_d-H-i-s') . '.xml';
$fh = fopen($xml_file, 'w') or die();
fwrite($fh, $xml_post);
fclose($fh);
// Return, as we don't want to cause a loop by processing the code below.
return;
}
?>
When i run the post script i get this error :
Notice: Array to string conversion in C:\xampp\htdocs\iPM\main.php on line 17
which refers to line :
curl_setopt($ch, CURLOPT_POSTFIELDS, $xml_builder);
which i dont know what exactly does. The xml file i receive is created but when i open it i get this :
XML Parsing Error: syntax error
Location: file:///C:/xampp/htdocs/iPM/received_xml_2013_01_14-01-06-09.xml
Line Number 1, Column 1:
I tried to comment this specific line as i thought the problem lies there but then when i run my post script i get this error :
Request entity too large!
The POST method does not allow the data transmitted, or the data volume exceeds the capacity limit.
If you think this is a server error, please contact the webmaster.
Error 413
but the xml file is only 5kbs so this is not the problem.
Does anyone have any idea what i should do here? All i am trying to do is make a script to send an xml file and a script to receive it and save it as an xml.
curl_setopt($ch, CURLOPT_POSTFIELDS, $foo) sets your request's body, the data to be posted. It expects $foo to be a set of key-value pairs provided either as an array:
$foo = array(
'foo' => 'some value',
'bar' => 2
);
or as a percent-encoded string:
$foo = 'foo=some%20value&bar=2'
Instead, you're providing $xml_builder variable which is a SimpleXMLElement object returned by simplexml_load_file($file).
Try this:
$postfields = array(
'xml' => $your_xml_as_string; // get it with file_get_contents() for example
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
Then on the receiving end:
$received_xml = $_POST['xml'];
I'm trying to save a local copy of an xml file, and then open it with simple xml, but i'm getting some errors.. here's my code:
$feedURL = "https://gdata.youtube.com/feeds/api/users/manitobachildhealth/favorites";
//$xml = file_get_contents("$feedURL");
$xml = file_get_contents($feedURL);
file_put_contents("video.xml", $xml);
// read feed into SimpleXML object
//$sxml = simplexml_load_file($feedURL);
$sxml = simplexml_load_file('video.xml');
The error i'm getting is as follows:
Warning: file_get_contents(https://gdata.youtube.com/feeds/api/users/manitobachildhealth/favorites) [function.file-get-contents]: failed to open stream: Result too large in D:\wamp\www\videos2.php on line 48
I'm not sure why it would be too large of a result, it only returns 6kb of xml. what am i doing wrong?
Update:
This is running on a windows platform using WAMP server - not ideal, but i'm stuck with it.
Update 2:
I've tried using curl and fwrite to achieve a similar result, as suggested below, but it won't write the xml file to the local server. It doesn't give me any errors though.
update 3:
This is obviously a very specific problem with the hosting environment, but I'm not sure where to start looking for the problem. Using curl works great on a linux-based dev server, but is causing problems on this windows-based production server. An extra help in troubleshooting this issue would be most appreciated!
Correct answer for the question:
It is possible you are having the same problem as of this question: CURL and HTTPS, "Cannot resolve host" (DNS-Issue)
Other Details:
You can use SimpleXML to load and save the xml data
$xml = new SimpleXMLElement('https://gdata.youtube.com/feeds/api/users/manitobachildhealth/favorites', NULL, TRUE);
$xml->asXML('video.xml');
I have tested the code above in a WAMP server and it works fine.
Update:
If the above returns error message "[simplexmlelement.--construct]: I/O warning : failed to load external entity ...." It's possible that your server does not allow to include external data or the php file/script does not have the right permission.
Try the following:
1. echo the content of the xml file.
$xml = new SimpleXMLElement('https://gdata.youtube.com/feeds/api/users/manitobachildhealth/favorites', NULL, TRUE);
echo htmlentities($xml->asXML());
If you managed to retrieved the xml content and print it to the browser, then your server is allowing to include external content and most likely the problem with the file permission. Make sure file/script have the right to create xml file.
If the above still does not work try using cURL.
function getPageContent($options)
{
$default = array(
'agent' => $_SERVER['HTTP_USER_AGENT'],
'url' => '',
'referer' => 'http://'.$_SERVER['HTTP_HOST'],
'header' => 0,
'timeout' => 5,
'user' => '',
'proxy' => '',
);
$options = array_merge($default, $options);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $options['url']);
curl_setopt($ch, CURLOPT_HEADER, $options['header']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
if ($options['proxy'] != '') {
curl_setopt($ch, CURLOPT_PROXY, $options['proxy']);
}
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $options['timeout']);
curl_setopt($ch, CURLOPT_REFERER, $options['referer']);
curl_setopt($ch, CURLOPT_USERAGENT, $options['agent']);
if ($options['user'] != '') {
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $options['user']);
}
$result = array();
$result['content'] = curl_exec($ch);
$result['info'] = curl_getinfo($ch);
$result['error'] = curl_error($ch);
curl_close($ch);
return $result;
}
$result = getPageContent(array(
'proxy' => '[ip or address]:[port]', // if needed
'user' => '[username]:[password]', // if needed
'url' => 'http://gdata.youtube.com/feeds/api/users/manitobachildhealth/favorites'
));
if (empty($result['error'])) {
// ok
// content of xml file
echo htmlentities($result['content']);
// file
$filename = 'video.xml';
// Open File
if (!$fp = fopen($filename, 'wt')) {
die("Unable to open '$filename'\n\n");
}
// write content to file
fwrite($fp, $result['content']);
// close file
fclose($fp);
} else {
// failed
echo '<pre>';
echo 'Error details;';
print_r ($result['error']);
echo '<hr />Other info:';
print_r ($result['info']);
echo '</pre>';
}
Have you tried using curl and fwrite to get the contents and write them to a local file?
$ch = curl_init("https://gdata.youtube.com/feeds/api/users/manitobachildhealth/favorites");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
fwrite("video.xml",$output);