I'm trying to learn about creating web bots and I'm working my way through a book called Webbots, Spiders, and Screen Scrapers by Michael Schrenk. In the book he gives example code for a basic bot that downloads a webpage. I have copied the code exactly as it is in the book (sans comments):
<?
$target = "http://www.schrenk.com/nostarch/webbots/hello_world.html";
$downloaded_page_array = file($target);
for($xx=0; $xx<count($downloaded_page_array); $xx++)
echo $downloaded_page_array[$xx];
?>
I put this code in a php file and uploaded to my site. When I navigate to it in the browser however, nothing happens. It just loads a blank page. No content.
Earlier I tried another snippet that the author provided, again, this one was copied EXACTLY from the book, only with this one I didn't really get a blank page, the page just tried to load until it eventually timed out. Never got the correct content back:
$target = "http://www.schrenk.com/nostarch/webbots/hello_world.html";
$file_handle = fopen($target, "r");
while (!feof($file_handle))
echo fgets($file_handle, 4096);
fclose($file_handle);
I have checked the URL to make sure the file exists and it does. I have no idea why this wouldn't work. I've read through how to use the file(); and fopen(); functions in PHP but from what I can tell they are both being used correctly. What am I doing wrong here?
Accessing URLs via fopen() is a bad idea. It requires you to have allow_url_fopen enabled in your PHP config, which opens the door to a vast number of exploits (hosters disable it for a reason).
Try using cURL functions instead: they will give you much more flexibility and control. PHP documentation gives you some great examples to start with.
Not fgets($file_handle, 4096) but fread($file_handle, 4096) ;
$target = "http://www.schrenk.com/nostarch/webbots/hello_world.html";
$file_handle = fopen($target, "r");
while (!feof($file_handle))
echo fread($file_handle, 4096);
fclose($file_handle);
Then later if you want to create a new file from the extracted text :
// extracting text operation
$target = "http://www.schrenk.com/nostarch/webbots/hello_world.html";
$file_handle = fopen($target, "r");
$getText = fread($file_handle, 4096);
fclose($file_handle);
// writing file operation
$writeHandle = fopen ("folder/text.txt","w"); // file will be created if not existed
$writeFile = fwrite($writeHandle,$getText );
fclose($writeHandle );
First you should put error_reporting(E_ALL); ini_set('display_errors', '1'); to your script to enable displaying errors in your script as AbraCadaver mentioned in his comment.
A reason could be, that allow_url_fopen is disabled on your hosting.
This option enables the URL-aware fopen wrappers that enable accessing URL object like files. Default wrappers are provided for the access of remote files using the ftp or http protocol, some extensions like zlib may register additional wrappers.
See: http://php.net/manual/en/filesystem.configuration.php#ini.allow-url-fopen
You can check that via:
var_dump(ini_get('allow_url_fopen'));
Your script requires true to run correct.
If allow_url_fopen is not true or 1 you can try to use file_get_contents() to load a url.
<?php
$homepage = file_get_contents('http://www.example.com/');
echo $homepage;
?>
See: http://php.net/manual/en/function.file-get-contents.php
Related
I have a PHP code that connects to a FTP server, reads a file's content, makes a minor change, and than overrides the original file.
it looks like:
$stream_context = stream_context_create(array('ftp' => array('overwrite' => true)));
$file_content = file_get_contents($ftp_file); // this line works
$file_content = str_replace('some content', 'another content', $file_content); // this also..
file_put_contents($ftp_file, $file_content, 0, $stream_context); // this one doesn't :/
the real issue is that the "file_put_contents" worked for a long time, but now it doesn't.
what it does now is weird: it deletes the original file from the server..
also, if i'm changing it to something like:
file_put_contents($new_ftp_file, $file_content);
from what I know, it should create the new file and put the content in it, but it doesn't create it at all.
the hosting service i'm using has a PHP version change a few days ago. I don't remember the what the previous version was, but the current is: 5.2.17
thanks! :)
some changes
I found this piece: http://www.php.net/manual/en/function.file-put-contents.php#86864
doting the same as "file_put_contents" but with foen, fwrite and fclose (I send his example because of the results..). his functions is returning "false" if it couldn't to "fopen" the file, or the "bytes" if it succeeded. I got "false" :/
which means it couldn't even do the:
#fopen($filename, 'w');
although the "file_get_contents" with the same file address is working.
reading is working (but if you take the $filename and use it yourself on a client FTP - it works):
#fopen($filename, 'r');
the "open base_dir" for my hosting (which makes the action) is set to false, but the target hosting (which has the target-file) is set to be true.
I had an idea to save the new content on a new file, so I tried something like:
$f = #fopen($new_ftp_file, 'w'); //this one seems to work and connect
fwrite($f, $file_content); // this one seems to work either and returning the number of byes..
fclose($f);
the problem is that none of them really works. I logged in to the FTP address, using the same credentials that my script is using, and I haven't found the new file. It wasn't created at all. (as I remind you, "$new_ftp_file" is a path to a file that doesn't exists, so "w" mode on "fopen" should create it).
file_put_contents do erase the file by default if it already exists. You have to ask him to don't.
Look at here:
http://www.php.net/manual/en/function.file-put-contents.php
See: If filename does not exist, the file is created. Otherwise, the existing file is overwritten, unless the FILE_APPEND flag is set.
this is not the best solution (definitely), but I managed to figure something..
I used 2 different hosts on them same hosting service, both of the having "open base_dir = On" & "safe mode = Off".
it looks something like:
$ftpstring = "ftp://user:password#anotherhosteddomain.com";
$file = file_get_contents($ftpstring . 'index.html'); // this line works as expected. (yeah, the other hosting has this file);
and then, if you're trying to write something like:
$handler = fopen($ftpstring.'index.html', "w");
it wouldn't work, and tell you it cannot access on writing mode to an existing file.
so if you're doing something like:
$newfile_handler = fopen($ftpstring.'index_new_version.html', "w");
fwrite($newfile, "1122");
so yeah - it works!
but now is a tricky issue.. when i'm adding this line:
fclose($newfile_handler);
the new file is deleted from the hosting!!
I couldn't find any reason why "fclose" is deleting the file after it was create at "fopen" and written in at "fwrite".
so if you're not adding the "fclose" line - it works, but it doesn't close the connection, and also I have to actually delete the existing file before I can override it with a new content, which makes it silly..
although it works, I would really like someone to give me a better solution than mine.
I want to store some data retrieved using an API on my server. Specifically, these are .mp3 files of (free) learning tracks. I'm running into a problem though. The mp3 link returned from the request isn't to a straight .mp3 file, but rather makes an ADDITIONAL API call which normally would prompt you to download the mp3 file.
file_put_contents doesn't seem to like that. The mp3 file is empty.
Here's the code:
$id = $_POST['cid'];
$title = $_POST['title'];
if (!file_exists("tags/".$id."_".$title))
{
mkdir("tags/".$id."_".$title);
}
else
echo "Dir already exists";
file_put_contents("tags/{$id}_{$title}/all.mp3", fopen($_POST['all'], 'r'));
And here is an example of the second API I mentioned earlier:
http://www.barbershoptags.com/dbaction.php?action=DownloadFile&dbase=tags&id=31&fldname=AllParts
Is there some way to bypass this intermediate step? If there's no way to access the direct URL of the mp3, is there a way to redirect the file download prompt to my server?
Thank you in advance for your help!
EDIT
Here is the current snippet. I should be echoing something, correct?
$handle = fopen("http://www.barbershoptags.com/dbaction.php?action=DownloadFile&dbase=tags&id=31&fldname=AllParts", 'rb');
$contents = stream_get_contents($handle);
echo $contents;
Because this echos nothing.
SOLUTION
Ok, I guess file_get_contents is supposed to handle redirects just fine, but this wasn't happening. So I found this function: https://stackoverflow.com/a/4102293/2723783 to return the final redirect of the API. I plugged that URL into file_get_contents and volia!
You seem to be just opening the file handler and not getting the contents using fread() or another similar function:
http://www.php.net/manual/en/function.fread.php
$handle = fopen($_POST['all'], 'rb')
file_put_contents("tags/{$id}_{$title}/all.mp3", stream_get_contents($handle));
I'm writing a function in php, client side I have a canvas image which I use toDataUrl() along with a file name to save the image on the server. The here's the code:
<?php
$imageData=$GLOBALS['HTTP_RAW_POST_DATA'];
$data = json_decode($imageData, true);
$file = $data["file"];
$image = $data["data"];
$filteredData=substr($image, strpos($image, ",")+1);
$unencodedData=base64_decode($filteredData);
$fp = fopen( 'image/' . $file , 'wb' );
fwrite( $fp, $unencodedData);
fclose( $fp );
?>
The thing is that this code works. And for two out of three of the pages I used it on it works fine. The problem is when I copy and pasted it a third time to implement it again, for some reason the file is made on the server except that no data get's written into the file. I don't think it's a problem client side because I write in a debug alert message in the javascript and a debug echo into the PHP and both are able to print out the data fine. I made this short debug file:
<?php
$fp = fopen('data.txt', 'wb');
if(is_writable('data.txt')){
echo "file is writable<br>";
}
if(fwrite($fp, 'test') == FALSE){
echo "failed to write data<br>";
}
fclose($fp);
?>
And the output is
file is writable
failed to write data
I've tried using chmod and setting everything, the folder, the text file before I write to it to 0777 and I still get the same result; the file is made but no data is written into it. Is there anything I'm missing or any other approaches that might help. I haven't found anything on google and am still baffled as to why the same code worked exactly as expected twice before suddenly stopping for no apparent reason.
Thanks in advance.
I know this is an old post, but I had a very similar problem and found a solution (for me at least)! I ran out of disk space on my server, so it could create a 0 byte file, but wouldn't write to it. After I cleared out some space (deleted a 13gb error.log file) everything started working again as expected.
If fopen works but fwrite mysteriously doesn't, check your disk space. 'df -h' is the command to check disk space on a linux server.
instead of $fp = fopen('data.txt', 'wb'); give $fp = fopen('data.txt', 'w'); and try
Changed "wb" to "w"
When you write $fp = fopen('data.txt', 'w'); for your domain website.com having root at /var/www/website/ and if the php file is located at /var/www/website/php/server/file/admin.php or something similar, it will actually create a file at /var/www/website/data.txt
Try giving absolute path or path relative to your domain root to create files like,
$fp = fopen('php/server/file/data.txt', 'w');
Try the find command to see if the file is created anywhere else in the folder directory by using the following in Ubuntu,
find /var/www/website/ -name 'data.txt'
I had this issue, probably can help you solve if you have similar issue.
Which would be the best way to download a file from another domain in PHP?
i.e. A zip file.
The easiest one is file_get_contents(), a more advanced way would be with cURL for example. You can store the data to your harddrive with file_put_contents().
normally, the fopen functions work for remote files too, so you could do the following to circumvent the memory limit (but it's slower than file_get_contents)
<?php
$remote = fopen("http://www.example.com/file.zip", "rb");
$local = fopen("local_name_of_file.zip", 'w');
while (!feof($remote)) {
$content = fread($remote, 8192);
fwrite($local, $content);
}
fclose($local);
fclose($remote);
?>
copied from here: http://www.php.net/fread
You may use one code line to do this:
copy(URL, destination);
This function returns TRUE on success and FALSE on failure.
So yea, im working on a windows system and while this works locally, know it will break on other peoples servers. Whats a cross platform way to do the same as this
function fetch($get,$put){
file_put_contents($put,file_get_contents($get));
}
I don't see why that would fail unless the other computer is on PHP4. What you would need to do to make that backwards compatible is add functionality to provide replacements for file_get_contents & file_put_contents:
if(version_compare(phpversion(),'5','<')) {
function file_get_contents($file) {
// mimick functionality here
}
function file_put_contents($file,$data) {
// mimick functionality here
}
}
Here would be the solution using simple file operations:
<?php
$file = "http://www.domain.com/thisisthefileiwant.zip";
$hostfile = fopen($file, 'r');
$fh = fopen("thisisthenameofthefileiwantafterdownloading.zip", 'w');
while (!feof($hostfile)) {
$output = fread($hostfile, 8192);
fwrite($fh, $output);
}
fclose($hostfile);
fclose($fh);
?>
Ensure your directory has write permissions enabled. (CHMOD)
Therefore, a replacement for your fetch($get, $put) would be:
function fetch($get, $put) {
$hostfile = fopen($get, 'r');
$fh = fopen($put, 'w');
while (!feof($hostfile)) {
$output = fread($hostfile, 8192);
fwrite($fh, $output);
}
fclose($hostfile);
fclose($fh);
}
Hope it helped! =)
Cheers,
KrX
Shawn's answer is absolute correct, the only thing is that you need to make sure your $put varialable is a valid path on either the Windows Server on the Unix server.
well when i read your question I understood you wanted to bring a file from a remote server to your server locally, this can be done with the FTP extension from php
http://www.php.net/manual/en/function.ftp-fget.php
if this is not what you intent I believe what shawn says is correct
else tell me in the comments and i'll help you more
If the fopen wrappers are not enabled, the curl extension could be: http://php.net/curl