Copy file from url to my own server remains file to 0mb - php

I'm facing to a problem and I'm not really sure if this is the right way of doing this. I need to copy a file from a remote server to my server with php.
I use the following script :
public function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'w+b');
if (!$rh || !$wh) {
return false;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 4096)) === FALSE) {
return false;
}
echo ' ';
flush();
}
fclose($rh);
fclose($wh);
return true;
}
but in the end, the file size remains at 0.
EDIT : I update my question, because there are still some things I didn't understand :
About fread, I used 2048mb. But it didn't work.
I found the script above, which uses 4096mb.
My question : How to determine which quantity of memory (?) to use in order no get the file downloaded anytime ? Because this one works on a specific machine (dedicated), but will it on a shared host, if I cannot modify the php.ini ?
Thanks again

filesize() expects a filename/path. You're passing in a filehandle, which means filesize will FAIL and return a boolean false.
You then use that false as the size argument for your fread, which gets translated to an integer 0. So essentially you're sitting there telling php to read a file, 0 bytes at a time.
You cannot reliably get the size of a remote file anyways, so just have fread some fixed number of bytes, e.g. 2048, at a time.
while(!feof($handle)) {
$contents = fread($handle, 2048);
fwrite($f, $contents);
}
and if that file isn't too big and/or your PHP can handle it:
file_put_contents('local.mp4', file_get_contents('http://whatever/foo.mp4'));

Related

Characters being dropped retrieving files from simple php service using WinHttpReadData

I have a simple php service set up on a IIS web server. It is used by my client to retrieve files from the server. It looks like this:
<?php
if (isset($_GET['file']))
{
$filepath = "C:\\files\\" . $_GET['file'];
if (!strpos(pathinfo($filepath, PATHINFO_DIRNAME), "..") && file_exists($filepath) && !is_dir($filepath))
{
set_time_limit(0);
$fp = #fopen($filepath, "rb");
while(!feof($fp))
{
print(#fread($fp, 1024*8));
ob_flush();
flush();
}
}
else
{
echo "ERROR at www.testserver.com\r\n";
}
exit;
}
?>
I retrieve the files using WinHttp's WinHttpReadData in C++.
EDIT #2: Here is the C++ code. This is not exactly how it appears in my program. I had to pull pieces from multiple classes, but the gist should be apparent.
session = WinHttpOpen(appName.c_str(), WINHTTP_ACCESS_TYPE_NO_PROXY, WINHTTP_NO_PROXY_NAME, WINHTTP_NO_PROXY_BYPASS, 0);
if (session) connection = WinHttpConnect(session, hostName.c_str(), INTERNET_DEFAULT_HTTP_PORT, 0);
if (connection) request = WinHttpOpenRequest(connection, NULL, requestString.c_str(), NULL, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, 0);
bool results = false;
if (request)
{
results = (WinHttpSendRequest(request, WINHTTP_NO_ADDITIONAL_HEADERS, 0, WINHTTP_NO_REQUEST_DATA, 0, 0, 0) != FALSE);
}
if (results)
{
results = (WinHttpReceiveResponse(request, NULL) != FALSE);
}
DWORD bytesCopied = 0;
DWORD size = 0;
if (results)
{
do {
results = (WinHttpQueryDataAvailable(request, &size) != FALSE);
if (results)
{
// More available data?
if (size > 0)
{
// Read the Data.
size = min(bufferSize, size);
ZeroMemory(buffer, size);
results = (WinHttpReadData(request, (LPVOID)buffer, size, &bytesCopied) != FALSE);
}
}
if (bytesCopied > 0 && !SharedShutDown.GetValue())
{
tempFile.write((PCHAR)RequestBuffer, bytesCopied);
if (tempFile.fail())
{
tempFile.close();
return false;
}
fileBytes += bytesCopied;
}
} while (bytesCopied > 0 && !SharedShutDown.GetValue());
}
Everything works fine when I test (thousands of files) over the local network using the server computer name from either a Windows 7 or Windows 10 machine. It also works fine when I access the service over the internet from a Windows 7 machine. However, when I run the client on a Windows 10 machine accessing over the internet, I get dropped characters. The interesting thing is that it is a specific set of characters that gets dropped every time from XML files. (Other, binary, files are affected as well, but I have not yet determined what changes in them.)
If the XML file contains an element starting with "<Style", that text disappears. So, this:
<Element1>blah blah</Element1>
<Style_Element>hoopa hoopa</Style_Element>
<Element2>bip bop bam</Element2>
becomes this:
<Element1>blah blah</Element1>
_Element>hoopa hoopa</Style_Element>
<Element2>bip bop bam</Element2>
Notice that the beginning of the style element is chopped off. This is the only element that is affected, and it seems to only affect the first one if there are more than one in the file.
What perplexes me is why this doesn't happen running the client from Windows 7.
EDIT: Some of the other files, binary and text, are missing from 1 to 3 characters each. It seems that a drop only happens once in a file. The rest of the contents of the file are identical to the source.
I can't make sense of the above read routine, it is also incomplete. Just keep it simple like the example below.
The fact that you are having problems with binary files suggest you are not opening the output tempFile in binary mode.
std::ofstream tempFile(filename, std::ios::binary);
while(WinHttpQueryDataAvailable(request, &size) && size)
{
std::string buf(size, 0);
WinHttpReadData(request, &buf[0], size, &bytesCopied);
tempFile.write(buf.data(), bytesCopied);
}
Your php file can be simplified as follows:
<?php
readfile('whatever.bin');
?>
I solved the problem, it seems. My php service did not include header information (didn't think I needed it), so I figured I would try adding a header specification for content type application/octet-stream just to see what would result. My updated service looked like this:
if (isset($_GET['file']))
{
$filepath = "C:\\Program Files (Unrestricted)\\Sony Online Entertainment\\Everquest Yarko Client\\" . $_GET['file'];
if (!strpos(pathinfo($filepath, PATHINFO_DIRNAME), "..") && file_exists($filepath) && !is_dir($filepath))
{
header("Content-Type:application/octet-stream");
set_time_limit(0);
$fp = #fopen($filepath, "rb");
while(!feof($fp))
{
print(#fread($fp, 1024*8));
ob_flush();
flush();
}
}
else
{
echo "ERROR at www.lewiefitz.com\r\n";
}
exit;
}
Now, the files download without any corruption. Why I need such a header in this situation is beyond me. What part of the system is messing with the response message before it ended up in my buffer? I don't know.

Is there any way to speed this code up? Too many fopen()'s

foreach($streams as $stream) {
parse_str($stream, $data);
if(stripos($data['type'], $format) !== false && stripos($data['quality'], 'small') !== false) {
$video = fopen($data['url'] . '&signature='.$data['sig'], 'r');
$file = fopen($_GET['id'] . '.flv', 'w');
stream_copy_to_stream($video, $file);
fclose($video);
fclose($file);
echo echo_video($id);
exit;
}
}
I am making a YouTube downloader and for some reason, the conversion (of the smallest quality) is so small, that my server times out. Is there a way to replace these fopen()'s with file_put_contents()?
You can act as a URL proxy instead of first downloading the file then serving it. As you download and serve the data on the fly, you can also write the content on your local server in a file or as a BLOB in a database. Moreover, you can increase the timeout limit to avoid timing out.

PHP not writing to file from one source

I have an issue I can't seem to find the solution for. I am trying to write to a flat text file. I have echoed all variables out on the screen, verified permissions for the user (www-data) and just for grins set everything in the whole folder to 777 - all to no avail. Worst part is I can call on the same function from another file and it writes. I can't see to find the common thread here.....
function ReplaceAreaInFile($AreaStart, $AreaEnd, $File, $ReplaceWith){
$FileContents = GetFileAsString($File);
$Section = GetAreaFromFile($AreaStart, $AreaEnd, $FileContents, TRUE);
if(isset($Section)){
$SectionTop = $AreaStart."\n";
$SectionTop .= $ReplaceWith;
$NewContents = str_replace($Section, $SectionTop, $FileContents);
if (!$Handle = fopen($File, 'w')) {
return "Cannot open file ($File)";
exit;
}/*
if(!flock($Handle, LOCK_EX | LOCK_NB)) {
echo 'Unable to obtain file lock';
exit(-1);
}*/
if (fwrite($Handle, $NewContents) === FALSE) {
return "Cannot write to file ($File)";
exit;
}else{
return $NewContents;
}
}else{
return "<p align=\"center\">There was an issue saving your settings. Please try again. If the issue persists contact your provider.</p>";
}
}
Try with...
$Handle = fopen($File, 'w');
if ($Handle === false) {
die("Cannot open file ($File)");
}
$written = fwrite($Handle, $NewContents);
if ($written === false) {
die("Invalid arguments - could not write to file ($File)");
}
if ((strlen($NewContents) > 0) && ($written < strlen($NewContents))) {
die("There was a problem writing to $File - $written chars written");
}
fclose($Handle);
echo "Wrote $written bytes to $File\n"; // or log to a file
return $NewContents;
and also check for any problems in the error log. There should be something, assuming you've enabled error logging.
You need to check for number of characters written since in PHP fwrite behaves like this:
After having problems with fwrite() returning 0 in cases where one
would fully expect a return value of false, I took a look at the
source code for php's fwrite() itself. The function will only return
false if you pass in invalid arguments. Any other error, just as a
broken pipe or closed connection, will result in a return value of
less than strlen($string), in most cases 0.
Also, note that you might be writing to a file, but to a different file that you're expecting to write. Absolute paths might help with tracking this.
The final solution I ended up using for this:
function ReplaceAreaInFile($AreaStart, $AreaEnd, $File, $ReplaceWith){
$FileContents = GetFileAsString($File);
$Section = GetAreaFromFile($AreaStart, $AreaEnd, $FileContents, TRUE);
if(isset($Section)){
$SectionTop = $AreaStart."\n";
$SectionTop .= $ReplaceWith;
$NewContents = str_replace($Section, $SectionTop, $FileContents);
return $NewContents;
}else{
return "<p align=\"center\">There was an issue saving your settings.</p>";
}
}
function WriteNewConfigToFile($File2WriteName, $ContentsForFile){
file_put_contents($File2WriteName, $ContentsForFile, LOCK_EX);
}
I did end up using absolute file paths and had to check the permissions on the files. I had to make sure the www-data user in Apache was able to write to the files and was also the user running the script.

Trigger action when file download actually completes

Nowdays there are a lot of websites for files hosting (uploading websites) and it count for example point per complete download of certain file.
My question
I want to understand what is the idea they are using !
How does it only count on complete downloading of the file ?!
i mean if i canceled downloading of the file after it started , it won't count point!
how does it knew ! is there any php function that able to know if i canceled downloading certain exact file or not !
that question was all time in my mind and thinking about it but i can't understand how does it works or what is the idea behind it. ~ thanks
This can be done by using my other answer as base How can I give download access to files outside public_html directory? and replacing readfile( $filename )
with readfileWhileConnected( $filename ):
Read file until EOF or disconnect:
/** Read $filename until EOF or disconnect,
* if disconnect then error_log() count of bytes read already
*/
function readfileWhileConnected( $filename ) {
// Save and set ini values:
$user_abort = ignore_user_abort();
ignore_user_abort(false);
// Get file size and set bytes_sent to zero:
$fsize = filesize($filename);
$bytes_sent = 0;
// Open file:
$f = fopen($filename, 'r');
// Read file:
while($chunk = fread($f, 1024)) {
// Check if connection is still open:
if(!connection_aborted()) {
// Send $chunk to buffer (if any), then flush() buffers:
echo $chunk;
flush();
// Add $chunk length to $bytes_sent
$bytes_sent += strlen($chunk);
} else {
// Close file:
fclose($f);
error_log("Connection closed at $bytes_sent/$fsize");
exit();
}
// Close file:
fclose($f);
// Reset ini values:
ignore_user_abort($user_abort);
return $bytes_sent;
}
}
After you have your new shiny class myNewSuperDownloadHandlerClass { ... } ready, then make sure you only serve downloads through filedownload.php described here or if have done good myNewSuperDownloadHandlerClass(), then use that, just make sure that readfileWhileConnected() is used for every download requiring connection status polling.
You can easily add callback to be triggered if user closes connection, only 2 exit points here. (seen many functions that have every often return false; return true; return null; return false; return true; and so on..)

How to detect stream_copy_to_stream errors?

I've got a bit of code which, simplified, looks something like:
$fout = fsockopen($host, 80);
stream_set_timeout($fout, 10*3600); // 10 hours
$fin = fopen($file, 'rb'); // not really a file stream, but good enough proxy here
$readbytes = stream_copy_to_stream($fin, $fout);
if(!$readbytes) die('copy failed');
However, I'm sometimes getting the following type of error:
Notice: stream_copy_to_stream(): send of 952 bytes failed with errno=104 Connection reset by peer in ...
Notice: stream_copy_to_stream(): send of 952 bytes failed with errno=32 Broken pipe in ...
And the check on $readbytes there won't pick up the error.
I'm aware that it may be possible to check the total length of the file with the number of bytes copied, but this only works if the total length of the stream can be determined in advance.
As this happens randomly, I presume that the connection is just being dropped for some weird reason (but if anyone has any suggestions to reduce the likeliness of this happening, I'm all ears). But it'd be nice to be able to know whether the transfer fully succeeded or not.
Is there anyway to detect a problem without:
having to know the length of the stream in advance
hook into the PHP error handler to pick up the error
...or perhaps using a buffered fread/fwrite loop, checking the length of bytes written, the best solution here?
Thanks.
Okay, so here's my stream copy function, which tries to detect errors, but it seems to fail still (and I thought fwrite was meant to return the correct number of bytes)
function stream_copy($in, $out, $limit=null, $offset=null) {
$bufsize = 32*1024;
if(isset($offset)) fseek($in, $offset, SEEK_CUR);
while(!feof($in)) {
if(isset($limit)) {
if(!$limit) break;
$data = fread($in, min($limit,$bufsize));
} else
$data = fread($in, $bufsize);
$datalen = strlen($data);
$written = fwrite($out, $data);
if($written != $datalen) {
return false; // write failed
}
if(isset($limit)) $limit -= $datalen;
}
return true;
}
In above function, I'm still getting a 'true' returned even when an error is displayed.
So I'm just going to try hooking into PHP's error handler. Haven't tested the following, but my guess is that it should work
function stream_copy_to_stream_testerr($source, $dest) {
$args = func_get_args();
global $__stream_copy_to_stream_fine;
$__stream_copy_to_stream_fine = true;
set_error_handler('__stream_copy_to_stream_testerr');
call_user_func_array('stream_copy_to_stream', $args);
restore_error_handler();
return $__stream_copy_to_stream_fine;
}
function __stream_copy_to_stream_testerr() {
$GLOBALS['__stream_copy_to_stream_fine'] = false;
return true;
}
Ugly, but the only solution I can see.

Categories