It seems I successfully create an on-the-fly zip archive (filesize and file_exists both return the expected values) but when I attempt to actually download it, I receive an empty ZIP file.
Curiously enough this error occurs with both readfile and fread. This is my code
$filename = $zip;
$handle = fopen($filename, 'r');
if ($handle === false)
{
die('Could not read file "' . $filename . '"');
}
header('Content-type: application/zip');
header('Content-Disposition: attachment; filename="fsdownload.zip"');
header('Cache-Control: private');
while (!feof($handle))
{
echo fread($handle, 8192);
}
fclose($handle);
This works fine for zip-Files < 10 MB. Any thoughts on what the problem might be?
To avoid consuming too much memory, you can use ZipStream or PHPZip, which will send zipped files on the fly to the browser, divided in chunks, instead of loading the entire content in PHP and then sending the zip file.
Both libraries are nice and useful pieces of code. A few details:
ZipStream "works" only with memory, but cannot be easily ported to PHP 4 if necessary (uses hash_file())
PHPZip writes temporary files on disk (consumes as much disk space as the biggest file to add in the zip), but can be easily adapted for PHP 4 if necessary.
Related SO questions:
Generating ZIP files with PHP + Apache on-the-fly in high speed?
Create a zip file using PHP class ZipArchive without writing the file to disk?
The problem is a PHP limit. Check for memory and execution time limits.
Related
I'm using the PHP Flysystem package to stream content from my Amazon S3 bucket. In particular, I'm using $filesystem->readStream.
My Question
When I stream a file, it ends up in myzip.zip and the size is correct, but when unzip it, it become myzip.zip.cpgz. Here is my prototype:
header('Pragma: no-cache');
header('Content-Description: File Download');
header('Content-disposition: attachment; filename="myZip.zip"');
header('Content-Type: application/octet-stream');
header('Content-Transfer-Encoding: binary');
$s3 = Storage::disk('s3'); // Laravel Syntax
echo $s3->readStream('directory/file.jpg');
What am I doing wrong?
Side Question
When I stream a file like this, does it:
get fully downloaded into my server's RAM, then get transferred to the client, or
does it get saved - in chunks - in the buffer, and then get transferred to the client?
Basically, is my server being burdened if I have have dozens of GB's of data being streamed?
You are currently dumping the raw contents of the directory/file.jpg as the zip (which a jpg is not a zip) . You need to create a zip file with those contents.
Instead of
echo $s3->readStream('directory/file.jpg');
Try the following in its place using the Zip extension:
// use a temporary file to store the Zip file
$zipFile = tmpfile();
$zipPath = stream_get_meta_data($zipFile)['uri'];
$jpgFile = tmpfile();
$jpgPath = stream_get_meta_data($jpgFile)['uri'];
// Download the file to disk
stream_copy_to_stream($s3->readStream('directory/file.jpg'), $jpgFile);
// Create the zip file with the file and its contents
$zip = new ZipArchive();
$zip->open($zipPath);
$zip->addFile($jpgPath, 'file.jpg');
$zip->close();
// export the contents of the zip
readfile($zipPath);
Using tmpfile and stream_copy_to_stream, it will download it in chunks to a temporary file on disk and not into RAM
I want to upload a file to an FTP server, but the file content is held in a variable, not in an actual local file. I want to avoid using a file; this is to avoid security risks when dealing with sensitive data on a (possibly) not-so-secure system(*), as well as to minimize the (already low) overhead of file handling.
But PHP's FTP API only offers uploading files from local files via the function ftp_put or (when the file is already opened as a file handle) ftp_fput.
Currently, I use this function with a temporary file in which I write the contents before the upload:
$tmpfile = tmpfile();
fwrite($tmpfile, $content);
fseek($tmpfile, 0);
ftp_fput($ftp, $filename, $tmpfile, FTP_BINARY);
Is there a simpler way without using files on the local (PHP) site at all?
There is ftp_raw which can be used to send arbitrary commands, so I could issue the PUT command manually, however I don't see a way to manually write the data on the data channel...
I don't know if it is important, but the FTP connection is secured with SSL (ftp_ssl_connect).
(*) Consider the scenario where an attacker has read-only control over the entire file system.
This may be no ultimate solution but I guess is still better than the original approach:
You can avoid temporary files on the file system by using a PHP memory stream. It basically is a file handle wrapper which (behind the scenes) uses no actual file but instead some chunk of memory.
So virtually you still use a file handle (so ftp_fput is happy), but no actual file (so no file is written to the disk and the overhead is kept minimal).
$tmpfile = fopen('php://memory', 'r+');
fputs($tmpfile, $content);
rewind($tmpfile); // or fseek
Note that when uploading multiple files, you can further minimize overhead by reusing the same file handle for all files (unless you parallelize the procedure, of course). In this case, also rewind the file after ftp_fput as well as truncate it using ftruncate($tmpfile, 0).
With no local file involved (ftp_fput):
$stream = fopen('php://memory','r+');
fwrite($stream, $newFileContent);
rewind($stream);
$success = ftp_fput($connectionId, "remoteFileName", $stream, FTP_BINARY);
fclose($stream);
fopen should be able to connect via ftp:
http://php.net/manual/en/function.fopen.php
and then use fwrite to write string to connection:
http://php.net/manual/en/function.fwrite.php
Second parameter of fwrite is string that need to be written - you don't need file there.
-----FTP PUT contents (php 7.0) ---
$tmpFile = tmpfile();
fwrite($tmpFile, $contents);
rewind($tmpFile);
$tmpMetaData = stream_get_meta_data($tmpFile);
if (ftp_put($ftpObj, $remoteFile, $tmpMetaData['uri'], FTP_ASCII)) {
echo "success";
} else {
echo "fail";
}
fclose($tmpFile);
I want to allow users to download large video files. These files are outside of the public folder because of security reasons.
I'm using a combination of fopen(), feof(), and fread() to download the file in chuncks.
The download works fine. The video is downloaded and also works just fine. The problem is during the download. Any user who's downloading the file can't continue browsing the site until the file is downloaded. The browser is trying to establish a connection, but it hangs while the file is downloading. When the download is done, the connection is immediately established. Other users can browse the site just fine during the download, so it's not like the whole server hangs or whatever.
I'm working with PHP (CakePHP) installed on an IIS server.
A snippet of code:
$name = "filename.mp4";
$folder = "private/folder/";
$handle = fopen($folder.$name, "rb");
if(!$handle)
{
echo "File not found";
}
else
{
header("Content-length:".filesize($folder.$name));
header("Content-Type: video/mp4");
header("Content-Disposition: attachment; filename='filename.mp4'");
header("Content-Transfer-Encoding: binary");
session_write_close(); // this is the solution
while(!feof($handle))
{
$buffer = fread($handle, 1*(1024*1024));
echo $buffer;
ob_flush();
flush();
}
}
I finally solved the problem. As suggested above, the problem was indeed related to sessions. Even though session.auto_start was off, CakePHP itself was handling sessions at the moment. So, by inserting session_write_close() right before the while loop, the problem was solved.
I'm a novice, so I'll try and do my best to explain a problem I'm having. I apologize in advance if there's something I left out or is unclear.
I'm serving an 81MB zip file outside my root directory to people who are validated beforehand. I've been getting reports of corrupted downloads or an inability to complete the download. I've verified this happening on my machine if I simulate a slow connection.
I'm on shared hosting running Apache-Coyote/1.1.
I get a network timeout error. I think my host might be doing killing the downloads if they take too long, but they haven't verified either way.
I thought I was maybe running into a memory limit or time limit, so my host installed the apache module XSendFile. My headers in the file that handles the download after validation are being set this way:
<?php
set_time_limit(0);
$file = '/absolute/path/to/myzip/myzip.zip';
header("X-Sendfile: $file");
header("Content-type: application/zip");
header('Content-Disposition: attachment; filename="' . basename($file) . '"');
Any help or suggestions would be appreciated. Thanks!
I would suggest taking a look at this comment:
http://www.php.net/manual/en/function.readfile.php#99406
Particularly, if you are using apache. If not the code in the link above should be helpful:
I started running into trouble when I had really large files being sent to clients with really slow download speeds. In those cases, the
script would time out and the download would terminate with an
incomplete file. I am dead-set against disabling script timeouts - any
time that is the solution to a programming problem, you are doing
something wrong - so I attempted to scale the timeout based on the
size of the file. That ultimately failed though because it was
impossible to predict the speed at which the end user would be
downloading the file at, so it was really just a best guess so
inevitably we still get reports of script timeouts.
Then I stumbled across a fantastic Apache module called mod_xsendfile ( https://tn123.org/mod_xsendfile/ (binaries) or
https://github.com/nmaier/mod_xsendfile (source)). This module
basically monitors the output buffer for the presence of special
headers, and when it finds them it triggers apache to send the file on
its own, almost as if the user requested the file directly. PHP
processing is halted at that point, so no timeout errors regardless of
the size of the file or the download speed of the client. And the end
client gets the full benefits of Apache sending the file, such as an
accurate file size report and download status bar.
The code I finally ended up with is too long to post here, but in general is uses the mod_xsendfile module if it is present, and if not
the script falls back to using the code I originally posted. You can
find some example code at https://gist.github.com/854168
EDIT
Just to have a reference of code that does the "chunking" Link to Original Code:
<?php
function readfile_chunked ($filename,$type='array') {
$chunk_array=array();
$chunksize = 1*(1024*1024); // how many bytes per chunk
$buffer = '';
$handle = fopen($filename, 'rb');
if ($handle === false) {
return false;
}
while (!feof($handle)) {
switch($type)
{
case'array':
// Returns Lines Array like file()
$lines[] = fgets($handle, $chunksize);
break;
case'string':
// Returns Lines String like file_get_contents()
$lines = fread($handle, $chunksize);
break;
}
}
fclose($handle);
return $lines;
}
?>
When using readfile() -- using PHP on Apache -- is the file immediately read into Apache's output buffer and the PHP script execution completed, or does the PHP script execution wait until the client finishes downloading the file (or the server times out, whichever happens first)?
The longer back-story:
I have a website with lots of large mp3 files (sermons for a local church). Not all files in the audio archive are allowed to be downloaded, so the /sermon/{filename}.mp3 path is rewritten to really execute /sermon.php?filename={filename} and if the file is allowed to be downloaded then the content type is set to "audio/mpeg" and the file streamed out using readfile(). I've been getting complaints (almost exclusively from iPhone users who are streaming the downloads over 3G) that the files don't fully download, or that they cut off after about 10 or 15 minutes. When I switched from streaming out the file with a readfile() to simply redirecting to the file -- header("Location: $file_url"); -- all of the complaints went away (I even checked with a few users who could reliably reproduce the problem on demand previously).
This leads me to suspect that when using readfile() the PHP script engine is in use until the file is fully downloaded but I cannot find any references which confirm or deny this theory. I'll admit I'm more at home in the ASP.NET world and the dotNet equivalent of readfile() pushes the whole file to the IIS output buffer immediately so the ASP.NET execution pipeline can complete independently of the delivery of the file to the end client... is there an equivalent to this behavior with PHP+Apache?
You may still have PHP output buffering active while performing the readfile(). Check that with:
if (ob_get_level()) ob_end_clean();
or
while (ob_get_level()) ob_end_clean();
This way theonly remaining output Buffer should be apache's Output Buffer, see SendBufferSize for apache tweaks.
EDIT
You can also have a look at mod_xsendfile (an SO post on such usage, PHP + apache + x-sendfile), so that you simply tell the web server you have done the security check and that now he can deliver the file.
a few things you can do (I am not reporting all the headers that you need to send that are probably the same ones that you currently have in your script):
set_time_limit(0); //as already mention
readfile($filename);
exit(0);
or
passthru('/bin/cat '.$filename);
exit(0);
or
//When you enable mod_xsendfile in Apache
header("X-Sendfile: $filename");
or
//mainly to use for remove files
$handle = fopen($filename, "rb");
echo stream_get_contents($handle);
fclose($handle);
or
$handle = fopen($filename, "rb");
while (!feof($handle)){
//I would suggest to do some checking
//to see if the user is still downloading or if they closed the connection
echo fread($handle, 8192);
}
fclose($handle);
The script will be running until the user finishes downloading the file. The simplest, most efficient and surely working solution is to redirect the user:
header("Location: /real/path/to/file");
exit;
But this may reveal the location of the files. It's a good idea to password-protect the files that may not be downloaded by everyone anyway with an .htaccess file, but perhaps you use a database to detemine access and this is no option.
Another possible solution is setting the maximum execution time of PHP to 0, which disables the limit:
set_time_limit(0);
Your host may disallow this, though. Also PHP reads the file into the memory first, then goes through Apache's output buffer, and finally makes it to the network. Making users download the file directly is much more efficient, and does not have PHP's limitations like the maximum execution time.
Edit: The reason you get this complaint a lot from iPhone users is probably that they have a slower connection (e.g. 3G).
downloading files thru php isnt very efficient, using a redirect is the way to go. If you dont want to expose the location of the file, or the file isnt in a public location then look into internal redirects, here is a post that talks about it a bit, Can I tell Apache to do an internal redirect from PHP?
Try using stream_copy_to_stream() instead. I find is has fewer problems than readfile().
set_time_limit(0);
$stdout = fopen('php://output', 'w');
$bfname = basename($fname);
header("Content-type: application/octet-stream");
header("Content-Disposition: attachment; filename=\"$bfname\"");
$filein = fopen($fname, 'r');
stream_copy_to_stream($filein, $stdout);
fclose($filein);
fclose($stdout);
Under Apache, there is a nice elgant solution not involving php at all:
Just place an .htaccess config file into the folder containing the files to be offered for download with the following contents:
<Files *.*>
ForceType applicaton/octet-stream
</Files>
This tells the Apache to offer all files in this folder (and all its subfolders) for download, instead of directly displaying them in the browser.
See below url
http://php.net/manual/en/function.readfile.php
<?php
$file = 'monkey.gif';
if (file_exists($file)) {
header('Content-Description: File Transfer');
header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename='.basename($file));
header('Content-Transfer-Encoding: binary');
header('Expires: 0');
header('Cache-Control: must-revalidate');
header('Pragma: public');
header('Content-Length: ' . filesize($file));
ob_clean();
flush();
readfile($file);
exit;
}
?>