I'm using shared hosting(hostgator).
I have site with video content like youtube written in php.
Implemented via direct links to mp4 files and html video tag.
I want to limit connections for file downloads(plays) to around 350.
Because if I have more than ~350 connections hostgator blocks my site.
Is there any way to do that?
Any other suggestions how to deal with this situation will also be helpful.
You could use a php script which handles the actual file download. If the script is executed, increment your download counter and if the file is sent completely to the client, close the connection.
To detect if the file is completely sent, you should send the file in small chunks and check after each transmitted chunk, if the connection is still open.
To do this
send the correct mime-types and http headers
use ignore_user_abort to keep the script running if the client closes the connection
send the file in small chunks and check after each chunk if the connection is still alive. ob_flush and flush are used to keep the output buffer empty. connection_status or connection_aborted to test if the connection is still open.
after the whole file is submited, decrement your connection counter
In addition to this, you might also implement HTTP_RANGE, to resume incomplete downloads. This should be important especially for video downloads, if you want to seek somewhere in the middle of the stream.
Below a little .htaccess that rewrite all the requests for the PHP file.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule ^ yourFile.php [L]
</IfModule>
Below the PHP file
// code to increment the counter..
// increment_counter(); ...
// Use the request path (URI) to choose what file to send.
$filename = 'video.mp4';
$size = filesize($filename);
$f = fopen($filename, 'rb');
if (!$f) {
// error...
}
ignore_user_abort(true);
set_time_limit(0);
header("Content-Length: $size");
header("Content-Type: video/mp4");
while (!feof($f)) {
echo fread($f, 8192);
ob_flush();
flush();
if (connection_status() != 0) {
// download aborted... decrement the counter
// decrement_counter(); ...
break;
}
}
fclose($f);
// download completed - decrement counter
// decrement_counter(); ...
This script is pretty simple, but should give you an idea. You might add more logic (as said above HTTP_RANGE) or send other headers, but this should give you a good starting point.
References:
Below the links to the documentation of the functions that could be less known.
connection_status
ignore_user_abort
I want to download different feeds form some publishers. But the poor thing is, that they are first of all zipped as .gz and as second not in the right format. You can download one of the feeds and check it out. They do not have any filespec... So, I'm forced to add the .csv by myself..
My question now is, how can I unzip those files from the different urls?
How I do rename them, I know. But how do I unzip them?
I already searched for it and found this one:
//This input should be from somewhere else, hard-coded in this example
$file_name = '2013-07-16.dump.gz';
// Raising this value may increase performance
$buffer_size = 4096; // read 4kb at a time
$out_file_name = str_replace('.gz', '', $file_name);
// Open our files (in binary mode)
$file = gzopen($file_name, 'rb');
$out_file = fopen($out_file_name, 'wb');
// Keep repeating until the end of the input file
while (!gzeof($file)) {
// Read buffer-size bytes
// Both fwrite and gzread and binary-safe
fwrite($out_file, gzread($file, $buffer_size));
}
// Files are done, close files
fclose($out_file);
gzclose($file);
But with those feeds it doesn't work...
Here a two example files: file one | file two
Do you have an idea? - Would be very grateful!
Greetings!
windows 10 + php7.1.4 it's work.
The following code has the same effect.
ob_start();
readgzfile($file_name);
file_put_contents($output_filename', ob_get_contents());
ob_clean();
Or you can try to use the gzip command to decompress, and then use the it.
Program execution Functions
I try to create the shop where user could buy the video/audio files. The files will be placed at another remote server (Debian). I can't figure out how to let downloading for particular user only. I could calculate the control sum somehow by IP and the link will be something like this:
http://100.000.000.000/files/video.avi?hash=87a686d86d8868a6868a
But how to check this on the remote server? I don't know is the good idea to read whole movie file with PHP script.
Basically two methods are possible.
File system:
You could use the file system, create a password protected folder for each user and copy all their files to it, or better, if you use Linux, use symbolic links (ln -s).
PHP:
Or you could stream files through PHP while it checks access. I don't think that's a real problem. PHP doesn't need to do much if it just pushes through raw data.
$total = filesize($filepath);
$blocksize = (2 << 20); //2M chunks
$sent = 0;
$handle = fopen($filepath, "r");
// Push headers that tell what kind of file is coming down the pike
header('Content-type: '.$content_type);
header('Content-Disposition: attachment; filename='.$filename);
header('Content-length: '.$filesize*1024);
// Now we need to loop through the file and echo out chunks of file data
// Dumping the whole file fails at > 30M!
while($sent < $total) {
echo fread($handle, $blocksize);
$sent += $blocksize;
}
(code is short, no error checks, no password check, no file closure, etc)
It does depend on what kind of password system you have, and what you're allowed to do on your server.
I'm finding it difficult to phrase this question correctly, let me try to explain our problem...
We have an intranet running on Ubunutu box with Apache2/PHP 5.2.4. We have a bit of PHP code that reads a file from a directory that is not publically accessible and output it to the screen (code below):
$file_path = '/home/path/to/filename.gif';
if(file_exists($file_path)){
$output = FALSE;
//File Information
$path_parts = pathinfo($file_path);
$file_size = filesize($file_path);
$file_ext = (isset($path_parts['extension'])) ? strtolower($path_parts['extension']) : null;
$file_name = $path_parts['basename'];
//Sets up the headers
if($file_size > 0){
header('Content-Length: ' .$file_size);
}
header('Content-Disposition: attachment; filename="'.$file_name.'"');
header('Content-Type: application/octet-stream');
//Reads the File
if($file_size > 0){
$handle = fopen($file_path, "r");
$output = fread($handle, $file_size);
fclose($handle);
}
//Outputs the File
echo $output;
}
Inside our network when, browsing to the page that uses this code, the file is downloaded perfectly and quickly...
However, when accessing this page via our Cisco ASA/Proxy/VPN (not sure what to call it) this code locks up the browser, but does eventually download the file...
After a bit of experimenting, after taking out the headers and just echoing the contents of the file to the browser, it prints no problem. However as soon as I add the lines with the headers back into the code it causes the hanging again, but only when accessed via this box..
Anybody come across this problem before or have any idea what we can try to move forward?
Thanks for any advice...
Have you tried eliminating the content-size header entirely? The proxy may be taking that as a firm promise and if the data you're sending ends up being a different size, the proxy may wait for those last few "missing" bytes to show up.
Just as an aside, you should use [readfile()][1] instead of the fopen()/fread()/echo construct you have now.
As it stands now, you're slurping the contents of the entire file into memory and then echoing out. For large files and multiple requests, you'll kill the server with memory starvation. readfile will automatically stream the file in smaller chunks so that memory usage is minimal.
Your proxy obviously have problems with the Content-Type: application/octet-stream. Try setting it to the real MIME-type of each file. You can use the Fileinfo module to find out which MIME-type a certain file is, like this:
//You may need to specify the location of your system's magic file
//See http://php.net/finfo_open for more info
$finfo = new finfo(FILEINFO_MIME);
$mimetype = $finfo->file($file_path);
I have a php script on a server to send files to recipents: they get a unique link and then they can download large files. Sometimes there is a problem with the transfer and the file is corrupted or never finishes. I am wondering if there is a better way to send large files
Code:
$f = fopen(DOWNLOAD_DIR.$database[$_REQUEST['fid']]['filePath'], 'r');
while(!feof($f)){
print fgets($f, 1024);
}
fclose($f);
I have seen functions such as
http_send_file
http_send_data
But I am not sure if they will work.
What is the best way to solve this problem?
Regards
erwing
Chunking files is the fastest / simplest method in PHP, if you can't or don't want to make use of something a bit more professional like cURL, mod-xsendfile on Apache or some dedicated script.
$filename = $filePath.$filename;
$chunksize = 5 * (1024 * 1024); //5 MB (= 5 242 880 bytes) per one chunk of file.
if(file_exists($filename))
{
set_time_limit(300);
$size = intval(sprintf("%u", filesize($filename)));
header('Content-Type: application/octet-stream');
header('Content-Transfer-Encoding: binary');
header('Content-Length: '.$size);
header('Content-Disposition: attachment;filename="'.basename($filename).'"');
if($size > $chunksize)
{
$handle = fopen($filename, 'rb');
while (!feof($handle))
{
print(#fread($handle, $chunksize));
ob_flush();
flush();
}
fclose($handle);
}
else readfile($path);
exit;
}
else echo 'File "'.$filename.'" does not exist!';
Ported from richnetapps.com / NeedBee. Tested on 200 MB files, on which readfile() died, even with maximum allowed memory limit set to 1G, that is five times more than downloaded file size.
BTW: I tested this also on files >2GB, but PHP only managed to write first 2GB of file and then broke the connection. File-related functions (fopen, fread, fseek) uses INT, so you ultimately hit the limit of 2GB. Above mentioned solutions (i.e. mod-xsendfile) seems to be the only option in this case.
EDIT: Make yourself 100% that your file is saved in utf-8. If you omit that, downloaded files will be corrupted. This is, because this solutions uses print to push chunk of a file to a browser.
If you are sending truly large files and worried about the impact this will have, you could use the x-sendfile header.
From the SOQ using-xsendfile-with-apache-php, an howto blog.adaniels.nl : how-i-php-x-sendfile/
Best solution would be to rely on lighty or apache, but if in PHP, I would use PEAR's HTTP_Download (no need to reinvent the wheel etc.), has some nice features, like:
Basic throttling mechanism
Ranges (partial downloads and resuming)
See intro/usage docs.
We've been using this in a couple of projects and it works quite fine so far:
/**
* Copy a file's content to php://output.
*
* #param string $filename
* #return void
*/
protected function _output($filename)
{
$filesize = filesize($filename);
$chunksize = 4096;
if($filesize > $chunksize)
{
$srcStream = fopen($filename, 'rb');
$dstStream = fopen('php://output', 'wb');
$offset = 0;
while(!feof($srcStream)) {
$offset += stream_copy_to_stream($srcStream, $dstStream, $chunksize, $offset);
}
fclose($dstStream);
fclose($srcStream);
}
else
{
// stream_copy_to_stream behaves() strange when filesize > chunksize.
// Seems to never hit the EOF.
// On the other handside file_get_contents() is not scalable.
// Therefore we only use file_get_contents() on small files.
echo file_get_contents($filename);
}
}
For downloading files the easiest way I can think of would be to put the file in a temporary location and give them a unique URL that they can download via regular HTTP.
As part generating these links you could also remove files that were more than X hours old.
Create a symbolic link to the actual file and make the download link point at the symbolic link. Then, when the user clicks on the DL link, they'll get a file download from the real file but named from the symbolic link. It takes milliseconds to create the symbolic link and is better than trying to copy the file to a new name and download from there.
For example:
<?php
// validation code here
$realFile = "Hidden_Zip_File.zip";
$id = "UserID1234";
if ($_COOKIE['authvalid'] == "true") {
$newFile = sprintf("myzipfile_%s.zip", $id); //creates: myzipfile_UserID1234.zip
system(sprintf('ln -s %s %s', $realFile, $newFile), $retval);
if ($retval != 0) {
die("Error getting download file.");
}
$dlLink = "/downloads/hiddenfiles/".$newFile;
}
// rest of code
?>
<a href="<?php echo $dlLink; ?>Download File</a>
That's what I did because Go Daddy kills the script from running after 2 minutes 30 seconds or so....this prevents that problem and hides the actual file.
You can then setup a CRON job to delete the symbolic links at regular intervals....
This whole process will then send the file to the browser and it doesn't matter how long it runs since it's not a script.
When I have done this in the past I've used this:
set_time_limit(0); //Set the execution time to infinite.
header('Content-Type: application/exe'); //This was for a LARGE exe (680MB) so the content type was application/exe
readfile($fileName); //readfile will stream the file.
These 3 lines of code will do all the work of the download readfile() will stream the entire file specified to the client, and be sure to set an infinite time limit else you may be running out of time before the file is finished streaming.
If you are using lighttpd as a webserver, an alternative for secure downloads would be to use ModSecDownload. It needs server configuration but you'll let the webserver handle the download itself instead of the PHP script.
Generating the download URL would look like that (taken from the documentation) and it could of course be only generated for authorized users:
<?php
$secret = "verysecret";
$uri_prefix = "/dl/";
# filename
# please note file name starts with "/"
$f = "/secret-file.txt";
# current timestamp
$t = time();
$t_hex = sprintf("%08x", $t);
$m = md5($secret.$f.$t_hex);
# generate link
printf('%s',
$uri_prefix, $m, $t_hex, $f, $f);
?>
Of course, depending on the size of the files, using readfile() such as proposed by Unkwntech is excellent. And using xsendfile as proposed by garrow is another good idea also supported by Apache.
header("Content-length:".filesize($filename));
header('Content-Type: application/zip'); // ZIP file
header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename="downloadpackage.zip"');
header('Content-Transfer-Encoding: binary');
ob_end_clean();
readfile($filename);
exit();
I'm not sure this is a good idea for large files. If the thread for your download script runs until the user has finished the download, and you're running something like Apache, just 50 or more concurrent downloads could crash your server, because Apache isn't designed to run large numbers of long-running threads at the same time. Of course I might be wrong, if the apache thread somehow terminates and the download sits in a buffer somewhere whilst the download progresses.
I have used the following snippet found in the comments of the php manual entry for readfile:
function _readfileChunked($filename, $retbytes=true) {
$chunksize = 1*(1024*1024); // how many bytes per chunk
$buffer = '';
$cnt =0;
// $handle = fopen($filename, 'rb');
$handle = fopen($filename, 'rb');
if ($handle === false) {
return false;
}
while (!feof($handle)) {
$buffer = fread($handle, $chunksize);
echo $buffer;
ob_flush();
flush();
if ($retbytes) {
$cnt += strlen($buffer);
}
}
$status = fclose($handle);
if ($retbytes && $status) {
return $cnt; // return num. bytes delivered like readfile() does.
}
return $status;
}
I have had same problem,
my problem solved by adding this before starting session
session_cache_limiter('none');
This is tested on files of a size 200+ MB on a server that has 256MB memory limit.
header('Content-Type: application/zip');
header("Content-Disposition: attachment; filename=\"$file_name\"");
set_time_limit(0);
$file = #fopen($filePath, "rb");
while(!feof($file)) {
print(#fread($file, 1024*8));
ob_flush();
flush();
}