I call the script mywebsite.com/download.php in crontab.
download.php
$video_url = "www.example.com/sample.mp4"; //hour long video
$link = '/home/mywebsite.com/videos/test.mp4';
$chunkSize = 1;
$chunksize = $chunkSize*(1024*1024); // How many bytes per chunk
$data = '';
$bytesCount = 0;
$handle = fopen($video_url, 'rb');
$fp = fopen($link, 'w');
if ($handle === false) {
return false;
}
while (!feof($handle)) {
$data = fread($handle, $chunksize);
fwrite($fp, $data, strlen($data));
}
$status = fclose($handle);
fclose($fp);
The problem is the script timeouts with long videos. I don't have the option of increasing server timeout limits.
What is an alternate method to guarantee the video get's downloaded?
ignore_user_abort(); ?
set_time_limit(0); ? Will this be ignored by server?
I have a queue system so only 1 video downloads at a time.
crontab call
*/15 * * * * /usr/bin/php /home/example.com/index.php media get_video
Related
I have a 40M zip file that is at a web address of say http://info/data/bigfile.zip that I would like to download to my local server. What is the best way currently to download a zip file of that size using PHP or header requests such that it won't time out at 8M or give me a 500 error? Right now, I keep getting timed out.
In order to download large file via php try something like this (source http://teddy.fr/2007/11/28/how-serve-big-files-through-php/):
<?php
define('CHUNK_SIZE', 1024*1024); // Size (in bytes) of tiles chunk
// Read a file and display its content chunk by chunk
function readfile_chunked($filename, $retbytes = TRUE) {
$buffer = '';
$cnt = 0;
$handle = fopen($filename, 'rb');
if ($handle === false) {
return false;
}
while (!feof($handle)) {
$buffer = fread($handle, CHUNK_SIZE);
echo $buffer;
ob_flush();
flush();
if ($retbytes) {
$cnt += strlen($buffer);
}
}
$status = fclose($handle);
if ($retbytes && $status) {
return $cnt; // return num. bytes delivered like readfile() does.
}
return $status;
}
// Here goes your code for checking that the user is logged in
// ...
// ...
$filename = 'path/to/your/file'; // url of your file
$mimetype = 'mime/type';
header('Content-Type: '.$mimetype );
readfile_chunked($filename);
?>
**Second solution **
Copy the file one small chunk at a time
/**
* Copy remote file over HTTP one small chunk at a time.
*
* #param $infile The full URL to the remote file
* #param $outfile The path where to save the file
*/
function copyfile_chunked($infile, $outfile) {
$chunksize = 10 * (1024 * 1024); // 10 Megs
/**
* parse_url breaks a part a URL into it's parts, i.e. host, path,
* query string, etc.
*/
$parts = parse_url($infile);
$i_handle = fsockopen($parts['host'], 80, $errstr, $errcode, 5);
$o_handle = fopen($outfile, 'wb');
if ($i_handle == false || $o_handle == false) {
return false;
}
if (!empty($parts['query'])) {
$parts['path'] .= '?' . $parts['query'];
}
/**
* Send the request to the server for the file
*/
$request = "GET {$parts['path']} HTTP/1.1\r\n";
$request .= "Host: {$parts['host']}\r\n";
$request .= "User-Agent: Mozilla/5.0\r\n";
$request .= "Keep-Alive: 115\r\n";
$request .= "Connection: keep-alive\r\n\r\n";
fwrite($i_handle, $request);
/**
* Now read the headers from the remote server. We'll need
* to get the content length.
*/
$headers = array();
while(!feof($i_handle)) {
$line = fgets($i_handle);
if ($line == "\r\n") break;
$headers[] = $line;
}
/**
* Look for the Content-Length header, and get the size
* of the remote file.
*/
$length = 0;
foreach($headers as $header) {
if (stripos($header, 'Content-Length:') === 0) {
$length = (int)str_replace('Content-Length: ', '', $header);
break;
}
}
/**
* Start reading in the remote file, and writing it to the
* local file one chunk at a time.
*/
$cnt = 0;
while(!feof($i_handle)) {
$buf = '';
$buf = fread($i_handle, $chunksize);
$bytes = fwrite($o_handle, $buf);
if ($bytes == false) {
return false;
}
$cnt += $bytes;
/**
* We're done reading when we've reached the conent length
*/
if ($cnt >= $length) break;
}
fclose($i_handle);
fclose($o_handle);
return $cnt;
}
Adjust the $chunksize variable to your needs. This has only been mildly tested. It could easily break for a number of reasons.
Usage:
copyfile_chunked('http://somesite.com/somefile.jpg', '/local/path/somefile.jpg');
Not a lot of detail given, but it sounds like php.ini defaults are restricting the servers ability to transfer large files via php web interface in a timely manner.
Namely these settings post_max_size, upload_max_filesize, or max_execution_time
Might jump in your php.ini file, up the sizes, restart Apache, and retry the file transfer.
HTH
I have a script processing large text files.
I am limited however by the size of the files.
I've done some searching on this forum, and i've come to the conclusion that i must process the file line by line, however this brings up quite a bit of issues for me, as i need some detailed info from the file, before i can start processing it.
I've tried adding each line to a variable as below:
$content = "";
$handle = fopen($target_File, "r") or die("Couldn't get handle");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 256);
// Process buffer here..
$content .= $buffer;
}
fclose($handle);
}
And just as expected, it did not work.
Could anyone help me out ?
add this on top of your file
ini_set("memory_limit", -1);
Be careful, the page can use all the RAM on the server
$content = "";
$handle = fopen($target_File, "r") or die("Couldn't get handle");
if ($handle) {
while (($buffer = fgets($handle, 4096))!== false) {
// Process buffer here..
$content .= $buffer;
}
fclose($handle);
}
OR
$content = "";
$target_File ="/tmp/uploadfile.txt";
$handle = fopen($target_File, "r") or die("Couldn't get handle");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
// Process buffer here..
//$content .= $buffer;
}
fclose($handle);
}
NOTE:
If the problem is caused by hitting the memory limit, you can try setting it a higher value (this could work or not depending on php's configuration).
this sets the memory limit to 32 Mb
ini_set("memory_limit","32M");
If I have three get parameters:
$filename = $_GET['filename'];
$start = $_GET['start'];
$size = $_GET['size'];
And I am reading a chunk of the file like so:
$handle = fopen($basepath . $filename, "rb");
fseek($handle, $start);
$contents = fread($handle, $size);
echo md5($contents);
How may I read large portions of a file (anywhere from 1mb to 1gb) and create a hash or checksum of its contents without needing to allocate enough memory for the entire read?
At the moment if I try to hash a too large of part of the file I get a memory error since php can not allocated enough memory (roughly 400mb).
Is there a hashing function in which I can digest parts of the file at a time rather than the entire contents at once (for example starting at $start read 100kb blocks and feed it to the function until $size is met)? And how would I read the file in chunks so that I would start at $start and read $size bytes?
If there is not such a hashing or checksum function that supports feeding chunks of the data pieces at a time, would file_get_contents() fix the issue of allocating memory for large reads? I am not entirely sure how that function works.
Thanks.
http://php.net/manual/en/function.hash-update.php
<?php
define('CHUNK', 65536);
//$file = 'alargefile.img';
//$start = 256 * 1024 * 1024;
//$size = 512 * 1024 * 1024;
$fp = fopen($file, "r");
fseek($fp, $start);
$ctx = hash_init('md5');
while ($size > 0) {
$buffer = fread($fp, min($size, CHUNK));
hash_update($ctx, $buffer);
$size -= CHUNK;
}
$hash = hash_final($ctx);
fclose($fp);
print $hash;
?>
I tried to stream an audio file using this PHP code, but always get cut around 3/4 of the file size, especially for iPad and Android agents.
ob_clean();
flush();
set_time_limit(0);
$size = intval(sprintf("%u", filesize($filename)));
$chunksize = 0.5 * (1024 * 1024);
if ($size > $chunksize) {
$handle = fopen($filename, 'rb');
$buffer = '';
while (!feof($handle)) {
$buffer = fread($handle, $chunksize);
echo $buffer;
ob_flush();
flush();
}
fclose($handle);
} else {
readfile($filename);
}
Is there a better way to do this? I have tried completely with readfile() only, the result is even worse. Thanks in advance.
I'm getting a 30 second timeout error because the code keeps checking if the file is over 5mb when it's under. The code is designed to reject files over 5mb but i need it to also stop executing when the file is under 5mb. Is there a way to check the file transfer chunk to see if it's empty? I'm currently using this example by DaveRandom:
PHP Stop Remote File Download if it Exceeds 5mb
Code by DaveRandom:
$url = 'http://www.spacetelescope.org/static/archives/images/large/heic0601a.jpg';
$file = '../temp/test.jpg';
$limit = 5 * 1024 * 1024; // 5MB
if (!$rfp = fopen($url, 'r')) {
// error, could not open remote file
}
if (!$lfp = fopen($file, 'w')) {
// error, could not open local file
}
// Check the content-length for exceeding the limit
foreach ($http_response_header as $header) {
if (preg_match('/^\s*content-length\s*:\s*(\d+)\s*$/', $header, $matches)) {
if ($matches[1] > $limit) {
// error, file too large
}
}
}
$downloaded = 0;
while ($downloaded < $limit) {
$chunk = fread($rfp, 8192);
fwrite($lfp, $chunk);
$downloaded += strlen($chunk);
}
if ($downloaded > $limit) {
// error, file too large
unlink($file); // delete local data
} else {
// success
}
You should check if you have reached the end of the file:
while (!feof($rfp) && $downloaded < $limit) {
$chunk = fread($rfp, 8192);
fwrite($lfp, $chunk);
$downloaded += strlen($chunk);
}