Downloading big files and writing it locally - php

which is the best way to download from php large files without consuming all server's memory?
I could do this (bad code):
$url='http://server/bigfile';
$cont = file_get_contents($url);
file_put_contents('./localfile',$cont);
This example loads entry remote file in $cont and this could exceed memory limit.
Is there a safe function (maybe built-in) to do this (maybe stream_*)?
Thanks

You can use curl and the option CURLOPT_FILE to save the downloaded content directly to a file.
set_time_limit(0);
$fp = fopen ('file', 'w+b');
$ch = curl_init('http://remote_url/file');
curl_setopt($ch, CURLOPT_TIMEOUT, 75);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_exec($ch);
curl_close($ch);
fclose($fp);

Here is a function I use when downloading large files. It will avoid loading the entire file into the buffer. Instead, it will write to the destination as it receives the bytes.
function download($file_source, $file_target)
{
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'wb');
if (!$rh || !$wh) {
return false;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 1024)) === FALSE) {
return false;
}
}
fclose($rh);
fclose($wh);
return true;
}

Related

PHP error - fileSize returns the image string

I am downloading a public image and writing it to file. But the filesize(file) returns the entire encoded string of the image
My code is as follows
<?php header('Access-Control-Allow-Origin: *'); ?>
<?php
function download_and_write($src,$filename)
{
$ch = curl_init($src);
$fp = fopen($_SERVER["DOCUMENT_ROOT"]."/dashboard/uploads/".$filename, 'wb');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
$httpcode = curl_getinfo($curl, CURLINFO_HTTP_CODE);
curl_close($ch);
fclose($fp);
$filepath = $_SERVER["DOCUMENT_ROOT"]."/dashboard/uploads/".$filename;
return filesize($filepath);
}
$src_url = $_POST['source'];
$dest_url = $_POST['destination'];
$filename = $_POST['filename'];
try{
$download = download_and_write($src_url,$filename);
echo $download;
}
catch(Exception $e) {
$mess=$e->getMessage();
echo "exception";
echo $mess;
}
?>
And echo of the download variable is like this
����T۸\u001b\u0014��Yl����\u0000�\"#?d�k�ϵ.Ⱦ)�;�+.�t��x/�M�E>}࿝7�\u0015�ݎ��b۪N+\u001e�K\u000f/�T�]�W3���\bhyx����~\tvM����3�|��:o�)��\u0005���(�N�u��6�RwV\u001d��\u001e_���f_�t��/\u0003�|��:o�)��\u0005���(�]��[��\u000fg\f���lN�0��g]~*�\u0018ۖ_0�Zk\u0002'\u0016�Ѱ�\u0012������j+>�uo�x/�M�E>}࿝7�\u0015Lj �A\u0004w��K��E_����C�����\u001a��ޗfW�4��/\u0006u_�x/�M�E>}࿝7�\u0015���]��l�VvWv�ds\bi�wU3���uqe�\u001d�M�\u0012�\u0000�{}�K��m.�XV]ώ�����>o�)��\u0005���(�T�u��\u001b+C<�tM��\u0011\u0004\u0003�=#ڬ[�b�ݕ�d�Mo�F�t�d��𨫸�\u0003�|��:o�)��\u0005���(�ck\tn�>��\u0014�m��\u0000������ޣO��^x�����/F1� ����f��].;��\u001f\u001d�W����tߔS��\u000b��~Q\\����䓲$n\r{�Ѥ�\u0003�S\"�\\\u0016��j\t�2̍c\u001f$d\u000e�u��ݒ?\u0017ҥ�+��?>�_Λ��|��:o�+�d1���:z�6\u000er�J�\u0010���\u0014^�N�K��Ȼ��]9���R���ƕ\u0017g\u0015���}࿟7�\u0014����|ߔW'����tN�f��\u0004�0�3d\u000e��e��͙�RC�2\u001b6]\\=ёɣ��=�\u001e�n����eO���:\u0017ϼ\u0017�����>�_Λ����1��5��^h�y<�{\u000bC����C\u000b�\fq��\r/�#Op�ܗf\u001f�4��4�\u000e���\u0005���(�ϼ\u0017�����DX��O�ERw��\u001f��\u0012yOq�(�E$\u0012�)���aӚ��\u000f�.��Ɣ��W�:�\u0000ϼ\u0017�����>�_Λ��㨗fvޑ�x~N���\u0005���(�ϼ\u0017������%��zG%��;\u0017ϼ\u0017�����>�_Λ��㨗cm�\u001c����_>�_Λ��|��:o�+��]���r^\u001f��|��:o�)��\u0005���(�:�v6ޑ�x~N���\u0005���(�ϼ\u0017������%��zG%��;\u0017ϼ\u0017�����>�_Λ��㨗cm�\u001c����_>�_Λ��|��:o�+��]���r^\u001f��|��:o�*#��`;�i�\u0000(�*�vU��!p^\u001f���!`�?�\u0014\\T\"��v���x~C��^*�;=\n�G��>-�\u0011{�|\nr�\u0002���/yO�NS�PX�\u0017����)�(,x��S�S��\u0014\u0016<E�)�)�|\n\u000b\u001e\"���\u0014�>\u0005\u0005�\u0011{�|\nr�\u0002�Lj��>\u00059O�PX�h��\n���:�/����,���7G��֔,��a��X��\u001c_(��5�ӹ9G)>�-o��')�*�ǥI��y����n���de��kQ�\u0004V$�ɹ�(`��� ;����L�Cnk��r\t-?\u001f\u0014\u0006g�\u0006K#\u001c\t󏳦�:Zg)�NS����ۻ������7\u001brA\u00060�\u0016`���\u0018�c�\u0005���G�H�C���E���%���\t�\u0007\u0003�k��Ѣ\u0016��|\u0013�� 폧͙N\u0018��e��_��i�'��7�\u001d\u001f�g�����c!��\u001d\u0015AJ��w+y�A���<�V���>>�oY���h\u0017�;��7�}�3����l���}(b�����W�˕����Y��>&�۰>�#����s61#4Ɨ{�\u001bX̜\r��\u0015���y,\u000bo{�\u0014����\u0001�,\u0007)�NS��ZN(���m���q���P��,�Nh��$�D�z9��=�Yl���\\��n=��#����\u001f �\u0004\u0010�N���KH�>\u00059O�\u0016:\\�\\m��X�6�\u00172���\u0013�X�'g\b���/F�z9U�{`�#��ymVA\u00151\u0013��\u0001���\u001c�������T�5G��4\f��'lo�p�V9O�\u001d\u001e��x{�����m�\\�8btv\u0018׷\n��\u000f�\u000f�==��{\u000fn)p���*>z�y��-�\u0012�\u0017l?�\u001c�_E�r�\u0004�>\bb:d�����e��\u001b<T٦�����s���\u001b׽��\u0003ڶ\u001c��f\u0016hɧ\u0014��/\f�ٙ�h'n$�o�h����)�P�=%�Ia���mV�\u0006{�R%��\u001c�wd\u000b�\u001d�F�~�\u0005�F�g�ې\u000e\u001eYv6S���g�?x\fZ�)�*]��/\bD���������\u0003��zO���SjW��o�o��}̖\u0005ݶ##J\u000b\u0011׹+�{\\�;0��{o7�\n�fH�R�\u0016rM�'mT>f�5�����|V��|\u0013��!��\u000e1Q�w\u001e�\u001fs}�����\rl�5#\u0007d�\u0000,3K0\u000fi�#z\u001d=\u001a���.�l�\u0010��Z��Q����\u001f��>\u001e��r�\u0004�>\bt���Zѷ��m��\u0013��:v8|��h�{w0�}���h���z<�L��\u0013Dq5��p���=\u001a��j<���)�P�KrVq�g#����ʈ���KU͊g�2K?#~���v�\u0019,F��sb'F�?b\u001ad\u001b�\u0004m�w���S����SR�\\�����\u0000f�<�\u0019;�Gd v2Z�k!\u0012��lЏ��h�\u0000��K!��\r�֫GM�x��F����Z�)�NS�P��%{����;�w\u001b�\u0019��\u0013\u0007��]�G]ы\u0010�1�\u0018��\u0018�\u0000����kM����\u0004�\r\u0011�\ta\u000e\u0007����Z�>\n��G��qqk\u0006�<\u0006��:���I�����n\u0018����Y����z�ls��y\u000b]�K��O0=�4|���\"�߂���J���\u0011�gl?�]:-W��')�C�kv������r39+u����`�j�-y�~�\u0004l�ۣ�[\u0013�,Q�\u001d�\u0017��ZV��6��H�\u001av[��\u000e�+Ea{\u001e׳msN�\u001e��w�w��$SvMl�\u0019Lq5����Gz��KIQ�)��o�5��6|���\u001eF�\rǚ�b-k���{��\u001d��\u0011��������g�fI\u0019�\u001b:<\u0012�4h�;��r�\u0002���S5��U4��\u0000[�6�ݎ(gt6\u001a�\u000e\u001e(�k��s7m��k\u0013���Ⱏ�\u0012J�\u001e���#�:\u0007�X~S�������L��m���\u0000F㉵\u0013)a�}�����~g?�R;|�#�\u000fj����Î�%v\u0019�\u0019\u0006N9%�DQ��$�\u001e�=�P�>\t�|\u0010��$��s]ﻇ�}\rң+�a���j��߂nH�\u000f%��<�=���,��M��^\ta��r26J$�\u001c��?`\u0000��S����\u0007k�%\u001d۳~ٱ�x���\u0013g��\u0017�c�\u000e~�绨��ܬp����V���6�-.�Q���F����\u0004�>\u0005C�h�56�yݷ��}��\u0012�\u001ddJ�)�<2Z��Ń��z��][\u0011:�\t\u0005�Du�ҾNi#�\\G)פw�Z/)�)�|\u0015:�6W�^�����u�\u001d?\u000ed�<�IM����잎�\u001f�y���x~�6O\u0004v\f��\u0004�\u0006\u0002�\u001c\u000e���X>S�S��(qU�$䯕��Sz���U�\u00043R�\u000b��i�ILb^V����\u0011�V�İE&FyjX�c�6\u00199�\u000f-q\u001a�k��\u0002���\u0004�>\n���ҝX�q����x��S�S��\u0014<�<E�)�)�|\n\u000b\u001e\"���\u0014�>\u0005\u0005�\u0011{�|\nr�\u0002�Lj��>\u00059O�Ac�^��\u0002������/yO�NS�PX�\u0017����)�(,x��S�W�>\u0007࠱�E�\u0007��ȱ��
Why would this occur ? The file didn't download.
EDIT:
I'm not checking the size of a remote file. I'm checking the size of the file on my own server. So this can't be related to the possible duplicate you mentioned.
I think fopen is getting an error, so you're not redirecting the curl output to the file. Add error checking.
error_reporting(E_ALL);
$fp = fopen($_SERVER["DOCUMENT_ROOT"]."/dashboard/uploads/".$filename, 'wb');
if (!$fp) {
throw new Exception("Unable to open download file");
}

Downloading Images from URL in PHP

I have a file, where I read URLs line by line. The URL are links to images. All the image-links work in the browser. Now I want to download them to my current directory. For the record I need to use a PHP-script.
This is how I get my Images:
for ($j; $j< $count;$j++)
{
// Get Image-URL as String and then do getImage
$image = $arr[$j];
$newname = $j;
getImage($image, $newname);
}
function getImage($image, $newname)
{
$ch = curl_init($image);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,1);
$rawdata=curl_exec ($ch);
curl_close ($ch);
$fp = fopen("$newname.jpg",'w');
fwrite($fp, $rawdata);
fclose($fp);
}
But the problem is, I get all the images, only the last one is viewable. The others I can't open and they only have 1KB.
So what did I do wrong?
Plus I need to also download png-Files later on, so can I then just change the $newname.jpg into $newname.png?
Thanks in advance, I really need an answer fast, been sitting here for hours, trying to figure it out.
Why not use stream_copy_to_stream?
function getImage($image, $newname, $fileType = "jpg")
{
$in = fopen($image, "r");
$out = fopen("$newname.$fileType",'w');
stream_copy_to_stream($in, $out);
fclose($in); fclose($out);
}
You can also play with stream_set_read_buffer
stream_set_read_buffer($in, 4096);
Just tested this with our avatar pics.
$data = [
"https://www.gravatar.com/avatar/42eec337b6404f97aedfb4f39d4991f2?s=32&d=identicon&r=PG&f=1",
"https://www.gravatar.com/avatar/2700a034dcbecff07c55d4fef09d110b?s=32&d=identicon&r=PG&f=1",
];
foreach ($data as $i => $image) getImage($image, $i, "png");
Works perfectly.
Debug: Read the remote headers for more info?
$httpresponseheader
var_dump($http_response_header);
or stream_get_meta_data
var_dump(stream_get_meta_data($in), stream_get_meta_data($out));

How Can php know if the image has been loaded fully?

I wrote a PHP script which simply gets URL for images and tries to download/save them on server.
My problem here is that sometimes the image is not fully loaded and the codes blow only save images partially. I did some research but couldn't figure out if I can add something to it, so it can check whether it is saving the full image or not.
Images sizes and other properties of images are random so I can't check it with those factors unless there is a way that I can get those info before loading them image.
Thank you all.
if ( $leng >= "5" ) {
define('UPLOAD_DIR', dirname(__FILE__) . '/files/');
$length = 512000;
$handle = fopen($url, 'rb');
$filename = UPLOAD_DIR . substr(strrchr($url, '/'), 1);
$write = fopen($filename, 'w');
while (!feof($handle))
{
$buffer = fread($handle, $length);
fwrite($write, $buffer);
}
fclose($handle);
fclose($write);
}else {Echo "failed";}
I think that using cURL is a better solution than using fopen for url. Check this out:
$file = fopen($filename, 'wb'); //Write it as a binary file - like image
$c = curl_init($url);
curl_setopt($c, CURLOPT_FILE, $file);
curl_setopt($c, CURLOPT_FOLLOWLOCATION, true); //Allow cURL to follow redirects (but take note that this does not work with safemode enabled)
curl_exec($c);
$httpCode = curl_getinfo($c, CURLINFO_HTTP_CODE); //200 means OK, you may check it later, just for sure
curl_close($c);
fclose($file);
Partially based on Downloading a large file using curl

How to make sure file saved by using cURL

<?php
$source = 'http://www.xxx.com/1.jpg';
$fileBody = date('YmdHis') . rand(1000, 9999);
$extension = pathinfo($source, PATHINFO_EXTENSION);
$fileName = $fileBody . '.' . $extension;
$ch = curl_init($source);
$fp = fopen($path . $fileName, 'wb');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
clearstatcache();
return $fileName;
This is how I grab image from internet, the image saved successfully and I will return file name for ajax to make immediately thumbnail, but sometimes php return the $fileName when it still processing download, therefore JavaScript reveal empty image on the page, how response after the file indeed been download.
curl_exec returns true on success and false on failure. Use that information.
Also, You can check curl_getinfo to make sure the transfer completed successfully and was not empty. (You get http_code there for example, as well as content_type and size_download).
$downComplete = false;
while(!$downComplete)
{
if(!file_exist($filePath))
{
sleep(1);
}
else
{
$downComplete = true;
break;
}
}
Hi,Above is an idea for check if file is completely downloaded,i think it's useful,
main idea is check saved file all the time until its finished download,then you can
display the img to front end..you can add it after your curl code,i didn't run it
myself,so just an idea..

Need php script to download a file on a remote server and save locally

Trying to download a file on a remote server and save it to a local subdirectory.
The following code seems to work for small files, < 1MB, but larger files just time out and don't even begin to download.
<?php
$source = "http://someurl.com/afile.zip";
$destination = "/asubfolder/afile.zip";
$data = file_get_contents($source);
$file = fopen($destination, "w+");
fputs($file, $data);
fclose($file);
?>
Any suggestions on how to download larger files without interruption?
$ch = curl_init();
$source = "http://someurl.com/afile.zip";
curl_setopt($ch, CURLOPT_URL, $source);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec ($ch);
curl_close ($ch);
$destination = "/asubfolder/afile.zip";
$file = fopen($destination, "w+");
fputs($file, $data);
fclose($file);
file_get_contents shouldn't be used for big binary files because you can easily hit PHP's memory limit. I would exec() wget by telling it the URL and the desired output filename:
exec("wget $url -O $filename");
Since PHP 5.1.0, file_put_contents() supports writing piece-by-piece by passing a stream-handle as the $data parameter:
file_put_contents("Tmpfile.zip", fopen("http://someurl/file.zip", 'r'));
I always use this code,it's working very well.
<?php
define('BUFSIZ', 4095);
$url = 'Type The URL Of The File';
$rfile = fopen($url, 'r');
$lfile = fopen(basename($url), 'w');
while(!feof($rfile))
fwrite($lfile, fread($rfile, BUFSIZ), BUFSIZ);
fclose($rfile);
fclose($lfile);
?>
Use this solution if you do not know the format of the file that you are going to download.
$url = 'http:://www.sth.com/some_name.format' ;
$parse_url = parse_url($url) ;
$path_info = pathinfo($parse_url['path']) ;
$file_extension = $path_info['extension'] ;
$save_path = 'any/local/path/' ;
$file_name = 'name' . "." . $file_extension ;
file_put_contents($save_path . $file_name , fopen($url, 'r'))
Try out phpRFT:http://sourceforge.net/projects/phprft/files/latest/download?source=navbar
It have progress_bar and simple filename detactor...
A better and lighter script which is streaming file:
<?php
$url = 'http://example.com/file.zip'; //Source absolute URL
$path = 'file.zip'; //Patch & file name to save in destination (currently beside of PHP script file)
$fp = fopen($path, 'w');
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FILE, $fp);
$data = curl_exec($ch);
curl_close($ch);
fclose($fp);
?>

Categories