I am using the SabreDAV PHP library to connect to a WebDAV server and download some files but it is taking forever to download a 1MB file and I have to download up to 1GB files from that server. I looked at this link http://code.google.com/p/sabredav/wiki/WorkingWithLargeFiles but it is not helpful because it's telling me that I will get a stream when I do a GET but it is not the case.
Here is my code:
$settings = array(
'baseUri' => 'file url',
'userName' => 'user',
'password' => 'pwd'
);
$client = new \Sabre\DAV\Client($settings);
$response = $client->request('GET');
response is an array with a 'body' key that contains the content of the file. What am I doing wrong? I only need the file for read only. How can I can read through the file line by line as quick as possible?
Thanks in advance.
If its taking too long just to download a 1MB file, then I think its not SabreDAV problem but a problem with your server or network, or perhaps the remote server.
The google code link you mentioned just lists a way if you want to transfer very large files, for that you will have to use the stream and fopen way they mentioned, but I think I was able to transfer 1GB files without using that way and just normally when I last used it with OwnCloud.
If you have a VPS/Dedi server, open ssh and use wget command to test the speed and time it takes to download that remote file from WebDAV, if its same as what its taking with SabreDAV, then its a server/network problem and not SabreDAV, else, its a problem with Sabre or your code.
Sorry but I donot have any code to post to help you since the problem itself is not clear and there can be more than 10 things causing it.
PS: You need to increase php limits for execution time, max file upload and max post size too relatively
Related
I've been searching through the web quite a bit now, found several possible solutions, but none of them worked. Some say it's due to php.ini settings, some say it's due to the method I am using from the SDK. I'm a bit stuck here. I've tested it quite thoroughly, and with the current code I have, I am able to download a file from my S3 bucket without problems or corruptions, however it's ALWAYS limited to 64 megabytes.
Is there some way to up this limit? Or increment the download?
When I try to download a file over 64 megabytes the page cannot be reached. Sometimes it might actually download the file anyway (while it says cannot be reached), but only exactly 64 megabytes.
try {
$result = $s3->getObject([
'Bucket' => $bucket,
'Key' => $keyname
]);
set_time_limit(0);
header('Content-Description: File Transfer');
header("Content-Type: {$result['ContentType']}; charset=utf-8");
header("Content-Disposition: attachment; filename=".$filename);
echo $result['Body'];
} catch (S3Exception $e) {
echo $e->getMessage() . PHP_EOL;
}
I've set my memory_limit to no limit (0). I've also tried to set the memory limit to about 64 megabytes, but still no dice.
Tried tinkering with post_max_size etc, but still nothing. I'm not sure if the problem relies on my apache/php setup, the EC2 I'm running, or S3 SDK limitations.
The EC2 instance I'm running is a t2.xlarge, running: Ubuntu 20.04 LTS (Focal Fossa) LAMP Stack - Linux Apache MySQL/MariaDB PHP
Some of the things I've found with similar issues (The first one I've tried without luck):
Download large files from s3 via php <-- This link has solution
Max execution time out error when tried to download large object(2gb) from s3 bucket to window server using php
The bottom link I don't really understand, apparently the solution should be to increment the download (according to online Thomas), but I'm not sure how that would work. How would I combine the data, and how would I keep downloading from where I left off? I'm missing an example of how to work that solution. The OP of that post asked the same question.
The presigned request solution from OP from the first link, did solve my problem. However it's a different "way of downloading", before in my code we were using the SDK to download via the SDK, now we are using the SDK to create basically a download link. The 64 megabyte issue still persists. I can use this, but if anyone has a solution for how to download via the SDK over 64 megabytes, please let me know!
My solution:
$cmd = $s3->getCommand('GetObject', [
'Bucket' => $bucket,
'Key' => $keyname,
'ResponseContentDisposition' => 'attachment; filename="'.$filename.'"'
]);
$request = $s3->createPresignedRequest($cmd, '+15 min');
$presignedUrl = (string)$request->getUri();
And then basically just open that URL anywhere in HTML, JS etc. and it will begin to download the file.
This is regards users' uploads - which are hosted in an S3 bucket - and the best approach for downloading them. Currently I use the following:
return response()->streamDownload(function(){
// Fetch the file
print Storage::disk('s3')->get('file');
}, 'file-name.ext');
This works just fine, but as you can see, the file is first fetched from S3, then streamed via the server to the user's browser. Which, I think, unnecessary work (calls to S3 and bandwidth), since we could just force-download it off S3 servers instead.
I have two questions here: How to force-download the file off s3, and more importantly; am I giving this too much thought? But I really hate the idea of downloading the file twice and putting more pressure on the server!
The pre-signed urls feature was the way to go! Here is the code:
$url = Storage::disk('s3')->temporaryUrl($path, now()->addMinutes(5), [
'ResponseContentDisposition' => "attachment; filename=$fileName.txt"
]);
I would like people to upload any size file to my site but I need only first 1 kB of it. So I would like PHP to somehow stop receiving a file after it got this 1 kB and then just process this truncated file.
Try to check this question, Maybe it has the answer for you.
Receiving only chunks of an uploaded file in PHP?
Version 2. If you cant do it with basic php support, then:
Create a new php, with a tcp socket server. Start the socket on another port, for example 8080, and that will handle your file upload.
You wait for start of the content (trim headers, and other unneeded data.), and when you got 1kb of uploaded file, you can parse it, and maybe send a redirection back to client, and handle that data.
So a little tricky, but not impossible.
I have written a Symfony2(PHP MVC Framework) script to download a zip file from the server. But the file download stops in the midway. I have increased the max_execution_time in apache configuration. Still the problem is persisting.
Do anyone have the quick fix for this?
Thanks in advance.
It seems like you may have an issue with a large file (downloading an archive of videos). You should use a StreamedResponse. This way, you don't have to store the entire contents of your file in memory, it will just stream to the client. The way you are currently doing it makes the file load into memory before it can start to download. You can see why this could be a problem. Here is a simple example of how you can stream a file to the client:
$path = "//usr/www/users/jjdqlo/Wellness/web/yoga_videos/archive.zip";
return new StreamedResponse(
function () use ($path) { // first param is a callback, where you do the readfile()
readfile($path);
},
200, // second param is the http status code
array( // third param is an array of header settings
'Content-Disposition' => 'attachment;filename="archive.zip"',
'Content-Type' => 'application/zip'
)
);
Give this a shot. Assuming the problem is because of file size, this should solve the issue.
I am having trouble uploading files to S3 from on one of our servers. We use S3 to store our backups and all of our servers are running Ubuntu 8.04 with PHP 5.2.4 and libcurl 7.18.0. Whenever I try to upload a file Amazon returns a RequestTimeout error. I know there is a bug in our current version of libcurl preventing uploads of over 200MB. For that reason we split our backups into smaller files.
We have servers hosted on Amazon's EC2 and servers hosted on customer's "private clouds" (a VMWare ESX box behind their company firewall). The specific server that I am having trouble with is hosted on a customer's private cloud.
We use the Amazon S3 PHP Class from http://undesigned.org.za/2007/10/22/amazon-s3-php-class. I have tried 200MB, 100MB and 50MB files, all with the same results. We use the following to upload the files:
$s3 = new S3($access_key, $secret_key, false);
$success = $s3->putObjectFile($local_path, $bucket_name,
$remote_name, S3::ACL_PRIVATE);
I have tried setting curl_setopt($curl, CURLOPT_NOPROGRESS, false); to view the progress bar while it uploads the file. The first time I ran it with this option set it worked. However, every subsequent time it has failed. It seems to upload the file at around 3Mb/s for 5-10 seconds then drops to 0. After 20 seconds sitting at 0, Amazon returns the "RequestTimeout - Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed." error.
I have tried updating the S3 class to the latest version from GitHub but it made no difference. I also found the Amazon S3 Stream Wrapper class and gave that a try using the following code:
include 'gs3.php';
define('S3_KEY', 'ACCESSKEYGOESHERE');
define('S3_PRIVATE','SECRETKEYGOESHERE');
$local = fopen('/path/to/backup_id.tar.gz.0000', 'r');
$remote = fopen('s3://bucket-name/customer/backup_id.tar.gz.0000', 'w+r');
$count = 0;
while (!feof($local))
{
$result = fwrite($remote, fread($local, (1024 * 1024)));
if ($result === false)
{
fwrite(STDOUT, $count++.': Unable to write!'."\n");
}
else
{
fwrite(STDOUT, $count++.': Wrote '.$result.' bytes'."\n");
}
}
fclose($local);
fclose($remote);
This code reads the file one MB at a time in order to stream it to S3. For a 50MB file, I get "1: Wrote 1048576 bytes" 49 times (the first number changes each time of course) but on the last iteration of the loop I get an error that says "Notice: fputs(): send of 8192 bytes failed with errno=11 Resource temporarily unavailable in /path/to/http.php on line 230".
My first thought was that this is a networking issue. We called up the customer and explained the issue and asked them to take a look at their firewall to see if they were dropping anything. According to their network administrator the traffic is flowing just fine.
I am at a loss as to what I can do next. I have been running the backups manually and using SCP to transfer them to another machine and upload them. This is obviously not ideal and any help would be greatly appreciated.
Update - 06/23/2011
I have tried many of the options below but they all provided the same result. I have found that even trying to scp a file from the server in question to another server stalls immediately and eventually times out. However, I can use scp to download that same file from another machine. This makes me even more convinced that this is a networking issue on the clients end, any further suggestions would be greatly appreciated.
This problem exists because you are trying to upload the same file again. Example:
$s3 = new S3('XXX','YYYY', false);
$s3->putObjectFile('file.jpg','bucket-name','file.jpg');
$s3->putObjectFile('file.jpg','bucket-name','newname-file.jpg');
To fix it, just copy the file and give it new name then upload it normally.
Example:
$s3 = new S3('XXX','YYYY', false);
$s3->putObjectFile('file.jpg','bucket-name','file.jpg');
now rename file.jpg to newname-file.jpg
$s3->putObjectFile('newname-file.jpg','bucket-name','newname-file.jpg');
I solved this problem in another way. My bug was, that filesize() function returns invalid cached size value. So just use clearstatcache()
I have experienced this exact same issue several times.
I have many scripts right now which are uploading files to S3 constantly.
The best solution that I can offer is to use the Zend libraries (either the stream wrapper or direct S3 API).
http://framework.zend.com/manual/en/zend.service.amazon.s3.html
Since the latest release of Zend framework, I haven't seen any issues with timeouts. But, if you find that you are still having problems, a simple tweak will do the trick.
Simply open the file Zend/Http/Client.php and modify the 'timeout' value in the $config array. At the time of writing this it existed on line 114. Before the latest release I was running at 120 seconds, but now things are running smooth with a 10 second timeout.
Hope this helps!
There are quite a bit of solutions available. I had this exact problem but I don't wanted to write a code and figure out the problem.
Initially I was searching for a possibility to mount S3 bucket in the Linux machine, found something interesting:
s3fs - http://code.google.com/p/s3fs/wiki/InstallationNotes
- this did work for me. It uses FUSE file-system + rsync to sync the files in S3. It kepes a copy of all filenames in the local system & make it look like a FILE/FOLDER.
This saves BUNCH of our time + no headache of writing a code for transferring the files.
Now, when I was trying to see if there is other options, I found a ruby script which works in CLI, can help you manage S3 account.
s3cmd - http://s3tools.org/s3cmd - this looks pretty clear.
[UPDATE]
Found one more CLI tool - s3sync
s3sync - https://forums.aws.amazon.com/thread.jspa?threadID=11975&start=0&tstart=0 - found in the Amazon AWS community.
I don't see both of them different, if you are not worried about the disk-space then I would choose a s3fs than a s3cmd. A disk makes you feel more comfortable + you can see the files in the disk.
Hope it helps.
You should take a look at the AWS PHP SDK. This is the AWS PHP library formerly known as tarzan and cloudfusion.
http://aws.amazon.com/sdkforphp/
The S3 class included with this is rock solid. We use it to upload multi GB files all of the time.