I am building a large website where members will be allowed to upload content (images, videos) up to 20MB of size (maybe a little less like 15MB, we haven't settled on a final upload limit yet but it will be somewhere between 10-25MB).
My question is, should I go with HTTP or FTP upload in this case. Bear in mind that 80-90% of uploads will be smaller size like cca 1-3MB but from time to time some members will also want to upload large files (10MB+).
Is HTTP uploading reliable enough for such large files or should I go with FTP? Is there a noticeable speed difference between HTTP and FTP while uploading files?
I am asking because I'm using Zend Framework which already has HTTP adapter for file uploads, in case I choose FTP I would have to write my own adapter for it.
Thanks!
HTTP definitely puts less of a burden on your clients. A lot of places have proxies or firewalls that block all FTP traffic (in or out).
The big advantage of HTTP is that it goes over firewalls and it's very easy to encrypt---just use HTTPS on port 443 instead of HTTP on port 80. Both go through proxies and firewalls. And these days it's pretty easy to upload a 20MB files over HTTP/HTTPS using a POST.
The problem with HTTP is that it is not restartable for uploads. If you get 80% of the file sent and then there is a failure, you will need to restart at the beginning. That's why vendors are increasingly using flash-based, java-based or javascript-based uploaders and downloaders. These systems can see how much of the file has been sent, send a MAC to make sure that it has arrived properly, and resend the parts that are missing.
A MAC is more important than you might think. TCP checksums are only 32 bits, so there is a 1-in-4-billion chance of an error not being detected. That potentially happens a lot with today's internet.
Is HTTP uploading reliable enough for
such large files
One major advantage of FTP would be the ability to resume aborted uploads. Most FTP servers and clients support this, though it's not always activated. Whereas with HTTP, it's theoretically possible using special headers, but a normal client (i.e. browser) will not support it.
Another advantage would be bulk uploads: very simple in FTP, not so in HTTP.
But why not simply offer both options? HTTP for those who are behind proxies or won't/can't use an FTP client, and FTP for people who have to do upload many or large uploads over unreliable connections.
I do not want to be sarcastic, but File Transfer Protocol must be more reliable on file transfer :)
Resource availability / usage is more of an issue than reliability or speed. Each upload consumes resources - thread / memory / etc - on your web server for the duration of the upload. If content upload traffic is significant for large files it would be better to use FTP simply to free your HTTP server to be more responsive to page requests.
I definitely, opt for the HTTP approach as the rest of the people here. The reason for this is what you've said about most of the files being from one to three megabytes.
The problem is for the "rest", so:
Have you considered allowing users to send larger files through e-mail to a deamon script that gets the emails and uploads the emails to the account associated with the sender?
Or there is the solution of the flash uploader, in a facebook-like approach.
FTP will consume less bandwidth than HTTP, since the latter will need to encode(base64) the binary content into plain text thus increase the total transfer size. (by 1/3).
However, bandwidth consumption might not necessarily be the major concern, compare to other factors like usability and security, in which HTTP prevail.
Related
I have a page that uploads a file to my server, where it then gets copied to a permanent directory via move_uploaded_file. This all seems to work great, with the exception that in a real-life scenario I will be expecting much larger files than I have successfully sent up.
I have already tacked the timeout for the file upload by changing the connection timeout in my site settings in IIS - so the file continues to upload up to six hours ( -_- ) - but this is where I run into my current problem - It might just take six hours!
After getting the upload process to get past 10% or so ( on a 300 meg file ), I noticed that the file continues to push up, but my upload rate seems to be 'falling off' - as in, I observed faster speeds when I started the transfer, than I am seeing halfway through it. The numbers here aren't necessarily relevant, as I know that my upload ( while Im uploading, still 2 Mbps ) is capable of pushing faster than it is, and the server on the other end is on fiber.
I wonder if anyone has encountered this before, and if so, have you determined a work-around. Any help appreciated. Thanks.
You should not be using HTTP for this task. You may have observed that all the "file locker" services (and others which involve uploading files, such as Apple's online-music service) provide you with an "uploader" program rather than making use of the browser. There are reasons for this.
First off, the overhead of the transfer encoding is large. You take your (presumably binary) data, and Base64 encode it; that's 33% overhead. So if it would take four hours with HTTP, it would only take three with a binary protocol - and that's disregarding the chunked-transfer overhead, so the reality is probably more severe.
Second, there's no way to "resume" an upload in HTTP. So if your connection is broken, you'll either have to write application-specific code to handle the resumption, or start all over.
Third, HTTP servers are not designed for super-long-lived connections: they usually have a finite or small pool of workers to service the (usually seconds-long at the outset) client requests, and occasionally they have smallish limits on the size of request data (2GB is common, and PHP by default has only a few MB).
I strongly recommend using a file transfer protocol to transfer files (such as FTP). You don't have to give out a single username/password pair to everyone: you can have a gatekeeper which integrates with whatever authentication system you already have in place. FTP-over-TLS also exists and is relatively mature.
There is a fairly good summary of the differences between the two protocols here. Note that you gain nothing from any of the advantages of HTTP listed, due to your circumstances.
Don't feel limited to FTP - rsync is a great protocol for transferring files as well, especially if you only change part of the file (it even does binary deltas!). git can also efficiently transport large blobs over secure connections or even HTTP, if you insist on using that.
I am debating whether to use FTP or HTTP for large file upload and downloads. File uploads will consists of text and audio files, from a range of a couple of KB - 200MB. I have a couple of questions, such as:
Which would be faster? HTTP or FTP
Which would be more reliable?
Which would be of greater ease and convenience for the end user?
What other alternatives are there for larger file uploads.
Note: I need to somehow keep track of what files what users upload.
Thanks a bunch!
In my opinion File transfers which are part of a website navigation should use the same protocol, as switching protocol may require additional work on your server, and it will open a new connection.
Moreover, HTTP supports POST and PUT methods which are designed for that goal.
If you simply want to upload files, then FTP is the dedicated protocol, but it may not be implemented on all web browser.
I am a developing a website that involves uploading videos above 50MB.
Which is a better (faster) way of uploading the files to server:
uploading the video files via ftp
or
uploading the files via a form
Thanks
The best way would be with FTP.
FTP is much faster for larger file sizes. File sizes that are below 1MB won't matter as much.
P.S. If you are not the one uploading, then think which is easier for your users. Form is easier but ftp is still faster.
For user experience you should go with the form file upload; The speed of both depend on the internet connection speed and load of the server and client and won't differ that much. It might be a bit much for your webserver if it's handling a lot of users but you can use for example nginx to make that less of a problem.
edit:
here a comparison: http://daniel.haxx.se/docs/ftp-vs-http.html
I use Jupload
It splits the files and uploads them via http. It's also good because you don't need to care about file upload limitations in server config. Speed depends mostly on the client connection info both for HTTP and FTP. Of course there are some differences but not at all so big between them.
Why not offer both? (Seriously - I wrote an app about ten years ago that did this.) Look up "MOVEit DMZ" or research various FTP servers with web portal integration to see how it's being done today.
There's also a third way you should consider and was touched on by the Jupload comment: a local control (Flash, Java, ActiveX, Firefox plug-in, etc. that optimizes the upload experience). If people are uploading multiple large files to your site they may appreciate the speed/reliability boost.
For what reasons would it be 'bad' to upload big files (10 megs) via a normal file HTML tag on a server ?
(Of course, to do that, some PHP settings has to be changed)
Maybe it uses too much RAM for the server when downloading, or else ?
There is no general reason why this would be "bad".
However, a hosting provider with dozens or hundreds of customers on one server might feel compelled to set the maximum upload size to such a low limit, in order to prevent overloading through many concurrent uploads on many sites.
These uploads have to be temporarily stored somewhere, and twenty concurrent connections uploading a 200MB file each might put a lot of stress on the server.
I created an simple web interface to allow various users to upload files. I set the upload limit to 100mb but now it turns out that the client occasionally wants to upload files 500mb+.
I know what to alter the php configuration to change the upload limit but I was wondering if there are any serious disadvantages to uploading files of this size via php?
Obviously ftp would be preferable but if possible i'd rather not have two different methods of uploading files.
Thanks
Firstly FTP is never preferable. To anything.
I assume you mean that you transferring the files via HTTP. While not quite as bad as FTP, its not a good idea if you can find another of solving the problem. HTTP (and hence the component programs) are optimized around transferring relatively small files around the internet.
While the protocol supports server to client range requests, it does not allow for the reverse operation. Even if the software at either end were unaffected by the volume, the more data you are pushing across the greater the interval during which you could lose the connection. But the biggest problem is that caveat in the last sentence.
Regardless of the server technology you use (PHP or something else) it's never a good idea to push that big file in one sweep in synchronous mode.
There are lots of plugins for any technology/framework that will do asynchronous upload for you.
Besides the connection timing out, there is one more disadvantage in that file uploading consumes the web server memory. You don't normally want that.
PHP will handle as many and as large a file as you'll allow it. But consider that it's basically impossible to resume an aborted upload in PHP, as scripts are not fired up until AFTER the upload is completed. The larger the file gets, the larger the chance of a network glitch killing the upload and wasting a good chunk of time and bandwidth. As well, without extra work with APC, or using something like uploadify, there's no progress report and users are left staring at a browser showing no visible signs of actual work except the throbber chugging away.