By default you cannot access a file that is uploaded until it has been fully transferred to the server.
What is the best way to get round this and be able to access the 'byte stream' as the file upload is in progress?
I think the closest you will get to this in PHP is looking at the various progress bar solutions that give you progress bars while doing file uploads to a PHP script. I don't know any of them in detail (I waited until SWFUpload became available to do this in Flash) but if you can get an upload's progress, it's likely you can get the data chunks themselves, too. (Update: the APC based solution can't. It seems to give the upload progress only.)
Check out this question, especially this answer and this one.
None of the approaches runs in pure vanilla PHP, they all need Perl or APC or other server-side stuff to work.
Final update: A cursory glance at the Raditha Mega Upload progress bar's Perl code makes me think this is your best bet, as it seems to be interacting with the actual file that is being uploaded. No guarantees, though.
You will need to have some kind of buffer where new data will be appended and where you read in chunks. Then check if there is new data and read that again.
Related
I want to allow uploads of very large files into our PHP application (hundred of megs - 8 gigs). There are a couple of problems with this however.
Browser:
HTML uploads have crappy feedback, we need to either poll for progress (which is a bit silly) or show no feedback at all
Flash uploader puts entire file into memory before starting the upload
Server:
PHP forces us to set post_max_size, which could result in an easily exploitable DOS attack. I'd like to not set this setting globally.
The server also requires some other variables to be there in the POST vars, such as an secret key. We'd like to be able to refuse the request right away, instead of after the entire file is uploaded.
Requirements:
HTTP is a must.
I'm flexible with client-side technology, as long as it works in a browser.
PHP is not a requirement, if there's some other technology that will work well on a linux environment, that's perfectly cool.
upload_max_filesize can be set on a per-directory basis; the same goes for post_max_size
e.g.:
<Directory /uploadpath/>
php_value upload_max_filesize 10G
php_value post_max_size 10G
</IfModule>
Python Handler?
Using a Python POST handler instead of PHP. Generate a unique identifier from your PHP app that the client can put in the HTTP headers. With mod_python to reject or accept the large upload before the entire POST body is transmitted.
I think
http://www.modpython.org/live/current/doc-html/dir-handlers-hph.html
Allows you to check headers and decline the rest of the POST input. I haven't tried it but might be the right path?
Looking at the source of mod_python, the buffering of the input via read() seems to allow bit-at-a-time evaluation of the HTTP input. Headers are first.
https://svn.apache.org/repos/asf/quetzalcoatl/mod_python/trunk/src/filterobject.c
It's old I know, but maybe someone have this problem nowdays ,too.
Now you can do this with only Javascript and, say, PHP. No Flash or Java required on client side.
demo: http://dnduploader.filkor.org/
The idea is to slice the files with Javascript's Blob slice() method...
How about a Java applet? That's how we had to do it at a company I previously worked for. I know applets suck, especially in this day and age with all our options available, but they really are the most versatile solution to desktop-like problems encountered in web development. Just something to consider.
You can set the post_max_size for just scripts in 1 directory. Place your upload script there, and allow only that script to handle large sizes. It's still possible for that script to be attacked with large/useless files, but it avoids setting it globally.
Use that with APC and you might be able to work out something good:
IBM Developer works article on APC
Tried all of this... this is by far the best I have used yet...
http://www.uploadify.com/
Take a look at jumploader.com
A good java-applet for uploading.
I've used it for uploading images and it works fine. Haven't tried with bigger files than 10MB, but i should work for really big files too.
Have you looked into using APC to check the progress and total file size. Here is a good blog post about it. It might help.
Maybe you could use Webdav and Javascript in the browser
AJAX Big file upload, with progress, to WebDAV
http://www.webdavsystem.com/ajax/programming/upload_progress
A simple library
http://debris.demon.nl/projects/davclient.js/doc/README.html
You can then get the JS to redirect the user to a success page. Secret keys and what-not can be handled in a PHP prelude before handing off the JS Client->WebDAV
I would look into FTP, SSH or SCP this allows you to upload a large file and still have access control over the file as well. This might take a little longer to implement but its probably the most secure way I could think of.
I know it sucks to add another dependency but in my experience, most websites that are doing something like this are using flash on the client side, and uploading the large file as chunks
adobe as a howto on flash file uploads
I also found this tutorial on codeproject:
Multiple File Upload With Progress Bar Using Flash and ASP.NET
PS - I know you're using PHP and not .net, I figured the important part was the flash ;)
I've had success with uploadify, and I would recommend it. It's a jQuery/Flash script that handles large uploads, and you can pass extra parameters to it (like the secret key). To solve the server-side issues, simply use the following code. The changes take affect just for the script they're called in:
//Check to see if the key is there
if(!isset($_POST['secret_key']) || !isValid($_POST['secret_key']))
{
exit("Invalid request");
}
function isValid($key)
{
//Put your validation code here.
}
//This line changes the timeout.
//Give it a value in seconds (3600 = 1 hour)
set_time_limit(3600);
//Set these amounts to whatever you need.
ini_set("post_max_size","8192M");
ini_set("upload_max_filesize","8192M");
//Generally speaking, the memory_limit should be higher
//than your post size. So make sure that's right too.
ini_set("memory_limit","8200M");
EDIT In response to your comment:
Given what you've said, I'm afraid you may not be able to meet your requirements over http. All of the solutions out there are code that add features to http that it was never designed for.
Like you said yourself, it's a simple protocol. Apart from writing your own client software that runs outside of the browser, a java applet, or using a different protocol (like FTP, which was designed for this), you might not get what you want.
I've done the best I could within the given constraints. Sorry I couldn't do better.
Try this: http://www.simple2ftp.com uses a Java based FTP applet from within a clever PHP application wrapper.
I'm trying to figure out if there's a way to get the progress of a file upload with PHP and/or Kohana. My script can upload images, videos, zip, exe, whatever I want really. However the larger the file the longer the user has to wait without any indication.
I was hoping to use some AJAX here to initialise the upload and then report back the progress.
Is this possible with PHP... and can anyone give me an indication of where to start looking.
there is a file upload progress extension for php, see http://www.ultramegatech.com/blog/2010/10/create-an-upload-progress-bar-with-php-and-jquery/ for how to use it.
I like to use a server module to do this sort of thing, mainly because it makes my life as a web developer easier if all I need to do is grab upload statistics from a URL. Nothing has to be changed in your website.
For Nginx there is the Upload Progress module and it should work on all recent releases. You can find code examples on the Nginx Wiki: http://wiki.nginx.org/HttpUploadProgressModule
For Apache there is the Upload Progress too. I haven't used it myself, but it seems fairly straight forward.
If you don't have access to the server configuration, then you might want to fall back onto a pure flash / javascript solution. For this I had good luck with Uploadify in the past, but it requires a bit more work as you now have to upload files in a separate request. Someone should be able to suggest a good HTML5 upload progress plugin too.
you could do this also with apache and APC example
When the user selects a file to be uploaded, is there a way I can get the exact size of this file before the upload even begins? I'm guessing this needs to be done on the client side with jQuery or JavaScript. Any ideas how?
This cannot be done in pure Javascript in current browsers.
Instead, you can use Uploadify, which uses Flash.
In non-IE browsers, you can also use HTML5 to read files on the client.
$("#file_input_selector").bind('change', function(){
alert(this.files[0].size);
});
Not sure of all the compatibility issues, but this seems to be working just fine for me.
Take a look at this post:
http://forums.digitalpoint.com/showthread.php?t=6704
Javascript doesn't have the ability to check file sizes (or access the file system for that matter). You'll need to upload the file to get the size
I suggest you look at the HTML5 File API. This, combined with some JS might be able to help you. I only say might because I have not yet had a chance to browse at this part of the HTML5 standard.
http://www.w3.org/TR/FileAPI/#dfn-filereader
The way PHP file uploads work, it is very hard to check file details before, or during a file upload (since the file is uploaded before your code even gets loaded).
I know it is possible to do some fancy things in some other languages (possibly Perl or Python) that handle the file uploading directly with the script (where the script opens the socket and handles the whole transfer itself), however PHP does this for you and accepts any file on your script's behalf. The file gets discarded if it is not within PHP's acceptable limits, but only after the file is completely uploaded.
There have also been several file upload implementations made using Flash, but not being an ActionScript coder, I can't really help too much there either.
hey guys,
i know there are a lot of "for-me-too-complicated" versions of progress-bars for php uploads out there.
however i have only a really basic knowledge of php and i have no idea how to implement this stuff.
i did a working file-upload script that transferres files from the user to my ftp-server. i'm using ftp_connect and ftp_put to do so.
i wonder how complicated it is to print a SIMPLE percentage value on to the page, to let the user know how far the upload has progressed.
i don't want any animated javascript stuff, just a simple percentage that shows the progress.
do you know a tutorial or something, or can you maybe give me a little explanation how i could do that. at least which methods return a progress value.
thank you in advance,matt
I know you said the flash uploaders are too complicated for you and you need a simple solution but the truth is there are none. If you could start your project over I would recommend using some known CMS with file upload support.
I think you should really give something like uploadify another chance. If you have problems with it ask here! There is a uploadify tag and really helpful peoples.
edit after your commenht: As seen on this page theres the idea to use uploadify to get the file to yoru server and then move it normally using ftp to your other space/server.
PHP/Apache talks to the client in a single request only. There is no simple way to actually have the client (know) how far the server is in the process. On uploads the file travels from the client to the server, so we generaly use Flash that can give us that information.
client (flash) -> server
What you are asking is something a bit fancier
client -> server -> ftp
And you want to know the progress between the server and the FTP. Mind you that even if you don't realize it the files are actually beeing transfered to the server and then from the server to the FTP server.
You will probably want to have the server update a database on given intervals with the progress so far and have the client AJAX the server to find out where the server is at.
You can also give socket.io a look!
I code primarily in PHP and Perl. I have a client who is insisting on seeking video submissions (any encoding) from the public via one of their pages rather than letting YouTube do its job.
Server in question is a virtual machine and I can adjust ini settings for max post, max upload size etc as needed.
My initial thought is to use a Flash based uploader with PHP on the back end but I wondered if someone might have useful advice and experience on the subject?
Doing large file transfers of HTTP is not usually fun -- but sometimes it's necessary.
For large files, you'll definitely want to provide some kind of progress gauge for end-users.
There are flash-based tools that do this (swfUpload comes to mind).
If you want to avoid flash and do it with pretty html/javascript/css, you can leverage PHP's APC extension, which for some reason provides support for getting upload status from the server, as explained here
You can adjust the post size and use a normal html form. The big problem is not Apache, its http. If anything goes wrong in the transmission you will have no way to detect the error. Further more there is no way to resume the transfer. This is exactly why BitTorrent is so popular.
I don't know how against youtube your client is, but you can use their api to do the uploads from a page on your site.
http://code.google.com/apis/youtube/2.0/developers_guide_protocol.html#Uploading_Videos
See: browser based uploading.
For web-based uploads, there's not many options. Regardless of web platform, web server, etc. you're still transferring over HTTP. The transfer is all or nothing.
Your best option might be to find a Flash, Java, or other client side option that can chunk files and upload them piecemeal, then do a checksum to verify. That will allow for resuming uploads. Unfortunately, I don't know of any such open source component that does this.
Try to convince your client to change point of view.
Using http (and the browser, hell, the browser!) for this kind of issue is rarely a good deal; Will his users wait 40 minutes with the computer and the browser running until the upload is complete?
I dont think so.
Maybe, you could set up a public ftp account, where users can upload but not download and see the others user's files.. then, who want to use FTP software can, who like to do it via browser can too.
The big problem dealing using a browser is that, if something go wrong, you cant resume but have to restart from zero again.
the past year i had the same issue, i gave a look to ZUpload
, but i didnt use it so i can suggest (we wrote a small python script that we send to our customer; the python script create a torrent of the folder our costumer need to send to us, and we download it via utorrent ;)
p.s: again, sorry for my bad english ;)
I used jupload. Yes it looks horrible, but it just works.
With that said, it's still a better idea to convince the client that doing so is stupid.
I would agree with others stating that using HTML is a poor option. I believe there is a size limitation using Flash as well. I know of a script that uses a JavaScript Applet to perform an actual FTP transfer. It is called Simple2FTP and can be found at http://www.simple2ftp.com
Not sure but perhaps worth a try?