So far I've figured out HOW to upload files asynchronously with Ajax and PHP, no problem there. But I want to get the percentage of the file that's already been uploaded, as it's uploading, and, after hours of research, I can't find a good way to do this without cheating.
Some implementations I've seen used Flash to upload, and getting the percentage in Flash is apparently fairly common, but I'd like to avoid this if I can.
Any ideas?
The core problem is that RFC 1867, the specification for file uploads over HTTP via the multipart/form-data MIME type, does not provide any method for providing file upload progress.
A file upload is actually just a fancy form submit. CGI scripts, PHP, and all other web technologies that rely on a front-end web server to first accept the request might not actually begin executing until the entire upload has completed. This means that they generally can't even know when the upload has started, only when it's been completed.
New versions of PHP's APC extension include a workaround for this problem that performs some level of black magic that allows it to know about uploads earlier. It only works as part of mod_php, though. The devs don't seem to have plans to support it under FastCGI.
Another server-side option would be the "uploadprogress" PECL extension. I'm not entirely sure what kind of black magic it uses. The source suggests that it actually hooks into the processing of the multipart MIME parts. (This suggests that at least some SAPIs stream form data to PHP as the client uploads it. I know that at least some FastCGI servers buffer the entire request before passing it along, so this might not work for you. YMMV.)
Both of these options are for normal file uploads. Ajax -- or rather, XMLHttpRequest -- does not support file upload operations. Most of the workarounds in this area involve creating an iframe and submitting a form there, and that also implies someone else's client-side work. If you're going to go through that level of hoop jumping, you'd may as well use one of the modern file upload widgets.
Personally, I use Plupload, a Javascript widget that can work with everyone's favorite Javascript library, jQuery. Some others swear by Uploadify. Regardless, both of these widgets offer a high degree of user feedback as to upload progress. They are likely to be easier for you to implement than APC or uploadprogress and have the advantage of being built and tested thoroughly by other people.
Plupload supports multiple upload engines, including HTML5, Gears, Flash, Silverlight, oldschool HTML4 and more. Between HTML5, Flash and Silverlight, you've pretty much just covered 100% of your audience. It also allows you to subscribe to events and have your own code perform magic. For example, if you need server-side file upload progress information, you can have the client regularly send updates to a different script. This would be useful if you regularly have clients uploading huge files and you want to know about it in real time.
tl;dr: Uploading is hard, let's go client-side!
Yeah,I dont like that "cheating" method either, In my opinon, the best method is to use APC , and its method, apc_fetch
Using ajax to make a apc_fetch, with a unique key specifying the upload, will return what you need .. ie bytes uploaded / total bytes.
Then simply do a progress bar with javascript.
I have heard chrome and safari dont allow you to do ajax calls during post upload, the work arround includes using an iframe to do the calls with the apc identifier.
Related
As all of us know, PHP finishes the upload and the enables you to use move_uploaded_file(); before this, however, it creates a temp file and then does the job. I want to know is it possible to get the name of this uploaded file during the file upload and before populating it into $_FILES?
I want to get the upload progress, while $_SESSION and Javascript onprogress solution both suck..
$_FILES['file']['tmp_name']; is the filename. It is not possible in PHP (without using ugly tricks) to get the filename before the upload is finished.
To do this, you have to fallback on either Flash (uploadify) or CGI (Perl / Python / C++ / Other)
A "reliable" progress bar, which seems to be your goal, will always require some sort of server and client support. In its most general and portable instance, PHP will see only the completed upload and you'll get no progress bar, but only the filled $_FILES structure.
On some platforms the information can be garnered from the system itself. For example under Linux/Apache you can inspect what temporary files Apache has opened in the /proc pseudo-filesystem, where available; so you need to put in the requisites "Linux, Apache, php5_module, /proc".
You can use a dedicated POST endpoint that does not terminate on the Web server, but on a specially crafted uploader process (I worked on a Perl script doing this years ago; I recall it used POE, and the architecture):
POST (from browser) ==> (server, proxying) ==> UPLOADER
The uploader immediately echoes a crafted GET to the server, activating
a PHP "pre-upload" page, and then might call a progress GET URL periodically
to update the upload status. When completed, it would issue a pseudo POST
to PHP "almost" as if it came from the client, sending $_POST['_FILES']
instead of $_FILES.
The $_SESSION solution is a good compromise but relies on the server not doing buffering.
A better and more "modern" solution would be to leverage the chunked upload AJAX trick and get resumable uploads, reliable progress and large file support all in one nifty package. See for example this other answer. Now you get wider server support but the solution won't work on some older browsers.
You could offer the user the choice between old-style FILE upload, Flash uploader (which bypasses all problems as it doesn't rely on the browser but on Flash code), Java FTP upload control (same thing, but sometimes with some protocol and firewall issues since it doesn't use HTTP as the container web page does), and AJAX HTML5 chunking, possibly based on browser capabilities.
I.e., a user with IE6 would see a form saying
SORRY!
Your browser does not support large file uploads and progress bar.
To send a file of no more than XXX meg,
[ ] [Choose file...] [ >> BEGIN UPLOAD >>> ]
I want to allow uploads of very large files into our PHP application (hundred of megs - 8 gigs). There are a couple of problems with this however.
Browser:
HTML uploads have crappy feedback, we need to either poll for progress (which is a bit silly) or show no feedback at all
Flash uploader puts entire file into memory before starting the upload
Server:
PHP forces us to set post_max_size, which could result in an easily exploitable DOS attack. I'd like to not set this setting globally.
The server also requires some other variables to be there in the POST vars, such as an secret key. We'd like to be able to refuse the request right away, instead of after the entire file is uploaded.
Requirements:
HTTP is a must.
I'm flexible with client-side technology, as long as it works in a browser.
PHP is not a requirement, if there's some other technology that will work well on a linux environment, that's perfectly cool.
upload_max_filesize can be set on a per-directory basis; the same goes for post_max_size
e.g.:
<Directory /uploadpath/>
php_value upload_max_filesize 10G
php_value post_max_size 10G
</IfModule>
Python Handler?
Using a Python POST handler instead of PHP. Generate a unique identifier from your PHP app that the client can put in the HTTP headers. With mod_python to reject or accept the large upload before the entire POST body is transmitted.
I think
http://www.modpython.org/live/current/doc-html/dir-handlers-hph.html
Allows you to check headers and decline the rest of the POST input. I haven't tried it but might be the right path?
Looking at the source of mod_python, the buffering of the input via read() seems to allow bit-at-a-time evaluation of the HTTP input. Headers are first.
https://svn.apache.org/repos/asf/quetzalcoatl/mod_python/trunk/src/filterobject.c
It's old I know, but maybe someone have this problem nowdays ,too.
Now you can do this with only Javascript and, say, PHP. No Flash or Java required on client side.
demo: http://dnduploader.filkor.org/
The idea is to slice the files with Javascript's Blob slice() method...
How about a Java applet? That's how we had to do it at a company I previously worked for. I know applets suck, especially in this day and age with all our options available, but they really are the most versatile solution to desktop-like problems encountered in web development. Just something to consider.
You can set the post_max_size for just scripts in 1 directory. Place your upload script there, and allow only that script to handle large sizes. It's still possible for that script to be attacked with large/useless files, but it avoids setting it globally.
Use that with APC and you might be able to work out something good:
IBM Developer works article on APC
Tried all of this... this is by far the best I have used yet...
http://www.uploadify.com/
Take a look at jumploader.com
A good java-applet for uploading.
I've used it for uploading images and it works fine. Haven't tried with bigger files than 10MB, but i should work for really big files too.
Have you looked into using APC to check the progress and total file size. Here is a good blog post about it. It might help.
Maybe you could use Webdav and Javascript in the browser
AJAX Big file upload, with progress, to WebDAV
http://www.webdavsystem.com/ajax/programming/upload_progress
A simple library
http://debris.demon.nl/projects/davclient.js/doc/README.html
You can then get the JS to redirect the user to a success page. Secret keys and what-not can be handled in a PHP prelude before handing off the JS Client->WebDAV
I would look into FTP, SSH or SCP this allows you to upload a large file and still have access control over the file as well. This might take a little longer to implement but its probably the most secure way I could think of.
I know it sucks to add another dependency but in my experience, most websites that are doing something like this are using flash on the client side, and uploading the large file as chunks
adobe as a howto on flash file uploads
I also found this tutorial on codeproject:
Multiple File Upload With Progress Bar Using Flash and ASP.NET
PS - I know you're using PHP and not .net, I figured the important part was the flash ;)
I've had success with uploadify, and I would recommend it. It's a jQuery/Flash script that handles large uploads, and you can pass extra parameters to it (like the secret key). To solve the server-side issues, simply use the following code. The changes take affect just for the script they're called in:
//Check to see if the key is there
if(!isset($_POST['secret_key']) || !isValid($_POST['secret_key']))
{
exit("Invalid request");
}
function isValid($key)
{
//Put your validation code here.
}
//This line changes the timeout.
//Give it a value in seconds (3600 = 1 hour)
set_time_limit(3600);
//Set these amounts to whatever you need.
ini_set("post_max_size","8192M");
ini_set("upload_max_filesize","8192M");
//Generally speaking, the memory_limit should be higher
//than your post size. So make sure that's right too.
ini_set("memory_limit","8200M");
EDIT In response to your comment:
Given what you've said, I'm afraid you may not be able to meet your requirements over http. All of the solutions out there are code that add features to http that it was never designed for.
Like you said yourself, it's a simple protocol. Apart from writing your own client software that runs outside of the browser, a java applet, or using a different protocol (like FTP, which was designed for this), you might not get what you want.
I've done the best I could within the given constraints. Sorry I couldn't do better.
Try this: http://www.simple2ftp.com uses a Java based FTP applet from within a clever PHP application wrapper.
http://www.plupload.com - "Allows you to upload files using HTML5 Gears, Silverlight, Flash, BrowserPlus or normal forms, providing some unique features such as upload progress, image resizing and chunked uploads." This is the uploader used in current WordPress v3.4.1 and the best one out there in my opinion.
It comes with upload.php file (full file: http://ideone.com/xbPUS).
I have doubts about its security: When I have upload.php on my server and even if I don't setup any Javascript for Plupload anyone is still able to relatively easy send request to upload.php file and upload anything, anytime... TRUE OR FALSE?
How do I prevent that?
It's not a security issue. Across the internet you can try to upload anything you want to servers (addresses) that support POST method. It's up to server-side software to accept or reject such upload - it's always been this way. Of course there can be some restrictions put onto who uploads what (using tokens, authorization etc), but that's up to you (as developer) to handle.
As to upload.php file from plupload, I think it's suppose to be just a quick and dirty example, that makes trying out plupload a little bit easier.
I think you want Wordpress to work with upload.php and not the other way arround. So if anyone would call upload.php directly it will fail. Can you set some specific information that will only be available from within your Wordpress functions. In upload.php you can ask for this information if not available it will stop. Hope this is what you need.
I am making an online tool for identifying certain file types. I need to access some byte values from the file header to do this.
The user selects the file on the client machine. Somehow, I need to get the key byte values from the file, and then these are looked up in a server side database to categorize the file.
How can I read bytes from a client-side file?
I know I could have the user upload the file to the server, but these files are very large, and I only need a few bytes, so it would be slow and wasteful to upload the whole file.
Could I somehow upload part of the file? It seems it is difficult to cancel a html form upload and the file-part is not available after cancel. Is this correct?
Is it possible to read a file in javascript? I have googled this, but the answer is unclear. I have read that it is possible with a java applet, but only if the applet is signed.
Is there some other way?
You can use html5, but will need to fallback on flash or some other non-javascript method for older browsers.
http://www.html5rocks.com/en/tutorials/file/dndfiles/
So. as Said above you must use non-javascript methodds. But each of this methods has some minus.
FLASH - bad work with proxy. Really bad. Of course you can use flash obly for get base64 code of file and give it to js. In this case this will be work greate.
Java Applet - greate work but not many users have JVM or versions of JVM may not be sasme (but if you will use JDK1.4 or 1.5 thi is no problem).
ActiveX - work only in IE and on Windows
HTML5 File Api - not cross browsers solution. Will be work only on last browsers and not in all.
of course much better use server side - in php for example getmimetype and other functions.
But I can manually change headers of my file. For example i can add to php file headers from jpeg or png - and your script will be think that is image.
So this is bad solution : use headers. For check filetype maybe simple use mimetype of file of trust to user and generate icon through file extension
I code primarily in PHP and Perl. I have a client who is insisting on seeking video submissions (any encoding) from the public via one of their pages rather than letting YouTube do its job.
Server in question is a virtual machine and I can adjust ini settings for max post, max upload size etc as needed.
My initial thought is to use a Flash based uploader with PHP on the back end but I wondered if someone might have useful advice and experience on the subject?
Doing large file transfers of HTTP is not usually fun -- but sometimes it's necessary.
For large files, you'll definitely want to provide some kind of progress gauge for end-users.
There are flash-based tools that do this (swfUpload comes to mind).
If you want to avoid flash and do it with pretty html/javascript/css, you can leverage PHP's APC extension, which for some reason provides support for getting upload status from the server, as explained here
You can adjust the post size and use a normal html form. The big problem is not Apache, its http. If anything goes wrong in the transmission you will have no way to detect the error. Further more there is no way to resume the transfer. This is exactly why BitTorrent is so popular.
I don't know how against youtube your client is, but you can use their api to do the uploads from a page on your site.
http://code.google.com/apis/youtube/2.0/developers_guide_protocol.html#Uploading_Videos
See: browser based uploading.
For web-based uploads, there's not many options. Regardless of web platform, web server, etc. you're still transferring over HTTP. The transfer is all or nothing.
Your best option might be to find a Flash, Java, or other client side option that can chunk files and upload them piecemeal, then do a checksum to verify. That will allow for resuming uploads. Unfortunately, I don't know of any such open source component that does this.
Try to convince your client to change point of view.
Using http (and the browser, hell, the browser!) for this kind of issue is rarely a good deal; Will his users wait 40 minutes with the computer and the browser running until the upload is complete?
I dont think so.
Maybe, you could set up a public ftp account, where users can upload but not download and see the others user's files.. then, who want to use FTP software can, who like to do it via browser can too.
The big problem dealing using a browser is that, if something go wrong, you cant resume but have to restart from zero again.
the past year i had the same issue, i gave a look to ZUpload
, but i didnt use it so i can suggest (we wrote a small python script that we send to our customer; the python script create a torrent of the folder our costumer need to send to us, and we download it via utorrent ;)
p.s: again, sorry for my bad english ;)
I used jupload. Yes it looks horrible, but it just works.
With that said, it's still a better idea to convince the client that doing so is stupid.
I would agree with others stating that using HTML is a poor option. I believe there is a size limitation using Flash as well. I know of a script that uses a JavaScript Applet to perform an actual FTP transfer. It is called Simple2FTP and can be found at http://www.simple2ftp.com
Not sure but perhaps worth a try?