I have a site that is receiving 30-40k photo uploads a day and I've been seeing an issue pop up with more frequency now. This issue is this:
Our upload script receives (via $_FILES['name']['tmp_name']) a file (photo) that was NOT uploaded by the user & the majority of the time the file received is a "partial" upload.
Of course at first I thought it was my PHP code making a simple mistake and I've spent days looking over it to make sure, but after placing checks in the code I've found that the file received via a HTTP POST upload to PHP is actually the wrong file. So the issue is happening before it reaches my code. The tmp file (phpxxxx) received by the script is sometimes incorrect, as if it was somehow being overwritten by another process and its usually overwritten by a file that was partially uploaded.
Has anyone every seen an issue like this? Any help is greatly appreciated. I'm turning to this as a last resort after days of searching/asking other PHP devs
So to recap:
User uploads a photo
PHP script receives a file that was not uploaded by the user (pre code, via $_FILES in /var/tmp)
Usually the incorrect file received is a partial upload or a broken upload
It seems to happen randomly and not all the time
First off, check you PHP version.
Second, check your file upload limits and POST_MAX_SIZE in php.ini
It might just be that someone tries to upload a file that's too large :-)
Can you try different names for the temp file to avoid its being overwritten? Can you identify the origin of the new, incorrect and incomplete file?
Is this a development environment? Is it possible that more than one user is uploading files at the same time?
Try your program with very small images to check if SchizoDuckie is correct about filesize problems.
Try with different navigators to eliminate the admittedly remote possibility that it is a local problem.
Check permissions on the directory where the temp file is stored.
PHP's built-in file handling does not support partial uploads.
Turn off KeepAlives and/or send a 'Connection: close' header after each upload.
Configure your webserver to send the header 'Allow-Ranges: none'.
Related
I am using the script posted here by w3school for uploading files to a webserver. I am using Google chrome going to an Apache server with php7.
When I upload a file from my windows desktop the script works as advertised and the file is uploaded. However when I attempt to upload a file from any other folder the $_FILES variable is either not there or it's values are empty (It actually appears to be a size issue, see edit below). I'm not sure how to troubleshoot this as the devtools on chrome don't seem to give me any information about what it is sending in this POST message.
I am assuming the problem is on the client side as it appears the server is getting no information from the client.
EDIT
I am aware of how to look at the headers for requests and nothing about the file except for its content length are included in the header. There is no difference between the headers that work and those that don't except the content boundary, which appears to be random, and the content-length header, which obviously changes with the size of the file.
I had a thought that this was related to the size of the file, and it is. I have a folder of different size photos and I sorted and uploaded increasingly larger files until it failed. It worked fine until around 8M files, and fails when they are larger than 8,200KB.
I checked the php.ini and I have it set to a max upload size of 150M, so it's not the ini file. I also check all the logs and nothing other than the post request shows up when I try to upload the file.
I was able to find the answer here: why would $_FILES be empty when uploading files to PHP.
I did not know there was a post_max_size option in the php.ini file, which was set to 8M. Now that this is set to 100M. If you go over this size of a POST request php does not parse the POST information, thus giving you a blank $_FILES variable.
My application is keeping watch on a set of folders where users can upload files. When a file upload is finished I have to apply a treatment, but I don't know how to detect that a file has not finish to upload.
Any way to detect if a file is not released yet by the FTP server?
There's no generic solution to this problem.
Some FTP servers lock the file being uploaded, preventing you from accessing it, while the file is still being uploaded. For example IIS FTP server does that. Most other FTP servers do not. See my answer at Prevent file from being accessed as it's being uploaded.
There are some common workarounds to the problem (originally posted in SFTP file lock mechanism, but relevant for the FTP too):
You can have the client upload a "done" file once the upload finishes. Make your automated system wait for the "done" file to appear.
You can have a dedicated "upload" folder and have the client (atomically) move the uploaded file to a "done" folder. Make your automated system look to the "done" folder only.
Have a file naming convention for files being uploaded (".filepart") and have the client (atomically) rename the file after upload to its final name. Make your automated system ignore the ".filepart" files.
See (my) article Locking files while uploading / Upload to temporary file name for an example of implementing this approach.
Also, some FTP servers have this functionality built-in. For example ProFTPD with its HiddenStores directive.
A gross hack is to periodically check for file attributes (size and time) and consider the upload finished, if the attributes have not changed for some time interval.
You can also make use of the fact that some file formats have clear end-of-the-file marker (like XML or ZIP). So you know, that the file is incomplete.
Some FTP servers allow you to configure a hook to be called, when an upload is finished. You can make use of that. For example ProFTPD has a mod_exec module (see the ExecOnCommand directive).
I use ftputil to implement this work-around:
connect to ftp server
list all files of the directory
call stat() on each file
wait N seconds
For each file: call stat() again. If result is different, then skip this file, since it was modified during the last seconds.
If stat() result is not different, then download the file.
This whole ftp-fetching is old and obsolete technology. I hope that the customer will use a modern http API the next time :-)
If you are reading files of particular extensions, then use WINSCP for File Transfer. It will create a temporary file with extension .filepart and it will turn to the actual file extension once it fully transfer the file.
I hope, it will help someone.
This is a classic problem with FTP transfers. The only mostly reliable method I've found is to send a file, then send a second short "marker" file just to tell the recipient the transfer of the first is complete. You can use a file naming convention and just check for existence of the second file.
You might get fancy and make the content of the second file a checksum of the first file. Then you could verify the first file. (You don't have the problem with the second file because you just wait until file size = checksum size).
And of course this only works if you can get the sender to send a second file.
First 3 bytes of FLV file are signature "FLV". Now my question:
Is there any possibility in PHP to handle file uploads so that we can hook into the input stream of uploading file and check the first 3 bytes?
The scenario is i don't want the complete file to be uploaded (in temporary folder) and then check if the file is FLV or not, i just want to read first few bytes of uploading stream and if it is not "FLV" then return/exit.
The file needs to be uploaded by HTML based form. Can't rely on javascript,flash uploader or any client side validation workarounds i.e. need a pure PHP based solution.
I already know that PHP does not hand over control to our script until it finishes uploading file in temporary folder and populating global variables i.e $_POST $_GET $_FILES etc.
Also there is a somewhat similar question here:
How to upload a file byte by byte in php
but answers does not satisfy my requirement/question.
Any ideas are really appreciated!
Thanks
First, set session.upload_progress.enabled in your php.ini.
Then, use session.upload_progress to track how many bytes have uploaded. Once you have reached the minimum threshold, check the temporary file being uploaded, it will be in $_SESSION[unique_key]['files'][0]['tmp_name']. If the file doesn't match, set $_SESSION[unique_key]["cancel_upload"] to TRUE, and the file will be rejected.
To get unique_key:
ini_get("session.upload_progress.prefix") . $_POST[ini_get("session.upload_progress.name")];
If the above does not work (I haven't tested it), then your only recourse would be to create your own custom handler for PHP either as an Apache module (or better, as a custom CGI application). There you could do your filtering.
#burhan-khalid provided a more up to date and correct answer above.
Short Answer is no with your constraints.
You can not access that file with PHP until it has been uploaded to the server. After it is uploaded you can read it but not before, at least not without some type of client side software that would allow you to stream it to PHP instead of the normal form submission.
I built a website in that I upload 10 too big size(10MB) images. When uploading start, it continues to some time then a blank page will come. I tried to change php_values in .htaccess file, because I don't have permission to change the settings in php.ini file (it's shared server). I have some doubts regarding this.
1) what happen if file will going to post request, because I want fastly uploded the files.
2) it takes time when posting the request or uploding the file, I am cropping the images (loop) using php GD functions.
It is because of the limits your web hosting provider set. Which values did you try to change in the .htaccess?
You could try using some flash uploader, it should work despite the limits imposed by the server. A good one is SWFUpload.
That is because of the exection time of a script.You can edit your php.ini file. If that is not permitted you can set the *MAX_EXECUTION_TIME* for a script using your .htaccess file.
I have a PHP script that opens a local directory in order to copy and process some files. But these files may be incomplete, because they are being uploaded by a slow FTP process, and I do not want to copy or process any files which have not been completely uploaded yet.
Is is possible in PHP to find out if a file is still being copied (that is, read from), or written to?
I need my script to process only those files that have been completely uploaded.
The ftp process now, upload files in parallel, and it take more than 1 second for each filesize to change, so this trick is not working for me now, any other method suggests
Do you have script control over the FTP process? If so, have the script that's doing the uploading upload a [FILENAME].complete file (blank text file) after the primary upload completes, so the processing script knows that the file is complete if there's a matching *.complete file there also.
+1 to #MidnightLightning for his excellent suggestion. If you don't have control over the process you have a couple of options:
If you know what the final size of the file should be then use filesize() to compare the current size to the known size. Keep checking until they match.
If you don't know what the final size should be it gets a little trickier. You could use filesize() to check the size of the file, wait a second or two and check it again. If the size hasn't changed then the upload should be complete. The problem with the second method is if your file upload stalls for whatever reason it could give you a false positive. So the time to wait is key.
You don't specify what kind of OS you're on, but if it's a Unix-type box, you should have fuser and/or lsof available. fuser will report on who's using a particular file, and lsof will list all open files (including sockets, fifos, .so's, etc...). Either of those could most likely be used to monitor your directory.
On the windows end, there's a few free tools from Sysinternals that do the same thing. handle might do the trick