In my PHP web page, I allow my clients to upload a file. That file can be very large (1-2 GB). I already modified the php.ini in order to allow the users to upload big files.
Now, I want to know how to upload only 1 chunk of the file (the size of the chunk is selected by the user). For example, I have a 1 GB file, but I only want to upload the first 150 MB of that file (the file will still work even it has been chunked, but it will have less samples to work with).
How can I do that? If I do so, I will let the user not to have to wait much time uploading the 1 GB file, and the user will upload a 150 MB file instead (150/1000 MB).
I've found some ways to do it, like PLUPLOAD, but I prefer not to use JavaScript because I want to do all the process/job in my server and not in the client's side. Or with fread(), fopen() and fwrite(), but in this case, I can't use these commands without having uploaded the full file first.
P.D.: I can also use Python to chunk the file before uploading it, if it's necessary, but I think that I'll have the same problem as fread(), fopen(), etc...
Thank you.
If you're not willing to split up the file on the client before uploading it, you'll need to use some client-side code (on a supported browser) to perform partial uploads.
For an example, see https://github.com/blueimp/jQuery-File-Upload/wiki/Chunked-file-uploads
Related
I have a cron script that compresses images. It basically iterates over folders and then compresses the files in the folder. My problem is that some images are getting processed halfway. My theory is that users are uploading a image, and before the image has finished uploading the file, the compressor tries to compress the file. Thus compressing a half-uploaded image, and resulting in half an image being displayed.
Is there a way in PHP to confirm that a file has finished uploading? So that I can only do the compression once i know the file has been fully written?
Or alternatively, is there a way to check if a file is being used by another process?
Or alternatively, would it be reliable enough to look at when the file was "written to disk" and not process it until 10 minutes has gone by?
PHP doesn't trigger your action until the files are fully uploaded, but it is possible for your cron job to start interacting with files before they're fully saved.
When saving something from $_FILES, save it to a version with a . prefix on it to tag it as incomplete. Make sure your cron job skips any such files.
Then, once the save operation is complete, rename the file without the . prefix to make it eligible for processing.
There are two ways to handle the scenario
Flags
Set flag that files before modify/write it.
Our App handles lots of files, we set flags before taking them to process once it's done we remove the flag, as it runs on cron flag is the best way to process files.
Usually, you can an extra column on the table on each file. or you can have an array where you can store all currently handling files.
filemtime()
As you mentioned you can check like if file mtime is more than 10 min that current time, then you compress them but if some other processes are using the file opened the at the same time. it causes the problem again.
So its better to go with flag. If other processes never modify the files often.
You can use flock to ensure file is not in use, see here for example. Alternatively you can check whether an image is broken or corrupted see here.
First 3 bytes of FLV file are signature "FLV". Now my question:
Is there any possibility in PHP to handle file uploads so that we can hook into the input stream of uploading file and check the first 3 bytes?
The scenario is i don't want the complete file to be uploaded (in temporary folder) and then check if the file is FLV or not, i just want to read first few bytes of uploading stream and if it is not "FLV" then return/exit.
The file needs to be uploaded by HTML based form. Can't rely on javascript,flash uploader or any client side validation workarounds i.e. need a pure PHP based solution.
I already know that PHP does not hand over control to our script until it finishes uploading file in temporary folder and populating global variables i.e $_POST $_GET $_FILES etc.
Also there is a somewhat similar question here:
How to upload a file byte by byte in php
but answers does not satisfy my requirement/question.
Any ideas are really appreciated!
Thanks
First, set session.upload_progress.enabled in your php.ini.
Then, use session.upload_progress to track how many bytes have uploaded. Once you have reached the minimum threshold, check the temporary file being uploaded, it will be in $_SESSION[unique_key]['files'][0]['tmp_name']. If the file doesn't match, set $_SESSION[unique_key]["cancel_upload"] to TRUE, and the file will be rejected.
To get unique_key:
ini_get("session.upload_progress.prefix") . $_POST[ini_get("session.upload_progress.name")];
If the above does not work (I haven't tested it), then your only recourse would be to create your own custom handler for PHP either as an Apache module (or better, as a custom CGI application). There you could do your filtering.
#burhan-khalid provided a more up to date and correct answer above.
Short Answer is no with your constraints.
You can not access that file with PHP until it has been uploaded to the server. After it is uploaded you can read it but not before, at least not without some type of client side software that would allow you to stream it to PHP instead of the normal form submission.
I'm using Jquery file upload to upload files onto php server. I would like to modify the uploader such that I can check whether the uploaded file is the same as the local file.
My current idea is to use checksum to check whether the files are identical. However, I need to support large files (1GB). As I understand, we cannot load 1GB file using HTML5 File API for the hashing function.
The way Jquery file upload handles large file is to load part of the file and send it. So is there any way to do checksum when files are chopped into pieces? Or there are any other ways to check whether a file is correctly uploaded?
You should check for the file size, since HTTP connections won't mess up chunk order nor send wrong bytes. The real corruption will be in truncated files.
We have a FreeBSD server with samba that employees copy image files on to, which then get uploaded to our web servers (this way they don't have to mess with ftp). Sometimes, if the upload script is running at the same time as files are being copied, it can upload an incomplete file.
We fixed this by getting the list of files along with the file sizes, then waiting 5 seconds and rechecking the file sizes. If the sizes match then its save to upload, if they don't match it checks again in another 5 seconds.
This seems like an odd way to check if the files are being written to. is there a better, more simple way of doing this?
Use a flock function http://php.net/flock - when writing a file obtain an exclusive lock flock($handle, LOCK_EX), after it is written release the lock flock($handle, LOCK_UN).
The upload script could try to obtain the exclusive writing lock too, if it succeeds it is Okay to move the file, otherwise no.
EDIT: Sorry, I forgot about the users copying the files to the server through samba... So there is no space to use flock while copying... But the upload script could still use flock($handle, LOCK_EX) to see, if it is successful or not.
I recommend to shell_exec() smbstatus(1), e.g. smbstatus -LB to check for locked files
Write a script to copy the files to a temp folder on the Samba server and then, when fully copied and flushed, move, (ie, unlink/link, not copy again), them to the upload folder.
I have a PHP script that opens a local directory in order to copy and process some files. But these files may be incomplete, because they are being uploaded by a slow FTP process, and I do not want to copy or process any files which have not been completely uploaded yet.
Is is possible in PHP to find out if a file is still being copied (that is, read from), or written to?
I need my script to process only those files that have been completely uploaded.
The ftp process now, upload files in parallel, and it take more than 1 second for each filesize to change, so this trick is not working for me now, any other method suggests
Do you have script control over the FTP process? If so, have the script that's doing the uploading upload a [FILENAME].complete file (blank text file) after the primary upload completes, so the processing script knows that the file is complete if there's a matching *.complete file there also.
+1 to #MidnightLightning for his excellent suggestion. If you don't have control over the process you have a couple of options:
If you know what the final size of the file should be then use filesize() to compare the current size to the known size. Keep checking until they match.
If you don't know what the final size should be it gets a little trickier. You could use filesize() to check the size of the file, wait a second or two and check it again. If the size hasn't changed then the upload should be complete. The problem with the second method is if your file upload stalls for whatever reason it could give you a false positive. So the time to wait is key.
You don't specify what kind of OS you're on, but if it's a Unix-type box, you should have fuser and/or lsof available. fuser will report on who's using a particular file, and lsof will list all open files (including sockets, fifos, .so's, etc...). Either of those could most likely be used to monitor your directory.
On the windows end, there's a few free tools from Sysinternals that do the same thing. handle might do the trick