Maybe it would be best to start by describing the scenario.
We have a Debian server with ffmpeg that we use to covert various video files into FLV.
The files are supplied by a number of different people via FTP and are kept in the "uploads" folder.
I need to write a PHP script that would go through all the files in the uploads folder, select the ones which are complete (i.e. not currently being uploaded or without any uploading errors) and then convert them to FLV using ffmpeg.
I can do the conversion and everything else, but how do I determine whether a file is complete and fully uploaded?
Many thanks!
Afaik you can't just figure out if a file is still being uploaded. You could run a cronjob everyminute getting the filesizes and store these in a database or file. Then if you run the cronjob the second time and the filesize is the same: convert them, if not.. wait another minute and then try again.
I don't believe there's a filesize stored with a file that contains the size it should be after the upload is done.
There is another way to go about this, which we have done for years.
Most ftp servers (proftpd does) will output a log which will tell you when a file upload has completed successfully. You can set this logging to go to a unix named pipe / fifo, and then have a daemonized script read this to determine which files to process. This works great, and only processes files after they are uploaded completely and successfully.
Related
Now the process is:
File upload
Encode file with ffmpeg when file has been uploaded
Can be done at the same time?, obviusly yes, but I don't know how.
The file upload process is by 8MB chunks stored in server, then, temporary I have videofile.ext.tmp that is growing up to final uploaded file.
I read about ffmpeg -stream_loop or -loop parameters, but I don't know if the upload process is good for real-time encoding or how to manage it.
Any help will be appreciated.
EDITED: I tried this feature and works good when: Internet connection is faster than ffmpeg encode speed. When uploaded file is big enough.
I personally ruled out this procedure because the internet speed may change or cut out.
It depends on the file format. Some formats like ts,mkv,flv will work this way. Mp4 may or may not work depending on how the file was created.
Currently the MP3 file exceeds the limit set in wordpress. And Although I am going to raise that limit with some information I found on the topic, the Mp3s are still rather large. If the site were for me, I'd simply compress them. So I need to find a method to compress them. I assume it needs to save that large file, to transcode it to a smaller file, and then delete the old one. Any ideas?
I assume it needs to save that large file, to transcode it to a smaller file, and then delete the old one
This is correct. You always have to store the content first. Encoding should not be done via PHP though, there are more effective libraries for that. For example, you can run lame mp3 encoder on the command line via a system() call in php. Be aware though that encoding might fail and also that it takes quite long. So you should run a cleaner script for the big files via cron instead of trying to delete them via the upload script.
Is it possible to upload files (even big files) to a ftp using PHP?
Been reading about ftp_connect() and it looks like I can, or can't I?
I had a look at this example, it's in Italian but you can read the code anyway, if that does what I'm asking, will I have to add an html form? I need to be able to pick up a file form my computer via a web page and upload it to an ftp basically.
Anyone?
Thanks
Especially on large files, you should make sure that the maximum execution time for the script is big enough to complete the transfer before the script is aborted. You can choose the maximum execution time in the php.ini file.
You will have to use an HTML form if you want to pick up the file from your computer via a web page.
As soon as the form is submitted, you can access the file using the $_FILES array. You can use this information to get a temporary path to where the file is stored, and can read it from there to upload it to a remote server using the FTP functions.
You could also split the two processes by using the PHP script only to drop the file into the local file system, and then use a second program which runs locally to do the upload. This has the advantage that you won't run into problems when multiple users upload simultaneously and your FTP is set up in a way that it allows only 1 simultaneous connection. You could program the second script also in PHP and run it using a Cronjob for example once per 30min.
If your goal is a direct stream from your computer to the FTP server, this is not easily possible using a pure PHP / HTML solution since the PHP script is only invoked when the file transfer from your computer to the machine serving the PHP script is complete.
Files are being pushed to my server via FTP. I process them with PHP code in a Drupal module. O/S is Ubuntu and the FTP server is vsftp.
At regular intervals I will check for new files, process them with SimpleXML and move them to a "Done" folder. How do I avoid processing a partially uploaded file?
vsftp has lock_upload_files defaulted to yes. I thought of attempting to move the files first, expecting the move to fail on a currently uploading file. That doesn't seem to happen, at least on the command line. If I start uploading a large file and move, it just keeps growing in the new location. I guess the directory entry is not locked.
Should I try fopen with mode 'a' or 'r+' just to see if it succeeds before attempting to load into SimpleXML or is there a better way to do this? I guess I could just detect SimpleXML load failing but... that seems messy.
I don't have control of the sender. They won't do an upload and rename.
Thanks
Using the lock_upload_files configuration option of vsftpd leads to locking files with the fcntl() function. This places advisory lock(s) on uploaded file(s) which are in progress. Other programs don't need to consider advisory locks, and mv for example does not. Advisory locks are in general just an advice for programs that care about such locks.
You need another command line tool like lockrun which respects advisory locks.
Note: lockrun must be compiled with the WAIT_AND_LOCK(fd) macro to use the lockf() and not the flock() function in order to work with locks that are set by fcntl() under Linux. So when lockrun is compiled with using lockf() then it will cooperate with the locks set by vsftpd.
With such features (lockrun, mv, lock_upload_files) you can build a shell script or similar that moves files one by one, checking if the file is locked beforehand and holding an advisory lock on it as long as the file is moved. If the file is locked by vsftpd then lockrun can skip the call to mv so that running uploads are skipped.
If locking doesn't work, I don't know of a solution as clean/simple as you'd like. You could make an educated guess by not processing files whose last modified time (which you can get with filemtime()) is within the past x minutes.
If you want a higher degree of confidence than that, you could check and store each file's size (using filesize()) in a simple database, and every x minutes check new size against its old size. If the size hasn't changed in x minutes, you can assume nothing more is being sent.
The lsof linux command lists opened files on your system. I suggest executing it with shell_exec() from PHP and parsing the output to see what files are still being used by your FTP server.
Picking up on the previous answer, you could copy the file over and then compare the sizes of the copied file and the original file at a fixed interval.
If the sizes match, the upload is done, delete the copy, work with the file.
If the sizes do not match, copy the file again.
repeat.
Here's another idea: create a super (but hopefully not root) FTP user that can access some or all of the upload directories. Instead of your PHP code reading uploaded files right off the disk, make it connect to the local FTP server and download files. This way vsftpd handles the locking for you (assuming you leave lock_upload_files enabled). You'll only be able to download a file once vsftp releases the exclusive/write lock (once writing is complete).
You mentioned trying flock in your comment (and how it fails). It does indeed seem painful to try to match whatever locking vsftpd is doing, but dio_fcntl might be worth a shot.
I guess you've solved your problem years ago but still.
If you use some pattern to find the files you need you can ask the party uploading the file to use different name and rename the file once the upload has completed.
You should check the Hidden Stores in proftp, more info here:
http://www.proftpd.org/docs/directives/linked/config_ref_HiddenStores.html
I'm writing a PHP process that will run on a Unix machine that will need to monitor a remote SMB server and detect new files that are being uploaded to that box via FTP. It's unlikely I'll be able to
It will need to detect:
New files being created
File upload completing
Files being deleted
If it was an NFS share, I'd try using FAM to detect the events, but I can't see a way of doing anything equivalent?
Doesn't sound like something I would use in production. But you could try something like this:
mount the SMB share with Samba on
the machine that is running a PHP
daemon
use SPL
RecursiveIteratorIterator with
DirectoryIterator to collect and
maintain a list of all the files and
folders on the shared drive
once in
a while refresh the folder list and
compare it with the current state,
if the file does not exist any more
you know it has been deleted, if
there is a new file put it in the
queue and mark it as "being
uploaded"
in the next "refresh run"
check the queued file, it the file
size did not change the file upload
probably completed, if the file size
changed put it in the queue again
and mark it as "being uploaded"