I want to be able to encode videos eith handbrake CLI once the user has uploaded them. I have no idea how to go about doing this, thr inly guess i can think of is to use php's exec() command and execute the program from there. But wouldnt that come with risks?
Or is there any other way to do this? I basically need to encode the videos into a web playable format with baseline 3 encoding so that they play through he JWPlayer on all devices.
How can i achieve this?
exec() doesn't have to be unsafe as long as you're careful with what you're executing.
Alternatively, and a better solution, would be to use a producer - consumer model. Everytime a video is uploaded, add a reference to it to a queue. Have a CLI script that monitors the queue, and as long as the queue has videos that need to be processed, the script does the encoding.
You can use a database, or even just a directory as a queue. For example, upload all new videos to a directory unprocessed. The consumer script can pick up videos from there, encode them and place them in a folder called processed.
This allows you to control how many videos should be processed concurrently (by varying the number of consumers). exec() on user upload doesn't allow that, and you risk the chance of bringing your server down to its knees.
Edit:
If you intend to use more than one consumer, I wouldn't recommend using the directory-as-queue model as you risk the chance of a race condition.
Related
I am building a web application that allows users to upload audio files, music in particular. Most of the time, I expect the duration of each song to normally be about several minutes and the file to be approximately 3-10MB in size. However, I would like to accept audio uploads up to about 100MB, possibly allowing for over an hour of audio. I am currently using a combination of FFmpeg, SoX, and LAME to convert from 7 possible formats to mp3 and perform audio modifications including equalization, trimming, and fading. The files are then stored and linked in the database.
My current strategy is to handle the entire process in one HTTP file upload request using PHP on the backend, in which I perform the following functions:
Validation
Transcode audio into multiple versions (using shell through PHP)
Store the original and transcoded versions in a temp directory
Upload all audio files to Amazon S3 for permanent storage
Commit the ID of each file to a database, linking them to the user
This works very similar to an image processing system I have already set up. However, while images can complete this whole process in just a few seconds, audio can take a lot longer. At most, audio could take about 5-10 minutes to be processed and stored.
My questions are:
For audio processing, would it be better to fork off the transcoding to another background process, writing its state to the database, and pinging it every few seconds to update the webpage vs. doing it all in one HTTP request?
With the intention of scaling in the future, would it be advisable to do all processing on a single server instance, leaving the frontend web instances free to replicate / be destroyed?
If yes, would this require cross-domain file uploading directly to that server? (Anyone know if this is how youtube or the big sites do it?)
Thanks!
If I understand your system correctly, your best approach is probably something more like this:
In your web front-end, store the audio and create a "task" indicating that the audio needs to be processed.
Run a background task that pulls tasks and does the processing. At the end of the task, the user can be notified (if necessary) and database state can be updated or whatever.
Your tasks should be written so that if they fail partway through, they can be re-executed from the start without causing problems. You can run multiple background tasks and web front-ends in this architecture.
A good way to write tasks is using a message passing system like AMQP. There are cheap services like rabbitmq that will do this for you. You can, of course, also build your own on top of any database, but this may require polling.
Finally, you might find it faster and more efficient to use a service like zencoder to do your transcoding, because they can parallelize the work and probably handle more input formats, but it may not be compatible with your processing.
you definitely want to throw the audio processing to a background process.
Depending on the scalability involved, you might need a computer dedicated to the processing. You might want to look into other resources you can offload audio stuff too (like PCIe cards and such)
Sorry to say I know nothing about cross domain file uploading or how the big dogs do it (youtube, soundcloud ect)
My application requires downloading many images from server(each image about 10kb large). And I'm simply downloading each of them with independent AsyncTask without any optimization.
Now I'm wondering what's the common practice to transfer these images. For example, I'm thinking about saving zipped images at server, then send zipped file for user's mobile to unzip. In this case, is it better to combine the zip files into one big zip file for user to download?
Or there's better solution? Thanks in advance!
EDIT:
It seems combining zip files is a good idea, but I feel it may take too long for user to wait downloading and unzipping all images. So I may put ten or twenty images in each zip file, so user can see some downloaded ones while waiting for more to come. Having multiple AsyncTask fired together can be faster right? But they won't finish at the same time even given same file size and same address to download?
Since latency is often the largest problem with mobile connections, reducing the number of connections you have to open is a great way to optimize the loading times. Sending a zip file with all the images sounds like a very good idea, and is probably worth the time implementing.
Images probably are already compressed (gif, jpg, png). You will not reduce filesize but will reduce the number of connections. Which is a good idea for mobile. If it is always the same set of images you can use some sprite technology (sending one bigger image file containing all the images but with different x/y offset, in html you can use the backround with an offset to show the right image).
I was looking at the sidebar and saw this topic, but you're asking about patching when I saw the comments.
The best way to make sure is that the user knows what to do with it. You want the user to download X file and have Y output for a different purpose. On the other hand, it appears common practice is that chunks of resources for those not native to the Android app and not able to fit in the APK.
A comparable example is the JDIC apps, which use the popular Japanese resource that are in tandem used for English translations. JDIC apps like WWWJDIC use online downloads for the extremely large reference files that would otherwise have bad latency (which have been mentioned before) on Google servers. It's also bad rep to have >200 MB on Google apps unless it is 3D, which is justifiable. If your images cannot be compressed without extremely long loading times on the app itself, you may need to consider this option. The only downside is to request online connection (also mentioned before).
Also, you could use 7zip and program Android to self-extract it to a location. http://www.wikihow.com/Use-7Zip-to-Create-Self-Extracting-excutables
On another note, it would be optimal for the user to perform routine checks on the app while having a one-time download on initial startup. You can then optionally put in an AsyncTask so that your files will be downloaded to the app and used after restart or however you want it, so you really need only one AsyncTask. The benefit of this is that the user syncs on the apps and he may need to check only once. The downside is that the user may not always be able to update and may need to use 4G or LTE, but that is a minor concern if he can use WiFi whenever he wants.
Hey just a quick question for anyone who has done this. I want to create a video tube site. I have done file uploads before but was wondering if anyone could give me suggestions on what I am planning to do.
The way I am planning is to have a folder in my web directory and to upload videos into the folder after virus scanning and checking mime. The video will then be converted and compressed using FFMPEG into flv.
I will change the name and store the video reference id in mysql so the file name can be fetched and served.
I will serve the files using HTTP_Download to a flash player
$dl = new HTTP_Download();
$dl->setFile("$path");
$dl->setContentDisposition(HTTP_DOWNLOAD_ATTACHMENT, "$path");
$dl->setContentType('video/flv');
$dl->send();
Anyone have any suggestions? Is it a good idea to put all videos in one directory?
You may want to consider a Java based uploader as PHP can run into timeout problems on large uploads.
Also do you FFMPEG processing as a CRON job not at upload as it takes a long time.
Look in something like Wowza Streaming Server to serve the videos. Allows streaming and everything is above the root. I name each video with a UID and send a parameter to the Flash video player to decide which one to play.
Where and how you store them will largely depend on how secure they need to be (i.e. should people be able to access the files in the directory directly? or should it be stored more securely than that?)
If direct access is fine, then putting them all in one folder is okay. If not, then you may want to obscure folder names, store them in a secure Database, or in a folder that is not accessible outside of the server.
Also, I'm hoping you're aware of the massive amounts of storage space and bandwidth such a service will consume? I hope you have a scaled solution ready to deploy if you're really serious about this..
I created an simple web interface to allow various users to upload files. I set the upload limit to 100mb but now it turns out that the client occasionally wants to upload files 500mb+.
I know what to alter the php configuration to change the upload limit but I was wondering if there are any serious disadvantages to uploading files of this size via php?
Obviously ftp would be preferable but if possible i'd rather not have two different methods of uploading files.
Thanks
Firstly FTP is never preferable. To anything.
I assume you mean that you transferring the files via HTTP. While not quite as bad as FTP, its not a good idea if you can find another of solving the problem. HTTP (and hence the component programs) are optimized around transferring relatively small files around the internet.
While the protocol supports server to client range requests, it does not allow for the reverse operation. Even if the software at either end were unaffected by the volume, the more data you are pushing across the greater the interval during which you could lose the connection. But the biggest problem is that caveat in the last sentence.
Regardless of the server technology you use (PHP or something else) it's never a good idea to push that big file in one sweep in synchronous mode.
There are lots of plugins for any technology/framework that will do asynchronous upload for you.
Besides the connection timing out, there is one more disadvantage in that file uploading consumes the web server memory. You don't normally want that.
PHP will handle as many and as large a file as you'll allow it. But consider that it's basically impossible to resume an aborted upload in PHP, as scripts are not fired up until AFTER the upload is completed. The larger the file gets, the larger the chance of a network glitch killing the upload and wasting a good chunk of time and bandwidth. As well, without extra work with APC, or using something like uploadify, there's no progress report and users are left staring at a browser showing no visible signs of actual work except the throbber chugging away.
I am a php newbie so i may be completely off here. But this is what i want to do.
Have a user upload a video to my site.
This then has a 30 second clip from our database added to it.
Which can be downloaded by the user for upload on any third party websites. Essentially the video is a flv or mpeg4, etc. But now with the 30 second clip added on.
These are the key point that make this difficult though.
I would want for certain links that are attached to the video file that is downloaded to be clickable from whereever it is uploaded to.
I would want for the 30 second slot to not be fast-forwardable. (Like on the youtube ads). <- I know they use FLASH for that./ But i would need my videos to be functional anywhere and youtube videos cant do that without attachment to youtube.
(This is something i intend to build myself. I just need your help in figuring out HOW to do it.)
All answers are GREATLY appreciated.
Thanks Alot!!
The challenges you bring up are not something you would ideally fix with PHP knowledge per se. PHP doesn't offer any video editing capabilities of its own, so you would effectively have to use PHP to run some other utilities.
The only PHP aspect here, really, is receiving an upload from the user, executing an external command on the uploaded file, and providing it as a download again. Those are the only parts of this process that I think you can realistically do with PHP alone, and they are quite easy. If you want to figure those out before you figure out the video editing aspect, then find information about making PHP receive a file from the user, how to make a PHP script downloadable instead of a page, and how to execute external commands.
That said, there are command-line utilities out there which you can use PHP to run on the file. I suggest you check out 'mencoder'. I've only used it a couple of times myself to do some video encoding conversions, but that's the first thing that comes to mind. It doesn't matter what you find... provided that it's a command-line utility, you can make PHP use it, even if you're running Windows as a server (imagine that!).
In short; I recommend trying to find command-line utilities that support video manipulation first, forgetting about PHP for a while, then figuring out how to use them, and only then figuring out how to make PHP do what you have to do.
The part with links in the video + no fastforward is not possible in my experience with php or anything else if this is only a video file.
If you go for a full embeded player or at least youtube, then it's possible.