I am building a web application that allows users to upload audio files, music in particular. Most of the time, I expect the duration of each song to normally be about several minutes and the file to be approximately 3-10MB in size. However, I would like to accept audio uploads up to about 100MB, possibly allowing for over an hour of audio. I am currently using a combination of FFmpeg, SoX, and LAME to convert from 7 possible formats to mp3 and perform audio modifications including equalization, trimming, and fading. The files are then stored and linked in the database.
My current strategy is to handle the entire process in one HTTP file upload request using PHP on the backend, in which I perform the following functions:
Validation
Transcode audio into multiple versions (using shell through PHP)
Store the original and transcoded versions in a temp directory
Upload all audio files to Amazon S3 for permanent storage
Commit the ID of each file to a database, linking them to the user
This works very similar to an image processing system I have already set up. However, while images can complete this whole process in just a few seconds, audio can take a lot longer. At most, audio could take about 5-10 minutes to be processed and stored.
My questions are:
For audio processing, would it be better to fork off the transcoding to another background process, writing its state to the database, and pinging it every few seconds to update the webpage vs. doing it all in one HTTP request?
With the intention of scaling in the future, would it be advisable to do all processing on a single server instance, leaving the frontend web instances free to replicate / be destroyed?
If yes, would this require cross-domain file uploading directly to that server? (Anyone know if this is how youtube or the big sites do it?)
Thanks!
If I understand your system correctly, your best approach is probably something more like this:
In your web front-end, store the audio and create a "task" indicating that the audio needs to be processed.
Run a background task that pulls tasks and does the processing. At the end of the task, the user can be notified (if necessary) and database state can be updated or whatever.
Your tasks should be written so that if they fail partway through, they can be re-executed from the start without causing problems. You can run multiple background tasks and web front-ends in this architecture.
A good way to write tasks is using a message passing system like AMQP. There are cheap services like rabbitmq that will do this for you. You can, of course, also build your own on top of any database, but this may require polling.
Finally, you might find it faster and more efficient to use a service like zencoder to do your transcoding, because they can parallelize the work and probably handle more input formats, but it may not be compatible with your processing.
you definitely want to throw the audio processing to a background process.
Depending on the scalability involved, you might need a computer dedicated to the processing. You might want to look into other resources you can offload audio stuff too (like PCIe cards and such)
Sorry to say I know nothing about cross domain file uploading or how the big dogs do it (youtube, soundcloud ect)
Related
My application requires downloading many images from server(each image about 10kb large). And I'm simply downloading each of them with independent AsyncTask without any optimization.
Now I'm wondering what's the common practice to transfer these images. For example, I'm thinking about saving zipped images at server, then send zipped file for user's mobile to unzip. In this case, is it better to combine the zip files into one big zip file for user to download?
Or there's better solution? Thanks in advance!
EDIT:
It seems combining zip files is a good idea, but I feel it may take too long for user to wait downloading and unzipping all images. So I may put ten or twenty images in each zip file, so user can see some downloaded ones while waiting for more to come. Having multiple AsyncTask fired together can be faster right? But they won't finish at the same time even given same file size and same address to download?
Since latency is often the largest problem with mobile connections, reducing the number of connections you have to open is a great way to optimize the loading times. Sending a zip file with all the images sounds like a very good idea, and is probably worth the time implementing.
Images probably are already compressed (gif, jpg, png). You will not reduce filesize but will reduce the number of connections. Which is a good idea for mobile. If it is always the same set of images you can use some sprite technology (sending one bigger image file containing all the images but with different x/y offset, in html you can use the backround with an offset to show the right image).
I was looking at the sidebar and saw this topic, but you're asking about patching when I saw the comments.
The best way to make sure is that the user knows what to do with it. You want the user to download X file and have Y output for a different purpose. On the other hand, it appears common practice is that chunks of resources for those not native to the Android app and not able to fit in the APK.
A comparable example is the JDIC apps, which use the popular Japanese resource that are in tandem used for English translations. JDIC apps like WWWJDIC use online downloads for the extremely large reference files that would otherwise have bad latency (which have been mentioned before) on Google servers. It's also bad rep to have >200 MB on Google apps unless it is 3D, which is justifiable. If your images cannot be compressed without extremely long loading times on the app itself, you may need to consider this option. The only downside is to request online connection (also mentioned before).
Also, you could use 7zip and program Android to self-extract it to a location. http://www.wikihow.com/Use-7Zip-to-Create-Self-Extracting-excutables
On another note, it would be optimal for the user to perform routine checks on the app while having a one-time download on initial startup. You can then optionally put in an AsyncTask so that your files will be downloaded to the app and used after restart or however you want it, so you really need only one AsyncTask. The benefit of this is that the user syncs on the apps and he may need to check only once. The downside is that the user may not always be able to update and may need to use 4G or LTE, but that is a minor concern if he can use WiFi whenever he wants.
I want to be able to encode videos eith handbrake CLI once the user has uploaded them. I have no idea how to go about doing this, thr inly guess i can think of is to use php's exec() command and execute the program from there. But wouldnt that come with risks?
Or is there any other way to do this? I basically need to encode the videos into a web playable format with baseline 3 encoding so that they play through he JWPlayer on all devices.
How can i achieve this?
exec() doesn't have to be unsafe as long as you're careful with what you're executing.
Alternatively, and a better solution, would be to use a producer - consumer model. Everytime a video is uploaded, add a reference to it to a queue. Have a CLI script that monitors the queue, and as long as the queue has videos that need to be processed, the script does the encoding.
You can use a database, or even just a directory as a queue. For example, upload all new videos to a directory unprocessed. The consumer script can pick up videos from there, encode them and place them in a folder called processed.
This allows you to control how many videos should be processed concurrently (by varying the number of consumers). exec() on user upload doesn't allow that, and you risk the chance of bringing your server down to its knees.
Edit:
If you intend to use more than one consumer, I wouldn't recommend using the directory-as-queue model as you risk the chance of a race condition.
I was thinking of using imagemagick to process images uploaded by a user in various ways (creating new images that are scaled, have drop shadows, etc.) but I've been worried about the speed. I don't want the user staring at a loading gif forever.
So I started looking around to see how other sites do it and I found http://www.redbubble.com. Users upload artwork and almost instantly there are tons of variations of the image in the shop processed in various ways. What does it use to process and generate images so fast?
it's relatively hard and inconvenient to maintain client-side image processing (it would be some kind of flash app similar to www.picnik.com with limited functionality)
I see use of ruby, nginx, remote xhr calls, json etc. that means that delayed_jobs/resque might be used to schedule asynchronous image processing using imagemagick, json/xhr to check the status. processed images are requested from ih*.redbubble.net (point to edgecastcdn.net) and seems like they produce them on the fly and let CDN cache them until user changes that image or it expires in the cache.
they have ~800k monthly visitors, you don't want to put load on app/web servers to process images, there is either delayed_jobs or resque behind the scene or ih* (image host?) servers that produce images on the fly (there are 4 of them, but who knows how many behind virtual host/proxy configuration)
all upload requests go to amazon (ec2, that might be a load balanced IP), originals are stored on amazon s3. they can scale by requesting more ec2 instances on demand.
hope you get an idea what's behind... back to your question: no client-side images processing, imagemagick is used, and there is a chance they do it on the fly.
I'm building a PHP based upload service for some of our clients. I am using SWFUpload so that I can view the progress of a file as it uploads. I've got it pretty much built, but am running into one last issue before we can release it to the public.
Many (almost all) of our clients are Mac-based and are uploading sets of files that include InDesign Files, Fonts, Illustrator Files, etc. Most of the times the images files are OK, but occasionally (and always with Type 1 Fonts) the file will become corrupted because it is losing the resource fork.
I understand why this is happening (moving from a multi-fork system to a single-fork system), but I can not find any elegant solution. In my research the best answer I've found so far is "have the user compress it". I know that works, but it's unreasonable - in our client's opinion - for us to require them to compress every set of files they are going to send.
Are there any better solutions for keeping those resource forks alive? Of course, I would prefer a solution that is straight javascript/php, but would settle for something that is flash based or (least preferably) java based.
My only requirements for the new solution would be:
View upload progress
User doesn't have to manually compress files
Here's some information about my system
Ubuntu 10.10 Server running a standard LAMP install
PHP5
SWFUpload (wtv the most recent version is)
Uploads handle files. If the browser and the underlying OS is not able to deal with forks in this procedure (map anything file onto the file model for uploads), then you're bound to what you get by the systems architecture.
Resource fork: The resource fork is a construct of the Mac OS operating system used to store structured data in a file, alongside unstructured data stored within the data fork. A resource fork stores information in a specific form, such as icons, the shapes of windows, definitions of menus and their contents, and application code (machine code).
If that's a blocker to you you might have chosen the wrong field to work in. Just saying, if you run into systematic borders, there is not much you can do about. Even if you work for graphic designers and mac users.
The swfupload would need a feature to deal with forks. For that, flash would need a feature to deal with forks. For that the browser would eventually need a feature to deal with forks. And so on.
Next to this chain, another question remains: How to deal with forks? As the upload only maps one file to a chunk of binary data, how to map the fork as well? Append it? Add an additional file?
So on the technical level this does not sound like easily solveable. All components and systems in the file input chain must support a feature that is commonly not supported at all.
So as you can not offer something to the user that does not exist, the only thing you can do is make your application more usable or user-friendly. E.g. by providing the right notes at the right time (e.g. when a user selects a Type 1 file for uploading, to remind him/her to select the fork as well). Communicating with the user can help, but keep in mind that a user needs to be spoken with in a language he/she understands.
So if you know that certain file types have forks, address the issue to someone who can solve it: The user. You can't.
You don't have to use swfupload to monitor progress.
Here are some file that demonstrate this: https://github.com/senica/Booger/tree/master/assets/js/jquery-upload
It is not documented very well, but it basically uses webkitSlice function for uploading the files in javascript. You can use the callback functions to display the progress of the files.
This would be a javascript/php solution.
I created an simple web interface to allow various users to upload files. I set the upload limit to 100mb but now it turns out that the client occasionally wants to upload files 500mb+.
I know what to alter the php configuration to change the upload limit but I was wondering if there are any serious disadvantages to uploading files of this size via php?
Obviously ftp would be preferable but if possible i'd rather not have two different methods of uploading files.
Thanks
Firstly FTP is never preferable. To anything.
I assume you mean that you transferring the files via HTTP. While not quite as bad as FTP, its not a good idea if you can find another of solving the problem. HTTP (and hence the component programs) are optimized around transferring relatively small files around the internet.
While the protocol supports server to client range requests, it does not allow for the reverse operation. Even if the software at either end were unaffected by the volume, the more data you are pushing across the greater the interval during which you could lose the connection. But the biggest problem is that caveat in the last sentence.
Regardless of the server technology you use (PHP or something else) it's never a good idea to push that big file in one sweep in synchronous mode.
There are lots of plugins for any technology/framework that will do asynchronous upload for you.
Besides the connection timing out, there is one more disadvantage in that file uploading consumes the web server memory. You don't normally want that.
PHP will handle as many and as large a file as you'll allow it. But consider that it's basically impossible to resume an aborted upload in PHP, as scripts are not fired up until AFTER the upload is completed. The larger the file gets, the larger the chance of a network glitch killing the upload and wasting a good chunk of time and bandwidth. As well, without extra work with APC, or using something like uploadify, there's no progress report and users are left staring at a browser showing no visible signs of actual work except the throbber chugging away.