I am using the YoutubeDL library in a project. My environment is based on WINDOWS with XAMPP as the webserver boundle (apache,php,mysql,etc). I am using the youtube-dl.exe file to download the video and then use ffmpeg.exe to convert the video to an MP3 audio file.
At the moment, I have an issue related to programming: I want to show live a progressbar while the video is downloaded with the youtube-dl.exe file. This exe creates a log file, that is updated while the video is downloaded. So my approach on this was to create a PHP file, that opens, parses the log file and get's the progress percent, and sends it as a json encoded value to an AJAX function that is called every 100MS. Indeed, if the video is too large, there will be a very high ammount of data while polling the PHP file to get the progress state. And sometimes, the browser either crashes or freezes because of this ajax polling.
My question is: is there any better approach to do this with PHP/AJAX? Rathar than poll the file every 100MS, or 50MS?
Poll only every second and animate the progress bar.
Related
I have numerous storage servers, and then more "cache" servers which are used to load balance downloads. At the moment I use RSYNC to copy the most popular files from the storage boxes to the cache boxes, then update the DB with the new server IDs, so my script can route the download requests to a random box which has the file.
I'm now looking at better ways to distribute the content, and wondering if it's possible to route requests to any box at random, and the download script then check if the file exists locally, if it doesn't, it would "get" the file contents from the remote storage box, and output the content in realtime to the browser, whilst keeping the file on the cache box so that the next time the same request is made, it can just serve the local copy, rather than connecting to the storage box again.
Hope that makes sense(!)
I've been playing around with RSYNC, wget and cURL commands, but I'm struggling to find a way to output the data to browser as it comes in.
I've also been reading up on reverse proxies with nginx, which sounds like the right route... but it still sounds like they require the entire file to be downloaded from the origin server to the cache server before it can output anything to the client(?) some of my files are 100GB+ and each server has a 1gbps bandwidth limit, so at best, it would take 100s to download a file of that size to the cache server before the client will see any data at all. There must be a way to "pipe" the data to the client as it streams?
Is what I'm trying to achieve possible?
You can pipe data without downloading the full file using streams. One example for downloading a file as a stream would be the Guzzle sink feature. One example for uploading a file as a stream would be the Symfony StreamedResponse. Using those the following can be done:
Server A has a file the user wants
Server B gets the user request for the file
Server B uses Guzzle to setup a download stream to server A
Server B outputs the StreamedResponse directly to the user
Doing so will serve the download in real-time without having to wait for the entire file to be finished. However I do not know if you can stream to the user and store the file on disk at the same time. There's a stream_copy_to_stream function in PHP which might allow this, but don't know that for sure.
Here's a thing- there is a file on external server.
My script needs to send the contents of this file to the visitor.
So it needs to download a file from the external server and then upload to the visitor. However the file is huge (>200 MB) so I cannot download file first then upload it- it would be too slow. And also I need to support resuming the download.
And that's the problem I am not sure how to handle properly.
One idea is:
start wget to download the file in background
then my script
in PHP will send the contents of the file as it is being downloaded.
But what if wget is slower then client's browser? Should my script
wait?
Another one:
Use curl in PHP to download a file and on progress output what I downloaded. But then I am not sure how to handle resuming
I am not really sure how should I handle this situation.
I think that one way can be to use two different sockets. One of them that download the data and put the readed byte in a buffer where the other socket reads and send these data to the server where you want to upload the file.
I work with radiology medical reports. Doctors record the report in a audio file and close the browser after this, so the browser doesn't have time to upload the file. I tried to put a loading with a warning about the time to upload and the need to stay with browser open, but they don't respect it and the accuracy of loading screen is not very correct.
My question is: since I have a audio file encoded in base64, there is a way to run this upload in background or keep this to upload in another screen request?
My point is, there is a way to upload the files without any dependency of the user?
If it helps, I`m using CodeIgniter.
Thanks
I have three weeks looking for this:
I have this page which have to download a file after verifying the data given by the user. I do the validation and execute an external script which gives me an URL. And I use that URL to download another file which I'm gonna execute later.
I'd tried to download the file with curl, wget, file_put_content, javascript, ajax and jquery with no luck, the file is 150MB+ so I created a nice progress bar to tell the user how the download is going and already have a method to read the downloaded size.
I fact I'm able to download the file with cURL, but the problem is that the "do_form.php" won't load until the file is completely downloaded, so I want to load the page first, then download the file - in the background -, so I can show the user the progress of the download.
Please tell me that this is possible... Thanks!!
Have a look at Gearman. Maybe a gearman background job fits your needs. This job can even run on the same machine as the web server. Of course the early response to the page has the disadvantage that it contains no success information. You will need to poll for that information using AJAX again.
If you decide to do so, you should also have a look at GearmanManager as this eases things a lot.
I'd like to have my PHP script upload a file with a certain filename in a directory of my choosing. However, the catch is that I need it to exist there immediately upon upload so I can moniter it on my server. I don't want to use a PHP extension or something - this should be very easy to transfer to any PHP setup.
So basically: Is there a way to guarantee that, from the very beginning of the file upload process, the file has a certain name and location on the server?
Not that I'm aware of.
PHP will use the php.ini-defined tmp folder to store uploads until you copy them to their correct location with move_uploaded_file(). So it's very easy to know its location, but the file name is random and I don't think you can define it.
If you're not going to have multiple concurrent uploads (for example if only you are going to upload files and you know you won't upload 2 files at the same time), you could check the most recent upload file in the tmp directory.
The common solution for monitoring uploads is apc.rfc1867
I know of three options:
RFC1867 (as mentioned by others) which allows you to poll upload progress using ajax
Flash-based uploaders like SWFUpload which allow you to poll upload progress using JavaScript
Create a PHP command line daemon listening on port 80 that accepts file uploads, and used shared memory (or some other mechanism) to communicate upload progress. Wish I could find the link, but I read a great article about a site that allowed users to upload their iTunes library XML file, and it was processed live by the server as it was being uploaded. Very cool, but obviously more involved than the previous options.
I have had decent luck with SWFUpload in the past.
I don't think you can configure the name, as it will be a random name in the temporary folder. You should be able to change the directory, but I can't seem to find the answer on Google (check out php.ini).
As far as I know, this isn't possible with PHP, as a file upload request submits the entire file to the system in one request. So there is no way for the PHP server to know what is happening until it receives the whole request.
There is not a way to monitor file upload progress using PHP only, as PHP does not dispatch progress events during the upload. This is possible to do using a Flash uploader even if Flash is uploading via a PHP script. Flash polls the temporary file on the server during the upload to dispatch progress events. Some of the javascript frameworks like YUI use a SWF to manage uploads. Check out YUI's Uploader widget.
http://developer.yahoo.com/yui/uploader/