Here's a thing- there is a file on external server.
My script needs to send the contents of this file to the visitor.
So it needs to download a file from the external server and then upload to the visitor. However the file is huge (>200 MB) so I cannot download file first then upload it- it would be too slow. And also I need to support resuming the download.
And that's the problem I am not sure how to handle properly.
One idea is:
start wget to download the file in background
then my script
in PHP will send the contents of the file as it is being downloaded.
But what if wget is slower then client's browser? Should my script
wait?
Another one:
Use curl in PHP to download a file and on progress output what I downloaded. But then I am not sure how to handle resuming
I am not really sure how should I handle this situation.
I think that one way can be to use two different sockets. One of them that download the data and put the readed byte in a buffer where the other socket reads and send these data to the server where you want to upload the file.
Related
I have numerous storage servers, and then more "cache" servers which are used to load balance downloads. At the moment I use RSYNC to copy the most popular files from the storage boxes to the cache boxes, then update the DB with the new server IDs, so my script can route the download requests to a random box which has the file.
I'm now looking at better ways to distribute the content, and wondering if it's possible to route requests to any box at random, and the download script then check if the file exists locally, if it doesn't, it would "get" the file contents from the remote storage box, and output the content in realtime to the browser, whilst keeping the file on the cache box so that the next time the same request is made, it can just serve the local copy, rather than connecting to the storage box again.
Hope that makes sense(!)
I've been playing around with RSYNC, wget and cURL commands, but I'm struggling to find a way to output the data to browser as it comes in.
I've also been reading up on reverse proxies with nginx, which sounds like the right route... but it still sounds like they require the entire file to be downloaded from the origin server to the cache server before it can output anything to the client(?) some of my files are 100GB+ and each server has a 1gbps bandwidth limit, so at best, it would take 100s to download a file of that size to the cache server before the client will see any data at all. There must be a way to "pipe" the data to the client as it streams?
Is what I'm trying to achieve possible?
You can pipe data without downloading the full file using streams. One example for downloading a file as a stream would be the Guzzle sink feature. One example for uploading a file as a stream would be the Symfony StreamedResponse. Using those the following can be done:
Server A has a file the user wants
Server B gets the user request for the file
Server B uses Guzzle to setup a download stream to server A
Server B outputs the StreamedResponse directly to the user
Doing so will serve the download in real-time without having to wait for the entire file to be finished. However I do not know if you can stream to the user and store the file on disk at the same time. There's a stream_copy_to_stream function in PHP which might allow this, but don't know that for sure.
I am trying to process the user uploaded file real time on the websever,
but it seems, APACHE invokes PHP, only once complete file is uploaded.
When i uploaded the file using CURL, and set
Transfer-Encoding : "Chunked"
I had some success, but can't do same thing via browser.
I used Dropzone.js but when i tried to set same header, it said Transfer -Encoding is an unsafe header, hence not setting it.
This answer explains what is the issue there.
Can't set Transfer-Encoding :"Chunked from Browser"
In a Nutshell problem is , when a user uploads the file to webserver, i want webserver to start processing it as soon as first byte is available.
by process i mean, PIPING it to a Named Pipe.
Dont want 500mb first getting uploaded to a server, then start processing it.
But with current Webserver (APACHE - PHP), I cant seem to be able to accomplish it.
could someone please explain, what technology stack or workarounds to use, so that i can upload the large file via browser and start processing it, as soon as first byte is available.
It is possible to use NodeJS/Multiparty to do that. Here they have an example of a direct upload to Amazon S3. This is the form, which sets content type to multipart/form-data. And here is the function for form parts processing. part parameter is of type ReadableStream, which will allow per-chunk processing of the input using data event.
More on readable streams in node js is here.
If you really want that (sorry don`t think thats a good idea) you should try looking for a FUSE Filesystem which does your job.
Maybe there is already one https://github.com/libfuse/libfuse/wiki/Filesystems
Or you should write your own.
But remember as soon as the upload is completed and the post script finishes his job the temp file will be deleted
you can upload file with html5 resumable upload tools (like Resumable.js) and process uploaded parts as soon as they received.
or as a workaround , you may find the path of uploaded file (usually in /tmp) and then write a background job to stream it to 3rd app. it may be harder.
there may be other solutions...
If I do
$file='http://notmywebsite.com/verybigimage.png';
$newfile='test.png';
copy($file,$newfile);
will my server download http://notmywebsite.com/verybigimage.png?
If yes, how can I make my user download http://notmywebsite.com/verybigimage.png without that my server download it?
It's not possible to have the client (the user) download a file and then copy this file in the server in a seamless way, because these two environments are living in different systems — in order to do this, the client needs to upload the file back to the server after downloading it, which in turn would be a pain for the user and slower than simply letting the server download the file and copying it there, if the ultimate goal is having a copy of the file in the server.
Is it possible to upload files (even big files) to a ftp using PHP?
Been reading about ftp_connect() and it looks like I can, or can't I?
I had a look at this example, it's in Italian but you can read the code anyway, if that does what I'm asking, will I have to add an html form? I need to be able to pick up a file form my computer via a web page and upload it to an ftp basically.
Anyone?
Thanks
Especially on large files, you should make sure that the maximum execution time for the script is big enough to complete the transfer before the script is aborted. You can choose the maximum execution time in the php.ini file.
You will have to use an HTML form if you want to pick up the file from your computer via a web page.
As soon as the form is submitted, you can access the file using the $_FILES array. You can use this information to get a temporary path to where the file is stored, and can read it from there to upload it to a remote server using the FTP functions.
You could also split the two processes by using the PHP script only to drop the file into the local file system, and then use a second program which runs locally to do the upload. This has the advantage that you won't run into problems when multiple users upload simultaneously and your FTP is set up in a way that it allows only 1 simultaneous connection. You could program the second script also in PHP and run it using a Cronjob for example once per 30min.
If your goal is a direct stream from your computer to the FTP server, this is not easily possible using a pure PHP / HTML solution since the PHP script is only invoked when the file transfer from your computer to the machine serving the PHP script is complete.
I'd like to have my PHP script upload a file with a certain filename in a directory of my choosing. However, the catch is that I need it to exist there immediately upon upload so I can moniter it on my server. I don't want to use a PHP extension or something - this should be very easy to transfer to any PHP setup.
So basically: Is there a way to guarantee that, from the very beginning of the file upload process, the file has a certain name and location on the server?
Not that I'm aware of.
PHP will use the php.ini-defined tmp folder to store uploads until you copy them to their correct location with move_uploaded_file(). So it's very easy to know its location, but the file name is random and I don't think you can define it.
If you're not going to have multiple concurrent uploads (for example if only you are going to upload files and you know you won't upload 2 files at the same time), you could check the most recent upload file in the tmp directory.
The common solution for monitoring uploads is apc.rfc1867
I know of three options:
RFC1867 (as mentioned by others) which allows you to poll upload progress using ajax
Flash-based uploaders like SWFUpload which allow you to poll upload progress using JavaScript
Create a PHP command line daemon listening on port 80 that accepts file uploads, and used shared memory (or some other mechanism) to communicate upload progress. Wish I could find the link, but I read a great article about a site that allowed users to upload their iTunes library XML file, and it was processed live by the server as it was being uploaded. Very cool, but obviously more involved than the previous options.
I have had decent luck with SWFUpload in the past.
I don't think you can configure the name, as it will be a random name in the temporary folder. You should be able to change the directory, but I can't seem to find the answer on Google (check out php.ini).
As far as I know, this isn't possible with PHP, as a file upload request submits the entire file to the system in one request. So there is no way for the PHP server to know what is happening until it receives the whole request.
There is not a way to monitor file upload progress using PHP only, as PHP does not dispatch progress events during the upload. This is possible to do using a Flash uploader even if Flash is uploading via a PHP script. Flash polls the temporary file on the server during the upload to dispatch progress events. Some of the javascript frameworks like YUI use a SWF to manage uploads. Check out YUI's Uploader widget.
http://developer.yahoo.com/yui/uploader/