I am trying to process the user uploaded file real time on the websever,
but it seems, APACHE invokes PHP, only once complete file is uploaded.
When i uploaded the file using CURL, and set
Transfer-Encoding : "Chunked"
I had some success, but can't do same thing via browser.
I used Dropzone.js but when i tried to set same header, it said Transfer -Encoding is an unsafe header, hence not setting it.
This answer explains what is the issue there.
Can't set Transfer-Encoding :"Chunked from Browser"
In a Nutshell problem is , when a user uploads the file to webserver, i want webserver to start processing it as soon as first byte is available.
by process i mean, PIPING it to a Named Pipe.
Dont want 500mb first getting uploaded to a server, then start processing it.
But with current Webserver (APACHE - PHP), I cant seem to be able to accomplish it.
could someone please explain, what technology stack or workarounds to use, so that i can upload the large file via browser and start processing it, as soon as first byte is available.
It is possible to use NodeJS/Multiparty to do that. Here they have an example of a direct upload to Amazon S3. This is the form, which sets content type to multipart/form-data. And here is the function for form parts processing. part parameter is of type ReadableStream, which will allow per-chunk processing of the input using data event.
More on readable streams in node js is here.
If you really want that (sorry don`t think thats a good idea) you should try looking for a FUSE Filesystem which does your job.
Maybe there is already one https://github.com/libfuse/libfuse/wiki/Filesystems
Or you should write your own.
But remember as soon as the upload is completed and the post script finishes his job the temp file will be deleted
you can upload file with html5 resumable upload tools (like Resumable.js) and process uploaded parts as soon as they received.
or as a workaround , you may find the path of uploaded file (usually in /tmp) and then write a background job to stream it to 3rd app. it may be harder.
there may be other solutions...
Related
As all of us know, PHP finishes the upload and the enables you to use move_uploaded_file(); before this, however, it creates a temp file and then does the job. I want to know is it possible to get the name of this uploaded file during the file upload and before populating it into $_FILES?
I want to get the upload progress, while $_SESSION and Javascript onprogress solution both suck..
$_FILES['file']['tmp_name']; is the filename. It is not possible in PHP (without using ugly tricks) to get the filename before the upload is finished.
To do this, you have to fallback on either Flash (uploadify) or CGI (Perl / Python / C++ / Other)
A "reliable" progress bar, which seems to be your goal, will always require some sort of server and client support. In its most general and portable instance, PHP will see only the completed upload and you'll get no progress bar, but only the filled $_FILES structure.
On some platforms the information can be garnered from the system itself. For example under Linux/Apache you can inspect what temporary files Apache has opened in the /proc pseudo-filesystem, where available; so you need to put in the requisites "Linux, Apache, php5_module, /proc".
You can use a dedicated POST endpoint that does not terminate on the Web server, but on a specially crafted uploader process (I worked on a Perl script doing this years ago; I recall it used POE, and the architecture):
POST (from browser) ==> (server, proxying) ==> UPLOADER
The uploader immediately echoes a crafted GET to the server, activating
a PHP "pre-upload" page, and then might call a progress GET URL periodically
to update the upload status. When completed, it would issue a pseudo POST
to PHP "almost" as if it came from the client, sending $_POST['_FILES']
instead of $_FILES.
The $_SESSION solution is a good compromise but relies on the server not doing buffering.
A better and more "modern" solution would be to leverage the chunked upload AJAX trick and get resumable uploads, reliable progress and large file support all in one nifty package. See for example this other answer. Now you get wider server support but the solution won't work on some older browsers.
You could offer the user the choice between old-style FILE upload, Flash uploader (which bypasses all problems as it doesn't rely on the browser but on Flash code), Java FTP upload control (same thing, but sometimes with some protocol and firewall issues since it doesn't use HTTP as the container web page does), and AJAX HTML5 chunking, possibly based on browser capabilities.
I.e., a user with IE6 would see a form saying
SORRY!
Your browser does not support large file uploads and progress bar.
To send a file of no more than XXX meg,
[ ] [Choose file...] [ >> BEGIN UPLOAD >>> ]
Folks
I have an image at some server (SOURCE)
i.e. http://stagging-school-images.s3.amazonaws.com/2274928daf974332ed4e69fddc7a342e.jpg
Now I want to upload it to somewhere else (DESTINATION)
i.e. example.mysite.com/receiveImage.php
First, I am copying image from source to my local server and then uploading it to destination.
It's perfectly working but taking too much time as it copy the image and then uploads...
I want to make it more simple and optimized by directly uploading image from source URL to destination URL.
Is there a way to handle this ?
I am using php/cURL to handle my current functionality.
Any help would be very much appreciated.
Cheers !!
If example.mysite.com/receiveImage.php is your own service, then you may
pass SOURCE URL to your PHP script as GET or POST parameter
in PHP script, use file_get_contents() function to obtain image by URL, and save it to your storage
Otherwise it's impossible by means of HTTP.
However, there are some ways to increase files uploading speed a little:
If files are huge, you may use two threads: one for downloading (it will store all downloaded data to some buffer) and one for uploading (it will get all available data from buffer and upload it to site). As far as I know, this can't be done easily with PHP, because multi-threading is currently not supported yet.
If there are too many files, you may use many threads / processes, which will do download/upload simultaneously.
By the way, these means do not eliminate double traffic for your intermediate service.
One of the services may have a form somewhere that will allow you to specify a URL to receive from/send to, but there is no generic HTTP mechanism for doing so.
copy($url, $uploadDir.'/'.$fileName);
The only way to transfer the image directly from source to destination is to initiate the transfer from either the source or the destination. You can't magically beam the data between these two locations without them talking to each other directly. If you can SSH login to your mysite.com server you could download the image directly from there. You could also write a script that runs on mysite.com and directly downloads the image from the source.
If that's not possible, the best alternative may be to play around with fread/fwrite instead of curl. This should allow you to read a little bit from the source, then directly upload that bit to the destination so download and upload can work in parallel. For huge files this should make a real difference, for small files on a decent connection it probably won't.
create two textfield one url, other filename
in php, use :
uploadDir is path to your file directory ;)
copy($url, $uploadDir.'/'.$fileName);
I have created a PHP script to upload a file, unfortunately I don't have permission to save files on the disk. I have to upload an excel file (using phpexcel), then I have to read all the rows in the file and save to disk, Is there any way for me to process this file without saving to disk, I tried to read $_FILES['file1']['tmp_name'] but it doesn't work.
could u please suggest a method to process this file
Thank you for the consideration
By "save to disk" you mean to send it back to the user for him to download it?
Usually, you shall have write access to (at least) the PHP temporary directory. Have you tried whether the form and script work in a local environment? Maybe there is something elso wrong with the upload?!
Finally: Why so you not have the persmission to save files? Are you allowed to create a subdirectory below you PHP file (via FTP) and give that one full permissions?
I tried to read $_FILES['file1']['tmp_name']
most probably you have just encountered an error.
that happens to beginner programmers very often
you have to repair that error instead of looking for odd workarounds.
Start from checking $_FILES['file1']['error']
what does
var_dump($_FILES['file1']['error']);
say?
Instead of sending your files with a form (multidata over HTTP POST), you can send your files with a little bit of Javascript with the HTTP PUT method to your server.
This scenario is described in the official documentation of PHP -> PUT method support.
Due some restrictions described in the documentation you have to do some workarounds to be able to work it properly.
You can read the direct input stream from your Webserver. The data will be piped from your Webserver to your PHP programm and will be only saved in memory.
To do a PUT Ajax call with jQuery was answered here. You can use a jQuery upload plugin like Uploadify.
I'd like to have my PHP script upload a file with a certain filename in a directory of my choosing. However, the catch is that I need it to exist there immediately upon upload so I can moniter it on my server. I don't want to use a PHP extension or something - this should be very easy to transfer to any PHP setup.
So basically: Is there a way to guarantee that, from the very beginning of the file upload process, the file has a certain name and location on the server?
Not that I'm aware of.
PHP will use the php.ini-defined tmp folder to store uploads until you copy them to their correct location with move_uploaded_file(). So it's very easy to know its location, but the file name is random and I don't think you can define it.
If you're not going to have multiple concurrent uploads (for example if only you are going to upload files and you know you won't upload 2 files at the same time), you could check the most recent upload file in the tmp directory.
The common solution for monitoring uploads is apc.rfc1867
I know of three options:
RFC1867 (as mentioned by others) which allows you to poll upload progress using ajax
Flash-based uploaders like SWFUpload which allow you to poll upload progress using JavaScript
Create a PHP command line daemon listening on port 80 that accepts file uploads, and used shared memory (or some other mechanism) to communicate upload progress. Wish I could find the link, but I read a great article about a site that allowed users to upload their iTunes library XML file, and it was processed live by the server as it was being uploaded. Very cool, but obviously more involved than the previous options.
I have had decent luck with SWFUpload in the past.
I don't think you can configure the name, as it will be a random name in the temporary folder. You should be able to change the directory, but I can't seem to find the answer on Google (check out php.ini).
As far as I know, this isn't possible with PHP, as a file upload request submits the entire file to the system in one request. So there is no way for the PHP server to know what is happening until it receives the whole request.
There is not a way to monitor file upload progress using PHP only, as PHP does not dispatch progress events during the upload. This is possible to do using a Flash uploader even if Flash is uploading via a PHP script. Flash polls the temporary file on the server during the upload to dispatch progress events. Some of the javascript frameworks like YUI use a SWF to manage uploads. Check out YUI's Uploader widget.
http://developer.yahoo.com/yui/uploader/
I need to upload potentially big (as in, 10's to 100's of megabytes) files from a desktop application to a server. The server code is written in PHP, the desktop application in C++/MFC. I want to be able to resume file uploads when the upload fails halfway through because this software will be used over unreliable connections. What are my options? I've found a number of HTTP upload components for C++, such as http://www.chilkatsoft.com/refdoc/vcCkUploadRef.html which looks excellent, but it doesn't seem to handle 'resume' of half done uploads (I assume this is because HTTP 1.1 doesn't support it). I've also looked at the BITS service but for uploads it requires an IIS server. So far my only option seems to be to cut up the file I want to upload into smaller pieces (say 1 meg each), upload them all to the server, reassemble them with PHP and run a checksum to see if everything went ok. To resume, I'd need to have some form of 'handshake' at the beginning of the upload to find out which pieces are already on the server. Will I have to code this by hand or does anyone know of a library that does all this for me, or maybe even a completely different solution? I'd rather not switch to another protocol that supports resume natively for maintenance reasons (potential problems with firewalls etc.)
I'm eight months late, but I just stumbled upon this question and was surprised that webDAV wasn't mentioned. You could use the HTTP PUT method to upload, and include a Content-Range header to handle resuming and such. A HEAD request would tell you if the file already exists and how big it is. So perhaps something like this:
1) HEAD the remote file
2) If it exists and size == local size, upload is already done
3) If size < local size, add a Content-Range header to request and seek to the appropriate location in local file.
4) Make PUT request to upload the file (or portion of the file, if resuming)
5) If connection fails during PUT request, start over with step 1
You can also list (PROPFIND) and rename (MOVE) files, and create directories (MKCOL) with dav.
I believe both Apache and Lighttpd have dav extensions.
You need a standard size (say 256k). If your file "abc.txt", uploaded by user x is 78.3MB it would be 313 full chunks and one smaller chunk.
You send a request to upload stating filename and size, as well as number of initial threads.
your php code will create a temp folder named after the IP address and filename,
Your app can then use MULTIPLE connections to send the data in different threads, so you could be sending chunks 1,111,212,313 at the same time (with separate checksums).
your php code saves them to different files and confirms reception after validating the checksum, giving the number of a new chunk to send, or to stop with this thread.
After all thread are finished, you would ask the php to join all the files, if something is missing, it would goto 3
You could increase or decrease the number of threads at will, since the app is controlling the sending.
You can easily show a progress indicator, either a simple progress bar, or something close to downthemall's detailed view of chunks.
libcurl (C api) could be a viable option
-C/--continue-at
Continue/Resume a previous file transfer at the given offset. The given offset is the exact number of bytes that will be skipped, counting from the beginning of the source file before it is transferred to the destination. If used with uploads, the FTP server command SIZE will not be used by curl.
Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.
If this option is used several times, the last one will be used
Google have created a Resumable HTTP Upload protocol. See https://developers.google.com/gdata/docs/resumable_upload
Is reversing the whole proccess an option? I mean, instead of pushing file over to the server make the server pull the file using standard HTTP GET with all bells and whistles (like accept-ranges, etc.).
Maybe the easiest method would be to create an upload page that would accept the filename and range in parameter, such as http://yourpage/.../upload.php?file=myfile&from=123456 and handle resumes in the client (maybe you could add a function to inspect which ranges the server has received)
# Anton Gogolev
Lol, I was just thinking about the same thing - reversing whole thing, making server a client, and client a server. Thx to Roel, why it wouldn't work, is clearer to me now.
# Roel
I would suggest implementing Java uploader [JumpLoader is good, with its JScript interface and even sample PHP server side code]. Flash uploaders suffer badly when it comes to BIIIGGG files :) , in a gigabyte scale that is.
F*EX can upload files up to TB range via HTTP and is able to resume after link failures.
It does not exactly meets your needs, because it is written in Perl and needs an UNIX based server, but the clients can be on any operating system. Maybe it is helpful for you nevertheless:
http://fex.rus.uni-stuttgart.de/
Exists the protocol called TUS for resumable uploads with some implementations in PHP and C++