I'm having a problem like a file upload code.
The user begins to upload files through the site (for large files. Like Wetransfer)
Showing percentage loading with Ajax.
When completed, showing warning.
But the problem starts here.
Because files are huge, it takes time to move to the appropriate folder and ziping.
If the user closes the browser in this process, the process can not be completed.
Even users close the browser, how do I ensure that the operation continues.
I tried to ignore_user_abort. But I was not successful.
So send response to the browser that you are moving file, and or do it as queue and execute it as background job or just do it in your script. That should help: https://stackoverflow.com/a/5997140/4099089
Related
we have a cron that runs a PHP script that processes xml files including processing images. (pulling them from a web address, resizing them and then uploading to CloudFiles.
we are finding that after 220 or so images that we get an error: Exception Received Retry Command Error:Unexpected response ():
we have coded the script to try 5 times to upload it (unfortunately it still fails) and then is to go to the NEXT IMAGE
Unfortunately it fails on the next image and then so on.
The container we are uploading to is not full, we only do 1 image at a time so below the 100/sec restrictions. Files are not large example: http://images.realestateview.com.au/pics/543/10157543ao.jpg" format="jpg"/>
We tried to then run the script again via our server with the image that failed and it worked successfully along with other images.
No idea why this is happening, RackSpace advise it is a issue with the script or the cron. But we are not convinced.
Happy to post script if it helps.
Are you doing 5 retries with any backoff time or just as fast as possible? If not currently, add exponential backoff to the retry attempts.
Im playing with MongoDB and Im trying to import .csv files to DB and Im getting strange error. In process of uploading script just ends for no reason and when I try to run it again nothing happens only solution is to restart apache. I have already set unlimited timeout in php.ini Here is the script.
$dir = "tokens/";
$fileNames = array_diff( scandir("data/"), array(".", "..") );
foreach($fileNames as $filename)
if(file_exists($dir.$filename))
exec("d:\mongodb\bin\mongoimport.exe -d import -c ".$filename." -f Date,Open,Next,Amount,Type --type csv --file ".$dir.$filename."");
I got around 7000 .csv files and it manage to insert only about 200 before script ends.
Can anyone help? I would appreciate any help
You are missing back end infrastructure. It is just insane to try to load 7000 files into a database as part of a web request that is supposed to be short lived and is expected, by some of the software components as well as the end user, to only last a few seconds or maybe a minute.
Instead, create a backend service and command and control for this procedure. In the web app, write each file name to be processed to a database table or even a plain text file on the server and then tell the end user that their request has been queued and will be processed within the next NN minutes. Then have a cron job that runs every 5 minutes (or even 1 minute) that looks in the right place for stuff to do and can create reports of success or failure and/or send emails to tell the original requestor that it is done.
If this is intended as an import script and you are set on using PHP, it would be preferable to at least use the PHP CLI environment instead of performing this task through a web server. As it stands, it appears the CSV files are located on the server itself, so I see no reason to get HTTP involved. This would avoid an issue where the web request terminates and abruptly aborts the import process.
For processing the CSV, I'd start by looking at fgetcsv or str_getcsv. The mongoimport command really does very little in the way of validation and sanitization. Parsing the CSV yourself will allow you to skip records that are missing fields, provide default values where necessary, or take other appropriate action. As you iterate through records, you can collect documents to insert in an array and then pass the results on to MongoCollection::batchInsert() in batches. The driver will take care of splitting up large batches into chunks to actually send over the wire in 16MB messages (MongoDB's document size limit, which also applies to wire protocol communication).
I have a PHP script that downloads files with direct link from a remote server that I own. Sometimes large files (~500-600 MB) and sometimes small files (~50-100 MB).
Some code from the script:
$links[0]="file_1";
$links[0]="file_2";
$links[0]="file_3";
for($i=0;$i<count($links);$i++){
$file_link=download_file($links[$i]); //this function downloads the file with curl and returns the path to the downloaded file in local server
echo "Download complete";
rename($file_link,"some other_path/..."); //this moves the downloaded file to some other location
echo "Downloaded file moved";
echo "Download complete";
}
My problem is if I download large file and run the script from web browser, it takes upto 5-10 minutes to complete and the script echos upto "Download complete" then it dies completely. I always find that the file that was being downloaded before the script dies is 100% downloaded.
On the other hand if I download small files like 50-100MB from web browser or run the script from command shell this problem does not occur at all and the script completes fully.
I am using my own VPS for this and do not have any time limit in the server. There is no fatal error or memory overload problem.
I also used ssh2_sftp to copy files from the remote server. But same problem when I run from web browser. It always downloads the file, executes the next line and then dies! Very strange!
What should I do to get over this problem?
To make sure you can download larger files, you will have to make sure that there is:
enough memory available for php
the maximum execution time limit is set high enough.
Judging from what you said about ssh2_sftp (i assume you are running it via php) your problem is the 2nd one. Check your error(-logs) to find if that truly is your error. If so you simply increase the maximum execution time in your settings/php.ini and that should fix it.
Note: I would encourage you not to let PHP handle these large files. Call some program (via system() or exec()) that will do the download for you as PHP still has garbage collection issues.
I have the following PHP Files:
fileUploadForm.php
handleUpload.php
fileUploadForm contains the following output:
output $_SESSION['errorMessage'] (if any)
Output a file upload form that posts to handleUpload.php
handleUpload.php performs the following actions:
Validates session (redirects to login if validation fails)
Validates file (sets $_SESSION['errorMessage'] if validation fails)
Scan File for Virus
MoveFile
Update database
The script is having trouble on large file uploads. I have set all of the php.ini settings regarding file uploads to be ridiculously huge, for testing purposes. So I don't believe the issue is a configuration issue.
The following behavior is confusing me:
When I watch the file grow in tmp, the file upload continues well past the max_input_time that was set. My understanding was that once the max_input_time is exceeded, the script will terminate, and in turn, so would the file upload. Any thoughts on why this isn't happening?
if I stop the file upload midstream and refresh fileUploadForm (not resubmit it), the script will output error messages related to file validation that are set in handleUpload. This seems to indicate that even though the file upload did not complete, lines of code in handleUpload are being executed. Does php execute a script and receive the form data asynchronously? I would have thought that the script would wait until all form data was received before executing any code. But this assumption is contradicted by the behavior I am seeing. What is the order in which a data POST / script execution occurs?
When max_input_time, along with the rest of the config values, is set to be ridiculously large for testing, very large uploads will complete. However, the rest of the script just seems to die. i.e. the virus scan and file move never happen, nor do the database updates. I have error handling set for each action in the script, but no errors are thrown. The page just seems to have died. Any thoughts on why this might happen / how to catch such an error?
Thanks in advance.
Kate
This quote from your second question answers (at least partially) the other two:
I would have thought that the script would wait until all form data was received before executing any code.
In order for PHP to be able to handle all input data, Apache (or whatever HTTP server you are using) will first wait for the file upload to be complete and only after that it will process the PHP script. So, PHP's max_input_time check will come into play after the file upload process is completed.
Now, in that case you'd probably ask then why your virus scanning, file moving and any other script procedures don't happen, since it's logical that any time counter related to PHP should start with the script's execution and this should happen after all input data is received. Well, that SHOULD be the case and to be honest - my thoughts on this are a kind of shot in the dark, but well ... either some other limit is exceeded or the script is started with the request, but is being suspended by the httpd until ready to proceed with it and effectively - some of those counters might expire during this time.
I don't know how to answer your second question as a refresh would mean that all of the data is re-POST-ed and should be re-processed. I doubt that you'd do the other thing - simply loading handleUpload.php without re-submitting the form, but it's a possibility that I should mention. A secound guess would be that if the first request was terminated unexpectedly - some garbage collection and/or recovery process happens the second time.
Hope that clears it up a bit.
I am using a cURL based php application to make requests to another webserver that does asynchronous requests. So what I am doing is creating files with the name as .req with the info I will need on the return and as the identification in the request. The requests are done using HTTP-XML-POST. The file is written using: -
file_get_contents(reqs/<databaseid>.req, FILE_APPEND);
What happens is that while the requests are being generated in bulk (about 1500 per second), the responses start coming back from the webserver. The response is caught by a another script which received the from the response and opens the request file based on it using: -
$aResponse = file(reqs/<databaseid>.req);
Now what happens is that in about 15% of requests, the file() request fails and generates a log entry in apache log like this: -
file(reqs/<databaseid>.req): failed to open stream: No such file or directory in <scriptname> on line <xyz>
It has been verified using a cleaner script that runs later that the file did exist.
Any ideas?!!!
There are some functions to handle simultaneous file access such as flock() but it's normally easier to simply use a database. Any decent DBMS has already worked it out for you.