Given a simple code like :
$file = 'hugefile.jpg';
$bckp_file = 'hugeimage-backup.jpg';
// here comes some manipulation on $bckp_file.
The assumed problem is that if the file is big or huge - let´s say a jpg - One would think that it will take the server some time to copy it (by time I mean even a few milliseconds) - but one would also assume that the execution of the next line would be much faster ..
So in theory - I could end up with "no such file or directory" error when trying to manipulate file that has not yet created - or worse - start to manipulate a TRUNCATED file.
My question is how can I assure that $bckp_file was created (or in this case -copied) successfully before the NEXT line which manipulates it .
What are my options to "pause" , "delay" the next line execution until the file creation / copy was completed ?
right now I can only think of something like
if (!copy($file, $bckp_file)) {
echo "failed to copy $file...\n";
}
which will only alert me but will not resolve anything (same like having the php error)
or
if (copy($file, $bckp_file)) {
// move the manipulation to here ..
}
But this is also not so valid - because let´s say the copy was not executed - I will just go out of the loop without achieving my goal and without errors.
Is that even a problem or am I over-thinking it ?
Or is PHP has a bulit-in mechanism to ensure that ?
Any recommended practices ?
any thoughts on the issue ? ??
What are my options to "pause" , "delay" the next line execution until the file is creation / copy was completes
copy() is a synchronous function meaning that code will not continue after the call to copy() until copy() either completely finishes or fails.
In other words, there's no magic involved.
if (copy(...)) { echo 'success!'; } else { echo 'failure!'; }
Along with synchronous IO, there is also asynchronous IO. It's a bit complicated to explain in technical detail, but the general idea of it is that it runs in the background and your code hooks into it. Then, whenever a significant event happens, the background execution alerts your code. For example, if you were going to async copy a file, you would register a listener to the copying that would be notified when progress was made. That way, your code could do other things in the mean time, but you could also know what progress was being made on the file.
PHP handles file uploads by saving the whole file in a temporary directory on the server before executing any of script (so you can use $_FILES from the beginning), and it's safe to assume all functions are synchronous -- that is, PHP will wait for each line to execute before moving to the next line.
Related
I have a PHP script which the execution time used to be a few milliseconds.
But yesterday I got complaints that it loads forever. I checked into it and nailed down the problem to a line that uses require_once(); -- execution time is about two minutes just for that line!
The file to be included contains a bunch of functions and in itself doesn't do anything at that point. It also is just about 35kb in size.
I went through the script logging the microtime() and this is the output:
0.10887400 1442934181 // start of script
0.13321200 1442934181 // line before "require_once()"
0.16033800 1442934307 // log time again in first line
// of the included functions file
0.16048000 1442934307 // back to original script,
// line after require_once()
0.16054300 1442934307 // end of script
Just for curiosity I've tried to replace require_once() with require() -- to change.
I don't know what the cause could be and where I should start to debug. It has worked before without problem, and I haven't done any change.
require_once and require need to access to the file, read it, and execute it. So you may have multiple problems, but you'll have to check :
Maybe your hard disk is going bad ?
Maybe the file is locked by some others process ?
The file contains now a big function doing too much things
Hello i need some help with this code in the install.php which has to run first before the program but it brings an error pointing on the fflush i don't know what to do please help?
<?php
fflush();
authTableCreate();
announceTableCreate();
classTableCreate();
studentTableCreate();
classmembersTableCreate();
attendanceTableCreate();
assignmentTableCreate();
messageTableCreate();
supportTableCreate();
if (!authCheckUserExists('admin')) { // this is their first install probably
$randpass = 'admin' . mt_rand();
authAddUser('admin', $randpass, 100, 100); // create default superuser account
announceAddAnnouncement('Welcome', 'This is the default start page for IntraSchool. You should change this to something that describes how the system works within your school.');
?>
<p>Congratulations! You have successfully setup <em>IntraSchool</em>. You may now login with the username <em>admin</em> and password <em><?=$randpass?></em>. Please change the password to something you can remember.</p>
<?php
} else {
?>
<p>The installation script has reinitialized any deleted tables.</p>
<?php
}
page_footer();
?>
fflush() requires the handle of the file to be flushed. It is likely a typo for flush(), however as it's apparently at the start of the file that would do nothing at all. You should just delete the line.
It's only a warning though, so the rest of the script has probably been executed. If it's a once-only setup script then you probably do not need to run it again.
Here's the documentation - always a good place to start.
My understanding of your code is limited, so I'm not sure what you're trying to accomplish here (in particular, it looks like you're doing database operations, for which fflush should not be necessary). That said, here's a little background:
fflush flushes an open file to disk. You need to provide it with a file handle to flush.
When you're writing to a file on your disk, the operating system will often store up a bunch of your data and write it all to the disk at one time, rather than writing each byte as you send it. This is primarily for performance reasons. Sometimes, however, you need to get that data written at a particular point in your program. That's what fflush is for. But for fflush to work, you need to tell it what file you're talking about - that's the file handle mentioned in the documentation.
I have a PHP page that should be accessible only by one user at a time. It's a kind of "poor man's cron": A "Javascript" file, that is requested in the background. At the moment I'm thinking of creating a lock file like this:
if(file_exists($lockfile) && filemtime($lockfile) + EXPIRES_AFTER > time() ) {
die("// Page is locked.");
}
touch($lockfile);
// do stuff
unlink($lockfile);
echo "// Cron stuff was run";
But I'm not sure if there could be a very short window of opportunity between the file_exists and the touch call where another page request could check for the file existence and see that it isn't there. We're probably talking microseconds here so I would like top know at which amount of requests I really need to start worrying.
If you want to do this really precisely, then use a different approach, because there IS some time between the check and the lock.
Two possible implementations:
Use flock:
https://secure.php.net/manual/en/function.flock.php
Use something like STM:
E.g. open the lockfile for append, write something into it, close the handle. Then read the file back, and if it only has what you wrote into it then you have acquired the lock.
Other than that, your original code would probably not cause any problems.
There is no atomicity in the code you wrote, so yes, there is a race condition.
Your code has a race condition. Instead, dio_open the file with O_EXCL. This will fail if the file already exists. Unlink it when you're done.
The only thing to watch out for is if the system or the script crashes while the file exists, the script will never run again. If you are worried about this, check the age of the file (if you fail to create it) and if it's older than the longest the script could ever take, unlink it.
Using flock is another option.
is there a way for me to check to see if a file is copied before continuing to execute a php loop?
i have a for loop, and within the loop it is going to copy a file. now, i want it so that it waits until the current file is copied before continuing the loop.
example:
for ($i = 1; $i <= 10; $i++)
{
$temp = $_FILES['tmp_name'];
$extension = '.jpg';
copy("$temp_$i_$extension", "$local_$i_$extension");
// not sure what to do here
if (FILE_DONE_COPYING())
{
CONTINUE_LOOP();
}
else
{
PAUSE_LOOP();
}
}
thats just an example. i have no clue how to do this...can anyone chime in?
That's what copy() does in PHP - it blocks until the file is copied. There's nothing you need to do, except checking the return value to see if the operation was successful.
PHP is taking it line by line, step by step, so it's waiting until copy() is completed
for ($i = 1; $i <= 10; $i++)
{
$temp = $_FILES['tmp_name'];
$extension = '.jpg';
$result = copy("$temp_$i_$extension", "$local_$i_$extension");
if($result){
//done
}
else{
//failed
}
}
copy returns true on success and false on failure. Check for that.
Unless you go through the trouble of using threading and have copy fired asynchronously, PHP will not move to the line after copy until after it has completed.
copy does wait for completion before continuing execution. It is a syncronous call. But, it can return false if it didn't work, and your copy wont work since $temp_ and $i_ are not defined variables. So maybe you are thinking the copy isn't finishing, when it actually just isn't working at all.
You should use:
copy("{$temp}_{$i}_$extension", "{$local}_{$i}_$extension");
OR
copy($temp.'_'.$i.'_'.$extension, $local.'_'.$i.'_'.$extension);
What makes you think that copy() will return before it has finished?
You could of course compare filesize of original file and copy to be sure the process is complete.
You could use a while loop with sleep calls to delay checking, and just exit the while loop once the file exists under the new name.
I know this is an ancient question but I feel I really need to talk about this problem. COPY is a great command - BUT - it does not work all of the time. I can honestly tell you this. Why? Why does it not always work? Simple - the Operating System is at fault. Here are two examples. One is using a standard disk drive and the second one deals with a Ram disk. The COPY command reads CHUNKS of a file and writes these chunks out to the destination. This means it really does NOT just do a File_get_contents but instead does the fopen(IN), fopen(OUT), while( !EOF(IN) ){ fread(IN), fwrite(OUT) } and then fclose(IN), and fclose(out). It should be noted that these commands try to make sure everything goes ok but if the disk drive buffers what it does - then the file might take a second or two to finish. This can be seen by having a file_exists() on the output file's name. It can come back as FALSE(it IS NOT there). This is because the disk drive's hardware has not caught up with the commands.
I even installed the AMD RamDisk software and ran a program using the above commands (both file_get_contents->file_put_contents and the fopen-fread/fwrite-fclose commands). The same thing happened then also. Every now and then (not always) the file_exists() function returned FALSE because the test got there before the file had finished being created. Don't ask me why - it just would do this.
So what do I suggest? Use the SLEEP() command. Maybe use three(3) seconds (so SLEEP(3);) -after- the COPY() command. I also determined that a CHMOD(, 0777); was a good idea also. With a SLEEP() command after it so it has time to apply the changes. (Which is probably closer to one second).
Now, remember - everyone's hardware is different. So some hardware might work better or faster than the one I am using. So - this is one of those - try it if you are having problems. Or don't - if you are not having problems. It is that simple. So - this is happening to me - I'm using it - it works now that the system gets three seconds to breath - but it might not do anything for you - who has an atomic powered Willy-Wonka mobile which does the impossible before breakfast.
Got it? Good. :-)
I have a hefty PHP script.
So much so that I have had to do
ini_set('memory_limit', '3000M');
set_time_limit (0);
It runs fine on one server, but on another I get: Out of memory (allocated 1653342208) (tried to allocate 71 bytes) in /home/writeabo/public_html/propturk/feedgenerator/simple_html_dom.php on line 848
Both are on the same package from the same host, but different servers.
Above Problem solved new problem below for bounty
Update: The script is so big because it rawls a site and parsers data from 252 pages, including over 60,000 images, which it makes two copies of. I have since broken it down into parts.
I have another problem now though. when I am writing the image from outside site to server like this:
try {
$imgcont = file_get_contents($va); // $va is an img src from an array of thousands of srcs
$h = fopen($writeTo,'w');
fwrite($h,$imgcont);
fclose($h);
} catch(Exception $e) {
$error .= (!isset($error)) ? "error with <img src='" . $va . "' />" : "<br/>And <img src='" . $va . "' />";
}
All of a sudden it goes to a 500 internal server error page and I have to do it again, at which point it works, because files are only copied it they don't already exist. Is there anyway I can receive the 500 response code and send it back it to the url to make it go again? As this is to all be an automated process?
If this is memory related, I would personally use copy() rather than file_get_contents(). It supports the file wrappers the same way, and I don't see any advantage in loading the whole file in memory just to write it back on the filesystem.
Otherwise, your error_log might give you more information as of why the 500 happens.
There are three parties involved here:
Remote - The server(s) that contain the images you're after
Server - The computer that is running your php script
Client - Your home computer if you are running the script from a web browser, or the same computer as the server if you are running it from Cron.
Is the 500 error you are seeing being generated by 'Remote' and seen by 'Server' (i.e. the images are temporarily unavailable);
Or is it being generated by 'Server' and seen by 'Client' (i.e. there is a problem with your script).
If it is being generated by 'Remote', then see Ali's answer for how to retry.
If it is being generated by your script on 'Server', then you need to identify exactly what the error is - the php error logs should give you more information. I can think of two likely causes:
Reaching PHP's time limit. PHP will only spend a certain amount of time working before returning a 500 error. You can set this to a higher value, or regularly re-set the timer with a call to set_time_limit(), but that won't work if your server is configured in safe mode.
Reaching PHP's memory limit. You seem to have encoutered this already, but worth making sure you're script still isn't eating lots of memory. Consider outputing debug data (possibly only if you set $config['debug_mode'] = true or something). I'd suggest:
try {
echo 'Getting '.$va.'...';
$imgcont = file_get_contents($va); // $va is an img src from an array of thousands of srcs
$h = fopen($writeTo,'w');
fwrite($h,$imgcont);
fclose($h);
echo 'saved. Memory usage: '.(memory_get_usage() / (1024 * 1024)).' <br />';
unset($imgcont);
} catch(Exception $e) {
$error .= (!isset($error)) ? "error with <img src='" . $va . "' />" : "<br/>And <img src='" . $va . "' />";
}
I've also added a line to remove the image from memory, incase PHP isn't doing this correctly itself (in theory that line shouldn't be necessary).
You can avoid both problems by making your script process fewer images at a time and calling it regularly - either using Cron on the server (the ideal solution, although not all shared webhosts allow this), or some software on your desktop computer. If you do this, make sure you consider what will happen if there are two copies of the script running at the same time - will they both fetch the same image at the same time?
So it sounds like you're running this process via a web browser. I'm guessing that you may be getting the 500 error from Apache timing out somehow after a certain period of time or the process dies or something funky. I would suggest you do one of the following:
A) Move the image downloading to a background process, you can run the crawl script in the browser which will write the urls of the images to be downloaded to the db or something and another script will fire up via cron and fetch all the images. You could also have this script work in batches of 100 or so at a time to keep memory consumption down
B) Call the script directly from the command line (this is really the preferred method for something like this anyway, and you should still probably separate the image fetching to another script)
C) If the command line is not an option for some reason, have your browser loaded script touch a file, and have a cron that runs every minute and looks for the file to exist. Then it fires up your script, you can have the output written to a file for you to check later or send an email when it's completed
Is there anyway I can receive the 500 response code and send it back it to the url to make it go again? As this is to all be an automated process?
Here's the simple version of how I would do it:
function getImage($va, $writeTo, $retries = 3)
{
while ($retries > 0) {
if ($imgcont = file_get_contents($va)) {
file_put_contents($writeTo, $imgcont);
return true;
}
$retries--;
}
return false;
}
This doesn't create the file unless we successfully get our image file, and will retry three times by default. You will of course need to add any require exception handling, error checking, etc.
I would definitely stop using file_get_contents() and write the files in chunks, like this:
$read = fopen($url, 'rb');
$write = fope($local, 'wb');
$chunk = 8096;
while (!feof($read)) {
fwrite($write, fread($read, $chunk));
}
fclose($fp);
This will be nicer to your server, and should hopefully solve your 500 problems. As for "catching" a 500 error, this is simply not possible. It is an irretrievable error thrown by your script and written to the client by the web server.
I'm with Swish, this is not really the kind of task that PHP is intended for - you'de be much better using some sort of server side scripting.
Is there anyway I can receive the 500 response code and send it back it to the url to make it go again?
Have you considered using another library? Fetching files from an external server seems to me more like a job for curl or ftp than file_get_content &etc. If the error is external, and you're using curl, you can detect the 500 return code and handle it appropriately without crashing. If not, then maybe you should split your program into two files - one of which fetches a single file/image, and the other that uses curl to repeatedly call the first one. Unless the 500 error means that all php execution crashes, you would be able to detect the failure and handle it.
Something like this pseudocode:
file1.php:
foreach(list_of_files as filename){
do {
x = call_curl('file2.php', filename);
}
while(x == 500);
}
file2.php:
filename=$_GET['filename'];
results = use_curl_to_get_page(filename);
echo results;
Thanks for all your input. I had seperated everything by the time I wrote this question, so the crawler, fired the image grabber, etc.
I took on board the solution to split the number of images, and that also helped.
I also added a try, catch round the file read.
This was only being called from the browser during testing, but now that it is all up and running it is going to be a cron job.
Thanks Swish and Benubird for your particularly detailed and educational answers. Unfortunately I had no cooperation with the developers on the backend where the images are coming from (long and complicated story).
Anyway, all good now so thanks. (Swish how do you call a script from the command line, my knowledge of this field is severely lacking?)