flock call within function always 'succeeds', ignoring previous lock - php

To prevent multiple instances of a PHP-based daemon I wrote from ever running simultaneously, I wrote a simple function to acquire a lock with flock when the process starts, and called it at the start of my daemon. A simplified version of the code looks like this:
#!/usr/bin/php
<?php
function acquire_lock () {
$file_handle = fopen('mylock.lock', 'w');
$got_lock_successfully = flock($file_handle, LOCK_EX);
if (!$got_lock_successfully) {
throw new Exception("Unexpected failure to acquire lock.");
}
}
acquire_lock(); // Block until all other instances of the script are done...
// ... then do some stuff, for example:
for ($i=1; $i<=10; $i++) {
echo "Sleeping... $i\n";
sleep(1);
}
?>
When I execute the script above multiple times in parallel, the behaviour I expect to see - since the lock is never explicitly released throughout the duration of the script - is that the second instance of the script will wait until the first has completed before it proceeds past the acquire_lock() call. In other words, if I run this particular script in two parallel terminals, I expect to see one terminal count to 10 while the other waits, and then see the other count to 10.
This is not what happens. Instead, I see both scripts happily executing in parallel - the second script does not block and wait for the lock to be available.
As you can see, I'm checking the return value from flock and it is true, indicating that the (exclusive) lock has been acquired successfully. Yet this evidently isn't preventing another process from acquiring another 'exclusive' lock on the same file.
Why, and how can I fix this?

Simply store the file pointer resource returned from fopen in a global variable. In the example given in the question, $file_handle is automatically destroyed upon going out of scope when acquire_lock() returns, and this releases the lock taken out by flock.
For example, here is a modified version of the script from the question which exhibits the desired behaviour (note that the only change is storing the file handle returned by fopen in a global):
#!/usr/bin/php
<?php
function acquire_lock () {
global $lock_handle;
$lock_handle = fopen('mylock.lock', 'w');
$got_lock_successfully = flock($lock_handle, LOCK_EX);
if (!$got_lock_successfully) {
throw new Exception("Unexpected failure to acquire lock.");
}
}
acquire_lock(); // Block until all other instances of the script are done...
// ... then do some stuff, for example:
for ($i=1; $i<=10; $i++) {
echo "Sleeping... $i\n";
sleep(1);
}
?>
Note that this seems to be a bug in PHP. The changelog from the flock documentation states that in version 5.3.2:
The automatic unlocking when the file's resource handle is closed was removed. Unlocking now always has to be done manually.
but at least for PHP 5.5, this is false; flock locks are released both by explicit calls to fclose and by the resource handle going out of scope.
I reported this as a bug in November 2014 and may update this question and answer pair if it is ever resolved. In case I get eaten by piranhas before then, you can check the bug report yourself to see if this behaviour has been fixed: https://bugs.php.net/bug.php?id=68509

Related

What is the correct way to perform lock file (for having critical section purpose) cleanup

By referring to flock(): removing locked file without race condition? and Will flock'ed file be unlocked when the process die unexpectedly? ,
I produce the following code. My intention is to allow only single thread / single process to run the critical section code, within any given time.
<?php
// Exclusive locking based on function parameters.
$lockFileName = '/tmp/cheok.lock';
// Create if doesn't exist.
$lockFile = fopen($lockFileName, "w+");
if (!flock($lockFile, LOCK_EX)) {
throw new \RumtimeException("Fail to perform flock on $lockFileName");
}
echo "start critical section...\n";
sleep(10);
echo "end critical section.\n";
// Warning: unlink(/tmp/cheok.lock): No such file or directory in
unlink($lockFileName);
flock($lockFile, LOCK_UN);
I will always get the warning
Warning: unlink(/tmp/cheok.lock): No such file or directory in
when the 2nd waiting process continues his execution, the 1st process already remove the physical disk file. The 2nd process tries to unlink file, which is already deleted by the 1st process.
And, what if, there is 3rd process join in, and tries the perform fopen while 2nd process is trying to perform unlink?
In short, what is the correct way to perform lock file cleanup?
The question is tagged with multithreading, so I'm going to answer the question in the context of multi-threading with pthreads.
First, a little bit about file locking. The PHP manual fails to fully explain what an advisory lock is.
Taken from the man page for flock
flock() places advisory locks only; given suitable permissions on a file, a process is free to ignore the use of flock() and perform I/O on the file.
In a ecosystem filled with people who simultaneously don't know much about permissions, and use cron jobs to execute long running scripts, this can be the cause of much pain ... although nobody really knows it.
In the context of multithreading (with pthreads), you want to stay well away from file locking, pthreads provides a much superior API for implementing mutual exclusion.
Follows is example code that creates a number of Threads, and implements mutual exclusion:
<?php
class Test extends Thread {
public function __construct(Threaded $monitor) {
$this->monitor = $monitor;
}
public function run () {
$this->monitor->synchronized(function(){
for ($i = 0; $i < 1000; $i++)
printf("%s #%lu: %d\n",
__CLASS__, $this->getThreadId(), $i);
});
}
private $monitor;
}
$threads = [];
$monitor = new Threaded();
for ($i = 0; $i < 8; $i++) {
$threads[$i] = new Test($monitor);
$threads[$i]->start();
}
foreach ($threads as $thread)
$thread->join();
Because all the Threads share the same $monitor, you can use the synchronization built into Threaded objects to implement reliable mutual exclusion, causing the output to resemble:
Test #140561163798272: 0
<-- snip for brevity -->
Test #140561163798272: 999
Test #140561151424256: 0
<-- snip for brevity -->
Test #140561151424256: 999
Test #140561138841344: 0
<-- snip for brevity -->
Test #140561138841344: 999
Test #140561059149568: 0
<-- snip for brevity -->
Test #140561059149568: 999
Test #140561050756864: 0
<-- snip for brevity -->
Test #140561050756864: 999
Test #140561042364160: 0
<-- snip for brevity -->
Test #140561042364160: 999
Test #140561033971456: 0
<-- snip for brevity -->
Test #140561033971456: 999
Test #140561025578752: 0
<-- snip for brevity -->
Test #140561025578752: 999
We can see that each Thread is mutually excluded from executing the code in the Closure passed to Threaded::synchronized.
You can unlock a file by using the excellent Process Explorer task manager. We’ve covered Process Explorer in detail before, so here we’ll just dive right into how to unlock a file. You won’t need to install it first—it’s a portable app—but you will need to run it with administrative privileges. You actually can do this from within Process Explorer by clicking the “File” menu and selecting “Show Details for All Processes.”

PHP locking not working. Can't figure out why

Hi guys I'm following up from my question Acquire_lock() not working. Bot still sending requests quickly. PHP + AJAX which I haven't been able to get an answer for.
I've simplified everything and have three files
abc.txt
file1.php
file2.php
All in the same directory and the contents of both php files are the same:
<?php
$x = fopen("/var/www/abc.txt", "w");
if (flock($x, LOCK_EX|LOCK_NB)) {
print "No problems, I got the lock, now I'm going to sit on it.";
while (true)
sleep(5);
} else {
print "Didn't quite get the lock. Quitting now. Good night.";
}
fclose($x);
?>
Yet when I load either of them, I get the second print message: "Didn't quite get the lock. Quitting now. Good night.".
Anyone have any idea, either to this question or the former as to what's going on? Literally at my wits end with this.
Thank you as always.
If you want the php script to not exit then you would want a blocking lock.
flock documentation says you can do it by specifying third parameter. Also removing LOCK_NB might help.
<?php
$x = fopen("/var/www/abc.txt", "w");
if (flock($x, LOCK_EX, 1)) {
print "No problems, I got the lock, now I'm going to sit on it.";
// wait for 5 seconds
sleep(5);
// Release the lock now so that next script is executed
flock($x , LOCK_UN);
} else {
print "Didn't quite get the lock. Quitting now. Good night.";
}
fclose($x);
?>
Here is what's going on in your script:
1) You code is going into an infinite loop while(true) and it is never reaching the fclose() statement at the end.
2) I tested both files on my local server: File1.php kept looping while the second file2.php gave the file is locked messaged immediately (which means the first file locked correctly). I tried refreshing both files afterwards and they both failed the lock test.
If you are using PHP> 5.3.2 then you have to unlock the file manually:
The automatic unlocking when the file's resource handle is closed was
removed. Unlocking now always has to be done manually.
Source
If you are using an older version of PHP, then the script will unlock the file once the script is done executing, since you are going into an infinite loop, the script is never done and therefore the file is never unlocked.
Also, even if you stop the script from running in your browser window, the process php-cgi.exe related to that script remains running and has to be terminated manually from the task manager (I verified that myself)
Solution:
1)
To fix this issue and make the script wait for the lock, you need to make sure first that the script actually stops gracefully by removing the infinite loop:
Here is a script that will lock the file for 30 seconds (loop removed):
<?php
$x = fopen("/var/www/abc.txt", "w");
if (flock($x, LOCK_EX|LOCK_NB)) {
print "No problems, I got the lock, now I'm going to sit on it.";
sleep(30);
fclose($x); // it is good practice to always close even your PHP <5.3.2
}
else {
print "Didn't quite get the lock. Quitting now. Good night.";
}
?>
2) If you are on a linux machine, you can use the LOCK_NB flag to determine whether the file is locked or not. The usage of LOCK_NB should be like this:
while ( ! flock($f, LOCK_NB) )
{
sleep(1);
}
this will force the script to check every second for the lock and wait for the other script to finish.
3) use flock($fp, LOCK_UN) to explicitly remove the lock when you are done instead of fclose();
In summary, this is how your code should look like:
<?php
$x = fopen("/var/www/abc.txt", "w");
while(!flock($x,LOCK_NB)
sleep(1);
if (flock($x, LOCK_EX,true)) {
print "No problems, I got the lock, now I'm going to sit on it.";
sleep(30);
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
print "Didn't quite get the lock. Quitting now. Good night.";
}
fclose($x);
?>

how to synchronise multiple processes in PHP to simulate wait()/notifyAll()

I'm trying to test a race condition in PHP. I'd like to have N PHP processes get ready to do something, then block. When I say "go", they should all execute the action at the same time. Hopefully this will demonstrate the race.
In Java, I would use Object.wait() and Object.notifyAll(). What can I use in PHP?
(Either Windows or linux native answers are acceptable)
Create a file "wait.txt"
Start N processes, each with the code shown below
Delete the "wait.txt" file.
...
<?php
while (file_exists('wait.txt')) {}
runRaceTest();
Usually with PHP file lock approach is used. One create a RUN_LOCK or similar file and asks for file_exists("RUN_LOCK"). This system is also used to secure potential endless loops in recursive threads.
I decided to require the file for the execution. Other approach may be, that existence of the file invokes the blocking algorithm. That depends on your situation. Always the safer situation should be the easier to achieve.
Wait code:
/*prepare the program*/
/* ... */
/*Block until its time to go*/
define("LOCK_FILE", "RUN_UNLOCK"); //I'd define this in some config.php
while(!file_exists(LOCK_FILE)) {
usleep(1); //No sleep will eat lots of CPU
}
/*Execute the main code*/
/* ... */
/*Delete the "run" file, so that no further executions should be allowed*/
usleep(1); //Just for sure - we want other processes to start execution phase too
if(file_exists(LOCK_FILE))
unlink(LOCK_FILE);
I guess it would be nice to have a blocking function for that, like this one:
function wait_for_file($filename, $timeout = -1) {
if($timeout>=0) {
$start = microtime(true)*1000; //Remember the start time
}
while(!file_exists($filename)) { //Check the file existence
if($timeout>=0) { //Only calculate when timeout is set
if((microtime(true)*1000-$start)>$timeout) //Compare current time with start time (current always will be > than start)
return false; //Return failure
}
usleep(1); //Save some CPU
}
return true; //Return success
}
It implements timeout. You don't need them but maybe someone else will.
Usage:
header("Content-Type: text/plain; charset=utf-8");
ob_implicit_flush(true);while (#ob_end_clean()); //Flush buffers so the output will be live stream
define("LOCK_FILE","RUN_FOREST_RUN"); //Define lock file name again
echo "Starting the blocking algorithm. Waiting for file: ".LOCK_FILE."\n";
if(wait_for_file(LOCK_FILE, 10000)) { //Wait for 10 secconds
echo "File found and deleted!\n";
if(file_exists(LOCK_FILE)) //May have been deleted by other proceses
unlink(LOCK_FILE);
}
else {
echo "Wait failed!\n";
}
This will output:
Starting the blocking algorithm. Waiting for file: RUN_FOREST_RUN
Wait failed!
~or~
Starting the blocking algorithm. Waiting for file: RUN_FOREST_RUN
File found and deleted!
PHP doesn't have multithreading. And its not planned to be implemented either.
You can try hacks with sockets though or 0MQ to communicate between multiple processes
See Why does PHP not support multithreading?
Php multithread

Can I close a file by unsetting the handle?

I'm a little puzzled if I can spare the fclose command by just unsetting the variable that carries the handle?
$handle = fopen($file);
...
fclose($handle);
... // script goes on for a long
Compared with:
$handle = fopen($file);
...
unset($handle);
... // script goes on for a long
Insights anyone?
Thanks to the reference-counting system introduced with PHP 4's Zend Engine, a resource with no more references to it is detected automatically, and it is freed by the garbage collector.
Consider the implications of this. It's safe to assume that all traces of the variable are gone after the garbage collection. In other words, at the end of PHP's execution, if PHP is not still tracking the reference, how would it close it? Thus, it seems fairly logical that it would close it when the garbage collector eats it.
This is a bad logical argument though because it assumes that garbage collections happens either immediately or shortly after unset and that PHP does not keep hidden references to variables that no longer exist in user land.
A more compelling case though could be a potential behavioural flaw if PHP did not close file handles when they go out of scope. Consider a daemon of some kind that opens lots of files. Now consider if fclose is never called. Instead, variables are allowed to fall out of scope or unset is explicitly called on them.
If these file handles were not closed, this long running daemon would run out of file handles.
Potentially behavior specific test script:
<?php
$db = mysql_connect(...);
if ($db) {
echo "Connected\n";
sleep(5); //netstat during this just for paranoia
unset($db);
echo "Unset\n";
sleep(5); //netstat during this and the connection is closed
}
On both Windows 7 and Debian 6, the connection has been closed after the unset.
Obviously though, this only proves that on my specific machines with my specific PHP version will this work. Has no meaning on file handles or the like :).
Am searching the PHP source now for hard proof
PHP docs hint that all resources with no remaining references are "freed", I assume for file handles this would include closing the file.
Simple test case:
$f = fopen("test.php", "r");
if (!flock($f, LOCK_EX)) {
print("still locked\n");
exit;
}
unset($f);
sleep(5);
print("goodbye\n");
(I've saved this as test.php, so it is locking itself; might need to change the filename in the fopen() to some existing file otherwise)
Run the script twice within 5 seconds; if you get "still locked", then apparently unsetting the handle did not release the lock. In my test, I did not get "still locked", so apparently unsetting the handle at least releases the lock, though it would seem silly to release locks upon garbage collection, but not close the file.
unset($handle) will destroy the $handle variable, but it won't close the file being pointed by $handle. You still need to call fclose() to close the file.
some research:
fclose makes $handle to be resource(5) of type (Unknown)
while
unset makes it NULL.
and after fclose php consumes 88 bytes of memory more.
so: they are different =)

How do I restore this script after a hardware failure?

I know this is a bit generic, but I'm sure you'll understand my explanation. Here is the situation:
The following code is executed every 10 minutes. Variable "var_x" is always read/written to an external text file when its refereed to.
if ( var_x != 1 )
{
var_x = 1;
//
// here is where the main body of the script is.
// it can take hours to completely execute.
//
var_x = 0;
}
else
{
// exit script as it's already running.
}
The problem is: if I simulate a hardware failure (do a hard reset when the script is executing) then the main script logic will never execute again because "var_x" will always be "1". (I already have logic to work out the restore point).
Thanks.
You should lock and unlock files with flock:
$fp = fopen($your_file);
if (flock($fp, LOCK_EX)) { )
{
//
// here is where the main body of the script is.
// it can take hours to completely execute.
//
flock($fp, LOCK_UN);
}
else
{
// exit script as it's already running.
}
Edit:
As flock seems not to work correctly on Windows machines, you have to resort to other solutions. From the top of my head an idea for a possible solution:
Instead of writing 1 to var_x, write the process ID retrieved via getmypid. When a new instance of the script reads the file, it should then lookup for a running process with this ID, and if the process is a PHP script. Of course, this can still go wrong, as there is the possibility of another PHP script obtaining the same PID after a hardware failure, so the solution is far from optimal.
Don't you think this would be better solved using file locks? (When the reset occurs file locks are reset as well)
http://php.net/flock
It sounds like you're doing some kind of manual semaphore for process management.
Rather than writing to a file, perhaps you should use an environment variable instead. That way, in the event of failure, your script will not have a closed semaphore when you restore.

Categories