Ok, I know that there's been similar questions on this site about this problem, but none of this questions and provided answers isn't exactly what I need.
I'm building flat-file based CMS.
What if, for example:
2, 3, 10..... fwrite in appending mode requestes come to same php file "contact_form_messages_container.php" at the same time?
2, 3, 10..... fwrite in "w" mode requestes come to same php file which holds the simpley nubmer of specific page visits, again at the same time?
I know about flock() function, but it could happen two or more flock() requests comes on the same time... Does anyone knows solution to this problem? Only thing I have on my mind is usleep()-ing the script using while looop for some amount of time, until the target file becomes availibile, but I do not have idea if it works, where and how to perform this?
Does anyone have practical expirience with this issue?
Thanks in advance!
The flock() function is designed to handle multiple concurrent readers and writers for file operations; by default flock() may suspend a process until a compatible lock can be obtained (i.e. shared or exclusive). Once obtained, a lock can later be released to allow other processes to operate on the file; locks are released implicitly when the file is closed or the process ends.
Unless your files are on NFS, I highly doubt you will ever run into a situation whereby two conflicting locks would be given simultaneously.
The following illustrates a basic example of using flock():
// open the file (take care to not use "w" mode)
$f = fopen('file.txt', 'r+');
// obtain an exlusive lock (may suspend the process)
if (flock($f, LOCK_EX)) {
// this process now holds the only exclusive lock
// make changes to the file
// release the lock
flock($f, LOCK_UN);
}
// don't perform any write operation on $f here
fclose($f);
Using the LOCK_NB flag together with LOCK_EX or LOCK_SH will prevent the process from being suspended; if the call returns false a third parameter can be passed to determine whether the process would have been suspended (not supported on Windows).
if (false === flock($f, LOCK_EX | LOCK_NB, $wouldblock)) {
if ($wouldblock) {
// the lock could not be obtained without suspending the process
} else {
// the lock could not be obtained due to an error
}
}
Fork the writing operation and put it into a while loop with a sleep while file is locked. As long as your application doesn't depend on making file system calls chronologically it should work.
This does however open for race conditions if your application depends on writing operations to happen in order.
Related
I have a PHP script which receives and saves invoices as files in Linux. Later, a C++ infinite loop based program reads each and does some processing. I want the latter to read each file safely (only after fully written).
PHP side code simplification:
file_put_contents("sampleDir/invoice.xml", "contents", LOCK_EX)
On the C++ side (with C filesystem API), I must first note that I want to preserve a code which deletes the files in the designated invoices folder which are empty, just as a means to properly deal with the edge case of an empty file being created from other sources (not the PHP script).
Now, here's a C++ side code simplification, too:
FILE* pInvoiceFile = fopen("sampleDir/invoice.xml", "r");
if (pInvoiceFile != NULL)
{
if (flock(pInvoiceFile->_fileno, LOCK_SH) == 0)
{
struct stat fileStat;
fstat(pInvoiceFile->_fileno, &fileStat);
string invoice;
invoice.resize(fileStat.st_size);
if (fread((char*)invoice.data(), 1, fileStat.st_size, pInvoiceFile) < 1)
{
remove("sampleDir/invoice.xml"); // Edge case resolution
}
flock(pInvoiceFile->_fileno, LOCK_UN);
}
}
fclose(pInvoiceFile);
As you can see, the summarizing key concept is the cooperation of LOCK_EX and LOCK_SH flags.
My problem is that, while this integration has been working fine, yesterday I noticed the edge case executed for an invoice which should not be empty, and thus it got deleted by the C++ program.
PHP manual on file_put_contents mentions the following for the LOCK_EX flag:
Acquire an exclusive lock on the file while proceeding to the writing. In other words, a flock() call happens between the fopen() call and the fwrite() call. This is not identical to an fopen() call with mode "x".
Could the problem be caused as a race condition by the LOCK_EX not being established right before file_put_contents calls fopen? If so, what could be done to solve this while keeping the edge case removal code?
Otherwise, may I be doing anything wrong overall?
Your code is assuming that the file_put_contents() operation is atomic, and that using FLOCK_EX and FLOCK_SH is enough to ensure no race conditions between the two programs happen. This is not the case.
As you can see from the PHP doc, the FLOCK_EX is applied after opening the file. This is important, because it leaves a short window of time for the C++ program to successfully open the file and lock it with FLOCK_SH. At that point the file was already truncated by the fopen() done by PHP, and it's empty.
What's most likely happening is:
PHP code opens the file for writing, truncating it and effectively wiping out its content.
C++ code opens the file for reading.
C++ code requests the shared lock on the file: the lock is granted.
PHP code requests the exclusive lock on the file: the call blocks, waiting for the lock to be available.
C++ code reads the file's contents: nothing, the file is empty.
C++ code deletes the file.
C++ code releases the shared lock.
PHP code acquires the exclusive lock.
PHP code writes to the file: the data does not reach the disk because the inode associated with the open file descriptor does not exist anymore.
You are effectively left with no file and the data is lost.
The problem with your code is that the operations you are doing on the file from two different programs are not atomic, and the way you are acquiring the locks does not help in ensuring that those don't overlap.
The only sane way of guaranteeing the atomicity of such an operation on a POSIX compliant system, without even worrying about file locking, is to take advantage of the atomicity of rename(2):
If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing.
If newpath exists but the operation fails for some reason, rename() guarantees to leave an instance of newpath in place.
The equivalent rename() PHP function is what you should use in this case. It's the simplest way to guarantee atomic updates to a file.
What I would suggest is the following:
PHP code:
$tmpfname = tempnam("/tmp", "myprefix"); // Create a temporary file.
file_put_contents($tmpfname, "contents"); // Write to the temporary file.
rename($tmpfname, "sampleDir/invoice.xml"); // Atomically replace the contents of invoice.xml by renaming the file.
// TODO: check for errors in all the above calls, most importantly tempnam().
C++ code:
FILE* pInvoiceFile = fopen("sampleDir/invoice.xml", "r");
if (pInvoiceFile != NULL)
{
struct stat fileStat;
fstat(fileno(pInvoiceFile), &fileStat);
string invoice;
invoice.resize(fileStat.st_size);
size_t n = fread(&invoice[0], 1, fileStat.st_size, pInvoiceFile);
fclose(pInvoiceFile);
if (n == 0)
remove("sampleDir/invoice.xml");
}
This way, the C++ program will always either see the old version of the file (if fopen() happens before PHP's rename()) or the new version of the file (if fopen() happens after), but it will never see an inconsistent version of the file.
I wanted to wait for all processes reading a certain file in PHP by obtaining an exclusive lock on that file, and after that delete (unlink) the file. This concerns files like profile pictures which a user can delete or change. The name of the file will be something like the user ID.
My code:
//Obtain lock
$file = fopen("path/to/file", "r"); //(I'm not sure which mode to use here btw)
flock($file, LOCK_EX);
//Delete file
unlink("path/to/file");
Line 3 waits for all locks to be released, which is good, but the unlink function throws an error: Warning: unlink(path/to/file): Resource temporarily unavailable in path/to/script on line xx
To prevent this I could release the lock before calling unlink, but this means another process could lock on the file again, which would cause the same error.
My questions are:
Is it possible to delete a file in PHP without releasing the lock? That is, without the risk of other processes trying to use the file at the same time.
If not:
Is this possible in Windows at all? How about Unix?
Should I involve my database for this matter and lock on rows in the database instead, or is there a better way?
Another option I can see is repeating this piece of code, including a release of the lock before calling unlink, until unlink succeeds, but this seems a bit messy, right?
Hey I'm struggling with this too, 2 years later. Kind of seems dumb you can't acquire an exclusive lock on a file when trying to rename or unlink it, or at least the documentation isn't there for doing this.
One solution is open the file for writing, acquire an exclusive lock, clear contents of the file using ftruncate, close it, and then unlink it. When you're reading from the file, you can check the size to make sure the file has contents.
When deleting (untested code):
$fh = fopen('yourfile.txt', 'c'); // 'w' mode truncates file, you don't want to do that yet!
flock($fh, LOCK_EX); // blocking, but you can use LOCK_EX | LOCK_NB for nonblocking and a loop + sleep(1) for a timeout
ftruncate($fh, 0); // truncate file to 0 length
fclose($fh);
unlink('yourfile.txt');
When reading (untested code):
if (!file_exists('yourfile.txt') || filesize('yourfile.txt') <= 0) {
print 'nah.jpg, must be dELeTeD :O';
}
Why can I not read a file locked with LOCK_EX? I am still able to write to it.
I wanted to know, what happens, if one process locks a file (with LOCK_SH or LOCK_EX) and another process tries to read this file or write to it, but ignores the lock at all. So I made a little script that has 3 functionalities:
Locking: Opens the target file, writes to it, locks the file (with specified lock), writes to it again, sleeps 10 seconds, unlocks it and closes it.
Reading: Opens the target file, reads from it and closes it.
Writing: Opens the target file, writes to it and closes it.
I tested it by having two consoles side by side and doing the folloing:
FIRST CONSOLE | SECOND CONSOLE
-----------------------------+-----------------------
php test lock LOCK_SH | php test read
php test lock LOCK_SH | php test write
php test lock LOCK_EX | php test read
php test lock LOCK_EX | php test write
LOCK_SH seems to have no effect at all, because the first process as well as the second process can read and write to the file. If the file is being locked with LOCK_EX by the first process, both processes can still write to it, but only the first process can read. Is there any reasoning behind this?
Here is my little test program (tested on Windows 7 Home Premium 64-bit):
<?php
// USAGE: php test [lock | read | write] [LOCK_SH | LOCK_EX]
// The first argument specifies whether
// this script should lock the file, read
// from it or write to it.
// The second argument is only used in lock-mode
// and specifies whether LOCK_SH or LOCK_EX
// should be used to lock the file
// Reads $file and logs information.
function r ($file) {
echo "Reading file\n";
if (($buffer = #fread($file, 64)) !== false)
echo "Read ", strlen($buffer), " bytes: ", $buffer, "\n";
else
echo "Could not read file\n";
}
// Sets the cursor to 0.
function resetCursor ($file) {
echo "Resetting cursor\n", #fseek($file, 0, SEEK_SET) === 0 ? "Reset cursor" : "Could not reset cursor", "\n";
}
// Writes $str to $file and logs information.
function w ($file, $str) {
echo "Writing \"", $str, "\"\n";
if (($bytes = #fwrite($file, $str)) !== false)
echo "Wrote ", $bytes, " bytes\n";
else
echo "Could not write to file\n";
}
// "ENTRYPOINT"
if (($file = #fopen("check", "a+")) !== false) {
echo "Opened file\n";
switch ($argv[1]) {
case "lock":
w($file, "1");
echo "Locking file\n";
if (#flock($file, constant($argv[2]))) {
echo "Locked file\n";
w($file, "2");
resetCursor($file);
r($file);
echo "Sleeping 10 seconds\n";
sleep(10);
echo "Woke up\n";
echo "Unlocking file\n", #flock($file, LOCK_UN) ? "Unlocked file" : "Could not unlock file", "\n";
} else {
echo "Could not lock file\n";
}
break;
case "read":
resetCursor($file);
r($file);
break;
case "write":
w($file, "3");
break;
}
echo "Closing file\n", #fclose($file) ? "Closed file" : "Could not close file", "\n";
} else {
echo "Could not open file\n";
}
?>
That's a very good question, but also a complex one because it depends on a lot of conditions.
We have to start with another pair of locking types - advisory and mandatory:
Advisory locking simply give you "status flags" by which you know if a resource is locked or not.
Mandatory locking enforces the locks, regardless of whether you're checking these "status flags".
... and that should answer your question, but I'll continue in order to explain your particular case.
What you seem to be experiencing is the behavior of advisory locks - there's nothing preventing you from reading or writing to a file, no matter if there is a lock for it or if you even checked for one.
However, you will find a note in the PHP manual for flock(), saying the following:
flock() uses mandatory locking instead of advisory locking on Windows. Mandatory locking is also supported on Linux and System V based operating systems via the usual mechanism supported by the fcntl() system call: that is, if the file in question has the setgid permission bit set and the group execution bit cleared. On Linux, the file system will also need to be mounted with the mand option for this to work.
So, if PHP uses mandatory locking on Windows and you've tested this on Windows, either the manual is wrong/outdated/inaccurate (I'm too lazy to check for that right now) or you have to read this big red warning on the same page:
On some operating systems flock() is implemented at the process level. When using a multithreaded server API like ISAPI you may not be able to rely on flock() to protect files against other PHP scripts running in parallel threads of the same server instance!
flock() is not supported on antiquated filesystems like FAT and its derivates and will therefore always return FALSE under this environments (this is especially true for Windows 98 users).
I don't believe it is even possible that your php-cli executable somehow spawns threads for itself, so there's the option that you're using a filesystem that simply doesn't support locking.
My guess is that the manual isn't entirely accurate and you do in fact get advisory locks on Windows, because you are also experiencing a different behavior between LOCK_EX (exclusive lock) and LOCK_SH (shared lock) - it doesn't make sense for them to differ if your filesystem just ignores the locks.
And that brings us to the difference between exclusive and shared locks or LOCK_EX and LOCK_SH respectively. The logic behind both is based around writing, but there's a slight difference ...
Exclusive locks are (generally) used when you want to write to a file, because only one process at a time may hold an exclusive lock over the same resource. That gives you safety, because no other process would read from that resource while the lock-holder writes to it.
Shared locks are used to ensure that a resource is not being written to while you read from it. Since no process is writing to the resource, it is not being modified and is therefore safe to read from for multiple processes at the same time.
I want to have a PHP file that is used as a counter. It will a) echo the current value of a txt file, and b) increment that file using an exclusive lock so no other scripts can read or write to it while it's being used.
User A will write and increment this number, while User B requests to read the file. Is it possible that User A can lock this file so no one can read or write to it until User A's write is finished?
I've used flock in the past, but I'm not sure how to get the file to wait until it is available, rather than quitting if it's already been locked
My goal is:
LOCK counter.txt; write to counter.txt;
while at the same time
Read counter.txt; realize it's locked so wait until that lock is finished.
//
$fp = fopen("counter.txt", 'w+');
if(flock($fp, LOCK_EX)) {
fwrite($fp, $counter + 1);
flock($fp, LOCK_UN);
} else {
// try again??
}
fclose($fp);
From documentation: By default, this function will block until the requested lock is acquired
So simply use flock in your reader (LOCK_SH) and writer (LOCK_EX), and it is going to work.
However I highly discourage use of blocking flock without timeout as this means that if something goes wrong then your program is going to hang forever. To avoid this use non-blocking request like this (again, it is in doc):
/* Activate the LOCK_NB option on an LOCK_EX operation */
if(!flock($fp, LOCK_EX | LOCK_NB)) {
echo 'Unable to obtain lock';
}
And wrap it in a for loop, with sleep and break after failed n-tries (or total wait time).
EDIT: You can also look for some examples of usage here. This class is a part of ninja-mutex library in which you may be interested too.
Will fopen() fail if a file exists, but is currently locked with LOCK_EX?
Or do I have to open it, and then try and set a lock, in order to determine if one already exists?
I've also read that flock() will;
pause [the script] untill you get the lock for indefinite amount of time or till your script times out
http://www.php.net/manual/en/function.flock.php#95257
If so, is it true this 'pause' can be by-passed with;
if (!flock($f, LOCK_SH | LOCK_NB)) {
// file locked, do something else
}
flock() doesn't actually prevent you from reading/writing to a file, it only allows you to "communicate" the ideas of locking to other scripts. You can detect if there is a lock on a file using the snippet you posted.