Why can I not read a file locked with LOCK_EX? I am still able to write to it.
I wanted to know, what happens, if one process locks a file (with LOCK_SH or LOCK_EX) and another process tries to read this file or write to it, but ignores the lock at all. So I made a little script that has 3 functionalities:
Locking: Opens the target file, writes to it, locks the file (with specified lock), writes to it again, sleeps 10 seconds, unlocks it and closes it.
Reading: Opens the target file, reads from it and closes it.
Writing: Opens the target file, writes to it and closes it.
I tested it by having two consoles side by side and doing the folloing:
FIRST CONSOLE | SECOND CONSOLE
-----------------------------+-----------------------
php test lock LOCK_SH | php test read
php test lock LOCK_SH | php test write
php test lock LOCK_EX | php test read
php test lock LOCK_EX | php test write
LOCK_SH seems to have no effect at all, because the first process as well as the second process can read and write to the file. If the file is being locked with LOCK_EX by the first process, both processes can still write to it, but only the first process can read. Is there any reasoning behind this?
Here is my little test program (tested on Windows 7 Home Premium 64-bit):
<?php
// USAGE: php test [lock | read | write] [LOCK_SH | LOCK_EX]
// The first argument specifies whether
// this script should lock the file, read
// from it or write to it.
// The second argument is only used in lock-mode
// and specifies whether LOCK_SH or LOCK_EX
// should be used to lock the file
// Reads $file and logs information.
function r ($file) {
echo "Reading file\n";
if (($buffer = #fread($file, 64)) !== false)
echo "Read ", strlen($buffer), " bytes: ", $buffer, "\n";
else
echo "Could not read file\n";
}
// Sets the cursor to 0.
function resetCursor ($file) {
echo "Resetting cursor\n", #fseek($file, 0, SEEK_SET) === 0 ? "Reset cursor" : "Could not reset cursor", "\n";
}
// Writes $str to $file and logs information.
function w ($file, $str) {
echo "Writing \"", $str, "\"\n";
if (($bytes = #fwrite($file, $str)) !== false)
echo "Wrote ", $bytes, " bytes\n";
else
echo "Could not write to file\n";
}
// "ENTRYPOINT"
if (($file = #fopen("check", "a+")) !== false) {
echo "Opened file\n";
switch ($argv[1]) {
case "lock":
w($file, "1");
echo "Locking file\n";
if (#flock($file, constant($argv[2]))) {
echo "Locked file\n";
w($file, "2");
resetCursor($file);
r($file);
echo "Sleeping 10 seconds\n";
sleep(10);
echo "Woke up\n";
echo "Unlocking file\n", #flock($file, LOCK_UN) ? "Unlocked file" : "Could not unlock file", "\n";
} else {
echo "Could not lock file\n";
}
break;
case "read":
resetCursor($file);
r($file);
break;
case "write":
w($file, "3");
break;
}
echo "Closing file\n", #fclose($file) ? "Closed file" : "Could not close file", "\n";
} else {
echo "Could not open file\n";
}
?>
That's a very good question, but also a complex one because it depends on a lot of conditions.
We have to start with another pair of locking types - advisory and mandatory:
Advisory locking simply give you "status flags" by which you know if a resource is locked or not.
Mandatory locking enforces the locks, regardless of whether you're checking these "status flags".
... and that should answer your question, but I'll continue in order to explain your particular case.
What you seem to be experiencing is the behavior of advisory locks - there's nothing preventing you from reading or writing to a file, no matter if there is a lock for it or if you even checked for one.
However, you will find a note in the PHP manual for flock(), saying the following:
flock() uses mandatory locking instead of advisory locking on Windows. Mandatory locking is also supported on Linux and System V based operating systems via the usual mechanism supported by the fcntl() system call: that is, if the file in question has the setgid permission bit set and the group execution bit cleared. On Linux, the file system will also need to be mounted with the mand option for this to work.
So, if PHP uses mandatory locking on Windows and you've tested this on Windows, either the manual is wrong/outdated/inaccurate (I'm too lazy to check for that right now) or you have to read this big red warning on the same page:
On some operating systems flock() is implemented at the process level. When using a multithreaded server API like ISAPI you may not be able to rely on flock() to protect files against other PHP scripts running in parallel threads of the same server instance!
flock() is not supported on antiquated filesystems like FAT and its derivates and will therefore always return FALSE under this environments (this is especially true for Windows 98 users).
I don't believe it is even possible that your php-cli executable somehow spawns threads for itself, so there's the option that you're using a filesystem that simply doesn't support locking.
My guess is that the manual isn't entirely accurate and you do in fact get advisory locks on Windows, because you are also experiencing a different behavior between LOCK_EX (exclusive lock) and LOCK_SH (shared lock) - it doesn't make sense for them to differ if your filesystem just ignores the locks.
And that brings us to the difference between exclusive and shared locks or LOCK_EX and LOCK_SH respectively. The logic behind both is based around writing, but there's a slight difference ...
Exclusive locks are (generally) used when you want to write to a file, because only one process at a time may hold an exclusive lock over the same resource. That gives you safety, because no other process would read from that resource while the lock-holder writes to it.
Shared locks are used to ensure that a resource is not being written to while you read from it. Since no process is writing to the resource, it is not being modified and is therefore safe to read from for multiple processes at the same time.
Related
I am writing a wee wrapper (in PHP 7.0.33) around a complex binary which takes its input from a named file. As this will be processing secrets, I don't want to commit the data to the filesystem, hence using a FIFO rather than conventional file. The binary will happily read its data from a FIFO (tested using 2 shell sessions - in one I created the fifo and started the binary, in the second I catted a file into the fifo).
However in PHP the call to fopen() blocks, regardless if I specify w, a or c
if (posix_mkfifo("myfifo", 0600)) {
$writer=fopen("myfifo", 'w'); // this blocks
`binary myfifo`;
fputs($writer, $mydata);
}
While I would expect writes to block if nothing is reading the data, I did not expect fopen() to block.
It does appear to work (execution continues, and the binary is started) with "w+" however the binary fails with
QIODevice::read (QFile, "filename"): device not open
To investigate further, I wrote a simple replacement for the binary. Again this works when I cat a file into the FIFO:
$in='';
$fh=fopen($argv[1],'r');
if (is_resource($fh)) {
print "File opened\n";
while (!feof($fh)) {
$in.=fgets($fh);
}
} else {
print "failed to open file\n";
}
file_put_contents("output", $in);
but when I write to the FIFO from the PHP code....
fopen(import): failed to open stream: No such file or directory in ...
By default, opening a FIFO will block until there is at least one reader and writer. The rationale for this is that the kernel has no place to stash the pipe data if no process is there to consume it. man page for fifo:
... the FIFO special file has no contents on the file system, the file system entry merely serves as a reference point so that processes can access the pipe using a name in the file system.
The kernel maintains exactly one pipe object for each FIFO special file that is opened by at least one process. The FIFO must be opened on both ends (reading and writing) before data can be passed. Normally, opening the FIFO blocks until the other end is opened also.
You can bypass this behaviour though. One way is like you've done - by opening the read and write end yourself. The other is to set O_NONBLOCK flag when opening the file (you can set it back to block afterwards). AFAIK you can't do that with fopen. Example with dio library:
<?php
echo "Opening\n";
$writer = dio_open("myfifo", O_CREAT | O_WRONLY | O_NONBLOCK) or die("Could not create FIFO\n");
echo "Open. Writing\n";
dio_write($writer, "DATA");
echo "Done\n";
The caveat with doing this is, if there is no reader, the process above will write the data, then exit immediately and then the data is lost forever.
For the benefit of anyone finding this question in Google and seeking a more detailled solution than comment on semisecure's answer:
if (pcntl_fork()) {
`binary myfifo`;
} else {
$fh=fopen('myfifo', 'w');
fputs($fh, $data);
fclose($fh);
}
(but you might also want to add a SIGALRM on the writer in case "binary" does not execute / flush the pipe).
I have a PHP script which receives and saves invoices as files in Linux. Later, a C++ infinite loop based program reads each and does some processing. I want the latter to read each file safely (only after fully written).
PHP side code simplification:
file_put_contents("sampleDir/invoice.xml", "contents", LOCK_EX)
On the C++ side (with C filesystem API), I must first note that I want to preserve a code which deletes the files in the designated invoices folder which are empty, just as a means to properly deal with the edge case of an empty file being created from other sources (not the PHP script).
Now, here's a C++ side code simplification, too:
FILE* pInvoiceFile = fopen("sampleDir/invoice.xml", "r");
if (pInvoiceFile != NULL)
{
if (flock(pInvoiceFile->_fileno, LOCK_SH) == 0)
{
struct stat fileStat;
fstat(pInvoiceFile->_fileno, &fileStat);
string invoice;
invoice.resize(fileStat.st_size);
if (fread((char*)invoice.data(), 1, fileStat.st_size, pInvoiceFile) < 1)
{
remove("sampleDir/invoice.xml"); // Edge case resolution
}
flock(pInvoiceFile->_fileno, LOCK_UN);
}
}
fclose(pInvoiceFile);
As you can see, the summarizing key concept is the cooperation of LOCK_EX and LOCK_SH flags.
My problem is that, while this integration has been working fine, yesterday I noticed the edge case executed for an invoice which should not be empty, and thus it got deleted by the C++ program.
PHP manual on file_put_contents mentions the following for the LOCK_EX flag:
Acquire an exclusive lock on the file while proceeding to the writing. In other words, a flock() call happens between the fopen() call and the fwrite() call. This is not identical to an fopen() call with mode "x".
Could the problem be caused as a race condition by the LOCK_EX not being established right before file_put_contents calls fopen? If so, what could be done to solve this while keeping the edge case removal code?
Otherwise, may I be doing anything wrong overall?
Your code is assuming that the file_put_contents() operation is atomic, and that using FLOCK_EX and FLOCK_SH is enough to ensure no race conditions between the two programs happen. This is not the case.
As you can see from the PHP doc, the FLOCK_EX is applied after opening the file. This is important, because it leaves a short window of time for the C++ program to successfully open the file and lock it with FLOCK_SH. At that point the file was already truncated by the fopen() done by PHP, and it's empty.
What's most likely happening is:
PHP code opens the file for writing, truncating it and effectively wiping out its content.
C++ code opens the file for reading.
C++ code requests the shared lock on the file: the lock is granted.
PHP code requests the exclusive lock on the file: the call blocks, waiting for the lock to be available.
C++ code reads the file's contents: nothing, the file is empty.
C++ code deletes the file.
C++ code releases the shared lock.
PHP code acquires the exclusive lock.
PHP code writes to the file: the data does not reach the disk because the inode associated with the open file descriptor does not exist anymore.
You are effectively left with no file and the data is lost.
The problem with your code is that the operations you are doing on the file from two different programs are not atomic, and the way you are acquiring the locks does not help in ensuring that those don't overlap.
The only sane way of guaranteeing the atomicity of such an operation on a POSIX compliant system, without even worrying about file locking, is to take advantage of the atomicity of rename(2):
If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing.
If newpath exists but the operation fails for some reason, rename() guarantees to leave an instance of newpath in place.
The equivalent rename() PHP function is what you should use in this case. It's the simplest way to guarantee atomic updates to a file.
What I would suggest is the following:
PHP code:
$tmpfname = tempnam("/tmp", "myprefix"); // Create a temporary file.
file_put_contents($tmpfname, "contents"); // Write to the temporary file.
rename($tmpfname, "sampleDir/invoice.xml"); // Atomically replace the contents of invoice.xml by renaming the file.
// TODO: check for errors in all the above calls, most importantly tempnam().
C++ code:
FILE* pInvoiceFile = fopen("sampleDir/invoice.xml", "r");
if (pInvoiceFile != NULL)
{
struct stat fileStat;
fstat(fileno(pInvoiceFile), &fileStat);
string invoice;
invoice.resize(fileStat.st_size);
size_t n = fread(&invoice[0], 1, fileStat.st_size, pInvoiceFile);
fclose(pInvoiceFile);
if (n == 0)
remove("sampleDir/invoice.xml");
}
This way, the C++ program will always either see the old version of the file (if fopen() happens before PHP's rename()) or the new version of the file (if fopen() happens after), but it will never see an inconsistent version of the file.
I am trying to understand the right way to synchronize file read/write using the flock in PHP.
I have two php scripts.
testread.php:
<?
$fp=fopen("test.txt","r");
if (!flock($fp,LOCK_SH))
echo "failed to lock\n";
else
echo "lock ok\n";
while(true) sleep(1000);
?>
and testwrite.php:
<?
$fp=fopen("test.txt","w");
if (flock($fp,LOCK_EX|LOCK_NB))
{
echo "acquired write lock\n";
}
else
{
echo "failed to acquire write lock\n";
}
fclose($fp);
?>
Now I run testread.php and let it hang there. Then I run testwrite.php in another session. As expected, flock failed in testwrite.php. However, the content of the file test.txt is cleared when testwrite.php exits. The fact is, fopen always succeeds even if the file has been locked in another process. If the file is opened with "w" mode, the file content will be erased regardless of the lock. So what is the point of flock here? It doesn't really protect anything.
You are using fopen() with the w mode in testwrite.php. When using the w option fopen() will truncate the file after opening it. (see fopen()).
Because of that the file gets truncated in your example before you try to obtain the exclusive lock. However you'll need an open file descriptor in order to use flock().
The way out of this dilemma is to use a lock file different from the file you are working on. The flock() manual page mentions this:
Because flock() requires a file pointer, you may have to use a special lock file to protect access to a file that you intend to truncate by opening it in write mode (with a "w" or "w+" argument to fopen()).
The accepted answer is overly complicated. You can simply open the file using a "c" argument, which doesn't truncate the file. Then call ftruncate() only if you acquire the lock.
From the documentation:
'c' Open the file for writing only. If the file does not exist, it is
created. If it exists, it is neither truncated (as opposed to 'w'),
nor the call to this function fails (as is the case with 'x'). The
file pointer is positioned on the beginning of the file. This may be
useful if it's desired to get an advisory lock (see flock()) before
attempting to modify the file, as using 'w' could truncate the file
before the lock was obtained (if truncation is desired, ftruncate()
can be used after the lock is requested).
I want to have a PHP file that is used as a counter. It will a) echo the current value of a txt file, and b) increment that file using an exclusive lock so no other scripts can read or write to it while it's being used.
User A will write and increment this number, while User B requests to read the file. Is it possible that User A can lock this file so no one can read or write to it until User A's write is finished?
I've used flock in the past, but I'm not sure how to get the file to wait until it is available, rather than quitting if it's already been locked
My goal is:
LOCK counter.txt; write to counter.txt;
while at the same time
Read counter.txt; realize it's locked so wait until that lock is finished.
//
$fp = fopen("counter.txt", 'w+');
if(flock($fp, LOCK_EX)) {
fwrite($fp, $counter + 1);
flock($fp, LOCK_UN);
} else {
// try again??
}
fclose($fp);
From documentation: By default, this function will block until the requested lock is acquired
So simply use flock in your reader (LOCK_SH) and writer (LOCK_EX), and it is going to work.
However I highly discourage use of blocking flock without timeout as this means that if something goes wrong then your program is going to hang forever. To avoid this use non-blocking request like this (again, it is in doc):
/* Activate the LOCK_NB option on an LOCK_EX operation */
if(!flock($fp, LOCK_EX | LOCK_NB)) {
echo 'Unable to obtain lock';
}
And wrap it in a for loop, with sleep and break after failed n-tries (or total wait time).
EDIT: You can also look for some examples of usage here. This class is a part of ninja-mutex library in which you may be interested too.
Ok, I know that there's been similar questions on this site about this problem, but none of this questions and provided answers isn't exactly what I need.
I'm building flat-file based CMS.
What if, for example:
2, 3, 10..... fwrite in appending mode requestes come to same php file "contact_form_messages_container.php" at the same time?
2, 3, 10..... fwrite in "w" mode requestes come to same php file which holds the simpley nubmer of specific page visits, again at the same time?
I know about flock() function, but it could happen two or more flock() requests comes on the same time... Does anyone knows solution to this problem? Only thing I have on my mind is usleep()-ing the script using while looop for some amount of time, until the target file becomes availibile, but I do not have idea if it works, where and how to perform this?
Does anyone have practical expirience with this issue?
Thanks in advance!
The flock() function is designed to handle multiple concurrent readers and writers for file operations; by default flock() may suspend a process until a compatible lock can be obtained (i.e. shared or exclusive). Once obtained, a lock can later be released to allow other processes to operate on the file; locks are released implicitly when the file is closed or the process ends.
Unless your files are on NFS, I highly doubt you will ever run into a situation whereby two conflicting locks would be given simultaneously.
The following illustrates a basic example of using flock():
// open the file (take care to not use "w" mode)
$f = fopen('file.txt', 'r+');
// obtain an exlusive lock (may suspend the process)
if (flock($f, LOCK_EX)) {
// this process now holds the only exclusive lock
// make changes to the file
// release the lock
flock($f, LOCK_UN);
}
// don't perform any write operation on $f here
fclose($f);
Using the LOCK_NB flag together with LOCK_EX or LOCK_SH will prevent the process from being suspended; if the call returns false a third parameter can be passed to determine whether the process would have been suspended (not supported on Windows).
if (false === flock($f, LOCK_EX | LOCK_NB, $wouldblock)) {
if ($wouldblock) {
// the lock could not be obtained without suspending the process
} else {
// the lock could not be obtained due to an error
}
}
Fork the writing operation and put it into a while loop with a sleep while file is locked. As long as your application doesn't depend on making file system calls chronologically it should work.
This does however open for race conditions if your application depends on writing operations to happen in order.