Is there some url/stream that fopen will successfully open on most PHP installations? /dev/null is not available or openable on some systems. Something like php://temp should be a fairly safe bet, right?
The application for this code that guarantees a file resource, instead of the mixed filetype of bool|resource you have with fopen:
/**
* #return resource
*/
function openFileWithResourceGuarantee() {
$fh = #fopen('/write/protected/location.txt', 'w');
if ( $fh === false ) {
error_log('Could not open /write/protected/location.txt');
$fh = fopen('php://temp');
}
return $fh;
}
In PHP 7 with strict types, the above function should guarantee a resource and avoid bools. I know that resources are not official types, but still want to be as type-safe as possible.
php://memory should be universally available.
If you need a stream for writing errors to why are you not writing to php://stderr?
Example from the docs:
When logging to apache on windows, both error_log and also
trigger_error result in an apache status of error on the front of the
message. This is bad if all you want to do is log information. However
you can simply log to stderr however you will have to do all message
assembly:
LogToApache($Message) {
$stderr = fopen('php://stderr', 'w');
fwrite($stderr,$Message);
fclose($stderr);
}
Note: php://stderr is sometimes the same as php://stdout, but not always.
For streams see: http://php.net/manual/en/wrappers.php.php
Something like php://temp should be a fairly safe bet, right?
As #weirdan already pointed out php://memory is probably safer as it does not even need to create any file. Memory access MUST be possible. From the docs:
php://memory and php://temp are read-write streams that allow
temporary data to be stored in a file-like wrapper. The only
difference between the two is that php://memory will always store its
data in memory, whereas php://temp will use a temporary file once the
amount of data stored hits a predefined limit (the default is 2 MB).
The location of this temporary file is determined in the same way as
the sys_get_temp_dir() function.
Not sure if this answers your question completely but does it lead you into the right direction?
Related
I have a PHP script which receives and saves invoices as files in Linux. Later, a C++ infinite loop based program reads each and does some processing. I want the latter to read each file safely (only after fully written).
PHP side code simplification:
file_put_contents("sampleDir/invoice.xml", "contents", LOCK_EX)
On the C++ side (with C filesystem API), I must first note that I want to preserve a code which deletes the files in the designated invoices folder which are empty, just as a means to properly deal with the edge case of an empty file being created from other sources (not the PHP script).
Now, here's a C++ side code simplification, too:
FILE* pInvoiceFile = fopen("sampleDir/invoice.xml", "r");
if (pInvoiceFile != NULL)
{
if (flock(pInvoiceFile->_fileno, LOCK_SH) == 0)
{
struct stat fileStat;
fstat(pInvoiceFile->_fileno, &fileStat);
string invoice;
invoice.resize(fileStat.st_size);
if (fread((char*)invoice.data(), 1, fileStat.st_size, pInvoiceFile) < 1)
{
remove("sampleDir/invoice.xml"); // Edge case resolution
}
flock(pInvoiceFile->_fileno, LOCK_UN);
}
}
fclose(pInvoiceFile);
As you can see, the summarizing key concept is the cooperation of LOCK_EX and LOCK_SH flags.
My problem is that, while this integration has been working fine, yesterday I noticed the edge case executed for an invoice which should not be empty, and thus it got deleted by the C++ program.
PHP manual on file_put_contents mentions the following for the LOCK_EX flag:
Acquire an exclusive lock on the file while proceeding to the writing. In other words, a flock() call happens between the fopen() call and the fwrite() call. This is not identical to an fopen() call with mode "x".
Could the problem be caused as a race condition by the LOCK_EX not being established right before file_put_contents calls fopen? If so, what could be done to solve this while keeping the edge case removal code?
Otherwise, may I be doing anything wrong overall?
Your code is assuming that the file_put_contents() operation is atomic, and that using FLOCK_EX and FLOCK_SH is enough to ensure no race conditions between the two programs happen. This is not the case.
As you can see from the PHP doc, the FLOCK_EX is applied after opening the file. This is important, because it leaves a short window of time for the C++ program to successfully open the file and lock it with FLOCK_SH. At that point the file was already truncated by the fopen() done by PHP, and it's empty.
What's most likely happening is:
PHP code opens the file for writing, truncating it and effectively wiping out its content.
C++ code opens the file for reading.
C++ code requests the shared lock on the file: the lock is granted.
PHP code requests the exclusive lock on the file: the call blocks, waiting for the lock to be available.
C++ code reads the file's contents: nothing, the file is empty.
C++ code deletes the file.
C++ code releases the shared lock.
PHP code acquires the exclusive lock.
PHP code writes to the file: the data does not reach the disk because the inode associated with the open file descriptor does not exist anymore.
You are effectively left with no file and the data is lost.
The problem with your code is that the operations you are doing on the file from two different programs are not atomic, and the way you are acquiring the locks does not help in ensuring that those don't overlap.
The only sane way of guaranteeing the atomicity of such an operation on a POSIX compliant system, without even worrying about file locking, is to take advantage of the atomicity of rename(2):
If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing.
If newpath exists but the operation fails for some reason, rename() guarantees to leave an instance of newpath in place.
The equivalent rename() PHP function is what you should use in this case. It's the simplest way to guarantee atomic updates to a file.
What I would suggest is the following:
PHP code:
$tmpfname = tempnam("/tmp", "myprefix"); // Create a temporary file.
file_put_contents($tmpfname, "contents"); // Write to the temporary file.
rename($tmpfname, "sampleDir/invoice.xml"); // Atomically replace the contents of invoice.xml by renaming the file.
// TODO: check for errors in all the above calls, most importantly tempnam().
C++ code:
FILE* pInvoiceFile = fopen("sampleDir/invoice.xml", "r");
if (pInvoiceFile != NULL)
{
struct stat fileStat;
fstat(fileno(pInvoiceFile), &fileStat);
string invoice;
invoice.resize(fileStat.st_size);
size_t n = fread(&invoice[0], 1, fileStat.st_size, pInvoiceFile);
fclose(pInvoiceFile);
if (n == 0)
remove("sampleDir/invoice.xml");
}
This way, the C++ program will always either see the old version of the file (if fopen() happens before PHP's rename()) or the new version of the file (if fopen() happens after), but it will never see an inconsistent version of the file.
in my PHP project, I use some kind of counter that appends to an existing (or new) file very often:
$f = fopen($filename, 'ab');
fwrite($f);
fclose($f);
When a new file is created, I have to edit this file's permissions, so another user may access the file as well:
$existed = file_exists($filename);
// Do the append
$f = fopen($filename, 'ab');
fwrite($f);
fclose($f);
// Update permissions
if (!$existed) {
#chmod($filename, 0666);
}
Is there any way to find out, whether 'a' (append) created a new file or appended to an exiting one without using file_exists()? To my understanding, file_exists() retrieves the file stats, which causes some unnecessary overhead compared to a simple file-append. As the function is used very often, I wonder if there's an option to tell if fopen(..., 'a') created a new file without using file_exists()?
Note: This is mostly a question of style and interest, not a true performance issue. But if I am mistaken and fopen() already retrieves the file stats, please let me know!
Update
Okay, it really is a rather academic question. Here're some performance tests run on a windows system (Apache, Win8.1 - no UNIX file permissions) and a linux machine (Nginx, Ubuntu 14.04, virutal machine).
Each test run with 1000 repetitions, file deleted before the first repetition.
Win Linux
simply append one byte 1.8ms 9.4ms
append + clearstatcache() 1.8ms 9.3ms
test fileexists() + append 2.2ms 10.5ms
fileexists() + append + clear 2.2ms 11.0ms
append + chmod() 2.7ms 12.3ms
append + fileexists() -> chmod() 3.3ms 10.6ms
Note: The last one is the only one that uses and IF within the test loop.
The php fopen is just a call to the libc fopen, that automatically creates a file for the modes w,w+,a and a+. As far as I can see, there is no way to get the stat with the permission bits from the returned file pointer.
It seems that PHP stores the stat array for each opened file and you can access it with fstat($fp) with the opened file handle $fp. But the mode field contains inode permission bits. I can't immediately see how "inode permission bits" are related to the "UNIX file mode". The stat system call does not use this term.
You can use "r+" mode to open your file and create it if that fails. If not you need to SEEK to then end to achieve something similar.
But finally it's best to check for existence before you open the file.
No, fopen just returns the resource, it doesn't return or set a flag that indicates whether the file already existed - http://php.net/manual/en/function.fopen.php
EDIT: see the performance test in the edited question.
Why not calling chmod() every times?
Your file_exist() is probably (maybe a little performance test...) more expensive than a chmod().
// Do the append
$f = fopen($filename, 'ab');
fwrite($f);
fclose($f);
// Update persmissions
#chmod($this->filename, 0666);
I have a blob resource from my db. I want to wrap temporaly this file into Symfony File object because I want to use specific methods like the extension guesser, and apply symfony file validators.
I want to store this temporary file into memory, because the blobs are small files and i dont want to create a file in disk in every request.
I tried to do this in that way:
$file = new File ('php://temp');
but symfony throws an error that says 'The file "php://temp" does not exist'. Looking at File source, the error is caused by a "is_file($path)" check that is made in the constructor, and I can invalidate this putting false in the second argument. But, if I do:
$file = new File ('php://temp', false);
the File is created, but then the error comes back later, e.g. when i use the guesser:
$file->guessExtension($file)
because in Symfony/Component/HttpFoundation/File/MimeType/MimeTypeGuesser.php:
public function guess($path)
{
if (!is_file($path)) {
throw new FileNotFoundException($path);
}
(...)
Ok. Then my question is: There is a way to load a 'php://temp' or 'php://memory' within a File object?
Pretty sure php://temp writes to memory until it is full and then writes to a file, whereas php://memory ensures only in memory with no fall back.
This likely happens because php://temp and php://memory are non-reusable, so once you've written to it the content may not still be there when you next want it. From the PHP manual:
php://memory and php://temp are not reusable, i.e. after the streams have been closed there is no way to refer to them again.
file_put_contents('php://memory', 'PHP');
echo file_get_contents('php://memory'); // prints nothing
How are you writing to php://temp to begin with? That will be the more important issue rather than with Symfony's File class. I suspect that by the time you are creating a File instance that php://temp has already gone.
It's worth noting that using php://temp will create a file on disk in the temporary location, so you might as well use write to a tempnam() handle anyway. At least then you will have a reference to a physical (but temporary) file.
I suggested to allow passing the file contents (instead of the path) to Symfony's MIME type guesser, to enable guessing "on-the-fly": https://github.com/symfony/symfony/issues/40916
Here's how I do it right now:
use Symfony\Component\Mime\MimeTypes;
$tmpFilename = tempnam(sys_get_temp_dir(), 'guessMimeType_');
file_put_contents($tmpFilename, $content);
$mimeTypes = new MimeTypes();
$guessedMimeType = $mimeTypes->guessMimeType($tmpFilename);
unlink($tmpFilename);
The first line is taken from https://www.php.net/manual/en/function.tempnam.php#93256
I am trying to understand the right way to synchronize file read/write using the flock in PHP.
I have two php scripts.
testread.php:
<?
$fp=fopen("test.txt","r");
if (!flock($fp,LOCK_SH))
echo "failed to lock\n";
else
echo "lock ok\n";
while(true) sleep(1000);
?>
and testwrite.php:
<?
$fp=fopen("test.txt","w");
if (flock($fp,LOCK_EX|LOCK_NB))
{
echo "acquired write lock\n";
}
else
{
echo "failed to acquire write lock\n";
}
fclose($fp);
?>
Now I run testread.php and let it hang there. Then I run testwrite.php in another session. As expected, flock failed in testwrite.php. However, the content of the file test.txt is cleared when testwrite.php exits. The fact is, fopen always succeeds even if the file has been locked in another process. If the file is opened with "w" mode, the file content will be erased regardless of the lock. So what is the point of flock here? It doesn't really protect anything.
You are using fopen() with the w mode in testwrite.php. When using the w option fopen() will truncate the file after opening it. (see fopen()).
Because of that the file gets truncated in your example before you try to obtain the exclusive lock. However you'll need an open file descriptor in order to use flock().
The way out of this dilemma is to use a lock file different from the file you are working on. The flock() manual page mentions this:
Because flock() requires a file pointer, you may have to use a special lock file to protect access to a file that you intend to truncate by opening it in write mode (with a "w" or "w+" argument to fopen()).
The accepted answer is overly complicated. You can simply open the file using a "c" argument, which doesn't truncate the file. Then call ftruncate() only if you acquire the lock.
From the documentation:
'c' Open the file for writing only. If the file does not exist, it is
created. If it exists, it is neither truncated (as opposed to 'w'),
nor the call to this function fails (as is the case with 'x'). The
file pointer is positioned on the beginning of the file. This may be
useful if it's desired to get an advisory lock (see flock()) before
attempting to modify the file, as using 'w' could truncate the file
before the lock was obtained (if truncation is desired, ftruncate()
can be used after the lock is requested).
I'm having trouble with reading and writing the php://temp stream in PHP 5.3.2
I basically have:
file_put_contents('php://temp/test', 'test');
var_dump(file_get_contents('php://temp/test'));
The only output I get is string(0) ""
Shouldn't I get my 'test' back?
php://temp is not a file path, it's a pseudo protocol that always creates a new random temp file when used. The /test is actually being ignored entirely. The only extra "arguments" the php://temp wrapper accepts is /maxmemory:n. You need to keep a file handle around to the opened temp stream, or it will be discarded:
$tmp = fopen('php://temp', 'r+');
fwrite($tmp, 'test');
rewind($tmp);
fpassthru($tmp);
fclose($tmp);
See http://php.net/manual/en/wrappers.php.php#refsect1-wrappers.php-examples
Each time, when you use fopen to get handler, content of php://temp will be flushed. Use rewind() and stream_get_contents() to get content. Or, use normal cachers, like APC or memcache :)
Finally found a documented small note, that explains why
Example 5 at the PHP Manual used almost your exact same code sample and says
php://memory and php://temp are not reusable, i.e. after the streams
have been closed there is no way to refer to them again.
file_put_contents('php://memory', 'PHP');
echo file_get_contents('php://memory'); // prints nothing
I guess this means that file_put_contents() closes the stream internally, which makes file_get_contents() unable to recover the data in the stream again
I know this is late, but in addition to #OZ_'s answer, i just discovered that 'fread' works too, after you rewind.
$handle = fopen('php://temp', 'w+');
fwrite($handle, 'I am freaking awesome');
fread($handle); // returns '';
rewind($handle); // resets the position of pointer
fread($handle, fstat($handle)['size']); // I am freaking awesome