I want to have a temp file that gets updated from time to time.
What I was thinking of doing is:
<!-- language: lang-php -->
// get the contents
$s = file_get_contents( ... );
// does it need updating?
if( needs_update() )
{
$s = 'some new content';
file_put_contents( ... );
}
The issue that I could see happening is that whatever condition causes 'needs_update()' to return true could cause more than one process to update the same file at, (almost), the same time.
In an ideal situation, I would have one single process updating the file and prevent all other processes from reading the file until I am done with it.
So as soon as 'needs_update()' return true is called I would prevent others processes from reading the file.
<!-- language: lang-php -->
// wait here if anybody is busy writing to the file.
wait_if_another_process_is_busy_with_the_file();
// get the contents
$s = file_get_contents( ... );
// does it need updating?
if( needs_update() )
{
// prevent read/write access to the file for a moment
prevent_read_write_to_file_and_wait();
// rebuild the new content
$s = 'some new content';
file_put_contents( ... );
}
That way, only one process could possibly update the file and the files would all be getting the latest values.
Any suggestions on how I could prevent such a conflict?
Thanks
FFMG
You are looking for the flock function. flock will work as long as everyone that acess the file is using it. Example from php manual:
$fp = fopen("/tmp/lock.txt", "r+");
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
ftruncate($fp, 0); // truncate file
fwrite($fp, "Write something here\n");
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
fclose($fp);
Manual: http://php.net/manual/en/function.flock.php
Related
When I run this function on multiple scripts one script generated warning:
fread(): Length parameter must be greater than 0
function test($n){
echo "<h4>$n at ".time()."</h4>";
for ($i = 0; $i<50; $i++ ){
$fp = fopen("$n.txt", "r");
$s = fread($fp, filesize("$n.txt") );
fclose($fp);
$fp = fopen("$n.txt", "w");
$s = $_SERVER['HTTP_USER_AGENT'].' '.time();
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
fwrite($fp, $s);
// fflush($fp);// flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
}
}
I try to write reading of the file for multiple users, but only one user can write the file. I know that when I use fwrite with flock - LOC_EX, next scripts must wait till the write is finished. But here it seems like filesize doesn't wait till the write operation is finished. My opinion is that it tries to reach the file when the file size is 0, and as a result this produces the problem: 0 bytes will be read from the file, when it is written by original script.
Is it possible to fix this for fread function?
Purpose of this script is to test fread with some limit and to check the data which I read later, if the data are really written when I did not used fflush.
function test($n){
echo "<h4>$n at ".time()."</h4>";
for ($i = 0; $i<50; $i++ ){
$start = microtime(true);
$fp = fopen("$n.txt", "r");
if(filesize($n.txt) > 0)
{
$s = fread($fp, filesize($n.txt) );
fclose($fp);
$fp = fopen("$n.txt", "w");
$s = $_SERVER['HTTP_USER_AGENT'].' '.time();
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
fwrite($fp, $s);
// fflush($fp);// flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
}
else
{
echo "Filesize must be greater than 0";
}
}
}
please change $s variables name its use same things two time
$fp = fopen("$n.txt", "r");
$s = fread($fp, filesize("$n.txt") );
fclose($fp);
The error occurs in the middle line of the above three lines.
Firstly, these three lines could be rewritten into a single line as follows:
$s = file_get_contents("$n.txt");
However, these isn't necessary, as these three lines are entirely redundant in your code. They don't do anything useful.
What they do is open a file, store its contents to $s and then close it.
But you are then immediately setting $s to a different value, thus throwing away the previous value, and making it pointless to have read it from the file in the first place.
If you need to keep the original contents of the file, then use file_get_contents() and make sure you don't overwrite the contents of the variable.
If you don't need the original contents of the file, then just delete those three lines from your code.
Incidentally, this error highlights a couple of good coding practices that you should take on board: Firstly, never re-use a variable for two different things, and secondly always give your variables (and functions) good names. $s is not a good name; $previousFileContents would be a better name; it would have made the error much more obvious.
I am using JSON as a data store and over time I foresee that several parties might be wanting to write to my JSON file like a chat log within a short space of time.
<?php
$foo = json_decode(file_get_contents("foo.json"), true);
if (! is_array($foo["bar"])) { $foo["bar"] = array(); }
array_push($foo["bar"], array("time" => time(), "who" => $_SERVER['REMOTE_ADDR'], msg => $_GET['m']));
file_put_contents("foo.json", json_encode($foo, JSON_PRETTY_PRINT));
?>
So the above code works, but I am worried what happens if the file is read before it's written out, or some case where they are writing out at the same time, leading to some data loss?
What's a better or safer design, preferably using flat file storage (i.e. not databases)?
As for a bonus I really don't want to return to my client who made the request this that were was some "lock". Ideally the request is made to wait until it's safe to return.
You can use the flock() function for this. What it does is it locks the file for all processes except the current one.
http://php.net/manual/en/function.flock.php
Basic usage:
<?php
$fp = fopen('path/to/data.json', 'r+');
if (flock($fp, LOCK_EX)) // locks the file
{
// write to the file
flock($fp, LOCK_UN); // remove the lock
}
fclose($fp);
flock() is blocking by default. That means a process is waiting until it gets permission to access the lock. Have a look at the docs on how to implement a nonblocking version.
How a bout you create individual JSON for each line so you do not need load Json every time. and you just append JSON to the file.
Each Joson will be in One line.
When you load it you will read each line in text file then convert it to Json within PHP.
The reason I recommend this way as we want to find other way to have better solution for file read and write using //http://php.net/manual/en/function.file-put-contents.php
file_put_contents("foo.json", json_encode($foo, JSON_PRETTY_PRINT).
JSON is just a format for text. So I believe that It is a good solution to do.
To write
<?php
$_SERVER['REMOTE_ADDR'], msg => $_GET['m']));
$foo = array("time" => time(), "who" => $_SERVER['REMOTE_ADDR'], msg => $_GET['m']);
//http://php.net/manual/en/function.file-put-contents.php
file_put_contents("foo.json", json_encode($foo, JSON_PRETTY_PRINT), FILE_APPEND | LOCK_EX));
?>
As you mention it is a log, so you may not need to load to see it all the time.
So we will to focus on writing. It may seem long process to read.
TO read
<?php
$log_array = array();
$handle = #fopen("foo.json", "r");
if ($handle) {
while (($buffer = fgets($handle, 4096)) !== false) {
$foo = json_decode($buffer, true);
$log_array[] = $foo;
}
if (!feof($handle)) {
echo "Error: unexpected fgets() fail\n";
}
fclose($handle);
}
print_r($log_array);
?>
I make this simple counter
$now = date ("d");
$filename = $now .".txt";
$lastcount="";
if (file_exists($filename))
{
if (time()-filemtime($filename) > 2 * 86400) {
} else {
$lastcount=strval(intval (file_get_contents($filename))+1);
}
}
file_put_contents($filename, $lastcount);
Basically it reads a file, then add 1, then rewrite
The problem is between the time I read the file, and writing it back, another copy of the program may read the file and write it.
So how do I make that atomic?
I also want to ensure that the whole script won't "crash" because of this locking.
So how to improve this counter?
You can lock file using flock. Use exclusive locking to write to file safely:
$fp = fopen($filename, "rw");
if (flock($fp, LOCK_EX)) {
// write here
// ...
// release the file
flock($fp, LOCK_UN);
} else {
// can't use it yet. Wait a little.
}
Nowdays there are a lot of websites for files hosting (uploading websites) and it count for example point per complete download of certain file.
My question
I want to understand what is the idea they are using !
How does it only count on complete downloading of the file ?!
i mean if i canceled downloading of the file after it started , it won't count point!
how does it knew ! is there any php function that able to know if i canceled downloading certain exact file or not !
that question was all time in my mind and thinking about it but i can't understand how does it works or what is the idea behind it. ~ thanks
This can be done by using my other answer as base How can I give download access to files outside public_html directory? and replacing readfile( $filename )
with readfileWhileConnected( $filename ):
Read file until EOF or disconnect:
/** Read $filename until EOF or disconnect,
* if disconnect then error_log() count of bytes read already
*/
function readfileWhileConnected( $filename ) {
// Save and set ini values:
$user_abort = ignore_user_abort();
ignore_user_abort(false);
// Get file size and set bytes_sent to zero:
$fsize = filesize($filename);
$bytes_sent = 0;
// Open file:
$f = fopen($filename, 'r');
// Read file:
while($chunk = fread($f, 1024)) {
// Check if connection is still open:
if(!connection_aborted()) {
// Send $chunk to buffer (if any), then flush() buffers:
echo $chunk;
flush();
// Add $chunk length to $bytes_sent
$bytes_sent += strlen($chunk);
} else {
// Close file:
fclose($f);
error_log("Connection closed at $bytes_sent/$fsize");
exit();
}
// Close file:
fclose($f);
// Reset ini values:
ignore_user_abort($user_abort);
return $bytes_sent;
}
}
After you have your new shiny class myNewSuperDownloadHandlerClass { ... } ready, then make sure you only serve downloads through filedownload.php described here or if have done good myNewSuperDownloadHandlerClass(), then use that, just make sure that readfileWhileConnected() is used for every download requiring connection status polling.
You can easily add callback to be triggered if user closes connection, only 2 exit points here. (seen many functions that have every often return false; return true; return null; return false; return true; and so on..)
I'm trying to write a PHP script that I want to ensure only has a single instance of it running at any given time. All of this talk about different ways of locking, and race conditions, and etc. etc. etc. is giving me the willies.
I'm confused as to whether lock files are the way to go, or semaphores, or using MySQL locks, or etc. etc. etc.
Can anyone tell me:
a) What is the correct way to implement this?
AND
b) Point me to a PHP implementation (or something easy to port to PHP?)
One way is to use the php function flock with a dummy file, that will act as a watchdog.
On the beginning of our job, if the file raise a LOCK_EX flag, exit, or wait, can be done.
Php flock documentation: http://php.net/manual/en/function.flock.php
For this examples, a file called lock.txt must be created first.
Example 1, if another twin process is running, it will properly quit, without retrying, giving a state message.
It will throw the error state, if the file lock.txt isn't reachable.
<?php
$fp = fopen("lock.txt", "r+");
if (!flock($fp, LOCK_EX|LOCK_NB, $blocked)) {
if ($blocked) {
// another process holds the lock
echo "Couldn't get the lock! Other script in run!\n";
}
else {
// couldn't lock for another reason, e.g. no such file
echo "Error! Nothing done.";
}
}
else {
// lock obtained
ftruncate($fp, 0); // truncate file
// Your job here
echo "Job running!\n";
// Leave a breathe
sleep(3);
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
}
fclose($fp); // Empty memory
Example 2, FIFO (First in, first out): we wants the process to wait, for an execution after the queue, if any:
<?php
$fp = fopen("lock.txt", "r+");
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
ftruncate($fp, 0); // truncate file
// Your job here
echo "Job running!\n";
// Leave a breathe
sleep(3);
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
}
fclose($fp);
It is also doable with fopen into x mode, by creating and erasing a file when the script ends.
Create and open for writing only; place the file pointer at the
beginning of the file. If the file already exists, the fopen() call
will fail by returning FALSE
http://php.net/manual/en/function.fopen.php
However, into a Unix environment, for fine tuning, I found easier to list the PID's of every background scripts with getmypid() into a DB, or a separate JSON file.
When one task ends, the script is responsible to declare his state in this file (eq: success/failure/debug infos, etc), and then remove his PID. This allows from my view to create admins tools and daemons in a simpler way. And use posix_kill() to kill a PID from PHP if necessary.
Micro-Services are composed using Unix-like pipelines.
Services can call services.
https://en.wikipedia.org/wiki/Microservices
See also: Prevent PHP script using up all resources while it runs?
// borrow from 2 anwsers on stackoverflow
function IsProcessRunning($pid) {
return shell_exec("ps aux | grep " . $pid . " | wc -l") > 2;
}
function AmIRunning($process_file) {
// Check I am running from the command line
if (PHP_SAPI != 'cli') {
error('Run me from the command line');
exit;
}
// Check if I'm already running and kill myself off if I am
$pid_running = false;
$pid = 0;
if (file_exists($process_file)) {
$data = file($process_file);
foreach ($data as $pid) {
$pid = (int)$pid;
if ($pid > 0 && IsProcessRunning($pid)) {
$pid_running = $pid;
break;
}
}
}
if ($pid_running && $pid_running != getmypid()) {
if (file_exists($process_file)) {
file_put_contents($process_file, $pid);
}
info('I am already running as pid ' . $pid . ' so stopping now');
return true;
} else {
// Make sure file has just me in it
file_put_contents($process_file, getmypid());
info('Written pid with id '.getmypid());
return false;
}
}
/*
* Make sure there is only one instance running at a time
*/
$lockdir = '/data/lock';
$script_name = basename(__FILE__, '.php');
// The file to store our process file
$process_file = $lockdir . DS . $script_name . '.pid';
$am_i_running = AmIRunning($process_file);
if ($am_i_running) {
exit;
}
Use semaphores:
$key = 156478953; //this should be unique for each script
$maxAcquire = 1;
$permissions =0666;
$autoRelease = 1; //releases semaphore when request is shut down (you dont have to worry about die(), exit() or return
$non_blocking = false; //if true, fails instantly if semaphore is not free
$semaphore = sem_get($key, $maxAcquire, $permissions, $autoRelease);
if (sem_acquire($semaphore, $non_blocking )) //blocking (prevent simultaneous multiple executions)
{
processLongCalculation();
}
sem_release($semaphore);
See:
https://www.php.net/manual/en/function.sem-get.php
https://www.php.net/manual/en/function.sem-acquire.php
https://www.php.net/manual/en/function.sem-release.php
You can go for the solution that fits best your project, the two simple ways to achieve that are file locking or database locking.
For implementations of file locking, check http://us2.php.net/flock
If you already use a database, create a table, generate known token for that script, put it there, and just remove it after the end of the script. To avoid problems on errors, you can use expiry times.
Perhaps this could work for you,
http://www.electrictoolbox.com/check-php-script-already-running/
In case you are using php on linux and I think the most practical way is:
<?php
if(shell_exec('ps aux | grep '.__FILE__.' | wc -l')>3){
exit('already running...');
}
?>
Another way to do it is with file flag and exit callback,
the exit callback will ensures that the file flag will be reset to 0 in any case of php execution end also fatal errors.
<?php
function exitProcess(){
if(file_get_contents('inprocess.txt')!='0'){
file_put_contents('inprocess.txt','0');
}
}
if(file_get_contents('inprocess.txt')=='1'){
exit();
}
file_put_contents('inprocess.txt','1');
register_shutdown_function('exitProcess');
/**
execute stuff
**/
?>