I'm trying to find a safe way to prevent a cron job collision (ie. prevent it from running if another instance is already running).
Some options I've found recommend using a lock on a file.
Is that really a safe option? What would happen if the script dies for example? Will the lock remain?
Are there other ways of doing this?
This sample was taken at http://php.net/flock and changed a little and this is a correct way to do what you want:
$fp = fopen("/path/to/lock/file", "w+");
if (flock($fp, LOCK_EX | LOCK_NB)) { // do an exclusive lock
// do the work
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
fclose($fp);
Do not use locations such as /tmp or /var/tmp as they could be cleaned up at any time by your system, thus messing with your lock as per the docs:
Programs must not assume that any files or directories in /tmp are preserved between invocations of the program.
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.html
Do use a location that is under your control.
Credits:
Michaƫl Perrin - for proposing to use w+ instead of r+
In Symfony Framework you could use the lock component symfony/lock
https://symfony.com/doc/current/console/lockable_trait.html
I've extended the concept from zerkms to create a function that can be called from the start of a cron.
Using the Cronlocker you specify a lock name, then the name of a callback function to be called if the cron is OFF. Optionally you may give an array of parameters to pass to the callback function. There's also an optional callback function if you need to do something different if the lock is ON.
In some cases I got a few exceptions and wanted to be able to trap them, and I added a function for handling fatal exceptions, which should be added. I wanted to be able to hit the file from a browser and bypass the cronlock, so that's built in.
I found as I used this a lot there were cases where I wanted to block other crons from running while this cron is running, so I added an optional array of lockblocks, which are other lock names to block.
Then there were cases where I wanted this cron to run after other crons had finished, so there's an optional array of lockwaits, which are other lock names to wait until none of which are running.
simple example:
Cronlocker::CronLock('cron1', 'RunThis');
function RunThis() {
echo('I ran!');
}
callback parameters and failure functions:
Cronlocker::CronLock('cron2', 'RunThat', ['ran'], 'ImLocked');
function RunThat($x) {
echo('I also ran! ' . $x);
}
function ImLocked($x) {
echo('I am locked :-( ' . $x);
}
blocking and waiting:
Cronlocker::CronLock('cron3', 'RunAgain', null, null, ['cron1'], ['cron2']);
function RunAgain() {
echo('I ran.<br />');
echo('I block cron1 while I am running.<br />')
echo('I wait for cron2 to finish if it is running.');
}
class:
class Cronlocker {
private static $LockFile = null;
private static $LockFileBlocks = [];
private static $LockFileWait = null;
private static function GetLockfileName($lockname) {
return "/tmp/lock-" . $lockname . ".txt";
}
/**
* Locks a PHP script from being executed more than once at a time
* #param string $lockname Use a unique lock name for each lock that needs to be applied.
* #param string $callback The name of the function to call if the lock is OFF
* #param array $callbackParams Optional array of parameters to apply to the callback function when called
* #param string $callbackFail Optional name of the function to call if the lock is ON
* #param string[] $lockblocks Optional array of locknames for other crons to also block while this cron is running
* #param string[] $lockwaits Optional array of locknames for other crons to wait until they finish running before this cron will run
* #see http://stackoverflow.com/questions/5428631/php-preventing-collision-in-cron-file-lock-safe
*/
public static function CronLock($lockname, $callback, $callbackParams = null, $callbackFail = null, $lockblocks = [], $lockwaits = []) {
// check all the crons we are waiting for to finish running
if (!empty($lockwaits)) {
$waitingOnCron = true;
while ($waitingOnCron) {
$waitingOnCron = false;
foreach ($lockwaits as $lockwait) {
self::$LockFileWait = null;
$tempfile = self::GetLockfileName($lockwait);
try {
self::$LockFileWait = fopen($tempfile, "w+");
} catch (Exception $e) {
//ignore error
}
if (flock(self::$LockFileWait, LOCK_EX | LOCK_NB)) { // do an exclusive lock
// cron we're waiting on isn't running
flock(self::$LockFileWait, LOCK_UN); // release the lock
} else {
// we're wating on a cron
$waitingOnCron = true;
}
if (is_resource(self::$LockFileWait))
fclose(self::$LockFileWait);
if ($waitingOnCron) break; // no need to check any more
}
if ($waitingOnCron) sleep(15); // wait a few seconds
}
}
// block any additional crons from starting
if (!empty($lockblocks)) {
self::$LockFileBlocks = [];
foreach ($lockblocks as $lockblock) {
$tempfile = self::GetLockfileName($lockblock);
try {
$block = fopen($tempfile, "w+");
} catch (Exception $e) {
//ignore error
}
if (flock($block, LOCK_EX | LOCK_NB)) { // do an exclusive lock
// lock made
self::$LockFileBlocks[] = $block;
} else {
// couldn't lock it, we ignore and move on
}
}
}
// set the cronlock
self::$LockFile = null;
$tempfile = self::GetLockfileName($lockname);
$return = null;
try {
if (file_exists($tempfile) && !is_writable($tempfile)) {
//assume we're hitting this from a browser and execute it regardless of the cronlock
if (empty($callbackParams))
$return = $callback();
else
$return = call_user_func_array($callback, $callbackParams);
} else {
self::$LockFile = fopen($tempfile, "w+");
}
} catch (Exception $e) {
//ignore error
}
if (!empty(self::$LockFile)) {
if (flock(self::$LockFile, LOCK_EX | LOCK_NB)) { // do an exclusive lock
// do the work
if (empty($callbackParams))
$return = $callback();
else
$return = call_user_func_array($callback, $callbackParams);
flock(self::$LockFile, LOCK_UN); // release the lock
} else {
// call the failed function
if (!empty($callbackFail)) {
if (empty($callbackParams))
$return = $callbackFail();
else
$return = call_user_func_array($callbackFail, $callbackParams);
}
}
if (is_resource(self::$LockFile))
fclose(self::$LockFile);
}
// remove any lockblocks
if (!empty($lockblocks)) {
foreach (self::$LockFileBlocks as $LockFileBlock) {
flock($LockFileBlock, LOCK_UN); // release the lock
if (is_resource($LockFileBlock))
fclose($LockFileBlock);
}
}
return $return;
}
/**
* Releases the Cron Lock locking file, useful to specify on fatal errors
*/
public static function ReleaseCronLock() {
// release the cronlock
if (!empty(self::$LockFile) && is_resource(self::$LockFile)) {
var_dump('Cronlock released after error encountered: ' . self::$LockFile);
flock(self::$LockFile, LOCK_UN);
fclose(self::$LockFile);
}
// release any lockblocks too
foreach (self::$LockFileBlocks as $LockFileBlock) {
if (!empty($LockFileBlock) && is_resource($LockFileBlock)) {
flock($LockFileBlock, LOCK_UN);
fclose($LockFileBlock);
}
}
}
}
Should also be implemented on a common page, or built into your existing fatal error handler:
function fatal_handler() {
// For cleaning up crons that fail
Cronlocker::ReleaseCronLock();
}
register_shutdown_function("fatal_handler");
Related
I am building a log parser program in PHP. Log parser reads the data from the log file created by ProFTPD and then runs some actions if it detects specific commands. To be able to detect changes in the log file, I am using Inotify. If log file gets too large, I want to rotate the log by sending a signal to the log parser to finish processing the current file and then terminate the log parser. Logrotate would then restart the log parser again after it makes sure that the original file that is being read is emptied.
The problem is that when I use Inotify and when the inotify is in blocking state, the interrupts won't work.
For example:
#!/usr/bin/php -q
<?php
declare(ticks = 1);
$log_parser = new LogParser();
$log_parser->process_log();
class LogParser {
private $ftp_log_file = '/var/log/proftpd/proftpd.log';
# file descriptor for the log file
private $fd;
# inotify instance
private $inotify_inst;
# watch id for the inotifier
private $watch_id;
public function process_log() {
// Open an inotify instance
$this->inotify_inst = inotify_init();
$this->watch_id = inotify_add_watch($this->inotify_inst, $this->ftp_log_file, IN_MODIFY);
$this->fd = fopen($this->ftp_log_file, 'r');
if ($this->fd === false)
die("unable to open $this->ftp_log_file!\n");
pcntl_signal(SIGUSR1, function($signal) {
$this->sig_handler($signal);
});
while (1) {
# If the thread gets blocked here, the signals do not work
$events = inotify_read($this->inotify_inst);
while ($line = trim(fgets($this->fd))) {
// Parse the log ...
}
}
fclose($this->fd);
// stop watching our directory
inotify_rm_watch($this->inotify_inst, $this->watch_id);
// close our inotify instance
fclose($this->inotify_inst);
}
private function sig_handler($signo) {
switch ($signo) {
case SIGUSR1:
// Do some action ...
}
}
}
I know that one solution could be that I start the parent process and then add the signal handler to that parent process. The parent process should start the log parser and the log parser would get blocked by inotify_read, but parent process wouldn't, but was wondering if there is a solution not involving the parent process - if the inotify is able to support interrupts?
Thanks
Found a solution here: php inotify blocking but with timeout
Final code:
#!/usr/bin/php -q
<?php
declare(ticks = 1);
$log_parser = new LogParser();
$log_parser->process_log();
class LogParser {
private $ftp_log_file = '/var/log/proftpd/proftpd.log';
# file descriptor for the log file
private $fd;
# inotify instance
private $inotify_inst;
# watch id for the inotifier
private $watch_id;
public function process_log() {
// Open an inotify instance
$this->inotify_inst = inotify_init();
stream_set_blocking($this->inotify_inst, false);
$this->watch_id = inotify_add_watch($this->inotify_inst, $this->ftp_log_file, IN_MODIFY);
$this->fd = fopen($this->ftp_log_file, 'r');
if ($this->fd === false)
die("unable to open $this->ftp_log_file!\n");
pcntl_signal(SIGUSR1, function($signal) {
$this->sig_handler($signal);
});
while (1) {
while (1) {
$r = array($this->inotify_inst);
$timeout = 60;
$w = array();
$e = array();
$time_left = stream_select($r, $w, $e, $timeout);
if ($time_left != 0) {
$events = inotify_read($this->inotify_inst);
if ($events) {
break;
}
}
}
while ($line = trim(fgets($this->fd))) {
// Parse the log ...
}
}
fclose($this->fd);
// stop watching our directory
inotify_rm_watch($this->inotify_inst, $this->watch_id);
// close our inotify instance
fclose($this->inotify_inst);
}
private function sig_handler($signo) {
switch ($signo) {
case SIGUSR1:
// Do some action ...
}
}
}
The suggested solution does not block interrupts and it also sets the thread in a non blocking state.
In order to protect script form race hazard, I am considering approach described by code sample
$file = 'yxz.lockctrl';
// if file exists, it means that some other request is running
while (file_exists($file))
{
sleep(1);
}
file_put_contents($file, '');
// do some work
unlink($file);
If I go this way, is it possible to create file with same name simultaneously from multiple requests?
I know that there is php mutex. I would like to handle this situation without any extensions (if possible).
Task for the program is to handle bids in auctions application. I would like to process every bid request sequentially. With most possible latency.
From what I understand you want to make sure only a single process at a time is running a certain piece of code. A mutex or similar mechanism could be used for this. I myself use lockfiles to have a solution that works on many platforms and doesn't rely on a specific library only available on Linux etc.
For that, I have written a small Lock class. Do note that it uses some non-standard functions from my library, for instance, to get where to store temporary files etc. But you could easily change that.
<?php
class Lock
{
private $_owned = false;
private $_name = null;
private $_lockFile = null;
private $_lockFilePointer = null;
public function __construct($name)
{
$this->_name = $name;
$this->_lockFile = PluginManager::getInstance()->getCorePlugin()->getTempDir('locks') . $name . '-' . sha1($name . PluginManager::getInstance()->getCorePlugin()->getPreference('EncryptionKey')->getValue()).'.lock';
}
public function __destruct()
{
$this->release();
}
/**
* Acquires a lock
*
* Returns true on success and false on failure.
* Could be told to wait (block) and if so for a max amount of seconds or return false right away.
*
* #param bool $wait
* #param null $maxWaitTime
* #return bool
* #throws \Exception
*/
public function acquire($wait = false, $maxWaitTime = null) {
$this->_lockFilePointer = fopen($this->_lockFile, 'c');
if(!$this->_lockFilePointer) {
throw new \RuntimeException(__('Unable to create lock file', 'dliCore'));
}
if($wait && $maxWaitTime === null) {
$flags = LOCK_EX;
}
else {
$flags = LOCK_EX | LOCK_NB;
}
$startTime = time();
while(1) {
if (flock($this->_lockFilePointer, $flags)) {
$this->_owned = true;
return true;
} else {
if($maxWaitTime === null || time() - $startTime > $maxWaitTime) {
fclose($this->_lockFilePointer);
return false;
}
sleep(1);
}
}
}
/**
* Releases the lock
*/
public function release()
{
if($this->_owned) {
#flock($this->_lockFilePointer, LOCK_UN);
#fclose($this->_lockFilePointer);
#unlink($this->_lockFile);
$this->_owned = false;
}
}
}
Usage
Now you can have two process that run at the same time and execute the same script
Process 1
$lock = new Lock('runExpensiveFunction');
if($lock->acquire()) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Process 2
$lock = new Lock('runExpensiveFunction');
// Check will be false since the lock will already be held by someone else so the function is skipped
if($lock->acquire()) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Another alternative would be to have the second process wait for the first one to finish instead of skipping the code.
$lock = new Lock('runExpensiveFunction');
// Process will now wait for the lock to become available. A max wait time can be set if needed.
if($lock->acquire(true)) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Ram disk
To limit the number of writes to your HDD/SSD with the lockfiles you could crate a RAM disk to store them in.
On Linux you could add something like the following to /etc/fstab
tmpfs /mnt/ramdisk tmpfs nodev,nosuid,noexec,nodiratime,size=1024M 0 0
On Windows you can download something like ImDisk Toolkit and create a ramdisk with that.
I have a Wordpress plugin that I created that simply exports orders to a 3rd party system. To prevent any possible issues of the plugin running more than once at the same time, I am using the following Mutex code, however on occasion, the mutex file does not remove which stops my plugin from running until I manually remove the file.
<?php
class System_Mutex
{
var $lockName = "";
var $fileName = null;
var $file = null;
public function __construct($lockName) {
$this->lockName = preg_replace('/[^a-z0-9]/', '', $lockName);
$this->getFileName();
}
public function __destruct() {
$this->releaseLock();
}
public function isLocked() {
return ! flock($this->file, LOCK_SH | LOCK_NB);
}
public function getLock() {
return flock($this->file, LOCK_EX | LOCK_NB);
}
public function releaseLock() {
if ( ! is_resource($this->file) ) return true;
$success = flock($this->file, LOCK_UN);
fclose($this->file);
return $success;
}
public function getFileName() {
$this->fileName = dirname(__FILE__) . "/../Locks/" . $this->lockName . ".lock";
if ( ! $this->file = fopen($this->fileName, "c") ) {
throw new Exception("Cannot create temporary lock file.");
}
}
}
The Mutex itself is used like this:
try {
$mutex_id = "ef_stock_sync";
$mutex = new System_Mutex($mutex_id);
//mutex is locked- exit process
if ( $mutex->isLocked() || ! $mutex->getLock() ) {
//
return;
}
} catch ( Exception $e ) {
//
return;
}
$this->_syncStock();
$mutex->releaseLock();
Any idea why this would be happening? I thought that the destructor of the class would ensure it is removed even if the code was to stop halfway?
Too broad to answer with certainty and the block is most likely triggered from your plugin code. However, you do have a logical error in your System_Mutex class.
You are acquiring an exclusive lock in getLock(), but isLocked() attempts to acquire a shared lock as the check. You shouldn't do that for two reasons:
A shared lock may be held by multiple processes simultaneously (hence its name).
A shared lock is not prevented by an exclusive lock acquired by the same process.
I'm not sure what you're using isLocked() for, but because of the above 2 rules, regardless of the purpose, you may get false positives. What I can tell you is this:
Don't mix the lock types.
Be careful with your lock check, because that does acquire a lock on its own.
In this particular case, use only LOCK_EX.
And also this: https://bugs.php.net/bug.php?id=53769
Im trying to implement a cache for a high traffic wp site in php. so far ive managed to store the results to a ramfs and load them directly from the htaccess. however during peak hours there are mora than one process generatin certain page and is becoming an issue
i was thinking that a mutex would help and i was wondering if there is a better way than system("mkdir cache.mutex")
From what I understand you want to make sure only a single process at a time is running a certain piece of code. A mutex or similar mechanism could be used for this. I myself use lockfiles to have a solution that works on many platforms and doesn't rely on a specific library only available on Linux etc.
For that, I have written a small Lock class. Do note that it uses some non-standard functions from my library, for instance, to get where to store temporary files etc. But you could easily change that.
<?php
class Lock
{
private $_owned = false;
private $_name = null;
private $_lockFile = null;
private $_lockFilePointer = null;
public function __construct($name)
{
$this->_name = $name;
$this->_lockFile = PluginManager::getInstance()->getCorePlugin()->getTempDir('locks') . $name . '-' . sha1($name . PluginManager::getInstance()->getCorePlugin()->getPreference('EncryptionKey')->getValue()).'.lock';
}
public function __destruct()
{
$this->release();
}
/**
* Acquires a lock
*
* Returns true on success and false on failure.
* Could be told to wait (block) and if so for a max amount of seconds or return false right away.
*
* #param bool $wait
* #param null $maxWaitTime
* #return bool
* #throws \Exception
*/
public function acquire($wait = false, $maxWaitTime = null) {
$this->_lockFilePointer = fopen($this->_lockFile, 'c');
if(!$this->_lockFilePointer) {
throw new \RuntimeException(__('Unable to create lock file', 'dliCore'));
}
if($wait && $maxWaitTime === null) {
$flags = LOCK_EX;
}
else {
$flags = LOCK_EX | LOCK_NB;
}
$startTime = time();
while(1) {
if (flock($this->_lockFilePointer, $flags)) {
$this->_owned = true;
return true;
} else {
if($maxWaitTime === null || time() - $startTime > $maxWaitTime) {
fclose($this->_lockFilePointer);
return false;
}
sleep(1);
}
}
}
/**
* Releases the lock
*/
public function release()
{
if($this->_owned) {
#flock($this->_lockFilePointer, LOCK_UN);
#fclose($this->_lockFilePointer);
#unlink($this->_lockFile);
$this->_owned = false;
}
}
}
Usage
Now you can have two process that run at the same time and execute the same script
Process 1
$lock = new Lock('runExpensiveFunction');
if($lock->acquire()) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Process 2
$lock = new Lock('runExpensiveFunction');
// Check will be false since the lock will already be held by someone else so the function is skipped
if($lock->acquire()) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Another alternative would be to have the second process wait for the first one to finish instead of skipping the code.
$lock = new Lock('runExpensiveFunction');
// Process will now wait for the lock to become available. A max wait time can be set if needed.
if($lock->acquire(true)) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Ram disk
To limit the number of writes to your HDD/SSD with the lockfiles you could create a RAM disk to store them in.
On Linux you could add something like the following to /etc/fstab
tmpfs /mnt/ramdisk tmpfs nodev,nosuid,noexec,nodiratime,size=1024M 0 0
On Windows you can download something like ImDisk Toolkit and create a ramdisk with that.
I agree with #gries, a reverse proxy is going to be a really good bang-for-the-buck way to get high performance out of a high-volume Wordpress site. I've leveraged Varnish with quite a lot of success, though I suspect you can do so with nginx as well.
I need a way to prevent multiple run on a process on symfony 1.4.
Something like: when a user is running this process, the other user who tries to run this process will get a warning message inform that the process is running.
Is there a way to implement it without using database?
kirugan's method will probably work in most cases, but it's susceptible to race conditions and can get stuck in the event of a crash.
Here's a more robust solution. It uses PHP's file locking so you know the lock is atomic, and if you forget to release the lock later or your process crashes, it gets released automatically. By default, getting a lock is non-blocking (i.e. if the lock is already held by another process, getLock() will instantly return FALSE). However, you can have the call block (i.e. wait until the lock becomes available) if you'd like. Finally, you can have different locks for different parts of your code. Just use a different name for the lock.
The only requirement is that the directory returned by getLockDir() must be server-writable. Feel free to change the location of the lock dir.
Note: I think flock() may behave differently on Windows (I use linux), so double check that if its an issue for you.
myLock.class.php
class myLock
{
/**
* Creates a lockfile and acquires an exclusive lock on it.
*
* #param string $filename The name of the lockfile.
* #param boolean $blocking Block until lock becomes available (default: don't block, just fail)
* #return mixed Returns the lockfile, or FALSE if a lock could not be acquired.
*/
public static function getLock($name, $blocking = false)
{
$filename = static::getLockDir() . '/' . $name;
if (!preg_match('/\.lo?ck$/', $filename))
{
$filename .= '.lck';
}
if (!file_exists($filename))
{
file_put_contents($filename, '');
chmod($filename, 0777); // if the file cant be opened for writing later, getting the lock will fail
}
$lockFile = fopen($filename, 'w+');
if (!flock($lockFile, $blocking ? LOCK_EX : LOCK_EX|LOCK_NB))
{
fclose($lockFile);
return false;
}
return $lockFile;
}
/**
* Free a lock.
*
* #param resource $lockFile
*/
public static function freeLock($lockFile)
{
if ($lockFile)
{
flock($lockFile, LOCK_UN);
fclose($lockFile);
}
}
public static function getLockDir()
{
return sfConfig::get('sf_root_dir') . '/data/lock';
}
}
How to use
$lockFile = myLock::getLock('lock-name');
if ($lockFile) {
// you have a lock here, do whatever you want
myLock::freeLock($lockFile);
}
else {
// you could not get the lock. show a message or throw an exception or whatever
}
Here is my functions for this:
function lock(){
$file = __DIR__ . '/file.lock';
if(file_exists($file)){
/* exit or whatever you want */
die('ALREADY LOCKED');
}
touch($file);
}
function unlock(){
$file = __DIR__ . '/parser.lock';
if(file_exists($file)){
unlink($file);
}else{
echoln("unlock function: LOCK FILE NOT FOUND");
}
}
function exitHandler(){
echoln('exitHandler function: called');
unlock();
}
Use lock function in the beginning, and set exitHandler function in register_shutdown_function(). You can save this snippets as class with the same methods or save it like helper file