I am building a log parser program in PHP. Log parser reads the data from the log file created by ProFTPD and then runs some actions if it detects specific commands. To be able to detect changes in the log file, I am using Inotify. If log file gets too large, I want to rotate the log by sending a signal to the log parser to finish processing the current file and then terminate the log parser. Logrotate would then restart the log parser again after it makes sure that the original file that is being read is emptied.
The problem is that when I use Inotify and when the inotify is in blocking state, the interrupts won't work.
For example:
#!/usr/bin/php -q
<?php
declare(ticks = 1);
$log_parser = new LogParser();
$log_parser->process_log();
class LogParser {
private $ftp_log_file = '/var/log/proftpd/proftpd.log';
# file descriptor for the log file
private $fd;
# inotify instance
private $inotify_inst;
# watch id for the inotifier
private $watch_id;
public function process_log() {
// Open an inotify instance
$this->inotify_inst = inotify_init();
$this->watch_id = inotify_add_watch($this->inotify_inst, $this->ftp_log_file, IN_MODIFY);
$this->fd = fopen($this->ftp_log_file, 'r');
if ($this->fd === false)
die("unable to open $this->ftp_log_file!\n");
pcntl_signal(SIGUSR1, function($signal) {
$this->sig_handler($signal);
});
while (1) {
# If the thread gets blocked here, the signals do not work
$events = inotify_read($this->inotify_inst);
while ($line = trim(fgets($this->fd))) {
// Parse the log ...
}
}
fclose($this->fd);
// stop watching our directory
inotify_rm_watch($this->inotify_inst, $this->watch_id);
// close our inotify instance
fclose($this->inotify_inst);
}
private function sig_handler($signo) {
switch ($signo) {
case SIGUSR1:
// Do some action ...
}
}
}
I know that one solution could be that I start the parent process and then add the signal handler to that parent process. The parent process should start the log parser and the log parser would get blocked by inotify_read, but parent process wouldn't, but was wondering if there is a solution not involving the parent process - if the inotify is able to support interrupts?
Thanks
Found a solution here: php inotify blocking but with timeout
Final code:
#!/usr/bin/php -q
<?php
declare(ticks = 1);
$log_parser = new LogParser();
$log_parser->process_log();
class LogParser {
private $ftp_log_file = '/var/log/proftpd/proftpd.log';
# file descriptor for the log file
private $fd;
# inotify instance
private $inotify_inst;
# watch id for the inotifier
private $watch_id;
public function process_log() {
// Open an inotify instance
$this->inotify_inst = inotify_init();
stream_set_blocking($this->inotify_inst, false);
$this->watch_id = inotify_add_watch($this->inotify_inst, $this->ftp_log_file, IN_MODIFY);
$this->fd = fopen($this->ftp_log_file, 'r');
if ($this->fd === false)
die("unable to open $this->ftp_log_file!\n");
pcntl_signal(SIGUSR1, function($signal) {
$this->sig_handler($signal);
});
while (1) {
while (1) {
$r = array($this->inotify_inst);
$timeout = 60;
$w = array();
$e = array();
$time_left = stream_select($r, $w, $e, $timeout);
if ($time_left != 0) {
$events = inotify_read($this->inotify_inst);
if ($events) {
break;
}
}
}
while ($line = trim(fgets($this->fd))) {
// Parse the log ...
}
}
fclose($this->fd);
// stop watching our directory
inotify_rm_watch($this->inotify_inst, $this->watch_id);
// close our inotify instance
fclose($this->inotify_inst);
}
private function sig_handler($signo) {
switch ($signo) {
case SIGUSR1:
// Do some action ...
}
}
}
The suggested solution does not block interrupts and it also sets the thread in a non blocking state.
Related
I am currently running a queue system with beanstalk + supervisor + PHP.
I would like my workers to automatically die when a new version is available (basically code update).
My current code is as follow
class Job1Controller extends Controller
{
public $currentVersion = 5;
public function actionIndex()
{
while (true) {
// check if a new version of the worker is available
$file = '/config/params.php';
$paramsContent = file_get_contents($file);
$params = eval('?>' . file_get_contents($file));
if ($params['Job1Version'] != $this->currentVersion) {
echo "not the same version, exit worker \n";
sleep(2);
exit();
} else {
echo "same version, continue processing \n";
}
}
}
}
When I will update the code, the params file will change with a new version number which will force the worker to terminate. I cannot use include as the file will be loaded in memory in the while loop. Knowing that the file params.php isn't critical in terms of security I wanted to know if there was another way of doing so?
Edit: the params.php looks as follow:
<?php
return [
'Job1Version' => 5
];
$params = require($file);
Since your file has a return statement, the returned value will be passed along.
After few tests I finally managed to find a solution which doesn't require versionning anymore.
$reflectionClass = new \ReflectionClass($this);
$lastUpdatedTimeOnStart = filemtime($reflectionClass->getFileName());
while (true) {
clearstatcache();
$reflectionClass = new \ReflectionClass($this);
$lastUpdatedTime = filemtime($reflectionClass->getFileName());
if ($lastUpdatedTime != $lastUpdatedTimeOnStart) {
// An update has been made, exit
} else {
// worker hasn't been modified since running
}
}
Whenever the file will be updated, the worker will automatically exit
Thanks to #Rudie who pointed me into the right direction.
I have a Wordpress plugin that I created that simply exports orders to a 3rd party system. To prevent any possible issues of the plugin running more than once at the same time, I am using the following Mutex code, however on occasion, the mutex file does not remove which stops my plugin from running until I manually remove the file.
<?php
class System_Mutex
{
var $lockName = "";
var $fileName = null;
var $file = null;
public function __construct($lockName) {
$this->lockName = preg_replace('/[^a-z0-9]/', '', $lockName);
$this->getFileName();
}
public function __destruct() {
$this->releaseLock();
}
public function isLocked() {
return ! flock($this->file, LOCK_SH | LOCK_NB);
}
public function getLock() {
return flock($this->file, LOCK_EX | LOCK_NB);
}
public function releaseLock() {
if ( ! is_resource($this->file) ) return true;
$success = flock($this->file, LOCK_UN);
fclose($this->file);
return $success;
}
public function getFileName() {
$this->fileName = dirname(__FILE__) . "/../Locks/" . $this->lockName . ".lock";
if ( ! $this->file = fopen($this->fileName, "c") ) {
throw new Exception("Cannot create temporary lock file.");
}
}
}
The Mutex itself is used like this:
try {
$mutex_id = "ef_stock_sync";
$mutex = new System_Mutex($mutex_id);
//mutex is locked- exit process
if ( $mutex->isLocked() || ! $mutex->getLock() ) {
//
return;
}
} catch ( Exception $e ) {
//
return;
}
$this->_syncStock();
$mutex->releaseLock();
Any idea why this would be happening? I thought that the destructor of the class would ensure it is removed even if the code was to stop halfway?
Too broad to answer with certainty and the block is most likely triggered from your plugin code. However, you do have a logical error in your System_Mutex class.
You are acquiring an exclusive lock in getLock(), but isLocked() attempts to acquire a shared lock as the check. You shouldn't do that for two reasons:
A shared lock may be held by multiple processes simultaneously (hence its name).
A shared lock is not prevented by an exclusive lock acquired by the same process.
I'm not sure what you're using isLocked() for, but because of the above 2 rules, regardless of the purpose, you may get false positives. What I can tell you is this:
Don't mix the lock types.
Be careful with your lock check, because that does acquire a lock on its own.
In this particular case, use only LOCK_EX.
And also this: https://bugs.php.net/bug.php?id=53769
I'm trying to create a script that process a number of files simultanously, the rule is, each file can only be processed once, and the input file is deleted after it has been processed. I created this script :
<?php
// Libraries for reading files
require_once "spooler.php";
// Configuration section ///////////////////////////////////////////////////////
$config["data"] = "data";
$config["threads"] = 20;
$config["timer"] = 1;
// Array to store currently processed files
$config["processed_files"] = array();
// Processing section //////////////////////////////////////////////////////////
$timer = 0;
$pool = new Pool($config["threads"], \ProcessingWorker::class);
while (true) {
// Read a number of files from the data folder according to the number of thread
$files = Spooler::read_spool_file($config["data"], $config["threads"]);
foreach ($files as $file) {
// Check if the file is already processed
if (in_array($file, $config["processed_files"])) continue;
// Submit the file to the worker
echo "Submitting $file\n";
$config["processed_files"][$file] = $file;
$pool->submit(new ProcessingJob($config, $file));
}
sleep($config["timer"]);
$timer++;
}
$pool->shutdown();
// Processing thread section ///////////////////////////////////////////////////
class ProcessingJob extends Stackable {
private $config;
private $file;
public function __construct($config, $file)
{
$this->config = $config;
$this->file = $file;
$this->complete = false;
}
public function run()
{
echo "Processing $this->file\n";
// Pretend we're doing something that takes time
sleep(mt_rand(1, 10));
file_put_contents("_LOG", $this->file."\n", FILE_APPEND);
// Delete the file
#unlink($this->file);
// Remove the file from the currently processing list
unset($this->config["processed_files"][$this->file]);
}
}
class ProcessingWorker extends Worker {
public function run() {}
}
However, this code doesn't work well, it doesn't process the same files twice, but instead sometimes it skip processing some files. Here's the file list it should be processed, but it only process these files.
Where am I doing it wrong?
Output to the log file isn't synchronized, it's highly likely that two threads are concurrently calling file_put_contents on the log file and so corrupting it's output.
You should not write to a log file in this way.
If $config['processed_files'] is intended to be manipulated by multiple contexts then it should be a thread safe structure descended from pthreads, not a plain PHP array.
Is there a way you can abort a block of code if it's taking too long in PHP? Perhaps something like:
//Set the max time to 2 seconds
$time = new TimeOut(2);
$time->startTime();
sleep(3)
$time->endTime();
if ($time->timeExpired()){
echo 'This function took too long to execute and was aborted.';
}
It doesn't have to be exactly like above, but are there any native PHP functions or classes that do something like this?
Edit: Ben Lee's answer with pcnt_fork would be the perfect solution except that it's not available for Windows. Is there any other way to accomplish this with PHP that works for Windows and Linux, but doesn't require an external library?
Edit 2: XzKto's solution works in some cases, but not consistently and I can't seem to catch the exception, no matter what I try. The use case is detecting a timeout for a unit test. If the test times out, I want to terminate it and then move on to the next test.
You can do this by forking the process, and then using the parent process to monitor the child process. pcntl_fork is a method that forks the process, so you have two nearly identical programs in memory running in parallel. The only difference is that in one process, the parent, pcntl_fork returns a positive integer which corresponds to the process id of the child process. And in the other process, the child, pcntl_fork returns 0.
Here's an example:
$pid = pcntl_fork();
if ($pid == 0) {
// this is the child process
} else {
// this is the parent process, and we know the child process id is in $pid
}
That's the basic structure. Next step is to add a process expiration. Your stuff will run in the child process, and the parent process will be responsible only for monitoring and timing the child process. But in order for one process (the parent) to kill another (the child), there needs to be a signal. Signals are how processes communicate, and the signal that means "you should end immediately" is SIGKILL. You can send this signal using posix_kill. So the parent should just wait 2 seconds then kill the child, like so:
$pid = pcntl_fork();
if ($pid == 0) {
// this is the child process
// run your potentially time-consuming method
} else {
// this is the parent process, and we know the child process id is in $pid
sleep(2); // wait 2 seconds
posix_kill($pid, SIGKILL); // then kill the child
}
You can't really do that if you script pauses on one command (for example sleep()) besides forking, but there are a lot of work arounds for special cases: like asynchronous queries if you programm pauses on DB query, proc_open if you programm pauses at some external execution etc. Unfortunately they are all different so there is no general solution.
If you script waits for a long loop/many lines of code you can do a dirty trick like this:
declare(ticks=1);
class Timouter {
private static $start_time = false,
$timeout;
public static function start($timeout) {
self::$start_time = microtime(true);
self::$timeout = (float) $timeout;
register_tick_function(array('Timouter', 'tick'));
}
public static function end() {
unregister_tick_function(array('Timouter', 'tick'));
}
public static function tick() {
if ((microtime(true) - self::$start_time) > self::$timeout)
throw new Exception;
}
}
//Main code
try {
//Start timeout
Timouter::start(3);
//Some long code to execute that you want to set timeout for.
while (1);
} catch (Exception $e) {
Timouter::end();
echo "Timeouted!";
}
but I don't think it is very good. If you specify the exact case I think we can help you better.
This is an old question, and has probably been solved many times by now, but for people looking for an easy way to solve this problem, there is a library now: PHP Invoker.
You can use declare function if the execution time exceeds the limits. http://www.php.net/manual/en/control-structures.declare.php
Here a code example of how to use
define("MAX_EXECUTION_TIME", 2); # seconds
$timeline = time() + MAX_EXECUTION_TIME;
function check_timeout()
{
if( time() < $GLOBALS['timeline'] ) return;
# timeout reached:
print "Timeout!".PHP_EOL;
exit;
}
register_tick_function("check_timeout");
$data = "";
declare( ticks=1 ){
# here the process that might require long execution time
sleep(5); // Comment this line to see this data text
$data = "Long process result".PHP_EOL;
}
# Ok, process completed, output the result:
print $data;
With this code you will see the timeout message.
If you want to get the Long process result inside the declare block you can just remove the sleep(5) line or increase the Max Execution Time declared at the start of the script
What about set-time-limit if you are not in the safe mode.
Cooked this up in about two minutes, I forgot to call $time->startTime(); so I don't really know exactly how long it took ;)
class TimeOut{
public function __construct($time=0)
{
$this->limit = $time;
}
public function startTime()
{
$this->old = microtime(true);
}
public function checkTime()
{
$this->new = microtime(true);
}
public function timeExpired()
{
$this->checkTime();
return ($this->new - $this->old > $this->limit);
}
}
And the demo.
I don't really get what your endTime() call does, so I made checkTime() instead, which also serves no real purpose but to update the internal values. timeExpired() calls it automatically because it would sure stink if you forgot to call checkTime() and it was using the old times.
You can also use a 2nd script that has the pause code in it that is executed via a curl call with a timeout set. The other obvious solution is to fix the cause of the pause.
Here is my way to do that. Thanks to others answers:
<?php
class Timeouter
{
private static $start_time = FALSE, $timeout;
/**
* #param integer $seconds Time in seconds
* #param null $error_msg
*/
public static function limit($seconds, $error_msg = NULL)
: void
{
self::$start_time = microtime(TRUE);
self::$timeout = (float) $seconds;
register_tick_function([ self::class, 'tick' ], $error_msg);
}
public static function end()
: void
{
unregister_tick_function([ self::class, 'tick' ]);
}
public static function tick($error)
: void
{
if ((microtime(TRUE) - self::$start_time) > self::$timeout) {
throw new \RuntimeException($error ?? 'You code took too much time.');
}
}
public static function step()
: void
{
usleep(1);
}
}
Then you can try like this:
<?php
try {
//Start timeout
Timeouter::limit(2, 'You code is heavy. Sorry.');
//Some long code to execute that you want to set timeout for.
declare(ticks=1) {
foreach (range(1, 100000) as $x) {
Timeouter::step(); // Not always necessary
echo $x . "-";
}
}
Timeouter::end();
} catch (Exception $e) {
Timeouter::end();
echo $e->getMessage(); // 'You code is heavy. Sorry.'
}
I made a script in php using pcntl_fork and lockfile to control the execution of external calls doing the kill after the timeout.
#!/usr/bin/env php
<?php
if(count($argv)<4){
print "\n\n\n";
print "./fork.php PATH \"COMMAND\" TIMEOUT\n"; // TIMEOUT IN SECS
print "Example:\n";
print "./fork.php /root/ \"php run.php\" 20";
print "\n\n\n";
die;
}
$PATH = $argv[1];
$LOCKFILE = $argv[1].$argv[2].".lock";
$TIMEOUT = (int)$argv[3];
$RUN = $argv[2];
chdir($PATH);
$fp = fopen($LOCKFILE,"w");
if (!flock($fp, LOCK_EX | LOCK_NB)) {
print "Already Running\n";
exit();
}
$tasks = [
"kill",
"run",
];
function killChilds($pid,$signal) {
exec("ps -ef| awk '\$3 == '$pid' { print \$2 }'", $output, $ret);
if($ret) return 'you need ps, grep, and awk';
while(list(,$t) = each($output)) {
if ( $t != $pid && $t != posix_getpid()) {
posix_kill($t, $signal);
}
}
}
$pidmaster = getmypid();
print "Add PID: ".(string)$pidmaster." MASTER\n";
foreach ($tasks as $task) {
$pid = pcntl_fork();
$pidslave = posix_getpid();
if($pidslave != $pidmaster){
print "Add PID: ".(string)$pidslave." ".strtoupper($task)."\n";
}
if ($pid == -1) {
exit("Error forking...\n");
}
else if ($pid == 0) {
execute_task($task);
exit();
}
}
while(pcntl_waitpid(0, $status) != -1);
echo "Do stuff after all parallel execution is complete.\n";
unlink($LOCKFILE);
function execute_task($task_id) {
global $pidmaster;
global $TIMEOUT;
global $RUN;
if($task_id=='kill'){
print("SET TIMEOUT = ". (string)$TIMEOUT."\n");
sleep($TIMEOUT);
print("FINISHED BY TIMEOUT: ". (string)$TIMEOUT."\n");
killChilds($pidmaster,SIGTERM);
die;
}elseif($task_id=='run'){
###############################################
### START EXECUTION CODE OR EXTERNAL SCRIPT ###
###############################################
system($RUN);
################################
### END ###
################################
killChilds($pidmaster,SIGTERM);
die;
}
}
Test Script run.php
<?php
$i=0;
while($i<25){
print "test... $i\n";
$i++;
sleep(1);
}
I'm trying to find a safe way to prevent a cron job collision (ie. prevent it from running if another instance is already running).
Some options I've found recommend using a lock on a file.
Is that really a safe option? What would happen if the script dies for example? Will the lock remain?
Are there other ways of doing this?
This sample was taken at http://php.net/flock and changed a little and this is a correct way to do what you want:
$fp = fopen("/path/to/lock/file", "w+");
if (flock($fp, LOCK_EX | LOCK_NB)) { // do an exclusive lock
// do the work
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
fclose($fp);
Do not use locations such as /tmp or /var/tmp as they could be cleaned up at any time by your system, thus messing with your lock as per the docs:
Programs must not assume that any files or directories in /tmp are preserved between invocations of the program.
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.html
Do use a location that is under your control.
Credits:
Michaƫl Perrin - for proposing to use w+ instead of r+
In Symfony Framework you could use the lock component symfony/lock
https://symfony.com/doc/current/console/lockable_trait.html
I've extended the concept from zerkms to create a function that can be called from the start of a cron.
Using the Cronlocker you specify a lock name, then the name of a callback function to be called if the cron is OFF. Optionally you may give an array of parameters to pass to the callback function. There's also an optional callback function if you need to do something different if the lock is ON.
In some cases I got a few exceptions and wanted to be able to trap them, and I added a function for handling fatal exceptions, which should be added. I wanted to be able to hit the file from a browser and bypass the cronlock, so that's built in.
I found as I used this a lot there were cases where I wanted to block other crons from running while this cron is running, so I added an optional array of lockblocks, which are other lock names to block.
Then there were cases where I wanted this cron to run after other crons had finished, so there's an optional array of lockwaits, which are other lock names to wait until none of which are running.
simple example:
Cronlocker::CronLock('cron1', 'RunThis');
function RunThis() {
echo('I ran!');
}
callback parameters and failure functions:
Cronlocker::CronLock('cron2', 'RunThat', ['ran'], 'ImLocked');
function RunThat($x) {
echo('I also ran! ' . $x);
}
function ImLocked($x) {
echo('I am locked :-( ' . $x);
}
blocking and waiting:
Cronlocker::CronLock('cron3', 'RunAgain', null, null, ['cron1'], ['cron2']);
function RunAgain() {
echo('I ran.<br />');
echo('I block cron1 while I am running.<br />')
echo('I wait for cron2 to finish if it is running.');
}
class:
class Cronlocker {
private static $LockFile = null;
private static $LockFileBlocks = [];
private static $LockFileWait = null;
private static function GetLockfileName($lockname) {
return "/tmp/lock-" . $lockname . ".txt";
}
/**
* Locks a PHP script from being executed more than once at a time
* #param string $lockname Use a unique lock name for each lock that needs to be applied.
* #param string $callback The name of the function to call if the lock is OFF
* #param array $callbackParams Optional array of parameters to apply to the callback function when called
* #param string $callbackFail Optional name of the function to call if the lock is ON
* #param string[] $lockblocks Optional array of locknames for other crons to also block while this cron is running
* #param string[] $lockwaits Optional array of locknames for other crons to wait until they finish running before this cron will run
* #see http://stackoverflow.com/questions/5428631/php-preventing-collision-in-cron-file-lock-safe
*/
public static function CronLock($lockname, $callback, $callbackParams = null, $callbackFail = null, $lockblocks = [], $lockwaits = []) {
// check all the crons we are waiting for to finish running
if (!empty($lockwaits)) {
$waitingOnCron = true;
while ($waitingOnCron) {
$waitingOnCron = false;
foreach ($lockwaits as $lockwait) {
self::$LockFileWait = null;
$tempfile = self::GetLockfileName($lockwait);
try {
self::$LockFileWait = fopen($tempfile, "w+");
} catch (Exception $e) {
//ignore error
}
if (flock(self::$LockFileWait, LOCK_EX | LOCK_NB)) { // do an exclusive lock
// cron we're waiting on isn't running
flock(self::$LockFileWait, LOCK_UN); // release the lock
} else {
// we're wating on a cron
$waitingOnCron = true;
}
if (is_resource(self::$LockFileWait))
fclose(self::$LockFileWait);
if ($waitingOnCron) break; // no need to check any more
}
if ($waitingOnCron) sleep(15); // wait a few seconds
}
}
// block any additional crons from starting
if (!empty($lockblocks)) {
self::$LockFileBlocks = [];
foreach ($lockblocks as $lockblock) {
$tempfile = self::GetLockfileName($lockblock);
try {
$block = fopen($tempfile, "w+");
} catch (Exception $e) {
//ignore error
}
if (flock($block, LOCK_EX | LOCK_NB)) { // do an exclusive lock
// lock made
self::$LockFileBlocks[] = $block;
} else {
// couldn't lock it, we ignore and move on
}
}
}
// set the cronlock
self::$LockFile = null;
$tempfile = self::GetLockfileName($lockname);
$return = null;
try {
if (file_exists($tempfile) && !is_writable($tempfile)) {
//assume we're hitting this from a browser and execute it regardless of the cronlock
if (empty($callbackParams))
$return = $callback();
else
$return = call_user_func_array($callback, $callbackParams);
} else {
self::$LockFile = fopen($tempfile, "w+");
}
} catch (Exception $e) {
//ignore error
}
if (!empty(self::$LockFile)) {
if (flock(self::$LockFile, LOCK_EX | LOCK_NB)) { // do an exclusive lock
// do the work
if (empty($callbackParams))
$return = $callback();
else
$return = call_user_func_array($callback, $callbackParams);
flock(self::$LockFile, LOCK_UN); // release the lock
} else {
// call the failed function
if (!empty($callbackFail)) {
if (empty($callbackParams))
$return = $callbackFail();
else
$return = call_user_func_array($callbackFail, $callbackParams);
}
}
if (is_resource(self::$LockFile))
fclose(self::$LockFile);
}
// remove any lockblocks
if (!empty($lockblocks)) {
foreach (self::$LockFileBlocks as $LockFileBlock) {
flock($LockFileBlock, LOCK_UN); // release the lock
if (is_resource($LockFileBlock))
fclose($LockFileBlock);
}
}
return $return;
}
/**
* Releases the Cron Lock locking file, useful to specify on fatal errors
*/
public static function ReleaseCronLock() {
// release the cronlock
if (!empty(self::$LockFile) && is_resource(self::$LockFile)) {
var_dump('Cronlock released after error encountered: ' . self::$LockFile);
flock(self::$LockFile, LOCK_UN);
fclose(self::$LockFile);
}
// release any lockblocks too
foreach (self::$LockFileBlocks as $LockFileBlock) {
if (!empty($LockFileBlock) && is_resource($LockFileBlock)) {
flock($LockFileBlock, LOCK_UN);
fclose($LockFileBlock);
}
}
}
}
Should also be implemented on a common page, or built into your existing fatal error handler:
function fatal_handler() {
// For cleaning up crons that fail
Cronlocker::ReleaseCronLock();
}
register_shutdown_function("fatal_handler");