I have a script i use that checks an IP address stored within my hosts.allow file against what IP is mapped to my dyndns hostname so i can log into my servers once i've synced my current IP to that hostname. For some reason though the script seems to cause really intermittent issues.
within my hosts.allow file i have a section like this:
#SOme.gotdns.com
sshd : 192.168.0.1
#EOme.gotdns.com
#SOme2.gotdns.com
sshd : 192.168.0.2
#EOme2.gotdns.com
I have a script running on a cron (every minute) that looks like this:
#!/usr/bin/php
<?php
$hosts = array('me.gotdns.com','me2.gotdns.com');
foreach($hosts as $host)
{
$ip = gethostbyname($host);
$replaceWith = "#SO".$host."\nsshd : ".$ip."\n#EO".$host;
$filename = '/etc/hosts.allow';
$handle = fopen($filename,'r');
$contents = fread($handle, filesize($filename));
fclose($handle);
if (preg_match('/#SO'.$host.'(.*?)#EO'.$host.'/si', $contents, $regs))
{
$result = $regs[0];
}
if($result != $replaceWith)
{
$newcontents = str_replace($result,$replaceWith,$contents);
$handle = fopen($filename,'w');
if (fwrite($handle, $newcontents) === FALSE) {
}
fclose($handle);
}
}
?>
The problem i have is that intermittently characters are being dropped (i assume during the replace) that causes future updates to fail as it inserts something like:
#SOme.gotdns.com
sshd : 192.168.0.1
#EOme.gotdn
note the missing "s.com"
This of course means i lose access to the server, any ideas why this would be happening?
Thanks.
that might be because of script execution time - can be too short- OR 1 min interval is too short. While cron is doing the job, another process of script starts and it may effect the first one.
This is almost certainly because the script hasn't finished executing within the one minute time period before it's started again via cron. You need to implement some sort of locking, or use a tool that only allows once instance of the script to be run. There are several tools available out there that can do this, for example lockrun.
I would say that in order to do this safely, you should acquire an exclusive lock on the file at the beginning of the script, read it all into memory once, modify it in memory, then write it back to the file at the end. This would also be considerably more efficient in terms of disk I/O.
You should also alter the cron job to run less frequently. It is likely that the reason you currently have this problem is because two processes are running at the same time - by locking the file, if this is the case, you risk having the processes stack up waiting to acquire a lock. Setting it for every 5 minutes should be good enough - your IP shouldn't change that often!
So do this (FIXED):
#!/usr/bin/php
<?php
// Settings
$hosts = array(
'me.gotdns.com',
'me2.gotdns.com'
);
$filename = '/etc/hosts.allow';
// No time limit (shouldn't be necessary with CLI, but just in case)
set_time_limit(0);
// Open the file in read/write mode and lock it
// flock() should block until it gets a lock
if ((!$handle = fopen($filename, 'r+')) || !flock($handle, LOCK_EX)) exit(1);
// Read the file
if (($contents = fread($handle, filesize($filename)) === FALSE) exit(1);
// Will be set to true if we actually make any changes to the file
$changed = FALSE;
// Loop hosts list
foreach ($hosts as $host) {
// Get current IP address of host
if (($ip = gethostbyname($host)) == $host) continue;
// Find the entry in the file
$replaceWith = "#SO{$host}\nsshd : {$ip}\n#EO{$host}";
if (preg_match("/#SO{$host}(.*?)#EO{$host}/si", $contents, $regs)) {
// Only do this if there was a match - otherise risk overwriting previous
// entries because you didn't reset the value of $result
if ($regs[0] != $replaceWith) {
$changed = TRUE;
$contents = str_replace($regs[0], $replaceWith, $contents);
}
}
}
// We'll only change the contents of the file if the data changed
if ($changed) {
ftruncate($handle, 0); // Zero the length of the file
rewind($handle); // start writing from the beginning
fwrite($handle, $contents); // write the new data
}
flock($handle, LOCK_UN); // Unlock
fclose($handle); // close
Related
When I run this function on multiple scripts one script generated warning:
fread(): Length parameter must be greater than 0
function test($n){
echo "<h4>$n at ".time()."</h4>";
for ($i = 0; $i<50; $i++ ){
$fp = fopen("$n.txt", "r");
$s = fread($fp, filesize("$n.txt") );
fclose($fp);
$fp = fopen("$n.txt", "w");
$s = $_SERVER['HTTP_USER_AGENT'].' '.time();
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
fwrite($fp, $s);
// fflush($fp);// flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
}
}
I try to write reading of the file for multiple users, but only one user can write the file. I know that when I use fwrite with flock - LOC_EX, next scripts must wait till the write is finished. But here it seems like filesize doesn't wait till the write operation is finished. My opinion is that it tries to reach the file when the file size is 0, and as a result this produces the problem: 0 bytes will be read from the file, when it is written by original script.
Is it possible to fix this for fread function?
Purpose of this script is to test fread with some limit and to check the data which I read later, if the data are really written when I did not used fflush.
function test($n){
echo "<h4>$n at ".time()."</h4>";
for ($i = 0; $i<50; $i++ ){
$start = microtime(true);
$fp = fopen("$n.txt", "r");
if(filesize($n.txt) > 0)
{
$s = fread($fp, filesize($n.txt) );
fclose($fp);
$fp = fopen("$n.txt", "w");
$s = $_SERVER['HTTP_USER_AGENT'].' '.time();
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
fwrite($fp, $s);
// fflush($fp);// flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
}
else
{
echo "Filesize must be greater than 0";
}
}
}
please change $s variables name its use same things two time
$fp = fopen("$n.txt", "r");
$s = fread($fp, filesize("$n.txt") );
fclose($fp);
The error occurs in the middle line of the above three lines.
Firstly, these three lines could be rewritten into a single line as follows:
$s = file_get_contents("$n.txt");
However, these isn't necessary, as these three lines are entirely redundant in your code. They don't do anything useful.
What they do is open a file, store its contents to $s and then close it.
But you are then immediately setting $s to a different value, thus throwing away the previous value, and making it pointless to have read it from the file in the first place.
If you need to keep the original contents of the file, then use file_get_contents() and make sure you don't overwrite the contents of the variable.
If you don't need the original contents of the file, then just delete those three lines from your code.
Incidentally, this error highlights a couple of good coding practices that you should take on board: Firstly, never re-use a variable for two different things, and secondly always give your variables (and functions) good names. $s is not a good name; $previousFileContents would be a better name; it would have made the error much more obvious.
I'm working on a cron system and need to execute a script only once at a time. By using the following codes, I execute the script first time and while it's looping (for delaying purpose), executing it again but file_exists always returns false while first execution returns content of file after loop is done.
Cronjob.php:
include "Locker.class.php";
Locker::$LockName = __DIR__.'/OneTime_[cron].lock';
$Locker = new Locker();
for ($i = 0 ; $i < 1000000; $i++){
echo 'Z';
$z = true;
ob_end_flush();
ob_start();
}
Locker.class.php:
class Locker{
static $LockName;
function __construct($Expire){
if (!basename(static::$LockName)){
die('Locker: Not a filename.');
}
// It doesn't help
clearstatcache();
if (file_exists(static::$LockName)){ // returns false always
die('Already running');
} else {
$myfile = fopen(static::$LockName, "x"); // Tried with 'x' and 'w', no luck
fwrite($myfile, 'Keep it alive'); // Tried with file_put_content also, no luck
fclose($myfile);
}
// The following function returns true by the way!
// echo file_exists(static::$LockName);
}
function __destruct(){
// It outputs content
echo file_get_contents(static::$LockName);
unlink(static::$LockName);
}
}
What is the problem? Why file_exists returns false always?
I suspect the PHP parser has noticed that you never use the variable $Locker, so it immediately destroys the object, which runs the destructor and removes the file. Try putting a reference to the object after the loop:
include "Locker.class.php";
Locker::$LockName = __DIR__.'/OneTime_[cron].lock';
$Locker = new Locker();
for ($i = 0 ; $i < 1000000; $i++){
echo 'Z';
$z = true;
ob_end_flush();
ob_start();
}
var_dump($Locker);
If you're goal is to prevent a potentially long running job from executing multiple copies at the same time, you can take a simpler approach and just flock() the file itself.
This would go in cronjob.php
<?php
$wb = false;
$fp = fopen(__FILE__, 'r');
if (!$fp) die("Could not open file");
$locked = flock($fp, LOCK_EX | LOCK_NB, $wb);
if (!$locked) die("Couldn't acquire lock!\n");
// do work here
sleep(20);
flock($fp, LOCK_UN);
fclose($fp);
To address your actual question, I found that by running your code, the reason the file is going away is because on subsequent calls, it outputs Already running if a job is running, and then the second script invokes the destructor and deletes the file before the initial task finishes running.
The flock method above solves this problem. Otherwise, you'll need to ensure that only the process that actually creates the lock file is able to delete it (and take care that it never gets left around too long).
Let's consider a sample php script which deletes a line by user input:
$DELETE_LINE = $_GET['line'];
$out = array();
$data = #file("foo.txt");
if($data)
{
foreach($data as $line)
if(trim($line) != $DELETE_LINE)
$out[] = $line;
}
$fp = fopen("foo.txt", "w+");
flock($fp, LOCK_EX);
foreach($out as $line)
fwrite($fp, $line);
flock($fp, LOCK_UN);
fclose($fp);
I want to know if some user is currently executing this script and file "foo.txt" is locked, in same time or before completion of its execution, if some other user calls this script, then what will happen?
Will second users process wait for unlocking of file by first users? or line deletion by second users input will fail?
If you try to acquire an exclusive lock while another process has the file locked, your attempt will wait until the file is unlocked. This is the whole point of locking.
See the Linux documentation of flock(), which describes how it works in general across operating systems. PHP uses fcntl() under the hood so NFS shares are generally supported.
There's no timeout. If you want to implement a timeout yourself, you can do something like this:
$count = 0;
$timeout_secs = 10; //number of seconds of timeout
$got_lock = true;
while (!flock($fp, LOCK_EX | LOCK_NB, $wouldblock)) {
if ($wouldblock && $count++ < $timeout_secs) {
sleep(1);
} else {
$got_lock = false;
break;
}
}
if ($got_lock) {
// Do stuff with file
}
Nowdays there are a lot of websites for files hosting (uploading websites) and it count for example point per complete download of certain file.
My question
I want to understand what is the idea they are using !
How does it only count on complete downloading of the file ?!
i mean if i canceled downloading of the file after it started , it won't count point!
how does it knew ! is there any php function that able to know if i canceled downloading certain exact file or not !
that question was all time in my mind and thinking about it but i can't understand how does it works or what is the idea behind it. ~ thanks
This can be done by using my other answer as base How can I give download access to files outside public_html directory? and replacing readfile( $filename )
with readfileWhileConnected( $filename ):
Read file until EOF or disconnect:
/** Read $filename until EOF or disconnect,
* if disconnect then error_log() count of bytes read already
*/
function readfileWhileConnected( $filename ) {
// Save and set ini values:
$user_abort = ignore_user_abort();
ignore_user_abort(false);
// Get file size and set bytes_sent to zero:
$fsize = filesize($filename);
$bytes_sent = 0;
// Open file:
$f = fopen($filename, 'r');
// Read file:
while($chunk = fread($f, 1024)) {
// Check if connection is still open:
if(!connection_aborted()) {
// Send $chunk to buffer (if any), then flush() buffers:
echo $chunk;
flush();
// Add $chunk length to $bytes_sent
$bytes_sent += strlen($chunk);
} else {
// Close file:
fclose($f);
error_log("Connection closed at $bytes_sent/$fsize");
exit();
}
// Close file:
fclose($f);
// Reset ini values:
ignore_user_abort($user_abort);
return $bytes_sent;
}
}
After you have your new shiny class myNewSuperDownloadHandlerClass { ... } ready, then make sure you only serve downloads through filedownload.php described here or if have done good myNewSuperDownloadHandlerClass(), then use that, just make sure that readfileWhileConnected() is used for every download requiring connection status polling.
You can easily add callback to be triggered if user closes connection, only 2 exit points here. (seen many functions that have every often return false; return true; return null; return false; return true; and so on..)
I'm trying to write a PHP script that I want to ensure only has a single instance of it running at any given time. All of this talk about different ways of locking, and race conditions, and etc. etc. etc. is giving me the willies.
I'm confused as to whether lock files are the way to go, or semaphores, or using MySQL locks, or etc. etc. etc.
Can anyone tell me:
a) What is the correct way to implement this?
AND
b) Point me to a PHP implementation (or something easy to port to PHP?)
One way is to use the php function flock with a dummy file, that will act as a watchdog.
On the beginning of our job, if the file raise a LOCK_EX flag, exit, or wait, can be done.
Php flock documentation: http://php.net/manual/en/function.flock.php
For this examples, a file called lock.txt must be created first.
Example 1, if another twin process is running, it will properly quit, without retrying, giving a state message.
It will throw the error state, if the file lock.txt isn't reachable.
<?php
$fp = fopen("lock.txt", "r+");
if (!flock($fp, LOCK_EX|LOCK_NB, $blocked)) {
if ($blocked) {
// another process holds the lock
echo "Couldn't get the lock! Other script in run!\n";
}
else {
// couldn't lock for another reason, e.g. no such file
echo "Error! Nothing done.";
}
}
else {
// lock obtained
ftruncate($fp, 0); // truncate file
// Your job here
echo "Job running!\n";
// Leave a breathe
sleep(3);
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
}
fclose($fp); // Empty memory
Example 2, FIFO (First in, first out): we wants the process to wait, for an execution after the queue, if any:
<?php
$fp = fopen("lock.txt", "r+");
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
ftruncate($fp, 0); // truncate file
// Your job here
echo "Job running!\n";
// Leave a breathe
sleep(3);
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
}
fclose($fp);
It is also doable with fopen into x mode, by creating and erasing a file when the script ends.
Create and open for writing only; place the file pointer at the
beginning of the file. If the file already exists, the fopen() call
will fail by returning FALSE
http://php.net/manual/en/function.fopen.php
However, into a Unix environment, for fine tuning, I found easier to list the PID's of every background scripts with getmypid() into a DB, or a separate JSON file.
When one task ends, the script is responsible to declare his state in this file (eq: success/failure/debug infos, etc), and then remove his PID. This allows from my view to create admins tools and daemons in a simpler way. And use posix_kill() to kill a PID from PHP if necessary.
Micro-Services are composed using Unix-like pipelines.
Services can call services.
https://en.wikipedia.org/wiki/Microservices
See also: Prevent PHP script using up all resources while it runs?
// borrow from 2 anwsers on stackoverflow
function IsProcessRunning($pid) {
return shell_exec("ps aux | grep " . $pid . " | wc -l") > 2;
}
function AmIRunning($process_file) {
// Check I am running from the command line
if (PHP_SAPI != 'cli') {
error('Run me from the command line');
exit;
}
// Check if I'm already running and kill myself off if I am
$pid_running = false;
$pid = 0;
if (file_exists($process_file)) {
$data = file($process_file);
foreach ($data as $pid) {
$pid = (int)$pid;
if ($pid > 0 && IsProcessRunning($pid)) {
$pid_running = $pid;
break;
}
}
}
if ($pid_running && $pid_running != getmypid()) {
if (file_exists($process_file)) {
file_put_contents($process_file, $pid);
}
info('I am already running as pid ' . $pid . ' so stopping now');
return true;
} else {
// Make sure file has just me in it
file_put_contents($process_file, getmypid());
info('Written pid with id '.getmypid());
return false;
}
}
/*
* Make sure there is only one instance running at a time
*/
$lockdir = '/data/lock';
$script_name = basename(__FILE__, '.php');
// The file to store our process file
$process_file = $lockdir . DS . $script_name . '.pid';
$am_i_running = AmIRunning($process_file);
if ($am_i_running) {
exit;
}
Use semaphores:
$key = 156478953; //this should be unique for each script
$maxAcquire = 1;
$permissions =0666;
$autoRelease = 1; //releases semaphore when request is shut down (you dont have to worry about die(), exit() or return
$non_blocking = false; //if true, fails instantly if semaphore is not free
$semaphore = sem_get($key, $maxAcquire, $permissions, $autoRelease);
if (sem_acquire($semaphore, $non_blocking )) //blocking (prevent simultaneous multiple executions)
{
processLongCalculation();
}
sem_release($semaphore);
See:
https://www.php.net/manual/en/function.sem-get.php
https://www.php.net/manual/en/function.sem-acquire.php
https://www.php.net/manual/en/function.sem-release.php
You can go for the solution that fits best your project, the two simple ways to achieve that are file locking or database locking.
For implementations of file locking, check http://us2.php.net/flock
If you already use a database, create a table, generate known token for that script, put it there, and just remove it after the end of the script. To avoid problems on errors, you can use expiry times.
Perhaps this could work for you,
http://www.electrictoolbox.com/check-php-script-already-running/
In case you are using php on linux and I think the most practical way is:
<?php
if(shell_exec('ps aux | grep '.__FILE__.' | wc -l')>3){
exit('already running...');
}
?>
Another way to do it is with file flag and exit callback,
the exit callback will ensures that the file flag will be reset to 0 in any case of php execution end also fatal errors.
<?php
function exitProcess(){
if(file_get_contents('inprocess.txt')!='0'){
file_put_contents('inprocess.txt','0');
}
}
if(file_get_contents('inprocess.txt')=='1'){
exit();
}
file_put_contents('inprocess.txt','1');
register_shutdown_function('exitProcess');
/**
execute stuff
**/
?>