I have a PHP script that is used to query an API and download some JSON information / insert that information into a MySQL database, we'll call this scriptA.php. I need to run this script multiple times as minute, preferably as many times in a minute that I can without allowing two instances to run at the same exact time or with any overlap. My solution to this has been to create scriptB.php and put in on a one minute cron job. Here is the source code of scriptB.php...
function next_run()
{
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "http://somewebsite.com/scriptA.php");
curl_exec($curl);
curl_close($curl);
unset($curl);
}
$i = 0;
$times_to_run = 7;
$function = array();
while ($i++ < $times_to_run) {
$function = next_run();
sleep(3);
}
My question at this point is to how cURL performs when used in a loop, does this code trigger scriptA.php and THEN once it has finished loading it at that point start the next cURL request? Does the 3 second sleep even make a difference or will this literally run as fast as the time it takes each cURL request to complete. My objective is to time this script and run it as many times as possible in a one minute window without two iterations of it being run at the same time. I don't want to include the sleep statement if it is not needed. I believe what happens is cURL will run each request upon finishing the last, if I am wrong is there someway that I can instruct it to do this?
preferably as many times in a minute that I can without allowing two instances to run at the same exact time or with any overlap. - then you shouldn't use a cronjob at all, you should use a daemon. but if you for some reason have to use a cronjob (eg, if you're on a shared webhosting platform that doesn't allow daemons), guess you could use the sleep hack to run the same code several times a minute?
* * * * * /usr/bin/php /path/to/scriptA.php
* * * * * sleep 10; /usr/bin/php /path/to/scriptA.php
* * * * * sleep 20; /usr/bin/php /path/to/scriptA.php
* * * * * sleep 30; /usr/bin/php /path/to/scriptA.php
* * * * * sleep 40; /usr/bin/php /path/to/scriptA.php
* * * * * sleep 50; /usr/bin/php /path/to/scriptA.php
should make it execute every 10 seconds.
as for making sure it doesn't run in paralell if the previous execution hasn't finished yet, add this to the start of scriptA
call_user_func ( function () {
static $lock;
$lock = fopen ( __FILE__, "rb" );
if (! flock ( $lock, LOCK_EX | LOCK_NB )) {
// failed to get a lock, probably means another instance is already running
die ();
}
register_shutdown_function ( function () use (&$lock) {
flock ( $lock, LOCK_UN );
} );
} );
and it will just die() if another instance of scriptA is already running. however, if you want it to wait for the previous execution to finish, instead of just exiting, remove LOCK_NB... but that could be dangerous, if every, or even just a majority of the executions use more than 10 seconds, you'll have more and more processes waiting for the previous execution to finish, until you run out of ram.
as for your curl questions,
My question at this point is to how cURL performs when used in a loop, does this code trigger scriptA.php and THEN once it has finished loading it at that point start the next cURL request, that is correct, curl waits until the page has completely loaded, usually meaning the entire scriptA has completed. (you can tell scriptA to finish the pageload prematurely with the fastcgi_finish_request() function if you really want, but that's unusual)
Does the 3 second sleep even make a difference or will this literally run as fast as the time it takes each cURL request to complete - yes, the sleep will make the loop 3 seconds slower per iteration.
My objective is to time this script and run it as many times as possible in a one minute window without two iterations of it being run at the same time - then make it a daemon that never exits, rather than a cronjob.
I don't want to include the sleep statement if it is not needed. - it's not needed.
I believe what happens is cURL will run each request upon finishing the last - this is correct.
I need to run this script multiple times as minute, preferably as many times in a minute that I can without allowing two instances to run
Your in luck as I wrote a class to handle just such a thing. You can find it on my github here
https://github.com/ArtisticPhoenix/MISC/blob/master/ProcLock.php
I'll also copy the full code at the end of this post.
The basic idea is to create a file, I will call it afile.lock for this example. In this file is recorded the PID, or the process ID of the current process that is ran by cron. Then when cron attempts to run the process again, it checks this lock file and sees if there is a PHP process running that is using this PID.
if there is it updates the modified time of the file (and throws an exception)
if there is not then you are free to create a new instance of the "worker".
As a bonus th modified time of the lock file can be used by the script (whose PID we are tracking) as a way of shutting down in the event the file is not updated, so for example: if cron is stopped, or if the lock file is manually deleted you can set in in such a way that the running script will detect this and self destruct.
So not only can you keep multiple instances from running, you can tell the current instance to die if cron is turned off.
The basic usage is as follows. In the cron file that starts up the "worker"
//define a lock file (this is actually optional)
ProcLock::setLockFile(__DIR__.'/afile.lock');
try{
//if you didn't set a lock file you can pass it in with this method call
ProcLock::lock();
//execute your process
}catch(\Exception $e){
if($e->getCode() == ProcLock::ALREADY_LOCKED){
//just exit or what have you
}else{
//some other exception happened.
}
}
It's basically that easy.
Then in the running process you can every so often check (for example if you have a loop that runs something)
$expires = 90; //1 1/2 minute (you may need a bit of fudge time)
foreach($something as $a=>$b){
$lastAccess = ProcLock::getLastAccess()
if(false == $lastAccess || $lastAccess + $expires < time()){
//if last access is false (no lock file)
//or last access + expiration, is less then the current time
//log something like killed by lock timeout
exit();
}
}
Basically what this says is that either the lock file was deleted wile the process was running, or cron failed to update it before the expiration time. So here we are giving it 90 seconds and cron should be updating the lock file every 60 seconds. As I said the lock file is updated automatically if it's found when calling lock(), which calls canLock() which if it returns true meaning we can lock the process because its not currently locked, then it runs touch($lockfile) which updates the mtime (modified time).
Obviously you can only self kill the process in this way if it is actively checking the access and expiration times.
This script is designed to work both on windows and linux. On windows under certain circumstances the lock file won't properly be deleted (sometimes when hitting ctrl+c in the CMD window), however I have taken great pains to make sure this does not happen, so the class file contains a custom register_shutdown_function that runs when the PHP script ends.
When running something using the ProcLoc in the browser please note that the process id will always be the same no matter the tab its ran in. So if you open one tab that is Process locked, then open another tab, the process locker will see it as the same process and allow it to lock again. To properly run it in a browser and test the locking it must be done using two separate browsers such as crome and firefox. It's not really intended to be ran in the browser but this is one quirk I noticed.
One last note this class is completely static, as you can have only one Process ID per process that is running, which should be obvious.
The tricky parts are
making sure the lock file is disposed of in the event of even critical PHP failures
making sure another process didn't pick up the pid number when it was freed from PHP. This can be done with relative accuracy, in that we can tell if a PHP process is using it, and if so we assume its the process we need, there is much less chance a re-used PID would show up for another process very quickly, even less that it would be another PHP process
making all this work on both Linux and Windows
Lucky for you I have already invested sufficient time in this to do all these things, this is a more generic version of an original lock script I made for my job that we have used in this way successfully for 3 years in maintaining control over various synchronous cron jobs, everything from sFTP upload scanning, expired file clean up to RabbitMq message workers that run for an indefinite period of time.
In anycase here is the full code, enjoy.
<?php
/*
(c) 2017 ArtisticPhoenix
For license information please view the LICENSE file included with this source code GPL3.0.
Proccess Locker
==================================================================
This is a pseudo implementation of mutex since php does not have
any thread synchronization objects
This class uses files to provide locking functionality.
Lock will be released in following cases
1 - user calls unlock
2 - when this lock object gets deleted
3 - when request or script ends
4 - when pid of lock does not match self::$_pid
==================================================================
Only one Lock per Process!
-note- when running in a browser typically all tabs will have the same PID
so the locking will not be able to tell if it's the same process, to get
around this run in CLI, or use 2 diffrent browsers, so the PID numbers are diffrent.
This class is static for the simple fact that locking is done per-proces, so there is no need
to ever have duplate ProcLocks within the same process
---------------------------------------------------------------
*/
final class {
/**
* exception code numbers
* #var int
*/
const DIRECTORY_NOT_FOUND = 2000;
const LOCK_FIRST = 2001;
const FAILED_TO_UNLOCK = 2002;
const FAILED_TO_LOCK = 2003;
const ALREADY_LOCKED = 2004;
const UNKNOWN_PID = 2005;
const PROC_UNKNOWN_PID = 2006;
/**
* process _key
* #var string
*/
protected static $_lockFile;
/**
*
* #var int
*/
protected static $_pid;
/**
* No construction allowed
*/
private function __construct(){}
/**
* No clones allowed
*/
private function __clone(){}
/**
* globaly sets the lock file
* #param string $lockFile
*/
public static function setLockFile( $lockFile ){
$dir = dirname( $lockFile );
if( !is_dir( dirname( $lockFile ))){
throw new Exception("Directory {$dir} not found", self::DIRECTORY_NOT_FOUND); //pid directroy invalid
}
self::$_lockFile = $lockFile;
}
/**
* return global lockfile
*/
public static function getLockFile() {
return ( self::$_lockFile ) ? self::$_lockFile : false;
}
/**
* safe check for local or global lock file
*/
protected static function _chk_lock_file( $lockFile = null ){
if( !$lockFile && !self::$_lockFile ){
throw new Exception("Lock first", self::LOCK_FIRST); //
}elseif( $lockFile ){
return $lockFile;
}else{
return self::$_lockFile;
}
}
/**
*
* #param string $lockFile
*/
public static function unlock( $lockFile = null ){
if( !self::$_pid ){
//no pid stored - not locked for this process
return;
}
$lockFile = self::_chk_lock_file($lockFile);
if(!file_exists($lockFile) || unlink($lockFile)){
return true;
}else{
throw new Exception("Failed to unlock {$lockFile}", self::FAILED_TO_UNLOCK ); //no lock file exists to unlock or no permissions to delete file
}
}
/**
*
* #param string $lockFile
*/
public static function lock( $lockFile = null ){
$lockFile = self::_chk_lock_file($lockFile);
if( self::canLock( $lockFile )){
self::$_pid = getmypid();
if(!file_put_contents($lockFile, self::$_pid ) ){
throw new Exception("Failed to lock {$lockFile}", self::FAILED_TO_LOCK ); //no permission to create pid file
}
}else{
throw new Exception('Process is already running[ '.$lockFile.' ]', self::ALREADY_LOCKED );//there is a process running with this pid
}
}
/**
*
* #param string $lockFile
*/
public static function getPidFromLockFile( $lockFile = null ){
$lockFile = self::_chk_lock_file($lockFile);
if(!file_exists($lockFile) || !is_file($lockFile)){
return false;
}
$pid = file_get_contents($lockFile);
return intval(trim($pid));
}
/**
*
* #return number
*/
public static function getMyPid(){
return ( self::$_pid ) ? self::$_pid : false;
}
/**
*
* #param string $lockFile
* #param string $myPid
* #throws Exception
*/
public static function validatePid($lockFile = null, $myPid = false ){
$lockFile = self::_chk_lock_file($lockFile);
if( !self::$_pid && !$myPid ){
throw new Exception('no pid supplied', self::UNKNOWN_PID ); //no stored or injected pid number
}elseif( !$myPid ){
$myPid = self::$_pid;
}
return ( $myPid == self::getPidFromLockFile( $lockFile ));
}
/**
* update the mtime of lock file
* #param string $lockFile
*/
public static function canLock( $lockFile = null){
if( self::$_pid ){
throw new Exception("Process was already locked", self::ALREADY_LOCKED ); //process was already locked - call this only before locking
}
$lockFile = self::_chk_lock_file($lockFile);
$pid = self::getPidFromLockFile( $lockFile );
if( !$pid ){
//if there is a not a pid then there is no lock file and it's ok to lock it
return true;
}
//validate the pid in the existing file
$valid = self::_validateProcess($pid);
if( !$valid ){
//if it's not valid - delete the lock file
if(unlink($lockFile)){
return true;
}else{
throw new Exception("Failed to unlock {$lockFile}", self::FAILED_TO_UNLOCK ); //no lock file exists to unlock or no permissions to delete file
}
}
//if there was a valid process running return false, we cannot lock it.
//update the lock files mTime - this is usefull for a heartbeat, a periodic keepalive script.
touch($lockFile);
return false;
}
/**
*
* #param string $lockFile
*/
public static function getLastAccess( $lockFile = null ){
$lockFile = self::_chk_lock_file($lockFile);
clearstatcache( $lockFile );
if( file_exists( $lockFile )){
return filemtime( $lockFile );
}
return false;
}
/**
*
* #param int $pid
*/
protected static function _validateProcess( $pid ){
$task = false;
$pid = intval($pid);
if(stripos(php_uname('s'), 'win') > -1){
$task = shell_exec("tasklist /fi \"PID eq {$pid}\"");
/*
'INFO: No tasks are running which match the specified criteria.
'
*/
/*
'
Image Name PID Session Name Session# Mem Usage
========================= ======== ================ =========== ============
php.exe 5064 Console 1 64,516 K
'
*/
}else{
$cmd = "ps ".intval($pid);
$task = shell_exec($cmd);
/*
' PID TTY STAT TIME COMMAND
'
*/
}
//print_rr( $task );
if($task){
return ( preg_match('/php|httpd/', $task) ) ? true : false;
}
throw new Exception("pid detection failed {$pid}", self::PROC_UNKNOWN_PID); //failed to parse the pid look up results
//this has been tested on CentOs 5,6,7 and windows 7 and 10
}
/**
* destroy a lock ( safe unlock )
*/
public static function destroy($lockFile = null){
try{
$lockFile = self::_chk_lock_file($lockFile);
self::unlock( $lockFile );
}catch( Exception $e ){
//ignore errors here - this called from distruction so we dont care if it fails or succeeds
//generally a new process will be able to tell if the pid is still in use so
//this is just a cleanup process
}
}
}
/*
* register our shutdown handler - if the script dies unlock the lock
* this is superior to __destruct(), because the shutdown handler runs even in situation where PHP exhausts all memory
*/
register_shutdown_function(array('\\Lib\\Queue\\ProcLock',"destroy"));
I'm trying to write gameserver based on Workerman.
Main idea is: list of workers accept messages from clients and puts them to the queue (RabbitMQ), another group of workers get messages from queue, do some calculation and update GameWorld instance accordingly.
GameWorld instance itself is created on start of the main process, workers are created after creating of the GameWorld object. So, I wrote dummy class Server:
namespace Server;
use \Workerman\Worker;
use \Workerman\Lib\Timer;
class Server {
private $name;
public function setName(string $name){
$this->name = $name;
}
public function printName(){
echo $this->name;
}
}
Also I wrote two simple workers just for testing concept of updating object from different workers.
First (start_worker1.php):
use \Workerman\Worker;
use \Workerman\Lib\Timer;
use \Server\Server;
global $ws_worker;
$ws_worker = new Worker('Websocket://0.0.0.0:8000');
$ws_worker->name = 'FirstWorker';
$ws_worker->onWorkerStart = function($ws_worker)
{
$ws_worker->server = new Server();
$ws_worker->server->setName("FirstName");
echo "worker1 started\n";
$ws_worker->is_started = TRUE;
var_dump($ws_worker);
};
Here I created new Server object and give it a name, also changed property $ws_worker->is_started to TRUE (default value is False).
Second(start_worker2.php):
use \Workerman\Worker;
use \Workerman\Lib\Timer;
global $ws_worker;
$worker = new Worker('Websocket://0.0.0.0:8001');
$worker->name = 'SecondWorker';
// here I'm checking, if I had an object of the first worker
// and actually it outputs all the data about first worker object,
// but $ws_worker->server is NULL and $ws_worker->is_started = FALSE
// This is confusing me so much..
var_dump($ws_worker);
// here I'm trying to detect when first worker is started
// but it return false all the time..
while(!$ws_worker->is_started){
var_dump($ws_worker->is_started);
sleep(1);
}
$worker->onWorkerStart = function() use($ws_worker){
echo "worker2 started\n";
var_dump($ws_worker);
$ws_worker->sever->setName("NewName");
};
Here I created second worker and tried to access server property of the first worker object, but with no success..
All this stuff is started this way:
use Workerman\Worker;
use Server\Server;
if(strpos(strtolower(PHP_OS), 'win') === 0)
{
exit("start.php does not support windows, please use start_for_win.bat\n");
}
if(!extension_loaded('pcntl'))
{
exit("Error! <pcntl> extension not found! Please install pcntl extension.");
}
if(!extension_loaded('posix'))
{
exit("Error! <posix> extension not found! Please install posix extension.");
}
define('GLOBAL_START', 1);
require_once __DIR__ . '/vendor/autoload.php';
require_once __DIR__ . '/start_worker1.php';
require_once __DIR__ . '/start_worker2.php';
// Run all services
Worker::runAll();
I guess that in worker2 I can access an worker1 object before function $ws_worker->onWorkerStart is evaluated, but I have no idea, how to access worker1 object in real time ( I mean - get access to current state of the object).
I'm new to PHP OOP style programming, I should say. So please show me where is my mistake. Detailed explanation greatly appreciated.
In a Laravel 5 package I am making there is a class FileSelector that uses the Storage-facade in a certain method.
public function filterFilesOnDate($files, DateTime $date)
{
return array_filter($files, function($file) use($date){
return Storage::lastModified($file) < $date->getTimeStamp();
});
}
This class takes a path (to some files) and a Storage::disk()in it's constructor.
Now I am trying to write some basic unit tests for this specific class using the Orchestra Testbench.
The setUp-function looks like this:
protected $fileSelector;
protected $date;
public function setUp()
{
parent::setUp();
$this->date = new DateTime();
$this->fileSelector = new fileSelector('tests/_data/backups', Storage::disk('local'));
}
The failing test is:
public function test_if_files_are_filtered_on_date()
{
$files = Storage::allFiles('tests/_data/backups');
$filteredFiles = $this->fileSelector->filterFilesOnDate($files, $this->date);
}
Storage::allFiles('tests/_data/backups') returns no files at all.
The path is correct because using the File-facade returns the needed files but this isn't compatible with the filterFilesOnDate()-method because it uses Storage.
Using the File-facade generates the following error:
League\Flysystem\FileNotFoundException: File not found at tests/_data/backups/ElvisPresley.zip
Am I using the Storage-methods wrong in the test or have I stumbled on a limitation of Orchestra/Testbench?
Ok, turns out I didn't completely understand how Storageand disks worked.
Using things like Storage::lastModified() calls the default Filesystem specified in the filesystem-config.
Since this is a test there is no config.
What Storage::disk() does, is create an instance of FilesystemAdapter using a Filesystem-object So a Storage object needs to be 'recreated'.
So:
$this->fileSelector = new FileSelector('tests/_data/backups', Storage::disk('local'));
Becomes:
$this->disk = new Illuminate\Filesystem\FilesystemAdapter(
new Filesystem(new Local($this->root))
);
$this->fileSelector = new FileSelector($this->disk, $this->path);
($this->pathis the path the where the files I use for testing are stored)
It was also pointed out to me that I should set the lastModified-timestamps manually everytime the test is run to avoid differing test results.
foreach (scandir($this->testFilesPath) as $file)
{
touch($this->testFilesPath . '/' . $file, time() - (60 * 60 * 24 * 5));
}
Using touch you can create files or set timestamps of files. In this case, they are set to 5 days.
I'm trying to run a job queue to create a PDF file using SlmQueueBeanstalkd and DOMPDFModule in ZF".
Here's what I'm doing in my controller:
public function reporteAction()
{
$job = new TareaReporte();
$queueManager = $this->serviceLocator->get('SlmQueue\Queue\QueuePluginManager');
$queue = $queueManager->get('myQueue');
$queue->push($job);
...
}
This is the job:
namespace Application\Job;
use SlmQueue\Job\AbstractJob;
use SlmQueue\Queue\QueueAwareInterface;
use SlmQueue\Queue\QueueInterface;
use DOMPDFModule\View\Model\PdfModel;
class TareaReporte extends AbstractJob implements QueueAwareInterface
{
protected $queue;
public function getQueue()
{
return $this->queue;
}
public function setQueue(QueueInterface $queue)
{
$this->queue = $queue;
}
public function execute()
{
$sm = $this->getQueue()->getJobPluginManager()->getServiceLocator();
$empresaTable = $sm->get('Application\Model\EmpresaTable');
$registros = $empresaTable->listadoCompleto();
$model = new PdfModel(array('registros' => $registros));
$model->setOption('paperSize', 'letter');
$model->setOption('paperOrientation', 'portrait');
$model->setTemplate('empresa/reporte-pdf');
$output = $sm->get('viewPdfrenderer')->render($model);
$filename = "/path/to/pdf/file.pdf";
file_put_contents($filename, $output);
}
}
The first time you run it, the file is created and the work is successful, however, if you run a second time, the task is buried and the file is not created.
It seems that stays in an endless cycle when trying to render the model a second time.
I've had a similar issue and it turned out it was because of the way ZendPdf\PdfDocument reuses it's object factory. Are you using ZendPdf\PdfDocument?
You might need to correctly close factory.
class MyDocument extends PdfDocument
{
public function __destruct()
{
$this->_objFactory->close();
}
}
Try to add this or something similar to the PdfDocument class...
update : it seem you are not using PdfDocument, however I suspect this is the issue is the same. Are you able to regenerate a second PDF in a normal http request? It is your job to make sure the environment is equal on each run.
If you are unable to overcome this problem a short-term quick solution would be to set max_runs configuration for SlmQueue to 1. That way the worker is stopped after each job and this reset to a vanilla state...
I'm trying to access an FTP server from my PHP script using Codeigniter's FTP Library. These functions work great, but when testing the script I discovered that if I attempt to connect to a server that does not exist, the script does not terminate with an error message of any kind.
The page continues to execute, until the web server gives up, returning an empty document.
So I am wondering, is there a way to limit the amount of time that Codeigniter can try to connect to an FTP server, then display a message if that times out?
I tried using the php function set_time_limit(), but it does not behave how I expected it to.
Thanks for your help.
Codeigniter's ftp class uses the underlying ftp_connect php call that supports a 3rd optional parameter, timeout (http://ca2.php.net/manual/en/function.ftp-connect.php).
Codeigniter however does not use it, but allows for extending the default libraries it provides (providing that you're willing to do some work and check that any updates you do to the core will not break the functionality of your extended class). So to solve your problem you could create a new library in you application library folder:
<?php
class MY_FTP extends CI_FTP { //Assuming that in your config.php file, your subclass prefix is set to 'MY_' like so: $config['subclass_prefix'] = 'MY_';
var $timeout = 90;
/**
* FTP Connect
*
* #access public
* #param array the connection values
* #return bool
*/
function connect($config = array())
{
if (count($config) > 0)
{
$this->initialize($config);
}
if (FALSE === ($this->conn_id = ftp_connect($this->hostname, $this->port, $this->timeout)))
{
if ($this->debug == TRUE)
{
$this->_error('ftp_unable_to_connect');
}
return FALSE;
}
if ( ! $this->_login())
{
if ($this->debug == TRUE)
{
$this->_error('ftp_unable_to_login');
}
return FALSE;
}
// Set passive mode if needed
if ($this->passive == TRUE)
{
ftp_pasv($this->conn_id, TRUE);
}
return TRUE;
}
}
?>
and from your script, you could add to your configuration array the timeout option:
$this->load->library('ftp'); //if ftp is not autoloaded
$ftp_params = array('hostname'=>'1.2.3.4', 'port'=>21, 'timeout'=>10); //timout is 10 seconds instead of default 90
$ftp_conn = $this->ftp->connect($ftp_params);
if(FALSE === $ftp_conn) {
//Code to handle error
}
The ftp class is not designed to give error messages unless the debug parameter is set to TRUE in te config array, in which case it'll just display an error. However it can also be override, because all errors call the function _error() in the class. So you could set 'debug' => true in your $ftp_params array, and add a function in MY_ftp like so:
/**
* This function overrides
*/
function _error($line)
{
$this->error = $line;
}
And then have a function getError()
/**
* This function overrides
*/
function get_error()
{
return $this->error;
}
So if
$ftp_conn = $this->ftp->connect($ftp_params);
returns false, you can call
$error = $this->ftp->get_error();
to get your error and display it.
Now, you can always customize and have a more complex error handling mechanism by further customizing the class...
Hope it answers your question.
The answer is simple, don't attempt to connect to a server that doesn't exist.