If a job failed, it will be pushed back to the Queue. Is there a way to remember the value of property in the job class when processing job again?
For example:
class MailJob extends Job
{
public $tries = 3;
public $status;
public function __construct()
{
$this->status = false; // set to false
}
/**
* Execute the job.
*/
public function handle()
{
$this->status = true;
// Assume job has failed, it went back to the Queue.
// status should be true when this job start processing again
}
}
If you want to rerun the failed process again on the same moment. you can do something like this.
Here the object is in memory while rerunning the job so data will be available.
I haven't verified it by running it , but hope it will work
class MailJob extends Job{
public $tries = 3;
public $status;
public function __construct()
{
$this->status = false; // set to false
}
/**
* Execute the job.
*/
public function handle()
{
$this->status = true;
// Assume job has failed, it went back to the Queue.
// status should be true when this job start processing again
$failed = processFailedisConfirm();
if $failed == true && $this->tries > -1 {
$this->tries = $this->tries - 1;
$this->handel();
}
}}
Example of processFailedisConfirm could be
public function processFailedisConfirm(){
// Core Process to be done in the Job
$state = (Do Some Api Call); // Here just example, you may send email
// Or can do the core Job Process
// And depending on the Response of the
// Process return true or false
// Is Job failed or not ?
if ( $state == "200" ){
return false; // Job is not failed
} else {
return true; // Job is failed
}
Logic of process is failed or not is depened on the operation you are doing. As i am doing an api call if i get response of 200 my process is successfull.
Otherwise process is failed.
This just example, sucess reponse of different api can be differnt as desigend by api designer.
Related
I am not sure whether this question is related to stomp-php or ActiveMQ Docker (running with defaults).
I have a simple Queue helper class written in PHP that handles both sending the message to the queue (Queue::push), as well as consumes it (Queue::fetch). See code below.
As you can see, fetch() should subscribe to the queue, read one message and unsubscribe. The message should be acknowledged automatically (\Stomp\StatefulStomp::subscribe(), 3rd. argument).
For some reason, about 5-7% of the messages are received by the customer twice or even three times. Why messages are delivered multiple times and how to avoid it?
Publisher (pushing 1000 messages):
$mq = new Queue('tcp://activemq:61613','test');
for ($msgCount = 0; $msgCount < 1000; $msgCount++) {
$mq->push('Message #' . $msgCount);
}
Consumer (receiving ~1070 messages):
$mq = new Queue('tcp://activemq:61613','test');
$received = 0;
while (true) {
$message = $mq->fetch();
if (null === $message) { break; }
$received++;
}
Queue class code:
use Stomp\Client;
use Stomp\Network\Connection;
use Stomp\SimpleStomp;
use Stomp\StatefulStomp;
use Stomp\Transport\Message;
class Queue
{
/**
* #var \Stomp\StatefulStomp
*/
private $stomp;
private $queue;
public function __construct($uri, $queue) {
$connection = new Connection('tcp://activemq:61613');
$this->stomp = new StatefulStomp(new Client($connection));
$connection->setReadTimeout(1);
$this->queue = $queue;
}
public function push($body) {
$message = new Message($body, ['activemq.maximumRedeliveries' => 0]);
$this->stomp->send('/queue/' . $this->queue, $message);
}
public function fetch() {
$subscriptionId = $this->stomp->subscribe('/queue/' . $this->queue, null, 'auto', ['activemq.prefetchSize' => 1]);
$msg = $this->stomp->read();
$this->stomp->unsubscribe($subscriptionId);
return $msg;
}
}
After years of searching for answers in other people's posts, I finally have to ask.
I'm currently trying to implement a WatchDog / Shepherd pattern to monitor long processes (from 2-3 minutes to several hours) running in my app.
The app is fairly complicated, and I started working on it mid-project, so I don't grasp it entirely.
Two servers are running : The one we'll call FACADE, with an Apache/2.4.10 (Debian), and CALCULUS, with an Apache/2.4.6 (Red Hat Enterprise Linux).
I have designed things like this :
The Shepherd initializes on FACADE, when the user click on a button that triggers a long process
The process, running on CALCULUS, initialises a WatchDog that connects to the Shepherd using a TCP Socket.
At some key points of the process, the WatchDog 'barks', i.e sends a string to the Shepherd to tell him at which step the Process is (the string is like "M(essage)#S(tarting)#Indexing", "M#C(ompleted)#Indexing", "M#S#Transfert", "M#C#Transfert"...) and if there's been an error (in which case it sends "E#Indexing" => Error happened during Indexing)
When he gets a Message from the dog, the Shepherd does what he has to (process, displaying to the user, filling a bit the progress bar...)
That my friends, was purely theory. Now comes the Implementation :
/**
*Implements the Singleton Design Pattern
*And of course, the Watchdog Pattern, using a Socket to communicate with the Shepherd.
*/
class Watchdog{
/** Singleton Design pattern. Use $watchdog = Watchdog::getInstance() to use the doggy */
private static $instance = null;
private function __construct(){
$this->init();
}
public static function getInstance(){
if (Watchdog::$instance === null)
Watchdog::$instance = new Watchdog();
return Watchdog::$instance;
}
private $socket;
private $connected = false;
private function init(){
$this->socket = fsockopen("tcp://A.B.C.D", 4242, $errno, $errstr);
if($this->socket === false){
echo "$errno : $errstr";
}else{
$this->connected = true;
}
}
public function kill(){
fclose($this->socket);
Watchdog::$instance = null;
}
public function bark($message){
if($this->connected === true){
fwrite($this->socket, "M#".$message."\n");
}
}
public function alert($err){
if($this->connected === true){
fwrite($this->socket, "E#".$err."\n");
}
}
}
?>
And the Shepeherd :
/**Implements the Singleton Design Pattern,
* And the Shepherd / Watchdog Patter, using a Socket to communicate with the Watchdog.
*/
class Shepherd{
/** Singleton Design pattern. Use $shepherd = Shepherd::GetInstance() to use the shepherd */
private static $instance;
private function __construct()
{
$this->init();
}
public static function GetInstance(){
if (self::$instance === null)
self::$instance = new Shepherd();
return self::$instance;
}
/**Instance variables **/
private $socket;
public $initialised = false;
public $doggyConnected = false;
private $dogsocket;
/**Init : Initializes the shephrd by binding it to the watchdog on the 4242 port**/
private function init(){
ob_implicit_flush();
$this->socket = stream_socket_server("tcp://0.0.0.0:4242", $errno, $errstr);
$this->initialised = true;
}
/**Sit And Wait : Waits for Dog connection**/
public function sitandwait(){
//Waiting for doggy connection
do {
if (($this->dogsocket = stream_socket_accept($this->socket)) === false) {
echo "socket_accept() failed: reason: " . socket_strerror(socket_last_error($this->dogsocket)) . "\n";
break;
}else{
$this->doggyConnected = true;
}
}while($this->doggyConnected === false);
}
/**Listen to Dog : Waits for Dog to send a message and echoes it **/
public function listentodog(){
if($this->dogsocket !== false) {
$buf = fgets($this->dogsocket);
return $buf;
}
}
/**Kill : Kills the shepherd and closes connections **/
private function kill(){
stream_socket_shutdown($this->socket, STREAM_SHUT_RDWR);
stream_socket_shutdown($this->dogsocket, STREAM_SHUT_RDWR);
fclose ($this->dogsocket);
fclose ($this->socket);
Shepherd::$instance = null;
}
/**Run : Runs the Shepherd till he got a message from dog **/
public function run(){
if($this->doggyConnected === false)
$this->sitandwait();
return $this->listentodog();
}
}
After trying those without success on the real website, I decided to try and see what was happening on both servers, thanks to netcat command. There's the catch : When I fake the Shepherd thanks to netcat, the dog can connect and send accurate data about the process. When I fake the dog thanks to netcat --send-only, the Shepherd gets the data and does the right things with them.
But when I run both from the application, I get a "Connexion refused" at the dog->init() (fsockopen : Connexion refusée), and of course, the Shepherd dies from Timeout.
But wait there's more !! You might think that the problem comes from the connection, and with netcat It doesn't come up because I don't do the connection from PHP (or I don't connect to a PHP opened socket).
I thought that too; I wrote two scripts, test_dog.php and test_shepherd.php, that are used EXACTLY in the same way than in my real live application. When I try to make them communicate, It works ! It even works with a real dog(monitoring a real application process) and test_shepherd.php, or with a real Shepherd (from my app), and test_watchdog.php
I decided to come here to ask you guys from help, because I'm utterly lost. i don't understand why It doesn't work with the real code, but does with the test_ scripts. In those, I made sure to use the objects exactly the same way than in the real application.
To show you everything, here are my test_ scripts :
test_watchdog.php
require "Watchdog.php";
$dog = Watchdog::getInstance();
$dog->bark("Try try try");
$dog->bark("Trying hard !!");
sleep(5);
$dog->bark("Trying harder to see...");
sleep(2);
$dog->bark("END");
$dog->kill();
test_shepherd.php
require "Shepherd.php";
$shep = new _Shepherd();
echo $shep->run();
... i think that's All. Please answer if you have the faintest idea that might help me, you're my last hope, I'm lost and desperate...
Thank you in advance :)
EDIT : On CALCULUS, the Watchdog is called by a Thousand-lines-long class, called Process (that runs the main process). The point is to be able to call Watchdog nearly everywhere in the code, where the user might have to wait.
Here is for instance the __construct of Process, intializing the Watchdog, and one of the methods that calls the $doggy->bark();
public function __construct($photoId = 0) {
$this->params = array();
$this->doggy = Watchdog::GetInstance();
$this->params['photoid'] = $photoId ;
date_default_timezone_set ('Europe/Paris');
}
public function transfertProject() {
try {
$this->doggy->bark('s#transfert');
//Traitement long
set_time_limit (0);
ini_set('post_max_size', 0);
ini_set('upload_max_filesize', 0);
$response = false;
if (!isset($_FILES['file'])) {
$post_max_size = ini_get('post_max_size');
$upload_max_size = ini_get('upload_max_filesize');
return "Le fichier ne semble pas avoir été posté, vérifier la taille maximal d'upload";
}
$name = $_FILES['file']['name'];
$filename = "../Workspace/projects/".$name;
$tmp = $_FILES['file']['tmp_name'];
if (move_uploaded_file($_FILES['file']['tmp_name'], $filename)) {
$response = $this->unzipProjectArchive();
unlink($filename);
}
$this->doggy->bark('c#transfert');
return $response;
} // END TRY
catch (Exception $ex) {
//Watchdog telling shepherd
$this->doggy->alert('transfert');
}
}
I have to set up multiple cron jobs. Each cron will be a separate request to the server. So, I started by the following, where each request will be handled by a case inside the switch, but the cases are bound to increase and thus doesn't seem to me a very good idea.
require_once './invoice_cron.php';
$checkRequest = isset($_REQUEST['request']);
if($checkRequest) {
$request_name = $_REQUEST['request'];
switch($request_name) {
case 'send_invoice':
break;
default:
break;
}
}
What could be a better approach here?
Let there be a request handler interface:
<?php
// CronRequests.php
require_once __DIR__.'./autoload.php';
$request_name = isset($_REQUEST['request']) ?
(new _cron)->handler($_REQUEST['request']) :
null ;
Make a class that could handle each request:
class _cron {
/**
* List of possible requests
* #var array
*/
private static $REQUESTS = ['send_invoice','start_user_subscription'];
/**
* HTTP request handler for all cron jobs
* #param string $request_name Name of the request
*/
public function handler($request_name) {
$status = false;
if(isset($request_name)) {
$request_map = array_flip(self::$REQUESTS);
if(isset($request_map[$request_name])) {
$status = $this->$request_name();
}
}
return $status;
}
}
The list of requests are bound to increase, so it is necessary to search the list efficiently. So, here we do an array flip and check a key for existence.
for each request type must match a function
all the types of requests must be defined in the system before start the processing
if a request is not in the list => some logical response must happen
i will give you just simple example, you can develop it
class Request{
private $REQUEST_TYPES = ['get_invoice', 'set_invoice', 'print_invoice'];
public function handler($requestKey){
foreach($this->REQUEST_TYPES as $type){
if($requestKey == $type) $this->$type();
}
$this->handlerNotFound();
}
private function get_invoice(){
//do some thing here
}
private function set_invoice(){
//do some thing here
}
private function print_invoice(){
//do some thing here
}
private function handlerNotFound(){
//do some thing here
}
}
In order to protect script form race hazard, I am considering approach described by code sample
$file = 'yxz.lockctrl';
// if file exists, it means that some other request is running
while (file_exists($file))
{
sleep(1);
}
file_put_contents($file, '');
// do some work
unlink($file);
If I go this way, is it possible to create file with same name simultaneously from multiple requests?
I know that there is php mutex. I would like to handle this situation without any extensions (if possible).
Task for the program is to handle bids in auctions application. I would like to process every bid request sequentially. With most possible latency.
From what I understand you want to make sure only a single process at a time is running a certain piece of code. A mutex or similar mechanism could be used for this. I myself use lockfiles to have a solution that works on many platforms and doesn't rely on a specific library only available on Linux etc.
For that, I have written a small Lock class. Do note that it uses some non-standard functions from my library, for instance, to get where to store temporary files etc. But you could easily change that.
<?php
class Lock
{
private $_owned = false;
private $_name = null;
private $_lockFile = null;
private $_lockFilePointer = null;
public function __construct($name)
{
$this->_name = $name;
$this->_lockFile = PluginManager::getInstance()->getCorePlugin()->getTempDir('locks') . $name . '-' . sha1($name . PluginManager::getInstance()->getCorePlugin()->getPreference('EncryptionKey')->getValue()).'.lock';
}
public function __destruct()
{
$this->release();
}
/**
* Acquires a lock
*
* Returns true on success and false on failure.
* Could be told to wait (block) and if so for a max amount of seconds or return false right away.
*
* #param bool $wait
* #param null $maxWaitTime
* #return bool
* #throws \Exception
*/
public function acquire($wait = false, $maxWaitTime = null) {
$this->_lockFilePointer = fopen($this->_lockFile, 'c');
if(!$this->_lockFilePointer) {
throw new \RuntimeException(__('Unable to create lock file', 'dliCore'));
}
if($wait && $maxWaitTime === null) {
$flags = LOCK_EX;
}
else {
$flags = LOCK_EX | LOCK_NB;
}
$startTime = time();
while(1) {
if (flock($this->_lockFilePointer, $flags)) {
$this->_owned = true;
return true;
} else {
if($maxWaitTime === null || time() - $startTime > $maxWaitTime) {
fclose($this->_lockFilePointer);
return false;
}
sleep(1);
}
}
}
/**
* Releases the lock
*/
public function release()
{
if($this->_owned) {
#flock($this->_lockFilePointer, LOCK_UN);
#fclose($this->_lockFilePointer);
#unlink($this->_lockFile);
$this->_owned = false;
}
}
}
Usage
Now you can have two process that run at the same time and execute the same script
Process 1
$lock = new Lock('runExpensiveFunction');
if($lock->acquire()) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Process 2
$lock = new Lock('runExpensiveFunction');
// Check will be false since the lock will already be held by someone else so the function is skipped
if($lock->acquire()) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Another alternative would be to have the second process wait for the first one to finish instead of skipping the code.
$lock = new Lock('runExpensiveFunction');
// Process will now wait for the lock to become available. A max wait time can be set if needed.
if($lock->acquire(true)) {
// Some expensive function that should only run one at a time
runExpensiveFunction();
$lock->release();
}
Ram disk
To limit the number of writes to your HDD/SSD with the lockfiles you could crate a RAM disk to store them in.
On Linux you could add something like the following to /etc/fstab
tmpfs /mnt/ramdisk tmpfs nodev,nosuid,noexec,nodiratime,size=1024M 0 0
On Windows you can download something like ImDisk Toolkit and create a ramdisk with that.
I am using Laravel 5 on a LAMP stack server running Ubuntu. I am using the queue system:
http://laravel.com/docs/master/queues
I can see from the docs that there is the ability to run queue:listen which will listen forever and process past and future items in the queue.
There is also queue:work which just processes the first item in the queue.
Is there a way to just process every item in the queue and then stop listening?
I only want to process the queue periodically so how can I setup a cron job that will process the queue and then as soon as everything in the queue has been done just exit?
I was just looking at this, as well. I modified the built-in queue:work command to process the entire queue and exit.
php artisan make:console ProcessQueueAndExit
You can get the code at https://gist.github.com/jdforsythe/b8c9bd46250ee23daa9de15d19495f07
Or here it is, for permanence:
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Carbon\Carbon;
use Illuminate\Queue\Worker;
use Illuminate\Contracts\Queue\Job;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Input\InputArgument;
class ProcessQueueAndExit extends Command {
protected $signature = 'queue:workall {connection?} {--queue=} {--daemon} {--delay=} {--force} {--memory=} {--sleep=} {--tries=}';
protected $description = 'Process all jobs on a queue and exit';
protected $worker;
public function __construct(Worker $worker) {
parent::__construct();
$this->worker = $worker;
}
public function handle() {
if ($this->downForMaintenance() && ! $this->option('daemon')) {
return $this->worker->sleep($this->option('sleep'));
}
$queue = $this->option('queue');
$delay = $this->option('delay');
$memory = $this->option('memory');
$connection = $this->argument('connection');
// keep processing until there are no more jobs returned
do {
$response = $this->runWorker(
$connection, $queue, $delay, $memory, $this->option('daemon')
);
if (! is_null($response['job'])) {
$this->writeOutput($response['job'], $response['failed']);
}
} while (! is_null($response['job']));
}
protected function runWorker($connection, $queue, $delay, $memory, $daemon = false) {
if ($daemon) {
$this->worker->setCache($this->laravel['cache']->driver());
$this->worker->setDaemonExceptionHandler(
$this->laravel['Illuminate\Contracts\Debug\ExceptionHandler']
);
return $this->worker->daemon(
$connection, $queue, $delay, $memory,
$this->option('sleep'), $this->option('tries')
);
}
return $this->worker->pop(
$connection, $queue, $delay,
$this->option('sleep'), $this->option('tries')
);
}
protected function writeOutput(Job $job, $failed) {
if ($failed) {
$this->output->writeln('<error>['.Carbon::now()->format('Y-m-d H:i:s').'] Failed:</error> '.$job->getName());
}
else {
$this->output->writeln('<info>['.Carbon::now()->format('Y-m-d H:i:s').'] Processed:</info> '.$job->getName());
}
}
protected function downForMaintenance() {
if ($this->option('force')) {
return false;
}
return $this->laravel->isDownForMaintenance();
}
}
I use this, in a command file:
$queue = Queue::connection('yourqueueconnection');
while ($entry = $queue->pop()) {
// your task
}