I am using the laravel middleware withoutOverlapping in laravel but it does not seem to work when jobs are dispatched at the same time (and take < 1 second)
Example
class TestWaitJob implements ShouldQueue
{
use Dispatchable;
use InteractsWithQueue;
use Queueable;
use SerializesModels;
public Charge $charge;
public function __construct(Charge $charge)
{
$this->charge = $charge;
}
/**
* #return array
*/
public function middleware(): array
{
return [
(new WithoutOverlapping($this->charge->id))
->releaseAfter(30)
];
}
/**
* #return void
*/
public function handle()
{
$charge = $this->charge->current_state_key_name;
return;
}
}
This is an absolutely simple job. It does nothing and it is what I have used to test.
If I go to my application and dispatch two copies of the job at the same time (using tinker)
TestWaitJob::dispatch(Charge::Find(1)); TestWaitJob::dispatch(Charge::find(1));1;
Both jobs are processed by horizon at the same time.
If I add a super simple sleep(1) line to the handle method of the job I get the expected behaviour which is
Job begins processing and acquires lock
The next job cannot acquire lock so is released back to the queue.
So with this sleep line, I have a job processed immediately (in 1 second) and the next job completes 31 seconds later which matches up exactly with the releaseAfter(30 seconds)
I have been looking at this for hours and everytime I introduce a delay the jobs process as expected but with no delay they process at the same time. The application is financial in nature so I cannot afford to potentially process jobs at the same time
Any help/advice would be greatly appreciated. (ps I am using Redis as cache)
Related
I've got the tasks table, this table has status and deadline columns. How can status be changed to "expired" automatically when current date will become greater than task's deadline date? Is there any realtime event listeners in Laravel?
I guess that's how event listener class should look like, but I'm not sure what to do next.
<?php
namespace App\Events;
use App\Models\Task;
use Illuminate\Broadcasting\InteractsWithSockets;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Queue\SerializesModels;
class DeadlineExpired
{
use Dispatchable, InteractsWithSockets, SerializesModels;
/**
* The task instance.
*
* #var \App\Models\Task
*/
public $task;
/**
* Create a new event instance.
*
* #param \App\Models\Task $task
* #return void
*/
public function __construct(Task $task)
{
$this->task = $task;
}
}
Since you're checking only the date. Your cron needs to run only once at midnight. Use Laravel Scheduler to do your Job.
First create a class
class UpdateTasks
{
public function __invoke()
{
// do your task here...e.g.,
Tasks::whereDate('deadline','<',today())->update(['status'=>'expired']);
}
}
Then in your app\Console\Kernel.php, schedule method-
$schedule->call(new UpdateTasks())->daily();
Finally configure a cron job at your server to run the schedule command daily at midnight.
php artisan schedule:run
There are Realtime event listeners but thes require a action to fire. Example are when a model is Created, Updated or Deleted then these events fire.
There is no built in "listener" to ping every model waiting for a field you defined to change.
If there is further logic you would like to fire when the Task becomes expired (like send email) then your best would be to run a check for any new expired Tasks using the scheduler.
The Scheduler runs every minute - set by cron.
I've created a simple laravel queue, a controller with a method which dispatches jobs into the queue, and a job for handling logic.
My plan is to have multiple queue workers using Supervisor config. And it works fine.
Supervisor config:
/etc/supervisor/conf.d/worker-node.conf
[program:laravel-worker]
...
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=admin
numprocs=8 //8 concurrent workers
...
Problem is that i want to log whenever a worker takes on a job from the queue and starts processing it. It's actually easy, i can just do some Log::info(...) but i also want to log the worker's unique identifier. It could be process id, worker id, worker number, whatever is possible. I want to do this so i can inspect which worker handled the job. Is such thing possible in laravel ? I know worker processes are daemon, but i think it could be possible to get process id somehow. Expected log output:
laravel.log
[2021-09-28 14:12:54] local.INFO: [worker identifier here] started processing JOB id 156
[2021-09-28 14:12:54] local.INFO: [worker identifier here] started processing JOB id 187
[2021-09-28 14:12:54] local.INFO: [worker identifier here] started processing JOB id 1214
Job class:
class ProcessLocationJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $fetchLocation;
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
$this->fetchLocation->execute();
}
/**
* Handle a job failure.
*
* #param \Throwable $exception
* #return void
*/
public function failed(Throwable $exception)
{
$jobId=''; //Some JOB ID or PID is needed here
Log::error("job id {$jobId} failed: {$exception->getMessage()}" );
}
}
TL;DR: How can i get worker's PID or any other identifier inside job's handle() method.
Any thought and alternatives are appreciated. Thank you.
Edit #1: Added job class
I am developing a laravel 5.7 application.
I have created a command that should setup my database:
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Artisan;
class TestSetupCommand extends Command
{
protected $signature = 'test:data';
protected $description = 'Basic Setup for Test Data';
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* #return mixed
*/
public function handle()
{
Artisan::call('migrate:refresh', ['--seed' => '']);
Artisan::call('basis:cc');
Artisan::call('tick:exchange');
$this->info("DB freshly setup: DONE");
$this->info("Coin: DONE");
$this->info("Exchange Scrapped: DONE");
}
}
My problem is that each command takes several minutes to run through. In total it costs me 25 minutes to fill the whole database with data.
I would like to run the commands only for 1 minutes each and kill them afterwards.
Any suggestions how to accomplish this within my laravel command?
I think the best way to do this is to extract these commands into background job. This artisan command then becomes code to queue up that new job (or jobs).
Why? It's very easy to configure jobs to timeout after x amount of time, by overriding a value like so:
<?php
namespace App\Jobs;
class ProcessPodcast implements ShouldQueue
{
/**
* The number of seconds the job can run before timing out.
*
* #var int
*/
public $timeout = 120;
}
Also, why are you refreshing the database? That seems.... like a crazy idea unless this is purely an analytics platform (no user data at all). It's probably a bad thing if that refresh command times out - you may look into job chaining so that the refresh command is guaranteed to succeed, then the other commands (new jobs now) have set timeouts.
I have a queue that sends requests to a remote service. Sometimes this service undergoes a maintenance. I want all queue tasks to pause and retry in 10 minutes when such situation is encountered. How do I implement that?
You can use the Queue::looping() event listener to pause an entire queue or connection (not just an individual job class). Unlike other methods, this will not put each job in a cycle of pop/requeue while the queue is paused, meaning the number of attempts will not increase.
Here's what the docs say:
Using the looping method on the Queue facade, you may specify
callbacks that execute before the worker attempts to fetch a job from
a queue.
https://laravel.com/docs/5.8/queues#job-events
What this doesn't document very well is that if the callback returns false then the worker will not fetch another job. For example, this will prevent the default queue from running:
Queue::looping(function (\Illuminate\Queue\Events\Looping $event) {
// $event->connectionName (e.g. "database")
// $event->queue (e.g. "default")
if ($event->queue == 'default') {
return false;
}
});
Note: The queue property of the event will contain the value from the command line when the worker process was started, so if your worker was checking more than one queue (e.g. artisan queue:work --queue=high,default) then the value of queue in the event will be 'high,default'. As a precaution, you may instead want to explode the string by commas and check if default is in the list.
So for example, if you want to create a rudimentary circuit breaker to pause the mail queue when your mail service returns a maintenance error, then you can register a listener like this in your EventServiceProvider.php:
/**
* Register any events for your application.
*
* #return void
*/
public function boot()
{
parent::boot();
Queue::looping(function (\Illuminate\Queue\Events\Looping $event) {
if (($event->queue == 'mail') && (cache()->get('mail-queue-paused'))) {
return false;
}
});
}
This assumes you have a mechanism somewhere else in your application to detect the appropriate situation and, in this example, that mechanism would need to assign a value to the mail-queue-paused key in the shared cache (because that's what my code is checking for). There are much more robust solutions, but setting a specific well-known key in the cache (and expiring it automatically) is simple and achieves the desired effect.
<?php
namespace App\Jobs;
use ...
class SendRequest implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
const REMOTE_SERVER_UNAVAILABLE = 'remote_server_unavailable';
private $msg;
private $retryAfter;
public function __construct($msg)
{
$this->msg = $msg;
$this->retryAfter = 10;
}
/**
* Execute the job.
*
* #return void
*/
public function handle(){
try {
// if we have tried sending the request and get a RemoteServerException, we will
// redispatch the job directly and return.
if(Cache::get(self::REMOTE_SERVER_UNAVAILABLE)) {
self::dispatch($this->msg)->delay(Carbon::now()->addMinutes($this->retryAfter));
return;
}
// send request to remote server
// ...
} catch (RemoteServerException $e) {
// set a cache value expires in 10 mins if not exists.
Cache::add(self::REMOTE_SERVER_UNAVAILABLE,'1', $this->retryAfter);
// if the remote service undergoes a maintenance, redispatch a new delayed job.
self::dispatch($this->msg)->delay(Carbon::now()->addMinutes($this->retryAfter));
}
}
}
I have a scheduled task with Laravel defined as below to run every 10 minutes. I also need the same job to be run on-demand without it overlapping if it is already running or preventing the scheduled job starting to run if the on-demand job is running.
/**
* Define the application's command schedule.
*
* #param \Illuminate\Console\Scheduling\Schedule $schedule
* #return void
*/
protected function schedule(Schedule $schedule)
{
$schedule->call(function () {
$job = new \App\Jobs\ImportJob();
$job->handle();
})->name('Import')->everyTenMinutes()->withoutOverlapping();
}
Is there a nice, simple way of achieving this with the schedular API or should the Job take care of its own mutex flag?