Laravel Queue - Pause between jobs - php

I want to create a queue (AMAZON SQS) that only runs jobs every X sec. So if suddenly 50 jobs are submitted, the end up in the queue. The queue listener then pulls a job, does something and waits X sec. After that, the next job is pulled. Another X sec pause. Etc etc
For the queue listener, the sleep option option only determines how long the worker will "sleep" if there are no new jobs available. So it will only sleep if there is nothing in the queue.
Or should I just put in a pause(x) in my PHP code?
[edit] I just tested the sleep method with a FIFO and standard AWS SQS queue and this messes up the whole queue. Suddenly jobs are (sucesssfully) resubmitted 3 times after which the go into failed state. Moreover, the delay that is given in my code (3-4 min) was ignored, instead a one minute was taken
<?php
namespace App\Jobs;
use App\City;
class RetrieveStations extends Job
{
protected $cities;
/**
* Create a new job instance.
*
* #return void
*/
public function __construct ($cities)
{
$this->cities = $cities;
}
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
// code here
doSomething()
sleep(X);
}
}

I have the exact same problem to solve. I'm using Laravel 5.8 and I don't see how I can get the queue worker to wait a fixed period between jobs.
I'm now thinking of using a scheduled task to handle this. I can schedule a task to run, say, every 5 minutes and run the following artisan command:
$schedule->command('queue:work --queue=emails --once')->everyFiveMinutes();
This will take one job from the queue and run it. Unfortunately, there's not much more granular control over how often a job is processed.

Exactly, you need to set asleep your php code, there is no other way.
Php sleep

Related

Laravel Cron Scheduler job not running as expected

I have Laravel cron issue ,In Console Kernel I have defined job which will hit Rollovercron.php file every 10 mins and every time it will hit it will pass one country. atleast 100 Countries are defined in an array and will be passed one by one to Rollovercron.php according to foreach loop. Rollovercron.php file takes minimum 2 hrs to run for single country.
I have multiple issues with this cron job:
100 elements in an array not getting fetched one by one means I can see 'GH' country(Ghana) has run for 5 times continuously and many of the countries are skipped.
when ever I get country missing issue I do composer update and clear cache frequently.
I want my cron should run smoothly and fetch all countries not even single country should miss and I should not need to do composer update for this all the time.
Please help me in this ,struggling for this since many months.
bellow is Kernel.php file:
<?php
namespace App\Console;
use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;
use DB;
class Kernel
extends ConsoleKernel
{
/**
* The Artisan commands provided by your application.
*
* #var array
*/
protected $commands = [
\App\Console\Commands\preAlert::class,
\App\Console\Commands\blCron::class,
\App\Console\Commands\mainRollover::class,
\App\Console\Commands\refilingSync::class,
\App\Console\Commands\TestCommand::class,
\App\Console\Commands\rollOverCron::class,
\App\Console\Commands\FrontPageRedis::class,
\App\Console\Commands\filingStatusRejectionQueue::class,
\App\Console\Commands\VesselDashboardRedis::class,
\App\Console\Commands\Bookingcountupdate::class,
// \App\Console\Commands\Voyagetwovisit::class,
];
/**
* Define the application's command schedule.
*
* #param \Illuminate\Console\Scheduling\Schedule $schedule
* #return void
*/
protected function schedule(Schedule $schedule)
{
$countrylist=array('NL','AR','CL','EC','DE','PH','ID','TT','JM','KR','BE','VN','US','BR','CM','MG','ZA','MU','RU','DO','GT','HN','SV', 'PR','SN', 'TN', 'SI','CI','CR','GM','GN','GY','HR','LC','LR','MR','UY','KH','BD','TH','JP','MM','AT','IE','CH','LB','PY','KE','YT','TZ','MZ','NA','GQ','ME');
foreach ($countrylist as $country) {
$schedule->command('rollOverCron:send ' . $country)
->everyTenMinutes()
->withoutOverlapping();
}
foreach ($countrylist as $country) {
$schedule->command('mainRollover:send ' . $country)
->daily()
->withoutOverlapping();
}
$schedule->command('filingStatusRejectionQueue')
->hourly()
->withoutOverlapping();
$schedule->command('Bookingcountupdate')
->everyTenMinutes()
->withoutOverlapping();
$schedule->command('preAlert')
->hourly()
->withoutOverlapping();
}
protected function commands()
{
require base_path('routes/console.php');
}
}
/**
* Register the Closure based commands for the application.
*
* #return void
*/
Laravel scheduling, knowing how it works helps, so you can debug it when it doesn't work as expected. This does involve diving in the source.
You invoke command on the scheduler, this returns an event.
Let's check how Laravel decides what defines overlapping, we see it expires after 1440 minutes, aka 24 hours.
So after one day, if the scheduled items have not run these scheduled items just stop being scheduled.
We see that a mutex is being used here. Let's see where it comes from. It seems it's provided in the constructor.
So lets see which mutex is being provided. In the exec and the call functions the mutex defined in the Scheduler constructor is used.
The mutex used there is an interface, probably used as a Facade, and the real implementation is most likely in CacheSchedulingMutex, which creates a mutex id using the mutexName from the event and the current time in hours and minutes.
Looking at the mutexName we see that the id exists out of the expression and command combined.
To summarise, all events called in one Scheduler function, share the same mutex that is used in checking if method calls don't overlap, but the mutex generates an unique identifier for each command, including differing parameters, and based on the time.
Your scheduled jobs will expire after 24 hours, which means that with jobs that take 2 hours to complete, you'll get about 12 jobs in a day completed. More if the jobs are small, less if the jobs take longer. This is because PHP is a single threaded process by default.
First task 1, then task 2, then task 3. Etc... This means that if each tasks takes 2 hours, then after 12 tasks their queued jobs expire because the job has been running for 1440 minutes and then the new jobs are scheduled and it starts again from the top.
Luckily there is a way to make sure they run simultaneously.
I suggest you add ->runInBackground() to your scheduling calls.
$schedule->command('rollOverCron:send ' . $country)
->everyTenMinutes()
->withoutOverlapping()
->runInBackground()
->emailOutputTo(['ext.amourya#cma-cgm.com','EXT.KKURANKAR#cma-cgm.com']);cgm.com']);
}

Rate limiting PHP function

I have a php function which gets called when someone visits POST www.example.com/webhook. However, the external service which I cannot control, sometimes calls this url twice in rapid succession, messing with my logic since the webhook persists stuff in the database which takes a few ms to complete.
In other words, when the second request comes in (which can not be ignored), the first request is likely not completed yet however I need this to be completed in the order it came in.
So I've created a little hack in Laravel which should "throttle" the execution with 5 seconds in between. It seems to work most of the time. However an error in my code or some other oversight, does not make this solution work everytime.
function myWebhook() {
// Check if cache value (defaults to 0) and compare with current time.
while(Cache::get('g2a_webhook_timestamp', 0) + 5 > Carbon::now()->timestamp) {
// Postpone execution.
sleep(1);
}
// Create a cache value (file storage) that stores the current
Cache::put('g2a_webhook_timestamp', Carbon::now()->timestamp, 1);
// Execute rest of code ...
}
Anyone perhaps got a watertight solution for this issue?
You have essentially designed your own simplified queue system which is the right approach but you can make use of the native Laravel queue to have a more robust solution to your problem.
Define a job, e.g: ProcessWebhook
When a POST request is received to /webhook queue the job
The laravel queue worker will process one job at a time[1] in the order they're received, ensuring that no matter how many requests are received, they'll be processed one by one and in order.
The implementation of this would look something like this:
Create a new Job, e.g: php artisan make:job ProcessWebhook
Move your webhook processing code into the handle method of the job, e.g:
public function __construct($data)
{
$this->data = $data;
}
public function handle()
{
Model::where('x', 'y')->update([
'field' => $this->data->newValue
]);
}
Modify your Webhook controller to dispatch a new job when a POST request is received, e.g:
public function webhook(Request $request)
{
$data = $request->getContent();
ProcessWebhook::dispatch($data);
}
Start your queue worker, php artisan queue:work, which will run in the background processing jobs in the order they arrive, one at a time.
That's it, a maintainable solution to processing webhooks in order, one-by-one. You can read the Queue documentation to find out more about the functionality available, including retrying failed jobs which can be very useful.
[1] Laravel will process one job at a time per worker. You can add more workers to improve queue throughput for other use cases but in this situation you'd just want to use one worker.

Laravel second task schedule never runs

i have cron entry at shared hosting:
* * * * * php web/artisan schedule:run
The problem is that queue:work command works no matter the order, but second task (Closure) that updates video views never runs.
Code from Kernel.php:
/**
* Define the application's command schedule.
*
* #param \Illuminate\Console\Scheduling\Schedule $schedule
* #return void
*/
protected function schedule(Schedule $schedule)
{
//Each Video total_video_views will be accurately updated on daily basis.
$schedule->call(function () {
$videos = Video::get();
app('queue')->pushOn('high', new UpdateVideoViews($videos));
})->hourlyAt('30')
->name('updates all video views')
->withoutOverlapping();
//this is alternative to supervisor since shared hosting someties doesnt provide supervisor
//so we use cron instead to keep queue worker up
$schedule->command('queue:work --queue=high,default')
->everyMinute()
->name('keeps queue worker active')
->withoutOverlapping();
}
Is there any way to run it? My suspiction is that queue:work command along "withoutOverlapping()" is kinda maybe blocking any other tasks to process.

Send response but keep long running script going to prevent timeout?

I am wondering how to deal with this. I have a webhook endpoint which responds to a webhook call from Github.
It starts a long running process in where it clones the repository from which the webhook call was made.
/**
* The webhook endpoint.
*
* #param Request $request
* #return mixed
* #throws \Exception
*/
public function webhook(Request $request)
{
// The type of GitHub event that we receive.
$event = $request->header('X-GitHub-Event');
$url = $this->createCloneUrl();
$this->cloneGitRepo($url);
return new Response('Webhook received succesfully', 200);
}
The problem with this is that Github gives an error when the response is not provided soon enough.
We couldn’t deliver this payload: Service Timeout
This is rightfully so because my cloneGitRepo method is simply blocking the execution of the response and it is taking too long.
How can I still deliver a response to acknowledge to Github that the webhook call has been made successfully and start my long running process?
I am using Laravel for all of this with Redis, maybe something can be accomplished there? I am open to all suggestions.
What you're looking for is a queued job. Laravel makes this very easy with Laravel Queues.
With queues, you setup a queue driver (database, redis, Amazon SQS, etc), and then you have one to many queue workers that are continuously running. When you put a job on the queue from your webhook method, it will be picked up by one of your queue workers and run in a separate process. The act of dispatching a queued job to a queue is very quick, though, so your webhook method will return quickly while the real work is being done by the queue worker.
The linked documentation has all the details, but the general process will be:
Setup a queue connection. You mention you're already using redis, I would start with that.
Use php artisan make:job CloneGitRepo to create a CloneGitRepo job class.
It should implement the Illuminate\Contracts\Queue\ShouldQueue interface so that Laravel knows to send this job to a queue when it is dispatched.
Make sure to define properties on the class for any data you pass into the constructor. This is necessary so the worker can rebuild the job correctly when it is pulled off the queue.
The queue worker will call the handle() method to process the job. Any dependencies can be type hinted here and they will be injected from the IoC container.
To dispatch the job to the queue, you can either use the global dispatch() helper function, or call the static dispatch() method on the job itself.
dispatch(new CloneGitRepo($url));
CloneGitRepo::dispatch($url);
So, your webhook would look like:
public function webhook(Request $request)
{
// The type of GitHub event that we receive.
$event = $request->header('X-GitHub-Event');
$url = $this->createCloneUrl();
CloneGitRepo::dispatch($url);
return new Response('Webhook received succesfully', 200);
}

Run cron jobs with same cron-time on a 10s interval queue

So basically I couldnt find my answer elsewhere. I'm having an issue with Laravel 5.x in where I have models called 'Bumpers' which can have a certain cron-time value (say 'every 25 minutes').
Many bumpers can have the same cron-time, meaning that at a certain time, many of these bumpers may be executed using the Laravel Scheduler. However, I need to run these bumpers 10 seconds apart from eachother. So if 10 bumpers are triggered at 8:00:00PM, I need the queue to run with 10 seconds interval until all bumpers have been executed.
I've tried adding sleep(); to the $schedule->call(function(){executeStuff(); sleep(10)});. That worked for a bit but unfortunately it only works for queues not exceeding 60 seconds because then the Scheduler is ran again:
// This function is called on "artisan schedule:run"
protected function schedule(Schedule $schedule)
{
// Get only the active bumpers.
$bumpers = Bumper::where('status', 1)->get();
// Register all bumpers through their occurrence value.
foreach ($bumpers as $bumper) {
// Check for member/admin. All other bumpers are neglected.
if ($bumper->user->canBump()) {
$schedule->call(function () use ($bumper) {
sleep(10);
$bumper->post();
})->cron($bumper->occurrence);
}
}
}
Any solutions?

Categories