How to process all items in queue and exit - php

I am using Laravel 5 on a LAMP stack server running Ubuntu. I am using the queue system:
http://laravel.com/docs/master/queues
I can see from the docs that there is the ability to run queue:listen which will listen forever and process past and future items in the queue.
There is also queue:work which just processes the first item in the queue.
Is there a way to just process every item in the queue and then stop listening?
I only want to process the queue periodically so how can I setup a cron job that will process the queue and then as soon as everything in the queue has been done just exit?

I was just looking at this, as well. I modified the built-in queue:work command to process the entire queue and exit.
php artisan make:console ProcessQueueAndExit
You can get the code at https://gist.github.com/jdforsythe/b8c9bd46250ee23daa9de15d19495f07
Or here it is, for permanence:
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Carbon\Carbon;
use Illuminate\Queue\Worker;
use Illuminate\Contracts\Queue\Job;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Input\InputArgument;
class ProcessQueueAndExit extends Command {
protected $signature = 'queue:workall {connection?} {--queue=} {--daemon} {--delay=} {--force} {--memory=} {--sleep=} {--tries=}';
protected $description = 'Process all jobs on a queue and exit';
protected $worker;
public function __construct(Worker $worker) {
parent::__construct();
$this->worker = $worker;
}
public function handle() {
if ($this->downForMaintenance() && ! $this->option('daemon')) {
return $this->worker->sleep($this->option('sleep'));
}
$queue = $this->option('queue');
$delay = $this->option('delay');
$memory = $this->option('memory');
$connection = $this->argument('connection');
// keep processing until there are no more jobs returned
do {
$response = $this->runWorker(
$connection, $queue, $delay, $memory, $this->option('daemon')
);
if (! is_null($response['job'])) {
$this->writeOutput($response['job'], $response['failed']);
}
} while (! is_null($response['job']));
}
protected function runWorker($connection, $queue, $delay, $memory, $daemon = false) {
if ($daemon) {
$this->worker->setCache($this->laravel['cache']->driver());
$this->worker->setDaemonExceptionHandler(
$this->laravel['Illuminate\Contracts\Debug\ExceptionHandler']
);
return $this->worker->daemon(
$connection, $queue, $delay, $memory,
$this->option('sleep'), $this->option('tries')
);
}
return $this->worker->pop(
$connection, $queue, $delay,
$this->option('sleep'), $this->option('tries')
);
}
protected function writeOutput(Job $job, $failed) {
if ($failed) {
$this->output->writeln('<error>['.Carbon::now()->format('Y-m-d H:i:s').'] Failed:</error> '.$job->getName());
}
else {
$this->output->writeln('<info>['.Carbon::now()->format('Y-m-d H:i:s').'] Processed:</info> '.$job->getName());
}
}
protected function downForMaintenance() {
if ($this->option('force')) {
return false;
}
return $this->laravel->isDownForMaintenance();
}
}

I use this, in a command file:
$queue = Queue::connection('yourqueueconnection');
while ($entry = $queue->pop()) {
// your task
}

Related

Symfony command running inside another command dont work

Hellow , I need some hel regarding executing symfony commands inside another command. I am not new with this and I created many commands and run them from inside commands, controllers and it always work. But this one I do not understand why It do not working like the others are. I am running one command all the time and from time to time I created some extra workers when there are many jobs to get this one worker some help (one-check option).
I created command to run beanstalk worker with this class:
<?php
namespace AppBundle\Command;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Question\Question;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Command\LockableTrait;
/**
* Class BeanstalkWorkerCommand
* Command start the Beanstalk worker. Get the job from queue and preform job
*
* #method configure()
* #method execute(InputInterface $input, OutputInterface $output)
* #method authenticateUser(InputInterface $input, OutputInterface $output)
* #method checkAuthentication($username, $password)
*/
class BeanstalkWorkerCommand extends ContainerAwareCommand
{
use LockableTrait;
protected function configure()
{
$this
->setName('beanstalk:worker:start')
->setDescription('Start the Beanstalk infinitive worker. Get the job from queue and preform job')
->addOption(
'one-check',
'o',
InputOption::VALUE_NONE,
'If set, the worker will check tubes only once and died if there is no jobs in queue'
)
;
}
protected function execute(InputInterface $input, OutputInterface $output)
{
$output->writeln("\n<info>Beanstalk worker service started</info>");
$tubes = $this->getContainer()->get('app.job_manager.power_plant')->getTubes();
if ($input->getOption('one-check')) {
// run once
foreach ($tubes as $tubeName) {
if ($tubeName != "default") {
$this->getContainer()->get('app.queue_manager')->fetchQueue($tubeName);
}
}
$output->writeln("\n<info>Beanstalk worker completed check and stoped</info>");
} else {
// run forever
set_time_limit(0);
ini_set('xdebug.max_nesting_level', 1000);
if (!$this->lock()) {
$output->writeln('The command is already running in another process.');
return 0;
}
while (1) {
foreach ($tubes as $tubeName) {
if ($tubeName != "default") {
$this->getContainer()->get('app.queue_manager')->fetchQueue($tubeName);
}
sleep(0.1);
}
}
$output->writeln("\n<error>Beanstalk worker service has stoped</error>");
}
}
}
Than I run another command to create some extra workers with this functions:
public function startExtraWorkers(OutputInterface $output)
{
$numOfExtraWorkers = $this->getContainer()->getParameter('num_of_extra_beanstalk_workers');
for ($i=0; $i < $numOfExtraWorkers; $i++) {
$payload = [
"--one-check" => TRUE
];
$this->getContainer()->get('app.job_manager.queue')->createAndSendToQueue('beanstalk:worker:start', $payload, 10);
}
$output->writeln("\n<question>".$numOfExtraWorkers." extra benastalk workers started!</question>");
return TRUE;
}
public function createAndSendToQueue($command, $payload, $priority = 65536)
{
$jobData = $this->createJob($command, $payload);
return $this->job->enqueue($command, $jobData, $priority);
}
public function enqueue($job, array $args, $priority = 65536, $delay = 0, $ttr = 120)
{
if (!preg_match('/[a-z0-9\.]+/i', $job)) {
throw new InvalidArgumentException("Invalid job name");
}
$args = json_encode($args);
return $this->pheanstalk->put($args, $priority, $delay, $ttr);
}
And the problem is that if I run this command from terminal or with cron job it forks but if i run it like that with this function it do not work. I see that command has been executed but for some unknown reason it do not work.
If I executed this command i can see all commands has been executed bot they do not perform job like if i run the same command from terminal or with cron job:
ps ax |grep "beanstalk:worker:start --one-check"
Output (first one has been run from this function and second one with cron job. And only second one works):
31934 ? Ss 0:00 /bin/sh -c /usr/bin/php /var/www/mose-base/bin/console beanstalk:worker:start --one-check
31935 ? S 0:00 /usr/bin/php /var/www/mose-base/bin/console beanstalk:worker:start --one-check
Can any one give me some advice why is this not working like other commands? And why the same command run OK if i run it with cron job or inside terminal?
Thanks for help!
Try using:
/bin/sh -c '/usr/bin/php /var/www/mose-base/bin/console beanstalk:worker:start --one-check'
Mind the quotes.

Laravel Queue - remembering property state?

If a job failed, it will be pushed back to the Queue. Is there a way to remember the value of property in the job class when processing job again?
For example:
class MailJob extends Job
{
public $tries = 3;
public $status;
public function __construct()
{
$this->status = false; // set to false
}
/**
* Execute the job.
*/
public function handle()
{
$this->status = true;
// Assume job has failed, it went back to the Queue.
// status should be true when this job start processing again
}
}
If you want to rerun the failed process again on the same moment. you can do something like this.
Here the object is in memory while rerunning the job so data will be available.
I haven't verified it by running it , but hope it will work
class MailJob extends Job{
public $tries = 3;
public $status;
public function __construct()
{
$this->status = false; // set to false
}
/**
* Execute the job.
*/
public function handle()
{
$this->status = true;
// Assume job has failed, it went back to the Queue.
// status should be true when this job start processing again
$failed = processFailedisConfirm();
if $failed == true && $this->tries > -1 {
$this->tries = $this->tries - 1;
$this->handel();
}
}}
Example of processFailedisConfirm could be
public function processFailedisConfirm(){
// Core Process to be done in the Job
$state = (Do Some Api Call); // Here just example, you may send email
// Or can do the core Job Process
// And depending on the Response of the
// Process return true or false
// Is Job failed or not ?
if ( $state == "200" ){
return false; // Job is not failed
} else {
return true; // Job is failed
}
Logic of process is failed or not is depened on the operation you are doing. As i am doing an api call if i get response of 200 my process is successfull.
Otherwise process is failed.
This just example, sucess reponse of different api can be differnt as desigend by api designer.

Ubuntu 15 proceed codebird-php stream by cron not works

I have a laravel project with codebird-php lib there. And I trying to set a cron job to start scheduling my streaming commands. On my local machine with Ubuntu 16 it works perfectly, every minute creating a stream to twitter and on 55 second it dies.
But on the remoted VPS it not scheduled my command. I've tried it by cron, by tiping in terminal and have no efeect. Can anyone explain me where I've got a stupid mistake and what difference have in cron work Ubuntu 15.04 and Ubuntu 16.04?
Here is the code of commands:
connect command setting streams and start them
class ConnectToStreamingAPI extends Command
{
protected $signature = 'connect';
protected $run = [];
protected $users;
protected $description = 'Connect to the Twitter Streaming API to search tweets by keywords';
protected $twitterStream;
public function __construct(User $user)
{
$this->users = $user::where('id', '>', '1')->pluck('id')->toArray();
parent::__construct();
}
public function handle()
{
//set_time_limit(59);
$this->setRun();
$this->runUserStreamCommand();
return "Fired";
}
protected function setRun()
{
foreach ($this->users as $user) {
$this->addToRun($user);
}
}
protected function runUserStreamCommand()
{
foreach ($this->run as $process) {
$process->setPty(true);
$process->start();
}
$count = count($this->run);
while (count($this->run) > 0) {
foreach ($this->run as $key => $process) {
$childProcess = $process->getPid() + $count;
try {
$process->checkTimeout();
} catch (ProcessTimedOutException $e) {
exec("kill -15 " . $childProcess);
}
if (!$process->isRunning()) {
unset($this->run[$key]);
}
}
}
}
protected function addToRun($user_id)
{
$this->run[] = new Process("php /path/to/artisan stream $user_id", null, null, null, 50);
}
}
And here is Stream command
class StreamingCommands extends Command
{
protected $signature = 'stream {id}';
protected $description = 'create stream to twitter for one user';
public function __construct()
{
parent::__construct();
}
public function handle()
{
$id = $this->argument('id');
$user = User::find($id);
Codebird::setConsumerKey($user->consumer_key, $user->consumer_secret);
$stream = new TwitterStream();
$stream->setToken($user->twitterData->token, $user->twitterData->token_secret);
$keywordArray = array();
$ids = array();
foreach ($user->campaigns as $campaign) {
$ids[] = $campaign->id;
$keywordArray = array_merge($keywordArray, Keyword::whereCampaignId($campaign->id)->pluck('keyword')->toArray());
}
$keywords = Keyword::whereIn('campaign_id', $ids)->where('rejected', 0)->get();
$stream->setKeywords($keywords);
$stream->setStreamingCallback('processTweet');
$keywords = implode(',', $keywordArray);
$this->info("start stream to user $id");
$stream->statuses_filter('track=' . $keywords);
}
}
And cron job
* * * * * php /path/to/artisan schedule:run >> /dev/null 2>&1
And schedule is just a
$schedule->command('connect')->everyMinute();
As for the comments and the changes, you clearly have two different systems 16 and 15, with different software and dependencies installed. Unfortunately, there's no simple way of fixing this kind of glitch, you have to check everything, it can still be a not listed PHP dependency or even an operating system dependency and/or user acces rights (you are starting and killing processes, is it working or abending?), but your app is working fine in 16, so you can probably forget about Laravel and Composer and focus on PHP and the OS layers.
The thing is, from here, without acces to your server, it's very hard to guess, so sorry if it doesn't help much.

Listen laravel 'illuminate.query' events in artisan console commands

I would like to log laravel queries. I know that we can listen to laravel queries using the following code:
Event::listen('illuminate.query', function ($query, $bindings, $time, $name) {
// Log queries
};
I would like to place above code in an artisan command so that I only listen to queries when this program is running, and not all the time. I wrote the following code.
<?php
// ... namespace and use statements
class QueryListener extends Command
{
protected $signature = 'pots:query-listen';
protected $description = 'Runs forever and logs all database queries.';
public function __construct()
{
parent::__construct();
}
private function logDatabaseQueries()
{
Event::listen('illuminate.query', function ($query, $bindings, $time, $name) {
Log::info(PHP_EOL.$query.PHP_EOL);
});
}
public function handle()
{
$this->logDatabaseQueries();
echo 'Listening for database queries: Type \'stop\' to stop execution.'.PHP_EOL;
$stopExecution = false;
while ($stopExecution === false) {
$handle = fopen('php://stdin', 'r');
$line = fgets($handle);
if (strtolower(trim($line)) === 'stop') {
fclose($handle);
echo 'Aborting script.'.PHP_EOL;
$stopExecution = true;
}
}
}
}
My expectation was that as soon as I run this command using artisan, the logDatabaseQueries() method will start listening for events. However this is not working because I never receive any query events. Am I thinking in the right direction?
You can listen to DB query using this:
DB::listen(function($query) {
// Log this
\Log::info($query->sql);
});
For Those How Are Interested
Listen for illuminate queries
// routes web.php
\Event::listen('illuminate.query', function($sql) {
var_dump($sql);
\Log::info($sql);
});
And For More And Advanced Lisen Here

sending emails from cron tasks in symfony

I am trying to send emails using cron tasks with Symfony , so my question is how would I execute my command with using swiftmailer in my execute function ?Thanks in advance
I want the swiftmailer to be in my execute method, so I can send emails based on cron tasks
$mail = \Swift_Message::newInstance();
$mail->setFrom('from#example.com')
->setTo('to#example.com')
->setSubject('Email subject')
->setBody('email body, can be swift template')
$this->get('mailer')->send($mail);
my CronTasksRunCommand
protected function execute(InputInterface $input, OutputInterface $output)
{
$output->writeln('<comment>Running Cron Tasks...</comment>');
$this->output = $output;
$em = $this->getContainer()->get('doctrine.orm.entity_manager');
$crontasks = $em->getRepository('AppBundle:CronTask')->findAll();
foreach ($crontasks as $crontask) {
// Get the last run time of this task, and calculate when it should run next
$lastrun = $crontask->getLastRun() ? $crontask->getLastRun()->format('U') : 0;
$nextrun = $lastrun + $crontask->getInterval();
// We must run this task if:
// * time() is larger or equal to $nextrun
$run = (time() >= $nextrun);
if ($run) {
$output->writeln(sprintf('Running Cron Task <info>%s</info>', $crontask));
// Set $lastrun for this crontask
$crontask->setLastRun(new \DateTime());
try {
$commands = $crontask->getCommands();
foreach ($commands as $command) {
$output->writeln(sprintf('Executing command <comment>%s</comment>...', $command));
// Run the command
$this->runCommand($command);
}
$output->writeln('<info>SUCCESS</info>');
} catch (\Exception $e) {
$output->writeln('<error>ERROR</error>');
}
// Persist crontask
$em->persist($crontask);
} else {
$output->writeln(sprintf('Skipping Cron Task <info>%s</info>', $crontask));
}
}
// Flush database changes
$em->flush();
$output->writeln('<comment>Done!</comment>');
}
If your Command class extends ContainerAwareCommand class, then just replace
$this->get('mailer')->send($mail);
with
$this->getContainer()->get('mailer')->send($mail);

Categories