I understand the listener passes a Job instance to my worker and i can use that instance to delete the job, but how can i delete a job outside of worker?
Consider this scenario:
$job_id=Queue::push('DoTheJob', array('data'=>array(1,2,3)));
If(!someotherjobdone){
// delete job from Queue with job_id
?
}
Thank you
I'm not certain if this is correct, but I think you want to 'pop' the job off the queue:
$job_id=Queue::push('DoTheJob', array('data'=>array(1,2,3)));
If(!someotherjobdone){
Queue::pop($job_id);
}
If that doesn't work you could try:
$queue=Queue::getQueue($job_id);
Queue::pop($queue);
In my case, I am implementing SQS Queue. If I have the 'ReceiptHandle' of the message (an ID associated with a specific instance of receiving the message), I can delete it by accessing the SqsClient object through the SQSManager, because the manager is accessible by the IoC container.
$queue_manager = App::make("queue");
$sqs_queue = $queue_manager->connection('sqs');
$sqs_client = $sqs_queue->getSqs();
$sqs_client->deleteMessage(['QueueUrl' => $queue_url, 'ReceiptHandle' => $receipt_handle]);
Related
I am wondering how to deal with this. I have a webhook endpoint which responds to a webhook call from Github.
It starts a long running process in where it clones the repository from which the webhook call was made.
/**
* The webhook endpoint.
*
* #param Request $request
* #return mixed
* #throws \Exception
*/
public function webhook(Request $request)
{
// The type of GitHub event that we receive.
$event = $request->header('X-GitHub-Event');
$url = $this->createCloneUrl();
$this->cloneGitRepo($url);
return new Response('Webhook received succesfully', 200);
}
The problem with this is that Github gives an error when the response is not provided soon enough.
We couldn’t deliver this payload: Service Timeout
This is rightfully so because my cloneGitRepo method is simply blocking the execution of the response and it is taking too long.
How can I still deliver a response to acknowledge to Github that the webhook call has been made successfully and start my long running process?
I am using Laravel for all of this with Redis, maybe something can be accomplished there? I am open to all suggestions.
What you're looking for is a queued job. Laravel makes this very easy with Laravel Queues.
With queues, you setup a queue driver (database, redis, Amazon SQS, etc), and then you have one to many queue workers that are continuously running. When you put a job on the queue from your webhook method, it will be picked up by one of your queue workers and run in a separate process. The act of dispatching a queued job to a queue is very quick, though, so your webhook method will return quickly while the real work is being done by the queue worker.
The linked documentation has all the details, but the general process will be:
Setup a queue connection. You mention you're already using redis, I would start with that.
Use php artisan make:job CloneGitRepo to create a CloneGitRepo job class.
It should implement the Illuminate\Contracts\Queue\ShouldQueue interface so that Laravel knows to send this job to a queue when it is dispatched.
Make sure to define properties on the class for any data you pass into the constructor. This is necessary so the worker can rebuild the job correctly when it is pulled off the queue.
The queue worker will call the handle() method to process the job. Any dependencies can be type hinted here and they will be injected from the IoC container.
To dispatch the job to the queue, you can either use the global dispatch() helper function, or call the static dispatch() method on the job itself.
dispatch(new CloneGitRepo($url));
CloneGitRepo::dispatch($url);
So, your webhook would look like:
public function webhook(Request $request)
{
// The type of GitHub event that we receive.
$event = $request->header('X-GitHub-Event');
$url = $this->createCloneUrl();
CloneGitRepo::dispatch($url);
return new Response('Webhook received succesfully', 200);
}
I have 1000 queues with specific names. so I want to process these queues with one broker. is it possible?
the queue names is stored in mysql db so I should fetch theme and run the broker for each one. and of course it should run asynchronously and should be able to pass the queued item to a idle broker. is this possible? or I should make 1000 files with specific queue names as brokers?
Update:
this is a picture of my queues. the queues should run in a parallel manner not a serial one. so the users are producer and the worker is consumer that runs the send_message() method;
I can show you how to it with enqueue library. I must warn you, there is no way to consume messages asynchronously in one process. Though you can run a few processes that serve a set of queues. They could be divided into groups by the queue importance.
Install the AMQP transport and consumption library:
composer require enqueue/amqp-ext enqueue/enqueue
Create a consumption script. I assume that you have an array of queue names already fetched from DB. They are stored in $queueNames var. The example bound the same processor to all queues but you can set different ones, of course.
<?php
use Enqueue\AmqpExt\AmqpConnectionFactory;
use Enqueue\Consumption\QueueConsumer;
use Enqueue\Psr\PsrMessage;
use Enqueue\Psr\PsrProcessor;
// here's the list of queue names which you fetched from DB
$queueNames = ['foo_queue', 'bar_queue', 'baz_queue'];
$factory = new AmqpConnectionFactory('amqp://');
$context = $factory->createContext();
// create queues at RabbitMQ side, you can remove it if you do not need it
foreach ($queueNames as $queueName) {
$queue = $context->createQueue($queueName);
$queue->addFlag(AMQP_DURABLE);
$context->declareQueue($queue);
}
$consumer = new QueueConsumer($context);
foreach ($queueNames as $queueName) {
$consumer->bind($queueName, function(PsrMessage $psrMessage) use ($queueName) {
echo 'Consume the message from queue: '.$queueName;
// your processing logic.
return PsrProcessor::ACK;
});
}
$consumer->consume();
More in doc
I am using laravel 5.1 and i am using the dispatch method to push the job onto the queue.
But there are two kind of jobs and i have created and two queues for that in sqs.
How should i achieve this?
In order to specify the queue you need to call onQueue() method on your job object, e.g.:
$job = (new SendReminderEmail($user))->onQueue('emails');
$this->dispatch($job);
If you want to send the job to a connection other than default, you need to do fetch connection manually and send the job there:
$connection = Queue::connection('connection_name');
$connection->pushOn('queue_name', $job)
This worked for me.
//code to be used in the controller (taken from #jedrzej.kurylo above)
$job = (new SendReminderEmail($user))->onQueue('emails');
$this->dispatch($job);
I think this dispatches the job on to the queue named "emails".
To execute the job dispatched on 'emails' queue:
//Run this command in a new terminal window
php artisan queue:listen --queue=emails
I'd suggest this:
app('queue')->connection('connection_name')->pushOn('queue_name', $job);
From here: In Laravel how to create a queue object and set their connection without Facade
So Laravel 5 was finally released yesterday with the final implementation of the command bus but I was wandering, what's the real difference in using a command bus over event mechanisms that we have in the previous releases?
Ok, I see the reason that it can be used to create commands from Request objects which is pretty useful but beyond that it seems to behave in a similar way even down to the whole queuing functionality for events now?
Can you please provide examples of use cases and where the pros and cons of either are?
Commands are things about to happen right now. i.e. "CreateUser"
Events are things that have just occured right now - i.e. "UserSuccessfullyCreated"
The differences appear minor - but have some key differences.
Commands must be specifically called/dispatched. I.e. if you want to
do CommandX - you must call CommandX somewhere.
Events respond to an event firing anywhere in your application.
The great thing is multiple event handling classes can respond to the
same event.
Lets do an example to illustrate it best. Lets say we create a user, and we want to send them a welcome email and also update our newsletter list.
In a Command Scenario would would do
AdminController {
function create() {
Bus::dispatch(new CreateUser(Auth::user());
}
}
then in our CommandClass - we would do
public function handle(CreateUser $auth)
{
// 1. Create the user here
// 2. Send welcome email
// 3. Update our newsletter
}
But if we use events - we would do something like this in our CommandClass
public function handle(CreateUser $auth)
{
// 1. Create the user here
Event::fire(new UserWasCreated($user));
}
then we can create as many events as we want to listen to that event and do something:
EventClassA
Event::listen('UserWasCreated', function($event)
{
// 2. Send welcome email
});
EventClassB
Event::listen('UserWasCreated', function($event)
{
// 3. Update our newsletter
});
The great thing is separation of concerns. The command "createuser" now does not need to worry itself about what happens after a user is created. It just needs to CreateUser.
Also - if we want to add another function after a user signs up - say enter them in a lotto draw - you can just add another Event Class and add a new event listener.
EventClassC
Event::listen('UserWasCreated', function($event)
{
// 4. Register them in lotto
});
Notice how we didnt need to touch the command CreateUser class code at all? This provides a true separation concerns of classes in a OOP style approach.
I just want to share my understanding of this concept on top of the correct answer:
The main difference is that Commands can change a Model state, while Events just react to a state change.
COMMANDS:
Commands in Laravel represent the implementation of the Command design pattern.
The main adventages of Commands:
The can be accessed from anywhere
They are very easy to read by any other developer
To create a Command in Laravel 5:
You need to generate a command DTO (which can implement the SelfHandling interface). Using php artisan make:command {command-name}
Example: php artisan make:command Course/PostCourseCommand
The naming convention for commands: speak the business language and add postfix Command to it
To call (dispatch) the command from you controller, you can use:
$this->dispatch(new PostCourseCommand())
or
Bus::dispatch(new PostCourseCommand());
Side Note:
The "dispatch from request” feature is a nice way to skip passing the variables to the command constructor one by one, instead it will resolve this for you:
Example:
$test_request = Request::create('/test', 'GET', [
'name' => 'Mahmoud',
'nickname' => 'Mega'
]);
$result = Bus::dispatchFrom(
CreateCourse::class, $test_request
);
Finally:
You can separate the handler function and it’s logic from the command DTO to the Handlers directory, to do so:
Generate a command handler via artisan
art handler:command --command="Course/PoatCourseCommand"
remove the SelfHandling interface from the Command class so it will search for a handler to handle it.
EVENTS:
Events in Laravel represent the implementation of the Observer design pattern.
To create an Event in Laravel 5:
use artisan: art make:event {event-name}
Example: art make:event LogSomething
generate an event handler for that event
art handler:event LogSomething --event="LogSomething"
register the event and it’s handler in the event service provider (app/Providers/EventServiceProvider.php)
Example:
protected $listen = [
\Zzz\Events\LogSomething::class => [ // event.name
\Zzz\Handlers\Events\LogSomething::class, //EventListener
],
],
To call (fire) an Event:
use:
Event::fire(New LogSomething());
or you can use the event helper
event(New LogSomething());
Side Note:
alternatively you can generate an event by simply registering the event in the service provider then running this command.
php artisan event:generate << this will automatically add the two classes for you
Also you can listen to an event without creating an event handler or registering a lister in the listeners array, by just going to the event service prover and inside the boot function writing your event and its action (NOT RECOMMENDED). Example:
Event::listen('XXX\Events\DoSomethingElse', function($event)
{
dd('handle me :)');
});
Finally: you can queue an event or even subscribe to multiple events from within the class itself..
Hi i am new to GAE task queues, I created one queue with name anchorextractor, this is showing in queues list.
Then i created a task with the url ('/worker/extractor/1'). after creating if i echo the name of task, its showing name ( task3 ). After i checked the queues list is Taskqueue page, Tasks under this queue is 0. Actually there are 3 tasks created. I tried with all possibilities. I think i explained well and no need of code here. If u need more explination i will give. Please anyone help me. (I am updating the question with code for reference, following is the code):
require_once 'google/appengine/api/taskqueue/PushTask.php';
use google\appengine\api\taskqueue\PushTask;
require_once 'google/appengine/api/taskqueue/PushQueue.php';
use google\appengine\api\taskqueue\PushQueue;
$queue = new PushQueue('tagextractor');
$task = new PushTask('/worker/anchorextractor/1', ['content_id' => 'aa', 'content_type' => 'aa']);
echo "Task Name = ".$task_name = $task->add();
$queue->addTasks([$task]);
Try this syntax instead, it will log the new tasks name to the AppEngine logs as proof that the task was created:
require_once 'google/appengine/api/taskqueue/PushTask.php';
use \google\appengine\api\taskqueue\PushTask;
$task_name = (new PushTask('/worker/anchorextractor/1', array(
'content_id' => 'aa',
'content_type' => 'aa'
)))->add("tagextractor");
syslog(LOG_INFO, "new task=".$task_name);
Tasks do get processed very quickly, so it is sometimes difficult to "see" them in the queue, you can however go to the queue in the admin console and pause it, the tasks will then build up until you either run it manually or resume the queue.