Consume raw json queue messages in Laravel - php

Question

Normally Laravel expects that it queued up any messages that it later consumes. It creates a payload with a job attribute that later indicates how to handle the queue message. When you do queue up jobs with Laravel, and later process them with Laravel, it works great!
However, I have some non-Laravel apps that are posting json messages to a queue. I need Laravel to pick up these messages and handle them.
I can write a command bus job to handle the messages, but I haven't been able to figure out how to tell queue:work to send the messages on to my specific handler.
It seems Laravel has a hard assumption that any queue messages it is asked to handle will be properly formatted, serialized, and structured the way it expects them to be.
How can I have Laravel pick up these raw json payloads, ignore the structure (there's nothing there for it to understand), and simply hand the payload off to my handler?
For example, if I have a queue message similar to:
{
"foo" : "bar"
}
So again, there's nothing for Laravel to inspect or understand here.
But I have a job handler that knows how to handle this:
namespace App\Jobs;
class MyQueueHandler {
public function handle($payload) {
Log::info($payload['foo']); // yay!
}
}
Now how to get queue:work and queue:listen to simply hand off any payloads to this App\Jobs\MyQueueHandler handler, where I can do the rest on my own?

Answer 1

If you're using Laravel 5.6+, check out this package.

Answer 2

You didn't specify which version of Laravel, so I'm guessing 5.1 (huge difference in how this is handled in L4.2 and L5.x).
If you've already set up your App\Jobs\MyQueueHandler, and want to queue up a job from a controller, using any data you wish, you can just do this:
use App\Jobs\MyQueueHandler;
class MyController
{
public function myFunction()
{
$this->dispatch(new MyQueueHandler(['foo' => 'bar']));
}
}
In your MyQueueHandler-class, the payload actually enters your constructor. The handle-method is still fired when your queue is processed though. You can however use parameters on your handle-method if you rely on dependancy injection (read more here, just above "When Things Go Wrong") So something like this should do it:
namespace App\Jobs;
class MyQueueHandler
{
protected $payload;
public function __construct($payload)
{
$this->payload = $payload;
}
public function handle() {
Log::info($this->payload['foo']); // yay!
}
}
Note: If you want to dispatch the job from outside a main controller (that inherits from the standard App\Http\Controller-class, use the DispatchesJobs trait;
MyClass
{
use DispatchesJobs;
public function myFunction()
{
$this->dispatch(new MyQueueHandler(['foo' => 'bar']));
}
}
(Code tested with Laravel 5.1.19 and the beanstalkd queue-adapter).

Answer 3

What you ask for is not possible as Laravel tries to execute the Gearman payload (see \Illuminate\Bus\Dispatcher).
I was in the same situation and just created a wrapper command around the Laravel job class. This is not the nicest solution as it will re-queue events, coming on the json queue, but you don't have to touch existing job classes. Maybe someone with more experience knows how to dispatch a job without actually sending it over the wire again.
Lets assume we have one regular Laravel worker class called GenerateIdentApplicationPdfJob.
class GenerateIdentApplicationPdfJob extends Job implements SelfHandling, ShouldQueue
{
use InteractsWithQueue, SerializesModels;
/** #var User */
protected $user;
protected $requestId;
/**
* Create a new job instance.
*
* QUEUE_NAME = 'ident-pdf';
*
* #param User $user
* #param $requestId
*/
public function __construct(User $user, $requestId)
{
$this->user = $user;
$this->requestId = $requestId;
}
/**
* Execute the job.
*
* #return void
*/
public function handle(Client $client)
{
// ...
}
}
To be able to handle this class, we need to provide the constructor arguments our own. Those are the required data from our json queue.
Below is a Laravel command class GearmanPdfWorker, which does all the boilerplate of Gearman connection and json_decode to be able to handle the original job class.
class GearmanPdfWorker extends Command {
/**
* The console command name.
*
* #var string
*/
protected $name = 'pdf:worker';
/**
* The console command description.
*
* #var string
*/
protected $description = 'listen to the queue for pdf generation jobs';
/**
* #var \GearmanClient
*/
private $client;
/**
* #var \GearmanWorker
*/
private $worker;
public function __construct(\GearmanClient $client, \GearmanWorker $worker) {
parent::__construct();
$this->client = $client;
$this->worker = $worker;
}
/**
* Wrapper listener for gearman jobs with plain json payload
*
* #return mixed
*/
public function handle()
{
$gearmanHost = env('CB_GEARMAN_HOST');
$gearmanPort = env('CB_GEARMAN_PORT');
if (!$this->worker->addServer($gearmanHost, $gearmanPort)) {
$this->error('Error adding gearman server: ' . $gearmanHost . ':' . $gearmanPort);
return 1;
} else {
$this->info("added server $gearmanHost:$gearmanPort");
}
// use a different queue name than the original laravel command, since the payload is incompatible
$queueName = 'JSON.' . GenerateIdentApplicationPdfJob::QUEUE_NAME;
$this->info('using queue: ' . $queueName);
if (!$this->worker->addFunction($queueName,
function(\GearmanJob $job, $args) {
$queueName = $args[0];
$decoded = json_decode($job->workload());
$this->info("[$queueName] payload: " . print_r($decoded, 1));
$job = new GenerateIdentApplicationPdfJob(User::whereUsrid($decoded->usrid)->first(), $decoded->rid);
$job->onQueue(GenerateIdentApplicationPdfJob::QUEUE_NAME);
$this->info("[$queueName] dispatch: " . print_r(dispatch($job)));
},
[$queueName])) {
$msg = "Error registering gearman handler to: $queueName";
$this->error($msg);
return 1;
}
while (1) {
$this->info("Waiting for job on `$queueName` ...");
$ret = $this->worker->work();
if ($this->worker->returnCode() != GEARMAN_SUCCESS) {
$this->error("something went wrong on `$queueName`: $ret");
break;
}
$this->info("... done `$queueName`");
}
}
}
The class GearmanPdfWorker needs to be registered in your \Bundle\Console\Kernel like this:
class Kernel extends ConsoleKernel
{
protected $commands = [
// ...
\Bundle\Console\Commands\GearmanPdfWorker::class
];
// ...
Having all that in place, you can call php artisan pdf:worker to run the worker and put one job into Gearman via commandline: gearman -v -f JSON.ident-pdf '{"usrid":9955,"rid":"ABC4711"}'
You can see the successful operation then
added server localhost:4730
using queue: JSON.ident-pdf
Waiting for job on `JSON.ident-pdf` ...
[JSON.ident-pdf] payload: stdClass Object
(
[usrid] => 9955
[rid] => ABC4711
)
0[JSON.ident-pdf] dispatch: 1
... done `JSON.ident-pdf`
Waiting for job on `JSON.ident-pdf` ...