I have a PHP project (Symfony2) that uses RabbitMQ. I use its as simple message queue to delay some jobs (sending mails, important data from APIs). The consumers run on the webserver and their code is part of the webserver repo - they are deployed in the same with with the web.
The questions are:
How do I start the consumers as daemons and make sure they always run?
When deploying the app, how do I shut down consumers "gracefully" so that they stop consuming but finish processing the message they started?
If it's any important, for deployment I use Capifony.
Thank you!
It maybe worth looking at something supervisord which is written in python. I've used it before for running workers for Gearmand which is a job queue that fullfils a similar role to the way your using RabbitMQ.
Related
I have been working on a php mvc web application using codeigniter and need to process some long running tasks.
I checked through several options (RabbitMQ, Gearman, IronMQ etc) and decided to use Gearman of it's simplecity. I went through the samples and tutorials in gearman.org which shows how to start a GearmanWorker using worker.php.
my concern is, in mvc architecture where does this GearmanWorker is initiated and started?
Does it started through a controller method OR
Do we need to initiate the GearmanWorker from cli(console)? If it's started from cli then how to handle if the already started worker has stopped for some reason when we make a GearmanClient->do('some task')
a similar question but not clear enough for me
I wouldn't recommend to start worker from the controller. You can start several workers distributed over network and using workers text command for monitoring purpose. gearmand dispatch a job to the next idle worker.
Maybe SUBMIT_JOB_BG is a good option for you to avoid web server timeout, if job execution takes long.
I feel a little bit silly for asking this question but I can't seem to find an answer on the internet for this problem. After searching for several hours I figured out that on a linux server you use Supervisor to run "php artisan queue:listen" (either with or without daemon) continuously on your website to handle jobs pushed to the queue. This is all well and good, but what if I want to do this on a Windows Azure web app? After searching around the solutions I found were:
Make a chron job to run "php artisan queue:listen" every minute (or every X minutes), I really dislike this solution and wanted to avoid it specially if the site gets more traffic;
Add a WebJob that runs "php artisan queue:listen" continuously (the problem here is I don't know how to write the script for the WebJob...);
I want to ask you guys for help on to know which of these is the correct solution, if there is a better one and if the WebJob is the best one how do I write the script for this? Thanks in advance.
In short, Supervisor is a modern alternative to nohup (no hang up) with a few other bits and pieces tacked on. In short, there's other resources that can keep a task running in the background (daemon) and the solution I use for Windows based projects (very few tbh) is Forever which I discovered via: https://stackoverflow.com/a/18226392/5912664
C:\myprojectroot > forever -c php artisan queue:listen --queue=some_nice_queue --tries=3
How?
Install node for Windows, then with npm install Forever
C:\myprojectroot > npm install -g forever
If you're stuck for getting Node running on Windows, I recommend the Windows Package Manager, Chocolatey
https://chocolatey.org/packages?q=node
Be sure to check for any logfiles that Forever creates, as I had left one long enough to consume 30Gb of disk space!
For Azure you can make a new webjob to your web app, and upload a .cmd file including a command like this.
php %HOME%\site\wwwroot\artisan queue:work --daemon
and defining that as a triguered and 0 * * * * * frequency cron.
that way work for me.
best.
First of all you cannot use a WebJob with Laravel on Azure. The Azure PHP Web App is hosted on Linux. WebJobs do not work with Linux at this moment.
The best way to do chron jobs in Laravel on Azure is to create an Azure Logic App. You use the Recurrence trigger and then a HTTP action to send a POST request to your Laravel Web App. You use this periodic heartbeat to run whatever actions you need to do. Be sure to add authentication to your POST request.
The next problem you will have is that POST will be synchronous so the work you are doing cannot be extensive or your HTTP request will time out or you will reach the time limit on PHP scripts (60 seconds).
The solution is not Laravel Jobs because here again you need something running in the background to process the queues.
The solution is also not PHP threads. The standard Azure PHP Web App does not support PHP Threads. You can of course build your own Web App and enable PHP threads, but this is really swimming upstream.
You simply have to live with synchronous logic. So the work you are doing with the heartbeat should take no more than about 60 seconds.
If you need more extensive processing then you really need to off load it to another place: another Web App, an Azure Function, etc.
But why not do that in the first place? The reason is cost and complexity. If you have something simple...like a daily report...you simply connect the report to the heartbeat and all the facilities for producing the report are right there in Laravel. To separate the daily report into its own container would require setup and the Web App it runs in would incur costs...not worth it in my view for something simple.
How to connect phalconphp as consumer with rabbitmq?
As I understood I need to have a background process running for phalconphp to be able to listen for events/messages from rabbitmq and process some time intensive tasks (sending mail, writing to logs).
What would fire the consumer (in Phalcon), maybe supervisord?
I found some article that states just to run php worker.php containing a listen method:
http://www.sitepoint.com/php-rabbitmq-advanced-examples/
While just running php worker.php will work, if you don't use a supervisor service, and just use a while(1) and send it to the background, there is no way to handle the process dying.
supervisord is recommended because you need to daemonize the process, and ensure that if it dies, or if the system is rebooted, that the process will be restarted.
You might also want to check into upstart. It can achieve the same goal.
Im looking to build a distributed video encoding cluster of a few dozen machines. Ive never worked with a messaging queue before, but the 2 that I started playing around with were Gearman and Beanstalkd.
Beanstalk seems to be a lot simpler and easier to use than Gearman, but its not as feature rich as.
One thing I don't understand is... how do you spawn new workers on all the servers? I plan to use php. Is it as simple as running worker.php in CLI with "&" and just have it sit there waiting for work?
I noticed gearman doesn't actually kill the process after a job is done, but Beanstalk does, so I have to restart the script after every job, on every server.
Currently Im more inclined to use Beanstalk, the general flow of things I planned was:
Run a minutely cron on each server that checks if there are pre-defined amount of workers running. If its less than supposed to be, spawn new worker processes. Each process will take roughly 2-30 minutes.
Maybe I have a flaw in my logic here? Let me know what would be a "better" or "proper" way of doing this?
Terminology I will use just to try and be clear...
There is the concept of a producer and a consumer. The producer generates jobs that are put on a queue (i.e. the beanstalk service) that is then read by a consumer.
There are multiple ways to write a consumer. You can either every x time frame via a cron job run the task or just have a consumer running in a while 1 loop via php (or what have you).
Where to install the service is really dependent on what you are going after. For me I normally install the service either on a consumer(s) or on its separate box (with sometimes the latter being overkill depending on your needs).
If you want durability on the queue side then you should use Beanstalk's binlog parameter (-b ). If something happens to your beanstalk service this will allow you to restart with minimal loss of data in the queues (if not no information). Durability on the producer side can come from having multiple queues to try against.
We have a large web application built on PHP. This application allows scheduling tweets and wall posts and there are scheduled emails that go out from the server.
By 'scheduled', I mean that these are PHP scripts scheduled to run at particular time using cron. There are about 7 PHP files that do the above jobs.
I have been hearing about Message Queues. Can anyone explain if Message Queues are the best fit in this scenario? Do Message Queues execute PHP scripts? or do we need to configure this entirely differently? What are the advantages / disadvantages?
Using Crontab to make asynchronous tasks (asynchronous from your PHP code) is a basic approach where using a job/task queue manager is an elaborate one and give you more control, power and scalability/elasticity.
Crontab are very easy to deal with but does not offer a lot of functionalities. It is best for scheduled jobs rather than for asynchronous tasks.
On the other hand, deploying a Task queue (and its message broker) require more time. You have to choose the right tools first then learn how to implement them in your PHP code. But this is the way to go in 2011.
Thank God, I don't do PHP but have played around with Celery (coupled with RabbitMQ) on Python projects ; I am sure you can find something similar in the PHP world.