Execute shell from controller CakePHP 3.x - php

I have a specific task in a CakePHP Shell and it's executed by a CRON job. But I want the users to be able to execute it from a web interface (like a button or something like this) whenever he wants.
So, my question is, is this possible to execute a shell from a controller ?
Emulate this in a controller:
bin/cake MyShell
I know it was possible in the previous versions of CakePHP, but I didn't find something related to this in the newest version. And use exec("bin/cake MyShell") seems really dirty to me.

create a shell object , the call any of its function to want to excute
$myShell = new \App\Shell\MyShell;
$myShell->anyShellFun();

in order to call shell from your controller function you need to do this in your controller function :
namespace App\Controller;
use App\Controller\AppController;
use Cake\Console\ShellDispatcher; //use this shell dispacher
/**
* Example Controller
*
*/
class ExampleController extends AppController
{
/**
* Index method
*
* #return \Cake\Network\Response|null
*/
public function index()
{
$shell = new ShellDispatcher(); //create object of your Shell Dispatcher
$output = $shell->run(['cake', 'foo']); //here foo is your shell name
if (0 === $output) {
$this->Flash->success('Success from shell command.');
} else {
$this->Flash->error('Failure from shell command.');
}
return $this->redirect('/');
}
}
hope this answer your question, if any problem leave a comment.
thank you.

If you can't mitigate the need to do this as dogmatic suggests then, read on.
So you have a (potentially) long-running job you want to perform and you don't want the user to wait.
As the PHP code your user is executing happens during a request that has been started by Apache, any code that is executed will stall that request until it completion (unless you hit Apache's request timeout).
If the above isn't acceptable for your application then you will need to trigger PHP outwith the Apache request (ie. from the command line).
Usability-wise, at this point it would make sense to notify your user that you are processing data in the background. Anything from a message telling them they can check back later to a spinning progress bar that polls your application over ajax to detect job completion.
The simplest approach is to have a cronjob that executes a PHP script (ie. CakePHP shell) on some interval (at minimum, this is once per minute). Here you can perform such tasks in the background.
Some issues arise with background jobs however. How do you know when they failed? How do you know when you need to retry? What if it doesn't complete within the cron interval.. will a race-condition occur?
The proper, but more complicated setup, would be to use a work/message queue system. They allow you to handle the above issues more gracefully, but generally require you to run a background daemon on a server to catch and handle any incoming jobs.
The way this works is, in your code (when a user registers) you insert a job into the queue. The queue daemon picks up the job instantly (it doesn't run on an interval so it's always waiting) and hands it to a worker process (a CakePHP shell for example). It's instant and - if you tell it - it knows if it worked, it knows if it failed, it can retry if you want and it doesn't accidentally handle the same job twice.
There are a number of these available, such as Beanstalkd, dropr, Gearman, RabbitMQ, etc. There are also a number of CakePHP plugins (of varying age) that can help:
cakephp-queue (MySQL)
CakePHP-Queue-Plugin (MySQL)
CakeResque (Redis)
cakephp-gearman (Gearman)
and others.
I have had experience using CakePHP with both Beanstalkd (+ the PHP Pheanstalk library) and the CakePHP Queue plugin (first one above). I have to credit Beanstalkd (written in C) for being very lightweight, simple and fast. However, with regards to CakePHP development, I found the plugin faster to get up and running because:
The plugin comes with all the PHP code you need to get started. With Beanstalkd, you need to write more code (such as a PHP daemon that polls the queue looking for jobs)
The Beanstalkd server infrastructure becomes more complex. I had to install multiple instances of beanstalkd for dev/test/prod, and install supervisord to look after the processes).
Developing/testing is a bit easier since it's a self-contained CakePHP + MySQL solution. You simply need to type cake queue add user signup and cake queue runworker.

Related

Task/Job scheduling on PHP

Our system based on Zend 1 has a background process running as daemon to check hourly whether user has expired or not, if yes, the system will send verification email to these expired users.
The code is really simple:
while(1) {
$worker = ... //get verification worker
$worker->run();
sleep(3600); //run hourly
}
We would like to look for better approach with free tool to replace this code to schedule job in the system. Zend Job Queue is really good approach but it is commercial.
My thinking on two below approaches as Zend Job Queue alternative:
Instead of running daemon, configure cron jobs on crontab to run PHP script hourly, but I am not sure whether it is better on above code or not.
Run node cron on nodejs server to call PHP script directly.
I also have looked on Gearman and beanstalked whether they supports on job scheduling but really stumped on this way.
Please suggest if there is any better approach to schedule task on PHP.
Cron would definately be the preferred way to automate tasks, and not a php script in an infinite loop.
see : http://en.wikipedia.org/wiki/Cron
You many also want to consider using Zend Server which includes a component called Zend Job Queue which provides offline asynchronous processing of tasks and activities.
see : http://files.zend.com/help/Zend-Server-6/content/jobs_component.htm
see : http://www.zend.com/en/products/server
0 * * * * /usr/bin/php -q /yourpath/verification_worker.php
you can use crontab to run this task every hour
Hope this helps

Running multiple Symfony commands at once

I have implemented a command in my Symfony setup which grabs a job from the DB and then processes it.
How can I run multiple instances of command at once, to get through jobs quicker. I know that multithreading is not supported in PHP but seeing as the command is called from the shell, I was wondering if there was a workaround.
Call command using:
app/console job:process
The way I would solve this is to use a work queue with multiple workers. It's easier to manage and scale than manually running multiple processes and worrying about concurrency.
The simplest general-purpose queue I've found for working with php/symfony is beanstalkd which you can integrate into symfony2 with the LeezyPheanstalkBundle
In general, I'd suggest using enqueue library. You can choose from a variety of transports available, from the simplest like filesystem and Doctrine DBAL to real once like RabbitMQ and Amazon SQS.
Regarding the consumers, you need sort of process manager. There several options:
http://supervisord.org/ - You need extra service. It has to be configured properly.
A pure PHP process manager like this. Based on Symfony process component and pure PHP code. It can handle process reboot, correct exit on sigterm signal and a lot more.
A php\swoole process manager like this. It requires a swoole PHP extension but it is performance is amazing.
I have written a blog post on how to solve this exact problem. https://plume.baucum.me/~/Absolutely/running-multiple-processes-simultaneously-in-a-symfony-command
It is much too long to rehash everything here, but the basic concept is that your command optionally takes in the job's ID. The command will check if the ID was given. If not then it will grab all the jobs from the DB, loop over them, and recall itself with the job ID parameter. As each command is kicked off you store it in an array, and if the array is too big you sleep, for rate throttling. As commands finish you remove them from the array.
When the command is ran with the job ID it will create a lock using Symfony's lock component so that a job cannot accidentally be processed two times at once. It is important that you unlock the job when it either finishes or errors out. Once it has the ID and the lock it will then call whatever code you have written to actually process the job.
Using this technique I have taken commands that took hours to run, as it synchronously went through each task, into taking only minutes. Make sure to try different throttles to balance resource utilization and time it takes to execute your task.

Adding a new job with GearmanManager?

I'm still new to the whole Gearman and GearmanManager cycle. I have a working server and have verified my jobs run if they're already in my queue (MySQL table) when the server starts. However, I need to be able to add a new job to the queue from PHP, and from inside of the worker if possible.
Right now I have a job that I will be creating on the deployment of our new codebase. This will be the first job to run and it's purpose is to gather some data for reports and store it.
This needs to be run every hour on the hour, so I want to utilize the when_to_run column. I've been mulling over the documentation for Gearman but I'm still confused on how I'm actually supposed to add a job to the queue.
I've tried running:
<?php
$gm = new GearmanClient;
$gm->addServer();
$gm->doBackground('Metadata_Ingest_Report', '', com_create_guid());
On a side note, yes, I do have php-pecl-uuid installed.
The above code just hangs and doesn't do anything. No job is added to the DB, nothing happens.
This is due to me not fully understanding how a job gets sent, and I'm doing my best to RTM, but I'm not having any luck.
So if there is something you can point me to, or if someone has some time to explain how I'm supposed to setup and add jobs to the MySQL queue so GearmanManager's workers pick them up that would be awesome.
Edit: So it appears that you have to call $gm->addServer('127.0.0.1'). According to the documentation 127.0.0.1 is supposed to be the default, but that does not appear to be the case running PHP 5.4.11. I can now get the tasks to run if I call $gm->runTasks() after $gm->addTask(). I would expect to just have to call $gm->addTask() and the task would be added to the DB and GearmanManager would see it and spool it up. Still digging...
Best regards,
Andrew
So it appears that the when_to_run functionality is not exposed on pecl-gearman. Because this, we are unable to schedule jobs for the future using their built in methods. It also appears that the library does not create the DB records like it should (I'd assume this may actually be Gearmand not offloading the jobs to the DB before they're run.
To get around this we have decided to do the following.
Scheduling Future Jobs
Manually INSERT the job into gearman_queue.
Run a CRON every minute to ping the queue table and load jobs that have a when_to_run <= time()
Fire those jobs via addTask($function, $payload) and runTasks(). The $payload contains the UUID from the DB as well.
GearmanManager picks up the job and hands off the payload to their respective workers.
Worker runs, then on completion, removes the item from the DB with a DELETE.
Running Job Immediately
Manually INSERT the job into gearmand_queue with a when_to_run of NULL.
Run addTask($function, $payload) and runTasks(). The $payload contains the UUID from the DB as well.
GearmanManager picks up the job and hands off the payload to their respetive workers.
Worker runs, then on completion, removes the item from the DB with a DELETE.
Conclusion
Gearmand Job Server, GearmanManager, and pecl-gearman all seem to be out of sync when it comes to what is supported and how it's done. For the most part I think this issue lays within the core of pecl-gearman talking to Gearmand.
I have also opened a feature request on the pecl-gearman project for when_to_run: https://bugs.php.net/bug.php?id=64120
Before added task, nedded starting gearman server.
for linux:
/usr/bin/gearmand -d -L 127.0.0.1 -q libdrizzle /
--libdrizzle-user=your_db_user --libdrizzle-password=your_db_pass /
--libdrizzle-db=your_db_name --libdrizzle-mysql -vvvv
after adding task, create worker
look like worker.php:
<?php
$worker= new GearmanWorker();
$worker->addServer();
while ($worker->work());
function Metadata_Ingest_Report( $job )
{
// do something
}
and starting this worker
/usr/bin/php worker.php

Start & Stop PHP Script from Backend Administrative Webpage

I'm trying to create a webpage that will allow me to start and stop of a PHP script. The script is part of the backend of a site, and will need to access, read data from, process that data, and update a database on the same server the script exists on. Additionally, I'd like the webpage to allow an administrator to view the current status (running or stopped), and even link to logs created by the script.
I'm starting to go down the path of learning about PHP's exec, passthru, and related functions. Is this the correct path to take? Are there other ways to do this that would be more suitable? Is it possible to do this in a platform agnostic way? I'm developing on a LAMPP+CakePHP stack in Windows, and would like the functionality to exist on any webhost I choose.
I've done this in a recent job but it's probably overkill for you. I did a job processor and it basically sets 2 tables in the database, 2 objects at a minimum and 2 controllers at a minimum.
The first part is the job processing unit, it is composed of a job processor controller that manages the request to start or continue a job and it comes with two activerow models JobQueue and Job. You can remove the queue, but it's always practical to have queing in such systems so you can say that 2,3,4 jobs could execute at once.
The queue is only that, it's a slot that gets several jobs attached to it and it has a queue status to determine if it is running right now or not.
The job is a virtual object that maps to a job table describing what has to be done. In my implementation, i have created an interface that must be implemented into the called controller and a field + a type in the database. The Job instanciates the controller class to call (not the job processor controler, another controler that manages the operation to do) and calls a method in it to start the task processing.
Now, to get tricky, i forced my system to run on a dedicated server just for that portion because i didn't want the task to load the main server or jam the processing queue of Apache. So i had two servers and my Queue class was in charge of calling via an ip address a page on another server to run the job on that server specifically. When the job was done, it called itself back using a HTTP request to restart processing and do the next task. If no task was left, then it would simply die normally.
The advantage of doing it this way is that it doesn't require a cronjob (as long as your script is super stable and cannot crash) because it gets triggered by you when you want it and then you can let it go and it calls itself back with a fsockopen to trigger another page view that triggers the next job.
Work units
It is important to understand that if your jobs are very large, you should segment them. I used the principle of a "work unit" to describe 1 part the job has to do any number of times. Then the Queue Processor became a time manager too so that he could detect if a job took more than X seconds, it would simply defer the rest of the steps for later and call itself back and continue were he was at. That way, you don't need to "set time limit" and you don't jam your server while a 30s script gets executed.
I hope this helps!
To run a script which run continually, you need think to that:
Your php script should be launched as CLI (command line) by a job scheduler like cron or something else. Don't forget that your web server configuration defined a timeout on executed script.
To run 24h a day, maybe you imagine to implement an infinite loop. In that case, you can write a test like jobIsActive which read in a file or in the database every loop if the job should be executed or not. If you click on the button just change the job status (update file, db ...). Your both button can stop the treatment or activate it but doesn't stop the infinite loop.
An infinite loop isn't the most elegant solution, why don't you write an entry in the cron tab to execute the job each night and a click on a button can fired it manually.

Infrastructure for Running your Zend Queue Receiver

I have a simple messaging queue setup and running using the Zend_Queue object heirarchy. I'm using a Zend_Queue_Adapter_Db back-end. I'm interested in using this as a job queue, to schedule things for processing at a later time. They're jobs that don't need to happen immediately, but should happen sooner rather than later.
Is there a best-practices/standard way to setup your infrastructure to run jobs? I understand the code for receiving a message from the queue, but what's not so clear to me is how run the program that does that receiving. A cron that receives n messages on the command-line, run once a minute? A cron that fires off multiple web requests, each web request running the receiver script? Something else?
Tangential bonus question. If I'm running other queries with Zend_Db, will the message queue queries be considered part of that transaction?
You can do it like a thread pool. Create a command line php script to handle the receiving. It should be started by a shell script that automatically restarts the process if it dies. The shell script should not start the process if it is already running (use a $pid.running file or similar). Have cron run several of these every 1-10 minutes. That should handle the receiving nicely.
I wouldn't have the cron fire a web request unless your cron is on another server for some strange reason.
Another way to use this would be to have some backround process creating data, and a web user(s) consume it as they naturally browse the site. A report generator might work this way. Company-wide reports are available to all users but you don't want them all generating this db/time intensive report. So you create a queue and process one at a time possible removing duplicates. All users can view the report(s) when ready.
According to the docs it doens't look like the zend db is even using the same connection as your other zend_db queries. But of course the best way to find out is to make a simple test.
EDIT
The multiple lines in the cron are for concurrency. each line represents a worker for the pool. I was not clear, you don't want the pid as the identifier, you want to pass that as a parameter.
/home/byron/run_queue.sh Process1
/home/byron/run_queue.sh Process2
/home/byron/run_queue.sh Process3
The bash script would check for the $process.running file if it finds it exit.
otherwise:
Create the $process.running file.
start the php process. Block/wait until finished.
Delete the $process.running file.
This allows for the php script to die but not cause the pool to loose a worker.
If the queue is empty the php script exits immediately and is started again by the nex invocation of cron.

Categories