I am currently working on a Symfony 6.1 project and have a database insert that needs to generate quite a lot of entries (30k+). I would like to run this insert process as a background task and once all entries have been created in the database, I would like to send an email.
Now I found the following article about Background Task in the Symfony documentation: https://symfony.com/doc/current/components/process.html#running-processes-asynchronously
However, I don't quite understand how to get the process to run in background.
My current code:
$process = new Process(); //not sure what to put in for the constructor
$process->start(function ($data) {
foreach($data as $key => $value) {
$value->setExampleValue("example");
$this->entityManager->persist($value);
$this->entityManager->flush();
}
... email code
});
if (!$process->isSuccessful()) {
throw new ProcessFailedException($process);
}
UPDATE:
As someone noticed I have had a look on the Commands from Symfony. Now I tried to start my custom Command as a Process. However each time I start the Command it fails and the Exit Code is: ().
This is the code:
$process = new Process(['php bin\console app:generate-test']);
//$process->setWorkingDirectory(substr(getcwd(), 0, strrpos(getcwd(), '/')));
$process->start();
if (!$process->isSuccessful()) {
throw new ProcessFailedException($process);
}
However, if I start the command via the console "php bin/console app:generate-test" it works and I get a simple textline printed in my console window.
Has anyone an idea what I am doing wrong here?
FINAL SOLUTION:
Instead of the Process Component from Symfony I switched to the Messenger Component as recommended in the comments.
This works now like a charm!
I found a great tutorial series for understanding and implementing the messenger: https://www.youtube.com/watch?v=Xwn_BNfyjgk
I think you should look at commands (https://symfony.com/doc/current/console.html)
You can then run the command and minimize the console window or use a process manager (like Supervisor) to run it in the background.
I have the following (simple) lock code for a Laravel 5.3 command:
private $hash = null;
public final function handle() {
try {
$this->hash = md5(serialize([ static::class, $this->arguments(), $this->options() ]));
$this->info("Generated signature ".$this->hash,"v");
if (Redis::exists($this->hash)) {
$this->hash = null;
throw new \Exception("Method ".$this->signature." is already running");
}
Redis::set($this->hash, true);
$this->info("Running method","vv");
$this->runMutuallyExclusiveCommand(); //Actual command is not important here
$this->cleanup();
} catch (\Exception $e) {
$this->error($e->getMessage());
}
}
public function cleanup() {
if (is_string($this->hash)) {
Redis::del($this->hash);
}
}
This works fine if the command is allowed to go through its execution cycle normally (including handling when there's a PHP exception). However the problem arises when the command is interrupted via other means (e.g. CTRL-C or when the terminal window is closed). In that case the cleanup code is not ran and the command is considered to be still "executing" so I need to manually remove the entry from the cache in order to restart it. I have tried running the cleanup code in a __destruct function but that does not seem to be called either.
My question is, is there a way to set some code to be ran when a command is terminated regardless how it was terminated?
Short answer is no. When you kill the running process, either by Ctrl-C or just closing the terminal, you terminate it. You would need to have an interrupt in your shell that links to your cleanup code, but that is way out of scope.
There are other options however. Cron jobs can be run at intermittent intervals to perform clean up tasks and other helpful things. You could also create a start up routine that runs prior to your current code. When you execute the start up routine, it could do the cleanup for you, then call your current routine. I believe your best bet is to use a cron job that simply runs at given intervals that then looks for entries in the cache that are no longer appropriate, and then cleans them. Here is a decent site to get you started with cron jobs https://www.linux.com/learn/scheduling-magic-intro-cron-linux
I can see if a script is running in the console with the App::runningInConsole() command, but I would like to know (for audit logging purposes) which command has been run from console.
To add some context - I need to log whenever a system or user accesses a certain type of data. For users, that's pretty simple - I can use Auth::user() and get their IP address from Request, but for console commands, it's a little more difficult.
I can figure out if a console command is running, which is good, but I need to be able to record which console command is running.
I do not see a way to get that information directly Laravel. I'm also not sure how that would be possible, as you can have one command executing other commands or even creating a new application instance like some test tools do.
There is however a way to achieve that using plain PHP. You can check the server/environment variables to identify how application was executed.
Have a look at
dd($_SERVER['argv']);
If you run a command this should give you the command name in $_SERVER['argv'][1]
You can find more information here: http://php.net/manual/en/reserved.variables.argv.php
To get console command together with arguments you should use something like that:
$command = \Request::server('argv', null);
if (is_array($command)) {
$command = implode(' ', $command);
}
In Laravel you can use \Request:server to get $_SERVER variables.
The problem with $SERVER['arg'] is that it doesnt work when you execute the command on the GUI.
I fixed the problem adding the next code on the command.
private function getAttributes()
{
$arguments = $this->arguments();
unset($arguments[0]);
$arguments = collect($arguments)->implode(' ');
$options = collect($this->options())->filter(function($item) {
return $item;
})->map(function($item, $key) {
$return = '--'.$key;
if($item!==true) {
$return .= '='.$item;
}
return $return;
})->implode(' ');
return $arguments.' '.$options;
}
and you get it calling $this->getAttributes() and you will get all the command with the attributes
I am new on Symfony2 and I got blocked when trying to run an asynchronous command like this:
class MyCommand extends ContainerAwareCommand{
protected function configure()
{
$this
->setName('my:command')
->setDescription('My command')
->addArgument(
'country',
InputArgument::REQUIRED,
'Which country?'
)
;
}
protected function execute(InputInterface $input, OutputInterface $output)
{
$country = $input->getArgument('country');
// Obtain the doctrine manager
$dm = $this->getContainer()->get('doctrine_mongodb.odm.document_manager');
$users = $dm->getRepository('MyBundle:User')
->findBy( array('country'=>$country));
}}
That works perfectly when I call it from my command line:
php app/console my:command uk
But it doesn't work when I call it trowh a Symfony2 Process:
$process = new Process("php ../app/console my:command $country");
$process->start();
I get a database error: "[MongoWriteConcernException] 127.0.0.1:27017: not master"
I think that means that the process is not getting my database configuration...
I just want to run an asynchronous process, is there other way to do it?
Maybe a way to call the Application Command that do not require the answer to keep going ?
Maybe I need to use injection?
PS: My current command is just a test, at the end it should be an 'expensive' operation...
Well, I found out what happened...
I use multiple environments: DEV, TEST and PROD.
And I also use differents servers.
So the DEV environment is my own machine with a simple mongodb configuration.
But the TEST environment is on other server with a replica set configuration...
Now the error get full sense: "[MongoWriteConcernException] 127.0.0.1:27017: not master"
To solve it, I've just added the environment parameter (--env=) to the process and everything worked like a charm:
$process = new Process("php ../app/console my:command $country --env=test");
Actually, to get the correct environment I use this:
$this->get('kernel')->getEnvironment();
Which let's my code as follows:
$process = new Process("php ../app/console my:command $country --env=".$this->get('kernel')->getEnvironment());
Maybe is not a beautifull way to do it, but it works for me :)
Disclamer: This might be a bit overkill for what you're trying to do :)
I would choose an opposite way to do it: pthreads
First, quick examination of StackOverflow showed me a really nice example of using pthreads: Multi-threading is possible in php
Then, knowing that you could invoke your command from another command:
http://www.craftitonline.com/2011/06/calling-commands-within-commands-in-symfony2/
... lets you piece all the parts. It's a bit complicated but it does the job.
In case you want to execute your code completely async in Symfony2/3 there is AsyncServiceCallBundle for that.
You should just call it like this:
$this->get('krlove.async')->call('your_service_id', 'method_name', [$arg1, $arg2]);
Internally it uses this approach to run your code in background.
I am trying to write a cronjob controller, so I can call one website and have all modules cronjob.php executed. Now my problem is how do I do that?
Would curl be an option, so I also can count the errors and successes?
[Update]
I guess I have not explained it enough.
What I want to do is have one file which I can call like from http://server/cronjob and then make it execute every /application/modules/*/controller/CronjobController.php or have another way of doing it so all the cronjobs aren't at one place but at the same place the module is located. This would offer me the advantage, that if a module does not exist it does not try to run its cronjob.
Now my question is how would you execute all the modules CronjobController or would you do it a completly different way so it still stays modular?
And I want to be able to giveout how many cronjobs ran successfully and how many didn't
After some research and a lot procrastination I came to the simple conclusion that a ZF-ized cron script should contain all the functionality of you zend framework app - without all the view stuff. I accomplished this by creating a new cronjobfoo.php file in my application directory. Then I took the bare minimum from:
-my front controller (index.php)
-my bootstrap.php
I took out all the view stuff and focused on keeping the environment setup, db setup, autoloader, & registry setup. I had to take a little time to correct the document root variable and remove some of the OO functionality copied from my bootstrap.
After that I just coded away.. in my case it was compiling and emailing out nightly reports. It was great to use Zend_Mail. When I was confident that my script was working the way I wanted, I just added it my crontab.
good luck!
For Zend Framework I am currently using the code outlined bellow. The script only includes the portal file index.php, where all the paths, environment and other Zendy code is bootstrapped. By defining a constant in the cron script we cancel the final step , where the application is run.
This means the application is only setup, not even bootstrapped. At this point we start bootstraping the resources we need and that is that
//public/index.php
if(!defined('DONT_RUN_APP') || DONT_RUN_APP == false) {
$application->bootstrap()->run();
}
// application/../cron/cronjob.php
define("DONT_RUN_APP",true);
require(realpath('/srv/www/project/public/index.php'));
$application->bootstrap('config');
$application->bootstrap('db');
//cron code follows
I would caution putting your cronjobs accessible to the public because they could be triggered outside their normal times and, depending on what they do, cause problems (I know that is not what you intend, but by putting them into an actual controller it becomes reachable from the browser). For example, I have one cron that sends e-mails. I would be spammed constantly if someone found the cron URL and just began hitting it.
What I did was make a cron folder and in there created a heartbeat.php which bootstraps Zend Framework (minus MVC) for me. It checks a database which has a list of all the installed cron jobs and, if it is time for them to run, generates an instances of the cron job's class and runs it.
The cron jobs are just child classes from an abstract cron class that has methods like install(), run(), deactivate(), etc.
To fire off my job I just have a simple crontab entry that runs every 5 minutes that hits heartbeat.php. So far it's worked wonderful on two different sites.
Someone mentioned this blog entry a couple days ago on fw-general (a mailinglist which I recommend reading when you use the Zend Framework).
There is also a proposal for Zend_Controller_Request_Cli, which should address this sooner or later.
I have access to a dedicated server and I initially had a different bootstrap for the cron jobs. I eventually hated the idea, just wishing I could do this within the existing MVC setup and not have to bother about moving things around.
I created a file cron.sh, saved is within my site root (not public) and in it I put a series of commands I would like to run. As I wanted to run many commands at once I wrote the PHP within my controllers as usual and added curl calls to those urls within cron.sh. for example curl http://www.mysite.com/cron_controller/action Then on the cron interface I ran bash /path/to/cron.sh.
As pointed out by others your crons can be fired by anyone who guesses the url so there's always that caveat. You can find a solution to that in many different ways.
Take a look at zf-cli:
scripts at master from padraic/ZFPlanet - GitHub
This handles well all cron jobs.
Why not just create a crontab.php, including, or requiring the index.php bootstrap file?
Considering that the bootstrap is executing Zend_Loader::registerAutoload(), you can start working directly with the modules, for instance, myModules_MyClass::doSomething();
That way you are skipping the controllers. The Controller job is to control the access via http. In this case, you don't need the controller approach because you are accessing locally.
Do you have filesystem access to the modules' directories? You could iterate over the directories and determine where a CronjobController.php is available. Then you could either use Zend_Http_Client to access the controller via HTTP or use an approach like Zend_Test_PHPUnit: simulate the actual dispatch process locally.
You could set up a database table to hold references to the cronjob scripts (in your modules), then use a exec command with a return value on pass/fail.
I extended gregor answer with this post. This is what came out:
//public/index.php
// Run application, only if not started from command line (cli)
if (php_sapi_name() != 'cli' || !empty($_SERVER['REMOTE_ADDR'])) {
$application->run();
}
Thanks gregor!
My solution:
curl /cron
Global cron method will include_once all controllers
Check whether each of the controllors has ->cron method
If they have, run those.
Public cron url (for curl) is not a problem, there are many ways to avoid abuse. As said, checking remote IP is the easiest.
This is my way to run Cron Jobs with Zend Framework
In Bootstrap I will keep environment setup as it is minus MVC:
public static function setupEnvironment()
{
...
self::setupFrontController();
self::setupDatabase();
self::setupRoutes();
...
if (PHP_SAPI !== 'cli') {
self::setupView();
self::setupDbCaches();
}
...
}
Also in Bootstrap, I will modify setupRoutes and add a custom route:
public function setupRoutes()
{
...
if (PHP_SAPI == 'cli') {
self::$frontController->setRouter(new App_Router_Cli());
self::$frontController->setRequest(new Zend_Controller_Request_Http());
}
}
App_Router_Cli is a new router type which determines the controller, action, and optional parameters based on this type of request: script.php controller=mail action=send. I found this new router here: Setting up Cron with Zend Framework 1.11
:
class App_Router_Cli extends Zend_Controller_Router_Abstract
{
public function route (Zend_Controller_Request_Abstract $dispatcher)
{
$getopt = new Zend_Console_Getopt (array());
$arguments = $getopt->getRemainingArgs();
$controller = "";
$action = "";
$params = array();
if ($arguments) {
foreach($arguments as $index => $command) {
$details = explode("=", $command);
if($details[0] == "controller") {
$controller = $details[1];
} else if($details[0] == "action") {
$action = $details[1];
} else {
$params[$details[0]] = $details[1];
}
}
if($action == "" || $controller == "") {
die("Missing Controller and Action Arguments == You should have:
php script.php controller=[controllername] action=[action]");
}
$dispatcher->setControllerName($controller);
$dispatcher->setActionName($action);
$dispatcher->setParams($params);
return $dispatcher;
}
echo "Invalid command.\n", exit;
echo "No command given.\n", exit;
}
public function assemble ($userParams, $name = null, $reset = false, $encode = true)
{
throw new Exception("Assemble isnt implemented ", print_r($userParams, true));
}
}
In CronController I do a simple check:
public function sendEmailCliAction()
{
if (PHP_SAPI != 'cli' || !empty($_SERVER['REMOTE_ADDR'])) {
echo "Program cannot be run manually\n";
exit(1);
}
// Each email sent has its status set to 0;
Crontab runs a command of this kind:
* * * * * php /var/www/projectname/public/index.php controller=name action=send-email-cli >> /var/www/projectname/application/data/logs/cron.log
It doesn't make sense to run the bootstrap in the same directory or in cron job folder. I've created a better and easy way to implement the cron job work. Please follow the below things to make your work easy and smart:
Create a cron job folder such as "cron" or "crobjob" etc. whatever you want.
Sometimes we need the cron job to run on a server with different interval like for 1 hr interval or 1-day interval that we can setup on the server.
Create a file in cron job folder like I created an "init.php", Now let's say you want to send a newsletter to users in once per day. You don't need to do the zend code in init.php.
So just set up the curl function in init.php and add the URL of your controller action in that curl function. Because our main purpose is that an action should be called on every day. for example, the URL should be like this:
https://www.example.com/cron/newsletters
So set up this URL in curl function and call this function in init.php in the same file.
In the above link, you can see "cron" is the controller and newsletters is the action where you can do your work, in the same way, don't need to run the bootstrap file etc.