I have multiple supervisor in horizon, and they work normally, the problem is that I want to interact with them by my own web interface, and by interacting I mean pause them and continue (unpause them).
To do that I want to be able as much as possible, without using system (in artisan horizon:pause-supervisor it sends posix_kill($supervisor->pid, 12)).
I tried to instantiate the supervisor by doing this :
class HorizonManager
{
private SupervisorRepository $supervisors;
private MasterSupervisorRepository $masters;
private WorkloadRepository $workload;
private RedisJobRepository $jobRepository;
private QueueManager $queueManager;
public function __construct(MasterSupervisorRepository $masters, SupervisorRepository $supervisors, WorkloadRepository $workload, RedisJobRepository $jobRepository, QueueManager $manager)
{
$this->masters = $masters;
$this->supervisors = $supervisors;
$this->workload = $workload;
$this->jobRepository = $jobRepository;
$this->queueManager = $manager;
}
public function pauseSupervisor(string $supervisorName){
$supervisor = $this->supervisors->find($supervisorName);
$supervisorOpt = new SupervisorOptions(...$supervisor->options);
$sup = new Supervisor($supervisorOpt);
$sup->pause();
$sup->persist();
return $this->supervisors->find($supervisorName);
}
}
In the return from the function, I have the supervisor paused, but it's not really paused (even If I persist the instantiate supervisor it's still running as a process)
For those interested I failed doing it by instanciating it so instead I send the command using artisan call :
define('SIGUSR2', 12);
Artisan::call('horizon:pause-supervisor', ['name'=>$supervisorName]);
$supervisor = $this->supervisors->find($supervisorName);
$supervisor->status = 'paused';
Related
I followed this guide: https://symfony.com/doc/current/console/lockable_trait.html and implemented the command lock feature for my one of my commands to see how it works. It worked as described and then I was going to implement it for all of my commands. But the issue is that I have about 50 commands and:
I do not want spent time adding the necessary code to each command
I want to have the centralized management of commands locking. I mean, adding extra option to regular commands so that they will be used by my future management center. For now I will need a pretty simple option protected function isLocked() for a regular command which will help me to manage if a command should have lockable feature.
So, I went to the source of \Symfony\Component\Console\Command\LockableTrait and after some time created the following listener to the event console.command:
use Symfony\Component\Console\Event\ConsoleCommandEvent;
use Symfony\Component\Console\Exception\LogicException;
use Symfony\Component\Lock\Lock;
use Symfony\Component\Lock\LockFactory;
use Symfony\Component\Lock\LockInterface;
use Symfony\Component\Lock\Store\FlockStore;
use Symfony\Component\Lock\Store\SemaphoreStore;
class LockCommandsListener
{
/**
* #var array<string, Lock>
*/
private $commandLocks = [];
private static function init()
{
if (!class_exists(SemaphoreStore::class)) {
throw new LogicException('To enable the locking feature you must install the symfony/lock component.');
}
}
public function onConsoleCommand(ConsoleCommandEvent $event)
{
static::init();
$name = $event->getCommand()->getName();
$this->ensureLockNotPlaced($name);
$lock = $this->createLock($name);
$this->commandLocks[$name] = $lock;
if (!$lock->acquire()) {
$this->disableCommand($event, $name);
}
}
private function disableCommand(ConsoleCommandEvent $event, string $name)
{
unset($this->commandLocks[$name]);
$event->getOutput()->writeln('The command ' . $name . ' is already running');
$event->disableCommand();
$event->getCommand()->setCode()
}
private function createLock(string $name): LockInterface
{
if (SemaphoreStore::isSupported()) {
$store = new SemaphoreStore();
} else {
$store = new FlockStore();
}
return (new LockFactory($store))->createLock($name);
}
private function ensureLockNotPlaced(string $name)
{
if (isset($this->commandLocks[$name])) {
throw new LogicException('A lock is already in place.');
}
}
}
I made some tests and it kind of worked. But I am not sure this is the right way of doing things.
Another problem is that I can not find the proper exit code when I disabled a command. Should I just disable it? But it seems that exit code would be a great feature here. Specially when it comes to this listener testing (PHPUnit testing).
And I also have with testing itself. How can I run commands in parallel in my test class. For now I have this:
class LockCommandTest extends CommandTest
{
public function testOneCommandCanBeRun()
{
$commandTester = new ApplicationTester($this->application);
$commandTester->run([
'command' => 'app:dummy-command'
]);
$output = $commandTester->getDisplay();
dd($output);
}
}
It will allow only to run my commands one by one. But I would like to run them both so after running the first one, the second will fail (with some exit code).
As for me the best way to make background task is doing it via supervisor, create config file, like:
[program:your_service]
command=/usr/local/bin/php /srv/www/bin/console <your:app:command>
priority=1
numprocs=1
# Each 5 min.
startsecs=300
autostart=true
autorestart=true
process_name=%(program_name)s_%(process_num)02d
user=root
this is the best way to be sure that your command will be ran only in one process
I have been having a lot of trouble getting the Laravel Excel package to export a large amount of data. I need to export about 80-100k rows so I implemented the queued export as mentioned in the docs. It works fine when I export a smaller amount of rows, but when I try to do 60-80k rows, it fails every time. While the jobs are being processed, I watch the temp file that is created, and I can see that the size of the file is increasing. I also watch the jobs in the database (I'm using the database queue driver), and I can see the jobs completing for a while. It seems that the jobs take incremently more time until the job fails. I don't get why the first several jobs are quick, and then they start taking more and more time to complete.
I'm using supervisor to manage the queue, so here's my config for that:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/site/artisan queue:work --sleep=3 --tries=3 --timeout=120 --queue=exports,default
autostart=true
autorestart=true
user=www-data
numprocs=8
redirect_stderr=true
stdout_logfile=/var/log/supervisor/worker.log
loglevel=debug
And then my controller to create the export
(new NewExport($client, $year))->queue('public/exports/' . $name)->allOnQueue('exports')->chain([
new NotifyUserOfCompletedExport($request->user(), $name),
]);
I'm using:
Laravel 5.8,
PHP 7.2,
Postgresql 10.10
I should also mention that I have played around with the chunk size a bit, but in the end I've always run into the same problem. I tried chunk sizes of 500, 2000, 10000 but no luck.
In the failed_jobs table, the exception is MaxAttemptsExceededException, although I have also got exceptions for InvalidArgumentException File '/path/to/temp/file' does not exist. I'm not quite sure what else to do. I guess I could make it so it doesn't timeout, but that seems like it will just cause more problems. Any help would be appreciated.
EDIT
Here is the content of my Export Class:
class NewExport implements FromQuery, WithHeadings, WithMapping, WithStrictNullComparison
{
public function __construct($client, $year)
{
$this->year = $year;
$this->client = $client;
}
public function query()
{
$data = $this->getDataQuery();
return $data ;
}
public function headings(): array
{
$columns = [
//....
];
return $columns;
}
public function map($row): array
{
$mapping = [];
foreach($row as $key => $value) {
if(is_bool($value)) {
if($value) {
$mapping[$key] = "Yes";
} else {
$mapping[$key] = "No";
}
}else{
$mapping[$key] = $value;
}
}
return $mapping;
}
private function getDataQuery()
{
$query = \DB::table('my_table')->orderBy('my_field');
return $query;
}
The NotifyUserOfCompletedExport class is just creating a job to email the logged in user that the export is finished with a link to download it.
class NotifyUserOfCompletedExport implements ShouldQueue
{
use Queueable, SerializesModels;
public $user;
public $filename;
public function __construct(User $user, $filename)
{
$this->user = $user;
$this->filename = $filename;
}
public function handle()
{
// This just sends the email
$this->user->notify(new ExportReady($this->filename, $this->user));
}
}
EDIT 2:
So I read this post, and I verified that eventually my server was just running out of memory. That lead to the MaxAttemptsExceededException error. I added more memory to the server, and I am still getting the InvalidArgumentException File '/path/to/temp/file' does not exist after the jobs have completed. It's even more weird though, because I can see that /path/to/temp/file actually does exist. So I have no idea what is going on here, but it's super frustrating.
I have an API written using Symfony2 that I'm trying to write post hoc tests for. One of the endpoints uses an email service to send a password reset email to the user. I'd like to mock out this service so that I can check that the right information is sent to the service, and also prevent an email from actually being sent.
Here's the route I'm trying to test:
/**
* #Route("/me/password/resets")
* #Method({"POST"})
*/
public function requestResetAction(Request $request)
{
$userRepository = $this->get('app.repository.user_repository');
$userPasswordResetRepository = $this->get('app.repository.user_password_reset_repository');
$emailService = $this->get('app.service.email_service');
$authenticationLimitsService = $this->get('app.service.authentication_limits_service');
$now = new \DateTime();
$requestParams = $this->getRequestParams($request);
if (empty($requestParams->username)) {
throw new BadRequestHttpException("username parameter is missing");
}
$user = $userRepository->findOneByUsername($requestParams->username);
if ($user) {
if ($authenticationLimitsService->isUserBanned($user, $now)) {
throw new BadRequestHttpException("User temporarily banned because of repeated authentication failures");
}
$userPasswordResetRepository->deleteAllForUser($user);
$reset = $userPasswordResetRepository->createForUser($user);
$userPasswordResetRepository->saveUserPasswordReset($reset);
$authenticationLimitsService->logUserAction($user, UserAuthenticationLog::ACTION_PASSWORD_RESET, $now);
$emailService->sendPasswordResetEmail($user, $reset);
}
// We return 201 Created for every request so that we don't accidently
// leak the existence of usernames
return $this->jsonResponse("Created", $code=201);
}
I then have an ApiTestCase class that extends the Symfony WebTestCase to provide helper methods. This class contains a setup method that tries to mock the email service:
class ApiTestCase extends WebTestCase {
public function setup() {
$this->client = static::createClient(array(
'environment' => 'test'
));
$mockEmailService = $this->getMockBuilder(EmailService::class)
->disableOriginalConstructor()
->getMock();
$this->mockEmailService = $mockEmailService;
}
And then in my actual test cases I'm trying to do something like this:
class CreatePasswordResetTest extends ApiTestCase {
public function testSendsEmail() {
$this->mockEmailService->expects($this->once())
->method('sendPasswordResetEmail');
$this->post(
"/me/password/resets",
array(),
array("username" => $this->user->getUsername())
);
}
}
So now the trick is to get the controller to use the mocked version of the email service. I have read about several different ways to achieve this, so far I've not had much luck.
Method 1: Use container->set()
See How to mock Symfony 2 service in a functional test?
In the setup() method tell the container what it should return when it's asked for the email service:
static::$kernel->getContainer()->set('app.service.email_service', $this->mockEmailService);
# or
$this->client->getContainer()->set('app.service.email_service', $this->mockEmailService);
This does not effect the controller at all. It still calls the original service. Some write ups I've seen mention that the mocked service is 'reset' after a single call. I'm not even seeing my first call mocked out so I'm not certain this issue is affecting me yet.
Is there another container I should be calling set on?
Or am I mocking out the service too late?
Method 2: AppTestKernel
See: http://blog.lyrixx.info/2013/04/12/symfony2-how-to-mock-services-during-functional-tests.html
See: Symfony2 phpunit functional test custom user authentication fails after redirect (session related)
This one pulls me out of my depth when it comes to PHP and Symfony2 stuff (I'm not really a PHP dev).
The goal seems to be to change some kind of foundation class of the website to allow my mock service to be injected very early in the request.
I have a new AppTestKernel:
<?php
// app/AppTestKernel.php
require_once __DIR__.'/AppKernel.php';
class AppTestKernel extends AppKernel
{
private $kernelModifier = null;
public function boot()
{
parent::boot();
if ($kernelModifier = $this->kernelModifier) {
$kernelModifier($this);
$this->kernelModifier = null;
};
}
public function setKernelModifier(\Closure $kernelModifier)
{
$this->kernelModifier = $kernelModifier;
// We force the kernel to shutdown to be sure the next request will boot it
$this->shutdown();
}
}
And a new method in my ApiTestCase:
// https://stackoverflow.com/a/19705215
protected static function getKernelClass(){
$dir = isset($_SERVER['KERNEL_DIR']) ? $_SERVER['KERNEL_DIR'] : static::getPhpUnitXmlDir();
$finder = new Finder();
$finder->name('*TestKernel.php')->depth(0)->in($dir);
$results = iterator_to_array($finder);
if (!count($results)) {
throw new \RuntimeException('Either set KERNEL_DIR in your phpunit.xml according to http://symfony.com/doc/current/book/testing.html#your-first-functional-test or override the WebTestCase::createKernel() method.');
}
$file = current($results);
$class = $file->getBasename('.php');
require_once $file;
return $class;
}
Then I alter my setup() to use the kernel modifier:
public function setup() {
...
$mockEmailService = $this->getMockBuilder(EmailService::class)
->disableOriginalConstructor()
->getMock();
static::$kernel->setKernelModifier(function($kernel) use ($mockEmailService) {
$kernel->getContainer()->set('app.service.email_service', $mockEmailService);
});
$this->mockEmailService = $mockEmailService;
}
This works! However I now can't access the container in my other tests when I'm trying to do something like this:
$c = $this->client->getKernel()->getContainer();
$repo = $c->get('app.repository.user_password_reset_repository');
$resets = $repo->findByUser($user);
The getContainer() method returns null.
Should I be using the container differently?
Do I need to inject the container into the new kernel? It extends the original kernel so I don't really know why/how it's any different when it comes to the container stuff.
Method 3: Replace the service in config_test.yml
See: Symfony/PHPUnit mock services
This method requires that I write a new service class that overrides the email service. Writing a fixed mock class like this seems less useful than a regular dynamic mock. How can I test that certain methods have been called with certain parameters?
Method 4: Setup everything inside the test
Going on #Matteo's suggestion I wrote a test that did this:
public function testSendsEmail() {
$mockEmailService = $this->getMockBuilder(EmailService::class)
->disableOriginalConstructor()
->getMock();
$mockEmailService->expects($this->once())
->method('sendPasswordResetEmail');
static::$kernel->getContainer()->set('app.service.email_service', $mockEmailService);
$this->client->getContainer()->set('app.service.email_service', $mockEmailService);
$this->post(
"/me/password/resets",
array(),
array("username" => $this->user->getUsername())
);
}
This test fails because the expected method sendPasswordResetEmail wasn't called:
There was 1 failure:
1) Tests\Integration\Api\MePassword\CreatePasswordResetTest::testSendsEmail
Expectation failed for method name is equal to <string:sendPasswordResetEmail> when invoked 1 time(s).
Method was expected to be called 1 times, actually called 0 times.
Thanks to Cered's advice I've managed to get something working that can test that the emails I expect to be sent actually are. I haven't been able to actually get the mocking to work so I'm a bit reluctant to mark this as "the" answer.
Here's a test that checks that an email is sent:
public function testSendsEmail() {
$this->client->enableProfiler();
$this->post(
"/me/password/resets",
array(),
array("username" => $this->user->getUsername())
);
$mailCollector = $this->client->getProfile()->getCollector('swiftmailer');
$this->assertEquals(1, $mailCollector->getMessageCount());
$collectedMessages = $mailCollector->getMessages();
$message = $collectedMessages[0];
$this->assertInstanceOf('Swift_Message', $message);
$this->assertEquals('Reset your password', $message->getSubject());
$this->assertEquals('info#example.com', key($message->getFrom()));
$this->assertEquals($this->user->getEmail(), key($message->getTo()));
$this->assertContains(
'This link is valid for 24 hours only.',
$message->getBody()
);
$resets = $this->getResets($this->user);
$this->assertContains(
$resets[0]->getToken(),
$message->getBody()
);
}
It works by enabling the Symfony profiler and inspecting the swiftmailer service. It's documented here: http://symfony.com/doc/current/email/testing.html
I have a unit test, where I need to call a service.
I did it that way:
<?php
namespace FM\PriceBundle\Tests\Service;
use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;
class PriceServiceTest extends KernelTestCase
{
private $container;
public function testFiulPrice()
{
self::bootKernel();
$this->container = self::$kernel->getContainer();
$productId = 1;
$id = 1;
$what = ['postal', 'departement', 'region', 'country', 'insee'];
$date = new \DateTime('2016-06-23');
$price = $this->container->get('fm.price.get')->fiulPrice($productId, $id, $what[0], $date);
$this->assertNotEmpty($price);
}
}
But I have this error:
There was 1 error:
1) FM\PriceBundle\Tests\Service\PriceServiceTest::testFiulPrice
Predis\Connection\ConnectionException: Connection refused [tcp://localhost:6379]
I guess you use redis in your app.
But I don't understand why do you want to implement test like this.
That test literally takes fixed data, an tries to get a result from external service that uses dependency (redis).
In my opinion, you need unit test for PriceService, another one for serivce registered as 'fm.price.get', and (maybe) the third one that use redis, but in this case you have to populate cache with some sample data.
I'm trying to run a job queue to create a PDF file using SlmQueueBeanstalkd and DOMPDFModule in ZF".
Here's what I'm doing in my controller:
public function reporteAction()
{
$job = new TareaReporte();
$queueManager = $this->serviceLocator->get('SlmQueue\Queue\QueuePluginManager');
$queue = $queueManager->get('myQueue');
$queue->push($job);
...
}
This is the job:
namespace Application\Job;
use SlmQueue\Job\AbstractJob;
use SlmQueue\Queue\QueueAwareInterface;
use SlmQueue\Queue\QueueInterface;
use DOMPDFModule\View\Model\PdfModel;
class TareaReporte extends AbstractJob implements QueueAwareInterface
{
protected $queue;
public function getQueue()
{
return $this->queue;
}
public function setQueue(QueueInterface $queue)
{
$this->queue = $queue;
}
public function execute()
{
$sm = $this->getQueue()->getJobPluginManager()->getServiceLocator();
$empresaTable = $sm->get('Application\Model\EmpresaTable');
$registros = $empresaTable->listadoCompleto();
$model = new PdfModel(array('registros' => $registros));
$model->setOption('paperSize', 'letter');
$model->setOption('paperOrientation', 'portrait');
$model->setTemplate('empresa/reporte-pdf');
$output = $sm->get('viewPdfrenderer')->render($model);
$filename = "/path/to/pdf/file.pdf";
file_put_contents($filename, $output);
}
}
The first time you run it, the file is created and the work is successful, however, if you run a second time, the task is buried and the file is not created.
It seems that stays in an endless cycle when trying to render the model a second time.
I've had a similar issue and it turned out it was because of the way ZendPdf\PdfDocument reuses it's object factory. Are you using ZendPdf\PdfDocument?
You might need to correctly close factory.
class MyDocument extends PdfDocument
{
public function __destruct()
{
$this->_objFactory->close();
}
}
Try to add this or something similar to the PdfDocument class...
update : it seem you are not using PdfDocument, however I suspect this is the issue is the same. Are you able to regenerate a second PDF in a normal http request? It is your job to make sure the environment is equal on each run.
If you are unable to overcome this problem a short-term quick solution would be to set max_runs configuration for SlmQueue to 1. That way the worker is stopped after each job and this reset to a vanilla state...