Customized Phinx Symfony command with multiple app bootstrap - php

I'm using Phinx to execute migrations across 100s of applications on multiple servers. Every application should execute same migrations.
In order to do this there is a instance of app on central server which is aware of all configs and other information needed to do bootstrap process (which is being done based on applicationId).
That central instance (let's call it adminapp) executes command and receives applicationIds through STDIN and then does a loop which bootstraps application and runs migration command.
<?php
namespace Command\Db;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use App\Command\AppCommand;
class MigrateBulkCommand extends AppCommand
{
protected function configure()
{
$this
->setName('command:blah')
->setDescription('Executes SQL migrations accross multiple applications. Expects ApplicationIDs to be passed as new line delimited string on STDIN.')
;
}
protected function execute(InputInterface $input, OutputInterface $output)
{
$stdin = $this->getStdin();
if ($stdin === false) {
throw new \RuntimeException("Bulk migration command requires applicationIds to be passed to STDIN.");
}
$applicationIds = explode("\n", $stdin);
foreach($applicationIds as $applicationId) {
try {
$this->bootstrap($applicationId);
} catch (\Exception $e) {
$output->writeln(sprintf("<error>Bootstrap process failed for applicationId `%s`</error>", $applicationId));
}
$command = new \Phinx\Console\Command\Migrate();
$migrationInput = new \Symfony\Component\Console\Input\ArrayInput([
]);
$returnCode = $command->run($migrationInput, $output);
$output->writeln(sprinf("<info>Migrations for applicationId `%s` executed successfully.</info>", $applicationId));
}
}
}
Now Phinx expects it's configuration to be present in form of a config file. What I'm trying to do is reuse DB connection resource (PDO) and pass it to Phinx command Phinx\Console\Command\Migrate on the fly, together with db name.
I've seen in Phinx documentation that this is an option with PHP config file but I can't find a way to do this on the fly (during Phinx\Console\Command\Migrate class initialization).
Phinx doc suggests:
require 'app/init.php';
global $app;
$pdo = $app->getDatabase()->getPdo();
return array('environments' =>
array(
'default_database' => 'development',
'development' => array(
'name' => 'devdb',
'connection' => $pdo
)
)
);
Is there a way, without horrible hacking to pass PDO connection resource and db name to \Phinx\Console\Command\Migrate

I ended up extending Phinx Config class \Phinx\Config\Config and creating method fromArray.
$command = new \Phinx\Console\Command\Migrate();
$command->setConfig(\MyNamespace\Config::fromArray(
[
'paths' => [
'migrations' => APPLICATION_PATH . "/../db/migrations",
'seeds' => APPLICATION_PATH . "/../db/seeds"
],
'environments' => [
'default_database' => 'production',
'production' => [
'name' => $db->get('dbname'),
'adapter' => 'mysql',
'host' => $db->get('host'),
'port' => $db->get('port'),
'user' => $db->get('username'),
'pass' => $db->get('password'),
]
]
]
));

Related

Set autocommit off at doctrine config level

I am aware that I can switch this off at mysql level as part of its config. This is limiting as its once per instance, and I dont want to switch on and off/introduce docker to circumvent it.
However, I would like to specify this at configuration level, ie in the array that you pass to doctrine when instantiating the object, so I can have one config that rolls every back at the end of the PHP run:
$conn = array(
'driver' => 'pdo_sqlite',
'path' => __DIR__ . '/db.sqlite',
'autocommit' => false // <- Does this exist?
);
$entityManager = EntityManager::create($conn, $config);
The best I can come up with is to grab the entity manager and switch autocommit off in the setUp() and tearDown() functions, it would be nice to not to have to remember to do that or specify globally. That, and I can do ad-hoc operations outside of phpunit and get the auto rollback.
It's not tested, but you could try the following
$db = new PDO('sqlite:dogsDb_PDO.sqlite', $user, $pass, array(
PDO::ATTR_AUTOCOMMIT => false
));
$config = new \Doctrine\DBAL\Configuration();
$connectionParams = array(
'dbname' => 'mydb',
'user' => 'user',
'password' => 'secret',
'host' => 'localhost',
'pdo' => $db,
);
$conn = \Doctrine\DBAL\DriverManager::getConnection($connectionParams, $config);
$entityManager = EntityManager::create($conn, $config);
Source / simillar issue - https://github.com/doctrine/dbal/issues/2315
You could do something like:
/** #var EntityManagerInterface */
private $em;
public function setUp()
{
// initialize the entity manager in some manner .....
$this->em->beginTransaction();
}
public function tearDown()
{
$this->em->rollback();
}
Hope this help

How to accommodate Amazon FIFO SQS in Laravel queue?

Amazon has announced their new FIFO SQS service and I'd like to use it in Laravel Queue to solve some concurrency issues.
I've created several new queues and changed the configurations. However, I got a MissingParameter error which says
The request must contain the parameter MessageGroupId.
So I modified the file vendor/laravel/framework/src/Illuminate/Queue/SqsQueue.php
public function pushRaw($payload, $queue = null, array $options = [])
{
$response = $this->sqs->sendMessage(['QueueUrl' => $this->getQueue($queue), 'MessageBody' => $payload,
'MessageGroupId' => env('APP_ENV', getenv('APP_ENV'))]);
return $response->get('MessageId');
}
public function later($delay, $job, $data = '', $queue = null)
{
$payload = $this->createPayload($job, $data);
$delay = $this->getSeconds($delay);
return $this->sqs->sendMessage([
'QueueUrl' => $this->getQueue($queue), 'MessageBody' => $payload, 'DelaySeconds' => $delay,
'MessageGroupId' => env('APP_ENV', getenv('APP_ENV'))
])->get('MessageId');
}
I'm using APP_ENV as the group ID (it's a single message queue so actually it doesn't matter a lot. I just want everything to be FIFO).
But I'm still getting the same error message. How could I fix it? Any help would be appreciated.
(btw, where has the SDK defined sendMessage? I can find a stub for it but I didn't find the detailed implementation)
I want to point out to others who might stumble across the same issue that, although editing SqsQueue.php works, it will easily be reset by a composer install or composer update. An alternative is to implement a new Illuminate\Queue\Connectors\ConnectorInterface for SQS FIFO then add it to Laravel's queue manager.
My approach is as follows:
Create a new SqsFifoQueue class that extends Illuminate\Queue\SqsQueue but supports SQS FIFO.
Create a new SqsFifoConnector class that extends Illuminate\Queue\Connectors\SqsConnector that would establish a connection using SqsFifoQueue.
Create a new SqsFifoServiceProvider that registers the SqsFifoConnector to Laravel's queue manager.
Add SqsFifoServiceProvider to your config/app.php.
Update config/queue.php to use the new SQS FIFO Queue driver.
Example:
Create a new SqsFifoQueue class that extends Illuminate\Queue\SqsQueue but supports SQS FIFO.
<?php
class SqsFifoQueue extends \Illuminate\Queue\SqsQueue
{
public function pushRaw($payload, $queue = null, array $options = [])
{
$response = $this->sqs->sendMessage([
'QueueUrl' => $this->getQueue($queue),
'MessageBody' => $payload,
'MessageGroupId' => uniqid(),
'MessageDeduplicationId' => uniqid(),
]);
return $response->get('MessageId');
}
}
Create a new SqsFifoConnector class that extends Illuminate\Queue\Connectors\SqsConnector that would establish a connection using SqsFifoQueue.
<?php
use Aws\Sqs\SqsClient;
use Illuminate\Support\Arr;
class SqsFifoConnector extends \Illuminate\Queue\Connectors\SqsConnector
{
public function connect(array $config)
{
$config = $this->getDefaultConfiguration($config);
if ($config['key'] && $config['secret']) {
$config['credentials'] = Arr::only($config, ['key', 'secret']);
}
return new SqsFifoQueue(
new SqsClient($config), $config['queue'], Arr::get($config, 'prefix', '')
);
}
}
Create a new SqsFifoServiceProvider that registers the SqsFifoConnector to Laravel's queue manager.
<?php
class SqsFifoServiceProvider extends \Illuminate\Support\ServiceProvider
{
public function register()
{
$this->app->afterResolving('queue', function ($manager) {
$manager->addConnector('sqsfifo', function () {
return new SqsFifoConnector;
});
});
}
}
Add SqsFifoServiceProvider to your config/app.php.
<?php
return [
'providers' => [
...
SqsFifoServiceProvider::class,
],
];
Update config/queue.php to use the new SQS FIFO Queue driver.
<?php
return [
'default' => 'sqsfifo',
'connections' => [
'sqsfifo' => [
'driver' => 'sqsfifo',
'key' => 'my_key'
'secret' => 'my_secret',
'queue' => 'my_queue_url',
'region' => 'my_sqs_region',
],
],
];
Then your queue should now support SQS FIFO Queues.
Shameless plug: While working on the steps above I've created a laravel-sqs-fifo composer package to handle this at https://github.com/maqe/laravel-sqs-fifo.
FIFO message works in a different way than standard AWS SQS queues.
You need a separate driver for handling FIFO queues.
I had to face the same situation and the below package was a lifesaver.
https://packagist.org/packages/shiftonelabs/laravel-sqs-fifo-queue
in queue.php
'sqs-fifo' => [
'driver' => 'sqs-fifo',
'key' => env('SQS_KEY'),
'secret' => env('SQS_SECRET'),
'prefix' => env('SQS_PREFIX'),
'queue' => env('SQS_QUEUE'),
'region' => env('SQS_REGION'),
'group' => 'default',
'deduplicator' => 'unique',
],
then
dispatch(new TestJob([]))->onQueue('My_Mail_Queue.fifo');
NB:
you need to specify default queue name you are going to use in your application in the .env
SQS_QUEUE=My_Default_queue.fifo
Also, you need to specify all the queue names you are going to use in your application in the listener. (if you are using the same queue name for the whole application, you don't need to specify the queue name in the listener)
php artisan queue:listen --queue=My_Default_queue.fifo,My_Mail_Queue.fifo,My_Message_Queue.fifo
Apart from the MessageGroupId, it needs a MessageDeduplicationId or enabling content-based deduplication.

Can't access CakePHP fixture db when triggering test with Travis CI

so I have a project in CakePHP. When pushing my code, Travis CI should run my test. My Test is named ToolTest and its Fixture is ToolFixture.
My .travis.yml looks as following:
language: php
php: 5.3
services:
- mysql
before_script:
- sh -c "mysql -e 'CREATE DATABASE test;'"
- chmod -R 777 project/tmp
- echo "<?php
class DATABASE_CONFIG {
public \$test = array(
'datasource' => 'Database/Mysql',
'persistent' => false,
'database' => 'test',
'host' => 'localhost',
'login' => 'travis'
);
}" > project/database.php
script:
sudo project/Console/cake test app Model/Tool --stderr
The error strack trace on travis says:
Fatal error: Uncaught exception 'MissingConnectionException' with message 'Database connection "Mysql" is missing, or could not be created.' in /home/travis/build/project/lib/Cake/Model/Datasource/Database/Mysql.php:194
Error: Database connection "Mysql" is missing, or could not be created.
I already tried '127.0.0.1' instead of localhost, same error messages. When running my test on the VM, the test passes.
What I've noticed:
If I'm not running the script command, travis is successful, so creating the db test and writing the database.php should work fine, right?
My test and fixture are pretty minimalistic.
ToolTest:
public function setUp()
{
parent::setUp();
$this->Tool = ClassRegistry::init('Tool');
}
public function testFindListById()
{
$result = $this->Tool->findListById(2);
$expected = array(
2 => 'Java'
);
$this->assertEquals($expected, $result);
}
ToolFixture:
class ToolFixture extends CakeTestFixture
{
public $useDbConfig = 'test';
public $fields = array(
'id' => 'string',
'name' => 'string'
);
public $records = array(
array(
'id' => 1,
'name' => 'HTML'
),
array(
'id' => 2,
'name' => 'Java'
)
);
}
What am I missing? I've been stuck with this problem for days..Any ideas? Glad for any help!
So I found out that I didn't write my database config in the correct folder..
Instead of
>project/database.php
It should have been
> project/Config/database.php
Jeez..

Cakephp set database timezone

I'm setting timezone to php and mysql to internacionalize my CakePHP application.
When the server receives a request from a client, before process request, it connects to a GeoIp location server and gets the Timezone. Then I use date_default_timezone_set() to set php timezone. The problem comes up when I want to set database timezone. Once Cakephp connected, I need to execute sql query like SET time_zone='-06:00'.
In /lib/Cake/Model/Datasource/Database/Mysql.php I can see at connect() function the following code:
try {
$this->_connection = new PDO(
$dsn,
$config['login'],
$config['password'],
$flags
);
$this->connected = true;
if (!empty($config['settings'])) {
foreach ($config['settings'] as $key => $value) {
$this->_execute("SET $key=$value");
}
}
} catch (PDOException $e) {
throw new MissingConnectionException(array(
'class' => get_class($this),
'message' => $e->getMessage()
));
}
There is a $config['settings'] array that can be configured to do it. But I don't know how to fill settings array and where it's the best place to do that.
What I need is modify default datasource config on-the-fly
You can add an additional key to config array located at app/Config/database.php like this:
public $default = array(
'datasource' => 'Database/Mysql',
'persistent' => false,
'host' => 'localhost',
'login' => 'db_user',
'password' => 'db_pass',
'database' => 'db_name',
'prefix' => '',
'settings' => array(
'time_zone' => "'+01:00'", // note the quotes!
)
);
Related: CakePHP switch database (using same datasource) on the fly?
I solved it in the following way.
First of all, adding setOptions() method into DATABASE_CONFIG class as follows:
public function setOptions ($datasource, array $options){
$this->{$datasource} = array_merge($this->{$datasource}, $options);
}
Afterwards, extending ConnectionManager class to initialize it:
class ConnectionManagerCustomConfig extends ConnectionManager
{
public static function initialize(){
self::_init();
}
}
Now I initialize the class and add new options:
ConnectionManagerCustomConfig::initialize();
$configClass = ConnectionManagerCustomConfig::$config;
$configClass->setOptions('default', array(
'settings' => array(
'time_zone' => $offset
)
));

Phpunit grabbing Doctrine2 configuration from unknown place

I am using ZF2 + Doctrine2 + PHPUNIT, when setting up Phpunit, it works fine for a basic test, however as soon as I try to run a test that invokes Doctrine2, I get:
PDOException: SQLSTATE[HY000] [1045] Access denied for user 'username'#'localhost' (using password: YES)
But, I have never specified "username" nor "localhost" nor any sort of password. In-fact, my application runs perfectly fine, and the configuration I have specifies completely different settings.
So where is PHPUnit getting those settings and how to fix it?
My global.php:
'doctrine' => array(
'connection' => array(
'orm_default' => array(
'driverClass' => 'Doctrine\DBAL\Driver\PDOMySql\Driver',
'params' => array(
'host' => '127.0.0.1',
'port' => '3306',
'user' => 'sbapp',
'password' => 'myp#ss',
'dbname' => 'root'
)
)
),
The Test:
class ApplicationControllerTest extends AbstractHttpControllerTestCase
{
protected $traceError = true;
public function setUp()
{
$this->setApplicationConfig(
include '../../../config/application.config.php'
);
parent::setUp();
}
public function testAuthActionCanBeAccessed()
{
$postData = new \stdClass();
$postData->username = "someAppUser";
$postData->password = "12345";
$postData = json_decode(json_encode($postData),true);
$this->dispatch('/auth', 'POST', $postData);
$response = $this->getResponse();
$this->assertResponseStatusCode(200);
$this->assertActionName('auth');
}
}
The relative paths used within the config on setUp when done incorrectly can seriously affect the loading of entities and so on. So I was chasing around the right path.. when calling phpunit from root..or from vendor..or from within the test folder, etc..
Solution:
Call it from project root, and leave the routes exactly as the main project are.

Categories