I'm working on a project that does NOT have a copy of production DB on development environment.
Sometimes we have an issue with DB migrations - they pass on dev DB but fail in production/testing.
It's often beacuse Dev environent data is loaded from Fixtures that use the latest entities - filling all tables properly.
Is there any easy way to make sure Doctrine Migration(s) will pass in production?
Do you have/know any way to write an automatic tests that will make sure data will be migrated properly without downloading the production/testing DB and running the migration manually?
I would like to avoid downloading a production/testing DB to dev machine so I can check migrations becasue that DB contains private data and it can be quite big.
First, you need to create a sample database dump in state before the migration. For MySQL use mysqldump. For postgres pg_dump, e.g.:
mysqldump -u root -p mydatabase > dump-2018-02-20.sql
pg_dump -Upostgres --inserts --encoding utf8 -f dump-2018-02-20.sql mydatabase
Then create an abstract class for all migrations tests (I assume you have configured a separate database for integration testing in config_test.yml):
abstract class DatabaseMigrationTestCase extends WebTestCase {
/** #var ResettableContainerInterface */
protected $container;
/** #var Application */
private $application;
protected function setUp() {
$this->container = self::createClient()->getContainer();
$kernel = $this->container->get('kernel');
$this->application = new Application($kernel);
$this->application->setAutoExit(false);
$this->application->setCatchExceptions(false);
$em = $this->container->get(EntityManagerInterface::class);
$this->executeCommand('doctrine:schema:drop --force');
$em->getConnection()->exec('DROP TABLE IF EXISTS public.migration_versions');
}
protected function loadDump(string $name) {
$em = $this->container->get(EntityManagerInterface::class);
$em->getConnection()->exec(file_get_contents(__DIR__ . '/dumps/dump-' . $name . '.sql'));
}
protected function executeCommand(string $command): string {
$input = new StringInput("$command --env=test");
$output = new BufferedOutput();
$input->setInteractive(false);
$returnCode = $this->application->run($input, $output);
if ($returnCode != 0) {
throw new \RuntimeException('Failed to execute command. ' . $output->fetch());
}
return $output->fetch();
}
protected function migrate(string $toVersion = '') {
$this->executeCommand('doctrine:migrations:migrate ' . $toVersion);
}
}
Example migration test:
class Version20180222232445_MyMigrationTest extends DatabaseMigrationTestCase {
/** #before */
public function prepare() {
$this->loadDump('2018-02-20');
$this->migrate('20180222232445');
}
public function testMigratedSomeData() {
$em = $this->container->get(EntityManagerInterface::class);
$someRow = $em->getConnection()->executeQuery('SELECT * FROM myTable WHERE id = 1')->fetch();
$this->assertEquals(1, $someRow['id']);
// check other stuff if it has been migrated correctly
}
}
I've figured out simple "smoke tests" for Doctrine Migrations.
I have PHPUnit test perfoming following steps:
Drop test DB
Create test DB
Load migrations (create schema)
Load fixtures (imitate production data)
Migrate to some older version
Migrate back to the latest version
This way I can test for the major issues, we've had recently.
Example of PHPUnit tests can be found on my blog: http://damiansromek.pl/2015/09/29/how-to-test-doctrine-migrations/
Related
I followed this guide: https://symfony.com/doc/current/console/lockable_trait.html and implemented the command lock feature for my one of my commands to see how it works. It worked as described and then I was going to implement it for all of my commands. But the issue is that I have about 50 commands and:
I do not want spent time adding the necessary code to each command
I want to have the centralized management of commands locking. I mean, adding extra option to regular commands so that they will be used by my future management center. For now I will need a pretty simple option protected function isLocked() for a regular command which will help me to manage if a command should have lockable feature.
So, I went to the source of \Symfony\Component\Console\Command\LockableTrait and after some time created the following listener to the event console.command:
use Symfony\Component\Console\Event\ConsoleCommandEvent;
use Symfony\Component\Console\Exception\LogicException;
use Symfony\Component\Lock\Lock;
use Symfony\Component\Lock\LockFactory;
use Symfony\Component\Lock\LockInterface;
use Symfony\Component\Lock\Store\FlockStore;
use Symfony\Component\Lock\Store\SemaphoreStore;
class LockCommandsListener
{
/**
* #var array<string, Lock>
*/
private $commandLocks = [];
private static function init()
{
if (!class_exists(SemaphoreStore::class)) {
throw new LogicException('To enable the locking feature you must install the symfony/lock component.');
}
}
public function onConsoleCommand(ConsoleCommandEvent $event)
{
static::init();
$name = $event->getCommand()->getName();
$this->ensureLockNotPlaced($name);
$lock = $this->createLock($name);
$this->commandLocks[$name] = $lock;
if (!$lock->acquire()) {
$this->disableCommand($event, $name);
}
}
private function disableCommand(ConsoleCommandEvent $event, string $name)
{
unset($this->commandLocks[$name]);
$event->getOutput()->writeln('The command ' . $name . ' is already running');
$event->disableCommand();
$event->getCommand()->setCode()
}
private function createLock(string $name): LockInterface
{
if (SemaphoreStore::isSupported()) {
$store = new SemaphoreStore();
} else {
$store = new FlockStore();
}
return (new LockFactory($store))->createLock($name);
}
private function ensureLockNotPlaced(string $name)
{
if (isset($this->commandLocks[$name])) {
throw new LogicException('A lock is already in place.');
}
}
}
I made some tests and it kind of worked. But I am not sure this is the right way of doing things.
Another problem is that I can not find the proper exit code when I disabled a command. Should I just disable it? But it seems that exit code would be a great feature here. Specially when it comes to this listener testing (PHPUnit testing).
And I also have with testing itself. How can I run commands in parallel in my test class. For now I have this:
class LockCommandTest extends CommandTest
{
public function testOneCommandCanBeRun()
{
$commandTester = new ApplicationTester($this->application);
$commandTester->run([
'command' => 'app:dummy-command'
]);
$output = $commandTester->getDisplay();
dd($output);
}
}
It will allow only to run my commands one by one. But I would like to run them both so after running the first one, the second will fail (with some exit code).
As for me the best way to make background task is doing it via supervisor, create config file, like:
[program:your_service]
command=/usr/local/bin/php /srv/www/bin/console <your:app:command>
priority=1
numprocs=1
# Each 5 min.
startsecs=300
autostart=true
autorestart=true
process_name=%(program_name)s_%(process_num)02d
user=root
this is the best way to be sure that your command will be ran only in one process
Wondering if anyone knows how or of a package that allows migrations to be done on multiple databases without having the additional connection in the config file. Currently I have two different connections, one for the default and another for the tenants. The tenant connection has the credentials input, but it doesn't have the database name since each tenant will have a different database name. Problem is when I'm doing the initial setup and need the migrations to be run it will do it for the default or the tenant(if I edit the file to include the database name). I had found a package that would do it but it is not currently compatible with laravel 5. Obviously if I have to do it manually it can be done. It would just be a pain. Thanks in advance for suggestions.
Looking through the tenancy related questions here, I stumbled upon your open one.
What I've done is override the default MigrateCommand provided by Laravel and add the --tenant option. This will allow me to migrate one, more or all tenants. You can check the implementation out at my repository. Knowing the code of conduct here, I'll also add an example implementation:
<?php
namespace Hyn\MultiTenant\Commands\Migrate;
use Hyn\MultiTenant\Traits\TenantDatabaseCommandTrait;
use Illuminate\Database\Migrations\Migrator;
use PDOException;
class MigrateCommand extends \Illuminate\Database\Console\Migrations\MigrateCommand
{
use TenantDatabaseCommandTrait;
/**
* MigrateCommand constructor.
*
* #param Migrator $migrator
*/
public function __construct(Migrator $migrator)
{
parent::__construct($migrator);
$this->website = app('Hyn\MultiTenant\Contracts\WebsiteRepositoryContract');
}
public function fire()
{
// fallback to default behaviour if we're not talking about multi tenancy
if (! $this->option('tenant')) {
$this->info('No running tenancy migration, falling back on native laravel migrate command due to missing tenant option.');
return parent::fire();
}
if (! $this->option('force') && ! $this->confirmToProceed()) {
$this->error('Stopped no confirmation and not forced.');
return;
}
$websites = $this->getWebsitesFromOption();
// forces database to tenant
if (! $this->option('database')) {
$this->input->setOption('database', 'tenant');
}
foreach ($websites as $website) {
$this->info("Migrating for {$website->id}: {$website->present()->name}");
$website->database->setCurrent();
$this->prepareDatabase($website->database->name);
// The pretend option can be used for "simulating" the migration and grabbing
// the SQL queries that would fire if the migration were to be run against
// a database for real, which is helpful for double checking migrations.
$pretend = $this->input->getOption('pretend');
// Next, we will check to see if a path option has been defined. If it has
// we will use the path relative to the root of this installation folder
// so that migrations may be run for any path within the applications.
if (! is_null($path = $this->input->getOption('path'))) {
$path = $this->laravel->basePath().'/'.$path;
} else {
$path = $this->getMigrationPath();
}
try {
$this->migrator->run($path, $pretend);
} catch (PDOException $e) {
if (str_contains($e->getMessage(), ['Base table or view already exists'])) {
$this->comment("Migration failed for existing table; probably a system migration: {$e->getMessage()}");
continue;
}
}
// Once the migrator has run we will grab the note output and send it out to
// the console screen, since the migrator itself functions without having
// any instances of the OutputInterface contract passed into the class.
foreach ($this->migrator->getNotes() as $note) {
$this->output->writeln($note);
}
}
// Finally, if the "seed" option has been given, we will re-run the database
// seed task to re-populate the database, which is convenient when adding
// a migration and a seed at the same time, as it is only this command.
if ($this->input->getOption('seed')) {
$this->call('db:seed', ['--force' => true, '--tenant' => $this->option('tenant')]);
}
}
/**
* Prepare the migration database for running.
*
* #return void
*/
protected function prepareDatabase($connection = null)
{
if (! $connection) {
$connection = $this->option('database');
}
$this->migrator->setConnection($connection);
if (! $this->migrator->repositoryExists()) {
$options = ['--database' => $connection];
$this->call('migrate:install', $options);
}
}
/**
* #return array
*/
protected function getOptions()
{
return array_merge(
parent::getOptions(),
$this->getTenantOption()
);
}
}
If you have any questions to solve this issue, let me know.
I am beginner to Simpletest and facing an issue while creating fixtures. As I am using cakephp 1.3.14 version for my application.
Created fixture with filename complaint_fixture.php
class ComplaintFixture extends CakeTestFixture {
var $name = 'Complaint';
var $import = array('table' => 'complaints', 'records' => true);
// do not truncate movie_stars table between tests
public function truncate($db) {
return null;
}
// do not drop movie_stars table between tests
public function drop($db) {
return null;
}
}
Created test case with name complaint.test.php
App::import('Model', 'Complaint');
class ComplaintTestCase extends CakeTestCase {
var $fixtures = array('app.Complaint');
function setUp($method) {
parent::setUp();
$this->Complaint = & ClassRegistry::init('Complaint');
// load data
$this->loadFixtures('Complaint');
}
function testFixture() {
$numberOfResults = $this->Complaint->find('count');
var_dump($numberOfResults);
}
/*
function testupdateComplaintStatus(){
$result = $this->Complaint->updateComplaintStatus(47,'ACT');
$this->assertEqual($result,1,'Status updated successfully!');
} */
}
As you can see in the above code, a fixture is created with name Complaint and then a test case is being used to load that fixture. So, what I have read on it from developer guide
- we do create a fixture with specifying the fields name and a records set
- load that fixture in test model class.
BUT, what I am looking for is to perform CRUD operations on test data which is being inserted into the test database. And, when I try to do the same with above given script, It starts affecting the production database records instead of test database.
If you see in the above code I have even stopped truncate and drop for test data, yet not able to sort out the issue.
Can anyone let me know what I have missed in the above code?
I am trying to run my unit test and create a database during setup. For some reason I am getting the error Unknown database 'coretest'. If I create the database though manually and run the test then I get Can't create database 'coretest'; database exists.
The drop database statement works just now the create database.
Here is my setUP and tearDown methods:
class TestCase extends Illuminate\Foundation\Testing\TestCase {
/**
* Default preparation for each test
*/
public function setUp() {
parent::setUp();
DB::statement('create database coretest;');
Artisan::call('migrate');
$this->seed();
Mail::pretend(true);
}
public function tearDown() {
parent::tearDown();
DB::statement('drop database coretest;');
}
}
The reason why you get this error is simply because laravel tries to connect to database specified in config, which doesn't exist.
The solution is to build your own PDO connection from the settings without specifying database (PDO allows this) and run CREATE DATABASE $dbname statement using it.
We used this approach for testing in our project without any problem.
Here some code:
<?php
/**
* Bootstrap file for (re)creating database before running tests
*
* You only need to put this file in "bootstrap" directory of the project
* and change "bootstrap" phpunit parameter within "phpunit.xml"
* from "bootstrap/autoload.php" to "bootstap/testing.php"
*/
$testEnvironment = 'testing';
$config = require("app/config/{$testEnvironment}/database.php");
extract($config['connections'][$config['default']]);
$connection = new PDO("{$driver}:user={$username} password={$password}");
$connection->query("DROP DATABASE IF EXISTS ".$database);
$connection->query("CREATE DATABASE ".$database);
require_once('app/libraries/helpers.php');
// run migrations for packages
foreach(glob('vendor/*/*', GLOB_ONLYDIR) as $package) {
$packageName = substr($package, 7); // drop "vendor" prefix
passthru("./artisan migrate --package={$packageName} --env={$testEnvironment}");
}
passthru('./artisan migrate --env='.$testEnvironment);
require('autoload.php'); // run laravel's original bootstap file
neoascetic has the best answer because essentially you have to boot laravel's database config file again.
So, a clever hack is to create database again after you have dropped it. No need to touch config/database.
public function setUp() {
parent::setUp();
Artisan::call('migrate');
$this->seed();
Mail::pretend(true);
}
public function tearDown() {
parent::tearDown();
DB::statement('drop database coretest;');
DB::statement('create database coretest;');
}
create it like this:
public function setUp() {
parent::setUp();
$pdo = new PDO(
"mysql:host=localhost",
config('database.connections.mysql.username'),
config('database.connections.mysql.password')
);
$pdo->query('CREATE DATABASE test_database');
// set up the new database as the default
config()->set('database.connections.mysql.database', 'test_database');
}
I feel like I have a much cleaner way of doing this. Just execute the commands normally through the shell.
$host = Config::get('database.connections.mysql.host');
$database = Config::get('database.connections.mysql.database');
$username = Config::get('database.connections.mysql.username');
$password = Config::get('database.connections.mysql.password');
echo shell_exec('mysql -h ' . $host . ' -u ' . $username . ' -p' . $password . ' -e "DROP DATABASE ' . $database . '"');
echo shell_exec('mysql -h ' . $host . ' -u ' . $username . ' -p' . $password . ' -e "CREATE DATABASE ' . $database . '"');
In Laravel 5 it's possible to call migrations internally to the Laravel process which ends up running a good bit quicker than using external commands.
In TestCase::setUp (or earlier), call the migration command with:
$kernel = app('Illuminate\Contracts\Console\Kernel');
$kernel->call('migrate');
Is there a way to run a console command from a Symfony 2 test case? I want to run the doctrine commands for creating and dropping schemas.
This documentation chapter explains how to run commands from different places. Mind, that using exec() for your needs is quite dirty solution...
The right way of executing console command in Symfony2 is as below:
Option one
use Symfony\Bundle\FrameworkBundle\Console\Application as App;
use Symfony\Component\Console\Tester\CommandTester;
class YourTest extends WebTestCase
{
public function setUp()
{
$kernel = $this->createKernel();
$kernel->boot();
$application = new App($kernel);
$application->add(new YourCommand());
$command = $application->find('your:command:name');
$commandTester = new CommandTester($command);
$commandTester->execute(array('command' => $command->getName()));
}
}
Option two
use Symfony\Component\Console\Input\StringInput;
use Symfony\Bundle\FrameworkBundle\Console\Application;
class YourClass extends WebTestCase
{
protected static $application;
public function setUp()
{
self::runCommand('your:command:name');
// you can also specify an environment:
// self::runCommand('your:command:name --env=test');
}
protected static function runCommand($command)
{
$command = sprintf('%s --quiet', $command);
return self::getApplication()->run(new StringInput($command));
}
protected static function getApplication()
{
if (null === self::$application) {
$client = static::createClient();
self::$application = new Application($client->getKernel());
self::$application->setAutoExit(false);
}
return self::$application;
}
}
P.S. Guys, don't shame Symfony2 with calling exec()...
The docs tell you the suggested way to do it. The example code is pasted below:
protected function execute(InputInterface $input, OutputInterface $output)
{
$command = $this->getApplication()->find('demo:greet');
$arguments = array(
'command' => 'demo:greet',
'name' => 'Fabien',
'--yell' => true,
);
$input = new ArrayInput($arguments);
$returnCode = $command->run($input, $output);
// ...
}
Yes, if your directory structure looks like
/symfony
/app
/src
then you would run
phpunit -c app/phpunit.xml.dist
from your unit tests you can run php commands either by using
passthru("php app/console [...]") (http://php.net/manual/en/function.passthru.php)
exec("php app/console [...]") (http://www.php.net/manual/en/function.exec.php)
or by putting the command in back ticks
php app/consode [...]
If you are running the unit tests from a directory other than symofny, you'll have to adjust the relative path to the app directory for it to work.
To run it from the app:
// the document root should be the web folder
$root = $_SERVER['DOCUMENT_ROOT'];
passthru("php $root/../app/console [...]");
The documentation has been updated since my last answer to reflect the proper Symfony 2 way of calling an existing command:
http://symfony.com/doc/current/components/console/introduction.html#calling-an-existing-command