Symfony 4, Doctrine and AWS RDS read replica usage - php

I'm about to embark on using RDS with a master read/write and slave read-only setup.
I've read about the Doctrine MasterSlaveConnection type.
However, there will be some endpoints I create that I would like to use the read-only replica (most of my GET endpoints) and some will need to write data (PUT, PATCH, DELETE endpoints).
I'm also using API Platform.
What is the best way of achieving this in Symfony 4 and Doctrine 2?

What I have done in the past is to just use different connections.
Something like:
doctrine:
dbal:
default_connection: default
connections:
default:
# This is your Master
url: '%env(DATABASE_URL)%'
driver: 'pdo_mysql'
server_version: '5.7'
charset: utf8mb4
slave:
# This would be the slave
url: '%env(DATABASE_SLAVE_URL)%'
driver: 'pdo_mysql'
server_version: '5.7'
charset: utf8mb4
orm:
default_entity_manager: default
entity_managers:
default:
connection: default
mappings:
Main:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity'
prefix: 'App\Entity'
alias: Main
slave:
connection: slave
mappings:
Main:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity'
prefix: 'App\Entity'
alias: Main
https://symfony.com/doc/current/doctrine/multiple_entity_managers.html
Then in your controllers or business logic you can either choose to use the default entity manager:
// Controller
$this->getDoctrine()->getEntityManager();
Or you can get the slave connection:
// Controller
$this->getDoctrine()->getEntityManager('slave');
If you need this to work just on all requests without having to create special actions for everything then your best bet is to decorate the Collection and Item DataProviders for doctrine.
https://symfony.com/doc/current/service_container/service_decoration.html
https://github.com/api-platform/core/blob/master/src/Bridge/Doctrine/Orm/CollectionDataProvider.php
https://github.com/api-platform/core/blob/master/src/Bridge/Doctrine/Orm/ItemDataProvider.php
So basically you need to change what manager is chosen based on the $opperationName something like:
if($opperationName === 'GET'){
$manager = $this->managerRegistry->getManager('slave');
} else {
$manager = $this->managerRegistry->getManager();
}

You actually don't need to setup multiple entity managers, nor is it preferable as handling one entity with multiple entity managers is hard.
Using Doctrine 2.2, you can setup slaves/replicas directly from configuration without needing an extra entity manager:
See the config reference here:
https://www.doctrine-project.org/projects/doctrine-bundle/en/2.2/configuration.html#configuration-overview
Example:
doctrine:
dbal:
default_connection: default
connections:
default:
dbname: '%env(DATABASE_DBNAME)%'
user: '%env(DATABASE_USER)%'
password: '%env(DATABASE_PASSWORD)%'
host: '%env(DATABASE_HOST)%'
driver: 'pdo_mysql'
server_version: '5.7'
charset: utf8mb4
default_table_options:
charset: utf8mb4
collate: utf8mb4_unicode_ci
slaves:
ro_replica:
dbname: '%env(REPLICA_DBNAME)%'
user: '%env(REPLICA_USER)%'
password: '%env(REPLICA_PASSWORD)%'
host: '%env(REPLICA_HOST)%'
charset: utf8mb4

Thank you #Chase for the solution. You have made my day. Although it works for me in 'dev' environment I had a problem when switched to 'prod'. I was getting an error that an Entity can not be found. The solution came from this post - thanks #xabbuh. Basically I had to add default_entity_manager: name_of_default_em to doctrine.yml. Here is the copy of the code:
# config/packages/prod/doctrine.yaml
doctrine:
orm:
default_entity_manager: BOE <- add this line to let know prod about default em
auto_generate_proxy_classes: false
metadata_cache_driver:
type: service
id: doctrine.system_cache_provider
query_cache_driver:
type: service
id: doctrine.system_cache_provider
result_cache_driver:
type: service
id: doctrine.result_cache_provider
# ...

Related

How to handle migrations with multiple entity managers in DoctrineMigrationsBundle 3

On doctrine/doctrine-migrations-bundle 2.* this was relatively simple - use the --em option (and use ContainerAwareInterface to skip any migrations from a different em/connection).
Now (on doctrine/doctrine-migrations-bundle 3.2.2), it seems the --em option is ignored, and the default em/connection is always specified, meaning the migrations for the default em are applied to every database. Edit: As pointed out in comments - --em is not ignored, it's passed through directly, it's rather our ContainerAwareInterface approach that's no longer valid.
There is a lot of conflicting information on how to set this up, some suggesting it should "just work" (Symfony Docs) and other describing workarounds (Issue):
https://symfony.com/doc/current/doctrine/multiple_entity_managers.html
https://github.com/doctrine/DoctrineMigrationsBundle/issues/38
How does one configure this new version (3) of doctrine/doctrine-migrations-bundle to apply migrations only to their matching entity/db?
Edit: I've included below our config previous to upgrading, which along with the ContainerAwareInterface connection filtering approach, allowed filtering migrations to run only against the appropriate entity manager.
Our existing "doctrine/doctrine-bundle": "1.12.8" config (shortened, but shows multiple entity managers):
doctrine:
dbal:
connections:
default:
charset: utf8mb4
default_table_options:
charset: utf8mb4
collate: utf8mb4_unicode_ci
driver: '%database_driver%'
server_version: mariadb-10.4.11
host: '%database_host%'
port: '%database_port%'
dbname: autotempest
user: '%database_user%'
password: '%database_password%'
mapping_types:
enum: string
model:
charset: utf8mb4
default_table_options:
charset: utf8mb4
collate: utf8mb4_unicode_ci
driver: '%database_driver%'
server_version: mariadb-10.4.11
host: '%database_host%'
port: '%database_port%'
dbname: autotempest_models
user: '%database_user%'
password: '%database_password%'
mapping_types:
enum: string
wrapper_class: App\Doctrine\ConnectionWrapper\ConnectionModel
persistent: true
orm:
auto_generate_proxy_classes: '%kernel.debug%'
entity_managers:
default:
connection: default
mappings:
App:
type: 'annotation'
dir: '%kernel.project_dir%/src/Entity/Main'
prefix: 'App\Entity\Main'
model:
connection: model
mappings:
TempestModelBundle:
type: 'annotation'
dir: 'Entity'
prefix: 'Tempest\Bundle\ModelBundle\Entity'
Our "doctrine/doctrine-migrations-bundle": "2.1.2" config:
doctrine_migrations:
dir_name: '%kernel.project_dir%/src/Migrations'
namespace: Application\Migrations
Also mentioned in my question, there is an open issue on the DoctrineMigrationsBundle from 2012 describing the problem of dealing with migrations when using multiple entity managers: https://github.com/doctrine/DoctrineMigrationsBundle/issues/38. It seems there are several options for workarounds to this issue as described there, we just needed to dig and try each of them to find the best one for our situation.
Container Aware Migrations
On Symfony 3, we were using the ContainerAwareInterface approach. Described in the above issue:
Currently this can be achieved by using Container Aware Migrations. If one can have the service container injected, he can obtain an instance of some entity manager and its connection.
This is no longer really a valid solution when moving to Symfony 4 however, due to ContainerAware classes being deprecated in favor of dependency injection.
Pass configuration directly
Another approach mentioned in the github issue above. The idea here is to have a separate configuration file for each entity manager like the following:
# config/packages/migrations/base.yaml
em: default
transactional: false
migrations_paths:
Hyra\Migrations\Base: src/Migrations/Base
table_storage:
table_name: migration_versions
This is passed directly to the command, along with the entity manager like this: bin/console doctrine:migrations:migrate --em default --configuration config/packages/migrations/base.yaml. These separate config files replace the single config/packages/doctrine_migrations.yaml configuration file.
This was also not viable for us, as we still needed to inject services into our migrations using the services configuration option of DoctrineMigrationsBundle, and --configuration only passes configuration options directly through to doctrine/migrations, which doesn't support the services configuration option.
Initially on DoctrineMigrationsBundle 3.0, this approach was complicated by the fact that the --em and --conn options were dropped completely, so it was also necessary to create a wrapper on top of the DoctrineMigrationsBundle commands to re-implement these options (described in more detail here). This is no longer necessary on DoctrineMigrationsBundle 3.1+ (which restored these options).
Use DoctrineMigrationsMultipleDatabaseBundle
Also mentioned in the github issue thread, this bundle implements what we needed exactly (and ended up using) - per-entity configuration for DoctrineMigrationsBundle, so we can also include our services config for migration dependency injection. Initially I misconfigured this - it's important that the base doctrine_migrations.yaml config only includes config for the default entity manager. Sample working config provided by the package author (version 0.3.3):
# doctrine.yaml
doctrine:
dbal:
default_connection: default
connections:
default:
url: '%env(resolve:DATABASE_URL)%'
server_version: mariadb-10.1.26
charset: utf8mb4
default_table_options:
charset: utf8mb4
collate: utf8mb4_unicode_ci
mapping_types:
enum: string
geonames:
url: '%env(resolve:GEONAMES_DATABASE_URL)%'
server_version: mariadb-10.1.26
charset: utf8mb4
default_table_options:
charset: utf8mb4
collate: utf8mb4_unicode_ci
mapping_types:
enum: string
# IMPORTANT: You MUST configure your server version,
# either here or in the DATABASE_URL env var (see .env file)
#server_version: '5.7'
orm:
auto_generate_proxy_classes: true
default_entity_manager: default
entity_managers:
default:
connection: default
naming_strategy: doctrine.orm.naming_strategy.underscore_number_aware
mappings:
Main:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/App/Entity/Main'
prefix: 'App\Entity\Main'
alias: Main
geonames:
naming_strategy: doctrine.orm.naming_strategy.underscore_number_aware
connection: geonames
mappings:
Geonames:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/App/Entity/Geonames'
prefix: 'App\Entity\Geonames'
alias: Geonames
# doctrine_migrations.yaml
doctrine_migrations:
em: default
migrations_paths:
DoctrineMigrations: '%kernel.project_dir%/migrations/Main'
# doctrine_migrations_multiple_database.yaml
doctrine_migrations_multiple_database:
entity_managers:
default:
migrations_paths:
DoctrineMigrations\Main: '%kernel.project_dir%/migrations/Main'
geonames:
migrations_paths:
DoctrineMigrations\Geonames: '%kernel.project_dir%/migrations/Geonames'

Doctrine2 filter table in DB to make migrations

I have been searching to make Doctrine ignoring all tables in a DB except for entities in my app. I was trying to do it with the schema_filter option in doctrine config, however i have a large amount of schemas (i'm using PostgreSQL) and tables in the DB that I don't need to watch with the App. So the easiest solution, IMO, would be to ignore all tables except for App\Entity\Custom folder (this is a shared DB with other apps / users who can create schemas / tables).
Also i'm using multiple connection, and I don't know how to solve my problem from now...
My config/packages/doctrine.yaml file :
doctrine:
dbal:
default_connection: classic
connections:
classic:
url: "%env(resolve:DATABASE_URL)%"
driver: "pdo_pgsql"
server_version: "11.5"
charset: "utf8"
custom:
url: "%env(resolve:DATABASE_URL_CUSTOM)%"
driver: "pdo_pgsql"
server_version: "11.5"
charset: "utf8"
schema_filter: "How to filter all tables in DB except for entities table in App\Entity\Custom\* ?"
orm:
default_entity_manager: "classic"
entity_managers:
classic:
naming_strategy: doctrine.orm.naming_strategy.underscore_number_aware
connection: "api"
mappings:
Main:
is_bundle: false
type: annotation
dir: "%kernel.project_dir%/src/Entity/Classic"
prefix: 'App\Entity\Classic'
alias: "Classic"
custom:
connection: "custom"
mappings:
Main:
is_bundle: false
type: annotation
dir: "%kernel.project_dir%/src/Entity/Custom"
prefix: 'App\Entity\Custom'
alias: "custom"
And I have also 2 migrations config file, cuz this would be greate to separate migrations / DB (I don't know if this is the best way to make it work) :
config/classic_migration.yaml :
doctrine_migrations:
migrations_paths:
'DoctrineMigrations\Classic': '%kernel.project_dir%/migrations/Classic'
enable_profiler: '%kernel.debug%'
storage:
table_storage:
table_name: 'public.migrations'
em: classic
config/custom_migration.yaml:
doctrine_migrations:
migrations_paths:
'DoctrineMigrations\Custom': '%kernel.project_dir%/migrations/Custom'
enable_profiler: '%kernel.debug%'
storage:
table_storage:
table_name: 'public.migrations'
em: custom
Has someone any trick / advice to give me ?
Thank you.

What is the standard way to use a bundle's entities in symfony?

I've been trying to learn how to use a bundle in symfony, but ran into some problems making the bundle's entities appear in my database. I finally got it to work by adding the last couple lines to my doctrine.yaml file. However for some reason it seems unlikely that this is the correct way to go about it, giving that it only works if I set the 'is_bundle' property to false, while it's probably supposed to be true, given that it is an entity from a bundle.
What is the correct way to make this entity appear in my database?
My doctrine file:
doctrine:
dbal:
default_connection: default
connections:
default:
# configure these for your database server
url: '%env(DATABASE_URL)%'
driver: 'pdo_mysql'
server_version: '5.7'
charset: utf8mb4
unix_socket: '/Applications/MAMP/tmp/mysql/mysql.sock'
default_table_options:
charset: utf8mb4
collate: utf8mb4_unicode_ci
orm:
default_entity_manager: default
entity_managers:
default:
connection: default
mappings:
default:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity'
prefix: 'App\Entity'
alias: default
ch_cookie_consent_bundle:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/vendor/connectholland/cookie-consent-bundle/Entity'
prefix: 'ConnectHolland\CookieConsentBundle\Entity'
alias: ch_cookie_consent_bundle
The bundle itself can be found here: https://github.com/ConnectHolland/cookie-consent-bundle
The is_bundle flag instructs the doctrine extension to match the configuration key to the bundle name.
In your case the configuration should be
orm:
mappings:
default:
...
CHCookieConsentBundle: # This should match the bundle name (in %kernel.bundles%)
is_bundle: true
alias: ch_cookie_consent_bundle
All the others configuration keys are optional and auto-detected.

Best practice for database connection via Doctrine in a Symfony Bundle

In my bundle i create a separate database connection and an EntityManager for it. Everything works fine, except those two things don't show up in the development profiler. There is only the default EntityManager and the default connection.
So basically i created 3 new service definitions for an Doctrine\Common\EventManager, an Doctrine\DBAL\Connection and an Doctrine\ORM\EntityManager. I've already tried to add these new service definition to the ContainerBuilder with the same naming convention which is used by the doctrine bridge, but they still won't show up in the profiler. The connection works fine, but i want debug it with and integrate it in the Symfony lifecycle.
The question is:
What is the best practice to create a separate database connection via Doctrine inside of a Symfony Bundle if the Symfony application is only configured to support one connection?
I believe you should take a look at this doc. They described there how to add another EntityManager, which mean another connection. First step is to create configuration.
Especialy take a look at doctrine.yaml configuration:
# config/packages/doctrine.yaml
doctrine:
doctrine:
dbal:
default_connection: default
connections:
default:
# configure these for your database server
url: '%env(DATABASE_URL)%'
driver: 'pdo_mysql'
server_version: '5.7'
charset: utf8mb4
customer:
# configure these for your database server
url: '%env(DATABASE_CUSTOMER_URL)%'
driver: 'pdo_mysql'
server_version: '5.7'
charset: utf8mb4
orm:
default_entity_manager: default
entity_managers:
default:
connection: default
mappings:
Main:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity/Main'
prefix: 'App\Entity\Main'
alias: Main
customer:
connection: customer
mappings:
Customer:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity/Customer'
prefix: 'App\Entity\Customer'
alias: Customer
Above are two entity managers, 'default' and 'customer'. There are also two cennections, one for each manager.
If configuration is valid you will have access to those managers by passing its names to 'getManager' method.
$entityManager = $this->getDoctrine()->getManager('default');
$customerEntityManager = $this->getDoctrine()->getManager('customer');
If you cant edit configuration:
What what about creating custom class (Manager or something) in which you will manually create connection. Take a look at this, it should help you.
getting-a-connection

Using Multiple Database Connections To Access Repositories and Entities

We have a Main database with multiple tables. we are going to have an unknown amount of 'duplicate' databases depending on the client currently being used.
this was setup like the following
doctrine.yaml
doctrine:
dbal:
default_connection: default
connections:
default:
driver: pdo_mysql
server_version: '5.7'
charset: utf8mb4
url: '%env(resolve:DATABASE_URL)%'
School_A:
driver: pdo_mysql
server_version: '5.7'
charset: utf8mb4
url: 'mysql://nibbr:nibbr#127.0.0.1:3306/School_A'
School_B:
driver: pdo_mysql
server_version: '5.7'
charset: utf8mb4
url: 'mysql://nibbr:nibbr#127.0.0.1:3306/School_B'
orm:
default_entity_manager: default
entity_managers:
default:
connection: default
naming_strategy: doctrine.orm.naming_strategy.underscore
mappings:
App:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity'
prefix: 'App\Entity'
alias: App
4bf40159870dc1b23c97e7906a303f39:
connection: School_A
naming_strategy: doctrine.orm.naming_strategy.underscore
mappings:
App:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity'
prefix: 'App\Entity'
46f0618dadff645591073709906f006c:
connection: School_B
naming_strategy: doctrine.orm.naming_strategy.underscore
mappings:
App:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src'
prefix: 'App'
now the Databases exist,
migrations using
php bin/console doctrine:migrations:migrate --em=46f0618dadff645591073709906f006c
worked perfectly fine.
but i cannot switch to specific EntityManagers to manipulate the correct database. it always uses the Default.
I know this has been touched on alot but no real answers have been given
each database uses all the same Entities and repositories. all i want to change is which database symfony 4 uses.
$rep = $this->getDoctrine()->getRepository(SchoolStudent::class,'46f0618dadff645591073709906f006c');
when i persist a new entry though $rep, no matter what value gets passed it only uses the first entity manager and connection it comes across in the doctrine.yaml. unless i pass a string that isnt an EntityManager name, then i get a 500 error.
using ANY variation of ->getManager() causes 500 error too. im fairly new to symfony
thank you kindly
I use two different managers that way (in services with dependency injection):
In this example i inject the repository directly into the service constructor.
<service id="my_service_id" class="FQCN">
<argument type="service">
<service class="Doctrine\ORM\DocumentRepository">
<factory service="doctrine.orm.46f0618dadff645591073709906f006c_entity_manager" method="getRepository"/>
<argument>Namespace\SchoolStudent</argument>
</service>
</argument>
</service>

Categories