I'm writing a Symfony application, and I am facing a problem with Doctrine entities eager loading.
I'm not sure if I can load multiple one to many collections on the same entity instance while keeping good performance.
There are many examples on the internet with people load one entity and one of its relationships.
Ex :
$user = $em->createQuery('
select a, au
from OctiviTestBundle:Article a
left join a.authorList au
where a.id = ?1')
->setParameter(1, $id)
->getOneOrNullResult();
However if I also want to load the article comments, the following request retrieves too many results (nb authors * nb comments) => combinatory explosion
$user = $em->createQuery('
select a, au
from OctiviTestBundle:Article a
left join a.authorList au
left join a.commentList c
where a.id = ?1')
->setParameter(1, $id)
->getOneOrNullResult();
In fact, I found no way to reuse an object once it has been loaded from the database. I don't know how to make a second query to load more parts of it later.
Eg :
$user = $em->createQuery('
select a, au
from OctiviTestBundle:Article a
left join a.authorList au
where a.id = ?1')
->setParameter(1, $id)
->getOneOrNullResult();
$em->LoadRelation($user, 'commentList');
$em->LoadRelation($user, 'commentList.author')
$em->LoadRelation($user, 'commentList.author.school');
... load any relation I want, while keeping only one root object.
I would like to be able to have only the main entity instance variable, eager load its 2 relationships and then go through the hierarchy.
I know I can load the two lists in different php variables, but I'd like to only pass the "$user" variable to the view template.
Do you have ideas about how to resolve this issue ?
Thanks
The only (tricky) solution I found is on this website : https://tideways.io/profiler/blog/5-doctrine-orm-performance-traps-you-should-avoid
1) Load the related entities
$companies = array_map(function($employee) {
return $employee->getCompany();
}, $employees);
$repository = $entityManager->getRepository('Acme\Company');
$repository->findBy(['id' => $companies]);
2) Don't use the result (drop the $companies variable), but now Doctrine has got the results in cache, so when I do $employee->getCompany()->getName(), it should not generate new queries.
=> Doesn't work : Doctrine doesn't put the results in the cache to reuse them later.
If the result is always going to be massive no matter what where's and ands you put in, and if you have no intent on doing any write operations on the result you should fetch them as an array instead of as the default objects.
The resulting array is still used exactly the same within twig however as you no longer have to hydrate the objects your memory saving will be huge.
Related
I have a very particular use case, and i can't find a clean solution with the ORM. I've searched a lot, and maybe my database model is not correct, I'm not sure.
I use CakePHP 3.8.11.
So, I have a table "MaintenanceTypes" with 3 important fields : id, name, and periodicity. Periodicity (in days) means "this maintenance is to be done every (for instance) 30 days".
Periodicity are like 7 (week), 30 (month), 90 (trimester) and so on.
I also have a table "Operations", they are little unit tests that belongs to a "MaintenanceType" (fields are id, name, maintenance_type_id).
What is special in this case, is that, as a business rule, Operations belonging to a MaintenanceType with a periodicity of 7 days is "included" in every MaintenanceType with a greater periodicity; that means that every trimester, you should do every Operations associated directly to the trimester, but also every Operations associated with the month, and the week, etc.
In raw SQL it's trivial :slight_smile:
mt_ref is the reference MaintenanceType, mt_inc are the included MaintenanceTypes (with a lesser periodicity) and finally, every Operations belonging to any of the MaintenanceTypes found.
SELECT mt_ref.id, mt_ref.name, mt_ref.periodicity,
mt_inc.name, mt_inc.periodicity, o.name
FROM maintenance_types mt_ref
LEFT JOIN maintenance_types mt_inc
ON (mt_inc.periodicity <= mt_ref.periodicity)
LEFT JOIN operations o ON (o.maintenance_type_id = mt_inc.id)
WHERE mt_ref.id = 3
I've tried to declare the association between MaintenanceTypes, but I can't find a way to declare that the association is done on the periodicity field, and, extra points, not on a equality but on a "less or equal".
To add extra difficulties, I use this query for a (very good) JQuery Datatables CakePHP plugin (https://github.com/allanmcarvalho/cakephp-datatables), so I can't simply pass the raw SQL, and I must use the ORM...
I hope this is clear, and that someone could help me on this one !
Thanks a lot !
If you need query builder instances, then pretty much have two options here that are more or less straightforward, that is either use associations with custom conditions, or manual joins.
Custom association conditions
With associations you'd probably do something like a self-association with MaintenanceTypes with a disabled foreign key and custom conditions, like so in your MaintenanceTypesTable class:
$this
->hasMany('MaintenanceTypesInc')
->setClassName('MaintenanceTypes')
->setForeignKey(false)
->setConditions(function (
\Cake\Database\Expression\QueryExpression $exp,
\Cake\ORM\Query $query
) {
return $exp->lte(
$query->identifier('MaintenanceTypesInc.periodicity'),
$query->identifier('MaintenanceTypes.periodicity')
);
});
Disabling the foreign key will prevent the ORM from creating the default A.fk = B.fk condition when joining in the association. It should be noted that you cannot contain a hasMany association with a disabled foreign key, you can only join it in! You could use a hasOne or even belongsTo association instead, but it would kinda be a lie, as you don't have a 1:1 relationship here (at least as far as I understand it).
Also note that you don't neccesarily have to use a callback with all expressions for the conditions, you could pass the conditions as a key => value array with a manually instantiated identifier expression, or even as simple string (the latter however will not be recognized when using automatic identifier quoting):
->setConditions([
'MaintenanceTypesInc.periodicity <=' =>
new \Cake\Database\Expression\IdentifierExpression('MaintenanceTypes.periodicity');
]);
->setConditions('MaintenanceTypesInc.periodicity <= MaintenanceTypes.periodicity');
Assuming you also have an association for Operations in your MaintenanceTypesTable class, you should be able to join in both, the new association and the Operations association via the query builders *JoinWith() methods, like this:
$query = $maintenanceTypesTable
->find()
->select([
'MaintenanceTypes.id', 'MaintenanceTypes.name', 'MaintenanceTypes.periodicity',
'MaintenanceTypesInc.name', 'MaintenanceTypesInc.periodicity',
'Operations.name',
])
->leftJoinWith('MaintenanceTypesInc.Operations');
In the results, the association data will be put under the _matchingData key, ie you can obtain it like $entity->_matchingData->MaintenanceTypesInc and $entity->_matchingData->Operations. If you don't want that, then you need to use aliases for the fields of the associations, like:
->select([
'MaintenanceTypes.id', 'MaintenanceTypes.name', 'MaintenanceTypes.periodicity',
'mt_inc_name' => 'MaintenanceTypesInc.name', 'mt_inc_periodicity' => 'MaintenanceTypesInc.periodicity',
'op_name' => 'Operations.name',
])
If you don't want to select all the fields everytime, use a custom finder as in the manual joins example below.
Manual joins
Using manual joins gives you complete freedom, with the query builders *Join() methods you can create whatever joins you like, and you don't have to use possible workarounds with associations.
You can add them in a custom finder for proper reusability, it could look something like this in your MaintenanceTypesTable class:
public function findWithIncludedMaintenanceTypes(\Cake\ORM\Query $query, array $options)
{
return $query
->select(/* ... */)
->leftJoin(
['MaintenanceTypesInc' => 'maintenance_types'],
function (
\Cake\Database\Expression\QueryExpression $exp,
\Cake\ORM\Query $query
) {
return $exp->lte(
$query->identifier('MaintenanceTypesInc.periodicity'),
$query->identifier('MaintenanceTypes.periodicity')
);
}
)
->leftJoin(
['Operations' => 'operations'],
function (
\Cake\Database\Expression\QueryExpression $exp,
\Cake\ORM\Query $query
) {
return $exp->equalFields(
'Operations.maintenance_type_id ',
'MaintenanceTypesInc.id'
);
}
);
}
Then you simply use the finder wherever you need it, like this:
$query = $maintenanceTypesTable
->find('withIncludedMaintenanceTypes');
Note that just like in the associations example, you can use string or array conditions for the custom joins too.
See also
Cookbook > Database Access & ORM > Associations - Linking Tables Together
Cookbook > Database Access & ORM > Query Builder > Loading Associations
Cookbook > Database Access & ORM > Query Builder > Loading Associations > Using leftJoinWith
Cookbook > Database Access & ORM > Query Builder > Loading Associations > Adding Joins
Cookbook > Database Access & ORM > Retrieving Data & Results Sets > Using Finders to Load Data
Cookbook > Database Access & ORM > Retrieving Data & Results Sets > Custom Finder Methods
I have a Book entity which has a One2Many relationship with the Page entity. I wanted to create a query which retrieved all the books which had at least one page. I did:
$qb = $this->getDoctrine()
->getRepository('AcmeDemoBundle:Book')
->createQueryBuilder('b');
->leftJoin('b.pages','p')
->having($qb->expr()->gt($qb->expr()->count('p'), 0));
$books = $qb
->getQuery()
->getResult();
The problem is that, although there are many books which have pages, this query only returns a single book.
The query created is:
SELECT b FROM Acme\DemoBundle\Entity\Book b LEFT JOIN b.pages p HAVING COUNT(p) > 0
which looks fine to me. Any idea what may be wrong?
It is simpler to use
->innerJoin('b.pages','p')
instead of
->leftJoin('b.pages','p')
you don't need aggregate function. In short: this innerjoin will return only Books that can be joined with at least one Page.
You should add groupBy as you are using aggregate function. e.g.
$qb = $this->getDoctrine()
->getRepository('AcmeDemoBundle:Book')
->createQueryBuilder('b');
->leftJoin('b.pages','p')
->groupBy('b.id')
->having($qb->expr()->gt($qb->expr()->count('p'), 0));
I have an entity Party with an one-to-many bi-directional association to PartyContact. I am trying to write a DQL statement to return a filtered PartyContact collection while using object hydration. I can return the desired result using array hydration but when I attempt to return a pure object hydration I receive the total collection.
There are a total of 5 PartyContact instances association with a Party(with an id of 1), and 3 of the 5 PartyContact instances associated with this Party instance have a property named type with a value of PartyContact::TYPE_HOME; So I should be able to return a Party instance with filtered collection of 3 PartyContact instances.
In the past I have used event listeners, closures, and or filter iterators to handle filtering of collections on entities. Performance wise this seems like a bit much. I am aware of the new Filters with Doctrine 2.2 but would like to be able to handle this in DQL in my repositories.
FYI I am using 2.2.0-BETA2 (note: just checked this and I know that there has been a release of 2.2.2 so I will try that now)
I have the following dql:
$query = $this->_em->createQuery("SELECT p, c FROM Test\Party p LEFT JOIN p.contactMechanisms c WITH c.type = :type WHERE p.id = :id");
$query->setParameter('type', PartyContact::TYPE_HOME);
$query->setParameter('id', 1);
$query->setFetchMode("Test\Party", "contactMechanisms", "EAGER");
$results = $query->getResult(Query::HYDRATE_ARRAY);
$party = $results[0];
My results are as follows:
When using Query::getResult(Query::HYDRATE_ARRAY):
count($party['contactMechanisms']) = 3
When using Query::getResult(Query::HYDRATE_ARRAY):
$party->getContactMechanisms()->count() = 5
I have a simple entity with many-to-many and one-to-many associations. I'm aware of 'Joins' for fetching related associations which is a manual solution for my problem.
How can I fetch an entity with all of its associations using EntityManager in Doctrine2? e.g.:
$this->em
->getRepository('Entities\Patientprofile')
->findOneByuserid('555555557')
->fetchAllAssociations();
from http://doctrine-orm.readthedocs.org/en/latest/reference/dql-doctrine-query-language.html#temporarily-change-fetch-mode-in-dql
you can set eager fetch mode temporarily:
$query = $em->createQuery("SELECT u FROM MyProject\User u");
$query->setFetchMode("MyProject\User", "address", "EAGER");
$query->execute();
If you want do load dynamically all associations with this fetch mode, you can use the getAssociationMappings() method of the Doctrine\ORM\Mapping\ClassMetadataInfo, passing your entity name as parameter to the constructor of ClassMetadataInfo and then iterate over the returned array as $assoc and call:
$query->setFetchMode("MyProject\User", $assoc, "EAGER");
Doc: ClassMetadataInfo#getAssociationMappings()
Doctrine2 setFetchMode not working with "EAGER"
I tried also to fetch the associating entities "eagerly" using setFetchMode in my query, but the following didn't seem to work:
$query->setFetchMode("MyProject\User", "address", "EAGER");
When I jumped into the files I found out that the third parameter $fetchMode should be an integer. The constants are defined in Doctrine\ORM\Mapping:ClassMetadataInfo. When passing a string it will default to Mapping\ClassMetadata::FETCH_LAZY because of this if clause.
/**
* Specifies that an association is to be fetched when it is first accessed.
*/
const FETCH_LAZY = 2;
/**
* Specifies that an association is to be fetched when the owner of the
* association is fetched.
*/
const FETCH_EAGER = 3;
/**
* Specifies that an association is to be fetched lazy (on first access) and that
* commands such as Collection#count, Collection#slice are issued directly against
* the database if the collection is not yet initialized.
*/
const FETCH_EXTRA_LAZY = 4;
So setting the corresponding integer solved the problem:
$query->setFetchMode("MyProject\User", "address", 3);
Or declare the class use Doctrine\ORM\Mapping\ClassMetadata at the top and then use the constant:
$query->setFetchMode("MyProject\User", "address", ClassMetadata::FETCH_EAGER);
EDIT:
Since there seems to be a lot of confusion here on how to fetch associations the right way I will edit my answer and add some additional information on how you can fetch join using your repository.
According to the Doctrine documentation there are 2 types of joins:
Regular Joins: Used to limit the results and/or compute aggregate values.
Fetch Joins: In addition to the uses of regular joins: Used to fetch related entities and include them in the hydrated result of a
query.
So to get an entity including its associations you will need to "fetch-join" all these associations to make sure they are loaded eagerly.
I usually don't use DQL queries for getting entities and solving my fetch joins, instead I add a custom method to a repository where I use a query builder. This is more flexible and much more readable then using DQL. The correct DQL query will be created by the query builder when we call the createQuery method. You can check the created DQL query of course for debug purposes.
An example for such a custom method inside the Patientprofile entity repository from the question above:
public function findPatientByIdWithAssociations($id)(
// create a query builder for patient with alias 'p'
$qb = $this->createQueryBuilder('p')
->where('p.id = :patient_id')
->addSelect('pd')
->leftJoin('p.documentation', 'pd')
->addSelect('pa')
->leftJoin('p.address', 'pa')
->setParameter('patient_id', $id);
$query = $queryBuilder->getQuery();
return $query->getSingleResult();
}
And now you can use your custom repository method to get the patient by id (for example '555555557') including associations to the patient documentation and address:
$repository = $this->em->getRepository('Entities\Patientprofile');
$patient = $repository->findPatientByIdWithAssociations('555555557');
Make sure you use both addSelect and leftJoin to do eager loading.
Doctrine 2 uses Proxy classes for lazy loading, so you don't actually need to have the associations' data fetched until you use the objects. Since the Proxy classes inherit from your association classes, you're able to use the proxies exactly as you would use the fretch association classes.
but, if you really need to fetch the actual association classes, you need to tell the query to set the fetch mode to Doctrine\ORM\Mapping\ClassMetadata::FETCH_EAGER. If you're using the annotations, you can achieve this with:
e.g.
/**
* #ManyToMany(targetEntity="Item", fetch="EAGER")
*/
private $items;
You can use a DQL query:
$query = $em->createQuery("SELECT p, f FROM Entities\\Patientprofile p JOIN p.Foo f WHERE p.id = ?1");
$query->setParameter(1, 321);
$patient = $query->getSingleResult();
Faced the same problem.
It was necessary to pull out all chain of parents of an element.
$query->setFetchMode(EntityClass, "alias_in_entity", 3) gets only 1 lvl deep, other parents are just proxy.
This can be fixed by changed in entity class fetch mode to eager. But if it`s not if this is not possible for some reason (performance etc), this can be made as #wormhit mentioned by changing entity metadata "on fly"
Example:
$query = $this->entityManager->createQueryBuilder()->select('fields')
->from(FormField::class, 'fields');
$metadata = $this->entityManager->getClassMetadata(FormField::class);
$metadata->setAssociationOverride('parent', ['fetch' => \Doctrine\ORM\Mapping\ClassMetadata::FETCH_EAGER]);
return $query->getOneOrNullResult();
I have a tree structure with a parent field. Currently I am trying to get all parent nodes to display the path to the current node.
Basically I am doing a while-loop to process all nodes.
$current = $node->getParent();
while($current) {
// do something
$current = $current->getParent();
}
Using the default findById method works. Because the entity has some aggregated fields, I am using a custom repository method, to load all basic fields with one query.
public function findNodeByIdWithMeta($id) {
return $this->getEntityManager()
->createQuery('
SELECT p, a, c, cc, ca, pp FROM
TestingNestedObjectBundle:NestedObject p
JOIN p.actions a
LEFT JOIN p.children c
LEFT JOIN c.children cc
LEFT JOIN c.actions ca
LEFT JOIN p.parent pp
WHERE p.id = :id
')
->setParameter('id', $id)
->setHint(
\Doctrine\ORM\Query::HINT_CUSTOM_OUTPUT_WALKER,
'Gedmo\\Translatable\\Query\\TreeWalker\\TranslationWalker'
)
->getOneOrNullResult();
}
With that code, loading the parents fails. I only get the immediate parent (addressed by LEFT JOIN p.parent pp) but not the parents above. E.g. $node->getParent()->getParent() returns null.
Whats wrong with my code? Did I misunderstood the lazy loading thing?
Thanks a lot,
Hacksteak
It looks like your are using the adjacency model for storing trees in a relational database. Which in turn means, that you will need a join for every level to get all ancestors with a single query.
As you are already using the Doctrine Extension Library I recommend to have a look at the Tree component.
My Answer involves not using DQL and instead creating a NestedSetManager which has access to your DBAL connection so you can use SQL. I never felt like the ORM's did a good job with NestedSets query logic.
With a NestedSetManager, you can then write a bunch of clean methods and it's really simple because all these queries are well documented. See this link. Some of the method in my NestedSetManager are:
setNode();
setRoot();
loadNestedSet();
moveNodeUp();
modeNodeDown();
getRootNode();
addNodeSibling();
getNodesByDepth();
getParents();
getNodePath();
childExists();
addChildToNode();
renameNode();
deleteNode();
// And many more
You can have a ball and create a lot of create NestedSet functionality if you're not tied down by an ORM's somewhat complex functionality.
Also -- Symfony2 makes all this really really easy. You create your NestedSetManager class file and reference it in your Services.yml and pass in your Dbal connection. Mine looks like this:
services:
manager.nestedset:
class: Acme\CoreBundle\Manager\NestedSetManager
arguments: [ #database_connection ]
you can then access your nestedsets with:
$path = $this->get('manager.nestedset')->setNode(4)->getNodePath(); // in your controller
Moral of the story, ORM/NestedSets drove me bonkers and this solution work really well. If you're being forced to use DQL and have no other options, this answer probably wont be acceptable.