loading of related documents in doctrine ODM leads to too many queries - php

I'm stuck trying to reduce the number of database queries on a web api.
My database has 3 collections : playground, widget, token
One playground has many widgets, one widget has one token. Each relationship uses referencesOne/referenceMany.
So here are my simplified models
/**
* #MongoDB\Document()
*/
class Widget
{
/**
* #MongoDB\ReferenceOne(targetDocument="Token", inversedBy="widgets")
*/
protected $token;
/**
* #MongoDB\ReferenceOne(targetDocument="Playground", inversedBy="widgets")
*/
protected $playground;
}
/**
* #MongoDB\Document()
*/
class Playground
{
/**
* #MongoDB\ReferenceMany(targetDocument="Widget", mappedBy="playground")
*/
protected $widgets;
}
/**
* #MongoDB\Document()
*/
class Token
{
/**
* #MongoDB\ReferenceMany(targetDocument="Widget", mappedBy="token")
*/
protected $widgets;
}
I need to use the full playground with all its widgets and tokens but by default, Doctrine does too many queries : one to get the playground (ok), one to get all widgets of the mapping (ok) and for each widget, one query to get the token (not ok). Is there a way to query all tokens at once instead of getting them one by one ?
I've looked at prime but it does not seem to solve my problem...
Is there a way other than using the query builder and manually hydrate all objects to reduce the query count ?
Edit :
As I added in my comment, what I'm looking for is get the playground and all its dependencies as a big object, json encode it and return it into the response.
What I do for now is query the playground and encode it but Doctrine populates the dependencies in a non efficient way : first there is the query to get the playgroung, then, there is one more query to get the related widgets and there is one query for each widget to get its token.
As one playground can have hundreds of widgets, this leads to hundreds of database queries.
What I'm looking for is a way to tell Doctrine to get all this data using only 3 queries (one to get the playgroung, one to get the widgets and one to get the tokens).

update: since the ArrayCollection in the $playground should contain all the widgets at least as proxy objects (or should get loaded when accessed), the following should work to fetch all required tokens...
Since the document manager keeps all managed objects, it should prevent additional queries from occuring. (Notice the omitted assignment from the execute).
$qb = $dm->createQueryBuilder('Token')->findBy(['widget' => $playground->getWidgets()]);
$qb->getQuery()->execute();
inspired by this page on how to avoid doctrine orm traps - point 5
old/original answer
I'm not quite familiar with mongodb, tbh, but according to doctrine's priming references, you should be able to somewhat comfortably hydrate your playground by:
$qb = $dm->createQueryBuilder('Widget')->findBy(['playground' => $playground]);
$qb->field('token')->prime(true);
$widgets = $qb->getQuery()->execute();
however, I might be so wrong.

What about that:
class Foo
{
/** #var \Doctrine\ORM\EntityManagerInterface */
private $entityManager;
public function __construct(\Doctrine\ORM\EntityManagerInterface $entityManager)
{
$this->entityManager = $entityManager;
}
public function getAll(): array {
$qb = $this->entityManager->createQueryBuilder();
return $qb
->select('p,t,w')
->from(Playground::class, 'p')
->join(Widget::class, 'w')
->join(Token::class, 't')
->getQuery()
->getResult();
}
}
At least with mysql backend, this solves the "n+1 problem". Not sure about MongoDB, though.

Related

Symfony4 - How to update a doctrine ArrayCollection ?

I have a member of my entity is an arrayCollection. With a classic form builder is working fine, I can select multiple items and persist it. But when I try to update an object in controller I get the error : "Call to a member function setFaavailability() on array".
A resume of my entity :
/**
* #ORM\ManyToOne(targetEntity="App\Entity\FaAvailability",
inversedBy="faavailability")
* #ORM\JoinColumn(nullable=true)
* #ORM\Column(type="array")
*/
public $faavailability;
/**
* #return mixed
*/
public function getFaavailability()
{
return $this->faavailability;
}
/**
* #param mixed $faavailability
*/
public function setFaavailability($faavailability)
{
$this->faavailability = $faavailability;
}
In my controler :
$varFaavailability = $animal->faperson->getFaavailability();
foreach($varFaavailability as $availability){
if($availability->getName() == $animal->typepet->getName()){
$varFaavailability->removeElement($availability);
$faPerson = $em->getRepository(FaPerson::class) >findById($animal->faperson->getId());
$faPerson->setFaavailability($varFaavailability);
$em->persist($faPerson);
$em->flush();
}
}
Any ideas ?
If I remember well, when you set a field as an ArrayCollection it means that you have a oneToMany relationship between two entities.
From your code, I can tell you that you are trying to persist the data in the wrong entity. You usually add the owning_entity_id(1-to-N) in each item(1-to-N) and persist it. In your code, you are trying to set all the references at once, which is never going to happen. Delete the setFaavailability() or redefine the entities' relationships.
You should never try to mass-add foreign key relationships in one super duper setter function. Cycle through all the items and set the reference to the "parent" entity.
The problem is in this part: $faPerson = $em->getRepository(FaPerson::class)->findById($animal->faperson->getId());
The findBy* methods will try to find multiple entities and return them in a Collection.
If you're looking for a single person, you can use findOneById instead. Or (assuming id is configured as identifier in Doctrine) you can even use the find method: $faPerson = $em->getRepository(FaPerson::class)->find($animal->faperson->getId());
some general comments:
In Doctrine you never have to work with the IDs. Use the entity
objects! You only need to findById if you get the ID from a request parameter for example.
You should reconsider the naming of your variables to make it clear if it is a collection ($availabilities) or a single one ($availability).
Always use the getter/setter methods instead of the fields (typepet vs getTypepet()).
Call flush() one at the end to update all entities in one single transaction.
I've renamned the variables below as I understood them. However I am still not sure what $animal->faperson->getFaavailabilities() returns, since at the beginning you wanto to loop through the results and later set it to a single one via setFaavailability()?
//Should be a Doctrine ArrayCollection
$varFaavailabilities = $animal->faperson->getFaavailabilities();
foreach($varFaavailability as $availability){
if($availability->getName() == $animal->getTypepet()->getName()) {
//Why do you want to remove an element from the current loop?
$varFaavailability->removeElement($availability);
//No need to use Id
$faPerson = $animal->getFaperson();
//A single one?
$faPerson->setFaavailability($availability);
//More than one? addFaavailability should exist.
$faPerson->addFaavailability($availability);
$em->persist($faPerson);
}
}
$em->flush();

How to retrieve grandchildren through a polymorphic relation in Laravel 5.2?

My database design looks like the below diagram.
A course has many modules
Modules morphs to Scorm and other 2 tables (not used in this example)
A scorm Morph one Module and has Many Scoes
I am trying to access from my course model all the scoes related using:
$course->scoes.
I am using Laravel 5.2, using the Eloquent relations I know this is not possible. After few tests, using the query builder I actually can return the correct data, but, they come back as being an instance of Module::class instead of Scorm, and Sco.
This is the code I have now.
Thanks,
public function modules() {
return $this->hasMany(Module::class);
}
public function scorms(){
return $this->modules()->where('moduleable_type','=','scorms');
}
public function scoes(){
return $this->scorms()->select('scoes.*')
->join('scoes','modules.moduleable_id','=','scoes.scorm_id');
}
I'm not sure but this may be exactly what you want.
https://laravel.com/docs/5.2/eloquent-relationships#has-many-through
I found a way to do what I was trying to do. This way makes only 2 queries to the db and it's located in the Sco Model.
//returns a Course Object.
$sco->course()
/**
* Get the course
* #return App\Course|null
*/
public function Course(){
$o = $this
->join('scorms','scoes.scorm_id','=','scorms.id')
->join('modules',function($join){
$join
->on('moduleable_id','=','scorms.id')
->where('moduleable_type','=','scorms');
})
->join('courses','modules.course_id','=','courses.id')
->select('courses.id as course_id')
->first();
if(!$o) return null;
return Course::find($o->course_id);
}

How to get doctrine2 entity one to many entities that are set "active"

Let's assume there is a OneToMany doctrine2 association between blogposts and comments. A blogposts might have many comments. Every comment remains inactive and therefore hidden in the frontend until a moderator will activate the comment manually.
I'm now trying to have some kind of security facade to ensure that only "active" comments will be provided to the view by accessing them in a loop over the {{blogpost.comments}} variable in the twig template.
Trying to use the getComments() method in the blogpost entity I was trying to filter the ArrayCollection of comments like so
/**
* #return ArrayCollection
*/
public function getComments()
{
return $this->comments->filter(function ($condition) {
return $condition->getActive() === true;
});
}
unfortunately Doctrine will entirely load every single comment even if the relations fetch mode is set to "EXTRA_LAZY". So that would influence the performance of an application in a way i'd like to avoid.
Is there any way to hide the inactive comments globally or do I have to take care of filtering them every time I'm accessing the blogpost.comments relations in the view?
You should use the matching method of your collection. If your collection is not loaded, it will add filters to the SQL query to only load what you need. If your collection is already loaded, it will filter the PHP array.
use Doctrine\Common\Collections\Criteria;
public function getComments()
{
return $this->comments->matching(
Criteria::create()->where(
Criteria::expr()->eq('active', true)
)
);
}
More informations here: http://doctrine-orm.readthedocs.org/projects/doctrine-orm/en/latest/reference/working-with-associations.html#filtering-collections
Regards

Advanced Filtering of Associated Entity Collection in Symfony

If I have an associated entity which is a collection, what options do you have when fetching?
e.g. Lets say I have a $view entity with this definition inside it:
/**
* #ORM\OneToMany(targetEntity="\Gutensite\CmsBundle\Entity\View\ViewVersion", mappedBy="entity")
* #ORM\OrderBy({"timeMod" = "DESC"})
*/
protected $versions;
public function getVersions() {
return $this->versions;
}
And I want to get the all the versions associated with the entity like this:
$view->getVersions();
This will return a collection. Great. But is it possible to take that collection and filter it by criteria, e.g. newer than a certain date? Or order it by some (other) criteria?
Or at this point are you just expected to do a query on the repository:
$versions = $em->getRepository("GutensiteCmsBundle:View\ViewVersion")->findBy(array(
array(
'viewId', $view->getId(),
'timeMod', time()-3600
)
// order by
array('timeMod', 'DESC')
));
There is a surprisingly unknown feature in recent versions of Doctrine, which makes these sort of queries much easier.
It doesn't seem to have a name, but you can read about it in the Doctrine docs at 9.8 Filtering Collections.
Collections have a filtering API that allows to slice parts of data from a collection. If the collection has not been loaded from the database yet, the filtering API can work on the SQL level to make optimized access to large collections.
In your case you could write a method like this on your View entity.
use Doctrine\Common\Collections\Criteria;
class View {
// ...
public function getVersionsNewerThan(\DateTime $time) {
$newerCriteria = Criteria::create()
->where(Criteria::expr()->gt("timeMod", $time));
return $this->getVersions()->matching($newerCriteria);
}
}
This will do one of two things:
If the collection is hydrated, it will use PHP to filter the existing collection.
If the collection is not hydrated, it will fetch a partial collection from the database using SQL constraints.
Which is really great, because hooking up repository methods to your views is usually messy and prone to break.
I also like #igor-pantovic's answer, although I've seen the method cause some funny bugs.
I would personally avoid using order by on annotation directly. Yes, you are supposed to do a query, just as you would if you were using raw SQL without Doctrine at all.
However, I wouldn't do it at that point but even before. In your specific case I would create an ViewRepository class:
class ViewRepository extends EntityRepository
{
public function findWithVersionsNewerThan($id, \DateTime $time)
{
return $this->createQueryBuilder('view')
->addSelect('version')
->join('view.versions', 'version')
->where('view.id = :id')
->andWhere('version.timeMod > :time')
->setParameter('time', $time)
->setParameter('id', $id)
->getQuery()
->getOneOrNullResult();
}
}
Now you can do:
$yourDateTime = // Calculate it here ... ;
$view = $em->getRepository("GutensiteCmsBundle:View\ViewVersion")->findWithVersionsNewerThan($yourDateTime);
$versions = $view->getVersions(); // Will only contain versions newer than datetime provided
I'm writing code from the top of my head here directly so sorry if some syntax or method naming error sneaked in.

Doctrine 2 - How to use objects retrieved from cache in relationships

I'm working in a project that use Doctrine 2 in Symfony 2 and I use MEMCACHE to store doctrine's results.
I have a problem with objects that are retrieved from MEMCACHE.
I found this post similar, but this approach not resolves my problem: Doctrine detaching, caching, and merging
This is the scenario
/**
* This is in entity ContestRegistry
* #var contest
*
* #ORM\ManyToOne(targetEntity="Contest", inversedBy="usersRegistered")
* #ORM\JoinColumn(name="contest_id", referencedColumnName="id", onDelete="CASCADE"))
*
*/
protected $contest;
and in other entity
/**
* #var usersRegistered
*
* #ORM\OneToMany(targetEntity="ContestRegistry", mappedBy="contest")
*
*/
protected $usersRegistered;
Now imagine that Contest is in cache and I want to save a ContestRegistry entry.
So I retrieve the object contest in cache as follows:
$contest = $cacheDriver->fetch($key);
$contest = $this->getEntityManager()->merge($contest);
return $contest;
And as last operation I do:
$contestRegistry = new ContestRegistry();
$contestRegistry->setContest($contest);
$this->entityManager->persist($contestRegistry);
$this->entityManager->flush();
My problem is that doctrine saves the new entity correctly, but also it makes an update on the entity Contest and it updates the column updated. The real problem is that it makes an update query for every entry, I just want to add a reference to the entity.
How I can make it possible?
Any help would be appreciated.
Why
When an entity is merged back into the EntityManager, it will be marked as dirty. This means that when a flush is performed, the entity will be updated in the database. This seems reasonable to me, because when you make an entity managed, you actually want the EntityManager to manage it ;)
In your case you only need the entity for an association with another entity, so you don't really need it to be managed. I therefor suggest a different approach.
Use a reference
So don't merge $contest back into the EntityManager, but grab a reference to it:
$contest = $cacheDriver->fetch($key);
$contestRef = $em->getReference('Contest', $contest->getId());
$contestRegistry = new ContestRegistry();
$contestRegistry->setContest($contestRef);
$em->persist($contestRegistry);
$em->flush();
That reference will be a Proxy (unless it's already managed), and won't be loaded from the db at all (not even when flushing the EntityManager).
Result Cache
In stead of using you own caching mechanisms, you could use Doctrine's result cache. It caches the query results in order to prevent a trip to the database, but (if I'm not mistaken) still hydrates those results. This prevents a lot of issues that you can get with caching entities themselves.
What you want to achieve is called partial update.
You should use something like this instead
/**
* Partially updates an entity
*
* #param Object $entity The entity to update
* #param Request $request
*/
protected function partialUpdate($entity, $request)
{
$parameters = $request->request->all();
$accessor = PropertyAccess::createPropertyAccessor();
foreach ($parameters as $key => $parameter) {
$accessor->setValue($entity, $key, $parameter);
}
}
Merge requires the whole entity to be 100% fullfilled with data.
I haven't checked the behavior with children (many to one, one to one, and so on) relations yet.
Partial update is usually used on PATCH (or PUT) on a Rest API.

Categories