I am using Symfony's serializer to return a JSON response. I use groups to only return information that is relevant:
return new JsonResponse($this->serializer->serialize($userRepository->getActiveUser(), 'json', ['groups' => ['contact-information']]));
My User class has additional relationships I am not interested in for the above request, like paymentProfile. This property has a different group:
class User {
...
/**
* #ORM\ManyToOne(targetEntity=PaymentProfile::class)
* #Groups({"payment-information"})
*/
protected $paymentProfile;
}
The JSON returned does NOT include the paymentProfile property for Users (as expected), yet looking at the Symfony Profiler, I see that the request required 100s of individual paymentProfile queries (one for each users returned):
Repository:
public function getActiveUsers()
{
return $this->createQueryBuilder('u')
->andWhere('u.active= :active')
->setParameter('active', true)
->getQuery()->getResult()
;
}
Don't serialization groups prevent unnecessarily fetching data? If yes, what am I doing wrong? If not, how can I get rid of the extra queries?
EDIT:
Symfony\Component\Serializer\Normalizer\AbstractNormalizer::IGNORED_ATTRIBUTES is also not fixing the issue.
This is why:
There is a general performance consideration with Single Table Inheritance: If the target-entity of a many-to-one or one-to-one
association is an STI entity, it is preferable for performance reasons that it
be a leaf entity in the inheritance hierarchy, (ie. have no subclasses).
Otherwise Doctrine CANNOT create proxy instances of this entity and will ALWAYS load the entity eagerly. (https://www.doctrine-project.org/projects/doctrine-orm/en/2.6/reference/inheritance-mapping.html#performance-impact)
H/T https://stackoverflow.com/a/25754165/1550361
Related
API Platform version(s) affected:
/srv/api # composer show | grep api-platform
api-platform/core v2.6.8 Build a fully-featured hypermedia or GraphQL API in minutes!
Description
To define the response of our API endpoints, we have used attributes on the generated Doctrine entity such as:
/**
* #ORM\Table(name = "products")
* #ORM\Entity(repositoryClass=ProductRepository::class)
*/
#[ApiResource(
collectionOperations: [
'get' => [
'path' => '/products',
],
],
itemOperations: [
'get' => [
'path' => '/products/{id}',
],
],
normalizationContext: [
'groups' => [
'product:read',
],
],
output: ProductOutput::class,
)]
class Product {
.... // properties and getters+setters
}
The Product entity has a 1:n relation to the Variant entity which is also a ApiResource with an different endpoint /variants. The Variant entity has several relations to other entities and some values of all entities are translatable with https://github.com/doctrine-extensions/DoctrineExtensions/blob/main/doc/translatable.md.
The performance was as expected => good enough.
Later on, it was required to "enrich" the response of /products and /variants with some data, which was not mapped in relations between Product <> additional-data | Variant <> additional-data, so we decided to use Outputs DTO with DataTransformers, as documented in the API-Platform docs.
The DataTransformer's method transform puts the data into the DTO by using therespective getters of the entities, e. g.:
$output = new ProductOutput();
$output->id = $object->getId();
$output->category = null !== $object->getCategory() ?
$this->iriConverter->getIriFromItem($object->getCategory()) :
'';
$output->identifierValue = $object->getIdentifierValue();
$output->manufacturer = $object->getManufacturer();
$output->variants = $object->getVariants();
The $object is a Product entity, in this case.
The DTO contains only public properties, such as
/**
* #var Collection<int, Variant>
*/
#[Groups(['product:read'])]
public Collection $variants;
and the Groups attributes, which are also defined in the normalizationContext of the ApiResource attribute in the Product entity above.
After that, we found the performance had drastically deteriorated: A request to the /products endpoint which "lists" 30 products with the related variants needs around 25 seconds.
After analyzing, we determined the following:
without DTO: Doctrine runs one single query with a lot of joins to retrieve all the related data from the database.
with DTO: Doctrine runs in sum 3.155 single queries to get the data.
by default, API-Platform uses Eager-Fetching (see https://api-platform.com/docs/core/performance/#force-eager), but it seems so that will be ignored if the getters of a entity are used in the DTO.
the serialization process needs the most time. That is maybe (also) a Symfony issue.
In a try to reduce the Doctrine queries, we created a DataProvider to fetch the related data. This actually worked, as using the DataProvider reduced the number of queries to +/- 50, but the serialization process also needed around 25s. So the cause of the performance problem does not seem to be the lazy-loading of doctrine, which is now done.
The question is: Why is using a DTO so much slower how would it be it possible to get performance back to an acceptable level?
I may be wrong, but I guess you are mistaking EAGER and LAZY loading in doctrine. As far as I understand, EAGER loaded associations will be fully loaded immediately, while LAZY loaded associations will only be fully loaded when the property is accessed via the property getter (what is actually happening in your transformer).
https://www.doctrine-project.org/projects/doctrine-orm/en/current/reference/working-with-objects.html#by-eager-loading
(There is also the EXTRA_LAZY option that won't even fully load related entities if you only access them with a couple of allowed methods)
From my opinion, as long as you are not serializing the related entities as nested documents (but as IRIs only), as far as I understand you shouldn't even need to fully load those entities. So LAZY (and don't accessing the related entities via getter) would be the way to go. This way we've been able to speed up entity loading a lot for entities which only needed to be serialized as an IRI (although we aren't using DTO and transformers).
It was not possible to improve performance when using a DTO for this data-structure. Instead of a DTO and data transformer we used a Doctrine Event Listener (postLoad) to set the value which is a unmapped property (Doctrine/Symfony: Entity with non-mapped property) now.
I am trying to clone an entity record along with the relationships it holds among other entities. I have successfully cloned some entity objects but this one to many entity relationship has challenged me. I have reviewed similar questions regarding the error message I have been given without progress to the challenge.
The correct records are queried out, looped through and cloned then stored in an array. I have tried to persist the array but get error
EntityManager#persist() expects parameter 1 to be an entity object,
array given
I then tried to encode the array and persist but I get error
The class 'Symfony\Component\HttpFoundation\JsonResponse' was not
found in the chain configured namespaces NameOfBundle\Entity.
This below code is in my controller
$quoteItemAddWorkCollection = $em->getRepository('UniflyteBundle:QuoteItemAdditionalWork')->findBy($params);
$quoteItemDeliverableCollection = $em->getRepository('UniflyteBundle:QuoteItemDeliverable')->findBy($params);
if (!empty($quoteItemAddWorkCollection)) {
$quoteItemAddWorkArray = [];
foreach ($quoteItemAddWorkCollection as $quoteItemAddWorkItem) {
$quoteItemAddWorkItemClone = clone $quoteItemAddWorkItem;
array_push($quoteItemAddWorkArray, $quoteItemAddWorkItemClone);
}
$quoteItemAddWorkCollection = new JsonResponse($quoteItemAddWorkArray);
$em->persist($quoteItemAddWorkCollection);
I can't persist an array, I have to encode it to json first I believe. What am I doing wrong?
I think you have a misunderstanding of Doctrine concepts here. In terms of Doctrine, each entity:
UniflyteBundle:QuoteItemAdditionalWork
and
UniflyteBundle:QuoteItemDeliverable
, and any of its relationships, could get persisted, using a configuration named Mapping.
To get this into work, any In-Memory object, MUST be an instance of a managed entity class.
There is not such a magic in Doctrine, to persist so many unknown objects at once. You may persist them, one-by-one inside a loop:
foreach ($quoteItemAddWorkCollection as $quoteItemAddWorkItem) {
$quoteItemAddWorkItemClone = clone $quoteItemAddWorkItem;
$quoteItemAddWorkItemClone->setId(null);
// Set relationships here ...
$em->persist($quoteItemAddWorkItemClone);
}
Keep in mind to set any required relationships, before persisting your new cloned objects.
If you want to use, one persist, you can assign their relationships, inside a loop:
foreach ($quoteItemAddWorkCollection as $quoteItemAddWorkItem) {
$quoteItemAddWorkItemClone = clone $quoteItemAddWorkItem;
$quoteItemAddWorkItemClone->setId(null);
$someParentCollection->add($quoteItemAddWorkItemClone);
}
$em->persist($someParentCollection);
the latter method, needs you to set cascade on mapping configuration:
class SomeParent
{
// #ORM\OneToMany(targetEntity="QuoteItemAdditionalWork", mappedBy="parent", cascade={"persist"})
private $quoteItemAddWork;
}
Let's assume there is a OneToMany doctrine2 association between blogposts and comments. A blogposts might have many comments. Every comment remains inactive and therefore hidden in the frontend until a moderator will activate the comment manually.
I'm now trying to have some kind of security facade to ensure that only "active" comments will be provided to the view by accessing them in a loop over the {{blogpost.comments}} variable in the twig template.
Trying to use the getComments() method in the blogpost entity I was trying to filter the ArrayCollection of comments like so
/**
* #return ArrayCollection
*/
public function getComments()
{
return $this->comments->filter(function ($condition) {
return $condition->getActive() === true;
});
}
unfortunately Doctrine will entirely load every single comment even if the relations fetch mode is set to "EXTRA_LAZY". So that would influence the performance of an application in a way i'd like to avoid.
Is there any way to hide the inactive comments globally or do I have to take care of filtering them every time I'm accessing the blogpost.comments relations in the view?
You should use the matching method of your collection. If your collection is not loaded, it will add filters to the SQL query to only load what you need. If your collection is already loaded, it will filter the PHP array.
use Doctrine\Common\Collections\Criteria;
public function getComments()
{
return $this->comments->matching(
Criteria::create()->where(
Criteria::expr()->eq('active', true)
)
);
}
More informations here: http://doctrine-orm.readthedocs.org/projects/doctrine-orm/en/latest/reference/working-with-associations.html#filtering-collections
Regards
If I have an associated entity which is a collection, what options do you have when fetching?
e.g. Lets say I have a $view entity with this definition inside it:
/**
* #ORM\OneToMany(targetEntity="\Gutensite\CmsBundle\Entity\View\ViewVersion", mappedBy="entity")
* #ORM\OrderBy({"timeMod" = "DESC"})
*/
protected $versions;
public function getVersions() {
return $this->versions;
}
And I want to get the all the versions associated with the entity like this:
$view->getVersions();
This will return a collection. Great. But is it possible to take that collection and filter it by criteria, e.g. newer than a certain date? Or order it by some (other) criteria?
Or at this point are you just expected to do a query on the repository:
$versions = $em->getRepository("GutensiteCmsBundle:View\ViewVersion")->findBy(array(
array(
'viewId', $view->getId(),
'timeMod', time()-3600
)
// order by
array('timeMod', 'DESC')
));
There is a surprisingly unknown feature in recent versions of Doctrine, which makes these sort of queries much easier.
It doesn't seem to have a name, but you can read about it in the Doctrine docs at 9.8 Filtering Collections.
Collections have a filtering API that allows to slice parts of data from a collection. If the collection has not been loaded from the database yet, the filtering API can work on the SQL level to make optimized access to large collections.
In your case you could write a method like this on your View entity.
use Doctrine\Common\Collections\Criteria;
class View {
// ...
public function getVersionsNewerThan(\DateTime $time) {
$newerCriteria = Criteria::create()
->where(Criteria::expr()->gt("timeMod", $time));
return $this->getVersions()->matching($newerCriteria);
}
}
This will do one of two things:
If the collection is hydrated, it will use PHP to filter the existing collection.
If the collection is not hydrated, it will fetch a partial collection from the database using SQL constraints.
Which is really great, because hooking up repository methods to your views is usually messy and prone to break.
I also like #igor-pantovic's answer, although I've seen the method cause some funny bugs.
I would personally avoid using order by on annotation directly. Yes, you are supposed to do a query, just as you would if you were using raw SQL without Doctrine at all.
However, I wouldn't do it at that point but even before. In your specific case I would create an ViewRepository class:
class ViewRepository extends EntityRepository
{
public function findWithVersionsNewerThan($id, \DateTime $time)
{
return $this->createQueryBuilder('view')
->addSelect('version')
->join('view.versions', 'version')
->where('view.id = :id')
->andWhere('version.timeMod > :time')
->setParameter('time', $time)
->setParameter('id', $id)
->getQuery()
->getOneOrNullResult();
}
}
Now you can do:
$yourDateTime = // Calculate it here ... ;
$view = $em->getRepository("GutensiteCmsBundle:View\ViewVersion")->findWithVersionsNewerThan($yourDateTime);
$versions = $view->getVersions(); // Will only contain versions newer than datetime provided
I'm writing code from the top of my head here directly so sorry if some syntax or method naming error sneaked in.
Say I have a an entity in Doctrine called Post and it has a bidirectional many-to-one relationship to another entity called Comment.
Say I have a function in Post that serializes the post to JSON and includes a portion of the comments:
public function serialize(){
return array(
... other data here ....
'comments' => $this->getSerializedComments(5),
'total_comments' => $this->getComments()->count()
);
}
I would like to also write a function getSerializedComments(limit) that only loads up to limit comments in the association (i.e. NOT all of the comments for the post, just 5). If I understand correctly, if I make the association EXTRA_LAZY, the count() will only run a count query, and not hydrate the whole association.
I would prefer to do all of this in my entity class, and not have to do it in a separate manager or repository function.
I know there's an #OrderBy annotation for To-Many relationships. Doesn't seem like there's an #Limit though.
You can simply use Doctrine\Common\Collections\Collection::slice(), which doesn't initialize the collection if it is marked as EXTRA_LAZY