I need to fill query builder with filter conditions of page attributes. I got Enities "Page", "Attribute" and "Value"
Page: id, name, etc
Attribute: id, name, etc
Value: page_id, date, string, numeric
I need to get all pages having attribute values selected in form.
So i get Query Builder:
$qb->select('p')
->from(Page p)
->leftJoin(p.values);
Can I use "having" or "where" clauses to do like?
$qb->add('having', 'a.id = :attr1_id AND a.value = :attr1_val')
$qb->add('having', 'a.id = :attr2_id AND a.value = :attr2_val')
UPDATE: No i can't. in this way it can get page with attr1_id = attr2_value AND attr2_id = attr1_value and all conditions will be true, but result is wrong.
or I should add custom join for each value?
UPDATE: here's working exapmle:
//attr1.intval=:a1 (or LIKE, BETWEEN, etc compare)
$comp_expression=$aliace.'.'.$value_field.'=:a'.$this->getId();
//INNER JOIN attribute attr1 ON attr1.type=5 AND attr1.intval=:a1
$qb->innerJoin($value_class, $aliace , 'WITH', $aliace.'.type='.$this->getId().' AND '.$comp_expression);
//where attr1.intval IS NOT NULL
$qb->andWhere($aliace.'.'.$value_field.' IS NOT NULL');
$qb->setParameter('a'.$this->getId(),$value);
UPDATE: So the only way to do key+value filtring is to add join for each filter condition? I got 27 attributes now, so need to build query with 27 joins?
Is there another way to this better? Sorry if this is duplicate, cant find keywords to find same questions.
UPDATE: Maybe i shuld get out of sql query limits and create mysql procedure for it?
UPDATE: Can i use somthing like this in doctrine? MySQL optimization on filtering key-value pairs as records
Model
The model you are using here is called Entity-Attribute-Value (EAV). If you weren't aware of that I suggest you read about the pro's and con's first.
One of the con's is that searching in a EAV model is difficult and very inefficient.
AND queries
It seems you want to query for Entities that have a specific Value for a specific Attribute and another specific Value for a specific Attribute. These types of queries are not very efficient in the EAV model.
Let's assume your entities are set up as:
Entity: id, attributes (one-to-many), values (one-to-many)
Attribute: id, name, entity (many-to-one), values (one-to-many)
Value: id, content, entity (many-to-one), attribute (many-to-one)
Let's assume you want to query for all Entities that have an Attribute color with the value blue. A Doctrine query would look like this:
SELECT e FROM Entity e
JOIN e.values v JOIN v.attribute a
WHERE a.name = 'color' AND v.content = 'blue'
(Of course you should never use values in the DQL like this, but bind them as parameters.)
Now let's assume you want to query for all Entities that have an Attribute color with the value blue and an Attribute shape with the value square. The query becomes:
SELECT e FROM Entity e
JOIN e.values v1 JOIN v.attribute a1
JOIN e.values v2 JOIN v.attribute a2
WHERE a1.name = 'color' AND v1.content = 'blue'
AND a2.name = 'shape' AND v2.content = 'square'
Now if you want to query for 3 Attribute/Value pairs, you'll need 3 of those JOIN sets. If you want to query for 4 Attribute/Value pairs, you'll need 4 of those JOIN sets. Etc.
OR queries
Let's assume you want to query for all Entities that have an Attribute color with the value blue or an Attribute shape with the value square. The query becomes:
SELECT e FROM Entity e
JOIN e.values v JOIN v.attribute a
WHERE (a.name = 'color' AND v.content = 'blue')
OR (a.name = 'shape' AND v.content = 'square')
You can see this is more efficient, you won't need those additional JOIN sets for every additional Attribute/Value pair.
Mixing AND OR
I'm going to be short here: this will become a query nightmare. I strongly advise you not to do this.
Alternative
If your application is going to rely heavily on these types of queries (especially AND and mixing AND OR), I suggest you consider a different type of storage engine. Relational Databases are not really suited for this kind of thing.
You'll probably be better of using a Document Oriented Database, something like Elasticsearch, MongoDB, CouchDB, etc.
Hybrid alternative
You can also "copy" the EAV part to a Document Oriented Database and use it only for search functionality. I suggest you set up events for when changes are made to an Entity (create, update, delete). Then create listeners that persist those changes into the Document store.
This way your application can work with the Relational Database for normal operations, and the Document store for searching. In this case I advise Elasticsearch, which is extremely suitable for this kind of thing.
There is no such difference between the two methods, the two methods do exactly the same thing.
By the central point of question, I got the doubt is about the construction of query using like.
The first example you provide is workfull.
The second example you propose is not right.
On the second example, you need to define All the where clauses on andWhere method.
$qb->innerJoin('a1 ON p.id=a1.page_id');
$qb->andWhere('a1.attribute_id=:attr1_id ');
$qb->andWhere('a1.value=:attr1_val');
$qb->andWhere('a1 is not null');
All are you doing is creating a sql by a third api, Doctrine ORM is one of the most successful orm tool for php, knowing how to use it will let you to experience the ORM power, by doctrine implementation all you need to know are there:
Doctrine Create Query Builder -> The same are you using.
Doctrine Query Builder
Doctrine Create Query -> Simplest form
Query Language
Related
I'm currently building an eCommerce site using Symfony 3 that supports multiple languages, and have realised they way I've designed the Product entity will require joining multiple other entities on using DQL/the query builder to load up things like the translations, product reviews and discounts/special offers. but this means I am going to have a block of joins that are going to be the same in multiple repositories which seems wrong as that leads to having to hunt out all these blocks if we ever need to add or change a join to load in extra product data.
For example in my CartRepository's loadCart() function I have a DQL query like this:
SELECT c,i,p,pd,pt,ps FROM
AppBundle:Cart c
join c.items i
join i.product p
left join p.productDiscount pd
join p.productTranslation pt
left join p.productSpecial ps
where c.id = :id
I will end up with something similar in the SectionRepository when I'm showing the list of products on that page, what is the correct way to deal with this? Is there some place I can centrally define the list of entities needed to be loaded for the joined entity (Product in this case) to be complete. I realise I could just use lazy loading, but that would lead to a large amount of queries being run on pages like the section page (a section showing 40 products would need to run 121 queries with the above example instead of 1 if I use a properly joined query).
One approach (this is just off the top of my head, someone may have a better approach). You could reasonably easily have a centralised querybuilder function/service that would do that. The querybuilder is very nice for programattically building queries. The key difference would be the root entity and the filtering entity.
E.g. something like this. Note of course these would not all be in the same place (they might be across a few services, repositories etc), it's just an example of an approach to consider.
public function getCartBaseQuery($cartId, $joinAlias = 'o') {
$qb = $this->getEntityManager()->createQueryBuilder();
$qb->select($joinAlias)
->from('AppBundle:Cart', 'c')
->join('c.items', $joinAlias)
->where($qb->expr()->eq('c.id', ':cartId'))
->setParameter('cartId', $cartId);
return $qb;
}
public function addProductQueryToItem($qb, $alias) {
/** #var QueryBuilder $query */
$qb
->addSelect('p, pd, pt, ps')
->join($alias.'product', 'p')
->leftJoin('p.productDiscount', 'pd')
->join('p.productTranslation', 'pt')
->join('p.productSpecial', 'ps')
;
return $qb;
}
public function loadCart($cartId) {
$qbcart = $someServiceOrRepository->getCartBaseQuery($cartId);
$qbcart = $someServiceOrRepository->addProductQueryToItem($qbcart);
return $qbcart->getQuery()->getResult();
}
Like I said, just one possible approach, but hopefully it gives you some ideas and a start at solving the issue.
Note: If you religiously use the same join alias for the entity you attach your product data to you would not even have to specify it in the calls (but I would make it configurable myself).
There is no single correct answer to your question.
But if I have to make a suggestion, I'd say to take a look at CQRS (http://martinfowler.com/bliki/CQRS.html) which basically means you have a separated read model.
To make this as simple as possibile, let's say that you build a separate "extended_product" table where all data are already joined and de-normalized. This table may be populated at regular intervals with a background task, or by a command that gets triggered each time you update a product or related entity.
When you need to read products data, you query this table instead of the original one. Of course, nothing prevents you from having many different extended table with your data arranged in a separate way.
In some way it's a concept very similar to database "views", except that:
it is faster, because you query an actual table
since you create that table via code, you are not limited to a single SQL query to process data (think filters, aggregations, and so on)
I am aware this is not exactly an "answer", but hopefully it may give you some good ideas on how to fix your problem.
I'm using Yii2's ActiveRecord implementation in (hopefully) exactly the way it should be used, according to the docs.
Problem
In a quite simple setup with simple relations betweens the tables, fetching 10 results is fast, 100 is slow. 1000 is impossible. The database is extremely small and indexed perfectly. The problem is definitly Yii2's way to request data, not the db itself.
I'm using a standard ActiveDataProvider like:
$provider = new ActiveDataProvider([
'query' => Post::find(),
'pagination' => false // to get all records
]);
What I suspect
Debugging with the Yii2 toolbar showed thousands of single SELECTs for a simple request that should just get 50 rows from table A with some simple "JOINs" to table B to table C. In plain SQL everybody would solve this with one SQL statement and two joins. Yii2 however fires a SELECT for every relation in every row (which makes sense to keep the ORM clean). Resulting in (more or less) 1 * 50 * 30 = 1500 queries for just getting two relations of each row.
Question
Why is Yii2 using so many single SELECTs, or is this a mistake on my side ?
Addionally, does anybody know how to "fix" this ?
As this is a very important issue for me I'll provide 500 bounty on May 14th.
By default, Yii2 uses lazy loading for better performance. The effect of this is that any relation is only fetched when you access it, hence the thousands of sql queries. You need to use eager loading. You can do this with \yii\db\ActiveQuery::with() which:
Specifies the relations with which this query should be performed
Say your relation is comments, the solution is as follows:
'query' => Post::find()->with('comments'),
From the guide for Relations, with will perform an extra query to get the relations i.e:
SELECT * FROM `post`;
SELECT * FROM `comment` WHERE `postid` IN (....);
To use proper joining, use joinWith with the eagerLoading parameter set to true instead:
This method allows you to reuse existing relation definitions to perform JOIN queries. Based on the definition of the specified relation(s), the method will append one or multiple JOIN statements to the current query.
So
'query' => Post::find()->joinWith('comments', true);
will result in the following queries:
SELECT `post`.* FROM `post` LEFT JOIN `comment` comments ON post.`id` = comments.`post_id`;
SELECT * FROM `comment` WHERE `postid` IN (....);
From #laslov's comment and https://github.com/yiisoft/yii2/issues/2379
it's important to realise that using joinWith() will not use the JOIN query to eagerly load the related data. For various reasons, even with the JOIN, the WHERE postid IN (...) query will still be executed to handle the eager loading. Thus, you should only use joinWith() when you specifically need a JOIN, e.g. to filter or order on one of the related table's columns
TLDR:
joinWith = with plus an actual JOIN (and therefore the ability to filter/order/group etc by one of the related columns)
In order to use relational AR, it is recommended that primary-foreign key constraints are declared for tables that need to be joined. The constraints will help to keep the consistency and integrity of the relational data.
Support for foreign key constraints varies in different DBMS. SQLite 3.6.19 or prior does not support foreign key constraints, but you can still declare the constraints when creating tables. MySQL’s MyISAM engine does not support foreign keys at all.
In AR, there are four types of relationships:
BELONGS_TO: if the relationship between table A and B is one-to-many, then B belongs to A (e.g. Post belongs to User);
HAS_MANY: if the relationship between table A and B is one-to-many, then A has many B (e.g. User has many Post);
HAS_ONE: this is special case of HAS_MANY where A has at most one B (e.g. User has at most one Profile);
MANY_MANY: this corresponds to the many-to-many relationship in database. An associative table is needed to break a many-to-many relationship into one-to-many relationships, as most DBMS do not support many-to-many relationship directly. In our example database schema, the tbl_post_category serves for this purpose. In AR terminology, we can explain MANY_MANY as the combination of BELONGS_TO and HAS_MANY. For example, Post belongs to many Category and Category has many Post.
The following code shows how we declare the relationships for the User and Post classes.
class Post extends CActiveRecord
{
......
public function relations()
{
return array(
'author'=>array(self::BELONGS_TO, 'User', 'author_id'),
'categories'=>array(self::MANY_MANY, 'Category',
'tbl_post_category(post_id, category_id)'),
);
}
}
class User extends CActiveRecord
{
......
public function relations()
{
return array(
'posts'=>array(self::HAS_MANY, 'Post', 'author_id'),
'profile'=>array(self::HAS_ONE, 'Profile', 'owner_id'),
);
}
}
The query result will be saved to the property as instance(s) of the related AR class. This is known as the lazy loading approach, i.e., the relational query is performed only when the related objects are initially accessed. The example below shows how to use this approach:
// retrieve the post whose ID is 10
$post=Post::model()->findByPk(10);
// retrieve the post's author: a relational query will be performed here
$author=$post->author;
You are somehow doing it the wrong please go through from the documentaion here http://www.yiiframework.com/doc/guide/1.1/en/database.arr
I have two entities in Doctrine 2.1: Category and Site each category has many sites and each site has a parent category.
I would like to make a single update query (in DQL) which will update a field called count of the Category entity with the number of related sites.
So in SQL I would do something like this:
UPDATE categories c SET c.count = (SELECT COUNT(s.id) FROM sites s WHERE s.category_id = c.id);
This would work beautifuly, in DQL it might something like this:
UPDATE PackageNameBundle:Category c SET c.count = (SELECT COUNT(s.id) FROM PackageNameBundle:Site s WHERE s.category = c)
Such attempt raises [Syntax Error] line 0, col 61: Error: Expected Literal, got 'SELECT'.
Subqueries DO work in DQL, but the problem here (as far as I see it) is that Doctrine cannot assign the returned value from the subquery, to the c.count. This is understandable since I might fetch more than 1 field in the subquery and even more than one row. It magicaly works in MySQL since it sees one row, one field and for convenience returns a single integer value. Doctrine on the other hand has to be object oriented and has to work with different engines where such convertions might not be supported.
Finally, my question is:
What is the best way to do this in Doctrine, should I go with Native SQL or it can be done with DQL and how?
Thanks in advance!
EDIT: I just found this quote in the DQL Docs:
References to related entities are only possible in the WHERE clause and using sub-selects.
So, I guess assigning anything but a scalar value is impossible?
The main question remains though..
You can use native sql queries in Doctrine also, for that kind of specific queries. DQL is powerful in its own way, but it's also limited due to performance constraints. Using native sql queries and mapping the results will achieve the same thing, and there is no disadvantage in doing that.
The documentation explains it in detail.
I have a problem with a DQL query and entity specialization.
I have an Entity called Auction, which is OneToOne relation with Item. Item is a mappedSuperclass for Film and Book. I need a query that could back a search engine, allowing the user to look for auctions with different properties AND selling items with different properties (it is the AND part that makes it challenging).
The problem is that even though Auction has an association pointing to Item as such, I need to have access to Film- and Book-specific fields. The users will specify the Item type they're looking for, but I don't see any way of using this information other than using INSTANCE OF in my DQL query.
So far, I have tried using a query like:
SELECT a FROM Entities\Auction a
INNER JOIN a.item i
INNER JOIN i.bookTypes b
WHERE i INSTANCE OF Entities\Book
AND b.type = 'Fantasy'
AND ...".
Such a query results in an error saying that:
Class Entities\Item has no field or association named bookTypes
which is false for Book, yet true for Item.
I have also tried
SELECT a FROM Entities\Book i
INNER JOIN i.auction a ...
but I reckon Doctrine requires that I refer to the same Entity in SELECT and FROM statements.
If that's of importance, I am using class table inheritance. Still, I don't think switching to single table inheritance would do the trick.
Any ideas?
As Matt stated, this is an old issue that Doctrine Project won't fix (DDC-16).
The problem is that doctrine's DQL is a statically typed language that comes with a certain amount of complexity in its internals.
We thought about allowing upcasting a couple of times, but the effort to get that working is simply not worth it, and people would simply abuse the syntax doing very dangerous things.
As stated on DDC-16, it is also indeed not possible to understand which class the property belongs to without incurring in nasty problems such as multiple subclasses defining same properties with different column names.
If you want to filter data in subclasses in a CTI or JTI, you may use the technique that I've described at https://stackoverflow.com/a/14854067/347063 . That couples your DQL with all involved subclasses.
The DQL you would need in your case is most probably (assuming that Entities\Book is a subclass of Entities\Item):
SELECT
a
FROM
Entities\Auction a
INNER JOIN
a.item i
INNER JOIN
i.bookTypes b
WHERE
i.id IN (
SELECT
b.id
FROM
Entities\Book b
WHERE
b.type = 'Fantasy'
)
That is the pseudo-code for your problem. It is not nice, but keep in mind that SQL and DQL are very different and follow different rules.
Updated:
I've discovered a solution for this. See my answer for this related question:
Doctrine2: Polymorphic Queries: Searching on properties of subclasses
You can easily solve this by left-joining your base entity with your inheritance class using the id:
SELECT a FROM Entities\Auction a
INNER JOIN a.item i
INNER JOIN Entities\Book b WITH b.id = i.id
INNER JOIN b.bookTypes bt
WHERE bt.type = 'Fantasy'
AND...
or with a queryBuilder:
$queryBuilderb->select('a')
->from('Entities\Auction', 'a')
->innerJoin('a.item', 'i')
->innerJoin('Entities\Book', 'b', 'WITH', 'b.id = i.id')
->innerJoin('b.bookTypes', 'bt')
->where('bt.type = :type')
->andWhere(...
->setParameter('type', 'Fantasy');
This is based on the answer given by Ian Philips in the question here
The Doctrine team has stated that they're not going to add support for this:
https://github.com/doctrine/orm/issues/2237
Pertinent comments from that page:
Thats indeed tricky. That syntax alone can, however, never work,
because there might be several subclasses that have a field named "d",
so Doctrine would not know which field you mean.
I am closing this one.
The requirement of this issue is basically violating OO principles.
If you really need to filter across multiple child-entities in your
inheritance, then try something as following instead:
SELECT
r FROM
Root r WHERE
r.id IN (
SELECT
c.id
FROM
Child c
WHERE
c.field = :value
)
I had the same issue, and didn't find a solution without using separate queries for each subclass and merging them later on the application level.
One thing I'm sure, single table inheritance will not solve this, completely the same thing.
There is another alternative, although being logically dirty.
Define all the fields (the ones you need) in the superclass. If the record logically doesn't have that field it will be empty. Not a pretty sight, but hey, more optimized than 2-3-4-... queries. Also in this scenario single table inheritance is definitely the better way to go
Ok to make it more clear:
I am Using doctrine
I have a table Brands and Products
Brand
id
name
Product
id
name
brand_id
I have a lot of brands and Products of those brands in the database.
I would like to retrieve List of brands(+ count of its products) Grouped by Brand.name's first latter.
ex:
array(
n => array(
0 => array('Nike', 4 ),
1 => array('North Pole', 18)
.....
)
.....
)
So my question was can this be done with one query in a efficient way.
I really don't wan't to run separate queries for each brand.name's first latter.
Doctrines "Hierarchical Data" cross my mind but I believe it for different thing?.
thanks
If you are going to use this form of result more than once, it might be worthwhile to make the formatting into a Hydrator, as described here.
In your case, you can create a query that select 3 columns
first letter of brand.name
brand.name
count(product.id)
Then hydrate the result
$results = $q->execute(array(), 'group_by_first_column');
You cannot take it from database in that way, but you can fetch data as objects or arrays and then transform it to described form. Use foreach loops.
When using Doctrine you can also use raw SQL querys and hydrate arrays instead of objects. So my Solution would be to use a native SQL Query:
SELECT
brand.name,
count(product.id)
FROM
brand
JOIN
product ON
brand.id=product.brand_id
GROUP BY
brand.id ORDER BY brand.name;
And then iterate in PHP over the result to build the desired array. Because the Result is ordered by Brand Name this is quite easy. If you wasn't to keep database abstraction I think it should also be possible to express this query in DQL, just hydrate an array instead of objects.