I have 2 tables:
User:
id
schedule_id
name
Schedules:
id
date
I execute a query using eager loading:
User::with (['schedules'])->get()->first();
And got result like this:
[
'id' => '3',
'schedule' => [
'id' => '13',
'date' => '20.11.2020',
],
'name' => 'John',
];
But when I execute a similar query using join,
User::join ('schedules', 'user.schedule_id', '=', 'schedules.id')->get()->first();
I got result like this, with merged arrays:
[
'id' => '13',
'date' => '20.11.2020',
'name' => 'John',
];
How can I got a result with separated arrays, using join in Eloquent?
Note: in raw PHP and PDO I always got separated arrays for any queries with join.
Data coming from a database query will always be formatted as a flat array, that's just how databases work. Eloquent has a lot of magic going on behind the scene's that is going to map the correct related values to the correct models in the collection.
When using joins, there is no way for Eloquent to know what data should be mapped to which relation or property.
If you want to use join queries, you either will have to use aliases for these properties on the models. Or you can manually map the properties the way you want them before using the data.
Ok, I can suggest a more suitable way to get results ordered by relation's column in Laravel's Eloquent.
You should use 2 queries.
Firstly just get only ids, but ordered by any relation's column.
(You can't order results by relation's column using eager loading)
$idsRaw = User::join ('schedules', 'user.schedule_id', '=', 'schedules.id')
->select ('users.id')
->orderBy ('schedules.date')
->get ()->toArray ();
$ids = \array_column ($idsRaw, 'id');
Secondly get good results with subarrays for relations using eager loading:
User::with (['schedules'])
->whereIn ('user.id', \implode (',', $ids))
->orderByRaw('FIELD(' . $ids . ',' . \implode(',', $ids) . ')')
->get()->toArray ();
Advantages:
You don't need manually map columns and aliases into subarrays (It may be very difficult if you have many relations).
Disadvantages:
This method works as slow as much ids you have.
Yes, if you have a thousands ids in the query, you should use one complex query with joins for all relations and manually map aliases into subarrays for best performance.
Related
I use MongoDB PHP v1.3 and in my MongoDB I have multiple collections:
// COLLECTION NAMES:
- user_1_list_1
- user_1_list_2
- user_1_list_3
...
- user_1_list_55
All these collections have the same document-structure:
{
first_name
last_name
phone
}
How can I query the documents from all of these collections at the same time? In the documentation, it is explained how to query (find many) documents from one collection: https://docs.mongodb.com/php-library/v1.3/tutorial/crud/#find-many-documents.
For example, in my case, it would look something like this:
$collection_name = "user_1_list_1";
$collection = $this->db->{$collection_name};
$query = [];
$cursor = $collection->find(
$query,
[
'limit' => 10,
'skip' => 0,
'sort' => ['first_name' => 1],
]
);
... but this will find documents only from one collection (in this case, only from the collection with name "user_1_list_1").
How to find documents from all of these collections (user_1_list_1, user_1_list_2, user_1_list_3 ... ) (that have the same structure), not just from one specific?
Is this possible at all? If yes, how would you do that?
MongoDB is not a relation database and there is no good solution for your case.
You can get your collections and loop over it (but it’s a not good
solution).
You can change your database structure and use one collection with embedded
data
This is related to a previous question here: Doctrine/Symfony query builder add select on left join
I want to perform a complex join query using Doctrine ORM. I want to select 10 paginated blog posts, left joining a single author, like value for current user, and hashtags on the post. My query builder looks like this:
$query = $em->createQueryBuilder()
->select('p')
->from('Post', 'p')
->leftJoin('p.author', 'a')
->leftJoin('p.hashtags', 'h')
->leftJoin('p.likes', 'l', 'WITH', 'l.post_id = p.id AND l.user_id = 10')
->where("p.foo = bar")
->addSelect('a AS post_author')
->addSelect('l AS post_liked')
->addSelect('h AS post_hashtags')
->orderBy('p.time', 'DESC')
->setFirstResult(0)
->setMaxResults(10);
// FAILS - because left joined hashtag collection breaks LIMITS
$result = $query->getQuery()->getResult();
// WORKS - but is extremely slow (count($result) shows over 80,000 rows)
$result = new \Doctrine\ORM\Tools\Pagination\Paginator($query, true);
Strangely, count($result) on the paginator shows the total number of rows in my table (over 80,000) but traversing the $result with foreach outputs 10 Post entities, as expected. Do I need to do some additional configuration to properly limit my paginator?
If this is a limitation of the paginator class what other options do I have? Writing custom paginator code or other paginator libraries?
(bonus): How can I hydrate an array, like $query->getQuery()->getArrayResult();?
EDIT: I left out a stray orderBy in my function. It looks like including both groupBy and orderBy causes the slowdown (using groupBy rather than the paginator). If I omit one or the other, the query is fast. I tried adding an index on the "time" column in my table, but didn't see any improvement.
Things I Tried
// works, but makes the query about 50x slower
$query->groupBy('p.id');
$result = $query->getQuery()->getArrayResult();
// adding an index on the time column (no improvement)
indexes:
time_idx:
columns: [ time ]
// the above two solutions don't work because MySQL ORDER BY
// ignores indexes if GROUP BY is used on a different column
// e.g. "ORDER BY p.time GROUP BY p.id is" slow
You should simplify your query. That would shave off some execution time. I can't test your query but here are a few pointers:
don't do sort while executing count()
you could sort by orderBy('p.id', 'DESC'), index would be used
instead of leftJoin() you could use join() if at least one record always exists at joined table. Else that record is skipped.
KNP/Paginator uses DISTINCT() to read only distinct records, but that could lead to using disk tmp table
$query->getArrayResult() uses array hidration mode, which returns multidimension array and it is way faster than object hidration for large result set
you could use partial select('partial p.{id, other used fields}'), this way you would load only needed fields, maybe skip unneded relations when using object hydration
check SF profiler EXPLAIN on a given query under doctrine section, maybe indexes are not used
does p.hashtags and p.likes return only one row or is oneToMany, which multiplies result
maybe some Posts design changes, that would remove some joins:
have p.hashtags field defined as #ORM\Column(type="array") and have stored string values of tags. Later maybe using full text search on serialized array.
have p.likesCount field defined as #ORM\Column(type="integer") which would have count of likes
I use KnpLabs/KnpPaginatorBundle and can also have speed issues for complex queries.
Usually using LIMIT x,z is slow for DB, because it runs COUNT on whole dataset. If indexes are not used it is painfully slow.
You could use different approach and do some custom pagination by ID advancing, but that would complicate your approach. I have used this with large datasets like SYSLOG tables. But you loose sorting and total record count functionality.
At the end of the day, many of the queries used in my application are too complex to make proper use of the Paginator, and I wasn't able to use array hydration mode with the Paginator.
According to MySQL documentation, ORDER BY cannot be resolved by indexes if GROUP BY is used on a different column. Thus, I ended up using a couple post-processing queries to populate my base results (ORDERed and LIMITed) with one-to-many relations (like hashtags).
For joins that load a single row from the joined table, I was able to join the desired values in the base ordered query. For example, when loading the "like status" for a current user, only one like from the set of likes needs to be loaded to indicate whether or not the current post has been liked. Similarly, the presence of only one author for a given post produces a single joined author row. e.g.
$query = $em->createQueryBuilder()
->select('p')
->from('Post', 'p')
->leftJoin('p.author', 'a')
->leftJoin('p.likes', 'l', 'WITH', 'l.post_id = p.id AND l.user_id = 10')
->where("p.foo = bar")
->addSelect('a AS post_author')
->addSelect('l AS post_liked')
->orderBy('p.time', 'DESC')
->setFirstResult(0)
->setMaxResults(10);
// SUCCEEDS - because joins only join a single author and single like
// no collections are joined, so LIMIT applies only the the posts, as intended
$result = $query->getQuery()->getArrayResult();
This produces a result in the form:
[
[0] => [
['id'] => 1
['text'] => 'foo',
['author'] => [
['id'] => 10,
['username'] => 'username',
],
['likes'] => [
[0] => [
['post_id'] => 1,
['user_id'] => 10,
]
],
],
[1] => [...],
...
[9] => [...]
]
Then in a second query I load the hashtags for posts loaded in the previous query. e.g.
// we don't care about orders or limits here, we just want all the hashtags
$query = $em->createQueryBuilder()
->select('p, h')
->from('Post', 'p')
->leftJoin('p.hashtags', 'h')
->where("p.id IN :post_ids")
->setParameter('post_ids', $pids);
Which produces the following:
[
[0] => [
['id'] => 1
['text'] => 'foo',
['hashtags'] => [
[0] => [
['id'] => 1,
['name'] => '#foo',
],
[2] => [
['id'] => 2,
['name'] => '#bar',
],
...
],
],
...
]
Then I just traverse the results containing hashtags and append them to the original (ordered and limited) results. This approach ends up being much faster (even though it uses more queries), as it avoids GROUP BY and COUNT, fully leverages MySQL indexes, and allows for more complex queries, such as the one I posted here.
You can configure the paginator to use a simpler 'count' sql strategy by doing one or more of the optimizations below.
$paginator = new Paginator($query, false);
$paginator->setUseOutputWalkers(false);
If results are unexpected you may want to do a DISTINCT select (select('DISTINCT p'))
For us it made massive improvements and we had no need to write or use a custom paginator.
More details can be found on this site. Note that I am owner of that website.
I am trying to do a simple query in doctrine but struggling.
$query->select(array(
'app_title' => 'u.title',
'user_name' => 'u.user_name',
'first_used' => 'MIN(u.creation_time)',
'last_used' => 'MAX(u.stop_time)',
'total_usage' => 'SUM(u.stream_seconds)',
))
->from(self::USAGE_TABLE, 'u')
->orderBy('total_usage', 'DESC');
Obviously I get an error about the column name not being known because Doctrine is using it's own aliases (sclr4).
However, if I try and order by the actual value; SUM(u.stream_seconds), then I get an unexpected bracket in the order by clause, I'm pretty sure SQL doesnt support this.
So, I am simply trying to put data in a table and handle the sorting of the columns. This seems so simple, how do I do it? Any ideas?
You can orderBy the SUM result field by list it in query projection by aliasing result using AS.
If you want to use an aggregate function such as MIN(), MAX(), AVG(), you have to use GROUP BY.
Try simmilar to this, which works perfectly for me (BTW instead of associative array in select method):
$q = $this->em()->createQueryBuilder();
$q->select(['product.id', 'product.title'])
->addSelect('SUM(product.price) AS HIDDEN stat_sum_realised')
->from('ModuleAdmin\Entity\ProductEntity', 'product')
->groupBy('product.id');
$q->orderBy('stat_sum_realised', 'DESC');
Aggregate functions are detailed here (for e.x. for MySQL):
http://dev.mysql.com/doc/refman/5.0/en/group-by-functions.html
As of Doctrine ORM 2.3, you can also use the HIDDEN keyword, which will avoid (in this case) stat_sum_realised from getting hydrated into your resultset.
I have array like this
$conditions = array("Post.title" => "This is a post");
And using $conditions array in this method.
$this->Post->find('first', array('conditions' => $conditions));
I want convert the $conditions array to normal sql query.
I want use
$this->Post->query($converted_query);
instead of
$this->Post->find('first', array('conditions' => $conditions));
$null=null;
echo $this->getDataSource()->generateAssociationQuery($this, NULL, NULL, NULL, NULL, $query_array, false,$null);
To do what you want you could do two things:
1) Combine your $conditions arrays and let CakePHP build your new query so you can simply use $this->Model->find() again.
2) Use this. It's an expansion for the mysql datasource that adds the option to do $this->Model->find('sql', array('conditions' => $conditions)) which will return the SQL-query. This option might cause trouble, because for some find calls (especially when you're fetching associated models) CakePHP uses multiple queryies to fetch the associated models (especially in case of hasMany-associations).
If at all possible, option 1 will probably cause the least trouble. Another problem with going with 2 is that if you're trying to combine two queries with conflicting conditions (like 'name = Hansel' in query 1 and 'name = Gretel' in query 2) you will just find nothing unless you plan on writing extra code to parse the resulting queries and look for conflicts..
Going with 1 will probably be a lot simpler and will probably avoid lots of problems.
I am having the query like this
$criteria = new CDbCriteria(array(
'distinct' => true,
'select' => array('assets_id'),
'condition' => 'assets_id in (159)',
'with' => array('tbl_asset_mappings'=>array('select'=>array('catid')), 'tbl_assets_details'=>array('select'=>array('filetype','original_filename'))),
'together' => true
));
$result=TblAssets::model()->findAll($criteria);
But I am getting all the column values from firsttable only.I didnt get the column values from second tables.why?
My aim is getting assets_id from tblasset,tbl_asset_mappings.catid,tbl_assets_details.filetype,tbl_assets_details.original_filename
How can I achieve that.
You are querying for objects, so you will get the relations as child objects as relations like $post->author->name.
You need instead to do a join not a with. In this situation is more easier to write as Join raw-query.
Maybe would be more easier if you just write your own query rather than constructing throughout Yii
You can access related object like $model->relatedModel->attribute.
Set a break point after model->findAll() and look to $model->_related property. You must have a collection of related models there.