I have already read some articles on Doctrine performance, but this one query is so slow it seems just wrong:
public function getBetRoundMainDataBuilder(BetRound $betRound){
$qb = $this->createQueryBuilder('br')
->select('br, uG, u, b, gG, g, t1, t2')
->where('br.id = :bID')
->setParameter('bID', $betRound->getId())
->innerJoin('br.userGroup', 'uG')
->innerJoin('uG.users', 'u')
->innerJoin('br.gameGroup', 'gG')
->leftJoin('gG.games', 'g')
->leftJoin('g.team1', 't1')
->leftJoin('g.team2', 't2')
->leftJoin('br.bets', 'b' );
return $qb;
}
I know it has a lot of Joins, but I thought I rather query everything within one query instead of lazy load all dependent Data.
I have profiled the code and although this query does not select too much data it takes endlessly for an array_shift:
Am I missing something? I even changed the Hydration Mode to array, but still have problems.
UPDATE:
I have now tried to select only partially but didn't change too much:
->select('partial br.{id},
partial uG.{id},
partial u.{id, firstName, lastName, nickName, username, imageName},
partial b.{id, data},
partial gG.{id, title},
partial g.{id, data, date},
partial t1.{id, name, shortName, shortCode, logoName},
partial t2.{id, name, shortName, shortCode, logoName}')
Next step is to split the query up.
Update 2
It is getting better, I have several areas in my View where I need different Datasets. I tried to split my content up into those Areas and also Query accordingly:
Data for Main Area:
BetRound
GamGroup (only to get the Games)
Games
Teams
Bets (only for Current User!)
This looks now like this:
$qb = $this->createQueryBuilder('br')
->select(
'partial br.{id},
partial b.{id, data},
partial gG.{id, title},
partial g.{id, data, date},
partial t1.{id, name, shortName, shortCode, logoName},
partial t2.{id, name, shortName, shortCode, logoName}'
)
->where('br.id = :bID')
->setParameter('bID', $betRound->getId())
->innerJoin('br.gameGroup', 'gG')
->leftJoin('gG.games', 'g')
->leftJoin('g.team1', 't1')
->leftJoin('g.team2', 't2')
->leftJoin('g.bets', 'b', 'WITH', 'b.user = :user')
->setParameter('user', $user->getId());
return $qb;
Second Area is the PointTable
$qb = $this->createQueryBuilder('br')
->select('br, b, u, uBs')
->where('br.id = :bID')
->setParameter('bID', $betRound->getId())
->leftJoin('br.bets', 'b')
->innerJoin('b.user', 'u')
->innerJoin('u.betroundStatus', 'uBs', 'WITH', 'uBs.betRound = :bID');
return $qb;
And then I have a third area which has all the stats. It is basically the same as the first Query but includes Bets for all Users. I am now unsure, If I should use only one query which queries all the Users Bets or, than for the stats create one query for each User or somehow different.
UPDATE QUESTION:
Maybe you see that I have a root entity called BetRound that kind of starts all queries (I am now up to 4 compley quersies from my BetRound?
I just don't know how I can "add" the data to my relations after initial load.
As you may see there are several join "paths" going from my BetRound. One of them is the "games path" that looks like:
BetRound -> GameGroup -> Games -> Team1 & Team2
BetRound -> UserGroup -> Users -> UserBetRoundStatus
BetRound -> Bets
I need all this Data, but how do I get all the data in one root Entity with the correct relations? And If I query e.g. do I start with the e.g. GameGroup or always with my root entity(=BetRound)?
I think there's a lot of left joins which may be really slow. You should rebuild this query and remove those left joins to improve some performance. Also please check if every keys and indexes are created properly in database.
In conclusion - it's not Doctrine's fault but the query.
The actual query (PDOStatement::execute()) only takes 3% of the total. Seeing as interacting with a database should be the slowest part of a script, 3% of the total execution time is plenty fast.
The vast majority of the time is taken up by the following 3 method calls:
ArrayHydrator::hydrateRowData() // 17%
AbstractHydrator::gatherRowData() // 39%
DateTimeType::convertToPHPValue() // 16%
If you count that up, it is 72% of the total. This is totally unacceptable by any standard.
Is "hydrating" records really necessary?
Can you eliminate the date/time conversion?
You just need to go after the biggest sinners. Shave off the fat.
Related
I have an entity “coin” linked with a oneToMany on prices. Prices is updated every minute and has millions of entries.
What I’m trying to achieve is a DQL query that will grab the last price to date.
Right now I see 2 ways to do it and I’m wondering which one is best in term of performance:
I could look for prices in the DB with ‘prices’.’lastupdated’ equal to the last update.
OR I could grab the last 100 price ids (we update 100 coins every minute and add 100 new rows in the Database) with a LIMIT 100 and ORDER BY ‘id’ DESC on my left join.
I know it’s tricky to use LIMIT on a LEFT JOIN with doctrine but I found a solution here:
How to limit results of a left-join with Doctrine (DQL)
and here: https://www.colinodell.com/blog/201703/limiting-subqueries-doctrine-2-dql
I’m wondering what will take the least amount of ressources to execute that query.
Of course I’m using getArrayResults() and am using partials and doctrine cache.
What is your opinion on that?
Thanks!
I've been in similar situations. For example, I run a social commerce network and want to get all follower_ids from a business to update them that an action has been performed. It's the same if you want liker_ids, etc.
In this scenario, you are only interested in a value from one column (price for you) but based off queries involving different fields (coin_id, lastupdated). For this, I greatly advise using doctrine to send a native SQL query. It's orders of magnitude more efficient, evading costly doctrine hydration, etc.
I wrote a sample query in an entity repository for you.
<?php
namespace App\EntityRepository;
use Doctrine\ORM\EntityRepository;
use PDO;
class CoinPricesRepository extends EntityRepository
{
public function queryLatestPricesForCoinId(int $coin_id, int $limit)
{
$sql = 'SELECT price FROM coin_prices WHERE coin_id = :coin_id ORDER BY lastupdated DESC LIMIT = :limit;';
$params['coin_id'] = $coin_id;
$params['limit'] = $limit;
$stmt = $this->getEntityManager()->getConnection()->prepare($sql);
$stmt->execute($params);
return $stmt->fetchAll(PDO::FETCH_COLUMN);
}
}
I have been working on optimizing a my Doctrine request and got some awesome perf improvement I’ll be sharing here if anyone is looking at a similar solution.
First, limit your left join with a where clause as much as possible
Second, use partial objets
Third, use Array results. This actually changes everything.
/**
* #return Coins[] Returns an array of Crypto objects
*/
public function findOneByTickerRelationnal($ticker)
{
$em = $this->getEntityManager();
$updatesrepository = $em->getRepository(Updates::class);
$updates = $updatesrepository->findOneBy(['id'=> 1 ]);
// This is where I’ve been doing additional work to limit my left join as much as possible with a ‘with’ on left join
$recentMarkets = $updates->getMarket();
$recentPrices = $updates->getPrice();
$recentSources = $updates->getSources();
$cryptos = $this->createQueryBuilder('c')
->select('partial c.{id, name, ticker}’) //<= use of partial is a plus but you need to know exactly which fields you want
->leftJoin('c.prices', 'p','WITH', 'p.last_updated >= :recentPrices')
->addSelect('partial p.{id, price_usd, daily_volume_usd, change_1h, change_1d, change_7d, rank}')
->leftJoin('c.markets', 'm','WITH', 'm.last_updated >= :recentMarkets')
->addSelect('partial m.{id, cur_supply, market_cap, max_supply}')
->leftJoin('c.sources', 's','WITH', 's.last_updated >= :recentSources')
->addSelect('s')
->where('c.ticker = :ticker')
->setParameter('recentPrices', $recentPrices)
->setParameter('recentMarkets', $recentMarkets)
->setParameter('recentSources', $recentSources)
->setParameter('ticker', $ticker)
->getQuery()
->getArrayResult(); //<=Changes everything
$results = $cryptos[0];
return $results;
}
I have the following query:
$query = $qb->select('p')
->from(get_class($page), 'p')
->innerJoin('p1.translations', 't')
->groupBy('p.id')
->addGroupBy('t.id')
->getQuery();
Doctrine returns the above like:
Page entity -> [translation 1, translation 2, translation 3]
But I want the result like:
Page entity 1 -> translation 1
Page entity 1 -> translation 2
Page entity 1 -> translation 3
Does anyone know how I can do this? I want to return a list of entities.
First of all, both groupBy are completely superfluous, assuming both id fields you are grouping by are the primary keys of their respective tables. For any given (p.id, t.id) combination there will only be at most one result, whether you are grouping or not.
Second, even though you are joining the the translations table, you are not fetching any data from it (t is absent from your ->select()). It just appears that way since doctrine "magically" loads the data in the background when you call $page->getTranslations() (assuming your getter is called that way).
Third, your issue isn't with the groupBy. You are completely misunderstanding what an ORM actually does when hydrating a query result. The SQL query that doctrine generates from your code will actually return results in a fashion just like you expect, with "Page entity 1" repeated multiple times.
However, now comes the hydration step. Doctrine reads the first result row, builds a "Page entity 1" and a "Translation 1" entity, links them and adds them to it's internal entity registry. Then, for the second result, Doctrine notices that it has already hydrated the "Page entity 1" object and (this is the crucial part!) reuses the same object from the first row, adding the second translation to the ->translation collection of the existing object. Even if you read the "Page entity 1" again in a completely different query later in your code, you still get the same PHP object again.
This behaviour is at the core of what Doctrine does. Basically, any table row in the database will always be mirrored by a single object on the PHP side, no matter how often or in what way you actually read it from the database.
To summarize, your query should look like this:
$query = $qb->select('p, t')
->from(get_class($page), 'p')
->innerJoin('p.translations', 't')
->getQuery();
If you really need to iterate over all (p, t) combinations, do so with nested loops:
foreach ($query->getResult() as $p) {
foreach ($p->getTranslations() as $t) {
// do something
}
}
EDIT: If your relationship is bidirectional, you could also turn your query around:
$query = $qb->select('t, p')
->from('My\Bundle\Entity\Translation', 't')
->innerJoin('t.page', 'p')
->getQuery();
Now in your example you actually get 3 results that you can iterate over in a single foreach(). You would still only get a single "Page entity 1" object, with all the translations in the query result pointing to it.
I am building a NOTE system.
I am having multiple tables (notes, tags, ...) and I am joining tags to my main notes with $this -> db -> join(); to get everything into one object.
When there are 2 or more tags for one note, then I get two or more rows with only different TAGS. The rest is the same. I need to merge it to have only one note entry.
$this -> db -> where ('user', USER_ID);
$this -> db -> join ('tags', 'tags.note_id = note.id', 'inner');
$query = $this->db->get('notes');
There may be also other tables with same character as TAGS, for example places. There may be more than one place for a note.
How do I proceed from now? I would like to have one object NOTE with parameters such as note_id, note_text, and join TAGS to it and probably if more than ONE tag, then OBJECT PARAMETER = ARRAY containing all the NOTES.
How to achieve that?
Is that actually good idea for further development to have it in one object? Or should I go foreach and list all the tags for each of the rows?
When somebody is filtering according to the tags, where & how should I store one's filtering? I am so far using $this -> session;
Thank you, Jakub
You might want to use MySQL's GROUP_CONCAT()
$this->db->select('n.*, GROUP_CONCAT(t.name)', false)
->from('notes n')->join('tags t', 't.note_id = n.id', 'inner')
->where('n.user', USER_ID)->get();
I used t.name but whatever the field name it is, you get the point.
I have reached dead end with the brain o.O. In DB I have two tables:
store_module->caffe_id, module_id, position, order
module->id_module, name, description, image
I have query where I take all modules for set ID (store_module table), and I need to get all modules which appear in this query (module_id). What I need to do?
This is the code (I am awake for 30+ hours and my brain is refusing to communicate with me, deadline is almost here, and this on of the last things I need to do. So, please help :D):
function mar_get_modules($id){
$q = $this->db->get_where('store_module', array('caffe_id' => $id));
$modules = $q->result_array();
}
Start simple, by using a regular query (if I guess right, you need a JOIN there).
This query should work:
$sql = "SELECT m.*,sm.* FROM module m
LEFT JOIN store_module sm ON sm.id_module = m.module_id
WHERE sm.caffe_id = ?";
return $this->db->query($sql, array($id))->result_array();
Now, you can transform it into an AR query:
$query = $this->db->select('module.*,store_module.*')
->from('module')
->join('store_module', 'store_module.id_module = module.module_id','left')
->where('store_module.caffe_id',$id)
->get();
return $query->result_array();
While AR is quicker sometimes, I usually prefer writing my queries "by hand", taking advantage of the binding to prevent SQL injections; it's a lot easier to see how things are working if you have a query fully laid under your eyes
Sasha,
In the function above, you are not returning anything. You'll need to update the 3rd line something to the effect of return $q->result_array();
I am building a site where candidates can post their resume and employers can post their jobs.
The employers can post a job with multiple qualifications and I saved it in database like this PHP,Javascript,ASP. Now I want admin to be able to select the candidates who are eligible for a post.
I have written the query:
$sql = "
SELECT
cand_f_name,
cand_l_name,
cand_qualification,
cand_phone,
cand_email,
cand_experience_yr,
cand_experience_mn,
cand_message,
cand_id
FROM
tbl_cand_data
WHERE
cand_qualification LIKE '%$emp_q%'
But it is not showing the expected result. I think my condition cand_qualification LIKE '$emp_q' is wrong.
My tbl_cand_data :
If you are doing a LIKE query you should include wildcards, as they will match a string containing, otherwise just do an exact match using =:
// String match
WHERE
cand_qualification LIKE '%emp_q%';
// Exact match
WHERE
cand_qualification = '$emp_q';
// You could try a WHERE IN clause as well
WHERE cand_qualification IN ('$emp_q');
// Values have to be quoted individually
WHERE cand_qualification IN ('BA','BSc','BCom');
// If you have an array you can do this:
$myArray = array('BA', 'BSc', 'BCom');
$emp_q = "'" . implode("','", $myArray) . "'"; //Output 'BA', 'BSc', 'BCom'
I saved it in database like this PHP,Javascript,ASP
That's what you did utterly wrong.
you have to create a related table (that's why our ratabase called relational one!) storing qualifications, and interconnection table, consists of qualifications id linked with candidates ids.
And query them using basic joins.
Note that despite of your current decision, even if you decide to continue with your lame schema, you WILL have to remake it proper way, sooner or later (but sooner will make you less work).
That is the very basics of database architecture and noone can go against it.
SELECT fields FROM tbl_cand_data d, qualification q, qual_cand qc
WHERE q.name = 'ASP' AND q.id=qc.qid AND d.id=qc.did
Like requires percent %
try it