I have been racking my brains for ages trying to figure this out but just can't seem to get my head round it. There is plenty of information out there but I have been unable to get it right.
I understand that using the following method is not good:
WHERE id IN (SELECT.....
I am trying to optimize the query below as I have many of this type in one script and it is making the page dreadfully slow:
$res = mysqli_query($DB, 'SELECT * FROM nde WHERE id IN (SELECT relto FROM ft_n_rel WHERE rel_fr = '.$DB->real_escape_string($rid).' AND rel_ty = '.$DB->real_escape_string($FA).') LIMIT 1');
Trying to do that query optimizes with INNER JOIN is completely baffling me.
Can anyone help?
Regards
Sub-selects and in array clauses are always very slow.
Try:
SELECT n.*
FROM nde n
INNER JOIN ft_n_rel f ON n.id = f.relto
WHERE f.rel_fr = '.$DB->real_escape_string($rid).'
AND f.rel_ty = '.$DB->real_escape_string($FA).'
LIMIT 1
#daremachine has a good point as well, run your select with EXPLAIN or EXPLAIN EXTENDED to check if indexes exist and are properly used.
Update
To check the page with explain prefix you query as follows:
EXPLAIN
SELECT n.*
FROM nde n
INNER JOIN ft_n_rel f ON n.id = f.relto
WHERE f.rel_fr = '.$DB->real_escape_string($rid).'
AND f.rel_ty = '.$DB->real_escape_string($FA).'
LIMIT 1
Related
I'm trying to convert this I think simple mysql query into Doctrine dql, however, Im experience quite a struggle right now...
SELECT (c.prix-aggregates.AVG) AS test
FROM immobilier_ad_blank c
CROSS JOIN (
SELECT AVG(prix) AS AVG
FROM immobilier_ad_blank)
AS aggregates
Purpose of this: creating z-score.
Original implementation coming from this question Calculating Z-Score for each row in MySQL? (simple)
I thought about creating an association within the entity, but I mean its not necessary, its only for stats.
Edit: Btw, I dont wanna use raw SQL, I will extract the "subquery" from another query builder expression using getDQL. Otherwise, I will have to rewrite my dynamic query builder to take in account for rawSQL.
Edit 2:
Tried this
$subQb = $this->_em->createQueryBuilder();
$subQb->addSelect("AVG(subC.prix) as AMEAN")
->from("MomoaIntegrationBundle:sources\Common", "subC");
$subDql = $subQb->getDQL();
$dql = "SELECT c.prix FROM MomoaIntegrationBundle:sources\Common c INNER JOIN ($subDql) AS aggregates";
Raw dql is:
SELECT c.prix FROM MomoaIntegrationBundle:sources\Common c INNER JOIN (SELECT AVG(subC.prix) as AMEAN FROM MomoaIntegrationBundle:sources\Common subC) AS aggregates
Getting this strange error:line 0, col 70 near '(SELECT AVG(subC.prix)': Error: Class '(' is not defined.
Edit 3:
I found kinda of a hawkish way to make it work but doctrine is stubborn with its implementation of entities and such and forgot that STATISTICS do NOT need ENTITIES !
$subQb = $this->_em->createQueryBuilder();
$subQb->addSelect("AVG(subC.prix) as AMEAN")
->from("MomoaIntegrationBundle:sources\Common", "subC");
$sql = "SELECT (c.prix-aggregates.sclr_0) AS test FROM immobilier_ad_blank c CROSS JOIN "
. "({$subQb->getQuery()->getSQL()}) AS aggregates";
$stm = $stm = $this->_em->getConnection()->prepare($sql);
$stm->execute();
$data = $stm->fetchAll();
If you have a better solution, Im all ears ! I actually dislike this solution.
Starting with Doctrine 2.4 it is possible to JOIN without using a defined association, for example:
SELECT u FROM User u JOIN Items i WITH u.age = i.price
This one doesn't make any sense but you get the point. The WITH keyword is absolutely required in this case, otherwise it is a syntax error, but you can just provide a dummy condition, like so:
SELECT u FROM User u JOIN Items i WITH 0 = 0
This essentially results in a cross join. Whether this is a good idea in a given situation is a different question, but I have encountered situations where this was indeed very useful.
For complex queries you might want to consider bypassing DQL and using a native query - especially since you don't need the result in an entity.
$connection = $em->getConnection();
$statement = $connection->prepare("
select c.prix-aggregates, t1.avg
from immobilier_ad_blank
cross join (
select avg(prix) as avg
from immobilier_ad_blank
) t1
");
$statement->execute();
$results = $statement->fetchAll();
I have this database format below (taken from phpmyadmin, the tables are relational already):
I'm trying to get all "videos.Video_Name, videos.Video_URL" with a certain "tags.Tag_Name" via the "tagmap" relational mapping. I've never really used MySQL before for anything more than SELECT's and DELETE's and the syntax of JOIN is proving too much to bear, and at this point it'd just be faster to ask for help than to keep bashing my head against it.
I know I should be using JOIN but I have no idea of the syntax to accomplish what I want.
The completely invalid query I tried was:
SELECT videos.Video_URL, videos.Video_Name
FROM tagmap
INNER JOIN videos ON videos.Video_ID = tagmap.Video_ID
INNER JOIN tagmap ON tagmap.Tag_ID = tags.Tag_ID
WHERE tags.Tag_Name = '$_GET[tag]'
But it returned no rows.
If your query returned no raws and did not a return an error then it's not "completely invalid".
Indeed, looking at the code, it should do exactly what you say you are trying to achieve. Hence if it's returning no rows then the reason must be that there is no matching data.
Break it down to find out where the data is missing:
SELECT COUNT(*)
FROM tags
WHERE tags.Tag_Name = '$_GET[tag]';
If you get a non-zero value then try....
SELECT COUNT(DISTINCT tagmap.Video_ID), COUNT(*)
FROM tags INNER JOIN tagmp
ON tags.tag_ID=tagmap.tag_ID
WHERE tags.Tag_Name = '$_GET[tag]';
(BTW you might want to read up on SQL Injection).
Now Try this one.
SELECT videos.Video_Name, videos.Video_URL FROM videos,tags,tagmap
WHERE videos.Video_ID = tagmap.Video_ID AND tags.Tag_ID = tagmap.Tag_ID AND
tags.Tag_Name='$_GET[tag]'
Same Result with Joins
SELECT videos.Video_Name, videos.Video_URL FROM tagmap
RIGHT JOIN videos ON videos.Video_ID = tagmap.Video_ID
LEFT JOIN tags ON tags.Tag_ID = tagmap.Tag_ID
WHERE tags.Tag_Name = '$_GET[tag]'
Hope it will not give any error.
Thanks.
I've been editing a script for a company that had this application specifically developed for their business, now they have came to me and wanted some upgrades. The entire application is in PHP and MySQL, minus a couple Python scripts to import 200k records daily to the database. My problem is that I need to allow the ability to categorize and edit notes on each record depending on it's event type. The only way to do this is by the URL embedded within each record, for it's the only truly unique value. Successfully figured this out, but now the page script takes forever (24 secs) to load.
Could someone please assist me in optimizing this bit of code?
$notesq = mysql_query("SELECT * FROM `campaign_event_detail_v2` WHERE `call_recording_url`<>'' AND `event_type_name`='Call'") or die(mysql_error());
while($cnD = mysql_fetch_array($notesq)) {
$callid=$cnD[0];
$getD = mysql_query("SELECT campaign_notes.note, campaign_categories.category FROM campaign_notes LEFT JOIN campaign_categories ON campaign_notes.cid = campaign_categories.cid WHERE campaign_notes.cid='".$cnD['call_recording_url']."' OR campaign_categories.cid='".$cnD['call_recording_url']."'");
$getData = mysql_fetch_row($getD);
mysql_query("UPDATE `campaign_event_detail_v2` SET `note`='".$getData[0]."',`category_id`='".$getData[1]."' WHERE `id`='".$callid."'");
}
Your help is greatly appreciated!
Thanks,
J
I think you can probably manage this in a single query:
UPDATE campaign_event_detail_v2 d
LEFT JOIN campaign_notes n ON n.cid = d.call_recording_url
LEFT JOIN campaign_categories c ON c.cid = n.cid
SET d.note = n.note, d.category_id = c.category
WHERE d.call_recording_url != '' AND d.event_type_name = 'Call'
I'm not 100% sure if this is the proper logic, from what I understood it is. I must apologize if it's not. But, my point being: you can probably make it all in a single query.
You probably should add indices on columns like event_type_name, category_id and cid, if they're not already there. It will not affect your scripts, but will take some time to execute depending on how many records you have in your table(s).
Also, it probably would be better to use triggers instead of executing this at each request.
It seems like you could improve this query:
SELECT campaign_notes.note, campaign_categories.category
FROM campaign_notes
LEFT JOIN campaign_categories
ON campaign_notes.cid = campaign_categories.cid
WHERE campaign_notes.cid = {$cnD['call_recording_url']}
OR campaign_categories.cid = {$cnD['call_recording_url']}
To this:
SELECT campaign_notes.note, campaign_categories.category
FROM campaign_notes
LEFT JOIN campaign_categories
ON campaign_categories.cid = campaign_notes.cid
WHERE campaign_notes.cid = {$cnD['call_recording_url']}
Something like this:
SELECT campaign_notes.note,
campaign_categories.category
FROM campaign_event_detail_v2 AS detail
LEFT JOIN campaign_notes ON campaign_notes.cid = detail.call_recording_url
LEFT JOIN campaign_categories ON campaign_notes.cid = campaign_categories.cid
AND campaign_notes.cid = detail.call_recording_url
WHERE detail.call_recording_url<>''
AND detail.event_type_name='Call'
AND campaign_notes.cid IS NOT NULL -- filters any from campaign notes that are empty
AND campaign_categories.cid IS NOT NULL -- filters any from campaign categories that are empty;
This grabs the urls from notes and categories and filters that match the call_recording_url. It then filters out any that didn't match in the left join (the is null check). If that is still too slow, there are other ways of optimizing. Then you should be able to loop through and do your update statement.
I would also try to index the ids you are joining on, especially the call_recording_url if that is a string, which takes longer to search on.
I would create a stored procedure for this and i would create some new indexsed depend on the queryies (for example one index on: call_recording_url, event_type_name).
And -- of course -- you can create a nicer query from the two first by mixing them to only one.
So I've got a little forum I'm trying to get data for, there are 4 tables, forum, forum_posts, forum_threads and users. What i'm trying to do is to get the latest post for each forum and giving the user a sneak peek of that post, i want to get the number of posts and number of threads in each forum aswell. Also, i want to do this in one query. So here's what i came up with:
SELECT lfx_forum_posts.*, lfx_forum.*, COUNT(lfx_forum_posts.pid) as posts_count,
lfx_users.username,
lfx_users.uid,
lfx_forum_threads.tid, lfx_forum_threads.parent_forum as t_parent,
lfx_forum_threads.text as t_text, COUNT(lfx_forum_threads.tid) as thread_count
FROM
lfx_forum
LEFT JOIN
(lfx_forum_threads
INNER JOIN
(lfx_forum_posts
INNER JOIN lfx_users
ON lfx_users.uid = lfx_forum_posts.author)
ON lfx_forum_threads.tid = lfx_forum_posts.parent_thread AND lfx_forum_posts.pid =
(SELECT MAX(lfx_forum_posts.pid)
FROM lfx_forum_posts
WHERE lfx_forum_posts.parent_forum = lfx_forum.fid
GROUP BY lfx_forum_posts.parent_forum)
)
ON lfx_forum.fid = lfx_forum_posts.parent_forum
GROUP BY lfx_forum.fid
ORDER BY lfx_forum.fid ASC
This get the latest post in each forum and gives me a sneakpeek of it, the problem is that
lfx_forum_posts.pid =
(SELECT MAX(lfx_forum_posts.pid)
FROM lfx_forum_posts
WHERE lfx_forum_posts.parent_forum = lfx_forum.fid
GROUP BY lfx_forum_posts.parent_forum)
Makes my COUNT(lfx_forum_posts.pid) go to one (aswell as the COUNT(lfx_forum_threads.tid) which isn't how i would like it to work. My question is: is there some somewhat easy way to make it show the correct number and at the same time fetch the correct post info (the latest one that is)?
If something is unclear please tell and i'll try to explain my issue further, it's my first time posting something here.
Hard to get an overview of the structure of your tables with only one big query like that.
Have you considered making a view to make it easier and faster to run the query?
Why do you have to keep it in one query? Personally I find that you can often gain both performance and code-readability by splitting overly complicated queries into more parts.
But hard to get an overview so can't really give a good answer to your question:)
Just add num_posts column to your table. Don't count posts with COUNT().
Can we get some...
Show Tables;
Desc Table lfx_forum_posts;
Desc Table lfx_forum_threads;
Desc Table lfx_forum_users;
Desc Table lfx_forum;
Here's some pseudo code
select f.*, (select count(*) from forum_posts fp where fp.forum_id = f.id) as num_posts,
(select count(*) from forum_threads ft where ft.forum_id = f.id) as num_threads,
(select max(fp.id) from forum_posts fp
where fp.id = f.id
) as latest_post_id,
from forums f;
Then go on to use latest_post_id in a seperate query to get it's information.
if it doesn't like using f before it's declared then make a temporary table for this then you update every time the query is ran.
I have a database schema like this:
My database schema: http://i.stack.imgur.com/vFKRk.png
To explain the context: One user writes one message. He can send it to one or more users.
I succeeded to get the title of message, the author for one user. However Doctrine, which I use for this project, do it with 2 queries. It's a little bit strange for me and I'm looking to understand, why. Normally, we can do it with one SQL query.
My DQL query:
$q = Doctrine_Query::create()->select('id_me, users_id_us, state_me, type_me, mc.title_mc, us.login_us') ->from('messages m')->innerJoin('m.messages_content mc')->innerJoin('mc.Users us') ->where('users_id_us = ?', $user)->limit($opt['limit'])->offset($opt['offset'])->orderBy($opt['order']);return $q->fetchArray();
SQL queries returned by Doctrine:
SELECT DISTINCT m3.id_me FROM messages m3 INNER JOIN messages_content m4 ON m3.messages_content_id_mc = m4.id_mc INNER JOIN users u2 ON m4.users_id_us = u2.id_us WHERE m3.users_id_us = '6' ORDER BY m3.id_me DESC LIMIT 2
SELECT m.id_me AS m__id_me, m.users_id_us AS m__users_id_us, m.state_me AS m__state_me, m.type_me AS m__type_me, m2.id_mc AS m2__id_mc, m2.title_mc AS m2__title_mc, u.id_us AS u__id_us, u.login_us AS u__login_us FROM messages m INNER JOIN messages_content m2 ON m.messages_content_id_mc = m2.id_mc INNER JOIN users u ON m2.users_id_us = u.id_us WHERE m.id_me IN ('11') AND (m.users_id_us = '6') ORDER BY m.id_me DESC
Why my Doctrine query doesn't return the query like this:
SELECT m.id_me, m.users_id_us, m.state_me, m.type_me, mc.title_mc, u.login_us FROM messages m JOIN messages_content mc ON mc.id_mc = m.messages_content_id_mc JOIN users u ON u.id_us = mc.users_id_us WHERE m.users_id_us = 6;
Any idea to transform my DQL query and execute it one time ?
The ORM Limit issue
This has to do with the LIMIT part. :) Doctrine LIMIT works a bit different than MySQL limit does.
MySQL LIMIT just issues the query, and stops searching as soon as n rows are found that matches your SQL query. Since in ORM, this is really unexpected behaviour (it might well be that in a scalar layout the SQL SELECT * FROM myModel LEFT JOIN someOtherModel ON someCondition LIMIT 3 actually only returns one myModel instance rather than three, since a left join can result 3 rows.
What does Doctrine do?
If your DQL query is FROM school s INNER JOIN s.students LIMIT 15, it means: give me 15 instances of school that have at least one student associated (hence INNER JOIN), with ALL of their student associates. To do this, Doctrine first asks for DISTINCT school with the exact same query parameters and a LIMIT part, to figure out which 15 school IDs should be returned. After this is done, these IDs are queried next, without the LIMIT part.
How to solve your issue
If you are not having an actual problem, huzzah, find the explaination of this behaviour above. If your query output is other than you expected, make sure you take these steps into consideraton. If for instance your DQL is FROM school s INNER JOIN s.students LIMIT 15, and you are wondering why you get more than 15 students, try: FROM students s INNER JOIN s.school LIMIT 15. In MySQL this means basically the same (disregarding the order of the result), though in Doctrine, this means you will get 15 students instead of 15 schools.
This bothered me too with some of the more complex queries. The only solution that I found was to bypass the sillyness altogether:
$q = Doctrine_Manager::getInstance()->getCurrentConnection();
$my_result = $q->fetchAssoc(" ... PUT SQL HERE ... ");
The solution which works:
I changed the relation alias and specified the columns participating in ON joint between messages_content and users.
The right Doctrine query is:
$q = Doctrine_Query::create()
->select('id_me, users_id_us, state_me, type_me, mc.title_mc, us.login_us')
->from('messages m')
->innerJoin('m.messages_content mc')
->innerJoin('m.Users us ON mc.users_id_us=us.id_us')
->where('users_id_us = ?', $user)
->limit($opt['limit'])
->offset($opt['offset'])
->orderBy($opt['order']);
It gives a SQL query like this:
SELECT m.id_me AS m__id_me, m.users_id_us AS m__users_id_us, m.state_me AS m__state_me, m.type_me AS m__type_me, m2.id_mc AS m2__id_mc, m2.title_mc AS m2__title_mc, u.id_us AS u__id_us, u.login_us AS u__login_us FROM messages m INNER JOIN messages_content m2 ON m.messages_content_id_mc = m2.id_mc INNER JOIN users u ON (m2.users_id_us = u.id_us) WHERE (m.users_id_us = '7') ORDER BY m.id_me DESC LIMIT 2
Tom and Pelle ten Cate, thanks for your participation.