This query/database is not working well together - php

Here's a query:
SELECT
*,
COUNT(*) as `numauth`
FROM `favorites` as `f1`
INNER JOIN `story` as `s1` ON `f1`.`story_id` = `s1`.`story_id`
WHERE `f1`.`story_id` != '".addslashes($_REQUEST['storyid'])."'
AND `f1`.`story_id` != '".addslashes($_REQUEST['storyid2'])."'
AND EXISTS (
SELECT 1 FROM `favorites` as `f2`
WHERE `story_id` = '".addslashes($_REQUEST['storyid'])."'
AND `f2`.`auth_id` = `f1`.`auth_id`)
AND EXISTS (
SELECT 1 FROM `favorites` as `f3`
WHERE `story_id` = '".addslashes($_REQUEST['storyid2'])."'
AND `f3`.`auth_id` = `f1`.`auth_id`)
AND NOT EXISTS (
SELECT 1 FROM `favorites` as `f4`
WHERE `story_id` =
'".addslashes($_REQUEST['exclude'])."'
`f4`.`auth_id` = `f1`.`auth_id`)
GROUP BY `f1`.`story_id`
ORDER BY `numauth` DESC, `story_words` DESC
And here's a description of the tables...
CREATE TABLE IF NOT EXISTS `favorites` (
`fav_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`auth_id` int(10) unsigned NOT NULL,
`story_id` int(10) unsigned NOT NULL,
PRIMARY KEY (`fav_id`),
UNIQUE KEY `auth_id_2` (`auth_id`,`story_id`),
KEY `auth_id` (`auth_id`),
KEY `story_id` (`story_id`),
KEY `fav_id` (`fav_id`,`auth_id`,`story_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1577985 ;
CREATE TABLE IF NOT EXISTS `story` (
`story_id` int(10) unsigned NOT NULL,
`story_title` varchar(255) NOT NULL,
`story_desc` text NOT NULL,
`story_authid` int(8) unsigned NOT NULL,
`story_authname` varchar(255) NOT NULL,
`story_fandom` varchar(255) NOT NULL,
`story_genre1` tinyint(2) unsigned NOT NULL,
`story_genre2` tinyint(2) unsigned NOT NULL,
`story_created` int(10) unsigned NOT NULL,
`story_updated` int(10) unsigned NOT NULL,
`story_reviews` smallint(5) unsigned NOT NULL,
`story_chapters` smallint(3) unsigned NOT NULL,
`story_rating` tinyint(2) unsigned NOT NULL,
`story_words` mediumint(7) unsigned NOT NULL,
`story_chars` varchar(255) NOT NULL,
UNIQUE KEY `story_id` (`story_id`),
KEY `story_authid` (`story_authid`),
KEY `story_fandom` (`story_fandom`),
KEY `story_authid_2` (`story_authid`,`story_fandom`),
KEY `story_id_2` (`story_id`,`story_authid`),
KEY `story_id_3` (`story_id`,`story_words`),
KEY `story_id_4` (`story_id`,`story_fandom`,`story_words`),
KEY `story_id_5` (`story_id`,`story_reviews`,`story_words`),
KEY `story_words` (`story_words`),
KEY `story_reviews` (`story_reviews`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Now I've done a fair bit of optimizing to get the query down to this. I'm running on a dedicated server but the query is still taking 5-7 seconds, which is unacceptable. We're looking at about 800,000 records on favorites and 400,000 records on stories, and I'm lost at this point on where to look to next for improvements.
It seems a bit daunting, so even if someone can point me in the right direction I'll be happy.
EXPLAIN with sample inputs:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY s1 ALL story_id,story_id_2,story_id_3,story_id_4,story_id... NULL NULL NULL 129429 Using where; Using temporary; Using filesort
1 PRIMARY f1 ref story_id story_id 4 fanfic_jordanl_ffrecs.s1.story_id 2 Using where
4 DEPENDENT SUBQUERY f4 eq_ref auth_id_2,auth_id,story_id auth_id_2 8 fanfic_jordanl_ffrecs.f1.auth_id,const 1 Using index
3 DEPENDENT SUBQUERY f3 eq_ref auth_id_2,auth_id,story_id auth_id_2 8 fanfic_jordanl_ffrecs.f1.auth_id,const 1 Using index
2 DEPENDENT SUBQUERY f2 eq_ref auth_id_2,auth_id,story_id auth_id_2 8 fanfic_jordanl_ffrecs.f1.auth_id,const 1 Using index

try this:
SELECT f1.*, s1.*, COUNT(*) as `numauth`
FROM `favorites` as `f1`
INNER JOIN `story` as `s1` ON `f1`.`story_id` = `s1`.`story_id`
INNER JOIN (
SELECT auth_id
FROM favorites
WHERE story_id IN ('".addslashes($_REQUEST['storyid'])."', '".addslashes($_REQUEST['storyid2'])."', '".addslashes($_REQUEST['exclude'])."')
GROUP BY auth_id
HAVING Count(IF(story_id = '".addslashes($_REQUEST['exclude'])."', 1, NULL)) = 0 AND Count(*) = 2
) fv ON f1.auth_id = fv.auth_id
WHERE `f1`.`story_id` != '".addslashes($_REQUEST['storyid'])."'
AND `f1`.`story_id` != '".addslashes($_REQUEST['storyid2'])."'
GROUP BY `f1`.`story_id`
ORDER BY `numauth` DESC, `story_words` DESC
Since you are selecting * but not grouping by auth_id, what exactly are you trying to do?
--- UPDATE
since you do not need all the fav info for the stories, this query should perform better:
SELECT s.*, fv.cnt
FROM story s
JOIN (
SELECT fv.story_id, COUNT(*) cnt
FROM favorites fv
JOIN (
SELECT auth_id
FROM favorites
WHERE story_id IN ('".addslashes($_REQUEST['storyid'])."', '".addslashes($_REQUEST['storyid2'])."', '".addslashes($_REQUEST['exclude'])."')
GROUP BY auth_id
HAVING Count(IF(story_id = '".addslashes($_REQUEST['exclude'])."', 1, NULL)) = 0 AND Count(*) = 2
) ufv ON fv.auth_id = ufv.auth_id
WHERE story_id != '".addslashes($_REQUEST['storyid'])."' AND story_id != '".addslashes($_REQUEST['storyid2'])."'
GROUP BY fv.story_id
ORDER BY COUNT(*) DESC
LIMIT 25
) fv ON s.story_id = fv.story_id
ORDER BY fv.cnt DESC, `story_words` DESC

Related

PHP/MySQL: How to compare the sum of certain fields on a query to a number?

I have a list of users that can enroll into courses and pay for them.
And I want to do a list of those users, and the courses that they are enrolled in and show there how much they've paid for each course already.
These are my tables:
CREATE TABLE usuarios(
userID int unsigned not null auto_increment primary key,
userEmail char(50) null,
userDNI int(10) null
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE rolesUsuarios (
rolesUsuariosID int unsigned not null auto_increment primary key,
userID int not null,
nombreRol char(50) not null
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE cursos (
cursoID int unsigned not null auto_increment primary key,
nombreCurso char(100) not null,
cursoPrecio int(10) null
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE cursosUsuarios (
cursosUsuariosID int unsigned not null auto_increment primary key,
userID int not null,
cursoID int not null
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE pagos (
pagoID int unsigned not null auto_increment primary key,
userID int not null,
pagoMonto int null
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE pagosVerificados (
pagosVerificadosID int unsigned not null auto_increment primary key,
userID int not null,
pagoID int not null,
cursoID int not null
)ENGINE=InnoDB DEFAULT CHARSET=utf8;
And this is the query that I'm using:
SELECT usuarios.userID AS useridd, usuarios.userEmail, usuarios.userApellido, rolesUsuarios.userID, rolesUsuarios.nombreRol,
cursos.cursoID, cursos.nombreCurso,
cursosUsuarios.cursoID, cursosUsuarios.userID,
GROUP_CONCAT(DISTINCT rolesUsuarios.nombreRol SEPARATOR '||') AS 'roles',
GROUP_CONCAT(DISTINCT cursos.cursoID,' - ',cursos.nombreCurso SEPARATOR '||') AS 'cursos',
GROUP_CONCAT(DISTINCT '$ ',pagos.pagoMonto,' course ID ',pagosVerificados.cursoID,'payment ID',pagos.pagoID ORDER BY pagos.pagoID SEPARATOR '||') AS 'pagos'
FROM usuarios LEFT JOIN rolesUsuarios ON usuarios.userID = rolesUsuarios.userID
LEFT JOIN cursosUsuarios ON usuarios.userID = cursosUsuarios.userID
LEFT JOIN cursos ON cursosUsuarios.cursoID = cursos.cursoID
LEFT JOIN roles ON rolesUsuarios.nombreRol = roles.nombreRol
LEFT JOIN pagosVerificados
ON rolesUsuarios.userID = pagosVerificados.userID
LEFT JOIN pagos
ON pagosVerificados.userID = pagos.userID
AND pagosVerificados.pagoID = pagos.pagoID
GROUP BY useridd
This is the result I get: A list of all payments in the same list, and I want to separate them according to each course.
The ultimate goal is to be able to compare the price of each course with the sum of all payments done for each course and know if there's any debt in any course.
How may I do that? Could I do that using PHP directly and that query, or should I modify the query?
Again I don't speak whatever language this is, but I would start with this and this join to others tables as needed.
select A.userID, A.cursoID, B.cursoPrecio, IFNULL(sums.totalpaid, 0) `totalpaid`, (B.cursoPrecio - IFNULL(sums.totalpaid, 0)) `amount left`
from `cursosusuarios` A
inner join `cursos` B on (A.cursoID = B.cursoID)
left outer join (
select A.userID, A.cursoID, sum(B.pagoMonto) `totalpaid`
FROM `pagosverificados` A
inner join `pagos` B on (A.userID = B.userID and A.pagoID = B.pagoID)
group by A.userID, A.cursoID ) sums on (A.userID = sums.userID AND A.cursoID = sums.cursoID)

MySQL - slowly query

I have a problem with a search post on my forum.
1) I split search string using explode(" ", $string);
2) I search all tag_id where tag_value contains word - SELECT tag_id WHERE tag_value like '%{word}%'; all results are added to array $tags_array
3) I created INNER JOIN
foreach ($tags_array as $k => $v) {
if (!empty($v)) {
$short_name = "frt_".$i++;
$inner_array[] = " INNER JOIN forum_rel_tags as ".$short_name." ON (".$short_name.".tag_id IN (".implode(", ", $v).")) ";
$more[] = " ".$short_name.".post_id = fp.post_id ";
}
}
4) and finally I have something like:
SELECT COUNT(fp.post_id) AS total
FROM forum_categories c
JOIN forum_thread t USE INDEX (thread_id)
ON t.category_id = c.category_id
AND t.posts_limit <= 1
AND t.contest_limit <= 3
AND (t.register_limit*86400) <= 5841372
JOIN forum_auth a
ON a.auth_id = c.category_id
AND a.auth_group_id = 1
AND a.auth_type = 1
AND a.auth_visible = 1
JOIN forum_posts fp
ON fp.thread_id = t.thread_id
AND fp.post_deleted = 0
JOIN forum_rel_tags frt_0
ON frt_0.post_id = fp.post_id
AND frt_0.tag_id IN (1000 tag_id)
JOIN forum_rel_tags frt_1
ON frt_1.post_id = fp.post_id
AND frt_1.tag_id IN (200 tag_id)
JOIN forum_rel_tags frt_2
ON frt_2.post_id = fp.post_id
AND frt_2.tag_id IN (432 tag_id)
JOIN forum_rel_tags frt_3
ON frt_3.post_id = fp.post_id
AND frt_3.tag_id IN (50 tag_id)
But this query is very slow.
What can I change to make it faster?
EXPLAIN SELECT
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE t ref thread_id thread_id 4 const 1 Using where; Using index
1 SIMPLE c eq_ref PRIMARY PRIMARY 4 pionas.t.category_id 1 Using index
1 SIMPLE a eq_ref auth_type auth_type 10 const,pionas.c.category_id,const 1 Using where
1 SIMPLE frt_0 range post_id,tag_id tag_id 4 NULL 372226 Using where; Using join buffer
1 SIMPLE frt_1 range post_id,tag_id tag_id 4 NULL 37787 Using where; Using join buffer
1 SIMPLE fp eq_ref PRIMARY,thread_id,post_deleted PRIMARY 4 pionas.frt_1.post_id 1 Using where
1 SIMPLE u eq_ref PRIMARY PRIMARY 4 pionas.fp.user_id 1 Using index
1 SIMPLE frt_3 range post_id,tag_id tag_id 4 NULL 23608 Using where; Using join buffer
1 SIMPLE frt_2 ref post_id,tag_id post_id 4 pionas.frt_3.post_id 296144 Using where; Using index
Tables
CREATE TABLE `forum_auth` (
`auth_type` smallint(1) unsigned NOT NULL DEFAULT '0',
`auth_id` int(10) unsigned NOT NULL,
`auth_visible` int(10) unsigned NOT NULL,
`auth_group_id` int(10) unsigned NOT NULL,
`auth_last_post_id` int(11) unsigned NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
ALTER TABLE `forum_auth`
ADD UNIQUE KEY `auth_type` (`auth_type`,`auth_id`,`auth_group_id`);
CREATE TABLE `forum_categories` (
`category_id` int(10) unsigned NOT NULL,
`category_name` varchar(200) NOT NULL,
`category_desc` text NOT NULL,
`category_order` tinyint(2) NOT NULL,
`category_parent` int(10) unsigned NOT NULL DEFAULT '0',
`last_post_id` int(11) NOT NULL,
`total_thread` int(11) NOT NULL,
`total_posts` int(11) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
ALTER TABLE `forum_categories`
ADD PRIMARY KEY (`category_id`);
CREATE TABLE `forum_posts` (
`post_id` int(10) unsigned NOT NULL,
`post_subject` varchar(250) NOT NULL,
`post_message` text NOT NULL,
`post_create_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`post_last_modify` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`post_count_modify` int(11) NOT NULL,
`post_deleted` tinyint(1) NOT NULL DEFAULT '0',
`post_ip` int(11) NOT NULL,
`user_id` int(10) NOT NULL,
`thread_id` int(10) unsigned NOT NULL,
`guest_name` varchar(100) NOT NULL,
`guest_mail` varchar(150) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
ALTER TABLE `forum_posts`
ADD PRIMARY KEY (`post_id`),
ADD KEY `thread_id` (`thread_id`),
ADD KEY `user_id` (`user_id`),
ADD KEY `post_deleted` (`thread_id`,`post_deleted`),
ADD KEY `post_create_date` (`post_create_date`),
ADD FULLTEXT KEY `post_message` (`post_message`);
CREATE TABLE `forum_rel_tags` (
`post_id` int(10) unsigned NOT NULL,
`tag_id` int(10) unsigned NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
ALTER TABLE `forum_rel_tags`
ADD UNIQUE KEY `post_id` (`post_id`,`tag_id`),
ADD KEY `tag_id` (`tag_id`);
CREATE TABLE `forum_tags` (
`tag_id` int(10) unsigned NOT NULL,
`name` varchar(150) NOT NULL,
`tag_url` varchar(150) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
ALTER TABLE `forum_tags`
ADD PRIMARY KEY (`tag_id`),
ADD UNIQUE KEY `tag_url` (`tag_url`);
CREATE TABLE `forum_thread` (
`thread_id` int(10) unsigned NOT NULL,
`thread_subject` varchar(250) CHARACTER SET utf8 NOT NULL,
`thread_desc` text CHARACTER SET utf8 NOT NULL,
`thread_create_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`thread_view` int(10) unsigned NOT NULL DEFAULT '0',
`thread_reply` int(10) unsigned NOT NULL DEFAULT '0',
`thread_ip` int(10) unsigned NOT NULL,
`last_post_id` int(11) NOT NULL,
`user_id` varchar(100) CHARACTER SET utf8 NOT NULL,
`category_id` int(10) unsigned NOT NULL,
`guest_name` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`guest_mail` varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
`guest_can_reply` smallint(1) NOT NULL DEFAULT '0',
`thread_block` tinyint(1) NOT NULL,
`thread_sticky` tinyint(1) unsigned NOT NULL DEFAULT '0',
`posts_limit` int(2) NOT NULL DEFAULT '0',
`register_limit` int(3) NOT NULL DEFAULT '0',
`contest_limit` int(2) NOT NULL DEFAULT '0'
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
ALTER TABLE `forum_thread`
ADD PRIMARY KEY (`thread_id`),
ADD KEY `category_id` (`category_id`),
ADD KEY `thread_id` (`thread_id`,`category_id`,`posts_limit`,`register_limit`,`contest_limit`);
Second query:
SELECT fp.*, u.username, u.rang, u.post, u.date_register, u.avatar, u.pkt, f.city
FROM forum_categories c
JOIN forum_thread t USE INDEX (thread_id)
ON t.category_id = c.category_id
AND t.posts_limit <= 1
AND t.contest_limit <= 3
AND (t.register_limit*86400) <= 5841372
JOIN forum_auth a
ON a.auth_id = c.category_id
AND a.auth_group_id = 1
AND a.auth_type = 1
AND a.auth_visible = 1
JOIN forum_posts fp
ON fp.thread_id = t.thread_id
AND fp.post_deleted = 0
JOIN forum_rel_tags frt_0
ON frt_0.post_id = fp.post_id
AND frt_0.tag_id IN (1000 tag_id)
JOIN forum_rel_tags frt_1
ON frt_1.post_id = fp.post_id
AND frt_1.tag_id IN (200 tag_id)
JOIN forum_rel_tags frt_2
ON frt_2.post_id = fp.post_id
AND frt_2.tag_id IN (432 tag_id)
JOIN forum_rel_tags frt_3
ON frt_3.post_id = fp.post_id
AND frt_3.tag_id IN (50 tag_id)
LEFT JOIN users u
ON fp.user_id=u.user_id
LEFT JOIN users_field f
ON u.user_id=f.user_id
GROUP BY fp.post_id
ORDER BY fp.post_id DESC
LIMIT 0,30;
[EDIT]
What you think about it:
EXPLAIN SELECT fp.*, u.username, u.rang, u.post, u.date_register, u.avatar, u.pkt, f.city
FROM forum_categories c
JOIN forum_thread t USE INDEX (thread_id)
ON t.category_id = c.category_id
AND t.posts_limit <= 1
AND t.contest_limit <= 3
AND t.register_limit <= (5841372/86400)
JOIN forum_auth a
ON a.auth_id = c.category_id
AND a.auth_group_id = 1
AND a.auth_type = 1
AND a.auth_visible = 1
JOIN forum_posts fp
ON fp.thread_id = t.thread_id
AND fp.post_deleted = 0
JOIN forum_rel_tags frt_0
ON frt_0.post_id = fp.post_id
JOIN forum_tags ft_0
ON ft_0.tag_id = frt_0.tag_id
AND ft_0.tag_url like '%play%'
JOIN forum_rel_tags frt_1
ON frt_0.post_id = frt_1.post_id
JOIN forum_tags ft_1
ON ft_1.tag_id = frt_1.tag_id
AND ft_1.tag_url like '%how%'
JOIN forum_rel_tags frt_2
ON frt_1.post_id = frt_2.post_id
JOIN forum_tags ft_2
ON ft_2.tag_id = frt_2.tag_id
AND ft_2.tag_url like '%win%'
JOIN forum_rel_tags frt_3
ON frt_2.post_id = frt_3.post_id
JOIN forum_tags ft_3
ON ft_3.tag_id = frt_3.tag_id
AND ft_3.tag_url like '%to%'
LEFT JOIN users u
ON fp.user_id=u.user_id
LEFT JOIN users_field f
ON u.user_id=f.user_id
GROUP BY fp.post_id
ORDER BY fp.post_id DESC
Assuming that user_id is unique in the users table, the LEFT JOIN to users table is unnecessary.
There are four joins to the forum_rel_tags table; there seems to be a potential there for multiple rows to be be returned from each join; and each of those rows is going to get "matched" to the rows returned from the other joins... which is a a partial cross product i.e. the same fp.post_id can be "counted" multiple times.
The most likely explanation for poor performance is the absence of suitable indexes.
Some predicates e.g. (t.register_limit*86400)<=5841372 cannot make use of an range scan operation on an index... the expression on t.register_limit*86400 has to be evaluated for every row (that isn't excluded by some other predicate) before the comparison to the literal can be made. We typically prefer to reference a bare column in the predicate, where it's possible to do that to return an equivalent result... e.g.
t.register_limit <= (584137/86400)
We can re-write the query, removing the unnecessary join to the users table, to something like this:
SELECT COUNT(fp.post_id) AS total
FROM forum_categories c
JOIN forum_thread t
ON t.category_id = c.category_id
AND t.posts_limit <= 1
AND t.contest_limit <= 3
AND (t.register_limit*86400) <= 5841372
JOIN forum_auth a
ON a.auth_id = c.category_id
AND a.auth_group_id = 1
AND a.auth_type = 1
AND a.auth_visible = 1
JOIN forum_posts fp
ON fp.thread_id = t.thread_id
AND fp.post_deleted = 0
JOIN forum_rel_tags frt_0
ON frt_0.post_id = fp.post_id
AND frt_0.tag_id IN (1000 tag_id)
JOIN forum_rel_tags frt_1
ON frt_1.post_id = fp.post_id
AND frt_1.tag_id IN (200 tag_id)
JOIN forum_rel_tags frt_2
ON frt_2.post_id = fp.post_id
AND frt_2.tag_id IN (432 tag_id)
JOIN forum_rel_tags frt_3
ON frt_3.post_id = fp.post_id
AND frt_3.tag_id IN (50 tag_id)
This makes it much easier to decipher the query, especially in terms of figuring out what indexes are likely to be the most appropriate for this query.
... ON forum_thread (category_id, posts_limit, contest_limit, register_limit)
... ON forum_auth (auth_id, auth_group, auth_type, auth_visible)
... ON forum_posts (thread_id, post_deleted, post_id)
... ON forum_rel_tags (post_id, tag_id)
Since these are covering indexes for the query, we expect the EXPLAIN output to include "Using index" for these tables. This is just a first cut at some indexes; these may already exist, or there may be more appropriate ordering of the columns in the indexes.
But absent table definitions, including indexes and estimated cardinalities, it's not really possible to give a more definitive answer.
Again...
I'm very suspicious of the partial cross product between the references to forum_rel_tags table. I suspect that this query can return a value for "total" that is higher than is expected. This is just a suspicion, because I don't see any specification (beyond the query) as to what result is supposed to be returned.

MySQL Math on "join" function

Not too sure how to explain this, but, basically I have two tables, and I need to do some math on their values. I am having issues with anything other than addition (which it does automatically).
This is the MySQL statement that I am using.
select snippet_id, count(snippet_id) as cnt
from snippets_likes join snippets_engagement on snippet_id = snippets_engagement.snip_id
group by snippet_id
order by cnt desc
This statements pulls the total number of likes + engagements.
However, I want to have it equal likes + ( engagements/1000 ).
Table construction
CREATE TABLE `IOTunes`.`snippets_engagement` (
`ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`snip_id` int(10) unsigned NOT NULL DEFAULT '0',
`artist_id` int(10) unsigned NOT NULL DEFAULT '0',
`snip_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`engagement_type` int(10) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=6694 DEFAULT CHARSET=latin1;
CREATE TABLE `IOTunes`.`snippets_likes` (
`ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`snippet_id` int(10) unsigned NOT NULL DEFAULT '0',
`artist_id` int(10) unsigned NOT NULL DEFAULT '0',
`snippet_like` int(10) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=197 DEFAULT CHARSET=latin1;
I have tried so many different formulas, but, cannot get this to work. Any pointers?
Try this statement:
SELECT snippet_id, SUM(cnt) AS cnt
FROM (
SELECT snippet_id, COUNT(1) AS cnt
FROM snippets_likes
GROUP BY snippet_id
UNION
SELECT snippet_id, COUNT(1)/1000 AS cnt
FROM snippets_engagement
GROUP BY snippet_id
) AS subQ
GROUP BY snippet_id
ORDER BY cnt DESC
or if you insist on a join...
select sl.snippet_id, count(DISTINCT sl.ID) + count(DISTINCT se.ID)/1000 as cnt
from snippets_likes AS sl join snippets_engagement AS se
on sl.snippet_id = se.snip_id
group by snippet_id
order by cnt desc
I wouldn't recommend the join though, you'll only get values for "snippets" that have at least one "like" and one "engagement".

i want to use left and inner join in 3 tables in mysql?

these are my tables. first one is appusers table.
CREATE TABLE IF NOT EXISTS `appusers` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`email` varchar(50) NOT NULL,
`is_active` tinyint(2) NOT NULL DEFAULT '0',
`zip` varchar(20) NOT NULL,
`city` text NOT NULL,
`country` text NOT NULL,
`created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=23 ;
second table is stickeruses table.
CREATE TABLE IF NOT EXISTS `stickeruses` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`sticker_id` int(11) NOT NULL,
`count` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=24 ;
Third table is Devices
CREATE TABLE IF NOT EXISTS `devices` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`regid` varchar(300) NOT NULL,
`imei` varchar(50) NOT NULL,
`device_type` tinyint(2) NOT NULL,
`notification` tinyint(2) NOT NULL DEFAULT '1',
`is_active` tinyint(2) NOT NULL DEFAULT '0',
`activationcode` int(6) NOT NULL,
`created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=28 ;
I Want to find the Sum(stickeruses.count) and COUNT(devices.id) for all appusers.
Here is my query.
SELECT `Appuser`.`id`, `Appuser`.`email`, `Appuser`.`country`, `Appuser`.`created`,
`Appuser`.`is_active`, SUM(`Stickeruse`.`count`) AS total, COUNT(`Device`.`id`)
AS tdevice
FROM `stickerapp`.`appusers` AS `Appuser`
LEFT JOIN `stickerapp`.`stickeruses` AS `Stickeruse`
ON (`Stickeruse`.`user_id`=`Appuser`.`id`)
INNER JOIN `stickerapp`.`devices` AS `Device`
ON (`Device`.`user_id`=`Appuser`.`id`)
WHERE `Appuser`.`is_active` = 1
GROUP BY `Appuser`.`id`
LIMIT 10
When I am applying each join separately the results are right, but I want to combine both joins. And when I am doing it then results are wrong. please help.
When mixing JOIN and LEFT JOIN it is a good idea to use parentheses to make it clear what your intent is.
I don't know what you need, but these syntaxes might give you different results:
FROM a LEFT JOIN ( b JOIN c ON b..c.. ) bc ON a..bc..
FROM ( a LEFT JOIN b ON a..b.. ) ab JOIN c ON ab..c..
Also, you can rearrange them do FROM a JOIN c LEFT JOIN b (plus parentheses) or any of several other arrangements. Granted, some pairs rearrangements are equivalent.
Also, beware; aggregates (such as SUM()) get inflated values when JOINing. Think of it this way: first the JOINs get all appropriate combinations of rows from the tables, then the SUM adds them up. With that in mind, see if this works better:
SELECT a.`id`, a.`email`, a.`country`, a.`created`, a.`is_active`,
( SELECT SUM(`count`)
FROM stickerapp.stickeruses
WHERE user_id = a.id
) AS total,
( SELECT COUNT(*)
FROM stickerapp.devices
WHERE user_id = a.id
) AS tdevice
FROM stickerapp.`appusers` AS a
WHERE a.`is_active` = 1
GROUP BY a.`id`
LIMIT 10

Improve speed of MySQL query [closed]

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 10 years ago.
EDIT:
I reworked the query as follows:
SELECT
a.title, count(*),at.search_code
FROM
`qitz3_attributes_type` at
left join
qitz3_attributes a
on
a.attribute_type_id = at.id
left join
qitz3_attributes_property ap
on
ap.attribute_id = a.id
left join
qitz3_helloworld h
on
h.id = ap.property_id
where
at.id in (1,2,8,9,11)
and
a.search_filter = 1
and
h.area=506
and
h.expiry_date >= '2013-02-20 13:28:04'
group by
a.title
order by search_code
This seems much faster but I am still getting a using temporary and using filesort on the explain...
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE at range PRIMARY PRIMARY 4 NULL 5 Using where; Using temporary; Using filesort
1 SIMPLE a ref PRIMARY,Attribute type ID,Search filter,Attribute ... Attribute type ID 4 password.at.id 6 Using where
1 SIMPLE ap ref Property ID,Attribute ID,Attribute Property Attribute Property 4 password.a.id 142 Using where; Using index
1 SIMPLE h eq_ref PRIMARY,Area indexes,Expiry date PRIMARY 4 password.ap.property_id 1 Using where
ENDEDIT
I am working on a search component for a site I am developing and while it is all working there are a couple of queries that I would like to see running a touch faster.
In order to populate a set of search filters (display the count of the number of properties that have each facility or are of a particular type) I am using the following two queries. The first will get a list of IDs that I then plug into the second query as follows:
Based on the following info can anyone suggest a more efficient way to do this? I would really like to speed the first query up as it seems a little slow compared to the others.
Is 150ms actually that slow? 15ms would be better...
Thanks,
Adam
Query 1 (takes around 150ms):
SELECT h.id, h.parent_id, h.level, h.title as property_title, h.area, h.region, h.department, h.city,
LEFT(h.description, 250) as description, h.thumbnail, h.occupancy, h.swimming, g.path, (single_bedrooms + double_bedrooms + triple_bedrooms + quad_bedrooms + twin_bedrooms) as bedrooms, c.title as location_title, ( select min(tariff)
from qitz3_tariffs
where id = h.id ) as price, e.title as tariff_based_on, f.title as base_currency, a.title as property_type, a2.title as accommodation_type, ( select count(*)
from qitz3_reviews
where property_id = h.id
group by h.id ) as reviews
FROM qitz3_classifications c
LEFT JOIN qitz3_helloworld h
on c.id = h.area
LEFT JOIN qitz3_attributes_property ap
ON ap.property_id = h.id
LEFT JOIN qitz3_attributes_type at
ON at.id = ap.attribute_id
LEFT JOIN qitz3_attributes a
ON a.id = ap.attribute_id
LEFT JOIN qitz3_attributes_property ap2
ON ap2.property_id = h.id
LEFT JOIN qitz3_attributes_type at2
ON at2.id = ap2.attribute_id
LEFT JOIN qitz3_attributes a2
ON a2.id = ap2.attribute_id
LEFT JOIN qitz3_attributes e
ON e.id = h.tariff_based_on
LEFT JOIN qitz3_attributes f
ON f.id = h.base_currency
LEFT JOIN qitz3_classifications g
ON g.id = h.city
WHERE a.attribute_type_id = 1
AND a2.attribute_type_id = 2
AND c.id = 506
AND h.expiry_date >= '2013-02-20 12:05:13'
AND h.id > 1
Explain:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY c const PRIMARY PRIMARY 4 const 1
1 PRIMARY h ref PRIMARY,Area indexes,Expiry date Area indexes 4 const 615 Using where
1 PRIMARY ap ref Property ID,Attribute ID Property ID 4 password.h.id 21 Using where
1 PRIMARY at eq_ref PRIMARY PRIMARY 4 password.ap.attribute_id 1 Using index
1 PRIMARY a eq_ref PRIMARY,Attribute type ID PRIMARY 4 password.ap.attribute_id 1 Using where
1 PRIMARY ap2 ref Property ID,Attribute ID Property ID 4 password.ap.property_id 21 Using where
1 PRIMARY at2 eq_ref PRIMARY PRIMARY 4 password.ap2.attribute_id 1 Using index
1 PRIMARY a2 eq_ref PRIMARY,Attribute type ID PRIMARY 4 password.ap2.attribute_id 1 Using where
1 PRIMARY e eq_ref PRIMARY PRIMARY 4 password.h.tariff_based_on 1
1 PRIMARY f eq_ref PRIMARY PRIMARY 4 password.h.base_currency 1
1 PRIMARY g eq_ref PRIMARY PRIMARY 4 password.h.city 1
3 DEPENDENT SUBQUERY qitz3_reviews ref Property ID Property ID 4 password.h.id 1 Using index; Using temporary; Using filesort
2 DEPENDENT SUBQUERY qitz3_tariffs ref Property ID Property ID 4 password.h.id 2
Query 2 (takes around 30ms):
SELECT a.id,count(attribute_id) as count, a.title AS attribute, a.published, at.title as facility_type, at.search_code
FROM qitz3_attributes AS a
LEFT JOIN qitz3_attributes_type at
on at.id = a.attribute_type_id
LEFT JOIN qitz3_attributes_property ap
on ap.attribute_id = a.id
WHERE search_filter = 1
AND a.published = 1
AND property_id in (152843,103180,152845,4628,5653,3865,107553,155945,106029,107575,149052,837,104264,98635,98636,98637,98638,3667,106838,3672,157278,155743,157791,157792,151153,151155,100725,106109,157569,157576,107145,150666,103310,5780,3480,102041,154016,3490,154018,932,153518,151991,154041,154042,4288,5832,149451,157646,102094,148444,153822,157407,153839,151536,157393,157395,157428,157429,102236,104770,157378,157380,157381,157382,157383,157654,104768,103746,153683,150175,4618,101050,104942,157229,157230,98515,104771,104944,3612,148721,104212,5307,3432,156676,102706,404,5518,156359,5252,102697,5271,98979,101827,149524,102676,107551,5685,101736,156538,152703,4730,151881,95838,3759,149769,5269,98429,153729,5233,703,5579,98943,157433,3157,155661,107347,147545,147547,5216,106345,156533,93833,158091,96403,3491,968,158195,158196,157580,104148,3030,94686,154725,150582,103027,99062,102462,4384,5634,153874,157974,101669,47,105285,102481,102234,5749,156793,153748,96404,151467,154292,147645,97471,100551,102090,4841,3563,155643,4656,98424,157243,150601,157415,4701,102283,100719,100738,5643,98425,98972,103261,531,3105,98108,150592,5719,150616,157532,3974,3212,157581,97469,97470,149183,157638,149730,102114,156395,153621,102560,102913,94684,5609,157578,98423,98971,102151,146734,150585,104287,155296,104956,94592,102433,147575,156325,106344,101766,107058,106560,103026,157848,98973,4303,5620,149767,150563,4407,104268,97876,156784,156786,149922,701,154317,153821,102480,348,102778,102779,102780,102781,102479,352,103025,98677,5254,98697,3995,156322,100305,98532,3833,5374,150172,151435,102368,102380,157228,103171,147740,152870,103579,3870,104037,103016,4995,105104,157605,5811,955,147643,156648,107802,101502,94685,3569,148755,150293,4122,157013,157297,98676,156794,102848,157635,157640,95717,98980,102764,102777,102782,36,101765,154373,149829,154955,107683,158176,102557,157552,103163,5760,104627,153561,266,151335,151176,147620,147379,3085,155760,106339,151424,106759,5145,104990,97877,155495,5241,156407,156625,3236,152782,96066,147617,3860,4614,3497,147883,158207,102985,104622,101816,157275,149037,4792,149226,3496,101825,102538,150571,105015,97874,157391,158192,102562,155032,5383,102558,104194,156740,101446,147615,5815,107081,155992,97473,148817,945,101751,158074,4249,101792,4532,152828,104316,157319,156071,157508,157510,148836,4745,153823,157942,3859,157442,100017,102555,147629,149272,157845,151256,101481,154735,154737,157652,106232,97991,4660,3309,157597,407,157658,154152,157374,153385,148037,158214,100452)
GROUP BY a.id
This query gives a count of the number of properties with each attribute (e.g. Golf, Air conditioning, property type etc). The good thing about this is that only attributes are shown that have properties. So as the user drills down attributes with no properties are not shown. This is basically down to the first query where I get a list of the properties matching a particular set of attributes.
The tables are as follows:
--
-- Table structure for table qitz3_attributes
CREATE TABLE IF NOT EXISTS `qitz3_attributes` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`attribute_type_id` int(11) NOT NULL,
`title` varchar(75) NOT NULL,
`ordering` int(11) NOT NULL,
`state` tinyint(4) NOT NULL DEFAULT '1',
`published` int(11) NOT NULL,
`search_filter` bit(1) NOT NULL,
`language_code` varchar(6) NOT NULL DEFAULT 'en-GB',
PRIMARY KEY (`id`),
KEY `Attribute type ID` (`attribute_type_id`),
KEY `Search filter` (`search_filter`),
KEY `Published` (`published`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1522 ;
--
-- Table structure for table qitz3_attributes_property
CREATE TABLE IF NOT EXISTS `qitz3_attributes_property` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`property_id` int(11) NOT NULL,
`attribute_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `Property ID` (`property_id`),
KEY `Attribute ID` (`attribute_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=66261 ;
--
-- Table structure for table qitz3_attributes_type
CREATE TABLE IF NOT EXISTS `qitz3_attributes_type` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(75) NOT NULL,
`language_code` varchar(6) NOT NULL,
`field_name` varchar(25) NOT NULL,
`search_code` varchar(25) NOT NULL,
`state` int(11) NOT NULL DEFAULT '1',
`published` int(11) NOT NULL DEFAULT '1',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=34 ;
--
-- Table structure for table qitz3_classifications
CREATE TABLE IF NOT EXISTS `qitz3_classifications` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`parent_id` int(10) unsigned NOT NULL DEFAULT '0',
`lft` int(11) NOT NULL DEFAULT '0',
`rgt` int(11) NOT NULL DEFAULT '0',
`level` int(10) unsigned NOT NULL DEFAULT '0',
`title` varchar(255) NOT NULL,
`description` text NOT NULL,
`path` varchar(255) NOT NULL DEFAULT '',
`alias` varchar(255) NOT NULL,
`access` tinyint(3) unsigned NOT NULL DEFAULT '0',
`published` int(11) NOT NULL,
`longitude` float(10,6) NOT NULL,
`latitude` float(10,6) NOT NULL,
PRIMARY KEY (`id`),
KEY `idx_left_right` (`lft`,`rgt`),
KEY `Alias index` (`alias`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=158052 ;
--
-- Table structure for table qitz3_helloworld (Property table)
CREATE TABLE IF NOT EXISTS `qitz3_helloworld` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`parent_id` int(10) NOT NULL DEFAULT '0',
`lft` int(11) NOT NULL DEFAULT '0',
`rgt` int(11) NOT NULL DEFAULT '0',
`level` int(10) unsigned NOT NULL,
`alias` varchar(250) NOT NULL DEFAULT '',
`access` tinyint(3) unsigned NOT NULL DEFAULT '0',
`path` varchar(255) NOT NULL DEFAULT '',
`title` varchar(120) NOT NULL,
`area` int(11) NOT NULL DEFAULT '0',
`region` int(11) NOT NULL DEFAULT '0',
`department` int(11) NOT NULL DEFAULT '0',
`city` int(11) NOT NULL DEFAULT '0',
`params` text NOT NULL,
`created_by` int(10) NOT NULL DEFAULT '0',
`created_on` datetime NOT NULL,
`modified` datetime DEFAULT NULL,
`expiry_date` date DEFAULT NULL,
`availability_last_updated_on` datetime DEFAULT NULL,
`modified_by` int(11) DEFAULT NULL,
`lang` varchar(5) NOT NULL DEFAULT 'en-GB',
`description` mediumtext NOT NULL COMMENT 'The summary and description for this accommodation',
`internal_facilities_other` varchar(1000) NOT NULL,
`external_facilities_other` varchar(1000) NOT NULL,
`activities_other` varchar(5000) NOT NULL,
`location_details` varchar(5000) NOT NULL,
`getting_there` varchar(5000) NOT NULL,
`thumbnail` varchar(150) NOT NULL,
`occupancy` int(11) DEFAULT NULL,
`single_bedrooms` int(11) NOT NULL,
`double_bedrooms` int(11) NOT NULL,
`triple_bedrooms` int(11) DEFAULT NULL,
`quad_bedrooms` int(11) DEFAULT NULL,
`twin_bedrooms` int(11) DEFAULT NULL,
`childrens_beds` int(11) DEFAULT NULL,
`cots` int(11) DEFAULT NULL,
`extra_beds` int(11) DEFAULT NULL,
`bathrooms` int(11) NOT NULL,
`toilets` int(11) DEFAULT NULL,
`swimming` int(11) NOT NULL,
`latitude` decimal(10,7) DEFAULT NULL,
`longitude` decimal(10,7) DEFAULT NULL,
`nearest_town` varchar(50) DEFAULT NULL,
`distance_to_coast` int(11) DEFAULT NULL,
`additional_price_notes` varchar(3000) DEFAULT NULL,
`base_currency` int(11) DEFAULT NULL,
`tariff_based_on` int(11) DEFAULT NULL,
`linen_costs` varchar(250) DEFAULT NULL,
`changeover_day` int(11) DEFAULT NULL,
`published` tinyint(4) NOT NULL DEFAULT '0',
`video` tinyint(4) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
KEY `idx_left_right` (`lft`,`rgt`),
KEY `Area indexes` (`area`,`region`,`department`),
KEY `Expiry date` (`expiry_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=158249 ;
--
-- Table structure for table qitz3_reviews
CREATE TABLE IF NOT EXISTS `qitz3_reviews` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`property_id` int(11) NOT NULL,
`title` varchar(150) NOT NULL,
`review_text` varchar(4000) NOT NULL,
`date` date NOT NULL,
`rating` int(11) NOT NULL,
`guest_name` varchar(75) NOT NULL,
`guest_email` varchar(150) NOT NULL,
`state` tinyint(3) NOT NULL DEFAULT '0',
`published` tinyint(1) NOT NULL DEFAULT '0',
`created` datetime NOT NULL,
`created_by` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `Property ID` (`property_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=921 ;
--
-- Table structure for table qitz3_tariffs
CREATE TABLE IF NOT EXISTS `qitz3_tariffs` (
`tariff_id` int(11) NOT NULL AUTO_INCREMENT,
`id` int(11) NOT NULL COMMENT 'Denotes the property listing ID',
`start_date` date NOT NULL,
`end_date` date NOT NULL,
`tariff` int(11) NOT NULL COMMENT 'Price per booking period between the dates specified. dated spec',
PRIMARY KEY (`tariff_id`),
KEY `Property ID` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=9267 ;
Quick play, but trying to move the subselects out from the field list.
SELECT h.id, h.parent_id, h.level, h.title as property_title, h.area, h.region, h.department, h.city, LEFT(h.description, 250) as description, h.thumbnail, h.occupancy, h.swimming, g.path, (single_bedrooms + double_bedrooms + triple_bedrooms + quad_bedrooms + twin_bedrooms) as bedrooms, c.title as location_title,
Sub1.price,
e.title as tariff_based_on, f.title as base_currency, a.title as property_type, a2.title as accommodation_type,
Sub2.reviews
FROM qitz3_classifications c
LEFT JOIN qitz3_helloworld h on c.id = h.area
LEFT JOIN qitz3_attributes_property ap ON ap.property_id = h.id
LEFT JOIN qitz3_attributes_type at ON at.id = ap.attribute_id
LEFT JOIN qitz3_attributes a ON a.id = ap.attribute_id
LEFT JOIN qitz3_attributes_property ap2 ON ap2.property_id = h.id
LEFT JOIN qitz3_attributes_type at2 ON at2.id = ap2.attribute_id
LEFT JOIN qitz3_attributes a2 ON a2.id = ap2.attribute_id
LEFT JOIN qitz3_attributes e ON e.id = h.tariff_based_on
LEFT JOIN qitz3_attributes f ON f.id = h.base_currency
LEFT JOIN qitz3_classifications g ON g.id = h.city
LEFT JOIN ( SELECT id, MIN(tariff) AS price FROM qitz3_tariffs GROUP BY id) Sub1 ON Sub1.Id = h.id
LEFT JOIN ( SELECT property_id, COUNT(*) AS reviews FROM qitz3_reviews GROUP BY property_id ) as Sub2 ON Sub2.property_id = h.id
WHERE a.attribute_type_id = 1
AND a2.attribute_type_id = 2
AND c.id = 506
AND h.expiry_date >= '2013-02-20 12:05:13'
AND h.id > 1
When you need more speed in searching data you should take a look at Solr or Sphinx. With this index servers you can index your MySQL-Data and query them.
Its much more faster then MySQL.
There's quite a lot wrong in your first query:
The first 7 LEFT JOINs should be INNER JOINs. For qitz3_attributes you should consider normalizing the data less aggressively (use multiple columns instead of multiple rows to describe the data.

Categories