Getting Popular Topics on a Custom Made Forum - php

For this website we're working on, we're trying to get the most popular topics (based on how many posts have been made in them within the last 24 hours). We have a medium to large based forum, and the current MySQL query looks like this:
SELECT `forums_topics`.`id`,`forums_topics`.`name`,
(
SELECT COUNT(`id`)
FROM `forums_posts`
WHERE `postdate` > (UNIX_TIMESTAMP()-60*60*24)
AND `topicid`=`forums_topics`.`id`
) AS `trendy_threads`
FROM `forums_topics`
WHERE `deleted`=0
AND `lastpost` > (UNIX_TIMESTAMP()-60*60*24)
ORDER BY `trendy_threads` DESC,`postdate` DESC
LIMIT 3
The SQL is quite sluggish.
How can we get this information as quickly and as efficiently as possible?
forums_topics
Field Type Null Key Default Extra
id int(50) NO PRI NULL auto_increment
uid varchar(255) NO NULL
flag int(1) NO 0
boardid varchar(255) NO NULL
postdate varchar(255) NO NULL
lastpost bigint(255) NO NULL
name varchar(50) NO NULL
description text NO NULL
body text NO NULL
author varchar(25) NO NULL
deleted tinyint(3) unsigned NO 0
deletememberid int(10) unsigned NO 0
pinned tinyint(1) NO 0
flagged text NO NULL
privateaccess text NO NULL
lastposter int(255) NO 1
replycount int(255) NO 0
viewcount int(255) NO 0
movedfrom int(255) NO 0
forums_posts
Field Type Null Key Default Extra
id int(50) NO PRI NULL auto_increment
topicid int(10) unsigned NO 0
author varchar(25) NO NULL
postdate varchar(255) NO NULL
body text NO NULL
lastedit varchar(255) NO NULL
postcount tinyint(1) NO NULL
invincible tinyint(1) NO 0
deleted tinyint(3) unsigned NO 0
deletememberid int(10) unsigned NO 0
thumbsup int(255) NO 0
thumbsdown int(255) NO 0
thumbsupuser text NO NULL
thumbsdownuser text NO NULL

The problem is probably that MySQL evaluates the subquery for every row. You can give MySQL a hint that that it should execute the subquery only once by moving the subquery into a join:
SELECT *
FROM forum_topics ft
JOIN (
SELECT topicid
, COUNT(*) as cnt
FROM forums_posts
WHERE postdate > UNIX_TIMESTAMP()-60*60*24
GROUP BY
topicid
) fpc
ON ft.topicid = fpc.topicid
WHERE ft.deleted = 0
ORDER BY
fpc.cnt DESC
, ft.postdate DESC
LIMIT 3
An index on forum_posts(postdate, topicid) would further improve performance.

I'm going to take a stab in the dark, and I'll edit further if needed. An EXPLAIN query would help.
SELECT `forums_topics`.*
FROM (
SELECT `topicid`, COUNT(*) as num
FROM `forums_posts`
WHERE `postdate` > (UNIX_TIMESTAMP()-60*60*24)
GROUP BY `topicid`
ORDER BY num DESC, `postdate` DESC
LIMIT 3
) `trendy`
LEFT JOIN `forums_topics` ON `id`=`topicid`
WHERE `deleted`=0

Related

MySQL Math on "join" function

Not too sure how to explain this, but, basically I have two tables, and I need to do some math on their values. I am having issues with anything other than addition (which it does automatically).
This is the MySQL statement that I am using.
select snippet_id, count(snippet_id) as cnt
from snippets_likes join snippets_engagement on snippet_id = snippets_engagement.snip_id
group by snippet_id
order by cnt desc
This statements pulls the total number of likes + engagements.
However, I want to have it equal likes + ( engagements/1000 ).
Table construction
CREATE TABLE `IOTunes`.`snippets_engagement` (
`ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`snip_id` int(10) unsigned NOT NULL DEFAULT '0',
`artist_id` int(10) unsigned NOT NULL DEFAULT '0',
`snip_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`engagement_type` int(10) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=6694 DEFAULT CHARSET=latin1;
CREATE TABLE `IOTunes`.`snippets_likes` (
`ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`snippet_id` int(10) unsigned NOT NULL DEFAULT '0',
`artist_id` int(10) unsigned NOT NULL DEFAULT '0',
`snippet_like` int(10) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=197 DEFAULT CHARSET=latin1;
I have tried so many different formulas, but, cannot get this to work. Any pointers?
Try this statement:
SELECT snippet_id, SUM(cnt) AS cnt
FROM (
SELECT snippet_id, COUNT(1) AS cnt
FROM snippets_likes
GROUP BY snippet_id
UNION
SELECT snippet_id, COUNT(1)/1000 AS cnt
FROM snippets_engagement
GROUP BY snippet_id
) AS subQ
GROUP BY snippet_id
ORDER BY cnt DESC
or if you insist on a join...
select sl.snippet_id, count(DISTINCT sl.ID) + count(DISTINCT se.ID)/1000 as cnt
from snippets_likes AS sl join snippets_engagement AS se
on sl.snippet_id = se.snip_id
group by snippet_id
order by cnt desc
I wouldn't recommend the join though, you'll only get values for "snippets" that have at least one "like" and one "engagement".

Mysql query to sort and merge two tables

I have two tables in my mysql database called forum_topics and forum_replies.
I am looking to have an overview of the latest posts / replies added, like this: http://prntscr.com/6ixtz4
to do so, i need a way to makee it sort by the time in both tables, and if a forum_reply is in the result it need to get its topic from the forum_topics.
How can i make this work? I have no mysql query to referer to to make this work.
CREATE TABLE IF NOT EXISTS `forum_topics` (
`id` int(255) NOT NULL AUTO_INCREMENT,
`topic` varchar(255) NOT NULL,
`thread` varchar(255) NOT NULL,
`level` int(2) NOT NULL DEFAULT '0',
`creator` int(255) NOT NULL,
`time` varchar(255) NOT NULL,
`innlegg` text NOT NULL,
`ip` varchar(255) NOT NULL,
`locked` enum('yes','no') NOT NULL DEFAULT 'no',
`deleted` enum('yes','no') NOT NULL DEFAULT 'no',
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=5 ;
CREATE TABLE IF NOT EXISTS `forum_replies` (
`id` int(255) NOT NULL AUTO_INCREMENT,
`topicid` int(255) NOT NULL,
`userid` int(255) NOT NULL,
`time` int(255) NOT NULL,
`reply` text NOT NULL,
`deleted` enum('yes','no') NOT NULL DEFAULT 'no',
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=20 ;
exampledata:
in table forum_topics
id |topic| time|
1 | test | 10
2 | test2 | 29
in table forum replies:
id | topicid | time |
1 | 1 | 18
2 | 2 | 28
As in this example i would like the outcome to be sorted something like this:
10
18
28
29
if forum_replies is one of the results it would need some data from forum_topics, but if forum_topics is one of the results it wouldnt need any data from the forum_replies.
The topicid is the id from the forum_topics of the topic that the reply is a response to.
So in the end i want to create the example output as in the image.
EDIT:
it would be a result containing in following order:
10
18
28
29
By loading the data in the example data sets, this will get the time from both tables, but sort the time from both as if they were in one table. The below query gets the results, and then orders them in order of time. Since both tables have the same structure, you can "union" them together as if they were one table and sort by time using "order_by"
select time
from forum_topics
UNION
select time
from forum_replies
order by time
Note: I was able to load both tables in the above request but without the last line on each. In other words I trimmed the part referring to the engine, and then just created the tables, and then entered the data provided..
I'm sorry I have to say the table structure of a forum post is incorrect. In your case, you will be very hard to sort a post based on topic and reply. I would suggest a table structure like this:
CREATE TABLE IF NOT EXISTS `forum_topics` (
`tid` int(255) NOT NULL AUTO_INCREMENT, // Topic ID
`tsubject` varchar(255) NOT NULL, // Topic Subject
`level` int(2) NOT NULL DEFAULT '0',
`creator` int(255) NOT NULL,
`innlegg` text NOT NULL, // I don't know what this means ...
`locked` enum('yes','no') NOT NULL DEFAULT 'no',
`deleted` enum('yes','no') NOT NULL DEFAULT 'no',
PRIMARY KEY (`tid`),
UNIQUE KEY `id` (`tid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=5;
CREATE TABLE IF NOT EXISTS `forum_posts` (
`pid` int(255) NOT NULL AUTO_INCREMENT,
`tid` int(255) NOT NULL,
`userid` int(255) NOT NULL,
`time` int(255) NOT NULL,
`pcontent` text NOT NULL, // Post Content
`isauthorpost` int(1) NOT NULL, // Optional: If this is the first post of the topic
`deleted` enum('yes','no') NOT NULL DEFAULT 'no',
PRIMARY KEY (`pid`),
UNIQUE KEY `id` (`pid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=20 ;
So now you will have a topic id to follow by the first. And you can just search for all replies with ORDER BY pid, for the topic details, you can use LEFT JOIN forum_topics or INNER JOIN to finish.

MySQL Query 127 seconds of execution

I have this SQL Query:
SELECT * FROM magnitudes t1 INNER JOIN value_magnitudes t2 ON t1.id = t2.magnitude_id WHERE t1.floor_id = 42 AND t2.reading_date = (SELECT Max(reading_date) FROM value_magnitudes WHERE t2.magnitude_id = t1.id);
And my DB Schema is:
CREATE TABLE magnitudes (
id INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
name VARCHAR(255) NULL ,
sdi_id VARCHAR(255) NULL ,
sdi_id_floor VARCHAR(255) NULL ,
visible TINYINT(1) UNSIGNED NULL ,
history TINYINT(1) UNSIGNED NULL ,
created_at DATETIME NULL ,
updated_at DATETIME NULL ,
floor_id INT(11) NULL ,
unit VARCHAR(255) NULL ,
kind_id INT(11) NULL ,
sample_time INT(11) NULL ,
history_from DATETIME NULL DEFAULT 2011-02-22 11:18:07 ,
PRIMARY KEY(id));
CREATE TABLE value_magnitudes (
id INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
value FLOAT NULL ,
magnitude_id INT(11) NULL ,
sdi_belongs_id VARCHAR(255) NULL ,
reading_date DATETIME NULL ,
created_at DATETIME NULL ,
updated_at DATETIME NULL ,
PRIMARY KEY(id));
What I actually want is, get the last value_magnitudes depending on the last updated_at by magnitudes.id on values_magnitudes.magnitude_id
Thanks.
SELECT TOP 1 VM.* FROM value_magnitudes VM, magnitudes M WHERE M.id = VM.id ORDER BY updated_at DESC
SELECT VM.* FROM value_magnitudes VM, magnitudes M WHERE M.id = VM.id ORDER BY updated_at DESC LIMIT 0,1;

SQL select query taking long time

I am using the below SQL query but it takes more than 180 sec to execute. Is there a way to speed it up ? This SQL give me the pic_id and of all the females.
SELECT pic_id, small
FROM picture
WHERE hide =0
AND userhide =0
AND `fbid`
IN (
SELECT fbid
FROM user
WHERE gender = "female"
)
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200
The Explain SQL
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY picture index NULL PRIMARY 4 NULL 1700 Using where
2 DEPENDENT SUBQUERY user ALL NULL NULL NULL NULL 7496 Using where
--- Result of explain statement for Tim's sql answer --
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE picture ALL NULL NULL NULL NULL 41443 Using where; Using temporary; Using filesort
1 SIMPLE user ALL NULL NULL NULL NULL 7501 Using where; Using join buffer
-- Structure ---
CREATE TABLE `user` (
`sid` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) CHARACTER SET utf8 NOT NULL,
`first_name` varchar(255) CHARACTER SET utf8 NOT NULL,
`username` varchar(255) CHARACTER SET utf8 NOT NULL,
`birthday` date NOT NULL,
`location` varchar(255) CHARACTER SET utf8 NOT NULL,
`gender` varchar(6) CHARACTER SET utf8 NOT NULL,
`created` date NOT NULL,
`fbid` bigint(50) NOT NULL,
`token` varchar(255) CHARACTER SET utf8 NOT NULL,
`relationship_status` varchar(20) CHARACTER SET utf8 NOT NULL,
`smallest` varchar(255) CHARACTER SET utf8 NOT NULL,
`email` varchar(40) CHARACTER SET utf8 NOT NULL,
`ref` varchar(15) NOT NULL,
PRIMARY KEY (`sid`),
KEY `gender` (`gender`),
KEY `fbid` (`fbid`)
) ENGINE=MyISAM AUTO_INCREMENT=7595 DEFAULT CHARSET=latin
---- structure of picture table ---
CREATE TABLE `picture` (
`fbid` bigint(50) NOT NULL,
`pic_id` int(11) NOT NULL AUTO_INCREMENT,
`pic_location` varchar(255) NOT NULL,
`hide` int(1) NOT NULL,
`small` varchar(255) NOT NULL,
`userhide` int(1) NOT NULL,
`likes` int(10) NOT NULL,
`hot` int(1) NOT NULL,
PRIMARY KEY (`pic_id`),
UNIQUE KEY `pic_location` (`pic_location`),
UNIQUE KEY `small` (`small`),
KEY `fbid` (`fbid`),
KEY `hide` (`hide`),
KEY `userhide` (`userhide`)
) ENGINE=MyISAM AUTO_INCREMENT=42749 DEFAULT CHARSET=latin1
try something like this:
SELECT pic_id, small
FROM picture
INNER JOIN user ON ( picture.fbid = user.fbid and user.gender='female' )
WHERE hide =0
AND userhide =0
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200
I put gender in the join because a query will not return rows that don't have a match on an inner join.
You should also read this stack overflow topic
EDIT:
make sure you have indexed the following fields:
picture.fbid
user.fbid
user.gender
picture.hide
picture.userhide
Try using a join instead:
SELECT p.pic_id, p.small
FROM picture p
INNER JOIN fbid f USING ( fbid )
WHERE p.hide =0
AND p.userhide =0
AND f.gender = 'female'
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200

This query/database is not working well together

Here's a query:
SELECT
*,
COUNT(*) as `numauth`
FROM `favorites` as `f1`
INNER JOIN `story` as `s1` ON `f1`.`story_id` = `s1`.`story_id`
WHERE `f1`.`story_id` != '".addslashes($_REQUEST['storyid'])."'
AND `f1`.`story_id` != '".addslashes($_REQUEST['storyid2'])."'
AND EXISTS (
SELECT 1 FROM `favorites` as `f2`
WHERE `story_id` = '".addslashes($_REQUEST['storyid'])."'
AND `f2`.`auth_id` = `f1`.`auth_id`)
AND EXISTS (
SELECT 1 FROM `favorites` as `f3`
WHERE `story_id` = '".addslashes($_REQUEST['storyid2'])."'
AND `f3`.`auth_id` = `f1`.`auth_id`)
AND NOT EXISTS (
SELECT 1 FROM `favorites` as `f4`
WHERE `story_id` =
'".addslashes($_REQUEST['exclude'])."'
`f4`.`auth_id` = `f1`.`auth_id`)
GROUP BY `f1`.`story_id`
ORDER BY `numauth` DESC, `story_words` DESC
And here's a description of the tables...
CREATE TABLE IF NOT EXISTS `favorites` (
`fav_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`auth_id` int(10) unsigned NOT NULL,
`story_id` int(10) unsigned NOT NULL,
PRIMARY KEY (`fav_id`),
UNIQUE KEY `auth_id_2` (`auth_id`,`story_id`),
KEY `auth_id` (`auth_id`),
KEY `story_id` (`story_id`),
KEY `fav_id` (`fav_id`,`auth_id`,`story_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1577985 ;
CREATE TABLE IF NOT EXISTS `story` (
`story_id` int(10) unsigned NOT NULL,
`story_title` varchar(255) NOT NULL,
`story_desc` text NOT NULL,
`story_authid` int(8) unsigned NOT NULL,
`story_authname` varchar(255) NOT NULL,
`story_fandom` varchar(255) NOT NULL,
`story_genre1` tinyint(2) unsigned NOT NULL,
`story_genre2` tinyint(2) unsigned NOT NULL,
`story_created` int(10) unsigned NOT NULL,
`story_updated` int(10) unsigned NOT NULL,
`story_reviews` smallint(5) unsigned NOT NULL,
`story_chapters` smallint(3) unsigned NOT NULL,
`story_rating` tinyint(2) unsigned NOT NULL,
`story_words` mediumint(7) unsigned NOT NULL,
`story_chars` varchar(255) NOT NULL,
UNIQUE KEY `story_id` (`story_id`),
KEY `story_authid` (`story_authid`),
KEY `story_fandom` (`story_fandom`),
KEY `story_authid_2` (`story_authid`,`story_fandom`),
KEY `story_id_2` (`story_id`,`story_authid`),
KEY `story_id_3` (`story_id`,`story_words`),
KEY `story_id_4` (`story_id`,`story_fandom`,`story_words`),
KEY `story_id_5` (`story_id`,`story_reviews`,`story_words`),
KEY `story_words` (`story_words`),
KEY `story_reviews` (`story_reviews`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Now I've done a fair bit of optimizing to get the query down to this. I'm running on a dedicated server but the query is still taking 5-7 seconds, which is unacceptable. We're looking at about 800,000 records on favorites and 400,000 records on stories, and I'm lost at this point on where to look to next for improvements.
It seems a bit daunting, so even if someone can point me in the right direction I'll be happy.
EXPLAIN with sample inputs:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY s1 ALL story_id,story_id_2,story_id_3,story_id_4,story_id... NULL NULL NULL 129429 Using where; Using temporary; Using filesort
1 PRIMARY f1 ref story_id story_id 4 fanfic_jordanl_ffrecs.s1.story_id 2 Using where
4 DEPENDENT SUBQUERY f4 eq_ref auth_id_2,auth_id,story_id auth_id_2 8 fanfic_jordanl_ffrecs.f1.auth_id,const 1 Using index
3 DEPENDENT SUBQUERY f3 eq_ref auth_id_2,auth_id,story_id auth_id_2 8 fanfic_jordanl_ffrecs.f1.auth_id,const 1 Using index
2 DEPENDENT SUBQUERY f2 eq_ref auth_id_2,auth_id,story_id auth_id_2 8 fanfic_jordanl_ffrecs.f1.auth_id,const 1 Using index
try this:
SELECT f1.*, s1.*, COUNT(*) as `numauth`
FROM `favorites` as `f1`
INNER JOIN `story` as `s1` ON `f1`.`story_id` = `s1`.`story_id`
INNER JOIN (
SELECT auth_id
FROM favorites
WHERE story_id IN ('".addslashes($_REQUEST['storyid'])."', '".addslashes($_REQUEST['storyid2'])."', '".addslashes($_REQUEST['exclude'])."')
GROUP BY auth_id
HAVING Count(IF(story_id = '".addslashes($_REQUEST['exclude'])."', 1, NULL)) = 0 AND Count(*) = 2
) fv ON f1.auth_id = fv.auth_id
WHERE `f1`.`story_id` != '".addslashes($_REQUEST['storyid'])."'
AND `f1`.`story_id` != '".addslashes($_REQUEST['storyid2'])."'
GROUP BY `f1`.`story_id`
ORDER BY `numauth` DESC, `story_words` DESC
Since you are selecting * but not grouping by auth_id, what exactly are you trying to do?
--- UPDATE
since you do not need all the fav info for the stories, this query should perform better:
SELECT s.*, fv.cnt
FROM story s
JOIN (
SELECT fv.story_id, COUNT(*) cnt
FROM favorites fv
JOIN (
SELECT auth_id
FROM favorites
WHERE story_id IN ('".addslashes($_REQUEST['storyid'])."', '".addslashes($_REQUEST['storyid2'])."', '".addslashes($_REQUEST['exclude'])."')
GROUP BY auth_id
HAVING Count(IF(story_id = '".addslashes($_REQUEST['exclude'])."', 1, NULL)) = 0 AND Count(*) = 2
) ufv ON fv.auth_id = ufv.auth_id
WHERE story_id != '".addslashes($_REQUEST['storyid'])."' AND story_id != '".addslashes($_REQUEST['storyid2'])."'
GROUP BY fv.story_id
ORDER BY COUNT(*) DESC
LIMIT 25
) fv ON s.story_id = fv.story_id
ORDER BY fv.cnt DESC, `story_words` DESC

Categories