Mysql query to sort and merge two tables - php

I have two tables in my mysql database called forum_topics and forum_replies.
I am looking to have an overview of the latest posts / replies added, like this: http://prntscr.com/6ixtz4
to do so, i need a way to makee it sort by the time in both tables, and if a forum_reply is in the result it need to get its topic from the forum_topics.
How can i make this work? I have no mysql query to referer to to make this work.
CREATE TABLE IF NOT EXISTS `forum_topics` (
`id` int(255) NOT NULL AUTO_INCREMENT,
`topic` varchar(255) NOT NULL,
`thread` varchar(255) NOT NULL,
`level` int(2) NOT NULL DEFAULT '0',
`creator` int(255) NOT NULL,
`time` varchar(255) NOT NULL,
`innlegg` text NOT NULL,
`ip` varchar(255) NOT NULL,
`locked` enum('yes','no') NOT NULL DEFAULT 'no',
`deleted` enum('yes','no') NOT NULL DEFAULT 'no',
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=5 ;
CREATE TABLE IF NOT EXISTS `forum_replies` (
`id` int(255) NOT NULL AUTO_INCREMENT,
`topicid` int(255) NOT NULL,
`userid` int(255) NOT NULL,
`time` int(255) NOT NULL,
`reply` text NOT NULL,
`deleted` enum('yes','no') NOT NULL DEFAULT 'no',
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=20 ;
exampledata:
in table forum_topics
id |topic| time|
1 | test | 10
2 | test2 | 29
in table forum replies:
id | topicid | time |
1 | 1 | 18
2 | 2 | 28
As in this example i would like the outcome to be sorted something like this:
10
18
28
29
if forum_replies is one of the results it would need some data from forum_topics, but if forum_topics is one of the results it wouldnt need any data from the forum_replies.
The topicid is the id from the forum_topics of the topic that the reply is a response to.
So in the end i want to create the example output as in the image.
EDIT:
it would be a result containing in following order:
10
18
28
29

By loading the data in the example data sets, this will get the time from both tables, but sort the time from both as if they were in one table. The below query gets the results, and then orders them in order of time. Since both tables have the same structure, you can "union" them together as if they were one table and sort by time using "order_by"
select time
from forum_topics
UNION
select time
from forum_replies
order by time
Note: I was able to load both tables in the above request but without the last line on each. In other words I trimmed the part referring to the engine, and then just created the tables, and then entered the data provided..

I'm sorry I have to say the table structure of a forum post is incorrect. In your case, you will be very hard to sort a post based on topic and reply. I would suggest a table structure like this:
CREATE TABLE IF NOT EXISTS `forum_topics` (
`tid` int(255) NOT NULL AUTO_INCREMENT, // Topic ID
`tsubject` varchar(255) NOT NULL, // Topic Subject
`level` int(2) NOT NULL DEFAULT '0',
`creator` int(255) NOT NULL,
`innlegg` text NOT NULL, // I don't know what this means ...
`locked` enum('yes','no') NOT NULL DEFAULT 'no',
`deleted` enum('yes','no') NOT NULL DEFAULT 'no',
PRIMARY KEY (`tid`),
UNIQUE KEY `id` (`tid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=5;
CREATE TABLE IF NOT EXISTS `forum_posts` (
`pid` int(255) NOT NULL AUTO_INCREMENT,
`tid` int(255) NOT NULL,
`userid` int(255) NOT NULL,
`time` int(255) NOT NULL,
`pcontent` text NOT NULL, // Post Content
`isauthorpost` int(1) NOT NULL, // Optional: If this is the first post of the topic
`deleted` enum('yes','no') NOT NULL DEFAULT 'no',
PRIMARY KEY (`pid`),
UNIQUE KEY `id` (`pid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=20 ;
So now you will have a topic id to follow by the first. And you can just search for all replies with ORDER BY pid, for the topic details, you can use LEFT JOIN forum_topics or INNER JOIN to finish.

Related

Optimization Needed For Dual Left Join Query

I've always struggled with mysql joins but have started incorporating more but struggling to understand despite reading dozens of tutorials and mysql manual.
My situation is I have 3 tables:
/* BASICALLY A TABLE THAT HOLDS FAN RECORDS */
CREATE TABLE `fans` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`first_name` varchar(255) DEFAULT NULL,
`middle_name` varchar(255) DEFAULT NULL,
`last_name` varchar(255) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`join_date` datetime DEFAULT NULL,
`twitter` varchar(255) DEFAULT NULL,
`twitterCrawled` datetime DEFAULT NULL,
`twitterImage` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `email` (`email`)
) ENGINE=MyISAM AUTO_INCREMENT=20413 DEFAULT CHARSET=latin1;
/* A TABLE OF OUR TWITTER FOLLOWERS */
CREATE TABLE `twitterFollowers` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`screenName` varchar(25) DEFAULT NULL,
`twitterId` varchar(25) DEFAULT NULL,
`customerId` int(11) DEFAULT NULL,
`uniqueStr` varchar(50) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique` (`uniqueStr`)
) ENGINE=InnoDB AUTO_INCREMENT=13426 DEFAULT CHARSET=utf8;
/* TABLE THAT SUGGESTS A LIKELY MATCH OF A TWITTER FOLLOWER BASED ON THE EMAIL / SCREEN NAME COMPARISON OF THE FAN vs OUR FOLLOWERS
IF SOMEONE (ie. a moderator) CONFIRMS OR DENIES THAT IT'S A GOOD MATCH THEY PUT A DATESTAMP IN `dismissed` */
CREATE TABLE `contentSuggestion` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`userId` int(11) DEFAULT NULL,
`fanId` int(11) DEFAULT NULL,
`twitterAccountId` int(11) DEFAULT NULL,
`contentType` varchar(50) DEFAULT NULL,
`contentString` varchar(255) DEFAULT NULL,
`added` datetime DEFAULT NULL,
`dismissed` datetime DEFAULT NULL,
`uniqueStr` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unstr` (`uniqueStr`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
What I'm trying to get is:
SELECT [fan columns]
WHERE fan screen name IS IN twitterfollowers
AND WHERE fan screen name IS NOT IN contentSuggestion (with a datestamp in dismissed)
My attempts so far:
~33 seconds
SELECT fans.id, tf.screenName as col1, tf.twitterId as col2 FROM fans
LEFT JOIN twitterFollowers tf ON tf.screenName = fans.emailUsername
LEFT JOIN contentSuggestion cs ON cs.contentString = tf.screenName WHERE dismissed IS NULL
GROUP BY(fans.id) HAVING col1 != ''
~14 seconds
SELECT id, emailUsername FROM fans WHERE emailUsername IN(SELECT DISTINCT(screenName) FROM twitterFollowers) AND emailUsername NOT IN(SELECT DISTINCT(contentString) FROM contentSuggestion WHERE dismissed IS NULL) GROUP BY (fans.id);
9.53 seconds
SELECT fans.id, tf.screenName as col1, tf.twitterId as col2 FROM fans
LEFT JOIN twitterFollowers tf ON tf.screenName = fans.emailUsername WHERE tf.uniqueStr NOT IN(SELECT uniqueStr FROM contentSuggestion WHERE dismissed IS NULL)
I hope there is a better way. I've been struggling to really use JOINS outside of a single LEFT JOIN which has already helped me speed up other queries by a significant amount.
Thanks for any help you can give me.
I would go with a variation of the second method. Instead of IN, use EXISTS. Then add the correct indexes and remove the aggregation:
SELECT f.id, f.emailUsername
FROM fans f
WHERE EXISTS (SELECT 1
FROM twitterFollowers tf
WHERE f.emailUsername = tf.screenName
) AND
NOT EXISTS (SELECT 1
FROM contentSuggestion cs
WHERE f.emailUsername = cs.contentString AND
cs.dismissed IS NULL
) ;
Then be sure you have the following indexes: twitterFollowers(screenName) and contentSuggestion(contentString, dismissed).
Some notes:
When using IN, don't use SELECT DISTINCT. I'm not 100% sure that MySQL is always smart enough to ignore the DISTINCT in the subquery (it is redundant).
Historically, EXISTS was faster than IN in MySQL. The optimizer has improved in recent versions.
For performance, you need the correct indexes.
Then be sure you have the following indexes: twitterFollowers(screenName) and contentSuggestion(contentString, dismissed).
Assuming that fan.id is unique (a very reasonable assumption), you don't need the final group by.

How can I use joins to fetch mysql rows from one table based on rows that are found in another?

Sorry, I know the question title is really vague, but this might take a little to explain...
I am developing a custom forum with PHP and MySQL... I have tables for categories, boards, and threads. I also have a table for forum permissions. Rows are inserted into the forum permissions table based on what user group can do certain things in each forum. Permissions are read, reply, create, and moderate... just by example. So the columns in the table are like this:
board_id | group_id | read | reply | create | moderate ...
What I want to do is get a list of the 10 most recent threads from any board, but only from boards that the user can read(based on what group they are in and what boards that group can read). I hope I am being clear enough :)
I have the following query that gets boards that the user can read, so maybe that can help
$query_boards = "
SELECT *
FROM forum_boards
WHERE board_id IN (
SELECT board_id
FROM forum_permissions
WHERE group_id = ".$this->user['group']['id']."
AND `read` = 1
)
AND category_id = ".$category_id."
AND parent_id = 0
ORDER BY position";
The SQL:
CREATE TABLE IF NOT EXISTS `forum_boards` (
`board_id` int(15) NOT NULL AUTO_INCREMENT,
`parent_id` int(15) NOT NULL,
`category_id` int(15) NOT NULL,
`last_msg_id` int(15) NOT NULL,
`position` tinyint(1) NOT NULL,
`title` text NOT NULL,
`description` text NOT NULL,
`status` tinyint(1) NOT NULL,
`redirect_url` text NOT NULL,
PRIMARY KEY (`board_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `forum_permissions` (
`permission_id` int(15) NOT NULL AUTO_INCREMENT,
`board_id` int(15) NOT NULL,
`group_id` int(15) NOT NULL,
`read` tinyint(1) NOT NULL,
`start` tinyint(1) NOT NULL,
`reply` tinyint(1) NOT NULL,
`moderate` tinyint(1) NOT NULL,
PRIMARY KEY (`permission_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `forum_threads` (
`thread_id` int(15) NOT NULL AUTO_INCREMENT,
`board_id` int(15) NOT NULL,
`author_id` int(15) NOT NULL,
`updater_id` int(15) NOT NULL,
`first_msg_id` int(15) NOT NULL,
`last_msg_id` int(15) NOT NULL,
`title` text NOT NULL,
`date_posted` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`date_updated` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`views` int(15) NOT NULL,
`status` tinyint(1) NOT NULL,
`type` tinyint(1) NOT NULL COMMENT '0 normal, 1 sticky, 2 global.',
PRIMARY KEY (`thread_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Try this
"SELECT *
FROM forum_boards AS FB
JOIN forum_permissions FP ON FB.board_id=FP.board_id
WHERE FP.group_id=".$this->USER['group']['id'].
"AND FP.read=1
AND FB.category_id =". $category_id.
"AND FB.parent_id = 0
ORDER BY FB.position";

MySql Properly Join Complex Data/Tables

Abstract:
Every client is given a specific xml ad feed (publisher_feed table). Everytime there is a query or a click on that feed, it gets recorded (publisher_stats_raw table) (Each query/click will have multiple rows depending on the subid passed by the client (We can sum the clicks together)). The next day, we pull stats from an API to grab the previous days revenue numbers (rev_stats table) (Each revenue stat might have multiple rows depending on the country of the click (We can sum the revenue together)). Been having a hard time trying to link together these three tables to find the average RPC for each client for the previous day.
Table Structure:
CREATE TABLE `publisher_feed` (
`publisher_feed_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`alias` varchar(45) DEFAULT NULL,
`user_id` int(10) unsigned DEFAULT NULL,
`remote_feed_id` int(10) unsigned DEFAULT NULL,
`subid` varchar(255) DEFAULT '',
`requirement` enum('tq','tier2','ron','cpv','tos1','tos2','tos3','pv1','pv2','pv3','ar','ht') DEFAULT NULL,
`status` enum('enabled','disabled') DEFAULT 'enabled',
`tq` decimal(4,2) DEFAULT '0.00',
`clicklimit` int(11) DEFAULT '0',
`prev_rpc` decimal(20,10) DEFAULT '0.0000000000',
PRIMARY KEY (`publisher_feed_id`),
UNIQUE KEY `alias_UNIQUE` (`alias`),
KEY `publisher_feed_idx` (`remote_feed_id`),
KEY `publisher_feed_user` (`user_id`),
CONSTRAINT `publisher_feed_feed` FOREIGN KEY (`remote_feed_id`) REFERENCES `remote_feed` (`remote_feed_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `publisher_feed_user` FOREIGN KEY (`user_id`) REFERENCES `user` (`user_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=124 DEFAULT CHARSET=latin1$$
CREATE TABLE `publisher_stats_raw` (
`publisher_stats_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`unique_data` varchar(350) NOT NULL,
`publisher_feed_id` int(10) unsigned DEFAULT NULL,
`date` date DEFAULT NULL,
`subid` varchar(255) DEFAULT NULL,
`queries` int(10) unsigned DEFAULT '0',
`impressions` int(10) unsigned DEFAULT '0',
`clicks` int(10) unsigned DEFAULT '0',
`filtered` int(10) unsigned DEFAULT '0',
`revenue` decimal(20,10) unsigned DEFAULT '0.0000000000',
PRIMARY KEY (`publisher_stats_id`),
UNIQUE KEY `unique_data_UNIQUE` (`unique_data`),
KEY `publisher_stats_raw_remote_feed_idx` (`publisher_feed_id`)
) ENGINE=InnoDB AUTO_INCREMENT=472 DEFAULT CHARSET=latin1$$
CREATE TABLE `rev_stats` (
`rev_stats_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`date` date DEFAULT NULL,
`remote_feed_id` int(10) unsigned DEFAULT NULL,
`typetag` varchar(255) DEFAULT NULL,
`subid` varchar(255) DEFAULT NULL,
`country` varchar(2) DEFAULT NULL,
`revenue` decimal(20,10) DEFAULT NULL,
`tq` decimal(4,2) DEFAULT NULL,
`finalized` int(11) DEFAULT '0',
PRIMARY KEY (`rev_stats_id`),
KEY `rev_stats_remote_feed_idx` (`remote_feed_id`),
CONSTRAINT `rev_stats_remote_feed` FOREIGN KEY (`remote_feed_id`) REFERENCES `remote_feed` (`remote_feed_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=58 DEFAULT CHARSET=latin1$$
Context:
Each remote_feed has a specific subid/typetag given to it. So we need to match up the both the remote_feed_id and the subid columsn from the publisher_feed table to the remote_feed_id and typetag columns in the revenue stats table.
My current, non working, implementation:
SELECT
pf.publisher_feed_id, psr.date, sum(clicks), sum(rs.revenue)
FROM
xml_network.publisher_feed pf
JOIN
xml_network.publisher_stats_raw psr
ON
psr.publisher_feed_id = pf.publisher_feed_id
JOIN
xml_network.rev_stats rs
ON
rs.remote_feed_id = pf.remote_feed_id
WHERE
pf.requirement = 'tq'
AND
pf.subid = rs.typetag
AND
psr.date <> date(curdate())
GROUP BY
psr.date
ORDER BY
psr.date DESC
LIMIT 1;
The above keeps pulling the wrong data out of the rev_stats table (pulls the sum of the correct stats, but repeats it over because of a join). Any help with how I would be able to properly pull the correct data would be greatly helpful ( I could use multiple queries and PHP to get the correct results, but what's the fun in that!)
Figured out a way to get this accomplished. Its def not a fast method by any means, needing 4 selects to get it done, but it works flawlessly =)
SELECT
pf.publisher_feed_id,
round(
(
SELECT
SUM(rs.revenue)
FROM
xml_network.rev_stats rs
WHERE
rs.remote_feed_id = pf.remote_feed_id
AND
rs.typetag = pf.subid
AND
rs.date = subdate(current_date, 1)
),10)as revenue,
(
SELECT
MAX(rs.tq)
FROM
xml_network.rev_stats rs
WHERE
rs.remote_feed_id = pf.remote_feed_id
AND
rs.typetag = pf.subid
AND
rs.date = subdate(current_date, 1)
) as tq,
(
SELECT
SUM(psr.clicks)-SUM(psr.filtered)
FROM
xml_network.publisher_stats_raw psr
WHERE
psr.publisher_feed_id = pf.publisher_feed_id
AND
psr.date = subdate(current_date, 1)
) as clicks
FROM
xml_network.publisher_feed pf
WHERE
pf.requirement = 'tq';

SQL select query taking long time

I am using the below SQL query but it takes more than 180 sec to execute. Is there a way to speed it up ? This SQL give me the pic_id and of all the females.
SELECT pic_id, small
FROM picture
WHERE hide =0
AND userhide =0
AND `fbid`
IN (
SELECT fbid
FROM user
WHERE gender = "female"
)
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200
The Explain SQL
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY picture index NULL PRIMARY 4 NULL 1700 Using where
2 DEPENDENT SUBQUERY user ALL NULL NULL NULL NULL 7496 Using where
--- Result of explain statement for Tim's sql answer --
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE picture ALL NULL NULL NULL NULL 41443 Using where; Using temporary; Using filesort
1 SIMPLE user ALL NULL NULL NULL NULL 7501 Using where; Using join buffer
-- Structure ---
CREATE TABLE `user` (
`sid` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) CHARACTER SET utf8 NOT NULL,
`first_name` varchar(255) CHARACTER SET utf8 NOT NULL,
`username` varchar(255) CHARACTER SET utf8 NOT NULL,
`birthday` date NOT NULL,
`location` varchar(255) CHARACTER SET utf8 NOT NULL,
`gender` varchar(6) CHARACTER SET utf8 NOT NULL,
`created` date NOT NULL,
`fbid` bigint(50) NOT NULL,
`token` varchar(255) CHARACTER SET utf8 NOT NULL,
`relationship_status` varchar(20) CHARACTER SET utf8 NOT NULL,
`smallest` varchar(255) CHARACTER SET utf8 NOT NULL,
`email` varchar(40) CHARACTER SET utf8 NOT NULL,
`ref` varchar(15) NOT NULL,
PRIMARY KEY (`sid`),
KEY `gender` (`gender`),
KEY `fbid` (`fbid`)
) ENGINE=MyISAM AUTO_INCREMENT=7595 DEFAULT CHARSET=latin
---- structure of picture table ---
CREATE TABLE `picture` (
`fbid` bigint(50) NOT NULL,
`pic_id` int(11) NOT NULL AUTO_INCREMENT,
`pic_location` varchar(255) NOT NULL,
`hide` int(1) NOT NULL,
`small` varchar(255) NOT NULL,
`userhide` int(1) NOT NULL,
`likes` int(10) NOT NULL,
`hot` int(1) NOT NULL,
PRIMARY KEY (`pic_id`),
UNIQUE KEY `pic_location` (`pic_location`),
UNIQUE KEY `small` (`small`),
KEY `fbid` (`fbid`),
KEY `hide` (`hide`),
KEY `userhide` (`userhide`)
) ENGINE=MyISAM AUTO_INCREMENT=42749 DEFAULT CHARSET=latin1
try something like this:
SELECT pic_id, small
FROM picture
INNER JOIN user ON ( picture.fbid = user.fbid and user.gender='female' )
WHERE hide =0
AND userhide =0
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200
I put gender in the join because a query will not return rows that don't have a match on an inner join.
You should also read this stack overflow topic
EDIT:
make sure you have indexed the following fields:
picture.fbid
user.fbid
user.gender
picture.hide
picture.userhide
Try using a join instead:
SELECT p.pic_id, p.small
FROM picture p
INNER JOIN fbid f USING ( fbid )
WHERE p.hide =0
AND p.userhide =0
AND f.gender = 'female'
ORDER BY `picture`.`pic_id` ASC
LIMIT 1500 , 200

mysql compare takes too much time

I'm Using 3 tables here listed below:
CREATE TABLE `user` (
`uid` int(10) NOT NULL AUTO_INCREMENT,
`kid` int(3) NOT NULL,
`Email` varchar(255) DEFAULT NULL,
`del` tinyint(1) DEFAULT '0',
PRIMARY KEY (`uid`),
KEY `kid` (`kid`),
KEY `email` (`Email`)
) ENGINE=MyISA
and
CREATE TABLE `blacklist_global` (
`bgid` int(10) NOT NULL AUTO_INCREMENT,
`email` varchar(255) NOT NULL,
`kid` int(3) DEFAULT NULL,
`stmp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`bgid`),
UNIQUE KEY `email` (`email`),
KEY `kid` (`kid`)
) ENGINE=MyISAM
and
CREATE TABLE `verteiler_user` (
`vuid` int(10) NOT NULL AUTO_INCREMENT,
`uid` int(3) NOT NULL,
`vid` int(3) NOT NULL,
`del` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`vuid`),
KEY `uid` (`uid`)
) ENGINE=MyISAM
every entry in table user is also in table verteiler_user. now i want to delete every entry from user where user.uid = verteiler_user.uid and verteiler_user.vid=XX and user.uid = XXX
so atm im doing it from php fetch all entrys from table user, all from blacklist_global.
the problem is, in the blacklist i could enter *#heloooo.de so i want to delete every email from the domain helooo.de but its very very slow.
is it possible to do it only in mysql? without php ? or any tip, of doing this faster ?
This might get what you need. Test it first before using it on production data.
DELETE FROM user WHERE uid = ? AND uid in (SELECT uid from verteiler_user WHERE vid = ?)
Edited based on comment:
DELETE FROM user WHERE uid IN (SELECT uid FROM blacklist_global WHERE email LIKE '%#somedomain.com')

Categories