I have an UPDATE query where I explicitely reference the database, but MySQL still complains with the message: ERROR 1046 (3D000): No database selected.
Other queries that are similar of structure, but use an INSERT work fine. Other queries that only perform SELECTs also run fine.
To repeat the problem in a test case, try running these queries:
create table test.object1 (
id_object1 int unsigned not null auto_increment,
total int,
weight int,
dt datetime,
primary key (id_object1)
) engine=InnoDB;
create table test.object2 (
id_object2 int unsigned not null auto_increment,
primary key (id_object2)
) engine=InnoDB;
create table test.score (
id_object1 int unsigned not null,
id_object2 int unsigned not null,
dt datetime,
score float,
primary key (id_object1, id_object2),
constraint fk_object1 foreign key (id_object1) references object1 (id_object1),
constraint fk_object2 foreign key (id_object2) references object2 (id_object2)
) engine=InnoDB;
insert into test.object1 (id_object1, total, weight, dt) values (1, 0, 0, '2012-01-01 00:00:00');
insert into test.object1 (id_object1, total, weight, dt) values (2, 0, 0, '2012-01-02 00:00:00');
insert into test.object2 (id_object2) values (1);
insert into test.score (id_object1, id_object2, dt, score) values (1, 1, '2012-01-03 00:00:00', 10);
insert into test.score (id_object1, id_object2, dt, score) values (2, 1, '2012-01-04 00:00:00', 8);
update test.object1 p
join (
select ur.id_object1, sum(ur.score * ur.weight) as total, count(*) as weight
from (
select lur.*
from (
select s.id_object1, s.id_object2, s.dt, s.score, 1 as weight
from test.score as s
join test.object1 as o1 using(id_object1)
where s.dt > o1.dt
order by s.id_object1, s.id_object2, s.dt desc
) as lur
group by lur.id_object2, lur.id_object1, date(lur.dt)
order by lur.id_object1, lur.id_object2
) as ur
group by ur.id_object1
) as r using(id_object1)
set
p.total = p.total + r.total,
p.weight = p.weight + r.weight,
p.dt = now();
Note: I'm running these queries from a PHP environment and I have NOT explicitely used mysql_select_db('test'), because I prefer not to and none of the other (many!) queries require it. I'm sure that using mysql_select_db would solve my issue, but I would like to know why exactly this particular query does not work.
For comparison sake: if you'd run this simpler query, also without using mysql_select_db, everything works fine:
update test.object1 set total=1, weight=1, dt=now() where id_object1=1;
I've searched to no avail. The only thing I found that came close, was this bug report: http://bugs.mysql.com/bug.php?id=28551 and especially that last (unanswered) message...
You have fields named incorrectly, but even if you correct them, this is a bug in MySQL that won't let you do it if you don't have default database.
update test.object1 p
join (
select ur.id_object1, sum(ur.score * ur.weight) as total, count(*) as weight
from (
select lur.*
from (
select s.id_object1, s.id_object2, s.dt, s.score, 1 as weight
from test.score as s
join test.object1 as o1
using (id_object1)
where s.dt > o1.dt
order by
s.id_object1, s.id_object2, s.dt desc
) as lur
group by
lur.id_object1, lur.id_object1, date(lur.dt)
order by
lur.id_object1, lur.id_object1
) as ur
group by ur.id_object1
) as r
USING (id_object1)
SET p.total = p.total + r.total,
p.weight = p.weight + r.weight,
p.dt = now();
The problem is specific to UPDATE with double-nested queries and no default database (SELECT or single-nested queries or default database work fine)
You have some wrong field names in the UPDATE statement -
What is s.object? Shouldn't it be s.id_object2?
What is lur.object1? Shouldn't it be lur.id_object1?
What is lur.object2? Shouldn't it be lur.id_object2?
What is ur.id_object at the end?
Fix all these issues and try to update again;-)
First time I ran this script I got that error. My output:
1 row inserted [0,184s]
1 row inserted [0,068s]
1 row inserted [0,066s]
1 row inserted [0,147s]
1 row inserted [0,060s]
Error (32,1): No database selected
When I set default database name the problem disappeared.
Remember that you cannot use foreign keys when the Engine is set to MyISAM. Not only does the table that you are creating a foreign key in need to be InnoDB, but the table you are getting the key from also needs to be InnoDB.
I was getting the same error as you and pulling my hair out for days before I thought of this. I went into each of my tables and made sure the Engines were set to InnoDB for each one, and now I have no issues setting up foreign keys.
Related
I want to insert a new dataset into a MySQL table tab with external data, but also with data from another table otherTab using the others' table primary key and another condition. However, it could be that the requested row simply does not exist (anymore) or the result set is empty due to a mismatch in the other supplied data. Then, the original INSERT should fail. All columns are forbidden to be NULL.
My first attempt was:
INSERT INTO tab (id, extid1, extid2, value)
SELECT 1,
(SELECT id FROM otherTab WHERE id = 12 AND data = 'TXT'),
(SELECT id FROM otherTab WHERE id = 34 AND data = 'JPG'),
1234
but the problem with it is that a returned empty result set is cast to the type of the column in tab, leading to a 0 as entry data.
The query shall be efficient and avoid unnecessary querying. This is how I achieve it with four subqueries:
INSERT INTO tab (id, extid1, extid2, value)
SELECT 1,
(SELECT id FROM otherTab WHERE id = 12 AND data = 'TXT'),
(SELECT id FROM otherTab WHERE id = 34 AND data = 'JPG'),
1234
WHERE EXISTS (SELECT id FROM otherTab WHERE id = 12 AND data = 'TXT')
AND EXISTS (SELECT id FROM otherTab WHERE id = 34 AND data = 'JPG')
I tried with other constructs, e.g. (SELECT IFNULL(SELECT id FROM otherTab WHERE id = 12 AND data = 'TXT', NULL)) to enforce NULL or even a string into the target column, but it also gets casted to a 0 or some value instead.
Here is the code for dbFiddle:
code
CREATE TABLE `tab` (
`id` int NOT NULL,
`seUuid4` binary(16) NOT NULL,
`rxUuid4` binary(16) NOT NULL,
`text` varchar(16)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
CREATE TABLE `otherTab` (
`uuid4` binary(16) NOT NULL,
`lgUuid4` binary(16) NOT NULL,
`data` varchar(16)
) ENGINE = InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
ALTER TABLE `otherTab`
ADD PRIMARY KEY(`uuid4`);
ALTER TABLE `tab`
ADD CONSTRAINT `tab_ibfk_1` FOREIGN KEY (`rxUuid4`) REFERENCES `otherTab` (`uuid4`) ON DELETE RESTRICT ON UPDATE RESTRICT,
ADD CONSTRAINT `tab_ibfk_2` FOREIGN KEY (`seUuid4`) REFERENCES `otherTab` (`uuid4`) ON DELETE RESTRICT ON UPDATE RESTRICT;
INSERT INTO `otherTab` (uuid4, lgUuid4, data) VALUES
(UNHEX("22224444aaaa49c782408b2fe8c4dee0"), UNHEX("00001234aaaa4444aaaa432187654321"), "JPG"),
(UNHEX("11113333aaaa49c782408b2fe8c4dee0"), UNHEX("12340000bbbb6666bbbb432187654321"), "TXT");
INSERT INTO tab (id, seUuid4, rxUuid4, text)
SELECT
1,
(SELECT uuid4 FROM otherTab WHERE lgUuid4 = UNHEX('00001234aaaa4444aaaa432187654321') AND data = 'JPK' LIMIT 0,1),
(SELECT uuid4 FROM otherTab WHERE lgUuid4 = UNHEX('12340000bbbb6666bbbb432187654321') AND data = 'TXT' LIMIT 0,1),
'some text'
This interestingly works exactly as expected: Note the JPK instead of JPG. I verified my code and the PDO prepared statement fires out exactly the same command, but it gets inserted as INSERT INTO tab (id, seUuid4, rxUuid4) VALUES (1, 0x00000000000000000000000000000000, 0x00000000000000000000000000000000, 'datatext'); while the SQL client and phpMyadmin deliver the expected cannot insert null error message.
I could not find anything in the PDO options. If it helps, I use PDO with emulated prepared statements, but also tried without with no change.
PS: I posted already at dba.stackexchange.com/posts/276868
You could use a subquery:
INSERT INTO tab (id, extid1, extid2, value)
SELECT
FROM (
SELECT
1 id,
(SELECT id FROM otherTab WHERE id = 12 AND data = 'TXT') extid1,
(SELECT id FROM otherTab WHERE id = 34 AND data = 'JPG') extid2
1234 value
) t
WHERE extid1 IS NOT NULL and extid2 IS NOT NULL
Or, probably better yet, you can CROSS JOIN the two subqueries:
INSERT INTO tab (id, extid1, extid2, value)
SELECT 1, t1.id, t2.id, 1234
FROM (SELECT id FROM otherTab WHERE id = 12 AND data = 'TXT') t1,
CROSS JOIN (SELECT id FROM otherTab WHERE id = 34 AND data = 'JPG') t2
Actually, since you are reurning the same value that you are filtering on, two exists subqueries are probably sufficient:
INSERT INTO tab (id, extid1, extid2, value)
SELECT t.*
FROM (SELECT 1 id, 12 extid1, 34 extid2, 1234 value) t
WHERE EXISTS (SELECT 1 FROM otherTab t1 WHERE t1.id = t.extid1 AND t1.data = 'TXT')
AND EXISTS (SELECT 1 FROM otherTab t1 WHERE t1.id = t.extid2 AND t1.data = 'JPG')
The requirement is to find the first available identifier where an identifier is an alphanumeric string, such as:
ABC10000
ABC10345
ABC88942
ABC90123
The database table has a structure such as:
id, user, identifier
Note that the alpha component ABC is consistent throughout and won't change. The numeric component should be between 10000 and 99999.
How best to tackle this? It does not seem like an overly complex problem - looking for simplest solution using either just MySQL or a combination of SQL and PHP. The current solution pulls each record from the database into an array and then loops from 10000 onwards, prepending ABC and checking availability, which seems like it could be improved significantly.
Edit: Original question was not clear enough in that a certain amount of identifiers have been assigned already, and I am looking to fill in the gaps. From the short list I provided, the next available would be ABC10001. Eventually, however, it would be ABC10346 and then ABC88943 and so on
Edit: Sorry for a poorly structured question. To avoid any further confusion, here is the actual table structure:
CREATE TABLE IF NOT EXISTS `User_Loc` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user` int(11) DEFAULT NULL,
`value` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UNIQ_64FB41DA17323CBC` (`user`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=4028 ;
You have to self join the table and look for the first NULL value in the joined table
SELECT CONCAT('ABC', SUBSTRING(t1.value, 4)+1) AS next_value
FROM test t1
LEFT JOIN test t2 on SUBSTRING(t1.value, 4)+1 = SUBSTRING(t2.value, 4)
WHERE ISNULL(t2.value)
ORDER BY t1.value ASC
LIMIT 1
http://sqlfiddle.com/#!2/d69105/22
edit
With the comment about some 'specialities' at ncatnow. There are slight adjusments to make with the help of subselects for ridding the 'ABC' and UNION for having a default value
SELECT
CONCAT('ABC', t1.value+1) AS next_value
FROM
((SELECT '09999' AS value) UNION (SELECT SUBSTRING(value, 4) AS value FROM test)) t1
LEFT JOIN
((SELECT '09999' AS value) UNION (SELECT SUBSTRING(value, 4) AS value FROM test)) t2
ON t1.value+1 = t2.value
WHERE
ISNULL(t2.value)
AND t1.value >= '09999'
ORDER BY t1.value ASC
LIMIT 1
http://sqlfiddle.com/#!2/28acf6/50
Similar to the above reply by #HerrSerker, but this will cope with existing identifiers which have the numeric part starting with a zero.
SELECT CONCAT('ABC',SUBSTRING(CONCAT('00000', CAST((CAST(SUBSTRING(a.identifier, 4) AS SIGNED) + 1) AS CHAR)), -5)) AS NextVal
FROM SomeTable a
LEFT OUTER JOIN SomeTable b
ON b.identifier = CONCAT('ABC',SUBSTRING(CONCAT('00000', CAST((CAST(SUBSTRING(a.identifier, 4) AS SIGNED) + 1) AS CHAR)), -5))
WHERE b.identifier IS NULL
ORDER BY NextVal
LIMIT 1
what comes to my mind is one table with all indentifiers and use this sql
SELECT identifier FROM allIdentifiersTable WHERE identifier NOT IN (SELECT identifier FROM yourTable) LIMIT 1
Reconsidering this from your edit, you should go the PHP route and add another table or other means to store the last filled id:
$identifier = 0;
$i = mysql_query("SELECT identifier FROM last_identifier");
if ($row = mysql_fetch_assoc($i)) $identifier = $row["identifier"];
if ($identifier < 10000) $identifier = 10000;
do {
$identifier += 1;
$result = mysql_query("
INSERT IGNORE INTO table (id, user, identifier)
VALUES ('[...]', '[...]',
'ABC" . str_pad($identifier, 5, "0", STR_PAD_LEFT) . "'
)");
if (mysql_affected_rows($result) < 1) continue;
} while (false);
mysql_query("UPDATE last_identifier SET identifier = '$identifier'");
Of course, you need to add a UNIQUE index on the identifier field.
I have a table that looks like this:
CREATE TABLE `relations` (
`idA` VARCHAR(20),
`idB` VARCHAR(20).
PRIMARY KEY (idA,idB)
)
TYPE=MyISAM;
and it basically just maps two ids together from another table that looks like this:
CREATE TABLE `scores` (
`id` VARCHAR(20) PRIMARY KEY,
`score` INT(10) UNSIGNED NOT NULL DEFAULT ‘0’,
`friendsids` VARCHAR(1000)
)
TYPE=MyISAM;
So - if I want to add something to the relations table, I query
INSERT IGNORE INTO relations VALUES ('$idA', '$idB')
So the problem - it sometimes creates entries that have the same info but swapped between idA and idB. For example, if one entry is idA = 1, idB = 2 - I dont' want an entry that looks like idA = 2, idB = 1;
I tried:
INSERT IGNORE INTO relations VALUES ('$idA', '$idB') WHERE NOT EXISTS (SELECT * WHERE idA IS '$idB' AND idB IS '$idA');
It gives me a syntax error which I somehow can't figure out:
Query failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'IGNORE INTO relations VALUES ('saubua', 'deppata') WHERE NOT EXISTS (SELECT * WH' at line 1
Is there a chance I'm totally on the wrong way with this? Is there a simpler way?
can you try
INSERT INTO relations
SELECT '$idA','$idB'
FROM dual
WHERE NOT EXISTS (SELECT idA
FROM relations
WHERE (idA='$idA' OR idB='$idB')
OR (idA='$idB' OR idB='$idA')
)
so - I took Akhils code and mofified it, so not it works:
INSERT IGNORE INTO relations SELECT '$idA','$idB' FROM dual WHERE NOT EXISTS (SELECT idA FROM relations WHERE (idA='$idB' AND idB='$idA') )
thanks everyone!! :)
I know this is quite complicated, but I sincerely hope someone will check this out.
I made short version (to better understand the problem) and full version (with original SQL)
Short version:
[TABLE A] [TABLE B]
|1|a|b| |1|x
|2|c|d| |1|y
|3| | | |2|z
|5| | | |2|v
|4|w
How can I make MySQL query to get rows like that:
1|a|b|x|y
2|c|d|z|v
2 columns from A and 2 rows from B as columns, only with keys 1 and 2, no empty results
Subquery?
Full version:
I tried to get from Prestashop db in one row:
product id
ean13 code
upc code
feature with id 24
feature with id 25
It's easy to get id_product, ean13 and upc, as it's one row in ps_product table. To get features I used subqueries (JOIN didn't work out).
So, I selected id_product, ean13, upc, (subquery1) as code1, (subquery2) as code2.
Then I needed to throw out empty rows. But couldn't just put code1 or code2 in WHERE.
To make it work I had to put everything in subquery.
This code WORKS, but it is terribly ugly and I bet this should be done differently.
How can I make it BETTER?
SELECT * FROM(
SELECT
p.id_product as idp, p.ean13 as ean13, p.upc as upc, (
SELECT
fvl.value
FROM
`ps_feature_product` fp
LEFT JOIN
`ps_feature_value_lang` fvl ON (fp.id_feature_value = fvl.id_feature_value)
WHERE fp.id_feature = 24 AND fp.id_product = idp
) AS code1, (
SELECT
fvl.value
FROM
`ps_feature_product` fp
LEFT JOIN
`ps_feature_value_lang` fvl ON (fp.id_feature_value = fvl.id_feature_value)
WHERE fp.id_feature = 25 AND fp.id_product = idp
) AS code2,
m.name
FROM
`ps_product` p
LEFT JOIN
`ps_manufacturer` m ON (p.id_manufacturer = m.id_manufacturer)
) mainq
WHERE
ean13 != '' OR upc != '' OR code1 IS NOT NULL OR code2 IS NOT NULL
create table tablea
( id int,
col1 varchar(1),
col2 varchar(1));
create table tableb
( id int,
feature int,
cola varchar(1));
insert into tablea (id, col1, col2)
select 1,'a','b' union
select 2,'c','d' union
select 3,null,null union
select 5,null,null;
insert into tableb (id, feature, cola)
select 1,24,'x' union
select 1,25,'y' union
select 2,24,'z' union
select 2,25,'v' union
select 4,24,'w';
select a.id, a.col1, a.col2, b1.cola b1a, b2.cola b2a
from tablea a
inner join tableb b1 on (b1.id = a.id and b1.feature = 24)
inner join tableb b2 on (b2.id = a.id and b2.feature = 25);
SQLFiddle here.
What you want to do is called a Pivot Query. MySQL has no native support for pivot queries, though other RDBMSen do.
You can simulate a pivot query with derived columns, but you must specify each derived column. That is, it is impossible in MySQL itself to have the number of columns match rows of another table. This has to be known ahead of time.
It would be much easier to query the results as rows and then use PHP to do the aggregation into columns. For example:
while ($row = $result->fetch()) {
if (!isset($table[$row->id])) {
$table[$row->id] = array();
}
$table[$row->id][] = $row->feature;
This is not a simple question because it's not a standard query, by the way if you can make use of views you can do the following procedure. Assuming you're starting from this tables:
CREATE TABLE `A` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`firstA` char(1) NOT NULL DEFAULT '',
`secondA` char(1) NOT NULL DEFAULT '',
PRIMARY KEY (`id`)
);
CREATE TABLE `B` (
`id` int(11) unsigned NOT NULL,
`firstB` char(1) NOT NULL DEFAULT ''
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO `A` (`id`, `firstA`, `secondA`)
VALUES (1, 'a', 'b'), (2, 'c', 'd');
INSERT INTO `B` (`id`, `firstB`)
VALUES (1, 'x'), (1, 'y'), (2, 'z'), (2, 'v'), (4, 'w');
First create a view that joins the two tables:
create or replace view C_join as
select A.firstA, A.secondA, B.firstB
from A
join B on B.id=A.id;
Create the view that groups the rows in table B:
create or replace view d_group_concat as
select firstA, secondA, group_concat(firstB) groupconcat
from c_join
group by firstA, secondA
Create the view that does what you need:
create or replace view e_result as
select firstA, secondA, SUBSTRING_INDEX(groupconcat,',',1) firstB, SUBSTRING_INDEX(SUBSTRING_INDEX(groupconcat,',',2),',',-1) secondB
from d_group_concat
And that's all. Hope this helps you.
If you can't create views, this could be the query:
select firstA, secondA, SUBSTRING_INDEX(groupconcat,',',1) firstB, SUBSTRING_INDEX(SUBSTRING_INDEX(groupconcat,',',2),',',-1) secondB
from (
select firstA, secondA, group_concat(firstB) groupconcat
from (
select A.firstA, A.secondA, B.firstB
from A
join B on B.id=A.id
) c_join
group by firstA, secondA
) d_group_concat
Big thanks to everyone for the answers. James's answer was first, simplest and works perfectly in my case. The query runs several times faster than mine, with subqueries. Thanks, James!
Just a few words why I needed that:
It's a part of integration component for Prestashop and wholesale exchange platform. There are 4 product code systems that wholesalers use on the platform (ean13, upc and 2 other systems). Those 2 other product codes are added as product feature in Prestashop. There are thousands of products on the shop and hundreds of thousands of products on the platform. Which is why speed is crucial.
Here is the code for full version of my question. Maybe someone will find this helpful.
Query to get Prestashop product codes and certain features in one row:
SELECT
p.id_product, p.ean13, p.upc, fvl1.value as code1, fvl2.value as code2
FROM `ps_product` p
LEFT JOIN
`ps_feature_product` fp1 ON (p.id_product = fp1.id_product and fp1.id_feature = 24)
LEFT JOIN
`ps_feature_value_lang` fvl1 ON (fvl1.id_feature_value = fp1.id_feature_value)
LEFT JOIN
`ps_feature_product` fp2 ON (p.id_product = fp2.id_product and fp2.id_feature = 25)
LEFT JOIN
`ps_feature_value_lang` fvl2 ON (fvl2.id_feature_value = fp2.id_feature_value)
WHERE
ean13 != '' OR upc != '' OR fvl1.value IS NOT NULL OR fvl2.value IS NOT NULL;
This post is taking a substantial amount of time to type because I'm trying to be as clear as possible, so please bear with me if it is still unclear.
Basically, what I have are a table of posts in the database which users can add privacy settings to.
ID | owner_id | post | other_info | privacy_level (int value)
From there, users can add their privacy details, allowing it to be viewable by all [privacy_level = 0), friends (privacy_level = 1), no one (privacy_level = 3), or specific people or filters (privacy_level = 4). For privacy levels specifying specific people (4), the query will reference the table "post_privacy_includes_for" in a subquery to see if the user (or a filter the user belongs to) exists in a row in the table.
ID | post_id | user_id | list_id
Also, the user has the ability to prevent some people from viewing their post in within a larger group by excluding them (e.g., Having it set for everyone to view but hiding it from a stalker user). For this, another reference table is added, "post_privacy_exclude_from" - it looks identical to the setup as "post_privacy_includes_for".
My problem is that this does not scale. At all. At the moment, there are about 1-2 million posts, the majority of them set to be viewable by everyone. For each post on the page it must check to see if there is a row that is excluding the post from being shown to the user - this moves really slow on a page that can be filled with 100-200 posts. It can take up to 2-4 seconds, especially when additional constraints are added to the query.
This also creates extremely large and complex queries that are just... awkward.
SELECT t.*
FROM posts t
WHERE ( (t.privacy_level = 3
AND t.owner_id = ?)
OR (t.privacy_level = 4
AND EXISTS
( SELECT i.id
FROM PostPrivacyIncludeFor i
WHERE i.user_id = ?
AND i.thought_id = t.id)
OR t.privacy_level = 4
AND t.owner_id = ?)
OR (t.privacy_level = 4
AND EXISTS
(SELECT i2.id
FROM PostPrivacyIncludeFor i2
WHERE i2.thought_id = t.id
AND EXISTS
(SELECT r.id
FROM FriendFilterIds r
WHERE r.list_id = i2.list_id
AND r.friend_id = ?))
OR t.privacy_level = 4
AND t.owner_id = ?)
OR (t.privacy_level = 1
AND EXISTS
(SELECT G.id
FROM Following G
WHERE follower_id = t.owner_id
AND following_id = ?
AND friend = 1)
OR t.privacy_level = 1
AND t.owner_id = ?)
OR (NOT EXISTS
(SELECT e.id
FROM PostPrivacyExcludeFrom e
WHERE e.thought_id = t.id
AND e.user_id = ?
AND NOT EXISTS
(SELECT e2.id
FROM PostPrivacyExcludeFrom e2
WHERE e2.thought_id = t.id
AND EXISTS
(SELECT l.id
FROM FriendFilterIds l
WHERE l.list_id = e2.list_id
AND l.friend_id = ?)))
AND t.privacy_level IN (0, 1, 4))
AND t.owner_id = ?
ORDER BY t.created_at LIMIT 100
(mock up query, similar to the query I use now in Doctrine ORM. It's a mess, but you get what I am saying.)
I guess my question is, how would you approach this situation to optimize it? Is there a better way to set up my database? I'm willing to completely scrap the method I have currently built up, but I wouldn't know what to move onto.
Thanks guys.
Updated: Fix the query to reflect the values I defined for privacy level above (I forgot to update it because I simplified the values)
Your query is too long to give a definitive solution for, but the approach I would follow is to simply the data lookups by converting the sub-queries into joins, and then build the logic into the where clause and column list of the select statement:
select t.*, i.*, r.*, G.*, e.* from posts t
left join PostPrivacyIncludeFor i on i.user_id = ? and i.thought_id = t.id
left join FriendFilterIds r on r.list_id = i.list_id and r.friend_id = ?
left join Following G on follower_id = t.owner_id and G.following_id = ? and G.friend=1
left join PostPrivacyExcludeFrom e on e.thought_id = t.id and e.user_id = ?
(This might need expanding: I couldn't follow the logic of the final clause.)
If you can get the simple select working fast AND including all the information needed, then all you need to do is build up the logic in the select list and where clause.
Had a quick stab at simplifying this without re-working your original design too much.
Using this solution your web page can now simply call the following stored procedure to get a list of filtered posts for a given user within a specified period.
call list_user_filtered_posts( <user_id>, <day_interval> );
The whole script can be found here : http://pastie.org/1212812
I haven't fully tested all of this and you may find this solution isn't performant enough for your needs but it may help you in fine tuning/modifying your existing design.
Tables
Dropped your post_privacy_exclude_from table and added a user_stalkers table which works pretty much like the inverse of user_friends. Kept the original post_privacy_includes_for table as per your design as this allows a user restrict a specific post to a subset of people.
drop table if exists users;
create table users
(
user_id int unsigned not null auto_increment primary key,
username varbinary(32) unique not null
)
engine=innodb;
drop table if exists user_friends;
create table user_friends
(
user_id int unsigned not null,
friend_user_id int unsigned not null,
primary key (user_id, friend_user_id)
)
engine=innodb;
drop table if exists user_stalkers;
create table user_stalkers
(
user_id int unsigned not null,
stalker_user_id int unsigned not null,
primary key (user_id, stalker_user_id)
)
engine=innodb;
drop table if exists posts;
create table posts
(
post_id int unsigned not null auto_increment primary key,
user_id int unsigned not null,
privacy_level tinyint unsigned not null default 0,
post_date datetime not null,
key user_idx(user_id),
key post_date_user_idx(post_date, user_id)
)
engine=innodb;
drop table if exists post_privacy_includes_for;
create table post_privacy_includes_for
(
post_id int unsigned not null,
user_id int unsigned not null,
primary key (post_id, user_id)
)
engine=innodb;
Stored Procedures
The stored procedure is relatively simple - it initially selects ALL posts within the specified period and then filters out posts as per your original requirements. I have not performance tested this sproc with large volumes but as the initial selection is relatively small it should be performant enough as well as simplifying your application/middle tier code.
drop procedure if exists list_user_filtered_posts;
delimiter #
create procedure list_user_filtered_posts
(
in p_user_id int unsigned,
in p_day_interval tinyint unsigned
)
proc_main:begin
drop temporary table if exists tmp_posts;
drop temporary table if exists tmp_priv_posts;
-- select ALL posts in the required date range (or whatever selection criteria you require)
create temporary table tmp_posts engine=memory
select
p.post_id, p.user_id, p.privacy_level, 0 as deleted
from
posts p
where
p.post_date between now() - interval p_day_interval day and now()
order by
p.user_id;
-- purge stalker posts (0,1,3,4)
update tmp_posts
inner join user_stalkers us on us.user_id = tmp_posts.user_id and us.stalker_user_id = p_user_id
set
tmp_posts.deleted = 1
where
tmp_posts.user_id != p_user_id;
-- purge other users private posts (3)
update tmp_posts set deleted = 1 where user_id != p_user_id and privacy_level = 3;
-- purge friend only posts (1) i.e where p_user_id is not a friend of the poster
/*
requires another temp table due to mysql temp table problem/bug
http://dev.mysql.com/doc/refman/5.0/en/temporary-table-problems.html
*/
-- the private posts (1) this user can see
create temporary table tmp_priv_posts engine=memory
select
tp.post_id
from
tmp_posts tp
inner join user_friends uf on uf.user_id = tp.user_id and uf.friend_user_id = p_user_id
where
tp.user_id != p_user_id and tp.privacy_level = 1;
-- remove private posts this user cant see
update tmp_posts
left outer join tmp_priv_posts tpp on tmp_posts.post_id = tpp.post_id
set
tmp_posts.deleted = 1
where
tpp.post_id is null and tmp_posts.privacy_level = 1;
-- purge filtered (4)
truncate table tmp_priv_posts; -- reuse tmp table
insert into tmp_priv_posts
select
tp.post_id
from
tmp_posts tp
inner join post_privacy_includes_for ppif on tp.post_id = ppif.post_id and ppif.user_id = p_user_id
where
tp.user_id != p_user_id and tp.privacy_level = 4;
-- remove private posts this user cant see
update tmp_posts
left outer join tmp_priv_posts tpp on tmp_posts.post_id = tpp.post_id
set
tmp_posts.deleted = 1
where
tpp.post_id is null and tmp_posts.privacy_level = 4;
drop temporary table if exists tmp_priv_posts;
-- output filtered posts (display ALL of these on web page)
select
p.*
from
posts p
inner join tmp_posts tp on p.post_id = tp.post_id
where
tp.deleted = 0
order by
p.post_id desc;
-- clean up
drop temporary table if exists tmp_posts;
end proc_main #
delimiter ;
Test Data
Some basic test data.
insert into users (username) values ('f00'),('bar'),('alpha'),('beta'),('gamma'),('omega');
insert into user_friends values
(1,2),(1,3),(1,5),
(2,1),(2,3),(2,4),
(3,1),(3,2),
(4,5),
(5,1),(5,4);
insert into user_stalkers values (4,1);
insert into posts (user_id, privacy_level, post_date) values
-- public (0)
(1,0,now() - interval 8 day),
(1,0,now() - interval 8 day),
(2,0,now() - interval 7 day),
(2,0,now() - interval 7 day),
(3,0,now() - interval 6 day),
(4,0,now() - interval 6 day),
(5,0,now() - interval 5 day),
-- friends only (1)
(1,1,now() - interval 5 day),
(2,1,now() - interval 4 day),
(4,1,now() - interval 4 day),
(5,1,now() - interval 3 day),
-- private (3)
(1,3,now() - interval 3 day),
(2,3,now() - interval 2 day),
(4,3,now() - interval 2 day),
-- filtered (4)
(1,4,now() - interval 1 day),
(4,4,now() - interval 1 day),
(5,4,now());
insert into post_privacy_includes_for values (15,4), (16,1), (17,6);
Testing
As I mentioned before I've not fully tested this but on the surface it seems to be working.
select * from posts;
call list_user_filtered_posts(1,14);
call list_user_filtered_posts(6,14);
call list_user_filtered_posts(1,7);
call list_user_filtered_posts(6,7);
Hope you find some of this of use.