I want to allow my users to create blacklists of content sources (users/categories/words/?). They shouldn't be able to see any content from these sources.
For example: If user A blacklisted user B and then user B uploaded a picture, then user A request to see the gallery he won't see picture from B, but he'll be able to see pictures from user C, D, ...
The problem occurs when one user built a big blacklist (e.g. 100 sources). Then the SQL queries will be really long and complex ("...and author != 'B' and category != 'C'...") what will eventually kill my server.
What are the other ways to handle this problem?
It sounds to me like you're using dynamic SQL to build this query. You should have the blacklist stored in a table related by UserId, then you can write a stored procedure that uses NOT IN or NOT EXISTS to build the final resultset.
First of all, create indices on every column that can be ignored. This way, your match criteria will be found a little bit faster.
Then you could create an in-between table that collects relations between a user and the content he has blacklisted.
Maybe something like this:
userId | blacklistType | blacklistId
1 | user | 2
1 | category | 12
1 | word | 4
Now if you want all categories for user 1, you could make the query
SELECT *
FROM categories
WHERE NOT EXISTS (
SELECT userId
FROM blacklist
WHERE userId = 1
AND blacklist.blacklistType = 'category'
AND categories.id = blacklist.blacklistId
)
(I'm not quite sure about the syntax here, but you should get the idea, I hope)
The query can look complex but that doesn't really matter. Make sure you index these columns and preferably use numbers for the categories, authors, etc.
So then you'll have a query like
SELECT * FROM .... WHERE author_id NOT IN (1,12,567,6788) AND category_id NOT IN (6654,23245,89795)
Related
I have the following table structure:
----------------------------------
ID | Acces | Group | User | Gate |
----------------------------------
1 | 1 | TR | tsv | TL-23|
----------------------------------
And I have a page with 3 functions:
Select group to see all gates where selected group has access.
Select gate to see all groups which have access to selected gate.
Select group to see all users that belong to selected group.
So basically:
SELECT Gate WHERE Group = TR
SELECT Group WHERE Gate = TL-23
SELECT User WHERE Group = TR
What I am trying to achieve is: The user should be able to run the all three queries in any order without the results of the former queries dissappearing.
Now, I know multi-threading is no longer possible in PHP, but there must be a way to temporarily save the results of a specific query until the same query is made again.
Any help and suggestions would be appreciated.
Firstly, PHP has never done MT out of the box, but can (still) with the use of pcntl extensions.
That said, that isn't the sauce you seek.
If you simply want the user on the front-end to interact 3 separate times, without having to hit the DB once the first time, twice the second (to redo both the first and the new query) you may benifit from caching the results of each call in the user's session.
If you actually want to make the 3 queries in a relational way at one exact time, try JOINs.
If you simply want to make all 3 separate queries at (what is essentially) the same time, look into TRANSACTIONs.
Hope that helps.
I've recently been working on normalizing and restructuring my database to make it more effective in the long run. Currently I have around 500 records, and obviously I don't want to lose the users data.
I assume SQL through phpmyadmin is the easiest way to do this?
So let me give you guys an example
In my old table I would have something like this
records //this table has misc fields, but they are unimportant right now
id | unit |
1 | g |
With my new one, I have it split apart 3 different tables.
records
id
1
units
id | unit
1 | g
record_units
id | record_id | unit_id
1 | 1 | 1
Just to be clear, I am not adding anything into the units table. The table is there as a reference for which id to store in the record_units table
As you can see it's pretty simple. What moved in the second table is that I started using an index table to hold my units, since they would be repeated quite often. I then store that unit id, and the pairing record id in the record_units table, so I can later retrieve the fields.
I am not incredibly experienced with SQL, though i'd say I know average. I know this operation would be quite simple to do with my cakephp setup, because all my associations are already setup, but I can't do that.
If I understand correctly you want to copy related records from your old table to the new tables, in which case you can use something like this
UPDATE units u
INNER JOIN records r
ON u.id=r.id
SET u.unit = r.unit
This will copy the unit type from your old table to the matching id in the new units table and then you can do something similare on your 3rd table
Description:
I am building a rating system with mysql/php. I am confused as to how I would set up the database.
Here is my article setup:
Article table:
id | user_id | title | body | date_posted
This is my assumed rating table:
Rating table:
id | article_id | score | ? user_id ?
Problem:
I don't know if I should place the user_id in the rating table. My plan is to use a query like this:
SELECT ... WHERE user_id = 1 AND article_id = 10
But I know that it's redundant data as it stores the user_id twice. Should I figure out a JOIN on the tables or is the structure good as is?
It depends. I'm assuming that the articles are unique to individual users? In that case, I could retain the user_id in your rating table and then just alter your query to:
SELECT ... WHERE article_id = 10
or
SELECT ... WHERE user_id = 1
Depending on what info you're trying to pull.
You're not "storing the user_id twice" so much as using the user_id to link the article to unique data associated to the user in another table. You're taking the right approach, except in your query.
I don't see anything wrong with this approach. The user id being stored twice is not particularly relevant since one is regarding a rating entry and the other, i assume, is related to the article owner.
The benefit of this way is you can prevent multiple scores being recorded for each user by making article_id and user_id unique and use replace into to manage scoring.
There are many things to elaborate on this depending on whether or not this rating system needs to be intelligent to prevent gaming, etc. How large the user base is, etc.
I bet for any normal person, this setup would not be detrimental to even a relatively large scale system.
... semi irrelevant:
Just FYI, depending on the importance and gaming aspects of this score, you could use STDDEV() to fetch an average factoring the standard deviation on the score column...
SELECT STDDEV(`score`) FROM `rating` WHERE `article_id` = {article_id}
That would factor outliers supposing you cared whether or not it looked like people were ganging up on a particular article to shoot it down or praise it without valid cause.
you should not, due to 3rd normal form, you need to keep the independence.
"The third normal form (3NF) is a normal form used in database normalization. 3NF was originally defined by E.F. Codd in 1971.[1] Codd's definition states that a table is in 3NF if and only if both of the following conditions hold:
The relation R (table) is in second normal form (2NF)
Every non-prime attribute of R is non-transitively dependent (i.e. directly dependent) on every superkey of R."
Source here: http://en.wikipedia.org/wiki/Third_normal_form
First normal Form: http://en.wikipedia.org/wiki/First_normal_form
Second normal Form: http://en.wikipedia.org/wiki/Second_normal_form
you should take a look to normalization and E/R model it will help you a lot.
normalization in wikipedia: http://en.wikipedia.org/wiki/Database_normalization
i don't even know if calling it serialized column is right, but i'm going to explain myself, for example, i have a table for users, i want to store the users phone numbers(cellphone, home, office, etc), so, i was thinkin' to make a column for each number type, but at the same time came to my head an idea, what if i save a json string in a single column, so, i will never have a column that probably will never be used and i can turn that string into a php array when reading the data from database, but i would like to hear the goods and bads of this practice, maybe it is just a bad idea, but first i want to know what other people have to say about
thanks
Short Answer, Multiple columns.
Long Answer:
For the love of all that is holy in the world please do not store mutiple data sets in a single text column
I am assuming you will have a table that will either be
+------------------------------+ +----------------------+
| User | cell | office | home | OR | User | JSON String |
+------------------------------+ +----------------------+
First I will say both these solutions are not the best solution but if you were to pick the from the two the first is best. There are a couple reasons mainly though the ability to modify and query specifically is really important. Think about the algrothim to modify the second option.
SELECT `JSON` FROM `table` WHERE `User` = ?
Then you have to do a search and replace in either your server side or client side language
Finally you have to reinsert the JSON string
This solution totals 2 queries and a search and replace algorithm. No Good!
Now think about the first solution.
SELECT * FROM `table` WHERE `User` = ?
Then you can do a simple JSON encode to send it down
To modify you only need one Query.
UPDATE `table` SET `cell` = ? WHERE `User` = ?
to update more than one its again a simple single query
UPDATE `table` SET `cell` = ?, `home` = ? WHERE `User` = ?
This is clearly better but it is not best
There is a third solution Say you want a user to be able to insert an infinite number of phone numbers.
Lets use a relation table for that so now you have two tables.
+-------------------------------------+
+---------+ | Phone |
| Users | +-------------------------------------+
+---------+ | user_name| phone_number | type |
| U_name | +-------------------------------------+
+---------+
Now you can query all the phone numbers of a user with something like this
Now you can query the table via a join
SELECT Users., phone. FROM Phone, Users WHERE phone.user_name = ? AND Users.U_name = ?
Inserts are just as easy and type checking is easy too.
Remember this is a simple example but SQL really provides a ton of power to your data-structure you should use it rather than avoiding it
I would only do this with non-essential data, for example, the user's favorite color, favorite type of marsupial (obviously 'non-essential' is for you to decide). The problem with doing this for essential data (phone number, username, email, first name, last name, etc) is that you limit yourself to what you can accomplish with the database. These include indexing fields, using ORDER BY clauses, or even searching for a specific piece of data. If later on you realize you need to perform any of these tasks it's going to be a major headache.
Your best best in this situation is using a relational table for 1 to many objects - ex UserPhoneNumbers. It would have 3 columns: user_id, phone_number, and type. The user_id lets you link the rows in this table to the appropriate User table row, the phone_number is self explanatory, and the type could be 'home', 'cell', 'office', etc. This lets you still perform the tasks I mentioned above, and it also has the added benefit of not wasting space on empty columns, as you only add rows to this table as you need to.
I don't know how familiar you are with MySQL, but if you haven't heard of database normalization and query JOINs, now is a good time to start reading up on them :)
Hope this helps.
If you work with json, there are more elegant ways than MySQL. Would recommend to use either another Database working better with json, like mongoDB or a wrapper for SQL like Persevere, http://www.persvr.org/Documentation (see "Perstore")
I'm not sure what the advantages of this approach would be. You say "so, i will never have a column that probably will never be used..." What I think you meant was (in your system) that sometimes a user may not have a value for each type of phone number available, and that being the case, why store records with empty columns?
Storing records with some empty columns is not necessarily bad. However, if you wanted to normalize your database, you could have a separate table for user_phonenumber, and create a 1:many relationship between user and user_phonenumber records. The user_phonenumber table would basically have four columns:
id (primary key)
userid (foreign key to user table)
type (e.g. cellphone, home, office, etc.)
value (the phone number)
Constraints would be that id is a primary key, userid is a foreign key for user.id, and type would be an enum (of all possible phone number types).
Here is the scenario 1.
I have a table called "items", inside the table has 2 columns, e. g. item_id and item_name.
I store my data in this way:
item_id | item_name
Ss001 | Shirt1
Sb002 | Shirt2
Tb001 | TShirt1
Tm002 | TShirt2
... etc, i store in this way:
first letter is the code for clothes, i.e S for shirt, T for tshirt
second letter is size, i.e s for small, m for medium and b for big
Lets say in my items table i got 10,000 items. I want to do fast retrieve, lets say I want to find a particular shirt, can I use:
Method1:
SELECT * from items WHERE item_id LIKE Sb99;
or should I do it like:
Method2:
SELECT * from items WHERE item_id LIKE S*;
*Store the result, then execute second search for the size, then third search for the id. Like the hash table concept.
What I want to achieve is, instead of search all the data, I want to minimize the search by search the clothes code first, follow by size code and then id code. Which one is better in term of speed in mysql. And which one is better in long run. I want to reduce the traffic and not to disturb the database so often.
Thanks guys for solving my first scenario. But another scenario comes in:
Scenario 2:
I am using PHP and MySQL. Continue from the preivous story. If my users table structure is like this:
user_id | username | items_collected
U0001 | Alex | Ss001;Tm002
U0002 | Daniel | Tb001;Sb002
U0003 | Michael | ...
U0004 | Thomas | ...
I store the items_collected in id form because one day each user can collect up to hundreds items, if I store as string, i.e. Shirt1, pants2, ..., it would required a very large amount of database spaces (imagine if we have 1000 users and some items name are very long).
Would it be easier to maintain if I store in id form?
And if lets say, I want to display the image, and the image's name is the item's name + jpg. How to do that? Is it something like this:
$result = Select items_collected from users where userid= $userid
Using php explode:
$itemsCollected = explode($result, ";");
After that, matching each item in the items table, so it would like:
shirt1, pants2 etc
Den using loop function, loop each value and add ".jpg" to display the image?
The first method will be faster - but IMO it's not the right way of doing it. I'm in agreement with tehvan about that.
I'd recommend keeping the item_id as is, but add two extra fields one for the code and one for the size, then you can do:
select * from items where item_code = 'S' and item_size = 'm'
With indexes the performance will be greatly increased, and you'll be able to easily match a range of sizes, or codes.
select * from items where item_code = 'S' and item_size IN ('m','s')
Migrate the db as follows:
alter table items add column item_code varchar(1) default '';
alter table items add column item_size varchar(1) default '';
update items set item_code = SUBSTRING(item_id, 1, 1);
update items set item_size = SUBSTRING(item_id, 2, 1);
The changes to the code should be equally simple to add. The long term benefit will be worth the effort.
For scenario 2 - that is not an efficient way of storing and retrieving data from a database. When used in this way the database is only acting as a storage engine, by encoding multiple data into fields you are precluding the relational part of the database from being useful.
What you should do in that circumstance is to have another table, call it 'items_collected'. The schema would be along the lines of
CREATE TABLE items_collected (
id int(11) NOT NULL auto_increment KEY,
userid int(11) NOT NULL,
item_code varchar(10) NOT NULL,
FOREIGN KEY (`userid`) REFERENCES `user`(`id`),
FOREIGN KEY (`itemcode`) REFERENCES `items`(`item_code`)
);
The foreign keys ensure that there is Referential integrity, it's essential to have referential integrity.
Then for the example you give you would have multiple records.
user_id | username | items_collected
U0001 | Alex | Ss001
U0001 | Alex | Tm002
U0002 | Daniel | Sb002
U0002 | Daniel | Tb001
U0003 | Michael | ...
U0004 | Thomas | ...
The first optimization would be splitting the id into three different fields:
one for type, one for size, one for the current id ending (whatever the ending means)
If you really want to keep the current structure, go for the result straight away (option 1).
If you want to speed up for results you should split up the column into multiple columns, one for each property.
Step 2 is to create an index for each column. Remember that mysql only uses one index per table per query. So if you really want speedy queries and your queries vary a lot with these properties, then you might want to create an index on (type,size,ending), (type,ending,size) etc.
For example a query with
select * from items where type = s and size = s and ending = 001
Can benefit from the index (type,size,ending) but:
select * from items where size = s and ending = 001
Can not, because the index will only be used in order, so it needs type, then size, then ending. This is why you might want multiple indexes if you really want fast searches.
One other note, generally it is not a good idea to use * in queries, but to select only the columns you need.
You need to have three columns for the model, size and id, and index them this way:
CREATE INDEX ix_1 ON (model, size, id)
CREATE INDEX ix_2 ON (size, id)
CREATE INDEX ix_3 ON (id, model)
Then you'll be able to search efficiently on any subset of the parameters:
model-size-id, model-size and model queries will use ix_1;
size-id and size queries will use ix_2;
model-id and id queries will use ix_3
Index on your column as it is now is equivalent to ix_1, and you can use this index to efficiently search on the appropriate conditions (model-size-id, model-size and model).
Actually, there is a certain access path called INDEX SKIN SCAN that may be used to search on non-first columns of a composite index, but MySQL does not support it AFAIK.
If you need to stick to your current design, you need to index the field and use queries like:
WHERE item_id LIKE #model || '%'
WHERE item_id LIKE #model || #size || '%'
WHERE item_id = #model || #size || #id
All these queries will use the index if any.
There is not need to put in into multiple queries.
I'm comfortable that you've designed your item_id to be searchable with a "Starts with" test. Indexes will solve that quickly for you.
I don't know MySQL, but in MSSQL having an index on a "Size" column that only has choices of S, M, L most probably won't achieve anything, the index won't be used because the values it contains are not sufficiently selective - i.e. its quicker to just go through all the data rather than "Find the first S entry in the index, now retrieve the data page for that row ..."
The exception is where the query is covered by the index - i.e. several parts of the WHERE clause (and indeed, all of them and also the SELECT columns) are included in the index. In this instance, however, the first field in the index (in MSSQL) needs to be selective. So put the column with the most distinct values first in the index.
Having said that if your application has a picklist for Size, Colour, etc. you should have those data attributes in separate columns in the record - and separate tables with lists of all the available Colours and Sizes, and then you can validate that the Colour / Size given to a Product is actually defined in the Colour / Size tables. Cuts down the Garbage-in / Garbage-out problem!
Your item_selected needs to be in a separate table so that it is "normalised". Don't store a delimited list in a single column, store it using individual rows in a separate table
Thus your USERS table will contain user_id & username
Your, new, items_collected table will contains user_id & item_id (and possibly also Date Purchased or Invoice Number)
You can then say "What did Alex buy" (your design has that) and also "Who bought Ss001" (which, in your design, would require ploughing through all the rows in your USERS table and splitting out the items_collected to find which ones contained Ss001 [1])
[1] Note that using LIKE wouldn't really be safe for that because you might have an item_id of "Ss001XXX" which would match WHERE items_collected LIKE '%Ss001%'