I'm building my first app right now, but I'm new to mysql databases.
I want to store users personalized settings in database, and here are two scenarios to make that happen:
First one:
COLUMNS: "uid" | "app_settings"
ROWS: 1 | 0,1,0,1,ffff00,#ff0000
Which is storing them as an array, and breaking them up by PHP explode.
Second one:
COLUMNS: "uid" | "show_menu" | "show_toolbar" | "show_email | "menu_color" | "toolbar_color"
ROWS: 1 | 0 | 1 | 1 | #ffff00 | #ff0000
Which is storing each in a separate column.
Both ways work fine, but I want to know if it's a bad practice to use the first method.
Does the extra processes to break apart each value is overwhelming for the server resources in a large scale? (Using the PHP explode) or selecting multiple columns is somehow just like exploding them by php in terms of processing speed?
It all depends on what for do you intent to use this data.
Main purpose of using separate columns in database is to have ability to index such data.
If it is a matter of storage only you can use your storage format in one field, but it is much better to use well known format as json (json_encoding in PHP before storing in db, and json_decode after reading).
Also if you really want to save up space, then assuming, that things such as "show_menu" / "show_toolbar" are simply boolean flags, you can store them in one number as a bit fields. For example field named show_rights may have value of 6 which translates to binary 110, so [1: show_menu][1: show_toolbar][0: show_email].
I hope I can explain well enough what I am looking for. I tried doing some searches on here, but wasn't finding what I want. I want to do this with strictly mysql if I can with no PHP involved.
So I have a table with terms in it with URL links for each term, we'll call that terms. I have another table with short paragraphs of text we'll call paragraphs. What I want to do is select a paragraph from the table and search it for any of the words contained in the terms table. If the term is in the text, I want to replace it with a concatenated html anchor tag using data from the terms table. I then want the paragraph returned as a formatted string with the anchor tags in place. This way if I add or remove terms to or from the table, I always get a paragraph formatted with the most up to date terms.
Let me know if this is too confusing and I'll see if I can explain it better. Also, if I missed something in my searching that is already posted for this, feel free to guide me in that direction.
Thanks in advance for any help.
-------------------------------------------------
Edit: Here is an example.
-------------------------------------------------
`terms` table
-------------------------------------------------
ID | term | URL
---|----------|----------------------------------
1 | printer | http://t-shooting.link/printers
2 | scanner | http://t-shooting.link/scanners
3 | copier | http://t-shooting.link/copier
-------------------------------------------------
`problems` table
ID | problem
---|---------------------------------------------
1 | the two printers (side by side) need cleaning as the
| copies from them come out with streaks.
2 | My computer won't recognize scanner when I try to use
| it. I was able to use the copier to scan for now, but
| it would be nice to use the scanner that I have in my
| room.
So if I select ID 2 from the problems table I need it to search for any words contained in the terms table and replace them with a mysql "CONCAT('', terms.term, '')" or however I decide to format that. This would make the paragraph look like this:
My computer won't recognize scanner when I try to use it. I was able to use the copier to scan for now, but it would be nice to use the scanner that I have in my room.
The more I do on the mysql side, the less overhead I have on the PHP side. I know ways to do this in PHP, I am just wondering if there is a way to do it in mysql.
Does that help a bit?
About
I have this table in my database holding some information saved with a user id and time.
| CONTENT | USER ID | TIME |
| text | 1 | 1405085592 |
| hello | 2 | 1405085683 |
| hey | 1 | 1405086953 |
This example could be a data dump from my database, now as you can count there is "three" rows. However I only need to know how many users there have some information in my database. Therefor the result I'm really looking for is "two", because only two users have information in the database. User ID 1 is owning both "text"(1) & "hey"(3) where user ID 2 haves "hello"(2).
In short
I want to count how many users (regardless how many rows of information they have) there are inside my database.
** What I tried **
I tried to fetch every single row into an array and then using array_unique to count them together, works fine but I do not see this as a clean and best way to do this.
Then what?
I could use the array_unique and just use count to see how many rows there are, but I'm looking for something more clean. I tried to search for this, but I'm not actually sure what I should search for in term to hit something I'm looking for. After being stuck and though I wanted to learn something new, I wanted to post this problem here.
Note
I hope you guys can help me, I have tried to make it clear what I'm looking for and what I tried. If not please let me know. Sorry if some of the above contains misspelled words, incorrect grammar or is badly explained. I do not speak English daily, but I try my best.
You are looking for the DISTINCT keyword. It returns the count of unique values of a column:
SELECT COUNT(DISTINCT user_id)
FROM your_table
See example on SQL Fiddle.
This query:
SELECT DISTINCT user_id FROM table
will return just one row for every user in the table.
I have sample data in database with multiple different name,date and menu... I need to proceed data with the same name and date, but each data has a different menu...
DATABASE
the problem is that I need to store all the menu from the same name and date as a new variable since I need to call it for table view in web, and the table need to be look like this :
TABLE
I already using array to store the data menu for each same name and date, so the menu come out like this :
I'm really confused at how to continue make approach after this part,
How do I get output --> Candy Cake Cheese , so I can call it in table (?)
I think I can't seem to just use concat_ws in my query, since the values are from the same field of array, or is there another solution so I can get the output table I need ??
Thanks :)
You're probably looking for MySQL GROUP_CONCAT() aggregate function:
SELECT Name, Date, GROUP_CONCAT(Menu) AS Menu
FROM Table
GROUP BY Name, Date
This will give your data like that:
|--------|------------|-------------------|
| Name | Date | Menu |
|--------|------------|-------------------|
| Sylvie | 2001-01-01 | Candy,Cake,Cheese |
| Sylvie | 2001-02-01 | Milk,Tea |
|--------|------------|-------------------|
From that you just need to use PHP explode() function to get each menu record as an array.
Use serialize or json_encode to store the value into the database and use unserialize or json_decode when you get the value from the database.
I have a large database of artists, albums, and tracks. Each of these items may have one or more tags assigned via glue tables (track_attributes, album_attributes, artist_attributes). There are several thousand (or even hundred thousand) tags applicable to each item type.
I am trying to accomplish two tasks, and I'm having a very hard time getting the queries to perform acceptably.
Task 1) Get all tracks that have any given tags (if provided) by artists that have any given tags (if provided) on albums with any given tags (if provided). Any set of tags may not be present (i.e. only a track tag is active, no artist or album tags)
Variation: The results are also presentable by artist or by album rather than by track
Task 2) Get a list of tags that are applied to the results from the previous filter, along with a count of how many tracks have each given tag.
What I am after is some general guidance in approach. I have tried temp tables, inner joins, IN(), all my efforts thus far result in slow responses. A good example of the results I am after can be seen here: http://www.yachtworld.com/core/listing/advancedSearch.jsp, except they only have one tier of tags, I am dealing with three.
Table structures:
Table: attribute_tag_groups
Column | Type |
------------+-----------------------------+
id | integer |
name | character varying(255) |
type | enum (track, album, artist) |
Table: attribute_tags
Column | Type |
--------------------------------+-----------------------------+
id | integer |
attribute_tag_group_id | integer |
name | character varying(255) |
Table: track_attribute_tags
Column | Type |
------------+-----------------------------+
track_id | integer |
tag_id | integer |
Table: artist_attribute_tags
Column | Type |
------------+-----------------------------+
artist_id | integer |
tag_id | integer |
Table: album_attribute_tags
Column | Type |
------------+-----------------------------+
album_id | integer |
tag_id | integer |
Table: artists
Column | Type |
------------+-----------------------------+
id | integer |
name | varchar(350) |
Table: albums
Column | Type |
------------+-----------------------------+
id | integer |
artist_id | integer |
name | varchar(300) |
Table: tracks
Column | Type |
-------------+-----------------------------+
id | integer |
artist_id | integer |
album_id | integer |
compilation | boolean |
name | varchar(300) |
EDIT I am using PHP, and I am not opposed to doing any sorting or other hijinx in script, my #1 concern is speed of return.
If you want speed, I would suggest you look into Solr/Lucene. You can store your data, and have very speedy lookups by calling Solr and parsing the result from PHP. And as an added benefit you get faceted searches as well (which is task 2 of your question if I interpret it correctly). The downside is of course that you might have redundant information (once stored in DB, once in the Solr document store). And it does take a while to setup (well, you could learn a lot from Drupal Solr integration).
Just check out the PHP reference docs for Solr.
Here's on article on how to use Solr with PHP, just in case : http://www.ibm.com/developerworks/opensource/library/os-php-apachesolr/.
You probably should try to denormalize your data. Your structure is optimised for insert/update load, but not for queries. As I got it, your will have much more select queries than insert/update queries.
For example you can do something like this:
store your data in normalized structure.
create agregate table like this
track_id, artist_tags, album_tags, track_tags
1 , jazz/pop/, jazz/rock, /heavy-metal/
or
track_id, artist_tags, album_tags, track_tags
1 , 1/2/, 1/3, 4/
to spead up search you probably should create FULLTEXT index on *_tags columns
query this table with sql like
select * from aggregate where album_tags MATCH (track_tags) AGAINST ('rock')
rebuild this table incrementally once a day.
I think the answer greately depends on how much money you wish to spend on your project - there are some tasks that are even theoretically impossible to accomplish given strict conditions(for example that you must use only one weak server). I will assume that you are ready to upgrade your system.
First of all - your table structure forces JOIN's - I think you should avoid them if possible when writing high performace applications. I don't know "attribute_tag_groups" is, so I propose a table structure: tag(varchar 255), id(int), id_type(enum (track, album, artist)). Id can be artist_id,track_id or album_id depending on id_type. This way you will be able too lokup all your data in one table, but of cource it will use much more memory.
Next - you should consider using several databases. It will help even more if each database contains only part of your data(each lookup will be faster). Deciding how to spread your data between databases is usually rather hard task: I suggest you make some statistics about tag length, find ranges of length that will get similar trac/artists results count and hard-code it into your lookup code.
Of cource you should consider MySql tuning(I am sure you did that, but just in case) - all your tables should reside in RAM - if that is impossible try to get SSD discs, raids etc.. Proper indexing and database types/settings are really important too (MySql may even show some bottlenecks in internal statistics).
This suggestion may sound mad - but sometimes it is good to let PHP do some calculations that MySql can do itself. MySql databases are much harder to scale, while a server for PHP processing can be added in in the matter of minutes. And different PHP threads can run on different CPU cores - MySql have problems with it. You can increase your PHP performace by using some advanced modules(you can even write them yourself - profile your PHP scripts and hard code bottlenecks in fast C code).
Last but I think the most important - you must use some type of caching. I know that it is really hard, but I don't think that there was any big project without a really good caching system. In your case some tags will surely be much more popular then others, so it should greately increase performance. Caching is a form of art - depending on how much time you can spend on it and how much resources are avaliable you can make 99% of all requests use cache.
Using other databases/indexing tools may help you, but you should always consider theoretical query speed comparison(O(n), O(nlog(n))...) to understand if they can really help you - using this tools sometimes give you low performance gain(like constant 20%), but they may complicate your application design and most of the time it is not worth it.
From my experience most 'slow' MySQL database doesn't have correct index and/or queries. So I would check these first:
Make sure all data talbes' id fields is primary index. Just in case.
For all data tables, create an index on the external id fields and then the id, so that MySQL can use it in search.
For your glue tables, setting a primary key on the two fields, first the subject, then the tag. This is for normal browsing. Then create a normal index on the tag id. This is for searching.
Still slow? Are you using MyISAM for your tables? It is designed for quick queries.
If still slow, run an EXPLAIN on a slow query and post both the query and result in the question. Preferably with an importable sql dump of your complete database structure.
Things you may give a try:
Use a Query Analyzer to explore the bottlenecks of your querys. (In most times the underlying DBS is quite doing an amazing job in optimizing)
Your table structure is well normalized but personal experience showed me that you can archive much greater performance levels with structures that enable you to avoid joins& subquerys. For your case i would suggest to store the tag information in one field. (This requires support by the underlying DBS)
So far.
Check your indices, and if they are used correctly. Maybe MySQL isn't up to the task. PostgreSQL should be similiar to use but has better performance in complex situations.
On a completely different track, google map-reduce and use one of these new fancy no-SQL databases for really really large data sets. This can do distributed search on multiple servers in parallel.