I have a mysql table looking like this:
id
some_field1
some_field2
variable_fields
datetime
...
Now I want to store more than 1 value in variable_fields like this:
user_id:5;message_id:10
The reason why I do not create a separate field for every value I want to store is that these values differ throughout the project. So I am storing different values along the project.
At some time variable_fields contains this value:
user_id:5;message_id:10
And at some other time it contains this value:
car_id:56;payment_id:45
This wouldn't be a big problem but I want to be able to search in this field. So something like: variable_fields LIKE '%payment_id:45%'.
This obviously takes time for mysql.. Is there another way of handling this instead of creating a field for every value? So some kind of dynamic field in mysql?
I happy for every kind of help. Thank you in advance!
Best regards,
Freddy
If you'll add a myisam full-text index or employ any other full-text tools on that column (e.g. sphinx, lucene) those searches you described will work much better, however that isn't advisable.
I would suggest either to divide the dynamic meta data into different tables per case, and keep a type_id in the main table, or keep columns for all options that are set to NULL by default. Really depends if there is a simple division or is this really dynamic and changing over time. In case you're diving the data into several tables, a JOIN according to type_id will give the ability to query by those specific fields values. Be sure to create an index in both tables on the mutual id.
Related
I currently have a database structure for dynamic forms as such:
grants_app_id user_id field_name field_value
5--------------42434----full_name---John Doe
5--------------42434----title-------Programmer
5--------------42434----email-------example#example.com
I found this to be very difficult to manage, and it filled up the number rows in the database very quickly. I have different field_names that can vary up to 78 rows, so it proved to be very costly when making updates to the field_values or simply searching them. I would like to combine the rows and use either json or php serialize to greatly reduce the impact on the database. Does anyone have any advice on how I should approach this? Thank you!
This would be the expected output:
grants_app_id user_id data
5--------------42434----{"full_name":"John Doe", "title":"Programmer", "email":"example#example.com"}
It seems you don't have a simple primary key in those rows.
Speeding up the current solution:
create an index for (grants_app_id, user_id)
add an auto-incrementing primary key
switch from field_name to field_id
The index will make retrieving full-forms a lot more fun (while taking a bit extra time on insert).
The primary key allow you to update a row by specifying a single value backed by a unique index, which should generally be really fast.
You probably already have some definition of fields. Add integer-IDs and use them to speed up the process as less data is stored, compared, indexed, ...
Switching to a JSON-Encoded variant
Converting arrays to JSON and back can be done by using json_encode and json_decode since PHP 5.2.
How can you switch to JSON?
Possibly the current best way would be to use a PHP-Script (or similar) to retrieve all data from the old table, group it correctly and insert it into a fresh table. Afterwards you may switch names, ... This is an offline approach.
An alternative would be to add a new column and indicate by field_name=NULL that the new column contains the data. Afterwards you are free to convert data at any time or store only new data as JSON.
Use JSON?
While certainly it is tempting to have all data in one row there are somethings to remember:
with all fields preserved in a single text-field searching for a value inside a field may become a two-phase approach, as a % inside any LIKE can skip into other field's values. Also LIKE '%field:value%' is not easily optimized by indexing the column.
changing a single field means updating all stored fields. As long as you are sure only one process changes the data at any given time this is ok, otherwise there tend to be more problems.
JSON-column needs to be big enough to hold field-names + values + separators. This can be a lot. Also if you miss-calculate a long value in any field means a truncation with the risk of loosing all information on all fields after the long value
So in your case even with 78 different fields it may still be better two have a row per formular user and field. (It may even turn out that JSON is more practicable for formulars with few fields).
As explained in this question you have to remember that JSON is only some other text to MySQL.
I'm hitting a dead with the best practice for storing a large amount of options and values in my MYSQL database and then assigning them to properties. The way I usually do this (example is for real estate) is to create a table called "pool" then have an auto increment value as the ID and a varchar to store the value, in this case "Above Ground" and another row for "In-ground". Then in my property table I would have a column for "has_pool" with the proper ID value from the "pool" table assigned. Obviously the problem is that with hundreds of options (fireplace, water view, etc) for each property, my number of database tables will get very large, very fast and my left joins would become out of control on the front side.
Can someone point me in the right direction on what the best practice would be to easily populate new values for the property attributes and keep the query count down to a minimum? I feel like there is a simple solution but my research so far has not made it apparent to me. Thank you!
One way you could do this is create an 'options' table with four columns: id, menuId, value
Create another table called menus, with two fields; id and name.
Add the menu names (pool, fireplace etc.) to the menus table, and then add the possible values to the options table, including the id of the menu it is related to.
I'd store all the values serialized (e.g. JSON or XML or YAML) into a blob, and then define inverted index tables for attributes I want to be searchable.
I describe this technique and alternatives in my presentation Extensible Data Modeling with MySQL.
Also see http://bret.appspot.com/entry/how-friendfeed-uses-mysql
I have a MySQL/PHP performance related question.
I need to store an index list associated with each record in a table. Each list contains 1000 indices. I need to be able to quickly access any index value in the list associated to a given record. I am not sure about the best way to go. I've thought of the following ways and would like your input on them:
Store the list in a string as a comma separated value list or using JSON. Probably terrible performance since I need to extract the whole list out of the DB to PHP only to retrieve a single value. Parsing the string won't exactly be fast either... I can store a number of expanded lists in a Least Rencently Used cache on the PHP side to reduce load.
Make a list table with 1001 columns that will store the list and its primary key. I'm not sure how costly this is regarding storage? This also feels like abusing the system. And then, what if I need to store 100000 indices?
Only store with SQL the name of the binary file containing my indices and perform a fopen(); fseek(); fread(); fclose() cycle for each access? Not sure how the system filesystem cache will react to that. If it goes badly then there are many solutions available to adress the issues... but that's sounds a bit overkill no?
What do you think of that?
What about a good old one-to-many relationship?
records
-------
id int
record ...
indices
-------
record_id int
index varchar
Then:
SELECT *
FROM records
LEFT JOIN indices
ON records.id = indices.record_id
WHERE indices.index = 'foo'
The standard solution is to create another table, with one row per (record, index), and add a MySQL Index to allow fast search
CREATE TABLE IF NOT EXISTS `table_list` (
`IDrecord` int(11) NOT NULL,
`item` int(11) NOT NULL,
KEY `IDrecord` (`IDrecord`)
)
Change the item's type according to your needs - I used int in my example.
The most logical solution would be to put each value in it's own tuple. Adding a MYSQL index to each tuple will enable the DBMS to quickly ascertain the value, and should improve performance.
The reasons we're not going with your other answers are as follows:
Option 1
Storing multiple values in one MYSQL cell is a violation of the first stage of database normalisation. You can read up on it here.
Option 3
This has heavy reliance on other files. You want to localize your data storage as much as possible, to make it easier to maintain in the future.
I have a problem how to store some data in mysql.
I have website which when link is pressed pass some data to php file which read this data with get and write in database(mysql). I'm passing campaign_id and unknown number of parameters.
http://domain.com/somefile.php?campaignid=1¶meter1=sometext1¶meter2=sometext2¶meter3=sometext3,....etc..
I don't know actual number of parameters because user make them in some sort of cms. The problem I'm facing is how to store them in database. I was thinking to make it like this below but i'm not sure if it's the right and the most effective way:
Combinations Table
-combination_id (Primary key and auto increment)
-campaign_id
-parameter1
-parameter2
-parameter3
-parameter4
-parameter5
-parameter6
-parameter7
-parameter8
-parameter9
-parameter10
In this example I assume that user will not add/use more than 10 parameters(which I think is lame, but I can't get better solution)
Also if I use this design I assume I need to check in this file where is get them from passing and write to database, if each parameter exist(if it was passed).
You have to normalize your schema.
Assume the following tables:
Entity: id, campaign_id, other fields.
Parameter: id, entityId, parameterValue.
This is a Many-to-One relation.
What About storing all the parameters as json in one table row?
You could try something like this:
combination_id (primary key auto increment)
campaign_id ( indexed / foreign key / can't be unique!)
param_name
param_value
You'd have to create an entry for every parameter you're getting, but you could theoretically add a thousand parameters or more.
Might not be the fastest method though and can be a bit hard to work with.
I think this is the kind of data nosql databases are made for... At least, trying to force it into a sql database always ends up as some kind of kludge. (been there done it...)
as far as I can see, you have three different ways of storing it:
As you proposed. Probably the easiest way to handle it and also probably the most efficient. But, at the moment you get 11 parameters you are in for major problems...
Make a parameter table - parameter_id, - campaign_id parameter (possible parameter name if it matters) - this gives you total flexibility - but everything else, ecept for searching for single values gets more difficult,
Combine the parameters and store them all in a text or varchar field. This is probably even more efficient than 1, except for searching for single parameter values.
And if I may add
Use a database system with an array type, eg postgresql
If you don't know the actual number of parameters that will come through url, there is a best option to store the infinite number of values for a campaign_id.
For that you can create multiple rows in the table. Like,
insert into table_name values(<campaign_id>,<parameter1>,<sometext>)
insert into table_name values(<campaign_id>,<parameter2>,<sometext>)
insert into table_name values(<campaign_id>,<parameter3>,<sometext>)
insert into table_name values(<campaign_id>,<parameter4>,<sometext>)
Assuming the campaign_id is unique in url.
I have a table with some fixed columns id,title,created.., but there I need put some optional data from the user that I cant define a specif column, but I need this data for search from others users. What the best way?
Store a serialized array
Leave some column optionals (
exemple:leaves any 10 column
opt1,opt2,opt3...)
Create a new table (dataid,uid,key,value) and
new rows for each data.
the data would be like key:value pair
What should the best way to record any undefined optionals data from the user and optimize to search?
Good Day
Definitively 3).
Storing a serialized array means being dependend on a language (php, json etc), and selecting via sql is impossible.
2) ist bad, because of several reasons; mainly, you are less flexible in extending and selecting.
Create a meta table like 3), this is an established approach.
Update: 3) must be improved a bit: Create a table like this:
UID | key | value
not:
UID | key_value
The 3rd approach is the best because you can search easily inside the db and you will not worry about how many values per user