I'm developing a stock and warehouse management system using relational databases (MySQL) and PHP. Due to the fact that the stock products will have multiple characteristics (widths, heights, weights, measures, colors, etc) there raises the need of having a database model approach of storing the attributes and the possibility to add/edit new attributes, alter product types and so on.
So, in the current concept I can see only 3 viable models:
store all attributes in a single table, as separated column and
based on product type (probably category) to serve them to the end
user to fill
the EAV (Entity - Attribute - Value) model that will involve
something like this:
a category table containing classes of attributes
a class of attributes table that will contain separate classes with multiple attributes (in this manner we ensure that we can add to a category a class of attributes without the need to manually add to similar categories attributes one after the other)
a attributes table responsible for the attribute itself
a attributes values table where we store the values
Store all common attributes in a single table and create multiple tables for all different category type: this model would require to change the database every time we encounter a new category type
The second model is inspired from here.
After reading a lot regarding the EAV model I now have doubts over this model and I am little concern regarding the ways I will have to connect different product attributes in orders / invoices and so on.. Even the validation of forms seems that it will be a real pain of using the EAV model, but still.. I wouldn't like to have a single table with 100+ columns and then to be ready to add new columns whenever a new attribute is to be added..
So, the question would be: is there a cheaper solution? Or could the EAV model be improved?
I know it's a long and old debate, but everybody is just pointing to NoSQL and I only rely on RDBMS..
EDIT:
The downside of those approaches (or of most of the approaches found) is that:
for a specified attribute there probably should exist a measure unit
(eq. attribute weight should have a drop down with measuring units)
a specified attribute should be mandatory or not
all attributes should have a validation on form submit
Until now, the only feasible solution would be to create a new table for every new category, and deal in that table with all custom attributes and rules. But, yet again, it would end up to a real pain when a new category is to be set up.
EDIT 2:
The option of using a Json column in MySQL, does not solve from my point of view any of the downsides mentioned above.. OR, maybe I am wrong and I don't clearly see the big picture..
I gather that these are your primary requirements:
Flexible attributes
Your exact need here is unclear: it sounds like you either expect the attributes to change, or at least expect that all attributes will not always be applicable to all products (i.e. a sparse matrix)
Products are also categorized, and the category will (at least partially) determine what attributes are applicable to a product
The attributes themselves may have additional properties aside from their value, that must be provided by the user (i.e. a unit that goes with a weight)
Input validation is a must, and checks things like:
All required attributes are present
Attributes which are not applicable are not present
Attributes have valid values
User-provided attribute properties have valid values
You probably also want to make sure you can search/filter efficiently by attributes
These different requirements all result in different technical needs, and different technical solutions. Some are matters of database, and some will have to be solved in code regardless of database choice. Obviously you are aware of some of these issues, but I think it is worth really breaking it down:
Flexible Attributes
Having a list of flexible attributes (as you know) does not work well with RDBMS systems where your table schema has to be pre-defined. This includes pretty much all of the SQLs, and definitely MySQL. The issue is that changing the table schema is expensive and for large tables can take minutes or hours, making it practically impossible to add attributes if you have to add a column to a table to do it.
Even if your list of attributes rarely changes, a large table of attributes is very inefficient if most products don't have a value for most attributes (i.e. a sparse matrix).
In the long run, you just won't get anywhere if your attributes are stored as a column in tables. Even if you break it down per-category, you are still going to have large empty tables that you can't add columns to dynamically.
If you stick with an RDBMS your only option is really an EAV system. Having considered, researched, and implemented EAV systems, I wouldn't worry too much about all the hype you hear about them on the internet. I know that there are lots of articles out there talking about the EAV "anti-pattern", and I'm the kind of person who takes proper use of software design patterns seriously, but EAV does have a perfectly valid time and place, and this is it. In the long run you will not be able to do this on an RDBMS without EAV. You could certainly look at a NoSQL system that is designed for this specific kind of problem, but when the rest of your database is in a standard RDBMS, installing or switching to a NoSQL system just to store your attribute values is almost certainly overkill. You certainly aren't going to want to lose the ACID compliance that a RDMBS comes with, and most NoSQL systems don't guarantee ACID compliance. There is a wave of NewSQL systems out there that are designed to get the best of both worlds, but if this is just one part of a larger application (which I'm sure is the case), it probably isn't worth investigating completely new technologies just to make this one feature happen. You could also consider using something like JSON storage inside MySQL to store your attribute values. That is a viable option now that MySQL has better JSON support, but that only makes a small change to the big picture: you would still need all your other EAV tables to keep track of allowed attributes, categories, etc. It is only the attribute values that you would be able to place inside of the JSON data, so the potential benefits of JSON storage are relatively small (and have other issues that I will mention down the road).
So in summary, I would say that as long as the rest of your application runs on a RDBMS, it is perfectly reasonable to use EAV to manage flexible attributes. If you were trying to build your entire system in an EAV inside of a RDBMS, then you would definitely be wasting your time and I'd tell you to go find a good NoSQL database that fits the problem you are trying to solve. The disadvantages of EAV do still apply though: you can't easily perform consistency checks within your RDBMS system, and will have to do that yourself in code.
Categorized products with category-specific attributes
You've pretty much got it here. This is relatively straight-forward inside an EAV system. You will have your attributes table, you will have a category table, and then you will need a standard one-to-many or many-to-many relationship between the attributes and categories table which will determine which attributes are available to which category. You obviously also have a relationship between products and categories, so you know which products therefore need which attributes.
Your option #3 is designed to fulfill this requirement, but having a table with each attribute as a column will scale very poorly as your system grows, and will definitely break if you ever need to dynamically add attributes. You don't want to be running ALTER TABLE statements on the fly, especially if you have more than a few thousand records.
Managing attribute properties
It is one thing to store dynamic attributes and values. It is another problem entirely to store dynamic attributes, values, and associated meta data (i.e. store a weight as well as the unit the weight is in). This however is no longer a database problem, but rather a code problem. In terms of actually storing the information your best bet is to probably store your meta data inside your attribute values table, and rely upon some code abstractions to handle the input validation as well as form building. That can get quite complicated quite fast, especially if done wrong, and talking through such a system would take another entire post. However, I think you are on the right track: for a fancier attribute that requires both a value and meta data, you need to somehow assign a class that is responsible for input processing and form validation. For instance for a simple text field you have a "text" class that reads the user's value out of the form and stores it in the proper "attribute_values" table, with no meta data stored. Then for your "weight" attribute you would have a "weight" attribute that stores the number given by the user (i.e. 0.5) but then also stores the unit the user specified with that number (i.e. 'lbs') and persists both to the "attribute_values" table (in pseudo-SQL): INSERT INTO attribute_values value='0.5', meta_data='{"unit":"lbs"}', product_id=X, attribute_id=X. Ironically JSON probably would be a good way to store this meta data, since the exact meta data kept will also vary by attribute type, and I doubt you would another level of tables to handle that variation in your EAV tables.
Again, this is more of a code problem than storage problem. If you decided to do JSON tables the overall picture to meet this requirement wouldn't change: your "attribute type classes" would simply store the meta data in a different way. That would probably look something like: UPDATE products SET attributes='{"weight":0.5,"unit":"lbs"}' WHERE id=X
Input Validation
This will have to be handled exclusively by code regardless of how you store your data, so this requirement doesn't matter much in terms of deciding your database structure. A class-based system as described above will also be able to handle input validation, if properly executed.
Sort/Search/Filter
This doesn't matter if you are exclusively using your attributes for data storage/retrieval, but will you be searching on attributes at all? With a proper EAV system and good indexes, you can actually search/sort efficiently in an RDBMS system (although it can start to get painful if you search by more than a handful of indexes at a time). I haven't looked in detail, but I'm pretty sure that using JSON for storage won't scale well when it comes to searching. While MySQL can work with JSON now and search the columns directly, I seriously doubt that such searching/sorting makes use of MySQL indexes, which means that it won't work with large databases. I could be wrong on that one though. It would be worth digging into before committing to a MySQL/JSON storage setup, if you were going to do something like that.
Depending on your needs, this is also a good place to compliment an RDBMS system with a NoSQL system. Having managed large-ish (~1.5 million product) e-commerce systems before, I have found that MySQL tends to fall flat in the searching/sorting category, especially if you are doing any kind of text searching. In an e-commerce system a query like: "Show me the results that best match the term 'blue truck' and have the attribute 'For ages 3-5'" is common, but doing something like that in MySQL is about impossible, primarily because of the need for relevancy based sorting and scoring. We solved this problem by using Apache Solr (Elastic is a similar solution) and it managed our searching/sorting/search term scoring very well. In this case it was a two database solution. MySQL kept all the actual data and stored attributes in EAV tables, and anytime something got updated we pushed a record of everything to Apache Solr for additional storage. When a query came in from a user we would query Apache Solr which was an expert at text searching and could also handle the attribute filtering with no trouble, and then we would pull the full product record out of our MySQL database. The system worked beautifully. We had 1.5 million products, thousands of custom attributes, and had no trouble running the whole thing off of a single virtual server. Obviously there was a lot of code going on behind the scenes, but the point is that it definitely worked and wasn't difficult to maintain. Never had any issues with performance from either MySQL or Solr.
Well, this is just one approach. You could simplify this if you don't need or want all of this.
You could, for example, use a Json column in Mysql, to store all of the extra attributes. Another idea, in the product type, add a json column to store the custom attributes and types, and use this to draw the form on the screen.
I would recommend you to go through an EAV database first in order to understand the database creation & its values.
You can follow magento DB structure which uses EAV model.
EAV stands for Entity attribute and value model. Let’s closely have a look at all parts.
Entity: Data items are represented as entity, it can be a product or customer or a category. In the database each entity have a record.
Attribute: These are belongs to different entity, for example a Customer entity have attributes like Name, Age, Address etc. In Magento database all attributes are listed in a single table.
Value: Simply the values of the attributes, for example for the Name attribute the value will be “Rajat”.
EAV is used when you have many attributes for an entity and these attribute are dynamic (added/removed).
Also there is a high possibility that many of these attribute would have empty or null value most of the time.
In such a situation EAV structure has many advantages mainly with optimized mysql storage
For Your case - Category can also have attributes, products can also have attributes so on with customers etc ...
Let's take an example of categories. Following are the tables provided by magento:
1. catalog_category_entity
2. catalog_category_entity_datetime
3. catalog_category_entity_decimal
4. catalog_category_entity_int
5. catalog_category_entity_text
6. catalog_category_entity_varchar
7. catalog_category_flat
Follow this link to know more about table
Magento Category Tables
For attributes which are select box. You can put dropdown values under option values.
Follow this to link to understand magento eav structure which will give you clear picture about how EAV model work & how you can make a best use of it.
magento table structure
There are three approaches if you want to stick with a relational database.
The first is best if you know in advance the attributes for all the products. You chose one of the three ways to store polymorphic data in a relational model.
It's "clean" from a relational point of view - you're just using rows and columns, but each of the 3 options has its own benefits and drawbacks.
If you don't know your attributes at development time, I'd recommend against these solutions - they'd require significant additional tooling.
The next option is EAV. The benefits and drawbacks are well documented - but your focus on "validating input forms" is only one use case for the data, and I think you could easily find your data becomes "write only". Providing sorting/filtering, for instance, becomes really hard ("find all products with a height of at least 12, and sort by material_type" is almost impossible using the EAV model).
The option I prefer is a combination of relational data for the core, invariant data, and document-centric (JSON/XML) for the variant data.
MySQL can query JSON natively - so you can sort/filter by the variant attributes. You'd have to create your own validation logic, though - perhaps by integrating JSON Schema in your data entry applications.
By using JSON Schema, you can introduce concepts that "belong together", and provide lookup values. For instance, if you have product weight, your schema might say weight always must have a unit of measure, with the valid options being kilogram, milligram, ounce, pound etc.
If you have foreign key relationships in the variant data, you have a problem - for instance, "manufacturer" might link to a manufacturers table. You can either model this as an explicit column, or in the JSON and do without SQL's built-in foreign key tools like joins.
PHP noob here.
I have an Item model which is responsible for adding items to my cart.. However the items I'm adding are not simple products. Each item type has a complicated formula for generating a customized quote based on the users selections and personal discount, etc.
The simplest way I can think to do this is to have functions for each product within the item model with each unique formula in. However, this will get messy very fast.
What is the best practice for splitting these into separate files and then including those files in the model, just for keeping my code clean and seperate?
So for example I have aircraft, cars, boats as categories, each product within each category has it's own formula but I at least want to keep aircraft formulas separate from boat for ease of reading.
Is it best to create an aircraft model, boats model etc and have them all use the items DB table and then once they are added I can control them all from the Item model?
I may have just answered my own question, but I don't know :D Learning loads but still a noob.
I would suggest two things:
Extract logic from models
I would extract as much of the business logic as possible from the models - to, say, several formula classes (one for each different formula), which you can easily execute wherever you need to.
If the data you will save differs between each type, you may want to look into...
Polymorphic relationships
See the laravel docs on this for specifics.
You would have a Product model, that has a polymorphic morphTo relationship (called "type", maybe) to several other "product type" models (that all have an inverse morphOne back to the Product).
Polymorphic relationships were the first thing i thought of while reading your question, but on re-reading it, the data structure for each type might not be different. If that's the case, ignore the polymorphic bit!
I've been trying to create an application where everything is effectively an object with a series of fields. I've abstracted it to the level that you have the following tables:
ObjectTemplate
Field
LinkObjectTemplateField
FieldType
Each ObjectTemplate has a series of fields (a many-to-many relationship), which can be found in LinkObjectTemplateField. Field is linked to FieldType (many-to-one relationship). Field also has an ObjectTemplateID field - so let's suppose we have an object template called Section, and another object template called Question (as in for a questionnaire). Section would have Question as a field for questionnaire designers to use to define which questions appear in a section. Each Question would then be linked to a series of Values (or none at all in the case that it is of FieldType 'Text'.
We're able to create fields, field types and object templates so far. However I've come to realise that actually all 3 of these could be represented within the above tables, and I could probably kill off one of these tables too (so I only have ObjectTemplate and LinkObjectTemplateField, where Field is an ObjectTemplate in it's own right so there is a link simply between ObjectTemplate and itself via LinkObjectTemplateField).
My aim is to have one table structure for ALL object types, both as it currently stands and in the future. I'll have a class which picks up all of the fields for a particular object, and the fields it is expecting based on the objecttemplate, and decides how to present the fields based on the template. This seems to be getting very complex and I keep finding myself getting confused. I have a week left to work on this, so my questions are: should I plough on with this? Are there any better techniques to achieve this, or any flaws in my approach? Should I have stuck with the old structure (an entire table for each object type, with the same fields as most other object types for the core details - name, description, deleted etc.)?
Edit
I have been going over my approach again and come to the following conclusions:
Each object type, including object template itself, should have its' own record in the objecttemplate table.
Each object template, field and fieldtype should then have its' own row in the object table.
In this way, for example, Text, Dropdown etc. will be objects using the fieldtype object template. The IDs of these will be used in the functions for writing the forms - they will be declared as constants and referenced via MAIN::TEXT, MAIN::DROPDOWN and so on.
You are effectively trying to implement o form of EAV, and unless you actually need the flexibility it brings, is considered an anti-pattern.
Such "inner platform" is usually a poor replica of the real thing. In a nutshell:
It's difficult to enforce constraints that are otherwise available to "normal" tables and fields, including data types, NULL-ability, CHECKs, keys, and foreign keys.
You no longer have a good "target" for setting permissions or creating triggers.
It's difficult to limit an index to a specific "column", or make it use a "native" type.
It's difficult to reconstruct the "original" object. Usually, a lot of JOINing is required and the resulting object is not represented as a single row (which may be awkward to the client). Indexes and query optimizer can no longer work optimally.
So unless you absolutely have to be able to change data structure without changing database structure, just use what DBMS already provides through "normal" tables/columns/constraints...
My aim is to have one table structure for ALL object types, both as it currently stands and in the future.
Well, you kind of already have that built-in to your DBMS: it's called "data dictionary". Yes, you change it through CREATE/ALTER/DROP statements instead of INSERT/UPDATE/DELETE, but at the logical level it's a similar thing.
Should I have stuck with the old structure (an entire table for each object type, with the same fields as most other object types for the core details - name, description, deleted etc.)?
Probably.
BTW, if you have a lot of common fields (and/or constraints), consider putting them in a common "base" table and then "inheriting" other tables from it.
I need to make a lead convert (and relate to) 1 or more opportunity on SugarCRM. With just one lead it's easy, I just have to set the 'opp_id' field - however, with multiple leads I'm not able to do that.
I need at least to create a relationship 1:n between Leads and Opportunities, and to fill that relationship with PHP code (receiving both lead and opportunity ids).
Any ideas?
Thanks.
What I would do is create a one to many relationship between Leads and Opportunities in Studio, what this does is creates a relationship table that will be called something like leads_opportunities (or opportunities_leads depending on what table is the LHS, I think). This will also create a subpanel, the Opportunities can be added to lead via the Select or Create option on the Subpanel. If you want to create multiple relationship straight from the Lead screen, you could add a number of Opportunity relate fields. With the after_save logic hook, you could check for the existence of that specific Lead/Opportunity combo (since there isn't a multi-field index on the relationship tables) and write it if it doesn't exist. It depends more what the use case requirements are, but that should give you an idea of how to handle it a couple different ways...
So I'm working on a framework-esque system with one of my co-workers. Our current challenge is how to best implement statuses. Oftentimes, a status will carry with it unique data (a color for a table row, or text to be displayed to a user. etc). Currently, we have a statuses table which contains all this data. Contained in that table is a column: "css_class", which, whenever a record has that status, the specified CSS class is attached to the element (in this case a tr). Also, in order to assign another record a specific status, a foreign key is specified in that database table (in this case, a user has a specific status. So in the users table, there is a statuses_id foreign key). This implementation works alright, but there are a few problems. First, what if I need to perform a specific action in PHP if a record is in a specific status? The way we do it now is something like this:
if($user->status==0)
{
//execute some code
}
This really doesn't work well if statuses can change. Change one status, and the associated code either breaks or behaves differently than intended.
The other issue, and the main reason for posting a question is that the table contains the column "css_class". This is very versatile and allows us change the style of a specific status very quickly. But we really dislike the idea of putting code inside a database. Perhaps having CSS classes in a database isn't necessarily a bad thing, but I really don't know what the common practice is. Any ideas?
EDIT:
What I've gathered from the first few answers is that I should keep all my view stuff out of my model stuff in order to maintain an MVC framework. My argument is that if I keep the css_class name out of the database, then I'm checking the status id in the view in order to decide which class to assign it. So if I put the class in the database, I'm putting View information in the Model. If I don't put CSS classes in the database then I'm putting Model information in the View (checking which ID it belongs to). So by not muddying up the Model, I muddy up the view instead.......
The most elegant way I've seen this solved so far (and I've worked with a few MVC implementations now) is to store only the relevant data in the database. E.g. you'd store status="red" in the database, and leave it up to the view to know what to do with a red status, in terms of CSS. The problem is then solved by designing a sufficiently advanced View layer that creates reusable structures -- that way you don't need to always be updating things on a page-by-page basis when the css changes.
Passing this information up to the Model somewhat defeats the point of the content/presentation separation, because now your code needs to know to pull presentation information off the database and forward it along to the View level or, shudder, you'll be pulling that stuff from the database right in your View layer code, which makes maintenance a nightmare, as you've now lost control over the information flow.
If you want to continue your paradigm of storing this in the DB, you could make another table that maps VARCHAR names of the statuses to their corresponding INTEGER IDs.
However, if this was my framework. I would not be storing view information like this in the database. This would be handled by the V of my MVC setup.
From a data modelling point of view:
Have a different table for each "kind" of status; keep user statuses separate from page statuses (for example) - group the like entities together.
Don't put the CSS classes into the database, but use some form of status indicator - this could be an ENUM column, if you know the set of possible statuses up front. Transform this into the appropriate CSS class in the view layer. You don't want to end up in a situation where your CSS can't be changed because some data in the database prevents it.