I have a Magento shop (using MySql db) and just noticed that some developer introduced a custom db for capturing some structured data.
Now I noticed that the tables are not linked via foreign keys with each other, but just added a column e.g. priceListID = 01124 which is the same Id as on price list table. So linking the data together must happen within the code by firing different select statements I assume.
Now I am wondering if this needs to be fixed soon or if it actually is ok not to use foreign keys on db level to link data together?
What are the down sides of doing this and are there maybe some benefits (like flexibility?)
Hope you can help me with this! Thanks a lot!
There're few advantages of keeping such constraints inside a database:
Performance. Most of constraints, such as foreign keys, are better implemented if stored inside a database, close to the data. You want to check data integrity with additional select? You have to make an extra request to the database. It requires some time.
What if you have several applications that work with your database? You have to write code for checking data integrity in all of them, which implies additional expenses.
Synchronization. While you're checking data integrity with additional select, some other user may delete this data at the same time. And you will not know about it. Of course, these checks can be properly implemented, but this is still an extra work you have to do.
To me, this is all a smell of bad, not scalable design which can bring many problems. Data integrity is what databases are built for. And these types of verifications should stay inside a database.
From your description, I understand that tables are indeed functionaly related, as they share a common piece of information (priceListID in the new table relates to id in the original table). On the one hand, this set-up would still allow writing queries that join the tables together.
The downside of not creating a foreign key to represent that relationship, however, is that, from database perspective, the consistency of the relationship cannot be guaranteed. It is, for example, possible that records are created in the new table where priceListID do not exist in the original table. It would also be possible to delete records in the old table while related records exists in the new one, hence turning the children to orphans.
As a conclusion: by not using foreign keys, the developers rely solely on the application to maintain data integrity. There is no obvious benefit not using the built-in features that the RDBMS offers to protect data consistency, and chances are developers just forgot that critical part of the table definition. I would suggest having a talk with them and intimate them to create the missing foreign key (unless they can give a clear explanation why they did not).
This should be as simple as:
ALTER TABLE newtable
ADD CONSTRAINT fk_new_to_original_fk
FOREIGN KEY (priceListID )
REFERENCES originaltable(id);
Please note that this requires all values in the referrencing column to be available in the parent table.
I was wondering if logic duplication can be reduced on this one. Let's say I have a users table and email column, which should be unique per record. What I normally do, is having a unique index on the column and validation code that checks if the value is already used:
SELECT EXISTS (SELECT * FROM `users` WHERE `email` = 'foo#bar.com')
Is it possible and practical to skip the explicit check and just rely on the database error when trying to put non-unique value? If we repeat the logic of uniqueness in two layers (database and application code), it's not really DRY.
I do that pretty often. In my custom database class I throw a specific exception for violated restrictions (this can be easily deduced from the numeric error code returned by MySQL) and I catch such exception upon insert.
It has the benefit of simplicity and it also prevents race conditions—MySQL takes care of data integrity in both variants, data itself and concurrent accesses.
The only drawback is that it can be tricky to figure out which index is failing when you have more than one (for instance, you may want to have a unique email and a unique username). MySQL drivers only report the violated key name in the text of the error message, there's no specific API for it.
In any case, you may want to inform the user about available names in an earlier stage, so a prior check can also make sense.
It makes sense to enforce the uniqueness of the email address in the database. Only that way you can be sure it is really unique.
If you do it only in the PHP code then any error in that code may corrupt the database.
Doing it in both is not needed but does, in my opinion, not offend against the DRY rule. For instance, you might want to check the presence of an email address during registration of a new user, and not only rely on the database reporting an error.
I assume by "DRY" you mean Don't Repeat Yourself. Applying the same logic in more than one place is not intrinsically bad - there's the old adage "measure twice, cut once".
In a more general case, I usually follow the pattern of applying the insert and catching the constraint violation, but for users with email addresses it's a much more complicated story. If your email is the only attribute required to be unique, then we can skip over a lot of discussion about a person having more than one account and working out which attribute is not unique when a constraint violation is reported. That the email is the only unqiue attribute is implied in your question, but not stated.
Based on your comments you appear to be sending this SQL from your PHP code. Given that, there are 2 real issues with polling the record first:
1) Performance and Capacity: it's an extra round trip, parse and read operation on the database
2) Security: Giving your application user direct access (particularly to tables controlling access) is not good for security. Its is much safer to encapsulate this as a stored procedure/function running with definer privileges and returning messages more aligned to the application logic. Even if you still go down the route of implementing poll first / insert if absent, you are eliminating most of the overhead in issue 1.
You might also spend a moment considering the difference between your query and...
SELECT 1 FROM `users` WHERE `email` = 'foo#bar.com'
On top of the database constraint, you should check if the email given already exists in it before trying to insert. Handling it that way is cleaner and allows for better validation and response for the client, without throwing an error.
The same goes for classic constraints such as MIN / MAX (note that such constraints are ignored on MySQL). You should check, validate and return a validation error message to the client before committing any change to the database.
I am currently maintaining a rather large office web-application. I recently became aware that via various developer-tools within web-browsers that values of select-boxes can easily be modified by a user (among other things). On the server side I do validation if the the posted data is numerical or not (for drop-downs), but don't actually check if the value exists in a database table, for example I have a dropdown box for salutation ('mr','ms','mrs','Mr/ms') etc. which correspond with a numerical values.
Currently I use Mysql's Myisam tables which don't offer foreign keys referential integrity, so I am thinking about moving to Innodb, yet this posses the following issue:
If I want to apply referential integrity (to insure valid ID's are inserted), it would mean I'd have to index all columns (if using integrity checks) that do not necessarily need to be indexed for performance issues at all (e.g. a salutation dropdown). If a very large database client-table has say 10 simular dropdowns (e.g. clientgroup, no. of employees, country-region etc) it would seem an overkill to index every linked table.
My questions:
1) when using referential integrity, do columns really need to be indexed also?
2) are there other practical solutions I may be overlooking? (e.g. use a separate query for every dropdown-list to see if the value exists in a table?)
3) How do other web-applications deal with such issues?
Help Appreciated!
thanks
Patrick
You only have to index the fields used in the foreign key relationships, and recent version of mysql do this automatically for you anyways. It's not "overkill". it's actually an optimization.
Consider that anytime you update/delete/insert a record, the foreign tables have to be checked for matching records - without the indexes, those checks could be glacially slow.
InnoDB automatically creates an index when you define a foreign key. If an index on that column already exists, InnoDB uses it instead of creating a new index.
As #MarcB mentioned in his answer, InnoDB uses these indexes to make referential integrity checks more efficient during some types of data changes. These changes include updating or deleting values in the parent table, and cascading operations.
You could use the ENUM data type to restrict a column to a fixed set of values. But ENUM has some disadvantages too.
Some web developers eschew foreign keys. To provide the same data integrity assurances, they have to write application code for every such case. So if you like to write and test lots of repetitive code, unnecessarily duplicating features the RDBMS already provides more efficiently, then go ahead! :-)
Most developers who don't use foreign keys don't write those extra checks either. They just don't have data integrity enforcement. I.e. they have sacrificed quality.
PS: I do recommend switching to InnoDB, and referential integrity is just one of the reasons to do so. Basically, if you want a database that supports ACID, InnoDB supports all aspects of that and MyISAM supports none.
I want to implement a filter-function in my PHP project.
To implement a filter, I usually just add a WHERE clause in my query to show filtered results.
My Problem is:
These filters require not only a smple added WHERE clause, but a huge Query including multiple JOINs. The resulting Query has > 30 lines.
Later, there should also be a search function which would then also require this huge query.
I wonder if this is a good practice or if I should add a "redundant" Database column to my database table where I compute the attribute I need for filtering on every update.
With this column, I wouldnt have my huge query on different places over my project, but have a redundant column.
What do you think?
Greetings
As questioned, here the table structure/code. This is not the exact code, because there is also a revision system which makes it even more complex, but for understanding this is enough:
table submissions:
ID (primary)
(additionalColumns)
table reports:
ID (primary)
submissionID (reference to submission table)
(additionalColumns)
table report_objects:
reportID (reference to reports table, multiple report_object for one report)
table accounting:
ID (primary)
reportID (reference to reports table, multiple accountings for one report)
(additionalColumns)
table accounting_objects:
ID
accountingID (reference to accounting table, multiple accounting_object for one accounting)
(additionalColumns)
For a submission, one or multiple reports are being create with multiple objects to account (report_objects).
For each report, I can create multiple accountings, where each accounting is for a few objects of the report. The accounted report_objects are stored in accounting_object
My query/filter checks, if each report_object of a submissionID is accounted (accounting_object exists) for one submissionID.
There isn't one definitive answer and, in practice, if it works and runs quickly enough for your needs then you can leave it as is. Optimization is always something you can come back to.
Joining correctly
If you are simply checking for the existence of a join table and only including the results with that join you can do this through the correct LEFT / RIGHT JOIN expressions. This is always the first call.
Expressiveness
Also be as expressive as you can with SQL, you want to give it the best chance to optimize your query, there are keywords such as EXISTS, for example, make sure to use them.
Denormalization
You can add in a column that stores the computed value, the complexity that arises out of this is ensuring that the value is always up to date. This can be done by triggers or manually. The pros:
It is the easiest method of getting around slowness introduced by computed columns.
The cons:
Ruins your nice normalized schema
If you do it manually in code, you will forget to do it somewhere, causing headaches.
Triggers can be a bit of a pain.
Materialized view
This is like denormalization but prevents polluting your normalized tables by created a stored view. This is achieved in MySQL by storing the result of your complex select into a results table when the values change. Again, the same as denormalization, the complexity is keeping this up to date. It is typically done with triggers. This can be a pain but keeps the complexity out of your schema. As mentioned by#eggyal it isn't a supported feature of MySQL yet so you will have to DIY...
Materialized views with MySQL
Pros:
Keeps dirty denormalized stuff away from your nice normalized schema.
Cons:
Materialized views aren't supported so setting them up requires work.
If you trigger the refresh of your views in code you get stale data, but isn't quite as painful as the single column staleness of denormalization.
Triggers can be a bit of a pain.
If you aren't sure, and it really matters, do some benchmarking.
EDIT If you code has this query in one form or another across your code base then that has the possibility of cause headaches in future as you will have to remember to change the statements in all of the places if or when they change.
If by doing the above you have made your statements really simple and concise then they may differ enough from each other for it to not be a problem.
You can do some things to help you out:
Put all of the related queries in a single place, i.e. a single class or script that handles this query in its various forms. This way at least all of the changes are limited to the one file.
You can, to help yourself out a bit more, do a bit of refactoring it to remove duplication between the queries.
Also, If you feel the database information is too exposed to the code you may want to abstract it behind a view.
I am trying to create a column in a table that's a foreign key, but in MySQL that's more difficult than it should be. It would require me to go back and make certain changes to an already-in-use table. So I wonder, how necessary is it for MySQL to be sure that a certain value is appropriate? Couldn't I just do that with a language like PHP, which I'm using to access this database anyway?
Similarly with NOT NULL. If I only access this database with PHP, couldn't I simply have PHP ensure that no null value is entered?
Why should I use MySQL to do enforce these constraints, when I could just do it with PHP?
I realize that NOT NULL is a very stupid part to neglect for the above reasons. But MySQL doesn't enforce foreign keys without a serious degree of monkeying around.
In your opinion, would it still be bad to use the "fake" foreign keys, and simply check if the values to be entered are matched in other tables, with PHP?
You are going to make mistakes with PHP, 100% guaranteed. PHP is procedural. What you want are declarative constraints. You want to tell the entire stack: "These are the constraints on the data, and these constraints cannot be violated." You don't want to much around with "Step 1 ... Step 2 ... Step 3 ... Step 432 ...", as your method of enforcing constraints on data, because
you're going to get it wrong
when you change it later, you will forget what you did now
nobody else will know all of these implicit constraints like you know them now, and that includes your future self
it takes a lot of code to enforce constraints properly and all the time - the database server has this code already, but are you prepared to write it?
The question should actually be worded, "Why should I use PHP to enforce these constraints, when I could just do it with MySQL?"
You can't "just" do it with PHP for the same reason that programmers "just" can't write bug-free code. It's harder than you think. Especially if you think it's not that hard.
If you can swear for the life of you that nothing will ever access the DB though any other means then your (of course bug-free) PHP page, then doing it with PHP alone will be fine.
Since real-world scenarios always contain some uncertainty, it is good to have the DB server watching the integrity of your data.
For simple databases, referential integrity constraints might not be an absolute requirement, but a nice-to-have. The more complex the application gets, the more benefit can you draw from them. Planning them in early makes your life easier later.
Additionally, referential integrity does it's part in forcing you to design the database in a more by-the-book manner, because not every dirty hack is possible anymore. This is also a good thing.
They are quite important. You don't want to define your model entirely through PHP. What if there is a bug in your PHP code? You could easily have null'ed columns where your business rules state you should not. By defining it at the database level, you at least get that check for free. You're going to really hate it when there are bugs in your PHP or if any other tool ever uses your database. You're just asking for problem, IMHO.
Be advised, this is the very short version of the story.
It's important to implement constraints in the database because it's impossible to predict the future! You just never know when your requirements will change.
Also consider the possibility that you may have multiple developers working on the same application. You may know what all the constraints are, but a junior developer may not. With constraints on the database, the junior developer's code will generate an error, and he'll know that something needs to be fixed. Without the constraints, the code may not fail, and the data could get corrupt.
I'm usually in favor of declaring constraints in the database. Arguments for constraints:
Declarative code is easier to make bug-free than Imperative code. Constraints are enforced even if app code contains bugs.
Supports the "Don't Repeat Yourself" principle, if you have multiple applications or code modules accessing the same database and you need business rules to be enforced uniformly. If you need to change the constraint, you can do it in one place, even if you have many apps.
Enforces data integrity even when people try to bypass the application, using ad hoc query tools to tinker with the database.
Enforces consistency which means that you can always be certain the data is in a valid state before and after any data update. If you don't use constraints, you may need to run periodic queries to check for broken references and clean them up.
You can model cascading update/delete easily with constraints. Doing the same thing in application code is complex and inefficient, cannot apply changes atomically (though using transaction isolation is recommended), and is susceptible to bugs.
Constraints help databases be more self-documenting, just as column names and SQL data types help.
Arguments against constraints:
More complex business rules cannot be modeled by declarative constraints, so you have to implement some in application space anyway. Given that, why not implement all business rules in one place (your app) and in the same language? This makes it easier to debug, test, and track code revisions.
Constraints often involve indexes, which incur some amount of overhead during inserts/updates. On the other hand, even if you don't declare a constraint, you probably need an index anyway, because the column may be used in search criteria or join conditions frequently.
Constraints can complicate your attempts to "clean up" mistakes in the data.
In your current project, the incompatibility of MyISAM vs. InnoDB with respect to referential constraints is causing some grief.
Enabling these constraints in MySQL takes almost zero time. If they save you from even a single bug due to faulty PHP or other code, isn't that worth it?
Keep in mind that the sorts of bugs you'll save yourself from can be rather nasty. Finding and fixing the bug itself may not be hard; the nasty part is that once you've fixed the bug you'll be left with a bunch of faulty data that may not even be salvageable.
I wouldn't even approach this problem from the "well, something other than PHP might access your data someday" angle. That's true, but even more important in my mind are the the headaches, time (money) and data loss that you can save yourself simply by adding a few simple constraints.
Use the database for structural data integrity, and use the BR layer for the rest. And catch errors as early as possible. They work together.
With luck, when your code as matured, you won't experience databse RI errors; and you can proudly announce yourself to be the first.
Even if your PHP code is perfectly bug-free, it may stop mid-script (out of memory error, segfault in some library, etc), leaving half-inserted stuff in the database, hence the importance of using InnoDB and transactions.
Same for constraints, of course you should have proper form validation, and database constraints behind it to catch bugs.
Database constraints are easy to specify, finding bugs in the application is hard, and even harder without constraints.
My experience has been that improperly constrained databases, and anything that uses MyISAM, WILL have inconssitent data after a few months of use, and it is very hard to find where it came from.
Having your data tier enforce data consistency through constraints helps to ensure your data remains consistent and provides cheap runtime bug checking within your application.
If you think constraints are not worthwhile you either have a small/non mission critical system or you are passing up a huge opportunity to improve the quality of your system. This cannot be understated.
Choices include: choosing a different RDBMS, reinvent your own metadata system or manually manage constraints. Manual management in queries without a metadata system quickly becomes infeasible to maintain and audit properly as schema/system complexity grows and unecessarily complicates an evolving schema.
My recommendation is to choose a different RDBMS.
Consistency checking is much harder than you may think. For example MySQL uses transactional read consistency which means the values you are checking against may not be the same values in the scope of another transaction. Consistency scemantics for concurrent access are very very difficult to get right if not bound directly to the data tier.
When all is said and done, even with a modest amount of effort put into manual checking, the likely outcome is that one would still be able to drive a truck through the corner cases you have failed to consider or committed an error in forming.
On your NOT NULL question... The obvious data field requirements are a good starting point. Here are a couple of other things to consider when defining column nullability.
It provides a guarantee that can very helpful when writing queries. Various joins may use NULL conditions to show a non-match of a table row separate from a NULL value that cannot be assumed if the condition allows nulls. (If NULLs are allowed, a match can mean either the row did not match or the row did match but the column value is null.)
The use of NOT NULL also helps define the rules for simpler queries matching values. Since you cannot say "WHEN value1 = value2" if both value1 and value2 are NULL the result of the evaluation is still false.
The most important thing about using NOT NULL to me, is more the documentation part. When I return to the project after a few months I forget which columns it is acceptable to have nulls in. If the column says NOT NULL, then I know I will never ever have to deal with potential null values from it. And if it allows null, then I know for sure I have to deal with them.
The other thing is, as others have noted: You may miss something somewhere, and cleaning up data sucks, or may be entirely impossible. It's better to know for sure that all data in your database is consistent.
I don't think you can be certain that your database will only be accessed by PHP and if so, by developers who will use it to respect those constraints for the entire lifecyle of your database.
If you include these constraints in your schema, then one can get a good idea of how the data is used and related by investigating your schema. If you only put all that in the code, then someone would have to look in both the database and the PHP code.
But shouldn't that stuff be in the design documentation, data dictionary, and logical database design?
Yes, but these documents are notorious for getting out of date and stale. I know you would never allow that to happen, but some people who have experience with projects with less discipline may assume this about your project, and want to consult the actual code and schema rather than documentation.
I highly appreciate your question, as I am deeply convinced that default-value rules should be implemented on the code-side, not on the database-side, and this for a very simple reason: when users are the one that initiate database changes (INSERTS, SELECTS and UPDATES), these changes shall integrate all business rules, and default values are basically business rules:
There is no invoice without invoice number
There is no invoice line without a quantity, and 0 or nulls are not acceptable
There is no incoming mail without date of reception
etc
We have decided a few years ago to get rid of all these "database-side" artefacts like "not null", "(do not) allow empty strings", and other "default value" tricks, and it works perfectly. Arguments in favor of the default value mainly refer to a kind of "security" principle ("do it on the database side because you will forget to to it on the code side / your language is not made for that/it's easier to do it on the database side") that does not make any sense once you have chosen not to implement any default value on the database side: just check that your business rules are properly implemented while debugging.
For the last 2 years, nobody in the team ever thought of declaring a default value in a table. I guess that our younger trainee does not even know about something that is called "default value".
EDIT: rereading some of the answers here, my final comment would be: do it on any side, either DB or code, but make your choice and do it on one side only! There is nothing more dangerous than having such controls on both sides, because eventually (1) you'll never know if both sides are really implementing the same rule, meaning that (2) checking the rules will mean checking both sides, which can really become a mess! The worst situation is of course when one part of the job is done on the database side (ie the rules that were identified when the database was created) and the other part (ie the newly identitified rules) done on the client side ... nightmare ....
Implement default values and constraints at the database level; rules that will result in acceptable data to any consuming application. This insulates you from integrity issues.
Then, implement better default values and constraints at the application level. If you are prohibited technically (access to APIs external to the database) from implementing a constraint, or generating a default value at the database level, the application is where you'll need to do it. This will result in a better default value. However a segfault, or general failure of the application will not result in unacceptable data being persisted.
I'm afraid this is a religious topic.
From a puristic point-of-view, you want the database to do the referential integrity. This is ideal when you have a multiplicity of applications accessing the database, because the constraints are in one place. Unfortunately, the real world is not ideal.
If you have to enforce some sort of referential integrity, in my experience, your application will need to know how to do this. This is regardless of whether it is the final arbiter, or the database checks it as well. And even if the database does do the referential integrity, then the application has to know what to do if the database rejects an update, because referential integrity would be violated...
As a sidenote, setting up MySQL to support foreign key constraints is a bit of a process because you need to shift to InnoDB. If you do just that, you can get a lot of performance back by setting innodb_flush_log_at_tx_commit to 1. But it probably would be better if you can instead re-engineer your site to be transaction-aware. Then you get two benefits of InnoDB.