What are your methods of linking data spread over multiple databases architectures (think MySQL vs PostgreSQL etc), into a single application?
Would you create giant hashtables/arrays to match content against one another? Are there other, more effective and less memory-consuming options for doing this?
If you were to use data both from a MySQL & PostgreSQL source, with no way of converting one DB to the other (application constraints, lack of time, lack of knowledge, ... ), how would you go about it?
SQL Relay or another sql proxy.
http://sqlrelay.sourceforge.net/
At least in the case of MySQL, you can use data from multiple databases in a single query anyway, provided the databases are hosted by the same MySQL Server instance. You can distinguish tables from different databases by qualifying the table with a schema name:
CREATE TABLE test.foo (id SERIAL PRIMARY KEY) TYPE=InnoDB;
CREATE DATABASE test2;
CREATE TABLE test2.bar (foo_id BIGINT UNSIGNED,
FOREIGN KEY (foo_id) REFERENCES test.foo(id)) TYPE=InnoDB;
SELECT * FROM test.foo f JOIN test2.bar b ON (f.id = b.foo_id);
In PostgreSQL, you can also qualify table references with a schema name. I'm not sure if you can create foreign key constraints across databases, though.
If you're looking to create constraints across RDBMSes - you can't.
I'm facing the same issue with running part of an application off PostgreSQL for where it will benefit, and the rest of MySQL where it's better.
I'm doing multiple inserts keyed off the same format of primary information (in my case a generic user ID), so I'm letting the application handle the logic of making sure to ask for the same ID from both DBs.
There's not really a clean way to do this outside of abstracting it to a class or utility function, though, that I've found.
Related
What do you think is the best approach for a PHP and SQL based web application that will be used by a number of people?
For example, say we have a table called "sales" and a user wants to access his sales. The table should contain a foreign key of the user_id or it will be better to make a separate table for each user?
Any other implementations and opinions are also welcome!
In my opinion best approach would be using two tables and refer from a foreign key. Make sure to use indexes as well. MySQL has done various optimizations to WHERE clause on a PRIMARY KEY or a UNIQUE index[1]. So you will be fine when working with considerable number of records(ex: handling 100000 records won't be a issue if you have capable hardware for database instance and optimized database configurations accordingly).
Make sure to do database optimizations based on your system to increase performance as well. Better to do in-house testing to make sure system is upto your expectations in long run.
[1] http://dev.mysql.com/doc/refman/5.7/en/where-optimizations.html
By sales I take it you mean the users invoices or billing history. You would want to separate them out into two tables using foreign keys. You could have a users table and an invoices table with the userid as the foreign key in your invoices table.
Whenever the user wanted to view their invoices, you would select rows from the invoices table where the userid is that users id.
I know, I know, putting two related tables on different databases isn't exactly the best design practice. But for whatever's sake, suppose that I have to do it absolutely. And I have to break up two foreign-key-related tables that were previously located in a database into two databases, that are located on two different servers, but I still want to maintain the database(s) integrity. What is the best way to do this?
Edit: I am using MySQL and Symfony
I can't think of any way to do this with standard MySQL.
You could write a plugin for MySQL Proxy, that manages referential integrity between the parent and child tables on different servers:
Intercept INSERT and UPDATE against child table. Query for matching row in parent table. Fail INSERT/UPDATE if no match found in parent table.
Intercept DELETE against parent table. Query for dependent rows in child table. Fail DELETE if any dependent rows found in child table. If the constraint is intended to support cascading behavior, do that instead of failing.
Intercept UPDATE against parent table. If the primary key value is changing as part of the update, query for dependent rows found in child table. Fail UPDATE if any dependent rows found in child table. If the constraint is intended to support cascading behavior, do that instead of failing.
Note that you'd have to keep information about the referential integrity constraints in your MySQL Proxy plugin (or write a custom config file for your plugin that records the relationships). You can't use conventional FOREIGN KEY syntax to declare such constraints across MySQL instances.
Have you considered Federated tables? These are basically links to tables which are hosted on a different databases on a different/same host.
You can create a federated table locally and use that to enforce referential integrity. However, I cannot overemphasize the fact that this approach is fraught with future gotchas and not at all recommended.
I have two tables in mysql. When I insert/delete values in the first table I want that the values get duplicated in table 2 to keep them "aligned".
table1:
id - username
1 - test_user
table2:
Same id as table1 and username as table1 (on insert/delete)
I want to keep the data between the tables aligned without doing multiple queries. I've read about triggers not sure if it's the correct road, i am a beninner.
I said two tables but i will need to do this in multiple tables.
You can use Mysql triggers. This way you can auto insert/update/delete datas from second table.
MySql Using Triggers
When you INSERT new records, given that you don't want to do two inserts for some reason, using a trigger to insert into the second table will work. For UPDATE and DELETE you might want to look at the CASCADE option with foreign keys. If all you are doing is keeping the data consistent between tables, that's exactly what cascade is for.
When you create table2 you just add a foreign key like this:
FOREIGN KEY (id, username)
REFERENCES table1(id, username) ON UPDATE CASCADE ON DELETE CASCADE
Then whenever you alter table1 the changes will automatically get pushed through to table2.
Couple prerequisites for this to work:
You have to use a storage engine that supports foreign keys, something like InnoDB and not MyISAM
You need to have an index on (id,username) in table1; the foriegn key needs to match a key in the parent table
You should read the doc page for foreign keys. There are a couple other ways you can tweak them, and you should figure out what works best for your purposes.
You can certainly put triggers on your table1 to make parallel changes to your other tables as your application changes table1.
See here for the documentation: http://dev.mysql.com/doc/refman/5.0/en/trigger-syntax.html
But, you should think over your design. It will take multiple queries to do your inserts and updates; they'll just be done "behind your back" on the server. They'll still take time. Triggers can really slow things down.
Also, triggers are a little bit fragile. If you add a column to a table, you'll have to rework your triggers. Triggers are generally a pain in the neck to keep in a source-control system and a huge pain in the neck to test, so using them will make your application more troublesome to maintain.
Could you think of another approach to handling this need for duplication? Could you, for example, use a view or a join to present the data you need to your application program without actually duplicating tables and the rows in them? If you figure out how to do that you'll be much happier in the long run.
CREATE VIEW table2 AS
SELECT *
FROM table1;
will produce a "fake" table2 with the contents of table1.
Or if you're hoping to view only the test users in a second table, a view can do that for you too, for example:
CREATE VIEW table3 AS
SELECT *
FROM table1
WHERE usertype = 'test_user' ;
If you're using duplicate tables for "backup," that's a bad way to make sure your information is safe. Instead, you need to back up your MySQL server instance.
Formal relational database design principles teach us to duplicating data, but instead use view and joins to structure the data the way applications need to see it.
Does anybody has experience of using partitioning feature in conjunction with the Doctrine2 library?
The first problem is that Doctrine creates foreign keys for association columns, anybody knows how to prevent or disable that?
And the second problem is how to specify custom table definition (PARTITION BY ...)?
Thanks in advance!
You're not out of luck!!
First, drop all foreign keys from all the tables D2 is managing.
Copy & execute the result of this query:
SET SESSION group_concat_max_len=8192; -- // increase this if you do not see the full list of your tables
SELECT IFNULL(REPLACE(GROUP_CONCAT('ALTER TABLE ',TABLE_NAME,' DROP FOREIGN KEY ',CONSTRAINT_NAME,'; '), ',', ''), '') FROM information_schema.TABLE_CONSTRAINTS WHERE CONSTRAINT_TYPE='FOREIGN KEY';
Then override the supportsForeignKeyConstraints() method in /vendor/doctrine-dbal/lib/Doctrine/DBAL/Platforms/MySqlPlatform.php (or wherever this class is located) to:
public function supportsForeignKeyConstraints()
{
return false;
}
This will stop Doctrine from creating foreign key constraints on your next doctrine:schema:update command. After that you can simply execute an ALTER TABLE PARTITION BY... statement where needed (D2 doesn't support partitioning on a schema level). I recommend you backup & truncate your tables first (using --no-create-info) in order to have the structure changes executed as fast as possible and then restore them.
As this fellow says here, and based on my personal experience, D2 doesn't care whether you have FKs or not, as long as the proper relation definitions are in place.
P.S.: I'm currently working on extending the annotation syntax to support proper table & column definitions, including ENGINE (this might be useful), PARTITION BY & the #Column options array (i.e. {"fixed"=true, "unsigned"=true, "default"=0})
The overall effort amounts to a couple of sleepless nights for reverse-engineering & code patches, hope you do it faster :)
PARTITION engine in MySQL has major limitations with regard to keys. Please see latest docs, currently here: http://dev.mysql.com/doc/refman/5.1/en/partitioning-limitations-partitioning-keys-unique-keys.html
If Doctrine requires keys that Partition does not support, you are out of luck. Partition engine is very limited by design - it's intended for archival storage which is infrequently read. Few MySQL-aware apps will work with Partition, unless you make changes.
I would suggest using Partition as it was intended - archiving. Store your data in a more mainstream MySQL data engine would be the answer.
I just came across the idea of writing a special database which will fit for exactly one purpose. I have looked into several other database-systems and came to the conclusion that I need a custom type. However my question is not about if it is a good idea, but how to implement this best.
The application itself is written in php and needs to write to a custom database system.
Because there can be simultaneous read/write operations I can forget the idea of implementing the database directly into my application. (correct me please if I'm wrong).
That means I have to create 2 scripts:
The database-server-script
The application.
This means that the application has to communicate with the server. My idea was using php in cli mode for the database-server. The question is, if this is effective, or if I should look into a programming language like c++ to develop the server application? The second question is then the communication. When using php in cli mode I thought about giving a serialized-array-query as a param. When using c++ should I still do it serialized? or maybe in json, or whatever?
I have to note that a database to search through can consist of several thousands of entries. So i dont know exactly if php is realy the right choice.
Secondly i have to note that queries arent strings which have to be parsed, but an array giving a key,value filter or dataset. The only maybe complexer thing the database server has to be able to is to compare strings like the MySQL version of LIKE '%VALUE%', which could be slow at several thousand entries.
Thanks for the Help.
writing a special database which will fit for exactly one purpose
I presume you mean a custom database management system,
I'm having a lot of trouble undertanding why this would ever be necessary.
Datasbes and Tables like usual databases have. But i dont have columns. Each entry can have its own columns, except for the id
That's not a very good reason for putting yourself (and your users) through a great deal of pain and effort.
i could use mysql id | serialized data... but then much fun searching over a specific parameter in a entry
So what's wrong with a fully polymorphic model implemented on top of a relational database:
CREATE TABLE relation (
id INTEGER NOT NULL auto_increment,
....
PRIMARY KEY (id)
);
CREATE TABLE col_string (
relation_id NOT NULL /* references relation.id */
name VARCHAR(20),
val_string VARCHAR(40),
PRIMARY KEY (relation_id, name)
);
CREATE TABLE col_integer (
relation_id NOT NULL /* references relation.id */
name VARCHAR(20),
val_integer INTEGER,
PRIMARY KEY (relation_id, name)
);
CREATE TABLE col_float (
relation_id NOT NULL /* references relation.id */
name VARCHAR(20),
val_float INTEGER,
PRIMARY KEY (relation_id, name)
);
... and tables for BLOBs, DATEs, etc
Or if scalability is not a big problem....
CREATE TABLE all_cols (
relation_id NOT NULL /* references relation.id */
name VARCHAR(20),
ctype ENUM('string','integer','float',...),
val_string VARCHAR(40),
val_integer INTEGER,
val_float INTEGER,
...
PRIMARY KEY (relation_id, name)
);
Yes, inserts and selecting 'rows' is more complicated than for a normal relational table - but a lot simpler than writing your own DBMS from scratch. And you can wrap most of the functionality in stored procedures. The method described would also map easily to a NoSQL db.