I have 2 Databases that basically have the same structure, but different data. (The older Database has 2 extra tables) Is there any way for me to combine the 2 Databases into one, with the extra tables).
Is there any easy way to do this? Importing a dump of the old Database into the new one throws errors.
Notes:
I can SSH into the server to combine the databases, I can also use PHPMyAdmin.
1005/121 is usually a foreign key violation. If the table you're loading is a target of (or has) any foreign keys, most likely the corresponding parent (or child) records aren't available, killing the load.
Related
I'm creating a database on mysql for a small app.
Problem is there are too many fields that are identical on different Tables like
Table 1: Muncipal Issues:
ID,
UserID,
Title,
Location,
Description,
ImageURL,
Table 2: Harrasement Issues:
ID ,
UserID,
Title,
Location,
Description,
ImageURL
Tables 3 same as above
both tables have almost same coulmns.
i want to ask if it's better to use a relations and create a table for handling IDs and link it with other details or it's better to create a single table with an extra coulmn for these issues.
on one hand there'll be too many tables with identical columns.
on the other hand there'll few tables with too many rows in it.
What will be best for performance more rows or more tables.
i'm using Mysql.
Firstly, unless you expect millions of records don't care that much about performance but care more about the structure of your data and how easy it will be to access it. Literally write down a list of data that you plan to extract in your app e.g. "find all issues today", "find all unresolved issues older than 6 months" and then try to build real SQL queries on your expected structure. If they're going hard try to change the structure.
To answer your question: it depends. The current structure has following benefits:
It's easy to query certain type of issues
It's easy to build a PHP application - just make one template form (or model) and then copypaste it with slight changes for other tables
In case of performance problems it may be easier to create a cluster by simply putting each table on the different db server.
and following downsides:
It's inflexible. Adding new field that you forgot to add in the beginning will be painful since you'll have to change 3 (or more) tables and then the same amount of pieces in your app.
Adding new types of issues will be painful and require creating new table.
Creating SQL-s for getting data like "all non-resolved issues (regardless of type)" will require complicated UNION-s. Moreover this UNIONS will require creating virtual field with issue type otherwise you can't tell from which table did certain id come.
The classical db approach recommends using one table for common fields and create derived tables for fields that are different. So:
issues table should have all common fields and is identified by PK issue_id
municipal_issues uses the foreign key to issues.issue_id and has only the specific fields
harassment_issues uses the foreign key to issues.issue_id and has only the specific fields
also the issues table has the issue_type field that takes values "harassment", "municipal" etc and helps finding the table where the additional data are stored.
This pattern is called "Class Table inheritance" and you may check out the SQL antipatterns presentation for more info and other approaches. This solves the flexibility issue and still allows re-creating each of the original tables with only one simple JOIN that goes pretty fast.
Also as a side note you may look into the db schema of bug-trackers like Mantis since this looks like the same domain.
I know, I know, putting two related tables on different databases isn't exactly the best design practice. But for whatever's sake, suppose that I have to do it absolutely. And I have to break up two foreign-key-related tables that were previously located in a database into two databases, that are located on two different servers, but I still want to maintain the database(s) integrity. What is the best way to do this?
Edit: I am using MySQL and Symfony
I can't think of any way to do this with standard MySQL.
You could write a plugin for MySQL Proxy, that manages referential integrity between the parent and child tables on different servers:
Intercept INSERT and UPDATE against child table. Query for matching row in parent table. Fail INSERT/UPDATE if no match found in parent table.
Intercept DELETE against parent table. Query for dependent rows in child table. Fail DELETE if any dependent rows found in child table. If the constraint is intended to support cascading behavior, do that instead of failing.
Intercept UPDATE against parent table. If the primary key value is changing as part of the update, query for dependent rows found in child table. Fail UPDATE if any dependent rows found in child table. If the constraint is intended to support cascading behavior, do that instead of failing.
Note that you'd have to keep information about the referential integrity constraints in your MySQL Proxy plugin (or write a custom config file for your plugin that records the relationships). You can't use conventional FOREIGN KEY syntax to declare such constraints across MySQL instances.
Have you considered Federated tables? These are basically links to tables which are hosted on a different databases on a different/same host.
You can create a federated table locally and use that to enforce referential integrity. However, I cannot overemphasize the fact that this approach is fraught with future gotchas and not at all recommended.
So. I'm building a multi-tenant Laravel SaaS web-app, and am a little stuck when it comes to the database design. I have been searching around trying to find a solution, but i really can't decide on which one to go with. I really hope some of you with more experience and knowledge than me can come up with some advice. Sorry about the long post, but i hope you’ll hang in.
Problem:
In the app my users will be importing data from an external database of their own (with a know schema).
E.g.: I will be importing products with realtions to categories. The easiest way would just be to import the external product_id to the new primary key of the product.
BUT as the users product_id’s will probably conflict, i will have to assign each product with a new primary key, while still keeping the external product_id for reference when syncing back to the external db.
E.g.: external product_id will be ext_product_id and i will assign a new product_id as a primary key.
As of now i can think of 3 ways to do this:
Solution 1 - Single database with new primary keys:
So if i import a list of products and categories i will have to save each external product_id as ext_product_id and assign a new primary key to the product. I will then have to query the categories ext_category_id = the products ext_category_id and then create a new relation with the new primary key product_id and primary key category_id.
These looping queries takes forever when importing several thousands of rows, and some tables has 4 different relations which mean a lot of “ext_” columns to keep track of and sync.
Solution 2 - composite primary key:
As each user will have no reference to an external database i could create composite keys consisting of the tenant_id and e.g. the external product_id. This would allow my to just batch insert the external data with a key prefix consisting of the tenant. This way the relations should be working "out of the box".
But Laravel doesn't support the feature as far as i understand? Any ideas?
Solution 3 - multiple databases:
To create a separate database for each tenant would probably be the best solution performance and sanity wise (to begin with), as i would just be able to copy/batch insert the external database, and the relations would be working right away.
But i'm really worried about the scalability of this design: How many databases would i realistically be able to manage? Say i have 1000 or even 10000 customers?
What if i want to add a column in an update - would i be able to perform some kind of loop-migration to all databases?
I really hope that some of you can help me move on with this as i am stuck and have no experience with solution 2 and 3.
Thanks in advance!
I would personally go for Solution 2 as that is probably the safest.
Solution 1 should be ruled out since you don't want to confuse the users of your application by modifying their data.
Solution 3 would probably be a pain to maintain and is more likely to fail (back-end of the application) + you will lose all track of whose database it is.
As for solution 2 that seems to me like the ideal one:
I don't know what you are using (PHPMyAdmin or another type) but basically what you want to do is have 2 columns:
table
id(PK, AI) original_id(PK)
and then just the rest of your table.
Like this you will have your own Auto Increment (AI) key and you won't get any conflicts from your users since the combination of your auto_increment and that of the user is going to ALWAYS be unique.
for example:
user1:
id = 1 | original_id = 1
user2:
id = 2 | original_id = 1
This still works because the combination is unique.
Another pro of using this composite UID is that you can still use your own id to perform queries or actions on the desired rows etc...
Hope this helps
There are many things to consider when choosing an architecture, but from what you've described, I suggest you use Solution 3 because:
as you've very well pointed out, it's the best solution performance wise (especially if you end up with a lot of customers) and you won't need to handle the overhead of having large amounts of entries for all customers in one table
you have a clear database structure where only the necessary relations are present, no extra fuss to track different customers
As far as maintaining and updating database structure, you can can create Laravel Commands to automate running migrations for multiple databases. You can have a look at this answer to get an idea of how you could do that (although that situation is a little different from what you'll be needing, it offers some insight). Also anything else that needs to be handled in batch can be automated via Laravel commands or other scripts, so the amount of databases should not hinder maintenance.
A more modern way of doing this is to use UUID as primary keys. If you also,
when you import data have a source_uuid, import_time etc, in the table you can bookkeep all import (and export).
It might be hard to convince all parties to use UUID - but that is the best way go.
/gh
I am doing a tracking site like google adwords. I am doing only modifications. In that site i have seen they are creating tables for each month and merging that table into one this is done for the tables which stores information about clicks and search details and the merged tables having crores of records.And for querying they have used only the merged table which is having crores of records. Is there any advantage of using tables like this?And the query is taking more than 10 minutes to execute.
Advantages are as follows, If you dont get the following advantages then do not use it.
Easily manage a set of log tables. For example, you can put data from different months into separate tables, compress some of them with myisampack, and then create a MERGE table to use them as one.
Obtain more speed. You can split a large read-only table based on some criteria, and then put individual tables on different disks. A MERGE table structured this way could be much faster than using a single large table.
Perform more efficient searches. If you know exactly what you are looking for, you can search in just one of the underlying tables for some queries and use a MERGE table for others. You can even have many different MERGE tables that use overlapping sets of tables.
Perform more efficient repairs. It is easier to repair individual smaller tables that are mapped to a MERGE table than to repair a single large table.
Instantly map many tables as one. A MERGE table need not maintain an index of its own because it uses the indexes of the individual tables. As a result, MERGE table collections are very fast to create or remap. (You must still specify the index definitions when you create a MERGE table, even though no indexes are created.)
If you have a set of tables from which you create a large table on demand, you can instead create a MERGE table from them on demand. This is much faster and saves a lot of disk space.
Exceed the file size limit for the operating system. Each MyISAM table is bound by this limit, but a collection of MyISAM tables is not.
You can create an alias or synonym for a MyISAM table by defining a MERGE table that maps to that single table. There should be no really notable performance impact from doing this (only a couple of indirect calls and memcpy() calls for each read).
You can read more about this on basic info link, advantages disadvantages link.
A MRG_MYISAM only works over MyISAM tables, which are, by themselves, not the first option for a table. You would normally go for InnoDB tables.
The MRG_MYISAM engine was invented before MySQL had support for views and for partitions. Range partitioning (e.g. partition per month) is most probably what you want.
Partitioning is transparent to the user in terms of queries, but nevertheless uses pruning so as to only read from selected partitions for a query, thus optimizing it.
I would recomment that you use InnoDB tables, and check out range partitioning.
MRG_MYISAM and MyISAM are still in use. They could work out for you. It's just that MyISAM introduces so much trouble (no crash recovery, table level locking, more...) that it's many times out of the question.
What are your methods of linking data spread over multiple databases architectures (think MySQL vs PostgreSQL etc), into a single application?
Would you create giant hashtables/arrays to match content against one another? Are there other, more effective and less memory-consuming options for doing this?
If you were to use data both from a MySQL & PostgreSQL source, with no way of converting one DB to the other (application constraints, lack of time, lack of knowledge, ... ), how would you go about it?
SQL Relay or another sql proxy.
http://sqlrelay.sourceforge.net/
At least in the case of MySQL, you can use data from multiple databases in a single query anyway, provided the databases are hosted by the same MySQL Server instance. You can distinguish tables from different databases by qualifying the table with a schema name:
CREATE TABLE test.foo (id SERIAL PRIMARY KEY) TYPE=InnoDB;
CREATE DATABASE test2;
CREATE TABLE test2.bar (foo_id BIGINT UNSIGNED,
FOREIGN KEY (foo_id) REFERENCES test.foo(id)) TYPE=InnoDB;
SELECT * FROM test.foo f JOIN test2.bar b ON (f.id = b.foo_id);
In PostgreSQL, you can also qualify table references with a schema name. I'm not sure if you can create foreign key constraints across databases, though.
If you're looking to create constraints across RDBMSes - you can't.
I'm facing the same issue with running part of an application off PostgreSQL for where it will benefit, and the rest of MySQL where it's better.
I'm doing multiple inserts keyed off the same format of primary information (in my case a generic user ID), so I'm letting the application handle the logic of making sure to ask for the same ID from both DBs.
There's not really a clean way to do this outside of abstracting it to a class or utility function, though, that I've found.