I'm going to build a new version of my website. The old one is written in Cakephp and for the new one I'm going to use Laravel for that I need to split some large tables into smaller ones.
An example is given above.
But till the development of the new project complete, I need to sync data between these tables.
I cant use Cron or Replication in this situation. Coz Cron job will take time, I cant update the Old code coz it will also take time.
So how do I do this?
Related
I have a number of tables with a large number of rows, some of them nearing a million. There are background tasks which keep accessing some recent records in these tables. Because of the ever increasing size, the tasks keep on taking a longer time to complete. Besides, when showing data on the front end, the calls to server also take a very long time.
Hence, I thought it is better to create a replica of such tables (as an archive) and keep saving data in these 'archive' tables (for future use if any). The idea is that whenever a record is completely processed, it will be deleted from the 'live' tables and be stored in the 'archive' tables.
PHP clone does not work as it creates an entity exactly same as the orginal.
One definite way is to follow exact same steps to create the entity, and always simultaneously keep on modifying.
Is there a better way to do this?
What you are looking for is "Partitioning". Both MySQL and Postgres have some beefy manuals.
Probably the best way to implement this is to use a daemon script that runs the partitioning queries every X time.
I came across a situation where having hundreds of database tables and rewriting them all into Laravel Migrations does not seems... a very nice task.
I know Laravel Migrations is a really cool feature top keep track of database changes among with some VCS such an GIT
BUT... Not being able to update the database with php artisan migrate in the production server technically drops away the use of migrations making it real pain... manually changing table by table adding columns, index or foreign keys.
QUESTION: Is there any way for Laravel Migrations to write the changes (SQL statements) to a file instead of doing it directly to the database?
I've come accross the same problem many times, and i did the following to solve it
First when you finally finish the mysql database/structure and you are about to publish the application you must set a "mark" on the database,export it correctly and declare that its the version 1 of database. After that you can start writing migrations and you will be more confident plus you will avoid many problems such us invalid data types, bad structure and other, others.
Another way is making use of toSql make it output under a folder lets say rawSqlDatabaseMigrations with timestamps and such.
Also you could just keep manually writing SQL and use migration only with DB::raw.
This is probably any team will encounter at some point so I'm counting on experience other guys had.
We are in a process of migrating old MySQL database to a new database with structure changed quite a bit. Some tables were split into multiple smaller tables, some data was joined from multiple smaller to one larger table.
We ran a test and it takes a few hours to migrate database to a new form. The problem is, the old database is our production database, changes every minute. We cannot have a few hours downtime.
What approach do you think would be ok in such a situation?
Let's say you have table called "users" with 1M rows. It's being changed every second. Some fields are updated, some rows are added and some rows are deleted. That's the problem why we cannot make a snapshot at certain point of time because after the migration is done, we would have 3 hours of data unsynced.
One approach I've used in the past was to use replication.
We created a replication scheme between the old production database and a slave which was used for the data migration. When we started the migration, we switched off the replication temporarily, and used the slave database as the data source for the migration; the old production system remained operational.
Once the migration script had completed, and our consistency checks had run, we re-enabled replication from the old production system to the replicated slave. Once the replication had completed, we hung up the "down for maintenance" sign on production, re-ran the data migration scripts and consistency checks, pointed the system to the new database, and took down the "down for maintenance" sign.
There was downtime, but it was minutes, rather than hours.
This does depend on your database schema to make it easy to identify changed/new data.
If your schema does not lend itself to easy querying to find new or changed records, and you don't want to add new columns to keep track of this, the easiest solution is to create separate tables to keep track of the migration status.
For instance:
TABLE: USERS (your normal, replicated table)
----------------------
USER_ID
NAME
ADDRESS
.....
TABLE: USERS_STATUS (keeps track of changes, only exists on the "slave")
-----------------
USER_ID
STATUS
DATE
You populate this table via a trigger on the USERS table for insert, delete and update - for each of those actions, you set a separate status.
This allows you to quickly find all records that changed since you ran your first migration script, and only migrate those records.
Because you're not modifying your production environment, and the triggers only fire on the "slave" environment, you shouldn't introduce any performance or instability problems on the production environment.
There's one approach I used once and that should work for you too, however you'll need to do modify your production datasets for that. Just briefly:
Add a new column named "migrated" (or so) to every table you want to migrate. Give it a boolean type. Set it to 0 by default.
When your migration script runs it has to set this flag to 1 for every entry that has been migrated to the new db. All entries that are already "1" have to be ignored. That way you won't run into synchronization issues.
That way you can run the migration script as often as you like.
You will have a downtime, but it will be just a minimal one because during that downtime you only have to migrate a few datasets (practically the last "delta" between the last run of the migration script and now).
Could you run the new database in parallel with the current one? That way you can later migrate the old data from your old db to your new one and your "live" situation will already have been captured on the new one.
What I mean is: when you write something to the old db, you will also have to write the data to the new one.
The server is a shared Windows hosting server with Hostgator. We are allowed "unlimited" MS SQL databases and each is allowed "unlimited" space. I'm writing the website in PHP. The data (not the DB schema, but the data) needs to be versioned such that (ideally) my client can select the DB version he wants from a select box when he logs in to the website, and then (roughly once a month) tag the current data, also through a simple form on the website. I've thought of several theoretical ways to do this and I'm not excited about any of them.
1) Put a VersionNumber column on every table; have a master Version table that lists all versions for the select box at login. When tagged, every row without a version number in every table in the db would be duplicated, and the original would be given a version number.
This seems like the easiest idea for both me and my client, but I'm concerned the db would be awfully slow in just a few months, since every table will grow by (at least) its original size every month. There's not a whole lot of data, and there probably never will be, in any one version. But multiplying versions in the same table just scares me.
2) Duplicate the DB every time we tag.
It looks like this would have to be done manually by my client since the server is shared, so I already dislike the idea. But in addition, the old DBs would have to be able to work with the current website code, and as changes are made to the DB structure over time (which is inevitable) the old DBs will no longer work with the new website code.
3) Create duplicate tables (with the version in their name) inside the same database every time we tag. Like [v27_Employee].
The benefit here over idea (1) would be that no table would get humongous in size, allowing the queries to keep up their speed, and over idea (2) it could theoretically be done easily through the simple website tag form rather than manually by my client. The problems are that the queries in my PHP code are going to get all discombobulated as I try to explain which Employee table is joining with which Address table depending upon which version is selected, since they all have the same name, but different; and also that as the code changes, the old DB tables no longer match, same problem as (2).
So, finally, does anyone have any good recommendations? Best practices? Things they did that worked in the past?
Thanks guys.
Option 1 is the most obvious solution because it has the lowest maintenance overhead and it's the easiest to work with: you can view any version at any time simply by adding #VersionNumber to your queries. If you want or need to, this means you could also implement option 3 at the same time by creating views for each version number instead of real tables. If your application only queries one version at a time, consider making the VersionNumber the first column of a clustered primary key, so that all the data for one version is physically stored together.
And it isn't clear how much data you have anyway. You say it's "not a whole lot", but that means nothing. If you really have a lot of data (say, into hundreds of millions of rows) and if you have Enterprise Edition (you didn't say what edition you're using), you can use table partitioning to 'split' very large tables for better performance.
My conclusion would be to do the simplest, easiest thing to maintain right now. If it works fine then you're done. If it doesn't, you will at least be able to rework your design from a simple, stable starting point. If you do something more complicated now, you will have much more work to do if you ever need to redesign it.
You could copy your versionable tables into a new database every month. If you need to do a join between a versionable table and a non-versionable table, you'd need to do a cross-schema join - which is supported in SQL Server. This approach is a bit cleaner than duplicating tables in a single schema, since your database explorer will start getting unwieldy with all the old tables.
What I finally wound up doing was creating a new schema for each version and duplicating the tables and triggers and keys each time the DB is versioned. So, for example, I had this table:
[dbo].[TableWithData]
And I duplicated it into this table in the same DB:
[v1].[TableWithData]
Then, when the user wants to view old tables, they select which version and my code automatically changes every instance of [dbo] in every query to [v1]. It's conceptually fairly simple and the user doesn't have to do anything complicated to version -- just type in "v1" to a form and hit a submit button. My PHP and SQL does the rest.
I did find that some tables had to remain separate -- I made a different schema called [ctrl] into which I put tables that will not be versioned, like the username / password table for example. That way I just duplicate the [dbo] tables.
Its been operational for a year or so and seems to work well at the moment. They've only versioned maybe 4 times so far. The only problem I seem to have consistently that I can't figure out is that triggers seem to get lost somehow. That's probably a problem with my very complex PHP rather than the DB versioning concept itself though.
I have just taken over a project for a client and the database schema is in a total mess. I would like to rename a load of fields make it a relationship database.
But doing this will be a painstaking process as they have an API running of it also. So the idea would be to create a new database and start re-writing the code to use this instead. But I need a way to keep these tables in sync during this process.
Would you agree that I should use MYSQL EVENT's to keep updating the new table on Inserts / updates & deletes??
Or can you suggest a better way??
Hope you can advise !!
thanks for any input I get
I had the same problem in my project. I did the same thing like you - writing the whole database new. We developed the new database and the fitting code and after finishing that work we made a migration script (small application) which did the migration from old to new database.
Hope this gives you some ideas...
I had the same problem as well, and went by the way of duplicating data at the point of user input (basically saved to both databases at once, since the original schema lacked all the required information). After several months of development the customer realized that he is going to move to a completely new database and considered it too risky (and I agree with him).
Thus, I would advice to utter clearly to your customer that you are going to implement a new database instead of iterative refactoring of the current one.