Migrating databases using phpMyAdmin's tracking mechanism - php

Question

In a development database, I have phpMyAdmin Tracking enabled on all tables. It logs all the changes I make to the tables' structures (in this case I'm not interested in data tracking.) So far so good.
What I want to do then is to take out a report, for ALL tracked tables, with the changes made from a specific version (or a date would even work,) so that I can run the resulting SQL on my production database, when upgrading to new versions, and make sure that the databases are identical, without the worry of the errors that come with manual handling of this.
However, there is no function that I can find that generates such a report. All the tracking reports are for individual tables, and if I have to click through all tables (20+) it takes away the benefit of this function. All tables don't change, but I don't want to keep track of what's changed, that's what I want phpMyAdmin to do for me.
I have tried to make my own query against the pma_tracking table where the changes are stored, and had partial success. The problem is that all changes for one version are stored as one BLOB, and with each new version a DROP TABLE / CREATE TABLE statement is made, and I can't drop tables on the production db since there is data there (I'm not recreating the database every time, only adding incremental changes). I just want to upgrade the structure, and the only time I want CREATE TABLE statements is when I actually create a new table in the database. So I thought I could filter those out with SQL, but then it's stored as a blog, and then I would have to parse and mess with the blob text which seems overly complicated.
So, as a summary, this is what I'm looking for:
An automated tracking system/workflow that logs all structure updates, and can create incremental SQL reports for the whole database from a version or point in time.
I'd prefer to not use any additional third party apps (I'd like to use phpMyAdmin or MySQL only), if possible
Also, I would love comments on the workflow, if someone has ideas of a better one. Any help appreciated.

Answer 1

The algorithm for parsing the BLOB field of the "pma_tracking" table is located in the getTrackedData method of the PMA_Tracker class, in the libraries/Tracker.class.php source file.
Starting from that code, I've written a simple PHP script to extract all the data definition statements (except the "DROP TABLE" statements) from the "pma_tracking" table.
For example, suppose that you want to get the list of all the changes of all the tables of the "test" database since version "1":
<?php
$link = mysqli_init();
// Adjust hostname, username, password and db name before use!
$db = mysqli_real_connect($link, "localhost", "myuser", "mypass", "phpmyadmin")
or die(mysqli_connect_error());
// Adjust also target db name and tracking version
$db_name = "test";
$version = "1";
$sql = "SELECT schema_sql FROM pma_tracking
WHERE db_name='{$db_name}' AND version>='{$version}'
ORDER BY version,date_created";
$result = mysqli_query($link, $sql) or die(mysqli_error($link));
while ($myrow = mysqli_fetch_assoc($result)) {
$log_schema_entries = explode('# log ', $myrow['schema_sql']);
foreach ($log_schema_entries as $log_entry) {
if (trim($log_entry) != '') {
$statement = trim(strstr($log_entry, "\n"));
if (substr($statement, 0, 11) != "DROP TABLE ") {
echo "{$statement}\n";
}
}
}
}
?>
By redirecting the script output on a file, you'll obtain a SQL commands file with (almost) all the statements needed to replicate the schema changes on the target (eg. production) database; this file must be executed by specifying the "-f" (force) MySQL option:
-f, --force Continue even if we get an SQL error.
By doing so, MySQL will ignore all the "Table already exists" error that will be thrown each time that a CREATE TABLE statement for an existing table is encountered, thus creating only the tables that still does'nt exist in the target database.
This kind of approach obviously has some drawbacks:
ALL the DROP TABLE commands will be ignored (not only those automatically inserted from phpMyAdmin) so, if you have deleted a table in the source database, that table won't be deleted in the target database.
ALL the script errors will be ignored, so it may not be 100% affordable.
A final word of advice: always do a full backup of your target database before proceeding!

Answer 2

I don't know how you could solve this problem using phpMyAdmin, but there are other tools that might help you achieve the effect your looking for. Liquibase is one of them. I've used it some times in the past and it was pretty good. It takes a little to get the hang of it, but I think it might help you.

Answer 3

I'm not too familiar with SQL tools, so I cannot recommend anything to help you out there, but I can try and help with a custom workflow...
Create a table called structure_log
Create a PHP script called print_stucture.php that prints whatever info you desire to a file on the server, saves the file as a timestamp (this will be your version number), and saves the name in the structure_log table
Create a crontab that runs print_structure.php however often you desire
Create a PHP script called delete_dups.php that grabs the last two records from your structure_log table, compares those two files, and if they are the same (representing no change to structures), deletes the one with the latest timestamp (filename) and removes that record from the structure_log table
Create a crontab that runs delete_dups.php half as often as the one that runs print_structure.php
This will make a versioning folder on your server. You can manually run the print_structure.php script whenever you desire and compare it against the latest version log you have in your server folder to see if your database you just ran it on, is the same as the last time the version check was ran.

Answer 4

I've had some success with MySQL Workbench:
Import (reverse engineer) your dev database into workbench. You can do this by either exporting your schema to an SQL file and loading it into workbench, or workbench will get the schema directly from the server.
Next, generate your diff file with the "Synchronise model" option. You select the production database, then which tables to sync, and workbench generates an SQL file you can run to sync both models.
A word of caution: the first time, there will likely be quite a few apparently uneeded changes while the DB is updated to workbench "style". For subsequent updates, the tool is rather reliable, though I would never let an automated tool have free range over my production DB ;-)
Always check the SQL file for errors, in some cases, dropping a column then adding another of the same name but different type will generate an alter column which will fail.

Answer 5

I don't have anything that creates an incremental diff between two databases but here's the script I use to compare two MySQL databases:
<?php
//------------------------------------------------------------------------------
// Define the variables we'll be using.
//------------------------------------------------------------------------------
$db1_con = NULL;
$db1_constraints = array();
$db1_dbname = 'db1';
$db1_host = 'localhost';
$db1_password = 'password1';
$db1_tables = array();
$db1_username = 'username1';
$db2_con = NULL;
$db2_constraints = array();
$db2_dbname = 'db2';
$db2_host = '123.123.123.123';
$db2_password = 'password2';
$db2_tables = array();
$db2_username = 'username2';
//------------------------------------------------------------------------------
// Connect to the databases.
//------------------------------------------------------------------------------
try{
$db1_con = new PDO("mysql:host=$db1_host;dbname=information_schema", $db1_username, $db1_password);
$db1_con->setAttribute(PDO::ATTR_EMULATE_PREPARES, FALSE); // Try to use the driver's native prepared statements.
$db1_con->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION); // Let's use exceptions so we can try/catch errors.
}catch(PDOException $e){
echo "<p>Connection failed for $db1_host: " . $e->getMessage() . '</p>';
exit;
}
try{
$db2_con = new PDO("mysql:host=$db2_host;dbname=information_schema", $db2_username, $db2_password);
$db2_con->setAttribute(PDO::ATTR_EMULATE_PREPARES, FALSE); // Try to use the driver's native prepared statements.
$db2_con->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION); // Let's use exceptions so we can try/catch errors.
}catch(PDOException $e){
echo "<p>Connection failed for $db2_host: " . $e->getMessage() . '</p>';
exit;
}
if (NULL !== $db1_con && NULL !== $db2_con){
echo "<h2>Column Analysis</h2>";
$sql = 'SELECT * FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = ? ORDER BY TABLE_NAME, ORDINAL_POSITION';
$statement1 = $db1_con->prepare($sql);
$statement1->bindValue(1, $db1_dbname);
$statement2 = $db2_con->prepare($sql);
$statement2->bindValue(1, $db2_dbname);
if (TRUE === $statement1->execute()){
while ($row = $statement1->fetch(PDO::FETCH_ASSOC)){
$db1_tables[$row['TABLE_NAME']][$row['COLUMN_NAME']] = array();
foreach ($row AS $key => $value){
$db1_tables[$row['TABLE_NAME']][$row['COLUMN_NAME']][$key] = $value;
}
}
}
if (TRUE === $statement2->execute()){
while ($row = $statement2->fetch(PDO::FETCH_ASSOC)){
$db2_tables[$row['TABLE_NAME']][$row['COLUMN_NAME']] = array();
foreach ($row AS $key => $value){
$db2_tables[$row['TABLE_NAME']][$row['COLUMN_NAME']][$key] = $value;
}
}
}
foreach ($db1_tables AS $table => $info){
if (!isset($db2_tables[$table])){
echo "<p>Table <strong>$table</strong> does not exist in the SECOND database!</p>";
}else{
foreach ($info AS $column => $data){
if (!isset($db2_tables[$table][$column])){
echo "<p>Column <strong>$column</strong> does not exist in table <strong>$table</strong> in the SECOND database!</p>";
}else{
if (count($data)){
foreach ($data AS $key => $value){
if ($db1_tables[$table][$column][$key] !== $db2_tables[$table][$column][$key]){
echo "<p>Column <strong>$column</strong> in table <strong>$table</strong> has differing characteristics for <strong>$key</strong> (". $db1_tables[$table][$column][$key] ." vs. ". $db2_tables[$table][$column][$key] .")</p>";
}
}
}
}
}
}
}
foreach ($db2_tables AS $table => $info){
if (!isset($db1_tables[$table])){
echo "<p>Table <strong>$table</strong> does not exist in the FIRST database!</p>";
}else{
foreach ($info AS $column => $data){
if (!isset($db1_tables[$table][$column])){
echo "<p>Column <strong>$column</strong> does not exist in table <strong>$table</strong> in the FIRST database!</p>";
}else{
if (count($data)){
foreach ($data AS $key => $value){
if ($db2_tables[$table][$column][$key] !== $db1_tables[$table][$column][$key]){
echo "<p>Column <strong>$column</strong> in table <strong>$table</strong> has differing characteristics for <strong>$key</strong> (". $db2_tables[$table][$column][$key] ." vs. ". $db1_tables[$table][$column][$key] .")</p>";
}
}
}
}
}
}
}
echo "<h2>Constraint Analysis</h2>";
$sql = 'SELECT * FROM information_schema.KEY_COLUMN_USAGE WHERE TABLE_SCHEMA = ? ORDER BY TABLE_NAME, ORDINAL_POSITION';
$statement1 = $db1_con->prepare($sql);
$statement1->bindValue(1, $db1_dbname);
$statement2 = $db2_con->prepare($sql);
$statement2->bindValue(1, $db2_dbname);
if (TRUE === $statement1->execute()){
while ($row = $statement1->fetch(PDO::FETCH_ASSOC)){
foreach ($row AS $key => $value){
$db1_constraints[$row['TABLE_NAME']][$row['COLUMN_NAME']][$key] = $value;
}
}
}
if (TRUE === $statement2->execute()){
while ($row = $statement2->fetch(PDO::FETCH_ASSOC)){
foreach ($row AS $key => $value){
$db2_constraints[$row['TABLE_NAME']][$row['COLUMN_NAME']][$key] = $value;
}
}
}
foreach ($db1_constraints AS $table => $info){
foreach ($info AS $column => $data){
if (isset($db2_constraints[$table][$column])){
if (count($data)){
foreach ($data AS $key => $value){
if ('CONSTRAINT_NAME' !== $key && $db1_constraints[$table][$column][$key] !== $db2_constraints[$table][$column][$key]){
echo "<p>Column <strong>$column</strong> in table <strong>$table</strong> has differing characteristics for <strong>$key</strong> (". $db1_constraints[$table][$column][$key] ." vs. ". $db2_constraints[$table][$column][$key] .")</p>";
}
}
}
}else{
echo "<p>Column <strong>$column</strong> in table <strong>$table</strong> is missing a constraint in the SECOND database!</p>";
}
}
}
foreach ($db2_constraints AS $table => $info){
foreach ($info AS $column => $data){
if (isset($db1_constraints[$table][$column])){
if (count($data)){
foreach ($data AS $key => $value){
if ('CONSTRAINT_NAME' !== $key && $db2_constraints[$table][$column][$key] !== $db1_constraints[$table][$column][$key]){
echo "<p>Column <strong>$column</strong> in table <strong>$table</strong> has differing characteristics for <strong>$key</strong> (". $db2_constraints[$table][$column][$key] ." vs. ". $db1_constraints[$table][$column][$key] .")</p>";
}
}
}
}else{
echo "<p>Column <strong>$column</strong> in table <strong>$table</strong> is missing a constraint in the FIRST database!</p>";
}
}
}
}
?>
Edited to add code that shows differences in constraints as well.