I'm thinking about building a php script that flushes, locks and copys a mysql data folder. Because I need to lock the tables and typical dump takes 5 minutes plus, I was thinking if I do a flush, lock and file copy of the data folder, it should be quicker. Anyone any experience of this and know if this is a viable solution?
Look to a XtraBackup also if you are planning to do non-stop backups of your data.
The MySQL devs are way ahead of you. The rolled your exact method with some syntactic sugar and proper error checking in a command called mysqlhotcopy.
Might be too late now but what phase does all this break down into? If your server is spending most of the five minutes copying the files instead of the actual flushing then your problem is simply a slow disk.
I think the best answer to the question is the following Windows command:
set bin=C:\Program Files\MySQL\MySQL Server 5.6\bin
"%bin%/mysql" -e "FLUSH TABLES WITH READ LOCK; UNLOCK TABLES;" --user=UserName --password=Password DatabaseName
This quickly forces all MySQL data for a database to its MySQL three or four files in the MySQL data folder, from which they can be copied to some other folder. You'll have to customize this command for your particular version of MySQL and for your database and admin user.
I couldn't get the other answers to work, but this BAT/CMD command works fast and well in my experience.
The only other suggestion I can make is to use the MySQL Workbench (which comes with MySQL) to Stop the MySQL server. When it is stopped it is flushed to disk. Don't forget to Start your MySQL server when you are finished using the files directly (at which time MySQL reads the database files from disk).
Note: if you simply copy the data files for a database to the data folder of another instance of MySQL that is already open, you won't see the data in MySQL applications! You would think that MySQL would check the date/time modified of the disk files to detect updates, but it doesn't. And it doesn't keep its files locked, showing Windows that they are in use, which they are. So strange.
Related
Despite many questions regarding this MySQL error, I haven't found a solution to my problem.
My (PHP) application requires the generation of a DAL (database access layer). This is done through a php script called from the command line. This script reads all tables from the information schema and generates files according to that schema.
This has been working for as long as I can remember, until recently. Every time I start this generation process, my 'normal' application loses connection to the MySQL server.
The strange thing is, that I can however connect to the database using a program such as Sequel Pro. Same database, same connection settings, same machine.
Also, when I try to restart MySQL on the command line using sudo /usr/local/mysql/support-files/mysql.server restart I get the error:
ERROR! MySQL server PID file could not be found
Then Starting MySQL takes forever before I kill it. The only solution I found is to completely reboot my computer. It then works fine again until I rerun the generation script.
So, résumé, I've got 2 problems:
When I run the script using the information schema, my normal application loses the ability to connect to the MySQL server, while other applications still can.
When that happens, I'm unable to restart MySQL to fix the problem.
The second problem is obviously way less important than the first one. I'd rather not have to restart MySQL at all.
What the generation script does:
First:
SELECT *
FROM INFORMATION_SCHEMA.TABLES
WHERE
TABLE_SCHEMA=DATABASE() %s AND SUBSTRING(TABLE_NAME, 1, 1)!='_'"
Then a loop in PHP to loop over all tables.
Then this:
SELECT *
FROM INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_SCHEMA=DATABASE()
AND TABLE_NAME=:tableName
The rest of the script then generates PHP files from that information.
I am trying to make a complete file & mySQL backup of my site each night.
I was thinking the best way of doing it would be to have a cronjob run each night that would login to a remote server and replicate all of the local files.
Then, I need to figure out a way to take backups of all the mysql databases (currently there are three) and upload them all to the remote server as well.
This sounds like a huge project and I don't know whether to reinvent the wheel here, or if there is some script out there which basically does the same thing already.
Use cronjob to run a bash script
mysqldump the databases
tar -cvf the files
wput it all to your remote server
Also you can set a variable like now=$(date +"%Y_%m_%d") to use in your file names
You can use the mysqldump command to backup the database to a file and then upload to a different server
http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html
have you thought about MySQL replication? Maybe that fits better to your needs and you doesn't need php to do it
http://dev.mysql.com/doc/refman/5.5/en/replication.html
Ok, i have searched high and low and have been unable to find an answer that works for what i am trying to do. I have two databases, we'll call them DB1 and DB2. I have a cron job that runs every night at 4am that backs up DB1 and stores its data in a SQL file archive, we'll call this file db_backup.sql. The file is stored in a folder on the server, we'll call it ROOT/backups/db_backup.sql.
Info
database names: DB1 and DB2
backup filename: db_backup.sql
backup file path: ROOT/backups/db_backup.sql
What i'm trying to do:
I want to use the db_backup.sql file to build DB2. I am basically trying to set up database replication where i replicate DB1 out to DB2. Don't know of any other way to do this on shared hosting servers than what i'm trying to explain. I am trying to use php to import the db_backup.sql file into DB2.
My Environment:
The website and databases are on a shared hosting account with godaddy (yes, i would love to get dedicated servers to set up real replication, but can't afford it for now). The databases are mysql in phpmyadmin.
Is this something that is possible? Any help would be greatly appreciated! Let me know if you have any questions as well.
I'm not sure if I understand your problem. You need to copy the db_backup file to the second host where you have access to the database and load the sql file.
From a shell:
mysql -hhost -uusername -ppassword databasename < db_backup.sql
This will restore the tables on the second machine.
Should be as simple as, setting up a CRON job on server 2 to call a script on Server 1 which dishes out the SQL file, then that cron job script would import / rebuild the DB.
Make sure to require a hash (via get or post) before giving out the DB SQL on server 1, or else anyone could read it / have your database dump.
Can you not avoid PHP altogether and connect directly to the database remotely to perform the backup either via the command line or using a scripting language more suitable for long running processes?
I have a backup created of my mysql database (vbulletin forum v3.8) everyday. It's about 360mb in size. It's stored as one text file in a secure folder.
I'm thinking of getting another server, through a different host, and somehow automatically transferring the backup to my second server every day.
Any ideas on how I could automate this process? I'm thinking PHP and a cron job.
Cron definately. Php, if you like it, but using bash with mysqldump combined with gzip works wonders.
Schedule rsync to transfer the files (over ssh) with cron (if you're on Linux).
Cron + rsync may be your best bet. If the file is text as you say and the changes are "diff"able then rsync can be used to transfer only the updates to that file. For example, the crontab could look something like this:
20 4 * * * rsync -a --delete source/ username#remotemachine.com:/path/to/destination/
This will sync the remote machine once a day deleting any files in the remote copy that no longer exist on the source machine.
As a note, I just read again and noticed this is a mysql backup so they output of the dump could eventually contain binary and in that case you probably want to just use a replication server or copy the file whole each day. rsync could be used for the copy as well...
slightly different approach and technically not a backup solution, but might want to consider running mysql in a replication mode and replicating changes live. so if the worst happpens you will be up to date with the data and not a day behind.
I have a fairly small MySQL database (a Textpattern install) on a server that I do not have SSH access to (I have FTP access only). I need to regularly download the live database to my local dev server on demand; i.e., I would like to either run a script and/or have a cron job running. What are some good ways of doing this?
Some points to note:
Live server is running Linux, Apache 2.2, PHP 5.2 and MySQL 4.1
Local server is running the same (so using PHP is an option), but the OS is Windows
Local server has Ruby on it (so using Ruby is a valid option)
The live MySQL db can accept remote connections from different IPs
I cannot enable replication on the remote server
Update: I've accepted BlaM's answer; it is beautifully simple. Can't believe I didn't think of that. There was one problem, though: I wanted to automate the process, but the proposed solution prompts the user for a password. Here is a slightly modified version of the mysqldump command that passes in the password:
mysqldump -u USER --password=MYPASSWORD DATABASE_TO_DUMP -h HOST > backup.sql
Since you can access your database remotely, you can use mysqldump from your windows machine to fetch the remote database. From commandline:
cd "into mysql directory"
mysqldump -u USERNAME -p -h YOUR_HOST_IP DATABASE_TO_MIRROR >c:\backup\database.sql
The program will ask you for the database password and then generate a file c:\backup\database.sql that you can run on your windows machine to insert the data.
With a small database that should be fairly fast.
Here's what I use. This dumps the database from the live server while uploads it to the local server.
mysqldump -hlive_server_addresss -ulive_server_user -plive_server_password --opt --compress live_server_db | mysql -ulocal_server_user -plocal_server_password local_server_db
You can run this from a bat file. You can ever use a scheduled task.
Is MySQL replication an option? You could even turn it on and off if you didn't want it constantly replicating.
This was a good article on replication.
I would create a (Ruby) script to do a SELECT * FROM ... on all the databases on the server and then do a DROP DATABASE ... followed by a series of new INSERTs on the local copy. You can do a SHOW DATABASES query to list the databases dynamically. Now, this assumes that the table structure doesn't change, but if you want to support table changes also you could add a SHOW CREATE TABLE ... query and a corresponding CREATE TABLE statement for each table in each database. To get a list of all the tables in a database you do a SHOW TABLES query.
Once you have the script you can set it up as a scheduled job to run as often as you need.
#Mark Biek
Is MySQL replication an option? You could even turn it on and off if you didn't want it constantly replicating.
Thanks for the suggestion, but I cannot enable replication on the server. It is a shared server with very little room for maneuver. I've updated the question to note this.
Depending on how often you need to copy down live data and how quickly you need to do it, installing phpMyAdmin on both machines might be an option. You can export and import DBs, but you'd have to do it manually. If it's a small DB (and it sounds like it is), and you don't need live data copied over too often, it might work well for what you need.