So im new to exec and recently wanted to auutomate mysql Dumping via a php script.
My php file originates from
var/WWW/html/tool/script.php
The folder reads root / WWW-data when i do ls -l
It is doing
exec('mysqldump -user=user --password=pass db > selfDir\dump.sql')
Now what i would expect is the for the script to output to dump.sql inside the scripts Folder.
This is not Happening though.
Only once i did touch dump.sql and chmod 777 dump.sql was the dump actually being written.
I dont understand why. How can i alter my exec() to make sure it can CREATE the dump.file instead of only writing to it once it already exists ?
thanks
If you only want to automatize the database backup, it's better calling the mysqldump with any other system service, like cron.
Rely on php exec can generate too many errors.
Php limit time execution
Execute from request, then the request is killed the exec is killed or need to wait all the mysqldump time
More difficult permissions
Limitation memory in php and the server
In cron you the backup will be (example from The Java Guy):
0 0 * * * mysqldump -u 'username' -p'password' DBNAME > /home/eric/db_backup/liveDB_`date +\%Y\%m\%d_\%H\%M`.sql
Other link for crontab backup with mysql: Answer K1773R mysql-dump-cronjob
Related
I have a PHP script that takes about 15 hours to run and I have to run it once a week. The script downloads and extracts the contents of a massive (70gb) CSV file into our database.
I need the script to start running every Sunday at 6 PM and to keep running until it's finished processing the file.
My plan is to create a bash file cron.sh that has the following:
nohup php /var/www/html/cron.php > /var/www/html/outputs/`date "+%Y-%m-%d %H:%M:%S"`.log
And the cron being:
0 18 * * 0 /var/www/html/cron.sh
How would I be able to improve this?
Remember to redirect stderr as well:
nohup php /var/www/html/cron.php > dir/stdout.log 2> dir/stderr.log
or if you prefer to merge them all in the same file:
nohup php /var/www/html/cron.php > dir/stdout.log 2>&1
The 2>&1 means redirect stderr to the same file stdout is using.
The rest is all fine, putting the command inside a shell script makes it more readable if the line is too long. Also if you later want to make the task do more things, it is easier to update the script file.
I would probably rename the cron.sh file to something more meaningful.
Since your OP asked "How would I be able to improve this" -- You didn't mention if you HAD to use PHP.
If the CSV is that large .. I would create a bash file that does the same thing that your PHP file does (since you're already calling cron.sh). You would have to change your PHP configuration to less than ideal settings to be able to run it efficiently. IMHO straight bash is the way to go. That way nohup can be bypassed entirely.
Meaning I would:
1) Create a login path for MySQL to securely connect to your db from the .sh file. You can acheive this with the following command:
sudo mysql_config_editor set --login-path=client --host=yourHost --user=MySQL_user --password
You will be prompted for the MySQL password.
2) Once that is done, you can access MySQL in your bash file like such AND create your logs all in the same file:
#!/bin/bash
cd /place/where/you/want/the/csv/
wget http://your_file_location.com/csv/location/file.csv
# Or whatever you're doing to "download" it -- Just an example ..
mysql --login-path=local infile=1 3parsfdb -e "LOAD DATA LOCAL INFILE 'file.csv' INTO TABLE your_table FIELDS TERMINATED BY ','"
echo "Whatever you've collected" >> /var/www/html/outputs/`date "+%Y-%m-%d %H:%M:%S"`.log
Of course that's just an example .. But you get the general idea. Then store said bash file in a safe location.
3) Put your newly created bash file into the crontab. You can use your user's crontab, or if the user doesn't have access to the bash file or directory you put it in, you can use root's crontab -- IE:
0 18 * * 0 /path/to/your/bash/file/BackupCSVToDB.sh
The saved overhead of not having to use PHP could save the run-time literally hours.
I am trying to create periodic backups (poor man's cron) of my database using mysqldump with exec() function. I am using XAMPP/PHP7 on macOS.
$command = "$mysqldump_location -u$db_user -h$db_host -p$db_password $db_name > $backup_file_location";
exec($command);
When I run the PHP script, I get no SQL dump in the path mentioned in $backup_file_location but if I execute the same $command string on the terminal directly I get the desired SQL file in the desired location.
I am unable to understand what could be the problem here. Also open to suggestions on better ways to dump the entire DB.
Edit 1:
The value of $mysqldump_location is /Applications/XAMPP/xamppfiles/bin/mysqldump
The value of $backup_file_location is /Applications/XAMPP/xamppfiles/htdocs/app5/data/sqldumps/sql_data.sql
/app5/ is the folder in while I am developing my app.
Edit 2:
Possible duplicate suggestion does not apply since the issue here was not on how to dump SQL backups. The key issue here was that the backup using mysqldump was working through terminal, but not through PHP's exec() function.
The resolution of the issue, from above comments, was that the PHP request executes in XAMPP as a user that has limited privileges, and the mysqldump process inherits those privileges.
Checking the exit status of the process run by exec() confirmed that mysqldump exited with a nonzero exit status, indicating it failed for some reason.
Opening write privileges to 777 on the directory where the mysqldump process tries to write resolved the error.
It should also be adequate to figure out the specific uid & gid of Apache processes (check the User and Group config values in the Apache config file (e.g. xampp-home/apache/conf/httpd.conf) and make the output directory writeable by that uid or gid.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
Hi i have to move mysql database to another server ,
It's nearly 5 gb
i can have root access at both servers?
Usually you run mysqldump to create a database copy and backups as follows:
$ mysqldump -u user -p db-name > db-name.out
Copy db-name.out file using sftp/ssh to remote MySQL server:
$ scp db-name.out user#remote.box.com:/backup
Restore database at remote server (login over ssh):
$ mysql -u user -p db-name < db-name.out
OR
$ mysql -u user -p 'password' db-name < db-name.out
How do I copy a MySQL database from one computer/server to another?
Short answer is you can copy database from one computer/server to another using ssh or mysql client.
You can run all the above 3 commands in one pass using mysqldump and mysql commands (insecure method, use only if you are using VPN or trust your network):
$ mysqldump db-name | mysql -h remote.box.com db-name
Use ssh if you don't have direct access to remote mysql server (secure method):
$ mysqldump db-name | ssh user#remote.box.com mysql db-name
OR
$ mysqldump -u username -p'password' db-name | ssh user#remote.box.com mysql -u username -p'password db-name
You can just copy table called foo to remote database (and remote mysql server remote.box.com) called bar using same syntax:
$ mysqldump db-name foo | ssh user#remote.box.com mysql bar
OR
$ mysqldump -u user -p'password' db-name foo | ssh user#remote.box.com mysql -u user -p'password' db-name foo
Almost all commands can be run using pipes under UNIX/Linux oses.
More from Reference
Regards,
If you have Root, you might find it quicker to avoid mysqldump. You can create the DB on the destination server, and copy the database files directly. Assuming user has access to the destination server's mysql directory:
[root#server-A]# /etc/init.d/mysqld stop
[root#server-A]# cd /var/lib/mysql/[databasename]
[root#server-A]# scp * user#otherhost:/var/lib/mysql/[databasename]
[root#server-A]# /etc/init.d/mysqld start
Important things here are: Stop mysqld on both servers before copying DB files, make sure the file ownership and permissions are correct on the destination before starting mysqld on the destination server.
[root#server-B]# chown mysql:mysql /var/lib/mysql/[databasename]/*
[root#server-B]# chmod 660 /var/lib/mysql/[databasename]/*
[root#server-B]# /etc/init.d/mysqld start
With time being your priority here, the use of compression will depend on whether the time lost waiting for compression/decompression (with something like gzip) will be greater than the time wasted transmitting uncompressed data; that is, the speed of your connection.
For an automated way to backup your MySQL database:
The prerequisites to doing any form of backup is to find the ideal time of day to complete the task without hindering performance of running systems or interfere with users. On that note, the size of the database must be taken into consideration along with the I/O speed of the drives - this becomes exponentially more important as the database grows ( another factor that comes to mind is the number of drives, as having an alternate drive where the database is not stored would increase the speed due to the heads not performing both reads and writes.) For the sake of keeping this readable I'm going to assume the database is of a manageable size ( 100MB ) and the work environment is a 9am-5pm job with no real stress or other running systems on off hours.
The first step would be to log into your local machine with root privileges. Once at the root shell, a MySQL user will need to be created with read only privileges. To do this, enter the MySQL shell using the command:
mysql -uroot -ppassword
Next, a user will need to be created with read only privileges to the database that needs to be backed up. In this case, a specific database does not need to be assigned to a user in case the script or process would be used at a later time. To create a user with full read privileges, enter these commands in the MySQL shell:
grant SELECT on *.* TO backupdbuser#localhost IDENTIFIED BY ' backuppassword';
FLUSH PRIVILEGES;
With the MySQL user created, it's safe to exit the MySQL shell and drop back into the root shell using exit. From here, we'll need to create the script that we want to run our backup commands, this is easily accomplished using BASH. This script can be stored anywhere, as we'll be using a cron job to run the script nightly, for the sake of this example we'll place the script in a new directory we create called "backupscripts." To create this directory, use this command at the root shell:
mkdir /backupscripts
We'll also need to create a directory to store our backups locally. We'll name this directory "backuplogs." Issue this command at the root shell to create the directory:
mkdir /backuplogs
The next step would be to log into your remote machine with root credentials and create a "backup user" with the command:
useradd -c "backup user" -p backuppassword backupuser
Create a directory for your backups:
mkdir /backuplogs/
Before you log out, grab the IP address of the remote host for future reference using the command:
ifconfig -a
eth0 is the standard interface for a wired network connection. Note this IP_ADDR.
Finally, log out of the remote server and return to your original host.
Next we'll create the file that is our script on our host local machine, not the remote. We'll use VIM (don't hold it against me if you're a nano or emacs fan - but I'm not going to list how to use VIM to edit a file in here,) first create the file and using mysqldump, backup your database. We'll also be using scp to After the database has been created, compress your file for storage. Read the file to STDOUT to satisfy the instructions. Finally, check for files older than 7 days old. Remove them. To do this, your script will look like this:
vim /backupscripts/mysqldbbackup.sh
#!/bin/sh
# create a temporary file for the schema to be stored
BACKUPDIR = /backuplogs/
TMPFILE = tmpout.sql
CURRTIME = $(date +%Y%m%d).tgz
#backup your database
mysqldump -ubackupdbuser -pbackuppassword databasename > $BACKUPDIR$TMPFILE
#compress this file and store it locally with the current date
tar -zvcf /backuplogs/backupdb-$CURRTIME $BACKUPDIR$TMPFILE
#per instructions - cat the contents of the SQL file to STDOUT
cat $BACKUPDIR$TMPFILE
#cleanup script
# remove files older than 7 days old
find $BACKUPDIR -atime +7 -name 'backup-db-*.tgz' -exec rm {} \;
#remove the old backupdirectory from the remote server
ssh backupuser#remotehostip find /backuplogs/ -name 'backup-db-*.tgz' -exec rm {} \;
#copy the current backup directory to the remote server using scp
scp -r /backuplogs/ backupuser#remotehostip:/backuplogs/
#################
# End script
#################
With this script in place, we'll need to setup ssh keys, so that we're not prompted for a password every time our script runs. We'll do this with SSH-keygen and the command:
ssh-keygen -t rsa
Enter a password at the prompt - this creates your private key. Do not share this.
The file you need to share is your public key, it is stored in the file current_home/.ssh/id_rsa.pub. The next step is to transfer this public key to your remote host . To get the key use the command:
cat current_home/.ssh/id_rsa.pub
copy the string in the file. Next ssh into your remote server using the command:
ssh backupuser#remotehostip
Enter your password and then edit the file /.ssh/authorized_keys. Paste the string that was obtained from your id_rsa.pub file into the authorized_keys file. Write the changes to the file using your editor and then exit the editor. Log out of your remote server and test the RSA keys have worked by attempting to log into the remote server again by using the previous ssh command. If no password is asked for, it is working properly. Log out of the remote server again.
The final thing we'll need to do is create a cron job to run this every night after users have logged off. Using crontab, we'll edit the current users file (root) as to avoid all permission issues. *Note - this can have serious implications if there are errors in your scripts including deletion of data, security vulnerabilities, etc - double check all of your work and make sure you trust your own system, if this is not possible then an alternate user and permissions should be set up on the current server *. To edit your current crontab we'll issue the command:
crontab -e
While in the crontab editor (it'll open in your default text editor), we're going to set our script to run every night at 12:30am. Enter a new line into the editor:
30 0 * * * bash /backupscripts/mysqldbbackup.sh
Save this file and exit the editor. To get your cronjob to run properly, we'll need to restart the crond service. To do this, issue the command:
/etc/init.d/crond restart
I have a pretty complicated bunch of sql calls that I have stored to a .sql file, that I can't call using sqlsrv_query because there are a bunch of GOs.
From the (windows) command line I can run the sql file using
"path\to\sqlcmd" -U name -P pass -S loc -i "path\to\stuff.sql"
I can also write system( [all that] ) in a php file that I run from the command line using php -f.
However, if I call the same system( [allthat] ) from a php file that I call using ajax on a website, I get no feedback, the sql just doesn't run. I've checked sqlcmd, and it has permissions to allow all users to execute.
Any ideas?
Permissions issue. If you're running it from the command line, it's running under YOUR permissions. When you invoke it via a webserver, it's running with the webserver's permisisons. The webserver will need execute rights (and permission to REACH the sqlcmd) as well as read rights on the .sql file (and permission to reach it).
For that, you may use the shell_exec.
Update: Finally got this thing working but still not sure what the problem was. I am using a wamp server that I access through a networked folder.
The problem that still exists is that to execute the mysqldump I have to access the php file from the actual machine that is being used to host the WAMP server.
End of update
I am running a wamp server and trying to use mysqldump to backup a mysql database I have. The following is the PHP code I am using to run mysqldump.
exec("mysqldump backup -u$user -p$pass > $sql_file");
When I run the script the page just loads inifnately and the backup is not created.
A blank file is being created so I know something is happening.
Extra info:
* exec() is not disabled
* PHP is not running in safe mode
Any ideas??
Win XP, WAMP, MYSQL 5.0.51b
mysqldump is likely to exceed the maximal time php is supposed to run on your system. Try using the command in cmd or increase the max_execution_time in your php.ini .
Are you sure $pass is defined and doesn't have a space character at the start?
If it wasn't, mysqldump would be waiting for command line entry of the password.
I had the same thing happen a while back. A co-worker pointed me to the MySQL GUI tools and I have been making backups with that. The Query Browser that comes with it is nice, too.
MySQL GUI tools
It might help to look at the stderr output from mysqldump:
$cmd = "mysqldump backup -u$user -p$pass 2>&1 > $sql_file";
exec($cmd, $output, $return);
if ($return != 0) { //0 is ok
die('Error: ' . implode("\r\n", $output));
}
Also you should use escapeshellarg() if $user or $pass are user-supplied.
I've also struggled with using the mysqldump utility. I few things to check/try based on my experience:
Is your server set up to allow programs to run programs with an exec command? (My webhost's server won't let me.) Test with a different command.
Is the mysqldump utility installed? Check with whereis mysqldump.
Try adding the optimize argument --opt