I want to do an export of a table and we do not have mysqldump installed.
I thought I can do this:
root:~> mysql news media > news.media.7.26.2016.sql
where news is the database name and media is the table name
it doesn't seem to work correctly.
Your command tries to mimic mysqldump but mysql does not have a table parameter. You could run it like this:
mysql -D news -e "SELECT * FROM media" > news.media.7.26.2016.txt
That will work but you won't get nice SQL statements in the output, just tabular data export.
I mean that you may (or may not) run into problems when importing the data back. There's a chance to use
mysql -D news -e "LOAD DATA INFILE 'news.media.7.26.2016.txt' INTO TABLE media"
but I do not have much experience with that. First of your concerns is secure-file-priv setting that has been made strict starting in MySQL 5.7.6. Second, I would be a tad bit nervous about preserving data types.
Related
I want to export MySQL database through command line with following two conditions as below,
Export database tables structure only.
But at the time of Exporting data, I want to export specified table data and structure both that are mention in command.
I have a database which contains 60 tables, and from them, one table named Country contains the static values. So I want to export its data too in the exported file, and rest of the tables contain only structure in exported file.
Can any one suggest me MySQL command to dump the database based on above conditions?
First method:
mysqldump --no-data -h <host> -u <username> -p<password> <database> > tables.sql
This will dump your database table structures into the file tables.sql.
The --no-data option specifies that table content is not dumped.
Second method:
(This can be useful if you want to more with the generated .sql file in batch processing.)
Use the show create table command to see the actual command to create a table.
Create a Bash script that executes that MySQL command for every table that you want to export. Pipe the output from show create table <table_name> to a text file.
I think you'll need to fire 2 separate commands for that.
mysqldump -u root -p password --no-data --ignore-table=db_name.tbl_name db_name > db_name.sql
Above query will dump all the structure of database without your country table's structure and data.
mysqldump -u root -p password db_name tbl_name >> db_name.sql
And this one will dump Country table's structure and data into the same file.
I copied the contents of a large data table from one table to another with 2 additional columns,
the table1(original data) is getting queried by
select * from cc2;
But the same data with 2 more additional columns having NULL values throughout are not getting executed normally. i have to put limit clause to make it execute. like
select * from cc *limit 0,68000*;
the database is same, table and content are same. the question is WHY this weird behavior. and to parse my this data to foreach() loop, i am having to run for() loop and it is affecting the performance.
Any suggestions would be tried and tested asap.
Thanks in advance geniuses
Instead of using php for importing a lot of data, just try to execute directly from the commandline.
first dump your table:
mysqldump -u yourusername -p yourpassword yourdatabase tableName > text_file.sql
then change the tablename in the top of that file (and make sure the extra columns have default NULL ). And import with
mysql -u yourusername -p yourpassword yourdatabase < text_file.sql
Using a textfile containing the query is always preferred for large data sets, so you don't run into problems with PHP or the webserver.
I am using mysqldump to create DB dumps of the live application to be used by developers.
This data contains customer data. I want to anonymize this data, i.e. remove customer names / credit card data.
An option would be:
create copy of database (create dump and import dump)
fire SQL queries that anonymize the data
dump the new database
But this has to much overhead.
A better solution would be, to do the anonymization during dump creation.
I guess I would end up parsing all the mysqlsqldump output? Are there any smarter solutions?
You can try Myanon: https://myanon.io
Anonymization is done on the fly during dump:
mysqldump | myanon -f db.conf | gzip > anon.sql.gz
Why are you selecting from your tables if you want to randomize the data?
Do a mysqldump of the tables that are safe to dump (configuration tables, etc) with data, and a mysqldump of your sensitive tables with structure only.
Then, in your application, you can construct the INSERT statements for the sensitive tables based on your randomly created data.
I had to develop something similar few days ago. I couldn't do INTO OUTFILE because the db is AWS RDS. I end up with that approach:
Dump data in tabular text form from some table:
mysql -B -e 'SELECT `address`.`id`, "address1" , "address2", "address3", "town", "00000000000" as `contact_number`, "example#example.com" as `email` FROM `address`' some_db > addresses.txt
And then to import it:
mysql --local-infile=1 -e "LOAD DATA LOCAL INFILE 'addresses.txt' INTO TABLE \`address\` FIELDS TERMINATED BY '\t' ENCLOSED BY '\"' IGNORE 1 LINES" some_db
only mysql command is required to do this.
As the export is pretty quick (couple of seconds for ~30.000 rows), the import process is a bit slower, but still fine. I had to join few tables on the way and there was some foreign keys so it will surely be faster if you don't need that. Also if you disable foreign key checks while importing it will also speed up things.
You could do a select of each table (and not a select *) and specify the columns you want to have and omit or blank those you don't want to have, and then use the export option of phpmyadmin for each query.
You can also use the SELECT ... INTO OUTFILE syntax from a SELECT query to make a dump with a column filter.
I found to similar questions but it looks like there is no easy solution for what you want. You will have to write a custom export yourself.
MySQL dump by query
MySQL: Dump a database from a SQL query
phpMyAdmin provides an export option to the SQL format based on SQL queries. It might be an option to extract this code from PHPmyadmin (which is probably well tested) and use it in this application.
Refer to the phpMyAdmin export plugin - exportData method for the code.
I have an application at Location A (LA-MySQL) that uses a MySQL database; And another application at Location B (LB-PSQL) that uses a PostgreSQL database. (by location I mean physically distant places and different networks if it matters)
I need to update one table at LB-PSQL to be synchronized with LA-MySQL but I don't know exactly which are the best practices in this area.
Also, the table I need to update at LB-PSQL does not necessarily have the same structure of LA-MySQL. (but I think that isn't a problem since the fields I need to update on LB-PSQL are able to accommodate the data from LA-MySQL fields)
Given this data, which are the best practices, usual methods or references to do this kind of thing?
Thanks in advance for any feedback!
If both servers are in different networks, the only chance I see is to export the data into a flat file from MySQL.
Then transfer the file (e.g. FTP or something similar) to the PostgreSQL server and import it there using COPY
I would recommend to import the flat file into a staging table. From there you can use SQL to move the data to the approriate target table. That will give you the chance to do data conversion or do updates on existing rows.
If that transformation is more complicated you might want to think about using an ETL tool (e.g. Kettle) to do the migration on the target server .
Just create a script on LA that will do something like this (bash sample):
TMPFILE=`mktemp` || (echo "mktemp failed" 1>&2; exit 1)
pg_dump --column-inserts --data-only --no-password \
--host="LB_hostname" --username="username" \
--table="tablename" "databasename" \
awk '/^INSERT/ {i=1} {if(i) print} # ignore everything to first INSERT' \
> "$TMPFILE" \
|| (echo "pg_dump failed" 1>&2; exit 1)
(echo "begin; truncate tablename;"; cat "$TMPFILE"; echo 'commit;' ) \
| mysql "databasename" < "$TMPFILE" \
|| (echo "mysql failed" 1>&2; exit 1) \
rm "$TMPFILE"
And set it to run for example once a day in cron. You'd need a '.pgpass' for postgresql password and mysql option file for mysql password.
This should be fast enough for a less than a million of rows.
Not a turnkey solution, but this is some code to help with this task using triggers. The following assumes no deletes or updates for brevity. Needs PG>=9.1
1) Prepare 2 new tables. mytable_a, and mytable_b. with the same columns as the source table to be replicated:
CREATE TABLE mytable_a AS TABLE mytable WITH NO DATA;
CREATE TABLE mytable_b AS TABLE mytable WITH NO DATA;
-- trigger function which copies data from mytable to mytable_a on each insert
CREATE OR REPLACE FUNCTION data_copy_a() RETURNS trigger AS $data_copy_a$
BEGIN
INSERT INTO mytable_a SELECT NEW.*;
RETURN NEW;
END;
$data_copy_a$ LANGUAGE plpgsql;
-- start trigger
CREATE TRIGGER data_copy_a AFTER INSERT ON mytable FOR EACH ROW EXECUTE PROCEDURE data_copy_a();
Then when you need to export:
-- move data from mytable_a -> mytable_b without stopping trigger
WITH d_rows AS (DELETE FROM mytable_a RETURNING * ) INSERT INTO mytable_b SELECT * FROM d_rows;
-- export data from mytable_b -> file
\copy mytable_b to '/tmp/data.csv' WITH DELIMITER ',' csv;
-- empty table
TRUNCATE mytable_b;
Then you may import the data.csv to mysql.
I'm writing a PHP script to generate SQL dumps from my database for version control purposes. It already dumps the data structure by means of running the appropriate SHOW CREATE .... query. Now I want to dump data itself but I'm unsure about the best method. My requirements are:
I need a record per row
Rows must be sorted by primary key
SQL must be valid and exact no matter the data type (integers, strings, binary data...)
Dumps should be identical when data has not changed
I can detect and run mysqldump as external command but that adds an extra system requirement and I need to parse the output in order to remove headers and footers with dump information I don't need (such as server version or dump date). I'd love to keep my script as simple as I can so it can be hold in an standalone file.
What are my alternatives?
I think the mysqldump output parsing is still the easiest. I think there are really few meta-data you have to exclude, and thus dropping those should be only a few lines of code.
You could also look at the following mysqdump options: --tab, --compact
Could always try the system() function. Hope you are in linux :)
"Select * from tablename order by primary key"
you can get the table names from the information_schema database or the "show tables" command. Also can get the primary key from "show indexes from tablename"
and then loop and create insert statements strings