If I have exported a .sql file with my database in it, can I then only import "parts" of that database instead of the entire database to MySql?
The question appeared when I was trying it out on a test database.
I exported the testdatabase.
Then emptied some of the tables in the database.
Then I planned on importing from the .sql file and hope the emptied tables would be refilled with whatever they where populated with.
But I get an error:
#1007 Can't create database 'database_name' - database exists
Offcourse it exists, but is it possible to only import values of the already existing tables from the .sql backup?
Or must I remove the entire database and then import the database?
FYI I am using PhpMyAdmin for this currently.
It's straightforward to edit the file and remove the parts you're not interested in having restored, Camran.
Alternatively - import the entire file into a separate database (change the database name # the top of the file) and then use INSERT statements to copy the data from the tables in this new database to the other.
I solved this problem by writing a script to dump each table into each individual file and all the CREATE TABLE statements off in their own file. It's gotten a bit long and fancy over the years, so I can't really post it here.
The other approach is to tell MySQL to ignore errors. With the CLI, you provide the -f switch, but I'm not familiar enough with PhpMyAdmin to know how to do that.
Related
I have a MySQL database on which i want to import data from multiple csv files. For the data I provided a table on which I want to merge the several files into one (join). Unfortunately I have the problem that my data is too big and therefore it is quite time-consuming until I get everything stored in the table. Therefore the question: What is the best way to deal with a huge amount of data?
I took the liberty to create a temporary table for each csv file and load the data into it. Then I joined all tables and wanted to insert the result of my query into the big table and there I already had the problem with the long waiting time. I would like to limit the solutions to the following languages: MySQL, PHP. So far I used the GUI of datagrip and the sql-console for importing these files.
Use any data integration tool like Pentaho, then follow the below steps:
Pentaho has CSV import object
You could join multiple CSV file using join object
Select all the columns from merging output
Then push it to MySQL using DB connector output object
There is a pretty neat library that does exactly this. Helps you to migrate data from one source to another. And it does pretty quickly.
https://github.com/DivineOmega/uxdm
You could use a shell script to loop through the files (this one assumes they're in the current directory)
#!/bin/bash
for f in *.csv
do
mysql -e "load data infile '"$f"' into table my_table" -u username --password=your_password my_database
done
You can achieve this easily with the use of pentaho data integration (ETL tool).
It provided us csv data input in which you can mention your csv file. then link to table output step in which you can use jdbc or jndi connection of your mysql database.
I exported sql file with tables schema without any data inside. Then I wanted to import it to another, empty database, so that it will create empty tables there.
The problem is it imported the file in the first database, non-empty, and thus, it restored the tables to the empty state.
So now all of my tables are empty.
So again, I exported empty tables sql file from staging database, and imported them to fresh, empty database. But instead the file was imported, accidentaly, to the staging database, and all data was removed.
I checked "default target schema" in import options to the right database, so I don't know what went wrong here.
My question is - is there a way to resotre the removed data?
This is very bad news for your database. Your import file dropped your tables with lots of data and recreated them containing no rows. Ouch.
If you have a backup, restore it.
If you don't have a backup, you may (or may not) be able to restore some of your data by recovering deleted files. Your luck will be better if you used separate files for your InnoDB tables, or if you used MyISAM. At any rate, shut down your MySQL server instance and make the server on which it runs as quiet as possible, to reduce the probability that the OS will reclaim the deleted file space.
You'll have to ask on https://serverfault.com/ for further advice.
There is no escaping that this event will be difficult to recover from.
I am trying to import some data in one table from a database to another database.
I cannot just copy them, because format of both tables of the two databases are different.
With the fetched data from one database, I am able to create insert queries.
I want to know which is better:
Execute those queries in PHP itself by creating a new connection to second database.
Write all queries to .sql file and then import it directly in second database.
I am looking at the aspects of performance and ease of implementation.
Note: I am expecting the data in the table to be more than ten thousand rows
If you go with the first Option, There are chances that you could make some mistakes.
I prefer you to go with the Second option to Write all queries to .sql file and then import it directly in second database. Thanks
vJ
I would certainly go for the second option. Why use php for a one time action.
You can just solve this in the database with SQL only
I would go for the second option.
Then I would:
get an overview over both table structures
Export the data from the first table in a flat file format like CSV.
If necessary, transform the data from the first table to the second using a script or a tool.
Import the modified data into the second table.
The database vendors have good tools for exporting, manipulating and importing data.
If only the name of the tables are different, vendor tools importing feature often have good functionality for mapping data from one table to another. In my own case, I've used Oracle SQL developer, but please let me know your vendor and I can give you a pointer in the right direction.
We created a import script which imports about 120GB of data into a MySQL database. The data is saved in a few hunderd directories (all are seperated databases). Each directory contains files with table structures and table data.
The issue being; it works on my local machine with a subset of the actual data, but when the import is ran on the server (Which takes a few days). Not all the tables are created (even tables that are tested locally). The odd thing is that the script, when ran on the server does not show any errors on the creation of the tables.
Here is on a high level how the script works:
Find all directories that represent a database
Create all databases
Per database loop through the tables: create table, fill table
Added the code on gist: https://gist.github.com/3349872
Add more logging to see steps that succeded since you might be having problems with memory usage or execution times.
Why dont you create sql files from given CVS files and then just do normal importing in bash?
mysql -u root -ppassword db_something< db_user.sql
Auch, the problem was in the code. Amazingly stupid mistake.
When testing the code on a subset of all the files all table information and table content where available. When a table count not be created the function enters a logging statement and than returns. On the real data this was a the mistake because there are files with no data and no structure so after creating a few tables this creation of the tables of a certain database went wrong and did a return and so didn't create the other tables.
I'm creating locally a big database using MySQL and PHPmyAdmin. I'm constantly adding a lot of info to the database. I have right now more than 10MB of data and I want to export the database to the server but I have a 10MB file size limit in the Import section of PHPmyAdmin of my web host.
So, first question is how I can split the data or something like that to be able to import?
BUT, because I'm constantly adding new data locally, I also need to export the new data to the web host database.
So second question is: How to update the database if the new data added is in between all the 'old/already uploaded' data?
Don't use phpMyAdmin to import large files. You'll be way better off using the mysql CLI to import a dump of your DB. Importing is very easy, transfer the SQL file to the server and afterwards execute the following on the server (you can launch this command from a PHP script using shell_exec or system if needed) mysql --user=user --password=password database < database_dump.sql. Of course the database has to exist, and the user you provide should have the necessary privilege(s) to update the database.
As for syncing changes : that can be very difficult, and depends on a lot of factors. Are you the only party providing new information or are others adding new records as well? Are you going modify the table structure over time as well?
If you're the only one adding data, and the table structure doesn't vary then you could use a boolean flag or a timestamp to determine the records that need to be transferred. Based on that field you could create partial dumps with phpMyAdmin (by writing a SQL command and clicking Export at the bottom, making sure you only export the data) and import these as described above.
BTW You could also look into setting up a master-slave scenario with MySQL, where your data is transferred automatically to the other server (just another option, which might be better depending on your specific needs). For more information, refer to the Replication chapter in the MySQL manual.
What I would do, in 3 steps:
Step 1:
Export your db structure, without content. This is easy to manage on the exporting page of phpmyadmin. After that, I'd instert that into the new db.
Step 2:
Add a new BOOL column in your local db in every table. The function of this is, to store if a data is new, or even not. Because of this set the default to true
Step 3:
Create a php script witch connects to both databases. The script needs to get the data from your local database, and put it into the new one.
I would do this with following mysql methods http://dev.mysql.com/doc/refman/5.0/en/show-tables.html, http://dev.mysql.com/doc/refman/5.0/en/describe.html, select, update and insert
then you have to run your script everytime you want to sync your local pc with the server.