I have a large excel file of ship locations for the next 2 years. Currently, I am manually splitting this file into multiple files and then importing into MYSQL. This wont work for long though, as the Excel file gets updated everyday and needs an easier way to be imported.
The data doesnt start till row 10, Column D, and goes till Column I, then another Ship starts on Row 10 Column J to O, ect, for 22 ships. (Row 10 being the header titles)
Is there a way to automate this? I have done some research and found I probably need to convert the XLS to CSV which isnt a problem, but I havnt found a way to state Row 10 Column's D-I into Table1, Column J to O to Table2, Column's P-U to Table3, ect.
Can someone point me in the right direction or provide some assistance. Thanks for all your help!
It's hard to tell not knowing all the details but I would propose the following flow:
Save your file as CVS (can be automated via VBA, PowerShell...)
Load it as it is in a staging table using LOAD DATA INFILE
If necessary validate, clean, normalize the data in the staging table
Use INSERT INTO ... SELECT ... to copy data to factual tables (selecting only necessary columns e.g. from J to O...)
Truncate the staging table
If the rules of data extraction are deterministic steps 2-5 can wrapped in a stored procedure.
Related
I have the following problem:
I have got a dataset inside a text file (not xml or csv encoded or something, just field values separated by \t and \n) which is updated every 2 minutes. I need to put the data from the file into a MariaDB Database, which itself is not very difficult to do.
What I am unsure about however, is how I would go about updating the table if the file's contents change. I thought about truncating the table and then filling it again, but doing that every 2 minutes with about 1000 datasets would mean some nasty problems with the database being incomplete during those updates, which makes it not a usable solution (which it wouldn't have been with fewer datasets either :D)
Another solution I thought about was to append the new data to the existing table, and use a delimter on the unique column (e.g. use cols 1-1000 before update, append data, then use values 1001-2000 after the update and remove 1-1000, after 2 or so updates start at id 1 again).
Updating the changing fields is not an option, because the raw data format would make that really difficult to keep track of the column that has changed (or hasn't)
I am, however unsure about best practices, as I am relatively new to SQL and stuff, and would like to hear your opinion, maybe I am just overlooking something obvious...
Even better...
CREATE TABLE new LIKE real; -- permanent, not TEMPORARY
load `new` from the incoming data
RENAME TABLE real TO old, new TO real;
DROP TABLE old.
Advantages:
The table real is never invisible, nor empty, to the application.
The RENAME is "instantaneous" and "atomic".
As suggested by Alex, I will create a temporary table, insert my data into the temporary table, truncate the production table and then insert from the temporary table. Works like a charm!
I have built a database with 6 tables, roughly 175 fields. About 130 of these fields are to be populated from data on a CSV.
Currently, a handheld device exports this CSV and it is read into a spreadsheet but it's moving to a database. So, on the front end when someone uploads a CSV, it will populate the database.
Question:
I'm trying to figure out the best way to break that CSV up line by line and put certain info into certain tables. Is that possible? If so how?
I was hoping I could query to create a header for each CSV field and map it to database fields (Since the CSV will always be in the same order).
I don't think of it as a RBAR problem. If you load the file as-is into a single staging table, you can then run something like the following for each table:
INSERT INTO destTable (col1, col2)
SELECT col1, col2
FROM StageTable
WHERE col3 = 'criteria'
That way, you keep everything set-based. Of course it depends on the number of records involved, but processing data row by row and TSQL are generally not a good fit. SSIS does a much better job of that than TSQL.
tag it by associative array in columns example
id,name,color
1,jo,red
2,ma,blue
3,j,yellow
get the first line in one array, so just compare by index the value in a loop
I've been having an issue for days now and have hit a brick wall. Firstly, as the title suggests I have been working to import CSV files to a SQL database.
To be more specific, this is done through PHP scripts on the server and through MySQL into the DB.
I currently have around 30 CSV files (this number is projected to increase) which are updated daily, then a cron script is triggered once per day to update the new data. It loads the file through LOAD DATA INFILE. All of this works perfectly.
The problem is:
Each CSV file contains a different column count. The column count ranges between 50-56 columns. The data I am storing in this collective database only requires the first 8 columns. I already know how to skip individual columns using #dummy thanks to the following Q&A: How to skip columns in CSV file when importing into MySQL table using LOAD DATA INFILE?
However, as the dummy count will not always be the same due to the different column counts, I was wondering if there was a way to get the data from columns 1-8 then ignore all after regardless of column count?
A rather rough patch up would be to first read the beginning line in php, to count columns by commas. Then knowing the amount, subtract 8 and generate the sql command now knowing how many columns you need to ignore.
Just include the eight columns to populate and it will us the first eight from the CSV row:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (c1, c2, c3, c4, c5, c6, c7, c8)
I have a database table with 6 columns of 365 rows of data. I need to swap the 3rd column (named 'Date_line') with new data while leaving the other 5 columns in place, without exporting the whole table, but can't get phpMyAdmin to work with me.
Normally I'd just truncate the table and upload a revised CSV file for the whole table, but here's the catch: I have to update 232 data tables with this same exact column of data (the column data is common to all 232 tables). To do all 232 individually would mean exporting each table, opening it in Excel, swapping the old column for the new one, converting to CSV then re-uploading. It would be a lot easier if I could just import a single column CSV to overwrite the old one. But I don't know how.
I'd like to do this using the phpMyAdmin interface... I'm not much experienced in assigning scripts. Is there a way?
Hello this is my first time i post but hopefully i won't mess up to much.
Basically i'm trying to to copy two tables into a new table, the data in table 2 and 3 are temp data that i update with two csv files. It's just basic data that share the same ID so thats the Primary Key and i want these to be combined into a new table. This is supposed to be done just once a day handling about 2000 lines Below follows a better description of what i'm looking for.
3 tables, Core, temp_data1, temp_data2
temp_data1 has id, name, product
temp_data2 has id, description
id is a unique since it's the product_nr of the product
First copy the data from temp_data1 to Core. Insert new line if the product does not exist, if it do exist it should update the row with the information
Next update Core with the description where id=id and do not insert if id do not exist (it should not exist)
I'm looking for something that can be done in one push of a button, first i upload the csv file into the two different databases (two different files) next i push a button to merge the two tables to the Core one. I know you can do this right away with the two csv files and skip the two tables but i feel like that is so over my head it's not even funny.
I can handle programming php it's all the mysql stuff that's messing with my head.
Hopefully you guys can help me and in return i will help out any other place i can.
Thanks in advance.
I'm not sure I understand it correctly, but this can be done using only sql script, using INSERT INTO...SELECT...ON DUPLICATE KEY UPDATE... - see http://dev.mysql.com/doc/refman/5.6/en/insert-select.html