I want to upload a csv to a database with php, but before i do that i want to modify some of the content.
The database table the csv will come from has 3 columns: id, postcode, debt_amount
The database table the csv will go to has 4 columns: id, postcode, debt_amount, count
What i want to do first is modify the full postcode to just show the first part before the 'space'.
Then i want to consolidate all the rows that have the same modified postcode, this will do two things:
Count the number of rows with the same modified postcode and place the total number into the consolidated row in the column 'count'.
Add up the 'debt_amount' column for the same modified postcode and put the total amount into the consolidated row under the 'debt_amount' column.
These processes would need to run together.
After that is done i want to upload it to the database.
I don't know if this is the best way of doing it, or if i should process the data from the first database first and export it into a CSV, to just allow me to upload the CSV on the other database.
Any help on either process would be good.
Thanks
I think it is best to process this data in MySQL itself. You may decide if you would like to process it in the source database or the target database.
So, if the processing involves:
modify the postcode
count #rows with same modified-postcode
sum debt_amount for same modified-postcode
Either do the processing in the source database, store the results in a temporary table, generate the CSV and then import to the target database. Or generate the CSV from the source DB, import to the target database in a temporary table, do the processing and store the results to the final table.
Do the standard file upload.
Read the CSV content from the temporary upload file.
Process the CSV Data (e.g. with SplFileObjectDocs.
Insert the processed data into your database.
Related
I've been having an issue for days now and have hit a brick wall. Firstly, as the title suggests I have been working to import CSV files to a SQL database.
To be more specific, this is done through PHP scripts on the server and through MySQL into the DB.
I currently have around 30 CSV files (this number is projected to increase) which are updated daily, then a cron script is triggered once per day to update the new data. It loads the file through LOAD DATA INFILE. All of this works perfectly.
The problem is:
Each CSV file contains a different column count. The column count ranges between 50-56 columns. The data I am storing in this collective database only requires the first 8 columns. I already know how to skip individual columns using #dummy thanks to the following Q&A: How to skip columns in CSV file when importing into MySQL table using LOAD DATA INFILE?
However, as the dummy count will not always be the same due to the different column counts, I was wondering if there was a way to get the data from columns 1-8 then ignore all after regardless of column count?
A rather rough patch up would be to first read the beginning line in php, to count columns by commas. Then knowing the amount, subtract 8 and generate the sql command now knowing how many columns you need to ignore.
Just include the eight columns to populate and it will us the first eight from the CSV row:
LOAD DATA INFILE 'file.txt' INTO TABLE t1 (c1, c2, c3, c4, c5, c6, c7, c8)
I have a huge CSV File with 20000+ User Entries+User Fields, that i have to compare with our Users in our Database.
The aim is to archive Every User in our database that is not in the CSV File.
My solution would be:
Get Multidimensional Array out of the CSV File
Get every User of the Database
While fetching the User, iterate through CSV array and look if User is in CSV
It is a solution that works but it draws way too much performance.
20,000~ User in CSV * 20,000~ User in Database.
=>400,000,000 Iterations (If no User is found of course...)
Is there a way to reduce the iterations to 20000~?
Yes, you can import csv data into another table and use SQL join to fetch the desired result. That way your data will be fetch much faster than before. Use temp table to import csv file.
my Current MySQL table employee_data has 13k rows with 17 columns. The data in the table came from a CSV file Employees.csv. after importing my csv data I added a new column 'password' (so its not in the csv file) Password is edited and accessed via a web portal. I now have an updated csv file and I want to update my main table with that data but I don't want to lose my password info.
Should I import my new CSV file into a temp table in my database and some how compare them? I am not sure where to start and I am open to recommendations.
I am now realizing I should have kept my password info in a separate table. Doh!
I guess I could created a php file that compares each row based on the employee_id field but with 13k rows I am afraid it would time out possibly.
I would do it like this :
Create a temp table using CREATE TABLE new_tbl LIKE orig_tbl; syntax
use LOAD DATA INFILE to import the data from the CSV into the table
Use UPDATE to update the primary table using a primary key / unique column (perhaps employee_id)
I have worked with tables containing 120 million lines and imported CSV files containing 30 million lines into it - this is the method I use all of the time - much more efficient than anything in PHP (and thats my server side language of choice)
Try other tools other than php based ones phpMyAdmin
MySQL workbench is a great tool,
based on you connection it will take a while to query the database with your data.
There are workarounds with php timeout limit,
set_time_limit();
I need to upload csv into mysql. Regularly.
So I am planing to upload csv to a temporary table. Than move data from temporary_table to main_table.
Now, I need help:
How can I move data. And (a) skip duplicate, or (b) overwrite duplicate
The csv currently contains 55566 rows, and will increasing day by day. So, how to handle execute time.
Best practice to import csv to mysql.
How can I move data.
Use INSERT ... SELECT.
And (a) skip duplicate, or (b) overwrite duplicate
Define a UNIQUE key constraint on the columns that determine whether records are duplicate or not; then:
(a) use INSERT IGNORE; or
(b) use either INSERT ... ON DUPLICATE KEY UPDATE or REPLACE.
The csv currently contains 55566 rows, and will increasing day by day. So, how to handle execute time.
Rotate your CSV file after each upload so that past records are not repeatedly uploaded.
Best practice to import csv to mysql.
Use either mysqlimport or LOAD DATA INFILE.
I have been given an excel document which contains information about different embassies, their contact numbers etc. Most of this information is in the database already, but I need to extract one column from this file (emails) and insert it into a specific column (emails) in the database.
Is this possible, or can you only import an exact copy of what you want in the database?
Thanks for any help
Export the table from phpmyadmin as CSV for excel -
drag it into excel -
add the desired column -
Save -
export the table again from phpmyadmin as .sql -
copy the table creation(Not the data) -
delete the table from phpmyadmin-
Run the table creation query so you now have a blank table -
Import the excel csv.
IMPORTANT: backup your database
You may want to make a script with php-excel-reader which goes throught every row in the file, and updates the corresponding database row. I don't think that you can import a file "selectively" in phpMyAdmin.