I am using PHPSpreadsheet to take some spreadsheets a user can upload, add a column with certain values, save the file as CSV, and use the following query to import the csv file:
LOAD DATA LOCAL INFILE '{$file}'
INTO TABLE {$table}
FIELDS TERMINATED by ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
Alternatively I can do something like:
foreach($rows as $row){
// INSERT $row INTO table
}
The spreadsheets will all have the same columns/data-types.
What would be the most efficient way to do this? Going from Xlsx -> CSV -> MySQL Import seems like I am adding extra steps.
MySQL's direct CSV import is usually the fastest option, however it is not without limitations. One is that you need to import all or nothing in the file and you won't know how far along it is until it's done. As some imports can take hours, even days, you may not know where it's at. The entire insert operation on an InnoDB table takes place atomically for performance reasons but that means it's not visible until fully committed.
Another is the file must be present on the server. The LOCAL option is a quirky feature of the mysql command-line tool and probably doesn't work in your database driver unless emulated.
Inserting row-by-row with a CSV parser is almost always slower. If you must do a thing, be sure to prepare an INSERT statement once and re-use it in the loop, or do a "multi-INSERT" with as many rows as you can fit in your max statement size buffer.
Related
I have excel data more than 5k rows and 17 columns, I use the nested loop technique in php, but this takes a long time, to process the data using the xls file format takes 45 minutes, while using the csv file format takes 30 minutes , is there a technique to speed up uploading files from excel to the database (I use Postgresql).
I use a nested loop because how many columns depend on the parameters, and for the INSERT or UPDATE process to the database also depends on the parameters.
Here is my code for the import process
<?php
$row = 5000; // estimated row
$col = 17; // estimated col
for($i=1; $i<=$row; $i+=1){
for($j=1; $j<=$col; $j+=1){
$custno = $custno = $sheetData[$i][0];
$getId = "SELECT id from data WHERE 'custno' = $custno";
if($getId){
$update = "UPDATE data SET address = 'address 1' WHERE custno = $custno";
}else{
$insert = "INSERT INTO data (address) VALUES (address jon);
}
}
}
I use the PhpSpreadsheet library
First, try to find out what is the root of the issue, is it because operating over the file is slow or there are too many SQL queries being executed in the meantime?
Bear in mind that running queries in the loop is always asking for performance trouble. Maybe you can avoid that by asking for needed data before processing the file? You may not be able to define which data are needed on that step but fetching more than you need could be still faster than making separate queries one by one. Also, I would like to encourage you to limit INSERT or UPDATE queries. They are usually slower than the SELECT one. Try to collect data for database write operations and run it once after the loop.
For CSV operations I would prefer basic php methods like fgetcsv() and str_getcsv() than the separate library as long as the file is not overcomplicated. If you are keen to check some alternatives for PhpSpreadsheet take a look at Spout by box.com, it looks promising but I have never used that.
I'm sure that you can improve your performance by using PHP Genrators, they are perfect everytime you have to read a file content. Here you have some more links:
https://www.sitepoint.com/memory-performance-boosts-with-generators-and-nikiciter/
https://www.sitepoint.com/generators-in-php/
https://riptutorial.com/php/example/5441/reading-a-large-file-with-a-generator/
If not using php for this operation is an option for your, try exporting this spreadsheet as CSV and importing the file using COPY. It won't take more than a few seconds.
If your database is installed locally you just need to execute COPY in a client of your choice, e.g. pgAdmin. Check this answer for more information.
COPY your_table FROM '/home/user/file.csv' DELIMITER ',' CSV HEADER;
Keep in mind that the user postgres in your system must have the necessary permissions to access the CSV file. Check how to do that in your operating system, e.g. chown in Linux.
In case your database is installed in a remote server, you have to use the STDIN facility of COPY via psql
$ cat file.csv | psql your_db -c "COPY your_table FROM STDIN;"
I Have created a table on my db, and filled all the records, using CSV file.
I need to do this weekly to keep the table updated.
I want to upload the new records without disturbing the old one onto the same table using csv.
[I have to pick the data from remote host and upload it locally on my server, i dont have access to the remote db]
Kindly guide me.
You can upload records from a CSV into a table VERY quickly using the load data infile syntax (http://dev.mysql.com/doc/refman/5.1/en/load-data.html)
The syntax is pretty simple, but also flexible. This is an example:
LOAD DATA INFILE 'data.txt' INTO TABLE table2
FIELDS TERMINATED BY '\t';
You can kick these off from a console or via code.
This will append to the table, not replace it, so if you don't truncate it first, it should work a charm.
You can of course also load the data manually by parsing the CSV file in your code and manually creating an insert statement for each line of code, but if the format is fixed already, this will be quicker and more efficient.
Edit: It appends the data. By default, no database will delete data from a table unless you specifically tell it to. Any insert statement is what you consider an append statement.
I am trying to parse a large sql file to a csv file. I have considered using fread in php but cant figure out if sql is separated by lines...bc I am assuming that fread is loading the data into RAM and that would not work.
Any ideas on how quickly convert sql to csv? (also I am running on a different machine than my db is on...so I cant export as csv unfortunately).
"Large" - what does it mean to you.
You can save to a file on server (the machine DB is running) and compress/download.
Exaple:
SELECT name,lastname,age FROM profile
The query returns three columns of the mysql table. Now for redirecting/print into a file:
SELECT name,lastname,age FROM profile INTO OUTFILE '/tmp/userdata.txt'
This will output data into the passed file in the above statement.
To output data in terms of CSV format add more options to the query as following:
SELECT name,lastname,age FROM profile INTO OUTFILE '/tmp/userdata.txt'
FIELDS enclosed by '"' separated by "," LINES TERMINATED BY '\n'
original post
install mysql on your local machine. now you can import the sql file, then freely export as csv or whatever you want.
I am trying to import write a script that imports a csv file and parses it into mysql, then imports it in the db, I came across this article,but it seems to be written for ms sql, does anyone know of a tutorial for doing this in mysql, or better still a libary or script that can do it ?.
Thanks :-)
http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/
Using the LOAD DATA INFILE SQL statement
Example :
LOAD DATA LOCAL INFILE '/importfile.csv'
INTO TABLE test_table
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(field1, filed2, field3);
If you are looking for script / library. I want to refer below link. here you can find :
PHP script to import csv data into mysql
This is a simple script that will allow you to import csv data into your database. This comes handy because you can simply edit the appropriate fields, upload it along with the csv file and call it from the web and it will do the rest.
It allows you to specify the delimiter in this csv file, whether it is a coma, a tab etc. It also allows you to chose the line separator, allows you to save the output to a file (known as a data sql dump).
It also permits you to include an empty field at the beginning of each row, which is usually an auto increment integer primary key.
This script is useful mainly if you don’t have phpmyadmin, or you don’t want the hassle of logging in and prefer a few clicks solution, or you simply are a command prompt guy.
Just make sure the table is already created before trying to dump the data.
Kindly post your comments if you got any bug report.
I just downloaded this csv of infoboxes of wikipedia from dbpedia. However I have no idea how to use it :-S I want to import all this data into a database but am not so sure how to take it from here. I downloaded it from http://wiki.dbpedia.org/Downloads32#infoboxes
I'm working in Php
Just for the record - this csv file is around 1.8 GB. I'm actually going through all this trouble for well just to get a select set of infoboxes from a select set of articles form wikipedia. I would do it manually except I need the infoboxes for over 10 000 entries which includes countries and cities. I'm just looking for an easy way to do this and frankly have been using all my options :(
To import CSV data into MySQL you can use a LOAD DATA INFILE statement, e.g.
LOAD DATA LOCAL INFILE '/importfile.csv'
INTO TABLE test_table
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(field1, filed2, field3);
Sometimes such data might need a little massaging, it's not tricky to write a script in Perl or similar to parse a file line by line and spit out SQL statements.
If you want to massage the data before importing it, you could take a look at my CSV stream editor, CSVfix - it's FOSS. It can also generate SQL INSERT statements for your database if for some reason your database's bulk loading of CSV data doesn't suit you.