I have excel data more than 5k rows and 17 columns, I use the nested loop technique in php, but this takes a long time, to process the data using the xls file format takes 45 minutes, while using the csv file format takes 30 minutes , is there a technique to speed up uploading files from excel to the database (I use Postgresql).
I use a nested loop because how many columns depend on the parameters, and for the INSERT or UPDATE process to the database also depends on the parameters.
Here is my code for the import process
<?php
$row = 5000; // estimated row
$col = 17; // estimated col
for($i=1; $i<=$row; $i+=1){
for($j=1; $j<=$col; $j+=1){
$custno = $custno = $sheetData[$i][0];
$getId = "SELECT id from data WHERE 'custno' = $custno";
if($getId){
$update = "UPDATE data SET address = 'address 1' WHERE custno = $custno";
}else{
$insert = "INSERT INTO data (address) VALUES (address jon);
}
}
}
I use the PhpSpreadsheet library
First, try to find out what is the root of the issue, is it because operating over the file is slow or there are too many SQL queries being executed in the meantime?
Bear in mind that running queries in the loop is always asking for performance trouble. Maybe you can avoid that by asking for needed data before processing the file? You may not be able to define which data are needed on that step but fetching more than you need could be still faster than making separate queries one by one. Also, I would like to encourage you to limit INSERT or UPDATE queries. They are usually slower than the SELECT one. Try to collect data for database write operations and run it once after the loop.
For CSV operations I would prefer basic php methods like fgetcsv() and str_getcsv() than the separate library as long as the file is not overcomplicated. If you are keen to check some alternatives for PhpSpreadsheet take a look at Spout by box.com, it looks promising but I have never used that.
I'm sure that you can improve your performance by using PHP Genrators, they are perfect everytime you have to read a file content. Here you have some more links:
https://www.sitepoint.com/memory-performance-boosts-with-generators-and-nikiciter/
https://www.sitepoint.com/generators-in-php/
https://riptutorial.com/php/example/5441/reading-a-large-file-with-a-generator/
If not using php for this operation is an option for your, try exporting this spreadsheet as CSV and importing the file using COPY. It won't take more than a few seconds.
If your database is installed locally you just need to execute COPY in a client of your choice, e.g. pgAdmin. Check this answer for more information.
COPY your_table FROM '/home/user/file.csv' DELIMITER ',' CSV HEADER;
Keep in mind that the user postgres in your system must have the necessary permissions to access the CSV file. Check how to do that in your operating system, e.g. chown in Linux.
In case your database is installed in a remote server, you have to use the STDIN facility of COPY via psql
$ cat file.csv | psql your_db -c "COPY your_table FROM STDIN;"
I have a script file (.sql) which has almost 20k insert statements in it.
The sql file converted from a csv file. There's an option "load data infile" but it's disabled by the hosting company so i choose to upload a .sql file. As far as i learned, load data infile is not working with sql files. so i need to work like that command below on wordpress.
mysql < yourfile.sql
but there's another problem. The customer is not sharing the database information with me so i only have an editor page of wordpress and $wpdb object with me. My main objective is update a table with given csv file.
Database is a some version of MariaDB.
Use PHP constants defined in wp-config.php file to get access to the database. To import the table you can use exec command from PHP file:
exec('mysql -u '.DB_USER.' -p'.DB_PASSWORD.' '.DB_NAME.' table_name < path/to/table_dump.sql;');
Change table_name to your table name. Without table_name parameter data will be imported for all tables in DB_NAME (if table_dump.sql contains commands for another tables).
If the table is not empty, then you must have a backup of the table or database in case you need to restore the table after import.
Can any one explain how to import a Microsoft Excel file in to a MySQL database?
For example, my Excel table looks like this:
Country | Amount | Qty
----------------------------------
America | 93 | 0.60
Greece | 9377 | 0.80
Australia | 9375 | 0.80
There's a simple online tool that can do this called sqlizer.io.
You upload an XLSX file to it, enter a sheet name and cell range, and it will generate a CREATE TABLE statement and a bunch of INSERT statements to import all your data into a MySQL database.
(Disclaimer: I help run SQLizer)
Below is another method to import spreadsheet data into a MySQL database that doesn't rely on any extra software. Let's assume you want to import your Excel table into the sales table of a MySQL database named mydatabase.
Select the relevant cells:
Paste into Mr. Data Converter and select the output as MySQL:
Change the table name and column definitions to fit your requirements in the generated output:
CREATE TABLE sales (
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
Country VARCHAR(255),
Amount INT,
Qty FLOAT
);
INSERT INTO sales
(Country,Amount,Qty)
VALUES
('America',93,0.60),
('Greece',9377,0.80),
('Australia',9375,0.80);
If you're using MySQL Workbench or already logged into mysql from the command line, then you can execute the generated SQL statements from step 3 directly. Otherwise, paste the code into a text file (e.g., import.sql) and execute this command from a Unix shell:
mysql mydatabase < import.sql
Other ways to import from a SQL file can be found in this Stack Overflow answer.
Export it into some text format. The easiest will probably be a tab-delimited version, but CSV can work as well.
Use the load data capability. See http://dev.mysql.com/doc/refman/5.1/en/load-data.html
Look half way down the page, as it will gives a good example for tab separated data:
FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\'
Check your data. Sometimes quoting or escaping has problems, and you need to adjust your source, import command-- or it may just be easier to post-process via SQL.
There are actually several ways to import an excel file in to a MySQL database with varying degrees of complexity and success.
Excel2MySQL. Hands down, the easiest and fastest way to import Excel data into MySQL. It supports all verions of Excel and doesn't require Office install.
LOAD DATA INFILE: This popular option is perhaps the most technical and requires some understanding of MySQL command execution. You must manually create your table before loading and use appropriately sized VARCHAR field types. Therefore, your field data types are not optimized. LOAD DATA INFILE has trouble importing large files that exceed 'max_allowed_packet' size. Special attention is required to avoid problems importing special characters and foreign unicode characters. Here is a recent example I used to import a csv file named test.csv.
phpMyAdmin: Select your database first, then select the Import tab. phpMyAdmin will automatically create your table and size your VARCHAR fields, but it won't optimize the field types. phpMyAdmin has trouble importing large files that exceed 'max_allowed_packet' size.
MySQL for Excel: This is a free Excel Add-in from Oracle. This option is a bit tedious because it uses a wizard and the import is slow and buggy with large files, but this may be a good option for small files with VARCHAR data. Fields are not optimized.
Not sure if you have all this setup, but for me I am using PHP and MYSQL. So I use a PHP class PHPExcel. This takes a file in nearly any format, xls, xlsx, cvs,... and then lets you read and / or insert.
So what I wind up doing is loading the excel in to a phpexcel object and then loop through all the rows. Based on what I want, I write a simple SQL insert command to insert the data in the excel file into my table.
On the front end it is a little work, but its just a matter of tweaking some of the existing code examples. But when you have it dialed in making changes to the import is simple and fast.
the best and easiest way is to use "MySQL for Excel" app that is a free app from oracle. this app added a plugin to excel to export and import data to mysql. you can download that from here
When using text files to import data, I had problems with quotes and how Excel was formatting numbers. For example, my Excel configuration used the comma as decimal separator instead of the dot.
Now I use Microsoft Access 2010 to open my MySql table as linked table. There I can simply copy and paste cells from Excel to Access.
To do this, first install the MySql ODBC driver and create an ODBC connection.
Then in access, in the "External Data" tab, open "ODBC Database" dialog and link to any table using the ODBC connection.
Using MySql Workbench, you can also copy and paste your Excel data into the result grid of MySql Workbench. I gave detailed instructions in this answer.
Fastest and simpliest way is to save XLS as ODS (open document spreasheet) and import it from PhpMyAdmin
For a step by step example for importing Excel 2007 into MySQL with correct encoding (UTF-8) search for this comment:
"Posted by Mike Laird on October 13 2010 12:50am"
in the next URL:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
You could use DocChow, a very intuitive GIU for importing Excel into MySQL, and it's free on most common platforms (including Linux).
More especially if you are concerned about date, datetime datatypes, DocChow easily handles datatypes. If you are working with multiple Excel spreadsheets that you want to import into one MySQL table DocChow does the dirty work.
Step 1 Create Your CSV file
Step 2 log in to your mysql server
mysql -uroot -pyourpassword
Step 3
load your csv file
load data local infile '//home/my-sys/my-excel.csv' into table my_tables fields terminated by ',' enclosed by '"' (Country, Amount,Qty);
Another useful tool, and as a MySQL front-end replacement, is Toad for MySQL. Sadly, no longer supported by Quest, but a brilliant IDE for MySQL, with IMPORT and EXPORT wizards, catering for most file types.
If you are using Toad for MySQL steps to import a file is as follows:
create a table in MySQL with the same columns that of the file to be imported.
now the table is created, goto > Tools > Import > Import Wizard
now in the import wizard dialogue box, click Next.
click Add File, browse and select the file to be imported.
choose the correct dilimination.("," seperated for .csv file)
click Next, check if the mapping is done properly.
click Next, select the "A single existing table" radio button also select the table that to be mapped from the dropdown menu of Tables.
Click next and finish the process.
If you don't like plugins, VBA and external tools, I have an excel file that using formulas only allows you to create INSERT/UPDATES. You only have to put the data on the cells:
As an extra, there's another tab in the file to CREATE TABLES:
The file can be found on the following link:
EXCEL FILE
I've had good results with the Tools / Import CSV File feature in HeidiSQL, with CSV files directly exported from Excel 2019 with "Save As..."
It uses LOAD DATA INFILE internally but with a GUI interface and also analyzes the CSV file before passing it to LOAD DATA INFILE so it can, for example, create the table using the first row as column names and guessing the column data type (<New table> option as shown in the picture)
I have a PHP web app that can convert data into an excel file using PHPExcel. The web app is running inside a Linux machine.
These are the steps involved.
I need to run several join queries to get the data I want.
I then take the data and write them into the excel file using PHPExcel.
Repeat step 1-2 in batches of ~5000 rows.
I noticed that the time taken to generate ~50000 rows alone (which is about 10%) is nearly 15 minutes which is too long.
I need to generate this excel file once a day.
Off the top of my head, I have the following options.
a) Instead of doing step 1 and 2 repeatedly for about 10 times, I could retrieve all the data I need in step 1 and store the data into a separate MySQL table.
Then I try PHPExcel and read this table once and write the excel file once.
At the end, I will wipe this table clean again.
b) Do the same as a) but instead of using PHPExcel, find a way to run MySQL For Excel in a cloud Windows instance to generate the Excel file.
If need further formatting, I will use PHPExcel to format this Excel file.
The way to send data between Linux machine and the windows cloud instance is through email attachments because the Linux machine is within a firewall.
I am posting this question to see if anybody has a better idea.
In the meantime, I will try option a).
I have no idea how to run a windows instance and microsoft excel for option b).
As for me, the best way, is use mysql SELECT INTO OUTFILE. You can export your result to csv, that Microsoft Excel will open correctly
Example:
SELECT * INTO OUTFILE '/tmp/result.csv'
FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"'
ESCAPED BY ‘\\’
LINES TERMINATED BY '\n'
FROM table
After that, you can use php library like PHPexcel for convertion csv to xls
include 'PHPExcel/IOFactory.php';
$objReader = PHPExcel_IOFactory::createReader('CSV');
// If the files uses a delimiter other than a comma (e.g. a tab), then tell the reader
$objReader->setDelimiter("\t");
// If the files uses an encoding other than UTF-8 or ASCII, then tell the reader
$objReader->setInputEncoding('UTF-16LE');
$objPHPExcel = $objReader->load('MyCSVFile.csv');
$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel5');
$objWriter->save('MyExcelFile.xls');
This question already has answers here:
How to import an excel file in to a MySQL database
(15 answers)
Closed 9 years ago.
my boss wants me to create a mysql database for an existing excel file. I was wondering if there are any ways to do convert or import excel file? I have searched google but didn't find anything useful. I appreciate for any reply....Thanks.
Save as CSV
Use a COPY sql statement
Profit
If you have a web server up and running with PHP support, I highly recommend phpMyAdmin.
Configure it to connect to MySQL
Create the Database and table
Click the import tab and you can import a CSV.
If this is a simple one-time import this probably isn't worth the effort. If, on the other hand, you will ever have to work with MySQL again, you will never regret the time to install and configure phpMyAdmin.
PHPMyAdmin can import CSV files, as well as Excel files (though I've never tried it)
First you need to create your datebase, and add a table. There must be as many fields in that table as there are columns in your Excel document (yes, I know you know)
Then select that database and table in phpmyadmin and use the "Import" tab.
I wrote a tool that will let you do sql queries against a csv file. The output is saved as a csv as well. Maybe you will find it useful.
http://whitlock.ath.cx/EasyCSV/
From Excel, export the sheet as a text file. In MySQL, use LOAD DATA INFILE to import the text file.
easiest way to do it would be this:
insert into Table (col 1,col 2,col 3...col n) values (val1,...valn);
basically:
do 2 for loops in your excel:
dim i,j
dim sqlString,sqlBase
sqlString=""
sqlBase="insert into yourTable (col1....coln) values ("
for i=1 to myRowVariable
sqlString=""
for j=1 to myColVariable
if j=myColVariable then
sqlString=sqlString & Cells(i,j).value & ");"
else if j=1 then
sqlString=sqlBase & sqlString & ","
else
sqlString=sqlString & Cells(i,j).value & ","
end if
Next j
Next i
'write sqlString into a txt or something
this will do what you need in a bootstrap but fast and very intuitive way.
You can use an ODBC driver to "mount" an Excel file as database and then make SQL queries to it. All you need then, is a simple migration tool, to copy the tables to another databases system.
I believe there's even an mysqldump-like tool for ODBC driven databases.
A low tech solution would be to use the concatenation function in excel to convert the data into a series of insert statements and then copy and paste them into mysql query analyzer or whatever client you are using.