I'm attempting to clean up data in this (old) spreadsheet and need to remove things like single and double quotes, HTML tags and so on. Trouble is, it's a 3000 row file with 25 columns and every spreadsheet app I've tried (NeoOffice, MS Excel, Apple Numbers) chokes on it. Hard.
Any ideas on how else I can clean this thing up for import to MySQL? Clearly I could go through each record manually, row by row, but would like to avoid that if at all possible. Likewise, I could write a PHP script to handle it on import, but don't want to put the server into a death spiral either.
If it's in Excel format, you might try to access it through python.
http://www.python-excel.org/
I've never done this myself, but it might work in your case.
I found a solution:
Exported the spreadsheet as CSV, and with my desktop running PHP and MySQL I imported the csv file into MySQL using the "LOAD DATA" SQL command, and then used PHP + MySQL to clean up each row.
Much, much faster than asking a desktop app like Excel or Numbers to do a find & replace.
Related
i have been reading around looking for a way to get graphs and so on into my spread sheet i export out of my database and to make it look better for my employee who wants reports of everything in a certain way and so on. I have found PhpExcel.. which seems really good it can do everything i want it to plus more.. but i haven't been able to work out how to use data from my mysql table to fill in the cells since it seems like they all have to be manually set to a value...
i have read over this post How to set cell value Dynamically
i have attempted to use the code provided in this page to try and help me out but i havent been able to get it to work for some reason i am always receiving an error.
i am hoping someone may be able to help me out with trying to get this MySQL table data into a spreadsheet using php excel or even if there are any other links i may have missed i would really appreciate it...
If you install phpmyadmin to connect to your database.
Then you can write SQL queries for whatever reports you need.
And Export (see image 1) the results to CSV or CSV for MS Excel (see image 2).
Export option below the query results
Choose export data format
I have done this using the PEAR classes for PHP. You can read the data from the MYSQL database and then write the script using the PHP and PEAR classes to create an excel file for the same.
Here is link to the SpreadSheet Writer
This tutorial will help you to install PEAR and write the data to the excel sheet.
Hope this will solve the problem you are facing.
I'm doing a lot of work with PHPExcel... I wrote a little class to simplify exporting, you can download it here.
You just need to get your data from database into array, pass it to this class and specify the output format. I hope it helps.
I am looking for a way to automate an entire process that uses Excel & ODBC queries. I know that Excel can use to a ODBC driver to run queries against a Mysql database. But I need this query to be triggered programmatically.
The individual steps the program or script should be able to do are:
1. Open Excel file
2. Run Mysql query (query will not change but the values the query hits the database for will come from the excel file)
3. Save results of the query to the Excel file as a sheet
4. Use the results to do vlookups against another sheet in the same excel file
5. save results of vlookups and close the file
Flexible on the language or any add-ons necessary. Anything out there that would help? I am looking to run this both on windows and mac.
What you want to do is probably possible using COM (Python tutorial). It will be messy, hard to code and hard to debug. And no way it'll work on a mac.
Instead, if I was you I would try and take the problem out of excel. For instance in Python, I would first directly access mysql. I would then parse my second xls file using xlrd. With the results of the mysql query and the parsed xls file both in Python data structures, replicating the behaviour of VLOOKUP is easy. I would then write the results to my output xls file using xlwt.
Using PHPExcel I can run each tab separately and get the results I want but if I add them all into one excel it just stops, no error or any thing.
Each tab consists of about 60 to 80 thousand records and I have about 15 to 20 tabs. So about 1600000 records split into multiple tabs (This number will probably grow as well).
Also I have tested the 65000 row limitation with .xls by using the .xlsx extension with no problems if I run each tab it it's own excel file.
Pseudo code:
read data from db
start the PHPExcel process
parse out data for each page (some styling/formatting but not much)
(each numeric field value does get summed up in a totals column at the bottom of the excel using the formula SUM)
save excel (xlsx format)
I have 3GB of RAM so this is not an issue and the script is set to execute with no timeout.
I have used PHPExcel in a number of projects and have had great results but having such a large data set seems to be an issue.
Anyone every have this problem? work around? tips? etc...
UPDATE:
on error log --- memory exhausted
Besides adding more RAM to the box is there any other tips I could do?
Anyone every save current state and edit excel with new data?
I had the exact same problem and googling around did not find a valuable solution.
As PHPExcel generates Objects and stores all data in memory, before finally generating the document file which itself is also stored in memory, setting higher memory limits in PHP will never entirely solve this problem - that solution does not scale very well.
To really solve the problem, you need to generate the XLS file "on the fly". Thats what i did and now i can be sure that the "download SQL resultset as XLS" works no matter how many (million) row are returned by the database.
Pity is, i could not find any library which features "drive-by" XLS(X) generation.
I found this article on IBM Developer Works which gives an example on how to generate the XLS XML "on-the-fly":
http://www.ibm.com/developerworks/opensource/library/os-phpexcel/#N101FC
Works pretty well for me - i have multiple sheets with LOTS of data and did not even touch the PHP memory limit. Scales very well.
Note that this example uses the Excel plain XML format (file extension "xml") so you can send your uncompressed data directly to the browser.
http://en.wikipedia.org/wiki/Microsoft_Office_XML_formats#Excel_XML_Spreadsheet_example
If you really need to generate an XLSX, things get even more complicated. XLSX is a compressed archive containing multiple XML files. For that, you must write all your data on disk (or memory - same problem as with PHPExcel) and then create the archive with that data.
http://en.wikipedia.org/wiki/Office_Open_XML
Possibly its also possible to generate compressed archives "on the fly", but this approach seems really complicated.
I have a PHP script that imports up to 10 or so different CSV files. Each row of each file contains bank account info, including balance. After I've imported all the CSV data into my database, I'd like to make sure the data got in there correctly by comparing the total account balance in the database to the total account balance of the CSV files.
I see at least a few options:
Manually total up all the account balances in Excel - yuck.
Write a PHP script to read each CSV file and total up the account balances - also yuck.
Some third option that I hope exists. It would be amazing if I could do something like:
excel --file="cd.csv" | sum --column="E"
That's obviously not a real thing but hopefully you get the idea. Using some combination of PHP, MySQL, Linux commands, Excel and/or any other tools, is there a simple way to do this?
Don't have to complete answer for you, but AWK should be able to solve your problem: Have a look at these 2 posts:
https://superuser.com/questions/53652/transforming-csv-file-using-sed
Shell command to sum integers, one per line?
I'm not enough of a AWK expert to give the solution, but perhaps someone else can help us here.
Another option (which you may also consider yuck) is to use a library like PHPExcel
You can iterate over the CSV file using fgetcsv() which converts each line to an array of values. You can accumulate the value of the array element containing the balance as you move thru each iteration until you get the sum total. Use glob to get the list of CSV files in a folder.
You may not have to "manually" total up the account balances, if you can use Excel functions from your application, the Excel formula in VBA would be:
Application.Sum(Range("A:A"))
where A:A is for column A.
Try using CSVFix with it's summary option. It will get you more data than you need, but should be easy to use.
Otherwise, this sounds like a good use for Awk.
Why can't you just automagically total up the account balances in excel with a formula and export them with the rest of the data?
A bit of a different angle: Make use of the MySql CSV engine to expose your CSV files to Mysql and then do a normal SQL SUM.
See also: The CSV Storage Engine
This question already has answers here:
How to import an excel file in to a MySQL database
(15 answers)
Closed 9 years ago.
my boss wants me to create a mysql database for an existing excel file. I was wondering if there are any ways to do convert or import excel file? I have searched google but didn't find anything useful. I appreciate for any reply....Thanks.
Save as CSV
Use a COPY sql statement
Profit
If you have a web server up and running with PHP support, I highly recommend phpMyAdmin.
Configure it to connect to MySQL
Create the Database and table
Click the import tab and you can import a CSV.
If this is a simple one-time import this probably isn't worth the effort. If, on the other hand, you will ever have to work with MySQL again, you will never regret the time to install and configure phpMyAdmin.
PHPMyAdmin can import CSV files, as well as Excel files (though I've never tried it)
First you need to create your datebase, and add a table. There must be as many fields in that table as there are columns in your Excel document (yes, I know you know)
Then select that database and table in phpmyadmin and use the "Import" tab.
I wrote a tool that will let you do sql queries against a csv file. The output is saved as a csv as well. Maybe you will find it useful.
http://whitlock.ath.cx/EasyCSV/
From Excel, export the sheet as a text file. In MySQL, use LOAD DATA INFILE to import the text file.
easiest way to do it would be this:
insert into Table (col 1,col 2,col 3...col n) values (val1,...valn);
basically:
do 2 for loops in your excel:
dim i,j
dim sqlString,sqlBase
sqlString=""
sqlBase="insert into yourTable (col1....coln) values ("
for i=1 to myRowVariable
sqlString=""
for j=1 to myColVariable
if j=myColVariable then
sqlString=sqlString & Cells(i,j).value & ");"
else if j=1 then
sqlString=sqlBase & sqlString & ","
else
sqlString=sqlString & Cells(i,j).value & ","
end if
Next j
Next i
'write sqlString into a txt or something
this will do what you need in a bootstrap but fast and very intuitive way.
You can use an ODBC driver to "mount" an Excel file as database and then make SQL queries to it. All you need then, is a simple migration tool, to copy the tables to another databases system.
I believe there's even an mysqldump-like tool for ODBC driven databases.
A low tech solution would be to use the concatenation function in excel to convert the data into a series of insert statements and then copy and paste them into mysql query analyzer or whatever client you are using.