Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
Improve this question
I have an Excel file that has some Spanish characters (tildes, etc.) that I need to convert to a CSV file to use as an import file. However, when I do Save As CSV it mangles the "special" Spanish characters that aren't ASCII characters. It also seems to do this with the left and right quotes and long dashes that appear to be coming from the original user creating the Excel file in Mac.
Since CSV is just a text file I'm sure it can handle a UTF8 encoding, so I'm guessing it is an Excel limitation, but I'm looking for a way to get from Excel to CSV and keep the non-ASCII characters intact.
A simple workaround is to use Google Spreadsheet. Paste (values only if you have complex formulas) or import the sheet then download CSV. I just tried a few characters and it works rather well.
NOTE: Google Sheets does have limitations when importing. See here.
NOTE: Be careful of sensitive data with Google Sheets.
EDIT: Another alternative - basically they use VB macro or addins to force the save as UTF8. I have not tried any of these solutions but they sound reasonable.
I've found OpenOffice's spreadsheet application, Calc, is really good at handling CSV data.
In the "Save As..." dialog, click "Format Options" to get different encodings for CSV. LibreOffice works the same way AFAIK.
Save the Excel sheet as "Unicode Text (.txt)". The good news is that all the international characters are in UTF16 (note, not in UTF8). However, the new "*.txt" file is TAB delimited, not comma delimited, and therefore is not a true CSV.
(optional) Unless you can use a TAB delimited file for import, use your favorite text editor and replace the TAB characters with commas ",".
Import your *.txt file in the target application. Make sure it can accept UTF16 format.
If UTF-16 has been properly implemented with support for non-BMP code points, that you can convert a UTF-16 file to UTF-8 without losing information. I leave it to you to find your favourite method of doing so.
I use this procedure to import data from Excel to Moodle.
I know this is an old question but I happened to come upon this question while struggling with the same issues as the OP.
Not having found any of the offered solutions a viable option, I set out to discover if there is a way to do this just using Excel.
Fortunately, I have found that the lost character issue only happens (in my case) when saving from xlsx format to csv format. I tried saving the xlsx file to xls first, then to csv. It actually worked.
Please give it a try and see if it works for you. Good luck.
You can use iconv command under Unix (also available on Windows as libiconv).
After saving as CSV under Excel in the command line put:
iconv -f cp1250 -t utf-8 file-encoded-cp1250.csv > file-encoded-utf8.csv
(remember to replace cp1250 with your encoding).
Works fast and great for big files like post codes database, which cannot be imported to GoogleDocs (400.000 cells limit).
You can do this on a modern Windows machine without third party software. This method is reliable and it will handle data that includes quoted commas, quoted tab characters, CJK characters, etc.
1. Save from Excel
In Excel, save the data to file.txt using the type Unicode Text (*.txt).
2. Start PowerShell
Run powershell from the Start menu.
3. Load the file in PowerShell
$data = Import-Csv C:\path\to\file.txt -Delimiter "`t" -Encoding BigEndianUnicode
4. Save the data as CSV
$data | Export-Csv file.csv -Encoding UTF8 -NoTypeInformation
The only "easy way" of doing this is as follows. First, realize that there is a difference between what is displayed and what is kept hidden in the Excel .csv file.
Open an Excel file where you have the info (.xls, .xlsx)
In Excel, choose "CSV (Comma Delimited) (*.csv) as the file type and save as that type.
In NOTEPAD (found under "Programs" and then Accessories in Start menu), open the saved .csv file in Notepad
Then choose -> Save As... and at the bottom of the "save as" box, there is a select box labelled as "Encoding". Select UTF-8 (do NOT use ANSI or you lose all accents etc). After selecting UTF-8, then save the file to a slightly different file name from the original.
This file is in UTF-8 and retains all characters and accents and can be imported, for example, into MySQL and other database programs.
This answer is taken from this forum.
Another one I've found useful:
"Numbers" allows encoding-settings when saving as CSV.
Using Notepad++
This will fix the corrupted CSV file saved by Excel and re-save it in the proper encoding.
Export CSV from Excel
Load into Notepad++
Fix encoding
Save
Excel saves in CP-1252 / Windows-1252. Open the CSV file in Notepad++. Select
Encoding > Character Sets > Western European > Windows-1252
Then
Encoding > Convert to UTF-8
File > Save
First tell Notepad++ the encoding, then convert. Some of these other answers are converting without setting the proper encoding first, mangling the file even more. They would turn what should be ’ into 達. If your character does not fit into CP-1252 then it was already lost when it was saved as CSV. Use another answer for that.
"nevets1219" is right about Google docs, however if you simply "import" the file it often does not convert it to UTF-8.
But if you import the CSV into an existing Google spreadsheet it does convert to UTF-8.
Here's a recipe:
On the main Docs (or Drive) screen click the "Create" button and choose "Spreadsheet"
From the "File" menu choose "Import"
Click "Choose File"
Choose "Replace spreadsheet"
Choose whichever character you are using as a Separator
Click "Import"
From the "File" menu choose "Download as" -> CSV (current sheet)
The resulting file will be in UTF-8
Under Excel 2016 and up (including Office 365), there is a CSV option dedicated to the UTF-8 format.
In Office 365, do Save As; where previously one might have chosen CSV (Comma Delimited), now one of the file types you can save as is CSV UTF-8 (Comma delimited) (*.csv)
What about using Powershell.
Get-Content 'C:\my.csv' | Out-File 'C:\my_utf8.csv' -Encoding UTF8
For those looking for an entirely programmatic (or at least server-side) solution, I've had great success using catdoc's xls2csv tool.
Install catdoc:
apt-get install catdoc
Do the conversion:
xls2csv -d utf-8 file.xls > file-utf-8.csv
This is blazing fast.
Note that it's important that you include the -d utf-8 flag, otherwise it will encode the output in the default cp1252 encoding, and you run the risk of losing information.
Note that xls2csv also only works with .xls files, it does not work with .xlsx files.
Easiest way:
No need Open office and google docs
Save your file as "Unicode text file";
now you have an unicode text file
open it with "notepad" and "Save as" it with selecting "utf-8" or
other code page that you want
rename file extension from "txt" to "csv". This will result in a tab-delimited UTF-8 csv file.
If you want a comma-delimited file, open the csv file you just renamed and replace all tabs with commas. To do this in Notepad on Win 10, simply select one tab field then click Ctrl+H. In the window that opens, type a comma , in the "Replace with" field then click "Replace All". Save your file. The result will be a comma-delimited UTF-8 csv file.
Don't open it with MS-Office anyway!!!
Now you have a tab delimited CSV file.
Or, a comma-delimited one if you applied step number 5.
As funny as it may seem, the easiest way I found to save my 180MB spreadsheet into a UTF8 CSV file was to select the cells into Excel, copy them and to paste the content of the clipboard into SublimeText.
I was not able to find a VBA solution for this problem on Mac Excel. There simply seemed to be no way to output UTF-8 text.
So I finally had to give up on VBA, bit the bullet, and learned AppleScript. It wasn't nearly as bad as I had thought.
Solution is described here:
http://talesoftech.blogspot.com/2011/05/excel-on-mac-goodbye-vba-hello.html
Assuming an Windows environment, save and work with the file as usual in Excel but then open up the saved Excel file in Gnome Gnumeric (free). Save Gnome Gnumeric's spreadsheet as CSV which - for me anyway - saves it as UTF-8 CSV.
Easy way to do it: download open office (here), load the spreadsheet and open the excel file (.xls or .xlsx). Then just save it as a text CSV file and a window opens asking to keep the current format or to save as a .ODF format. select "keep the current format" and in the new window select the option that works better for you, according with the language that your file is been written on. For Spanish language select Western Europe (Windows-1252/ WinLatin 1) and the file works just fine. If you select Unicode (UTF-8), it is not going to work with the spanish characters.
Save xls file (Excel file) as Unicode text=>file will be saved in text format (.txt)
Change format from .txt to .csv (rename the file from XYX.txt to XYX.csv
I have also came across the same problem but there is an easy solution for this.
Open your xlsx file in Excel 2016 or higher.
In "Save As" choose this option: "(CSV UTF-8(Comma Delimited)*.csv)"
It works perfectly and a csv file is generated which can be imported in any software. I imported this csv file in my SQLITE database and it works perfectly with all unicode characters intact.
Came across the same problem and googled out this post. None of the above worked for me. At last I converted my Unicode .xls to .xml (choose Save as ... XML Spreadsheet 2003) and it produced the correct character. Then I wrote code to parse the xml and extracted content for my use.
I have written a small Python script that can export worksheets in UTF-8.
You just have to provide the Excel file as first parameter followed by the sheets that you would like to export. If you do not provide the sheets, the script will export all worksheets that are present in the Excel file.
#!/usr/bin/env python
# export data sheets from xlsx to csv
from openpyxl import load_workbook
import csv
from os import sys
reload(sys)
sys.setdefaultencoding('utf-8')
def get_all_sheets(excel_file):
sheets = []
workbook = load_workbook(excel_file,use_iterators=True,data_only=True)
all_worksheets = workbook.get_sheet_names()
for worksheet_name in all_worksheets:
sheets.append(worksheet_name)
return sheets
def csv_from_excel(excel_file, sheets):
workbook = load_workbook(excel_file,use_iterators=True,data_only=True)
for worksheet_name in sheets:
print("Export " + worksheet_name + " ...")
try:
worksheet = workbook.get_sheet_by_name(worksheet_name)
except KeyError:
print("Could not find " + worksheet_name)
sys.exit(1)
your_csv_file = open(''.join([worksheet_name,'.csv']), 'wb')
wr = csv.writer(your_csv_file, quoting=csv.QUOTE_ALL)
for row in worksheet.iter_rows():
lrow = []
for cell in row:
lrow.append(cell.value)
wr.writerow(lrow)
print(" ... done")
your_csv_file.close()
if not 2 <= len(sys.argv) <= 3:
print("Call with " + sys.argv[0] + " <xlxs file> [comma separated list of sheets to export]")
sys.exit(1)
else:
sheets = []
if len(sys.argv) == 3:
sheets = list(sys.argv[2].split(','))
else:
sheets = get_all_sheets(sys.argv[1])
assert(sheets != None and len(sheets) > 0)
csv_from_excel(sys.argv[1], sheets)
Excel typically saves a csv file as ANSI encoding instead of utf8.
One option to correct the file is to use Notepad or Notepad++:
Open the .csv with Notepad or Notepad++.
Copy the contents to your computer clipboard.
Delete the contents from the file.
Change the encoding of the file to utf8.
Paste the contents back from the clipboard.
Save the file.
A second option to "nevets1219" is to open your CSV file in Notepad++ and do a convertion to ANSI.
Choose in the top menu :
Encoding -> Convert to Ansi
Encoding -> Convert to Ansi will encode it in ANSI/UNICODE. Utf8 is a subset of Unicode. Perhaps in ANSI will be encoded correctly, but here we are talking about UTF8, #SequenceDigitale.
There are faster ways, like exporting as csv ( comma delimited ) and then, opening that csv with Notepad++ ( free ), then Encoding > Convert to UTF8. But only if you have to do this once per file. If you need to change and export fequently, then the best is LibreOffice or GDocs solution.
Microsoft Excel has an option to export spreadsheet using Unicode encoding. See following screenshot.
open .csv fine with notepad++. if you see your encoding is good (you see all characters as they should be) press encoding , then convert to ANSI
else - find out what is your current encoding
another solution is to open the file by winword and save it as txt and then reopen it by excel and it will work ISA
Save Dialog > Tools Button > Web Options > Encoding Tab
I have the same problem and come across this add in , and it works perfectly fine in excel 2013 beside excel 2007 and 2010 which it is mention for.
I am looking for a way to open a CSV file, that was created with a PHP script, in Excel - in such a way that Excel knows how to parse the file. Currently when i double click on a CSV file created with PHP, Excel opens the content in a single column so it does not parse each line. Also, if i do CTRL-O in Excel and select the CSV file to be opened, Excel launches a wizard where i am able to select parsing and encoding option.
Are there any 'headers' or flag characters that i could prepend to the CSV output in PHP to let Excel know how to open a file? I know, for example, that in order for Excel to handle UTF8 encoding, a U+FEFF character needs to be included as the first character in the CSV file, so maybe there is something similar for parsing?
Thanks.
Beware that depending on your xcel version you will or will not have the csv options dialog in Excel when opening direclty the csv file.
latest versions of Excel I've tested needs a special menu usage to get this dialog.
So you should provides what Excels wants.
And what he wants is a tab-separated csv (not comas, funny enough when he save a csv file he use comas but not in his auto-import), without " and without carriage returns in cells, and not in utf8.
Some says he need some sort of UTF16, I can't remember exactly, certainly the UTF-32LE BOM cited by Mark Baker. You will certainly have to transcode your chars.
Then do not forget to set tes text/csv mime type header.
When I see this broken auto-import csv without dialog of the new Excel I wonder if they didn't want to avoid complelty csv usage :-)
Ho, and I saw somewhere in past that there some mysterious formating commands you can use in an pure HTML table export that Excel will understand really better than the csv format.
You should search a lttle about it, maybe really simplier.
I need to implement a functionality where a user can upload an Excel file, that is, .xls file and i need to write a function that will read that file and then save the values in a table, compare them to values in another table and then print some kind of billing quotation.
From what I have read, it seems that the best way to do that is to first convert that file to an xml file and then write a function to parse that xml file.
I want to know, is it possible to write a function in php that will convert an xls file to xml. Assuming that the users will save and upload the file as xls.
Thanks.
i often let admin users upload a csv file they exported from an xcel file, its much simpler to parse through using http://php.net/manual/en/function.fgetcsv.php
A quick Google search for "PHP excel parser" turns up PHP Excel Reader, for an open source library to use for reading data from .xls files. There may be others that suit your needs.
I have been successfully using the 'ABC Excel Parser Pro' from Zakkis.
This allows converting from a XLS file directly into a MySQL table.
A few notes on limitations:
column count must match exactly to database columns
Excel 2007 format (xlsx) are not supported
I have tabulated data in an Excel spreadsheet (file size will likely never be larger than 1 mb). I want to use PHP to parse the data and insert in to a MySQL database.
Is there any advantage to keeping the file as an .xls/.xlsx and parsing it using a PHP Excel Parsing Library? If so, what are some good libraries to use?
Obviuously, I can save the .xls/.xlsx as a CSV and handle the file that way.
Thanks!
If you are just after the values, I would save it as a CSV. This is much easier to parse programatically, especially if you are trying to do this on a non-windows box.
That being said, there will be information lost in the export to CSV. It will only save the values of the cells - not their formatting information, formulas, etc. If you need to use that information, you're better off doing this straight from Excel.
Here is a PHP Excel Reading library. If you decide to read Excel files directly, this may help get you started.
If your excel files contain strictly data and contain no formulas, scripts, macros and etc., I would say parsing through Excel will only add development overhead, and will potentially slow down processing. It would probably be best to convert the files to CSV in this case.
Also consider that MySQL's 'LOAD DATA INFILE' command can be used to import entire CSV files into a table, this can potentially further uncomplicate matters for you.
when you provide a way for customers to upload excel/csv files, you should consider that
CSV files will only export one sheet
Having multiline cells will make the CSV parsing complicated
You cannot easily detect corrupted/incomplete CSV files
CSV files do not include formatting
Besides from that, importing CSV is a lot easier than importing XLS.
Remember that if you're importing the csv file directly into Mysql, that you may have problems with the date format (as Mysql uses a different date format to Excel). I find it easier to change the date fields in Excel first (to format yyyy-mm-dd) prior to saving as a csv file.
Edit: Although I've not used it myself, others have recommended Navicat as a very good tool for converting Excel spreadsheets or Access data into Mysql databases. May be worth a look.
With Office 2003 there's an XML format called SpreadsheetML which is a bit in-between XML and Excel. I've considered using this format to import/export data to a web site but the format turns out to be a bit complex. Internally, this format turns all references into relative references. (Relative from the current location.) Worse, some cells have an index, thus you might see a row with only two cells, but the second cell might be 6 columns away from the first cell. (In which case Index=5.) Basically, if you want to use the Excel format, you will need to have a good way to calculate the position of each cell and know how to translate the references in the cells in a proper way.
If you're onlyinterested in the data, CSV would be much, much easier to implement. As an in-between solution, you could define an XML schema and add an XML mapping to your spreadsheet to export the data to an XML file. It's more complex than CSV i9mport/export, but also a bit more robust. But the Excel or Excel XML formats themselves are horrible to implement. (Or just a nice challenge, if you're a real XML expert.)