I know this question might be redundant, but I had to go through them and I cannot solve my problem.
1 - The data is in the DB and stored in UTF-8 encoding.
2 - The connection charset also had been set to UTF8.
3 - Have tried manually encode the value to UTF8 encode while printing using encode_utf8($value) where the $value is the Chinese character.
I used PHP 5.621, and PHPExcel class from http://www.codeplex.com/PHPExcel .
The result is still showing the Chinese character as "赛维网络".
I am not printing the result on the page. I am creating a new xlsx file and printing onto it. The Chinese character is unreadable on the xlsx file, not on the page. On the page yes I can use the meta charset. But the problem is in the xlsx file.
Any advice for the best solution?
If your content is already UTF-8 and you encode it again you certainly will get broken characters.
So, just omit the third step.
I noticed some people with Non-Western had issues with XLSX/Excel-2077 too, maybe it's a bug. Check the PHPExcel GitHub tracker.
Related
I am doing the folloiwing steps to import data into my e-commerce shop:
convert excel sheet to csv in excel
open csv file in notepad++ and convert to UTF-8
import csv file in phpmyadmin
If I look at the front end of the webpage the french characters are displayed as ?. The charset of the page is utf-8
If I change the charset to iso-8859-1 everything displays correctly.
If I check the item in the phpmyadmin the accents are displayed correctly.
How come utf-8 is not displaying them correctly? I thought it should display é etc.
If i go to the back end of the website and edit the product, the french description displays properly in the WYSIWYG editor. If I save then the product the french characters then show correctly. But this is because the WYSIWYG editor is converting the characters to html entities.
A common issue when collecting Unicode DATA is leaving the Connection and database/table/column character set configurad as ISO-8859-1, but then inserting data that is actually utf-8. The database is essentially told, "here's some 8859-1-encoded data, store it in this 8859-1 table". It doesn't do any conversions because it doesn't realize the data isn't in 8859-1. So the data is utf-8 but the database has essentially been told it's in 8859-1.
It's an insidious problem because, as you say, the database will convert them wrongly if you change your charset to UtF-8, since it will convert the "8859-1" data (remmember the databae thinks it's 8859-1) to utf-8 - a conversion that fails of course, as the data really is in utf-8.
So basically the problem is that phpmyadmin is in 8859-1 but you told it to insert the data in 8859-1 and then told it you were providing data in 8859-1, and then gave it utf-8 data. The database thinks it's 8859-1 so the only easy way to solve the problem is to a) keep acting like it's 8859-1 even though it's not, and hope you never have to deal with sorting, searching, collation, etc ( may work in your case), or b) pulling out the data as 8859-1 ( leaving it unconverted ), then re-inserting it after setting the database and connection to utf-8 so the database knows what character set the data really is in.
Hope that makes sense. Let me know if it doesn't. This is a hard one to wrap your head around.
You might consider opening your csv with PHP (since you mention it in your tags), and use utf8_encode on the fields before saving them with queries.
This question is so old, but changing the encoding of the file from ISO-8859-1 to UTF-8 in various programs such as Excel etc was not working for me.
My issue is words like intérêt shows up as intérêt in the file.
In case this helps someone, here is what finally worked for me:
Starting with a CSV file, open in Notepad
Click "File > Save As".
In the dialog window that appears - select "ANSI" from the "Encoding" field. Then click "Save".
That's it! Opening this new CSV file using Excel should now show the non-English characters properly.
I'd really appreciate some help with this. I've wasted days on this problem and none of the suggestions I have found online seem to give me a fix.
I have a CSV file from a supplier. It appears to have been exported from an Microsoft system.
I'm using PHP to import the data into MySQL (both latest versions).
I have one particular record which contains a strange character that I can't get rid of. Manual editing to remove the character is possible, but I would prefer an automated solution as this will happen multiple times a day.
The character appears to be an interpretation of a “smart quote”. A hex editor tells me that the character codes are C2 and 92. In the hex editor it looks like a weird A followed by a smart quote. In other editors and Calc, Writer etc it just appears as a box. ﾒ
I'm using mb_detect_encoding to determine the encoding. All records in the CSV file are returned as ASCII, except the one with the strange character, which is returned as UTF-8.
I can insert the offending record into MySQL and it just appears in Workbench as a square.
MySQL tables are configured to utf-8 – utf8_unicode_ci and other unusual UTF characters (eg fractions) are ok.
I've tried lots of solutions to this...
How to detect malformed utf-8 string in PHP?
Remove non-utf8 characters from string
Removing invalid/incomplete multibyte characters
How to detect malformed utf-8 string in PHP?
How to replace Microsoft-encoded quotes in PHP
etc etc but none of them have worked for me.
All I really want to do is remove or replace the offending character, ideally with a search and replace for the hex values but none of the examples I have tried have worked.
Can anyone help me move forward with this one please?
EDIT:
Can't post answer as not enough reputation:
Thanks for your input. Much appreciated.
I'm just going to go with the hex search and replace:
$DodgyText = preg_replace("/\xEF\xBE\x92/", "" ,$DodgyText);
I know it's not the elegant solution, but I need a quick fix and this works for me.
Another solution is:
$contents = iconv('UTF-8', 'Windows-1251//IGNORE',$contents);
$contents = iconv('Windows-1251', 'UTF-8//IGNORE',$contents);
Where you can replace Windows-1251 to your local encoding.
At a quick glance, this looks like a UTF-8 file. (UTF-8 is identical with the first 128 characters in the ASCII table, hence everything is detected as ASCII except for the special character.)
It should work if your database connection is also UTF-8 encoded (which it may not be by default).
How to do that depends on your database library, let us know which one you're using if you need help setting the connection encoding.
updated code based on established findings
You can do search & replace on strings using hexadecimal notation:
str_replace("\xEF\xBE\x92", '', $value);
This would return the value with the special code removed
That said, if your database table is UTF-8, you shouldn't need that conversion; instead you could look at the connection (or session) character set (i.e. SET NAMES utf8;). Configuring this depends on what library you use to connect to your database.
To debug the value you could use bin2hex(); this usually helps in doing searches online.
I am using flash to read contents from a UTF8 page, which has unicode in it.
The problem is that when Flash loads the data it displays ???????? instead all unicode.
What could be the problem?
By default Flash treats strings as if they are encoded using UTF-8. The reason that you are seeing characters that possibly substitute non-printable characters or invalid / missing glyphs could be that you set System.useCodepage to true - if that's what happened, then why did you do that?
Otherwise, the font that is used to display the characters may be missing glyphs for the characters you need. You can check that by using Font.hasGlyphs("string with the glyphs"); to make sure the text can be displayed. This would normally only apply to embedded fonts.
Yet another possibility is that the source text you are trying to display is not a UTF-8 encoded string. Some particularly popular file formats such as XML and HTML some times use a declaration of the format in no correspondence to the actual payload (example XML tag: <?xml encoding="utf-8" ?> can be attached to any XML regardless of the actual encoding of the document). In order to make sure that the text is in UTF-8 - read it as ByteArray and verify that the first bit of every byte is set to 0. Single-byte encodings that use national characters use the first bit to encode their characters, while UTF-8 never does that.
Flash internally uses UTF-8 to represent strings, so there should not be a problem if the entire stack uses UTF-8 encoding.
You probably have an implicit decode/encode step somewhere along the way.
This could really be a million things, unfortunately. Start from the ground up, insert traces and/or log messages to see where the conversion fails. Make sure your XML-content uses UTF-8, and especially if you're using PHP, make sure that all the PHP source files are saved in UTF-8 encoding - editing PHP files in simple text editors often results in Windows/Mac format source files, which will then break your character encoding. Also, verify HTML request/response headers to see if there is an encoding mismatch.
I have two tables here - one is in UTF and holds Arabic text as it can be read. The other one has a different encoding however and the content is Arabic however in the database its displayed as
ÈöÓúãö Çááøåö ÇáÑøóÍúãóäö ÇáÑøóÍöíãö
I have to show data from both tables on the same page - the page is UTF encoded however I'm not sure if this can be done or if its possible. What do i do? My database is mysql and I'm using php.
Is it possible to convert the encoding of the contents of the other table into UTF8 btw?
You have to use mb_convert_encoding() first, on everything, to make sure it's all in UTF-8 to begin with. http://us3.php.net/manual/en/function.mb-convert-encoding.php Then it should display, assuming your HTML's charset is UTF-8 and the users have the appropriate fonts installed.
Also, virtually all consoles and a great many free online SQL commanders (like PHPMyAdmin) are not UTF-8 aware and print out jibberish. I have not yet found a free SSH client that supports UTF-8; if it's a big deal, invest in SecureCRT.
EDIT:
Excuse me. I don't read Arabic at all, but I did get Arabic back. please tell me if this is the correct text, and if so, accept this answer ;_)
ب?س?ك? افف?م? افر??ح?ك?ل? افر??ح?ٍك?
The code I used to get this was:
header('Content-Type: text/html;charset=utf-8');
echo mb_convert_encoding('ÈöÓúãö Çááøåö ÇáÑøóÍúãóäö ÇáÑøóÍöíãö', 'utf-8', 'iso-8859-6');
I found the Arabic encoding via this page: http://a4esl.org/c/charset.html
Cheers!
So, I have built on this system for quite some time, and it is currently outputting Latin1 (ISO-8859-1) to the web browser, and this is the components:
MySQL - all data is stored with the Latin1 character set
PHP - All PHP text files are stored on disk with Latin1 encoding
HTML - The output has the http-equiv="content-type" content="text/html; charset=iso-8859-1" meta tag
So, I'm trying to understand how the encoding of the different parts come into play in my workflow. If I open a PHP script and change its encoding within the text editor to UTF-8 and save it back to disk and reload the web browser, the text is all messed up - unless the text comes from the DB. If I change the encoding of the DB to UTF-8 and keep the PHP files in latin1 I have to use utf8_decode() for the data to display correctly. And if I change the HTML code the browser will read it incorrectly.
So yeah, I realise that if I want to "upgrade" to UTF8, I have to update all three parts of this setup for it to work correctly, but since it's a huge system with some 180k lines of PHP code and millions of posts in a lot of databases/tables, I don't want to start something like this without understanding everything correctly.
What haven't I thought about? What could mess this up beyond fixing? What are the procedures for changing the encoding of an entire MySQL installation and what's the easiest way to change the encoding of hundreds or thousands of PHP files on disk?
The META tag is luckily added dynamically, so I'll change that in one place only :)
Let me hear about your experiences with this.
It's tricky.
You have to:
change the DB and every table character set/encoding – I don't know much about MySQL, but see here
set the client encoding to UTF-8 in PHP (SET NAMES UTF8) before the first query
change the meta tag and possible the Content-type header (note the Content-type header has precedence)
convert all the PHP files to UTF-8 w/out BOM – you can easily do that with a loop and iconv.
the trickiest of all: you have to change most of your string function calls. Than means mb_strlen instead of strlen, mb_substr instead of substr and $str[index], etc.
Don't convert to UTF8 if you don't have to. Its not worth the trouble.
UTF8 is (becoming) the new standard, so for new projects I can recommend it.
Functions
Certain function calls don't work anymore. For latin1 it's:
echo htmlentities($string);
For UTF8 it's:
echo htmlentities($string, ENT_COMPAT, 'UTF-8');
strlen(), substr(), etc. Aren't aware of the multibyte characters.
MySQL
mysql_set_charset('UTF8') or mysql_query('SET NAMES UTF8') will convert all text to UTF8 coming from the database(SELECTs). It will also convert incoming strings(INSERT, UPDATE) from UTF8 to the encoding of the table.
So for reading from a latin1 table it's not necessary to convert the table encoding.
But certain characters are only available in unicode (like the snowman ☃, iPhone emoticons, etc) and can't be converted to latin1. (The data will be truncated)
Scripts
I try to prevent specials-characters in my php-scripts / templates.
I use the ë notation instead of ë etc. This way it doesn't matter if is saved in latin1 or utf8.