Quick Background: I inherited a large sql dump file containing a combination of english and arabic text and (I think) it was originally exported using 'latin1'. I changed all occurrences of 'latin1' to 'utf8' prior to importing the file. The the arabic text didn't appear correctly in phpmyadmin (which I guess is normal), but when I loaded the text to a web page with the following...
<meta http-equiv='Content-Type' content='text/html; charset=windows-1256'/>
...everything looked good and the arabic text displayed perfectly.
Problem: My client is really really really picky and doesn't want to change his...
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
...to the 'Windows-1256' equivalent. I didn't think this would be a problem, but when I changed the charset value to 'UTF-8', all of the arabic characters appeared as diamonds with question marks. Shouldn't UTF-8 display arabic text correctly?
Here are a few notes about my database configuration:
Database charset is 'utf8'
Database connection collation is 'utf8_general_ci'
All databases, tables, and applicable fields have been collated as 'utf8_general_ci'
I've been scouring stack overflow and other forums for anything the relates to my issue. I've found similar problems, but not of the solutions seem to work for my specific situation. Hope someone can help!
If the document looks right when declared as windows-1256 encoded, then it most probably is windows-1256 encoded. So it was apparently not exported using latin1—which would have been impossible, since latin1 has no Arabic letters.
If this is just about a single file, then the simplest way is to convert it from windows-1256 encoding to utf-8 encoding, using e.g. Notepad++. (Open the file in it, change the encoding, via File format menu, to Arabic, windows-1256. Then select Convert to UTF-8 in the File format menu and do File → Save.)
Windows-1256 and UTF-8 are completely different encodings, so data gets all messed up if you declare windows-1256 data as UTF-8 or vice versa. Only ASCII characters, such as English letters, have the same representation in both encodings.
We can't find the error in your code if you don't show us your code, so we're very limited in how we can help you.
You told the browser to interpret the document as being UTF-8 rather than Windows-1256, but did you actually change the encoding used from Windows-1256 to UTF-8?
For example,
$ cat a.pl
use strict;
use warnings;
use feature qw( say );
use charnames ':full';
my $enc = $ARGV[0] or die;
binmode STDOUT, ":encoding($enc)";
print <<"__EOI__";
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=$enc">
<title>Foo!</title>
</head>
<body dir="rtl">
\N{ARABIC LETTER ALEF}\N{ARABIC LETTER LAM}\N{ARABIC LETTER AIN}\N{ARABIC LETTER REH}\N{ARABIC LETTER BEH}\N{ARABIC LETTER YEH}\N{ARABIC LETTER TEH MARBUTA}
</body>
</html>
__EOI__
$ perl a.pl UTF-8 > utf8.html
$ perl a.pl Windows-1256 > cp1256.html
I think you need to go back to square one. It sounds like you have a database dump in Win-1256 encoding and you want to work with it in UTF-8 from now on. It also sounds like you are using PHP but you have lots of irrelevant tags on your question and are missing the most important one, PHP.
First, you need to convert the text dump into UTF-8 and you should be able to do that with PHP. Chances are that your conversion script will have two steps, first read the Win-1256 bytes and decode them into internal Unicode text strings, then encode the Unicode text strings into UTF-8 bytes for output to a new text file.
Once you have done that, redo the database import as you did before, but now you have correctly encoded the input data as UTF-8.
After that it should be as simple as reading the database and rendering a web page with the correct UTF-8 encoding.
P.S. It is actually possible to reencode the data every time you display it, but that does not solve the problem of having a database full of incorrectly encoded data.
inorder to display arabic characters correctly , you need to convert your php file to utf-8 without Bom
this happened with me, arabic characters was displayed diamonds, but conversion to utf-8 without bom will solve this problem
I seems that the db is configured as UTF8, but the data itself is extended ascii. If the data is converted to UTF8, it will display correctly in content type set to UTF8
Related
I work on a website that has different language interfaces, so far I use english and german.
when the german text is loaded, it shows weird characters like the following screenshot
though I use
header('Content-type: text/html; charset=utf-8');
and also in the html header
<META http-equiv="content-type" content="text/html; charset=utf-8">
what else can I do to solve it ?
Thanks
The content of the page needs to also be in UTF-8. Your content was probably made using MS Word, which uses Windows 1251 encoding. You need to re-save your document as UTF-8.
UTF-8 does not convert formats for you.
If those strings are saved in a file, the file has to be encoded in UTF-8 too.
If you're getting them from a database, they'll have to be stored as UTF-8 and you'll have to set the connection charset to utf-8.
You could also check whether your text is UTF-8 and if not, convert it with utf8_encode.
I have a Zope/Plone WS that calls some functions written in Python.
That WS are called by PHP pages (utf-8 into header) but characters aren't visible.
I've tried to decode (where possible) special chars into entities (into Python) and that works, but not all chars have corresponding HTML entities.
I've tried to save the original Python file in UTF-8 format, but I thought that wasn't the right way.
Can someone help?
note : I pass through some php include, if this could be an hint...
Edit it's weird, because if I log all the "pieces" singly, then I have the right chars encoded. If I go up to the "main php page" (where I include all pieces), that messes up everything.
Obviously, the "main php page" has that:
<meta http-equiv="content-type" content="text/html;charset=utf-8" />
496e73e972657220646174652064926172726976e9652065742064652064e970617274
That string is encoded in ISO-8859-1, not UTF-8.
Somewhere you're converting your strings to ISO-8859-1, which means they're not interpreted correctly when trying to interpret them as UTF-8, and all non-European characters will be discarded since ISO-8859-1 can't encode anything but a handful of European characters.
I just edited the file site.py of python.
I follow that guide: click here and everything is ok now.
Thank you all for help.
I never had this problem before, it was usually my database or the html page. But now i think its my php. I import text from a csv or from a text area and in both ways it goes wrong.
for example é changes to é. I used htmlentities to fix this but it didn't work. The htmlentities function didn't return é in html but é in html entities, so it already loses the real characters before htmlentities comes in to place... So does that mean my php file has the wrong encoding or something?
I hope someone can help me out..
Thanks!
Chris
A file is usually ISO-8859-1 (Latin) or UTF-8 ... ISO-8859-1 is 1 byte per char, UTF-8 is 1-4 bytes per char. So if you get 2 chars when you expect one, then you are reading UTF-8 and showing it as ISO-8859-1 ... if you get strange chars, then you are reading ISO-8859-1 and showing it as UTF-8.
If you provide more details, it would be easier to pinpoint, but in short, you have inconsistent charsets and need to convert one or the other so they're all the same. But from what it seems, you're using ISO-8859-1 in your project, but you are reading some UTF-8 from somewhere... use utf8_decode($text) if that data should be indeed be stored as UTF-8, or find the data and convert it manually.
EDIT: If you are using AJAX somewhere, then you will ALWAYS get UTF-8 from it, and you'll have to decode it yourself with utf8_decode() if you want to keep using ISO-8859-1.
Try opening your php file and change the encoding to UTF-8
if that doesn't help, add this to your php:
header('Content-Type: text/html; charset=utf-8');
Or this to your html:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
Take a look at PHP's iconv().
I have a problem in Language Encoding in PHP as my php file should display both English and Arabic Characters.
Some web page parts are static and others are dynamic (data comes from a Sybase database) and the language encoding of database is ok as data is displayed well in it.
My web page has some drop down lists that are dynamic but they display the data in a strange format which is not English or Arabic like squares and unknown symbols.
I checked the possible causes and did many solutions like:-
Changing the encoding of the PHP script:
Saving File with the Name : WebPage1 of Type : PHP and Encoding : ANSI or UTF-8 or Unicode.
Changing the HTML encoding declaration:
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=windows-1256" />
Changing the PHP encoding declaration:
header('Content-Type: text/html; charset=UTF-8);
header('Content-Type: text/html; charset=windows-1256');
Changing the database tables font and language:
Arial(Arabic).
The problem still exists and I do not know what I can do to solve that.
Can you suggest any solution?
Check your database connection, make sure the sybase_connect connects with UTF-8 as charset.
See http://php.net/manual/en/function.sybase-connect.php
From the comment that you are using ODBC to connect: There seems to be an issue with PHP/ODBC and UTF8. Some suggestions are mentioned in this thread: Php/ODBC encoding problem
Always use UTF-8.
Your first header is correct. Your first header is correct, except you should use single = instead of ==. Make sure you used header() function before sending any output to browser.
Open your files in a Unicode supporting editor like Editplus, notepad++ and while saving every source code or HTML file, use Save as and choose UTF-8 on the save as screen. If you use eclipse, import your project to eclipse, right click it and go to project settings, apply charset setting as utf-8 to all source code.
If there's something wrong with data coming from MySQL database, then use appropriate collation on any text storing column (varchar, blob etc). Those are the usual suggestions for it. If you use Sybase, then use Google for collation settings.
And don't change your font to Arabic; Arial already supports it.
You seem to confuse something. Neither UTF-8 nor windows-1256 describe languages, they denote character sets/encodings. Although the character sets may contain characters that are typically used in certain languages, their use doesn’t say anything about the language.
Now as the characters of Windows-1256 are contained in Unicode’s character set and thus can be encoded with UTF-8, you should choose UTF-8 for both languages.
And if you want to declare the language for your contents, read the W3C’s tutorial on Declaring Language in XHTML and HTML.
In your case you could declare your primary document language as en (English) and parts of your document as ar (Arabic):
header('Content-Language: en');
header('Content-Type: text/html;charset=UTF-8');
echo '<p>The following is in Arabic: <span lang="ar">العربية</span></p>';
Make sure to use UTF-8 for both.
I'm using Kohana 3, which has full support for Unicode.
I have this as the first child of my <head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
The Unicode character I am inserting into is é as in Café.
However, I am getting the triangle with a ? (as in could not decode character).
As far as I can tell in my own code, I am not doing any string manipulation on the text.
In fact, I have placed the accent straight into a view's PHP file and it is still not working.
I copied the character from this page: http://www.fileformat.info/info/unicode/char/00e9/index.htm
I've only just started examining PHP's Unicode limitations, so I could be doing something horribly wrong.
So, how do I display this character? Do I need to resort to the HTML entity?
Update
So this works
Caf<?php echo html_entity_decode('é', ENT_NOQUOTES, 'UTF-8'); ?>
Why does that work? If I copy the output accented e from that script and insert it into my document, it doesn't work.
View the http headers. You should see something like
Content-Type: text/html; charset=UTF-8
Browsers don't pay much attention to meta tags, if there was a real http header stating a different encoding.
update
Whatcha get from this?
echo bin2hex('é');
echo chr(0xc3) . chr(0xa9);
You should get c3a9é, otherwise I'd say file encoding issue.
I guess, you see �, the replacement character for invalid UTF-8 byte sequences. Your text is not UTF-8 encoded. Check your editor’s settings to control the encoding of the PHP file.
If you’re not sure about the encoding of your sources, you can enforce UTF-8 compatibilty as described here (German text): Force UTF-8.
You should never need entities except the basic ones.