I have a PHP application using Gettext as the i18n engine. The translation works fine, the only problem is that I'm having encoding issues with UTF8 characters. My PHP code to load gettext is something like this:
bindtextdomain( $domain, PATH_BASE . DS . "language" . DS );
$this->utf8Encode = strtolower($encoding) == "utf-8";
bind_textdomain_codeset($domain, $encoding);
textdomain($domain);
My templates render the pages using the utf8 charset and I've tried just about anything to load the proper charset. For the current locale I'm loading SL_sl, the names appear correctly but have issues with UTF8 chars, so where it should appear Država, it shows up Dr?ava
So, it has happened before, and now it happened again, I found the solution myself! The problem was that like I said to #bozdoz, I was converting UTF8 text already, but I didn't realized that the gettext function returned a UTF8 string, so if you do this:
$encoded = utf8_encode($utf8String);
Then you'll have a really nasty bug when $utf8String is an actual UTF8 string. Therefore I did some modifications to my code and the translation method (simplified) ended up like this:
$translation = gettext($singular);
$encoded = $this->utf8Encode ? $this->Utf8Encode($translation) : $translation;
And the Utf8Encode method is like this:
private function Utf8Encode( $text )
{
if ( mb_check_encoding($text, "utf8") == TRUE ){
return $text;
return utf8_encode($text);
}
I hope that if somebody has the same error this can help!
From the partial information I can suggest you take a look at the actual mo/po files, in poedit there are several warnings about utf8 encoding. Assuming that everything else is correct (meta, headers, etc) it's the only thing left to check
Try encoding it with utf8_encode(). I can't really tell from your code, but perhaps it could be implemented like this:
utf8_encode($domain);
Related
I'm using a simple form to write string to database, but the curly quotes (on retrieval) are being displayed as special characters.
So far I have tried changing the table collation to UTF-8 as well as using the CodeIgniter Typography class:
1. alter table test.vocab convert to character set utf8 collate
utf8_unicode_ci;
2. $this->load->library('typography');
$data['item'] = $this->typography->auto_typography($data['item'], FALSE);
This hasn't helped, though. Here's what the output looks like:
If needed I'll post more code. Please help.
Thank you everyone, but it seems like what I needed was the mb_convert_encoding() function:
$i['item'] = mb_convert_encoding($i['item'], 'HTML-ENTITIES', 'UTF-8');
Many thanks for taking out the time for commenting or answering.
I had a similar issue with a migrated DB. Turned out I needed to convert 'UTF-8' to 'Windows-1252'.
I found out by running the code below and checking which combination looked right:
$html = 'Your HTML here';
$charsets = array(
"UTF-8",
"ASCII",
"Windows-1252",
"ISO-8859-15",
"ISO-8859-1",
"ISO-8859-6",
"CP1256"
);
foreach ($charsets as $ch1) {
foreach ($charsets as $ch2){
echo "<h1>Combination $ch1 to $ch2 produces: </h1>".iconv($ch1, $ch2, $html);
}
}
Now I, in my own opionion, have tried everything there is on this encoding problem, looked through a lot of answered quistions but nothing worked for me, so here I go.
I have a MySQL database with a Users table. This table has a column for "firstname" which collation is set to utf8_general_ci (all varchar columns is). I have then inserted a row where the firstname-column is set to "Løw", with the scandinavian special character "ø".
I now use the php-ActiveRecord library, where the connection string is to ";charset=utf8", to retrieve the row and afterwards outputs the user as json, like so:
$user = User::find($ID);
$userArr = $user->to_array();
header('Content-Type: application/json; charset=utf-8');
print(json_encode($userArr));
Now the wired things starts. The firstname is now NOT "Løw" as displayed in the MySQL Database , but "L\u00f8w". I then tried to see if this was also the case without the json_encode function, like so:
$user = User::find($ID);
$userArr = $user->to_array();
header('Content-Type: text/plain; charset=utf-8');
print_r($userArr);
But here the output was correct, firstname was "Løw". I then tried to encode the fields in the array to utf-8, since everybody told me if the strings was utf-8 it should work, like so:
$return[] = array_map('utf8_encode', $userArr);
print_r(json_encode($return));
But this gave me "L\u00c3\u00b8w", so that didn't work. I then tried, since i was out of ideas to utf8_decode it:
$return[] = array_map('utf8_decode', $userArr);
print_r(json_encode($return));
But that made the string return as "null". I then tried to check what encoding my vars was when they came out of the database, like so:
header('Content-Type: text/plain; charset=utf-8');
print(mb_detect_encoding($userArr['firstname']));
But this returned UTF-8.
So as you, hopefully, can see, i have tried everything and i still don't know why my json_encode, changes the "ø" charcter to "\u00f8". Please help, i don't want to make my own json_encode-method.
Ok found an answer pretty quick, but ill let other scandinavian people know, since i coulden't find anything on the subject.
I solved the problem by adding the following to the json_encode method:
print(json_encode($userArr,JSON_UNESCAPED_UNICODE));
This tells the method NOT to escape unicode chars (i think) or as it says in the PHP doc:
JSON_UNESCAPED_UNICODE (integer)
Encode multibyte Unicode characters literally (default is to escape as
\uXXXX). Available since PHP 5.4.0.
I'm building a PHP web application, and it works in UTF-8. The database is UTF-8, the pages are served as UTF-8 and I set the charset using a meta tag to UTF-8. Of course, with users using Internet Explorer, and copying & pasting from Microsoft Office, I somehow manage to get not UTF-8 input occasionally.
The ideal solution would be to throw an HTTP 400 Bad Request error, but obviously I can't do that. The next best thing is converting $_GET, $_POST and $_REQUEST to UTF-8. Is there anyway to see what character encoding the input is in so I can pass it off to iconv? If not, what's the best solution for doing this?
Check out mb_detect_encoding() Example:
$utf8 = iconv(mb_detect_encoding($input), 'UTF-8', $input);
There's also utf8_encode() if you guarantee that the string is input as ISO-8859-1.
In some cases using just utf8_encode or general checks are ok but you might lose some characters within the string. If you can build out a basic array/string list based on various types, this example being windows, you can salvage quite a bit more.
if(!mb_detect_encoding($fileContents, "UTF-8", true)){
$checkArr = array("windows-1252", "windows-1251");
$encodeString = '';
foreach($checkArr as $encode){
if(mb_check_encoding($fileContents, $encode)){
$encodeString .= $encode.",";
}
}
$encodeString = substr($encodeString, 0, -1);
$fileContents = mb_convert_encoding($fileContents, "UTF-8", $encodeString);
}
I started a website some time ago using the wrong CHARSET in my DB and site. The HTML was set to ISO... and the DB to Latin... , the page was saved in Western latin... a big mess.
The site is in French, so I created a function that replaced all accents like "é" to "é". Which solved the issue temporarily.
I just learned a lot more about programming, and now my files are saved as Unicode UTF-8, the HTML is in UTF-8 and my MySQL table columns are set to ut8_encoding...
I tried to move back the accents to "é" instead of the "é", but I get the usual charset issues with the (?) or weird characters "â" both in MySQL and when the page is displayed.
I need to find a way to update my sql, through a function that cleans the strings so that it can finally go back to normal. At the moment my function looks like this but doesn't work:
function stripAcc3($value){
$ent = array(
'à'=>'à',
'â'=>'â',
'ù'=>'ù',
'û'=>'û',
'é'=>'é',
'è'=>'è',
'ê'=>'ê',
'ç'=>'ç',
'Ç'=>'Ç',
"î"=>'î',
"Ï"=>'ï',
"ö"=>'ö',
"ô"=>'ô',
"ë"=>'ë',
"ü"=>'ü',
"Ä"=>'ä',
"€"=>'€',
"′"=> "'",
"é"=> "é"
);
return strtr($value, $ent);
}
Any help welcome. Thanks in advance. If you need code, please tell me which part.
UPDATE
If you want the bounty points, I need detailed instructions on how to do it. Thanks.
Try using the following function instead, it should handle all the issues you described:
function makeStringUTF8($data)
{
if (is_string($data) === true)
{
// has html entities?
if (strpos($data, '&') !== false)
{
// if so, revert back to normal
$data = html_entity_decode($data, ENT_QUOTES, 'UTF-8');
}
// make sure it's UTF-8
if (function_exists('iconv') === true)
{
return #iconv('UTF-8', 'UTF-8//IGNORE', $data);
}
else if (function_exists('mb_convert_encoding') === true)
{
return mb_convert_encoding($data, 'UTF-8', 'UTF-8');
}
return utf8_encode(utf8_decode($data));
}
else if (is_array($data) === true)
{
$result = array();
foreach ($data as $key => $value)
{
$result[makeStringUTF8($key)] = makeStringUTF8($value);
}
return $result;
}
return $data;
}
Regarding the specific instructions of how to use this, I suggest the following:
export your old latin database (I hope you still have it) contents as an SQL/CSV dump *
use the above function on the file contents and save the result on another file
import the file you generated in the previous step into the UTF-8 aware schema / database
* Example:
file_put_contents('utf8.sql', makeStringUTF8(file_get_contents('latin.sql')));
This should do it, if it doesn't let me know.
You might want to investigate what is used to fix WP database encoding issues:
http://codex.wordpress.org/Converting_Database_Character_Sets
To cut a long story short, most old WP sites were created with Swedish/Latin1 collated tables, which were used to store UTF8 strings. To collate the tables properly, the approach is to change the column to binary type, and then to change it to UTF8 text.
This avoids that the text gets wrangled when converting from Latin1 to UTF8 directly.
You will need to convert the offending rows using for example iconv. The challenge for you will be to know what rows are already UTF-8 and which are latin-1.
I'm not completely sure I understand your question, but
if you have
a UTF-8 database
all special characters in there stored as HTML entities
then a
html_entity_decode($string, ENT_QUOTES, "UTF-8");
should do the trick and turn all entities back into their UTF-8 native characters.
Make sure, not just your tables use utf-8, your database connection should use utf-8 as well.
$this->db = mysql_connect(MYSQL_SERVER,DB_LOGIN,DB_PASS);
mysql_set_charset ('utf8',$this->getConnection());
If you want to discuss with your database in UTF-8 you have to tell the Database that the connexion flow is a UTF-8 flow. You have to sent a request before each request you make to the database, this request in the following :
"SET NAMES utf8";
Personnaly I use that in the connect.inc.php files which create the connection to the database. Which this statement the database know that your sending UTF-8 encoded string and works perfectly !
mysql_set_charset function isn't working well, i tried this function in the past but the truth is that it don't do the trick.
For your complete issue, if you want to convert latin1 string to UTF-8, you have to convert first the latin1 string to a binary string format. Then convert the binary string into UTF-8 string, all can be done inside the database with database commands. See that artile (in french) : http://www.noidea.ca/2009/06/15/comment-convertir-une-db-de-latin1-a-utf8/
I can tell you that this method works because i used it to transform data from a database I've created.
I'm using Amazon's API to obtain the description of books. The API returns XML responses and the description is marked up (with HTML) very poorly. To deal with this poorly marked up description, which oftentimes breaks the layout of my site, I'm trying to use HTML Tidy to "clean it up."
In order to prevent "weird" characters from being displayed on my web page, I think I need to tell Tidy what the input encoding is and what the desired output encoding is. I know I want the output to be UTF8. However, I'm not sure how to determine the encoding of the input (Amazon's book description).
I've tried something like this:
mb_detect_encoding($amazon_description);
It's helped, but I'm still occasionally getting weird characters (a black diamond with a question mark in it: �). My guess is that I'm not detecting the encoding properly.
Any suggestions what I need to do?
EDIT:
This is my current solution:
$sanitized_amazon_markup = preg_replace('/[^\w`~!##$%^&*()-=_+[\]{}|;\':",.\/<>? ]/', '', $sanitized_amazon_markup);
I'm not sure about this as this may delete stuff that I should be keeping.
Can you provide your tidy repairString call?
If you tried to use input-encoding and output-encoding from tidy options, try to not use these and use the third argument or repairString instead, something like this :
$oTidy = new tidy();
$page_content = $oTidy->repairString($page_content,
array("show-errors" => 0, "show-warnings" => false),
"utf8"
);
Edit :
After doing some tests, what I said before cannot work if you don't have utf8 encoding in $page_content already before calling repairString
But you will mostly end up with ISO-8859-1 (latin1) encoding if not UTF-8 already.
May I suggest you try :
$charset = mb_detect_encoding($amazon_description, 'UTF-8, ISO-8859-1');
if ($charset == "ISO-8859-1") {
$amazon_description = utf8_encode($amazon_description);
}
$oTidy = new tidy();
$amazon_description = $oTidy->repairString($amazon_description,
array("show-errors" => 0, "show-warnings" => false),
"utf8"
);