I have an unusual problem (this is linked to Browser displays � instead of ´)
I had mismatched character encoding settings on my server (UTF-8) and application (ISO-8859-1), so a third person tasked with entering Spanish translations, entered the words properly at his end, but they weren't saved correctly in the database.
I have subsequently fixed the problem and the server is now ISO-8859-1 as well. [I set
default_charset = "iso-8859-1"
in php.ini]
I do see a pattern in what is in the system, for example the following appears on the system:
Nombre de la organización*
This needs to be:
Nombre de la organización*
ie, I need to search and replace 'ó' with 'ó'.
How can I do so for an entire table (all fields)? (there will be other such corrections as well)
Use the replace function. Simple example:
SELECT REPLACE('www.mysql.com', 'w', 'Ww');
Result: 'WwWwWw.mysql.com'
Now, if you have a table called Foo and you want to replace those characters in a field called bar, you can do the following:
update Foo set bar = Replace(bar, 'ó', 'ó');
Do this for all the affected fields and the problem is solved.
Best regards,
Lajos Arpad.
Related
Plugin Used : http://www.tinybutstrong.com/plugins.php
Russian characters are not displaying correctly.
In mysql database they are stored correctly collation is utf8_general_ci.
I used define('OPENTBS_ALREADY_UTF8','already_utf8');
It looks like an UTF-8 problem.
You have to check that all the data chain is UTF-8 :
all your PHP scripts
the data injected in the template (usually a database), but you also have to check that your PHP script retrieve the data as UTF8. See « How do I make MySQL return UTF-8? »
the template (it is actually UTF8 since it is a LibreOffice or Ms Office template)
Since this chain is ok, you have to use the OPENTBS_ALREADY_UTF8 option to load the template.
$TBS->LoadTemplate('my_template.odt', OPENTBS_ALREADY_UTF8);
You can check that you chain is ok by a test like this :
echo "<!doctype html><html><head><title>Test</title><meta charset='UTF-8'></head><body>";
echo $my_data_from_database;
echo "</body></html>";
exit;
where $my_data_from_database is a data item retrieve from the database and that contains special characters like Russian characters.
I have a MySQL table that contains names and e-mail addresses. The data was originally imported from a .csv file, and it did not originally contain complete e-mail addresses. We had to append the #place.domain to the user's alias.
When the data is sitting in the MySQL table, it looks normal: person#place.domain; however, when I output the content in PHP, I get this: person #place.domain. There's always a space between the person and the #. It doesn't look like that in the MySQL column, nor does it look like that when I copy/paste the data into Notepad, Word, Excel, etc. Furthermore, if I erase the data in the column and manually replace it with person#place.domain, it displays normally in my PHP app. So I'm guessing there's some hidden character that PHP is picking up that I can't detect. Is there a way for me to clean this up? I've tried TRIM(), REPLACE,(), etc., all to no avail.
UPDATE: I've discovered that, when I click in the MySQL field and move among the characters using my arrow keys, I have to hit the arrow key TWICE to move past the # symbol, yet there is no visible space.
I made this sample code for you:
<?php
$test = "user #mail.com";
$aux = explode("#",$test);
$mailok = trim($aux[0])."#".trim($aux[1]);
echo $test." vs ".$mailok;
?>
This is likely something like non-breaking space (ascii 160). To get rid of it:
UPDATE my_table SET e_mail = REPLACE(e_mail, CHAR(160), '');
Try with a foreach cycle, and do the chr($char) function to every character in the string, it will display you the ascii code of each character in the string, and you will find the wrong character. It's the only solution I found. Hope to be useful
I have an array called $data and I am sending this data to my db insert method.
This works nice when used with:
'body' => $_POST['body']
However, when I try:
'body' => htmlentities($_POST['body'])
Only a first 3 words are saved but when it comes to character ě it stops there and other parts of the text is not saved.
in my db body is stored in body column which is a standard text type, utf_8_general_ci.
It has nothing to do with db settings I guess, because all tables is set to uf8_general_ci as well as the table itself (the same as on my localhost).
The funniest part is that on my localhost machine running wamp it wokrs with al these weird characters and the text is saved correctly when sing htmlentities.
Only on the server it didn't work. Btw. it's justhost if it helps.
Do you know how to run it the same way as on my localhost or any other function that can be used except htmlentities ?
I need htmlentities because I have html code like google maps and it needs to be saved. Without htmlentities it is saved incorrectly.
Try using mysql_real_escape_string or mysqli_real_escape_string or prepared statements when inserting into a DB.
Try and set the htmlentities encoding to match your form data encoding.
Note the dividing PHP version 5.4.0: default encoding for htmlentities can be UTF-8 or ISO-8859-1. Does your local machine have the same PHP version as server?
Note to all local machine users (MAMP, WAMP, LAMP, whatever): always check your PHP/MySQL versions if they match the server version. There is almost always some common function that doesn't work as expected when the testing environment isn't identical to production environment (been there, done that – once or twice...).
You can use set utf8 to mysql before insertion. It can insert special character in the database. FYI http://php.net/manual/en/function.mysql-set-charset.php
I have come across some problems when inputting certain characters into my mysql database using php. What I am doing is submitting user inputted text to a database. I cannot figure out what I need to change to allow any kind of character to be put into the database and printed back out through php as it's suppose to.
My MySQL collation is: latin1_swedish_ci
Just before I send the text to the database from my form I use mysql_real_escape_string() on the data.
Example below
this text:
�People are just as happy as they make up their minds to be.�
� Abraham Lincoln
is suppose to look like this:
“People are just as happy as they make up their minds to be.”
― Abraham Lincoln
As mentioned by others, you need to convert to UTF8 from end to end if you want to support "special" characters. This means your web page, PHP, mysql connection and mysql table. The web page is fairly simple, just use the meta tag for UTF8. Ideally your headers would say UTF8 also.
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
Set your PHP to use UTF8. Things would probably work anyway, but it's a good measure to do this:
mb_internal_encoding('UTF-8');
mb_http_output('UTF-8');
mb_http_input('UTF-8');
For mysql, you want to convert your table to UTF8, no need to export/import.
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8
You can, and should, configure mysql to default utf8. But you can also run the query:
SET NAMES UTF8
as the first query after establishing a connection and that will "convert" your database connection to UTF8.
That should solve all your character display problems.
The likeliest cause of the problem is that the database connection is set to latin1 but you are feeding it text encoded in UTF-8. The simplest way to solve this is to convert your input into what the client expects:
$quote = iconv("UTF-8", "WINDOWS-1252//TRANSLIT", $quote);
(What MySQL calls latin1 is windows-1252 in the rest of the world.) Note that many characters, such as the quotation dash U+2015 that you use there, cannot be represented in this encoding and will be converted into something else. Ideally you should change the column encoding to utf8.
An alternative solution: set the database connection to utf8. It doesn't matter how the columns are encoded: MySQL internally converts text from the connection encoding into the storage encoding, you can keep the columns as latin1 if you want to. (If you do, the quotation dash U+2015 will be turned into a question mark ? because it's not in latin1)
How to set the connection encoding depends on what library you are using: if you use the deprecated MySQL library it's mysql_set_charset, if MySQLi it's mysqli_set_charset, if PDO add encoding=utf8 to the DSN.
If you do this you'll have set the page encoding to UTF-8 with the Content-Type header.
Otherwise you would be having the same problem with the browser: feeding it text encoded in UTF-8 when it's expecting something else:
header("Content-Type: text/html; charset=utf-8");
The solutions provided are helpful if starting from scratch. Putting all possible connections to UTF-8 is indeed the safest. UTF-8 is the most used charset on the net for a variety of reasons.
Some suggestions and a word of warning:
copy the tables you want to sanitize with a unique prefix (tmp_)
although your db-connection is forced to utf8, check you General Settings collation, change to utf8_bin if that was not done yet
you need to run this on the local server
the funny char error is mostly due to mixing LATIN1 with UTF-8 configurations. This solution is designed for this. It could work with other used char-sets that LATIN1 but I haven't checked this
check these tmp_tables extensively before copying back to the original
Builds the 2 array needed for the magic:
$chars = get_html_translation_table(HTML_ENTITIES, ENT_QUOTES, "UTF-8");
$LATIN1 = $UTF8 = array();
while (list($key,$val) = each ($chars)) {
$UTF8[] = $key;
$LATIN1[] = $val;
}
Now build up the routines you need: (tables->)rows->fields and at each field call
$row[$field] = mysql_real_escape_string(str_replace($LATIN1 , $UTF8 , $row[$field]));
$q[] = "$field = '{$row[$field]}'";
Finally build up and send the query:
mysql_query("UPDATE $table SET " . implode(" , " , $q) . " WHERE id = '{$row['id']}' LIMIT 1");
change the MySQL collation to utf8_unicode_ci or utf8_general_ci, including the table and the database.
You will need to set your database in utf-8 yes. There is many ways to do it. By changin the config file, via phpmyadmin or by calling php function (sorry memory blank) right before insert and update the mysql.
Unfortunately, i think you will have to re-enter any data you entered before.
One thing you also need to know, from personnal experience, make sure all table with relation have the same collation or you won'T be able to JOIN them.
as reference: http://dev.mysql.com/doc/refman/5.6/en/charset-syntax.html
Also, i can be a apache setting. We've experienced the same issue on 'free-hosting' server as well as on my brother's server. Once switched to another server, all the charater's became neat. Verfiy you apache setting, sorry but i can't bting more light on apache's config.
Get rid of everything you just need to follow these two points, every problem regarding special languages characters will be resolved.
1- You need to define the collation of your table to be utf8_general_ci.
2- define <meta http-equiv="content-type" content="text/html; charset=utf-8"> in the HTML after head tag.
2- You need to define the mysql_set_charset('utf8',$link_identifier); in the file where you made connection with the database and right after the selection of database like 'mysql_select_db' use this 'mysql_set_charset' this will allow you to add and retrieve data properly in what ever the language it is.
If your text has been encoded and decoded with the wrong encoding and so the mojibake is actually "solidified" into unicode characters, then the solutions mentioned so far won't work. I ended up having success with the ftfy Python package to automatically detect/fix mojibake:
https://github.com/LuminosoInsight/python-ftfy
https://pypi.org/project/ftfy/
https://ftfy.readthedocs.io/en/latest/
>>> import ftfy
>>> print(ftfy.fix_encoding("(ง'⌣')ง"))
(ง'⌣')ง
Hopefully this helps people who are in a similar situation.
I am facing an small issue in my project When I am trying to store some German words into the MYSQL Database. When this German words contains umlauts i.e. characters ä, ö, ß, ü etc., they are not stored as they are.....?
I want to store them as it is into the Database.To do so I tried to change the COLLATION to UTF8-general-ci, and others in the list using PHP myAdmin. But none of them is working for me.
Am I in the right way or I have to do something else.
Please suggest some help.
Thanks In Advance......
You have to choose the right transfer encoding either. Call
SET NAMES utf8
before inserting the data and make sure that the german words are utf8-encoded before inserting.
Try to use utf8_encode($string) to encode your text into UTF8 first, before saving it into the database. In order for characters to display correctly in a certain language, you have to (1) set the text into the right charset and then also (2) set a database to the right charset (as you did).
Also, for example, file display.php will output the German text, you can open the file in any editors (EmEditor?) and then "save as", choose a right encoding scheme. After that, the display file, when outputting the text, will take care of the charset.
years ago I've faced the same problem. I've solved it by implicit setting NAMES option for mysql. In my code it looks like this:
//inside AbstractMapper class
public function __construct($modelClass, $dbTable) {
$this->setDbTable($dbTable);
$stmt = new Zend_Db_Statement_Pdo($this->getDbTable()->getAdapter(), 'set names utf8');
$stmt->execute();
$this->_model_class = $modelClass;
}
After connecting to the database, use the following codes:
SET NAMES XXX
replace XXX with your working charset.