I have a webform and an input if you put in Latté and POST it using JSON...
$.ajax({
type: "POST",
url: "http://"+document.domain+"/includes/rpc.php",
data: {method:"add_item",item:item},
dataType: "json",
timeout: 10000,
success:......
item will be the value Latté Latté is posted and the responding JSON is Latt\u00e9 which the browser interprets as Latté. Effectively this script is a WYSIWYG editor so what you type in you get back. Anyhue if I refresh the text is pulled out of mysql and comes out as Latté?. So I am guessing that MYSQL is not the correct collation?
Some more information - the query to edit the DB is
UPDATE menu_items SET description = 'Latté' WHERE item_id = '742'
the JSON reply is
{"description":"Latt\u00e9","id":"#recordsArray_742"}
To be precise, the collation is the way in which the strings are compared and sorted. It has a relationship with the character set, but your problem is an character set problem, not a collation problem.
A character set is a set of symbols and encodings. A collation is a set of rules for comparing characters in a character set.
The first thing that you've to know is which character set you're using. Are you using UTF-8 or LATIN1 or others?
After that I'd try to output the correct header from the PHP script generating the JSON string. For example for UTF-8:
header('Content-type: application/json; charset=utf-8');
This could already solve your problem, if it doesn't we have to look deeper on how do you connect to the DB and how you manipulate your data. Let me know in case which MySQL libraries or framework are you using to connect to the db, and post the relevant source code.
It could be MySQL's collation.
It could also be HTTP encoding.
Or you could be using string functions in PHP which are not multi-byte-safe.
This kind of error can happen at many points through your tool chain.
Either the database collation is wrong or the DBAL you are using (PDO?).
Use utf8_encode() around your fetched values in your PHP backend before you output it
Related
I have a data encoding problem. My database has accents in one of the columns, in the api return that column in a PDO query SQL SERVER in php. As soon as I return I transform into JSON by the json_encode method, plus the JSON comes NULL. When I give var_dump the question letters with accents this appears '�' and in json empty.
I know it's the encoding I need to convert to UTF8 but I'm not able to do this conversion in php. Can anyone help me?
Are you specifying the header for the right charset?
header('Content-type: text/html; charset=utf-8');
Notice also that your columns and tables should be utf8_unicode_ci.
And finally your connection to database should also be set accordingly charset=utf8.
I have come across some problems when inputting certain characters into my mysql database using php. What I am doing is submitting user inputted text to a database. I cannot figure out what I need to change to allow any kind of character to be put into the database and printed back out through php as it's suppose to.
My MySQL collation is: latin1_swedish_ci
Just before I send the text to the database from my form I use mysql_real_escape_string() on the data.
Example below
this text:
�People are just as happy as they make up their minds to be.�
� Abraham Lincoln
is suppose to look like this:
“People are just as happy as they make up their minds to be.”
― Abraham Lincoln
As mentioned by others, you need to convert to UTF8 from end to end if you want to support "special" characters. This means your web page, PHP, mysql connection and mysql table. The web page is fairly simple, just use the meta tag for UTF8. Ideally your headers would say UTF8 also.
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
Set your PHP to use UTF8. Things would probably work anyway, but it's a good measure to do this:
mb_internal_encoding('UTF-8');
mb_http_output('UTF-8');
mb_http_input('UTF-8');
For mysql, you want to convert your table to UTF8, no need to export/import.
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8
You can, and should, configure mysql to default utf8. But you can also run the query:
SET NAMES UTF8
as the first query after establishing a connection and that will "convert" your database connection to UTF8.
That should solve all your character display problems.
The likeliest cause of the problem is that the database connection is set to latin1 but you are feeding it text encoded in UTF-8. The simplest way to solve this is to convert your input into what the client expects:
$quote = iconv("UTF-8", "WINDOWS-1252//TRANSLIT", $quote);
(What MySQL calls latin1 is windows-1252 in the rest of the world.) Note that many characters, such as the quotation dash U+2015 that you use there, cannot be represented in this encoding and will be converted into something else. Ideally you should change the column encoding to utf8.
An alternative solution: set the database connection to utf8. It doesn't matter how the columns are encoded: MySQL internally converts text from the connection encoding into the storage encoding, you can keep the columns as latin1 if you want to. (If you do, the quotation dash U+2015 will be turned into a question mark ? because it's not in latin1)
How to set the connection encoding depends on what library you are using: if you use the deprecated MySQL library it's mysql_set_charset, if MySQLi it's mysqli_set_charset, if PDO add encoding=utf8 to the DSN.
If you do this you'll have set the page encoding to UTF-8 with the Content-Type header.
Otherwise you would be having the same problem with the browser: feeding it text encoded in UTF-8 when it's expecting something else:
header("Content-Type: text/html; charset=utf-8");
The solutions provided are helpful if starting from scratch. Putting all possible connections to UTF-8 is indeed the safest. UTF-8 is the most used charset on the net for a variety of reasons.
Some suggestions and a word of warning:
copy the tables you want to sanitize with a unique prefix (tmp_)
although your db-connection is forced to utf8, check you General Settings collation, change to utf8_bin if that was not done yet
you need to run this on the local server
the funny char error is mostly due to mixing LATIN1 with UTF-8 configurations. This solution is designed for this. It could work with other used char-sets that LATIN1 but I haven't checked this
check these tmp_tables extensively before copying back to the original
Builds the 2 array needed for the magic:
$chars = get_html_translation_table(HTML_ENTITIES, ENT_QUOTES, "UTF-8");
$LATIN1 = $UTF8 = array();
while (list($key,$val) = each ($chars)) {
$UTF8[] = $key;
$LATIN1[] = $val;
}
Now build up the routines you need: (tables->)rows->fields and at each field call
$row[$field] = mysql_real_escape_string(str_replace($LATIN1 , $UTF8 , $row[$field]));
$q[] = "$field = '{$row[$field]}'";
Finally build up and send the query:
mysql_query("UPDATE $table SET " . implode(" , " , $q) . " WHERE id = '{$row['id']}' LIMIT 1");
change the MySQL collation to utf8_unicode_ci or utf8_general_ci, including the table and the database.
You will need to set your database in utf-8 yes. There is many ways to do it. By changin the config file, via phpmyadmin or by calling php function (sorry memory blank) right before insert and update the mysql.
Unfortunately, i think you will have to re-enter any data you entered before.
One thing you also need to know, from personnal experience, make sure all table with relation have the same collation or you won'T be able to JOIN them.
as reference: http://dev.mysql.com/doc/refman/5.6/en/charset-syntax.html
Also, i can be a apache setting. We've experienced the same issue on 'free-hosting' server as well as on my brother's server. Once switched to another server, all the charater's became neat. Verfiy you apache setting, sorry but i can't bting more light on apache's config.
Get rid of everything you just need to follow these two points, every problem regarding special languages characters will be resolved.
1- You need to define the collation of your table to be utf8_general_ci.
2- define <meta http-equiv="content-type" content="text/html; charset=utf-8"> in the HTML after head tag.
2- You need to define the mysql_set_charset('utf8',$link_identifier); in the file where you made connection with the database and right after the selection of database like 'mysql_select_db' use this 'mysql_set_charset' this will allow you to add and retrieve data properly in what ever the language it is.
If your text has been encoded and decoded with the wrong encoding and so the mojibake is actually "solidified" into unicode characters, then the solutions mentioned so far won't work. I ended up having success with the ftfy Python package to automatically detect/fix mojibake:
https://github.com/LuminosoInsight/python-ftfy
https://pypi.org/project/ftfy/
https://ftfy.readthedocs.io/en/latest/
>>> import ftfy
>>> print(ftfy.fix_encoding("(ง'⌣')ง"))
(ง'⌣')ง
Hopefully this helps people who are in a similar situation.
I have a MySQL DB, encoded in UTF8, where some records have 'ā's in them (in case it doesn't show up right in SO, that's an 'a' with a line above it).
There is a PHP script that is getting the records, putting them in an array, and json_encoding them. No matter whether the script is being invoked by ajax or the webpage, the 'ā's show up as question marks. Where is the problem, and how do I fix it?
Thanks,
Jamie McClymont
EDIT: Forgot to mention that the 'ā's show up fine in PHPMyAdmin
For the text to print correctly you need to set the charset of the mysql connection and the page
For the connection the following query will work
set names utf8
Run this query right after connecting
If the charset is still incorrect try adding
header('Content-Type: application/json; charset=utf-8');
assuming you're outputting json
http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html
I have been struggling with this for three days now and this is what i have got and i cannot understand why i am seeing this behavior.
my problem is that i have a MySql spanish db with char set and collation defined as utf8_general_ci. when i query the data base in delete.php like this "DELETE FROM countryNames WHERE country = '$name'"
the specified row doesnot get deleted. i am setting the variable $name in delete.php through a post variable $name=$_post['data'] . mostly $name gets the value in spanish characters e.g español, México etc. the delete.php file gets called from main.php.if i send a post message from main.php $.post("delete.php", {data:filename}); , the query doesnot deletes the entry (although the 'filename' string is in utf8) but if i create a form and then post my data variable in main.php, the query works!! the big question to me is why do i have to submit a form for the query to work? what im seeing is my database rejects the value if it comes from a jquery post call but accepts it when its from a submitted form. (i make no code change for the query to work. just post the value by submiting the form)
First of all, to see what charset ìs used for requests, install something like Firebug and check the 'Content-Type' header of your request/response. It will look something like 'application/json; charset=...'. This should be charset=utf-8 in your case.
My guess why it worked when posting a form is probably because of x-www-form-urlencoded - non-alphanumeric characters are additionally encoded on the client side and again decoded on the server, that's the difference to posting the data directly.
This means that somewhere there is a wrong encoding at work. PHP treats your strings agnostic to its encoding by default, so I would tend to rule it out as the source of the error. jQuery.post also uses UTF-8 by default... so my suspect is the filename variable. Are you sure it is in UTF-8? Where and how do you retrieve it?
You should probably also ensure that the actual HTML page is also sent as UTF-8 and not, let's say iso-8859-1. Have a look at this article for a thorough explanation on how to get it right.
guys this was a Mac problem!! i just tested it on windows as my server and now everything works fine. So beware when u r using Mac as a server with MySql having UTF8 as charset and collation. I guess the Mac stores the folder and file name in some different encoding and not UTF-8.
You answer might be here: How to set encoding in .getJSON JQuery
As it says there, use $.ajax instead of $.post and you can set encoding.
OR, as it says in the 2nd answer use $.ajaxSetup to set the encoding accordingly.
Use .serialize() ! I think it will work. More info: http://api.jquery.com/serialize/
I'm having encoding problems in my webpage, and it is driving me crazy. Let me try to explain
I have a meta tag defining utf8 as charset.
I'm including the scripts as utf8 too (<script type="text/javascript src="..." charset="utf8"></script>).
In .php files, I declare header('Content-Type: text/html; charset=utf8');
In my database (postgreSQL), I've made the query show lc_collate; and the return was en_US.UTF-8
I'm using AJAX
When I try to save the field value "name" as "áéíóú", I get the value "áéÃóú" in the record set (using phpPgAdmin to view results).
What am I doing wrong? There's a way to fix it without using decode/encode? Someone have a good reference about theses issues?
Thank you all!
Maybe the client encoding is not set correctly? PostgreSQL automatically converts between the character encoding on the client and the encoding in the database. For this to work it needs to know what encoding the client is using. Safest is to set this when you open your connection using:
SET CLIENT_ENCODING TO 'UTF8';
For details see the docs
You might be storing the data as ISO-8859-1?
Try enconding to base64 and decoding on the other end.