I am parsing a XML and putting it into MySQL. Now MySQL is showing following error for some specific records inserting:
SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xEAm-Kh\xEA...'
When i check the data inside XML, it is showing like iêm-Khê
So the Special Character ê is transforming into xEA
What is it?
How can i get it solved to maintain & put its original Character into Database?
FYI, here is some info.:
My whole MySQL DB itself is using latin1_swedish_ci
But that TABLE & COLUMN are using utf8_general_ci
** When i manually Copy/Paste that iêm-Khê values into the MySQL Record directly, it is saved. (I mean, MySQL is accepting that Special Characters.) So i think the issue is at the Coding side.
After some additional research I ran across two possible approaches to a solution:
- the first option is to use php's utf8_encode() function.
- the second option uses some code with the PDO using set names (though there seemed to be some discussion as to whether or not this worked for everyone.
$handle = new PDO("mysql:host=localhost;dbname=dbname",
'username', 'password',
array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"));
The following link had some other useful information:
See here
You should define the UTF-8 charset to the mysql DB, by default it used to be latin charset , we can define the utf-8 along with table definition or on database creation,
To Table
create table `my_table` (.....
.... ) DEFAULT CHARSET=utf8;
To database
CREATE DATABASE mydb DEFAULT CHARACTER SET utf8
correct me if 'm worng
You need to set the character set of the database to UTF-8.
Command:
ALTER DATABASE db_name CHARACTER SET utf8 COLLATE utf8_general_ci;
For alter the column:
ALTER TABLE [table_name] MODIFY
[column_name] [data_type]
CHARACTER SET utf8
COLLATE utf8_general_ci;
Related
Table was created with:
CREATE TABLE IF NOT EXISTS `mathsqs` (
`questions` varchar(5000) NOT NULL,
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8;
I have inserted data through PHP using mysqli.
To confirm the insertion, I tried SELECT * FROM mathsqs LIMIT 1 on the Windows command line. It shows question marks for non english characters.
How do I see the exact posted data in MySql command line?
Example data I'm trying to handle:
இரு எண்களின் பெருக்கல் பலன் 3375 அவ்வெண்களின் மீ.பெ.வ 15
Assuming that you have already set the table char set to utf8 and it's collation is utf8.
Try adding this line just before you $mysqli command:
$mysqli->set_charset("utf8")
Also, set the utf 8 encoding in your page header where this output is coming.
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
Update:
When running the sql from command line, make sure you have set the default charset property before launching the mysql client. Something like:
Start the client with option --default-character-set=utf8.
mysql --default-character-set=utf8
To set it as a default option to be included automatically each time you run the mysql client add an entry in your my.cnf file, in the [mysql] section as:
[mysql]
default-character-set=utf8
Update #2:
#GopsAB I ran the DML statement to create the table as you specified. Followed the same process and surprisingly, I have having the same problem. I can't figure out why the question marks are displayed even after enforcing the character encoding.
So I digged further and made sure the command prompt was set to use use 'Lucidia Console' font and Active page code chcp 65001, By setting the property of the console to the use 'Lucidia Console' font and then running: chcp 65001 and followed the same process.
But now instead of '?' marks I am getting BOM character boxes....but the surprising thing is when I copy the console text that is displayed for the value of the column, I am getting the proper text: போக்குவரத்து (this is pasted directly from the console). Strange hah!
Important!
Turns out MySQL’s utf8 charset only partially implements proper UTF-8 encoding. It can only store UTF-8-encoded symbols that consist of one to three bytes; encoded symbols that take up four bytes aren’t supported.
In your case, the characters are stored perfectly and they are retrieved in php page and in mysql editors like sql workbench or Toad for SQL properly. Only the command line interface is unable to display them for some weird reason even after setting proper encoding and page type as discussed above. The text displayed in the console when copy/pasted displays correctly in notepad or any other place where you can type.
Running SET NAMES 'big5'; and SET NAMES 'utf8'; doesn't have any effect either
and neither did SET collation_connection = utf8_unicode_ci; SET NAMES utf8; did anything new but displayed only boxes which is the actual value when copy/pasted but on the console itself masked in boxes.
So until here you are good! nothing is wrong in your SQL and the values stored in the database are fine and fetched properly.
Something Extra:
MySQL’s utf8mb4
Luckily, MySQL 5.5.3 (released in early 2010) introduced a new encoding called utf8mb4 which maps to proper UTF-8 and thus fully supports Unicode, including astral symbols.
Switching from MySQL’s utf8 to utf8mb4
For each database:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
For each table:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
For each column:
ALTER TABLE table_name CHANGE column_name column_name VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
(Don’t blindly copy-paste this! The exact statement depends on the column type, maximum length, and other properties. The above line is just an example for a VARCHAR column.)
MySQL’s utf8mb4 Reference: How to support full Unicode in MySQL databases
When users try to save their name in german, they're saved like this:
Markus Müller ( Markus Müller)
Angela Eisenbl�tter ( Angela Eisenblätter )
Doris Vötter ( Doris Vötter )
I have inspected the values just before saving them with firebug and they show normally. But when saved they show like above.
The structure of my table is this
name varchar(250) utf8_unicode_ci
email varchar(250) utf8_unicode_ci
company varchar(250) utf8_unicode_ci
reading int(11)
rdate timestamp
Please help me
update
$con=mysqli_connect("localhost","englisch_root","b00t","englisch_efront");
mysql_set_charset('utf8', $con);
after i have added like this it give fullowing error
Warning: mysql_set_charset() expects parameter 2 to be resource,
Replace mysql_set_charset('utf8'); with mysqli_set_charset($con, 'utf8'); (or $con->set_charset('utf8');). You can't mix functions relative to databases of different PHP extensions (mysql vs mysqli): they work on different connections so they are mutually incompatible.
Notes:
MySQL uses utf8, not utf-8
never execute directly a SET NAMES statement, this is not safe:
If you must change the character set of the connection, use the mysql_set_character_set() function rather than executing a SET NAMES (or SET CHARACTER SET) statement. mysql_set_character_set() works like SET NAMES but also affects the character set used by mysql_real_escape_string(), which SET NAMES does not.
(from MySQL's documentation about mysql_real_escape_string, the C function behind mysql(i)_set_charset PHP functions)
Using mysqli_query($link,"SET CHARACTER SET utf8"); BEFORE the query solved the issue in my case. P.s. i suggest using mysqli_set_charset.
User table column data is in ASCII instead of utf8_unicode_ci.
Two issues...
SELECT HEX(col), col FROM ... WHERE ... -- For "Müller", you should get 4DC3BC6C6C6572 if it is correctly stored as utf8. If you don't get that, then you have either latin1 or a "double encoding", and fixing the data will be more complex.
If you are displaying this on a web page, you need a suitable tag near the top.
First Read this http://www.joelonsoftware.com/articles/Unicode.html For clarity of encoding.
Then follow this article http://www.toptal.com/php/a-utf-8-primer-for-php-and-mysql
You need to ensured that you are using UTF-8 encoding at all the places Including. Html page, Database schema, table, column collation, Db Connection etc.
I run the following query in mysql
UPDATE `gamequestions` SET a2 = '≠' WHERE id = 564
It runs successfully but the '?' is inserted in a2 field in place of '≠'
The datatype of a2 is text and also tried with varchar
Any Help greatly appreciated.
you need to change Collation to UTF-8 to store special characters
insert ≠ (not equal to ) in mysql field
The goal in these conversions is always to decide on what charset/collation combination you want to use (UTF8 being the best choice in almost all scenarios) then to convert all tables/columns in your database to use that charset. At that point you can set DB_COLLATE and DB_CHARSET` to the desired charset and collation to match.
Note:
In most cases if a collation is not defined MySQL will assume the default collation for the CHARSET which is specified. For UTF8 the default is utf8_general_ci, which is usually the right choice.
Changing the default charset of the database
ALTER DATABASE MyDb CHARACTER SET utf8;
Changing the default charset of individual tables
ALTER TABLE MyTable CHARACTER SET utf8;
https://dev.mysql.com/doc/refman/5.1/en/charset-unicode-utf8.html
You can add that option in the /mysql/my.cnf. In the [mysqld] section add ’’character-set-server=UTF8"; in the [client] section add “default-character-set=UTF8”.
You can find more information in these links:
http://dev.mysql.com/doc/refman/5.1/en/charset-… http://dev.mysql.com/doc/refman/5.0/en/server-o…
If you need to conver existing data, you can execute:
ALTER TABLE tbl_name CONVERT TO CHARACTER SET charset_name;
You need to check following things
use set names utf8 before you query/insert into the database
using Default CHARSET=utf8 when creating new tables
I am trying to import data from an XML file into a MYSQL DB using PHP. I am able to get the code to work just fine but when I look at the data in the DB there are special characters. For example, when I look at the XML in my browser it shows up as "outdoors in good weater..." but in the DB it appears to as "outdoors in good weather…".
I've cycled through all the different types of collation for that field in my DB but it does not seem to help much. Sometimes it shows up with the characters mentioned above and others as ???.
I have also tried to sync up the data with the following code in my PHP
$mysqli->query("SET NAMES 'utf8' COLLATE 'utf8_general_ci'");
But, again I have had no luck.
Thank you for reading this and for your help!
Akshay
You need to change the character set to UTF-8, along with your collation:
ALTER TABLE tablename CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
What you are seeing is a Unicode ellipsis character (…) being converted into another character set, which is probably Latin1. That is why it looks garbled.
I've moved database from one host to another. I've used PMA to export and bigdump to import. The whole database have latin2 charset set everywhere where it's possible. However in database, special chars (polish ąęłó, etc.) are broken. When I used SELECT i see "bushes" - "Ä�" insetad of "ą". Then I've set document encoding to utf-8... And the characters are good. How to fix this? Can it be done using CONVERT in query? I don't want to export/import database again, because it has over 200MB. What's wrong?
Every PHP/MySQL query solution will save me.
Sorry if you can't understand this, because I'm still learning english though.
If a table contains the wrong kind of charset (let's say utf-8 has slipped into latin1 column varhcar(255)):
ALTER TABLE tablename MODIFY colummname BINARY(255);
ALTER TABLE tablename MODIFY colummname VARCHAR(255) CHARSET utf8;
ALTER TABLE tablename MODIFY colummname VARCHAR(255) CHARSET latin1;
See also: http://dev.mysql.com/doc/refman/4.1/en/charset-conversion.html
However, it is more likely you just have a wrong character set in your default connection. What does a SET NAMES latin1; before selecting result in?