I have an application talking to a NodeJS server via socket, that server is talking to PHP via HTTP, and that PHP is using MySQL.
When I try to insert Emojis in a text field, and save it to DB, when I get it back I see "?".
If I copy Emojis from wikipedia (for example ✒️ ❤️ 🀄️ 🈚️) and force them to the DB, when I select them it shows ✒️ ❤️ ?️ ?️
Note: The field in DB is utf8mb4bin.
Why can't I store emojis directly from my device to DB?
Why does some emojis show, while others don't?
Note: I also tried only PHP and MySQL, but with no luck.
1.)Database :
Change Database default collection as **utf8mb4 ** .
2)Table :
Change Table collection as ** CHARACTER SET utf8mb4 COLLATE utf8mb4_bin **.
Query :
ALTER TABLE Tablename CONVERT TO CHARACTER SET utf8mb4 COLLATE
utf8mb4_bin
3)Code :
insert into tablename (column1,column2,column3,column4,column5,column6,column7) values ('273','3','Hdhdhdh😜😀😊😃hzhzhzzhjzj 我爱你 ❌',49,1,'2016-09-13 08:02:29','2016-09-13 08:02:29')
4)Set utf8mb4 in database connection :
$database_connection = new mysqli($server, $user,$password,$database_name);
$database_connection->set_charset("utf8mb4");
Related
Table was created with:
CREATE TABLE IF NOT EXISTS `mathsqs` (
`questions` varchar(5000) NOT NULL,
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8;
I have inserted data through PHP using mysqli.
To confirm the insertion, I tried SELECT * FROM mathsqs LIMIT 1 on the Windows command line. It shows question marks for non english characters.
How do I see the exact posted data in MySql command line?
Example data I'm trying to handle:
இரு எண்களின் பெருக்கல் பலன் 3375 அவ்வெண்களின் மீ.பெ.வ 15
Assuming that you have already set the table char set to utf8 and it's collation is utf8.
Try adding this line just before you $mysqli command:
$mysqli->set_charset("utf8")
Also, set the utf 8 encoding in your page header where this output is coming.
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
Update:
When running the sql from command line, make sure you have set the default charset property before launching the mysql client. Something like:
Start the client with option --default-character-set=utf8.
mysql --default-character-set=utf8
To set it as a default option to be included automatically each time you run the mysql client add an entry in your my.cnf file, in the [mysql] section as:
[mysql]
default-character-set=utf8
Update #2:
#GopsAB I ran the DML statement to create the table as you specified. Followed the same process and surprisingly, I have having the same problem. I can't figure out why the question marks are displayed even after enforcing the character encoding.
So I digged further and made sure the command prompt was set to use use 'Lucidia Console' font and Active page code chcp 65001, By setting the property of the console to the use 'Lucidia Console' font and then running: chcp 65001 and followed the same process.
But now instead of '?' marks I am getting BOM character boxes....but the surprising thing is when I copy the console text that is displayed for the value of the column, I am getting the proper text: போக்குவரத்து (this is pasted directly from the console). Strange hah!
Important!
Turns out MySQL’s utf8 charset only partially implements proper UTF-8 encoding. It can only store UTF-8-encoded symbols that consist of one to three bytes; encoded symbols that take up four bytes aren’t supported.
In your case, the characters are stored perfectly and they are retrieved in php page and in mysql editors like sql workbench or Toad for SQL properly. Only the command line interface is unable to display them for some weird reason even after setting proper encoding and page type as discussed above. The text displayed in the console when copy/pasted displays correctly in notepad or any other place where you can type.
Running SET NAMES 'big5'; and SET NAMES 'utf8'; doesn't have any effect either
and neither did SET collation_connection = utf8_unicode_ci; SET NAMES utf8; did anything new but displayed only boxes which is the actual value when copy/pasted but on the console itself masked in boxes.
So until here you are good! nothing is wrong in your SQL and the values stored in the database are fine and fetched properly.
Something Extra:
MySQL’s utf8mb4
Luckily, MySQL 5.5.3 (released in early 2010) introduced a new encoding called utf8mb4 which maps to proper UTF-8 and thus fully supports Unicode, including astral symbols.
Switching from MySQL’s utf8 to utf8mb4
For each database:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
For each table:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
For each column:
ALTER TABLE table_name CHANGE column_name column_name VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
(Don’t blindly copy-paste this! The exact statement depends on the column type, maximum length, and other properties. The above line is just an example for a VARCHAR column.)
MySQL’s utf8mb4 Reference: How to support full Unicode in MySQL databases
I am parsing a XML and putting it into MySQL. Now MySQL is showing following error for some specific records inserting:
SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xEAm-Kh\xEA...'
When i check the data inside XML, it is showing like iêm-Khê
So the Special Character ê is transforming into xEA
What is it?
How can i get it solved to maintain & put its original Character into Database?
FYI, here is some info.:
My whole MySQL DB itself is using latin1_swedish_ci
But that TABLE & COLUMN are using utf8_general_ci
** When i manually Copy/Paste that iêm-Khê values into the MySQL Record directly, it is saved. (I mean, MySQL is accepting that Special Characters.) So i think the issue is at the Coding side.
After some additional research I ran across two possible approaches to a solution:
- the first option is to use php's utf8_encode() function.
- the second option uses some code with the PDO using set names (though there seemed to be some discussion as to whether or not this worked for everyone.
$handle = new PDO("mysql:host=localhost;dbname=dbname",
'username', 'password',
array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"));
The following link had some other useful information:
See here
You should define the UTF-8 charset to the mysql DB, by default it used to be latin charset , we can define the utf-8 along with table definition or on database creation,
To Table
create table `my_table` (.....
.... ) DEFAULT CHARSET=utf8;
To database
CREATE DATABASE mydb DEFAULT CHARACTER SET utf8
correct me if 'm worng
You need to set the character set of the database to UTF-8.
Command:
ALTER DATABASE db_name CHARACTER SET utf8 COLLATE utf8_general_ci;
For alter the column:
ALTER TABLE [table_name] MODIFY
[column_name] [data_type]
CHARACTER SET utf8
COLLATE utf8_general_ci;
I am writing a PHP script to import some data from a Microsoft SQL Server database to my MySQL database.
Collation in the SQL Server is Modern_Spanish_CI_AI (so I guess the encoding is ASCII), and in my MySQL database I'm using UTF-8 encoding and utf8_unicode_ci as collation.
When I import the data I get wrong characters corresponding to the Spanish letter Ñ. I've tried to use the PHP functions utf8_encode($string) and mb_convert_encoding($string, "UTF-8") without any success, (I got different wrong characters but still wrong).
Perhaps this helps:
Table's definition in SQL Server:
CREATE TABLE [dbo].[myTable] (
[myField] varchar(2) COLLATE Modern_Spanish_CI_AS NOT NULL,
Table's definition in MySQL
CREATE TABLE `myTable` (
`myField` varchar(2) COLLATE utf8_unicode_ci NOT NULL,
I have no experience with SQL server, but from a cursory glance at this collation chart, Modern_Spanish_CI_AI seems to be essentially a Windows-1252 character set, probably with some Spanish sorting / comparison rules on top.
You should be able to convert the data like this:
$utf8 = iconv("windows-1252", "utf-8", $input);
try it out. However, when using PHP for this, the character set of the connection to each database also plays a role - if it still doesn't work, you need to show some more code.
I have a MySQL database table populated with non English data. When I view the data in Navicat MySQL browser the data appears fine. However when I run a php script to select and display the data on a web page it displays question marks instead. The page encoding is set to utf8 and even the MySQL collation is set to utf8 - something gets wrong in selecting and displaying... please help.
MySQL connection settings could be at fault here. Run this MySQL command when you connect to the database from PHP, before you run any other SQL commands:
SET names 'utf8';
This should set the connection's encoding to UTF-8. As you're saying, the page and the database is already in UTF-8 (that should also mean the page sends Content-Type: text/html; charset=utf-8); the connection itself can accidentaly have a different encoding by default :(
In addition, in HTML, inside we must write:
<meta charset="utf-8">
When we create a MySQL database, we must use something like this:
CREATE DATABASE `base` DEFAULT CHARACTER SET=utf8;
When we create tables, use:
CREATE TABLE `base`.`table` (
`key` int(5) unsigned NOT NULL,
`name` varchar(100),
...
PRIMARY KEY (`key`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
In PHP code, after mysql_connect() and mysql_select_db(), use:
mysql_query('SET NAMES UTF8');
Hey guys I am having trouble trying to convert my web app to support unicode characters. I have the following script which tries to insert russian characters into my mysql database but just outputs ?????? in my mysql database field. I have changed default charset to UTF-8 in my php.ini and have modified my table fields to collation: utf8_unicode_ci. Anyone have any ideas?
mb_language('uni'); mb_internal_encoding('UTF-8');
$sql = 'SET NAMES utf8';
$stmt = $conn->prepare($sql);
$result=$stmt->execute();
$sql = 'SET CHARACTER SET utf8';
$stmt = $conn->prepare($sql);
$result=$stmt->execute();
$sql = 'INSERT INTO topic (topic_id,topic_title) VALUES (?,?)';
$stmt6 = $conn->prepare($sql);
$result=$stmt6->execute(array(0,"дравствуйте"));
?>
show create table edit
CREATE TABLE `topic` (
`id` mediumint(8) NOT NULL AUTO_INCREMENT,
`topic_title` varchar(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`description` mediumtext CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
FULLTEXT KEY `description` (`description`),
FULLTEXT KEY `topic_title` (`topic_title`),
FULLTEXT KEY `topic_title_2` (`topic_title`,`description`),
FULLTEXT KEY `description_2` (`description`)
) ENGINE=MyISAM AUTO_INCREMENT=39 DEFAULT CHARSET=latin1
I got it to work by using the following sql:
ALTER DATABASE mydatabase charset=utf8;
If you are seeing those ?????? on mysql browser or on mysql cli, make sure you tune up the charset settings on the programs, even if the entries are correctly added to the database, displaying them correctly also depends on the client charset settings.
I'm inclined to think the problem isn't with MySQL.
You are entering an non ASCII string as a literal into a PHP script. Are you sure the "дравствуйте" string will be sent to MySQL server as UTF-8 encoded?
I've seen some problems with nos ASCII characters with PHP and MySQL with incompatible charset configurations amongst the operating system, the PHP interpreter (it inherits the default from the operating system), the database and the development tools (the encoding of .php files). If all four are configured to the same encoding, it should work.
You said that your php.ini is configured to UTF-8. If your source php file isn't in UTF-8, it will not work! So, check if your php files are encoded as UTF-8 and if the mysql client is configured to UTF-8 too.
A common mistake is to forget to configure the MySQL client to UTF-8 too and fail to check the data on the database because of charset problems with MySQL client application.
Home it helps!
Normally, (if this you're quoting file is also in utf-8) the PHP & MySQL side is OK, but the HTTP / HTML side of things isn't.
Do you send a Content-Type: text/html;charset=utf-8 header?
Is there a <meta> element on the HTML page that claims another charset then utf-8?
So, after more infomation: in 'normal' PHP scripts it works I gather (?), in PHPMyAdmin it doesn't (which should make it more of a candidate for superuser.com b.t.w.). A version of PHPMyAdmin used would be nice, but normally, a $cfg['DefaultCharset'] = 'utf-8'; in PHPMyAdmin config should work if the rest of non-PHPMyAdmin script works. If not: update to the latest PHPMyAdmin release, and if it still doesn't work file a bug report.
From PHPMYADMIN please change the Collation to utf32_general_ci, they will support and some cases it will make existing "????" to correct one as well.
PHP Myadmin illustration purpose
DB Activity for Storing Emojis
1) default-character-set = utf8mb4 under [client]
2) default-character-set = utf8mb4 under [mysql]
3) Following under [mysqld]
character-set-client-handshake = FALSE
character-set-server = `utf8mb4`
collation-server = `utf8mb4_unicode_ci`
4) Restart Mysql server using below command
`sudo service mysql stop`
`sudo service mysql start`
5) ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
6) ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
7) SET NAMES utf8mb4;