Non-english characters in MySQL command-line on Windows - php

Question

Table was created with:
CREATE TABLE IF NOT EXISTS `mathsqs` (
`questions` varchar(5000) NOT NULL,
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8;
I have inserted data through PHP using mysqli.
To confirm the insertion, I tried SELECT * FROM mathsqs LIMIT 1 on the Windows command line. It shows question marks for non english characters.
How do I see the exact posted data in MySql command line?
Example data I'm trying to handle:
இரு எண்களின் பெருக்கல் பலன் 3375 அவ்வெண்களின் மீ.பெ.வ 15

Answer 1

Assuming that you have already set the table char set to utf8 and it's collation is utf8.
Try adding this line just before you $mysqli command:
$mysqli->set_charset("utf8")
Also, set the utf 8 encoding in your page header where this output is coming.
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
Update:
When running the sql from command line, make sure you have set the default charset property before launching the mysql client. Something like:
Start the client with option --default-character-set=utf8.
mysql --default-character-set=utf8
To set it as a default option to be included automatically each time you run the mysql client add an entry in your my.cnf file, in the [mysql] section as:
[mysql]
default-character-set=utf8
Update #2:
#GopsAB I ran the DML statement to create the table as you specified. Followed the same process and surprisingly, I have having the same problem. I can't figure out why the question marks are displayed even after enforcing the character encoding.
So I digged further and made sure the command prompt was set to use use 'Lucidia Console' font and Active page code chcp 65001, By setting the property of the console to the use 'Lucidia Console' font and then running: chcp 65001 and followed the same process.
But now instead of '?' marks I am getting BOM character boxes....but the surprising thing is when I copy the console text that is displayed for the value of the column, I am getting the proper text: போக்குவரத்து (this is pasted directly from the console). Strange hah!
Important!
Turns out MySQL’s utf8 charset only partially implements proper UTF-8 encoding. It can only store UTF-8-encoded symbols that consist of one to three bytes; encoded symbols that take up four bytes aren’t supported.
In your case, the characters are stored perfectly and they are retrieved in php page and in mysql editors like sql workbench or Toad for SQL properly. Only the command line interface is unable to display them for some weird reason even after setting proper encoding and page type as discussed above. The text displayed in the console when copy/pasted displays correctly in notepad or any other place where you can type.
Running SET NAMES 'big5'; and SET NAMES 'utf8'; doesn't have any effect either
and neither did SET collation_connection = utf8_unicode_ci; SET NAMES utf8; did anything new but displayed only boxes which is the actual value when copy/pasted but on the console itself masked in boxes.
So until here you are good! nothing is wrong in your SQL and the values stored in the database are fine and fetched properly.
Something Extra:
MySQL’s utf8mb4
Luckily, MySQL 5.5.3 (released in early 2010) introduced a new encoding called utf8mb4 which maps to proper UTF-8 and thus fully supports Unicode, including astral symbols.
Switching from MySQL’s utf8 to utf8mb4
For each database:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
For each table:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
For each column:
ALTER TABLE table_name CHANGE column_name column_name VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
(Don’t blindly copy-paste this! The exact statement depends on the column type, maximum length, and other properties. The above line is just an example for a VARCHAR column.)
MySQL’s utf8mb4 Reference: How to support full Unicode in MySQL databases