I am using restler PHP API to host a REST service. I am having a problem with handling some European characters, otherwise it is working fine.
For example I have the string "Český rozhlas 1 - Radiožurnál" in a MySQL database. When the restler API converts the data in to JSON, it is converted like this "?esk\u00fd rozhlas 1 - Radio\u009eurn\u00e1l"
Here first character is converted as a question mark.
How can I convert the data properly using the restler PHP service?
When dealing with Unicode, we need to make sure we use utf-8 all the way
First you need to make sure MySQL database is using utf-8 encoding. You can run the following sql to make sure of that
ALTER TABLE your_table_name_here CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
Next you need to make sure MySQL spits utf-8 when talking to PHP.
You can use the following commands
mysql_query('SET NAMES utf8');
mysql_query('SET CHARACTER SET utf8');
If you are using PDO you need to use the following instead, for connecting to the database
$db = new PDO(
'mysql:host=localhost;dbname=data_pdo_mysql', 'username', 'password',
array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES 'utf8'")
);
After these changes, restler result should become
"\u010cesk\u00fd rozhlas 1 - Radio\u017eurn\u00e1l"
Which is valid JSON with complete data where unicode characters are escaped using unicode escape sequences.
Related
There are already plenty of posts about choosing the right charset for mysql, but it's again a different (and very frustrating) story for the rocksdb engine.
Firstly, I decided to use utf8-binary as charset (latin1, utf8-bin and binary are supported by myrocks) because my data may contain special chars and I want to be on the save side.
Furthermore, I am using PHP and PDO for loading data into mysql and the connection looks like this:
$pdo = new PDO('mysql:host=localhost;dbname=dbname;charset=utf8', 'user', 'password');
So I set the charset to utf8 (I also tried to use utf8_bin, but this is not supported by PDO). Although, I am able to insert some rows, sometimes I get errors like the following one:
Incorrect string value: '\xF0\x9F\x87\xA8\xF0\x9F...' for column 'column_name'
But what's the error now? This hex sequence encodes a unicode-smily (a regional indicator symbol letter c + regional indicator symbol letter n). Seems for me like valid utf8 and mysql as well as php are configured to use it.
You gotta have utf8mb4, not MySQL's subset utf8.
🇨 needs a 4-byte UTF-8 encoding, hex F09F87A8.
If rocksdb does not support it, then abandon either such characters, or rocksdb. Change the charset in the PDO call, and on the columns that need it.
I'm in a situation where I need to update some rows in a table named "matrículas'. The query looks something like this:
UPDATE `matrículas` SET...
When I run this query in my SQL program (HeidiSQL) directly, it executes without problems. When I do it in PHP via a PDO object, I get the following error:
SQLSTATE[HY000]: General error: 1300 Invalid utf8 character string: 'matr\xEDculas'
My PDO object is set up like this:
$db= new PDO(
'mysql:host='.$credentials['host'].';dbname='.$credentials['dbname'].';charset=utf8',
$credentials['user'],
$credentials['password'],
array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8")
);
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
The actual update is done by taking the above query and doing this:
$query = $this->db->prepare($sql);
$query->execute($params);
Both the table and the database were created using the utf8_general_ci collation.
Any ideas what I'm doing wrong? btw, I'm currently testing in Windows in case that has anything to do with it...
ERROR 1300 (HY000): Invalid utf8 character string: 'matr\xEDculas'
The \xNN notation gives the hex encoded value for the invalid byte(s) in the character string.
Unicode code point 237 (í), when encoded in utf-8, is a 2-byte character that is encoded as 0xC3 0xAD... but the error shows 0xED, which happens to be the ISO/IEC-8859-1 (Latin1) encoding for the character í.
Since the error is related to a column name being passed from the script rather than external data, that suggested what turned out to be the issue -- that the PHP script, itself, had the column name encoded incorrectly, since the character set in which the script had been saved was ISO-8859-1 rather than UTF-8.
`matrículas`
this is cp866-gp2312 encoding
please change it to utf-8 like matriculas
i having the different encoding style
If you must use accented letters in table names, then they must be encoded in UTF-8 in the client.
That is, it is not a PDO problem, but an encoding problem is your source editor/language/whatever.
Hello this is a follow up questing from yesterday,
I have a php script that is parsing a website. I am getting strings in UTF-8 now i want to insert those strings into my Oracle database which uses:
NLS_NCHAR_CHARACTERSET = AL16UTF16
NLS_CHARACTERSET = EE8ISO8859P2
I have tried with :
$rep = iconv("UTF-8","AL-16UTF-16",$string);
// FAILS - produces ?? in database or scripts fails with "wrong charset"
I have also tried with
$rep = iconv("UTF-8","ISO-8859-2",$string);
$rep1 = iconv("UTF-8","AL-16UTF-16",$rep);
same as above ... fails with ?? in database.
Anyone has any idea what should i try next?
The OCI driver implicitly handles charset conversion. When connecting, ensure you set your charset as UTF-8:
oci_connect($username, $password, $connection_string, 'UTF-8');
This tells OCI to expect you to provide strings in UTF8 format and to provide resultsets in UTF8, converted from the database charset. From the manual (emphasis mine):
Determines the character set used by the Oracle Client libraries. The character set does not need to match the character set used by the database. If it doesn't match, Oracle will do its best to convert data to and from the database character set. Depending on the character sets this may not give usable results. Conversion also adds some time overhead.
This means, assuming that the strings you want to input are in UTF8, that you shouldn't need to use iconv() at all. Just let OCI handle that for you.
what's up? :-)
I have one problem and i hope you can help me with it.
One friend of mine have a simple solid html website and i implemented little php; CRUD system for articles... problem i came across is placing and getting cyrillic characters from mysql database.
What i want to achive is next:
In the main navigation there are some separated sections, whose names, ids and item's order i want to place in mysql and than to pull names and to put each name as a link. Names are supposed to be cyrillic characters.
The problem comes when i, using php mysql_fetch_assoc function, try to display names which are inserted with cyrillic characters in database row, collation of row is utf8_general_ci, and i end with ????? insted of original characters. If i submit cyrillic characters via submit form to mysql it shows something like this У.
How can i solve this, thanks in advance!? :-)
Make sure you call this after connecting to database.
mysql_query("SET NAMES UTF8");
Also make sure that HTML file has charset meta tag set to UTF-8 or send header before output.
header("Content-Type: text/html; charset=utf-8");
I had the same problem until I encoded the 'Collation' column in my table to 'utf8_bin'.
if its really mysql fetch assoc messing up you should try:
mysql-set-charset
from the docs:
Note:
This is the preferred way to change
the charset. Using mysql_query() to
execute SET NAMES .. is not
recommended.
also make sure your files are saved as utf8 and check iconv_set_encoding / iconv_get_encoding
For anyone having more complex issues with legacy project upgrades from versions before PHP 5.6 and MYSQL 5.1 to PHP 7 & Latest MySQL/Percona/MariaDB etc...
If the project uses utf8_encode($value) you can either try removing the function from the value being prepared and use the accepted answer for setting UTF-8 encoding for all input.
--- OR ---
Try replacing utf8_encode($value) with mb_convert_encoding($value, 'utf-8')
PDO USERS
If you are using PDO here are two ways how to set utf8:
$options = [
\PDO::MYSQL_ATTR_INIT_COMMAND => 'SET NAMES utf8'
];
new \PDO($dsn, $username, $passwd, $options);
--- OR ---
$dsn = 'mysql:host=localhost;charset=utf8;'
new \PDO($dsn, $username, $passwd);
I can confirm that mb_convert_encoding($value, 'utf-8') to SQL table using utf8_unicode_ci works for Cyrillic and Umlaut.
I've a MySQL table that has a UTF-8 charset and upon attempting to insert to it via a PHP form, the database gives the following error:
PDOStatement::execute():
SQLSTATE[HY000]: General error: 1366
Incorrect string value: '\xE8' for
column ...
The character in question is 'è', yet I don't see why this should be a problem considering the database and table are set to UTF-8.
Edit
I've tried directly from the mysql terminal and have the same problem.
Your database might be set to UTF-8, but the database connection also needs to be set to UTF-8. You should do that with a SET NAMES utf8 statement. You can use the driver_options in PDO to have it execute that as soon as you connect:
$handle = new PDO("mysql:host=localhost;dbname=dbname",
'username', 'password',
array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"));
Have a look at the following two links for more detailed information about making sure your entire site uses UTF-8 appropriately:
UTF-8 all the way through…
UTF8, PHP and MySQL
E8 is greater than the maximum usable character 7F in a one-byte UTF8 character: http://en.wikipedia.org/wiki/UTF-8
It seems your connection is not set to UTF8 but some other 8 bit encoding like ISO Latin. If you set the database to UTF8 you only change the character set the database uses internally, connections may be on a different default value (latin1 for older MySQL versions) so you should try to send an initial SET CHARACTER SET utf-8 after connecting to the database. If you have access to my.cnf you can also set the correct default value there, but keep in mind that changing the default may break any other sites/apps running on the same host.
Before passing the value to Mysql you can use the following code:
$val = mb_check_encoding($val, 'UTF-8') ? $val : utf8_encode($val);
convert the string the to UTF-8, If it's matter of only one field.