There are already plenty of posts about choosing the right charset for mysql, but it's again a different (and very frustrating) story for the rocksdb engine.
Firstly, I decided to use utf8-binary as charset (latin1, utf8-bin and binary are supported by myrocks) because my data may contain special chars and I want to be on the save side.
Furthermore, I am using PHP and PDO for loading data into mysql and the connection looks like this:
$pdo = new PDO('mysql:host=localhost;dbname=dbname;charset=utf8', 'user', 'password');
So I set the charset to utf8 (I also tried to use utf8_bin, but this is not supported by PDO). Although, I am able to insert some rows, sometimes I get errors like the following one:
Incorrect string value: '\xF0\x9F\x87\xA8\xF0\x9F...' for column 'column_name'
But what's the error now? This hex sequence encodes a unicode-smily (a regional indicator symbol letter c + regional indicator symbol letter n). Seems for me like valid utf8 and mysql as well as php are configured to use it.
You gotta have utf8mb4, not MySQL's subset utf8.
🇨 needs a 4-byte UTF-8 encoding, hex F09F87A8.
If rocksdb does not support it, then abandon either such characters, or rocksdb. Change the charset in the PDO call, and on the columns that need it.
I am using restler PHP API to host a REST service. I am having a problem with handling some European characters, otherwise it is working fine.
For example I have the string "Český rozhlas 1 - Radiožurnál" in a MySQL database. When the restler API converts the data in to JSON, it is converted like this "?esk\u00fd rozhlas 1 - Radio\u009eurn\u00e1l"
Here first character is converted as a question mark.
How can I convert the data properly using the restler PHP service?
When dealing with Unicode, we need to make sure we use utf-8 all the way
First you need to make sure MySQL database is using utf-8 encoding. You can run the following sql to make sure of that
ALTER TABLE your_table_name_here CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
Next you need to make sure MySQL spits utf-8 when talking to PHP.
You can use the following commands
mysql_query('SET NAMES utf8');
mysql_query('SET CHARACTER SET utf8');
If you are using PDO you need to use the following instead, for connecting to the database
$db = new PDO(
'mysql:host=localhost;dbname=data_pdo_mysql', 'username', 'password',
array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES 'utf8'")
);
After these changes, restler result should become
"\u010cesk\u00fd rozhlas 1 - Radio\u017eurn\u00e1l"
Which is valid JSON with complete data where unicode characters are escaped using unicode escape sequences.
im facing a really stressing problem here.. i have everything in UTF-8 , all my DB and tables are utf8_general_ci but when trying to insert or update from a single PHP script all i see are symbols.. but if i edit in phpmyadmin the words are shown correctly.. i found that if i run the utf8_decode() function to my strings in php i can make it work, but im not planning to do that because is a mess and it should work without doing that :S
Here is a basic code im using to test this:
<?php
$conn=mysql_connect("localhost","root","root")
or die("Error");
mysql_select_db("mydb",$conn) or
die("Error");
mysql_query("UPDATE `mydb`.`Clients` SET `name` = '".utf8_decode("Araña")."' WHERE `Clients`.`id` =25;",
$conn) or die(mysql_error());
mysql_close($conn);
echo "Success.";
?>
This is what i get if i dont decode utf8 with php utf8_decode function:
instead of Araña, i get : Araña
I've run into the same issue many times. Sometimes it's because the type of database link I'm selecting from isn't the same type that I'm using for inserting and other times, it's from file data into a database.
For the later instance, mysql_set_charset('utf8',$link); is the magic answer.
Place the call to mysql_set_charset just after you select your database via mysql_select_db.
#ref http://php.net/manual/en/function.mysql-set-charset.php
"Araña" IS UTF-8. The characters "ñ" represent the two bytes into which the Spanish ñ are encoded in UTF-8. Whatever you're reading it back with is not handling the UTF-8 and is displaying it as (it appears) ISO-8859-1.
That DDL you mentioned has to do with the collation, not the character set. The correct statement would be:
ALTER TABLE Clients CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
You still need to make sure the client library (libmysql or whatever driver PHP is using) is not transcoding the data back to ISO-8859. mysql_set_charset('utf8') will explicitly set the client encoding to UTF-8. Alternatively, you can send a SET NAMES UTF8; right after you connect to the database. To do that implicitly, you can change the my.cnf [client] block to have utf-8 as the client character encoding (and /etc/init.d/mysql reload to apply). Either way, make sure the client doesn't mangle the results it's pulling.
[client]
default-character-set=utf8
You do not need to use utf8_decode if you're using mbstrings. The following php.ini configuration should ensure UTF-8 support on the PHP side:
mbstring.internal_encoding = utf-8
mbstring.http_output = utf-8
mbstring.func_overload = 6
Finally, when you display the results in HTML, verify that the page's encoding is explicitly UTF-8.
I've a MySQL table that has a UTF-8 charset and upon attempting to insert to it via a PHP form, the database gives the following error:
PDOStatement::execute():
SQLSTATE[HY000]: General error: 1366
Incorrect string value: '\xE8' for
column ...
The character in question is 'è', yet I don't see why this should be a problem considering the database and table are set to UTF-8.
Edit
I've tried directly from the mysql terminal and have the same problem.
Your database might be set to UTF-8, but the database connection also needs to be set to UTF-8. You should do that with a SET NAMES utf8 statement. You can use the driver_options in PDO to have it execute that as soon as you connect:
$handle = new PDO("mysql:host=localhost;dbname=dbname",
'username', 'password',
array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"));
Have a look at the following two links for more detailed information about making sure your entire site uses UTF-8 appropriately:
UTF-8 all the way through…
UTF8, PHP and MySQL
E8 is greater than the maximum usable character 7F in a one-byte UTF8 character: http://en.wikipedia.org/wiki/UTF-8
It seems your connection is not set to UTF8 but some other 8 bit encoding like ISO Latin. If you set the database to UTF8 you only change the character set the database uses internally, connections may be on a different default value (latin1 for older MySQL versions) so you should try to send an initial SET CHARACTER SET utf-8 after connecting to the database. If you have access to my.cnf you can also set the correct default value there, but keep in mind that changing the default may break any other sites/apps running on the same host.
Before passing the value to Mysql you can use the following code:
$val = mb_check_encoding($val, 'UTF-8') ? $val : utf8_encode($val);
convert the string the to UTF-8, If it's matter of only one field.
I'm trying to store some data in a SQL Server database through php.
Problem is that special chars aren't converted properly. My app's charset is iso-8859-1
and the one used by the server is windows-1252.
Converting the data manually before inserting doesn't help, there seems to be some
conversion going on.
Running the SQL query 'set char_convert off' doesn't help either.
Anyone have any idea how I can get this to work?
EDIT: I have tried ini_set('mssql.charset', 'windows-1252'); as well, but no result with that one either.
Client charset is necessary but not sufficient:
ini_set('mssql.charset', 'UTF-8');
I searched for two days how to insert UTF-8 data (from web forms) into MSSQL 2008 through PHP. I read everywhere that you can't, you need to convert to UCS2 first (like cypher's solution recommends).
On Windows SQLSRV said to be a good solution, which I couldn't try, since I am developing on Mac OSX.
However, FreeTDS manual (what PHP mssql uses on OSX) says to add a letter "N" before the opening quote:
mssql_query("INSERT INTO table (nvarcharField) VALUES (N'űáúőűá球最大的采购批发平台')", +xon);
According to this discussion, N character tells the server to convert to Unicode.
https://softwareengineering.stackexchange.com/questions/155859/why-do-we-need-to-put-n-before-strings-in-microsoft-sql-server
I had the same problem and ini_set('mssql.charset', 'utf-8') did not work for me.
However, it worked in uppercase:
ini_set('mssql.charset', 'UTF-8');
I suggest looking at the following points:
Ensure that the columns that you're storing the information in are nchar or nvarchar as char and nvarchar don't support UCS-2 (SQLServer doesn't store in UTF-8 format btw)
If you're connecting with the mssql library/extension for PHP, run: ini_set('mssql.charset', 'utf-8'); as there's no function with a charset argument (connect, query etc)
Ensure that your browsers charset is also set to UTF-8
If ini_set('mssql.charset', 'UTF-8'); doesn't help AND you don't have root access to modify the system wide freetds.conf file, here's what you can do:
1. Set up /your/local/freetds.conf file:
[sqlservername]
host=192.168.0.56
port=1433
tds version=7.0
client charset=UTF-8
2. Make sure your connection DSN is using the servername, not the IP:
'dsn' => 'dblib:host=sqlservername;dbname=yourdb
3. Make FreeTDS to use your local freetds.conf file as an unprivileged user from php script via env variables:
putenv('FREETDSCONF=/your/local/freetds.conf');
If you are using TDS protocol version 7 or above, ALL communications over the wire are converted to UCS2. The server will convert from UCS2 into whatever the table or column collation is set to, unless the column is nvarchar or ntext. You can store UTF-8 into regular varchar or text, you just have to use a TDS protocol version lower than 7, like 6.0 or 4.2. The only drawback with this method is that you cannot query any nvarchar, ntext, or sys.* tables (I think you also can't do any CAST()ing) - as the server refuses to send anything that might possibly be converted to UTF-8 to any client using protocol version lower than 7.
It is not possible to avoid converting character sets when using TDS protocol version 7 or higher (roughly equivalent to MSSQL 2005 or newer).
In my case, It worked after I added the "CharacterSet" parameters into sqlsrv_connect() connection's option.
$connectionInfo = array(
"Database"=>$DBNAME,
"ConnectionPooling"=>0,
"CharacterSet"=>"UTF-8"
);
$LAST_CONNECTION = sqlsrv_connect($DBSERVER, $connectionInfo);
See documentation here :
https://learn.microsoft.com/en-us/sql/connect/php/connection-options?view=sql-server-2017
I've had luck in a similar situation (using a PDO ODBD connection) using the following code to convert the encoding before printing output:
$data = mb_convert_encoding($data, 'ISO-8859-1', 'windows-1252');
I had to manually set the source encoding, because it was erroneously being reported as 'ISO-8859-1' by mb_detect_encoding().
My data was also being stored in the database by another application, so I might be in a unique situation, although I hope it helps!
For me editing this file:
/etc/freetds/freetds.conf
...and changing/setting 'tds version' parameter to '7.0' helped. Edit your freetds.conf and try to change this parameter for your server configuration (or global).
It will work even without apache restart.
I did not notice someone to mention another way of converting results from MSSQL database. The good old iconv() function:
iconv (string $in_charset, string $out_charset, string $str): string;
In my case everything else failed to provide meaningful conversion, except this one when getting the results. Of course, this is done inside the loop of parsing the results of the query - from CP1251 to UTF-8:
foreach ($records as $row=>$col) {
$array[$row]['StatusName'] = iconv ('CP1251', 'UTF-8' , $records[$row]['StatusName']);
}
Ugly, but it works.
Can't you just convert your tables to your application encoding? Or use utf-8 in both?
I don't know whether MSSQL supports table-level encodings, though.
Also, try the MB (multibyte) string functions, if the above fails.
You should set the charset with ini_set('mssql.charset', 'windows-1252') before the connection. If you use it after the mssql_connect it has no effect.
Just adding ini_set('mssql.charset', 'UTF-8'); didn't help me in my case. I had to specify the UTF-8 character set on the column:
$age = 30;
$name = utf8_encode("Joe");
$select = sqlsrv_query($conn, "SELECT * FROM Users WHERE Age = ? AND Name = ?",
array(array($age), array($name, SQLSRV_PARAM_IN, SQLSRV_PHPTYPE_STRING('UTF-8')));
You can use the mysql_set_charset function:
http://it2.php.net/manual/en/function.mysql-set-charset.php