In my table, I have a row like this:
Amazing ... ๐ฐ๐ฐ๐ฐ
When I try do display it in my view, it show this:
Amazing ... ???
In the head of the html page, I have well the tag
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
In core.php I have:
Configure::write('App.encoding', 'UTF-8');
In my database.php, I have:
public $default = array(
'datasource' => 'Database/Mysql',
'persistent' => false,
'host' => 'localhost',
'login' => 'root',
'password' => 'xxx',
'database' => 'xxx',
'prefix' => '',
'encoding' => 'utf8',
);
I am converting a current python script in php and I can see this code:
'comment_text': row[2].encode('unicode-escape'),
I tried for find the equivalent for encode('unicode-escape') in php but nothing found.
Do I need to use a similar function for my php display or I don't need to use this function equivalent and something is wrong with my encoding setup ?
I also had same problem before. The thing is the utf8 encoding only supports three bytes per character. You can read detail here
MySQLโs utf8 isnโt UTF-8. So, you can't save some char and emoji and sometime it may cut off your text.
What I did is I applied utf8mb4 to all table and schema.
E.g
`ALTER TABLE your_table CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;`
`ALTER SCHEMA `your_schema` DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;`
After onward, happy to saved emoji character as well :).
Related
I'm getting this error:
SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\xBD Inch...' for column 'column-name' at row 1
My database, table, and column have the format utf8mb4_unicode_ci also column-name is type text and NULL.
This is the value of the column-name
[column-name] => Some text before 11 โ and other text after, and after.
However I wait that laravel adds quotes to column's values, because the values are separated by commas (,). It should be as follow:
[column-name] => 'Some text before 11 โ and other text after, and after.'
See below the Schema
Schema::create('mws_orders', function (Blueprint $table) {
$table->string('custom-id');
$table->string('name');
$table->string('description')->nullable();
$table->string('comment')->nullable();
$table->integer('count')->nullable();
$table->text('column-name')->nullable();
$table->timestamps();
$table->primary('custom-id');
});
I have been looking for on google but not any solution, yet.
Anyone has an idea how to solve this issue?
I'm using Laravel 5.5 and MariaDB 10.2.11.
I solved it, encoding to uft-8 all string columns that generated this error before insert. For example, the column that generated the error was column-name, I encoded as show bellow. Also I found other column with the same error, I used this solution, too.
$data [
//key=>values
];
$myModel = new MyModel();
$data['column-name'] = DB::connection()->getPdo()->quote(utf8_encode($data['column-name']));
$myModel->insert($data);
I ran into similar problems with Laravel 5.5 and MariaDB 10.2. When storing some user input t into a varchar column, an exception:
SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\xE2\x80\x86y\xE2\x80...' for column 'comment' at row 1
will be thrown.
Since this only happened on the stage server but not on local dev server, I compared the collation and charset of underlining table, it turns out database and table on stage server use latin1 while local dev server uses utf8mb4.
The problem was solved by changing database and table collation and char set to utf8mb4 and utf8mb4_unicode_ci.
ALTER DATABASE <db_name> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE <table_name> CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
For anyone who runs into this problem, please check collation and char set of your database and table. Chances are Laravel app is encoding with utf8 while databases uses something else.
If mb-convert-encoding or utf8-encode are not solving this problem for you, check if you're only using string functions in their multibyte variants.
e.g.
Instead of substr you must use mb_substr
Doc reference here: Multibyte String Functions
Written for future readers who might end up with my same problem :)
BD is the latin1 (and several others) encoding for ยฝ (one-half). The error message talks about storing that in a datetime. So, it sounds like there are at least two errors --
mismatch of CHARACTER SETs
poorly formulated query
You show us something about the CREATE TABLE, but why would "inches" be involved in that?
Just change database configuration (charset & collation) in
config/database.php
to:
'connections' => [
'mydb' => [
'driver' => 'mysql',
'host' => env('DB_HOST'),
'database' => env('DB_DATABASE'),
'username' => env('DB_USERNAME'),
'password' => env('DB_PASSWORD'),
'charset' => 'utf8mb4', // **for emoticons**
'collation' => 'utf8mb4_unicode_ci', // **for emoticons**
],
]
follow this steps for solve problem
1- change CHARACTER and COLLATE`s table
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
2-change config /database.php file
'mysql' => [
'driver' => 'mysql',
'host' => env('DB_HOST', '127.0.0.1'),
'port' => env('DB_PORT', '3306'),
'database' => env('DB_DATABASE', 'forge'),
'username' => env('DB_USERNAME', 'forge'),
'password' => env('DB_PASSWORD', ''),
'unix_socket' => env('DB_SOCKET', ''),
'charset' => 'utf8mb4',//****make sure these configs are set
'collation' => 'utf8mb4_unicode_ci',//****make sure these configs are set
'prefix' => '',
'strict' => true,
'engine' => null,
],
3- run this command to regenerate config cache
php artisan config:cache
Have an issue with a project in php(laravel), when i bring info from a database(mysql) and it contains accents i.e รฉ, รก . Php ends up showing a weird question mark, or crashin
'charset' => 'utf8mb4' is set in the database.php
CHARACTER SET=utf8mb4 is set on every table on the mysql database
the original project had
'charset' => 'latin1' and showed problems, changing to utf8mb4 solved the problem on OSX(the test local host) but the problem didnt fix when uploaded to the server(linux)
The data comes bad before even trying to show:
$example= DB::connection('mysql_db')->select("SELECT * FROM test)
Log::error($example);
log shows the query gets me the question mark character
Found a similar issue in:
scandir issue with accents in Linux work fine in OSX
But i am not sure if the answer applies only to scan dir, or to any php-linux accent issue
I had the same issue and I manage to fix it by adding these lines in the config/database.php
'mysql' => [
[...]
'charset' => 'latin1',
'collation' => 'latin1_swedish_ci',
[...]
],
Finally made it work, had to set
'mysql' => [
[...]
'charset' => 'utf8mb4',
'collation' => 'utf8mb4_unicode_ci',
[...]
],
On config/database.php
and set CHARACTER SET=utf8mb4; at the end of each Table in my sql file(works setting it to the whole database but another documents needed the database to be default latin1).
OSX overwrites the charset of the DB with the one you set on the connection, but apparently linux doesn't allow the same.
The same solution didnt work with utf8
Inserting UTF-8 encoded string into UTF-8 encoded table gives incorrect string value.
PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9D\x84\x8E i...' for column 'body_value' at row 1: INSERT INTO
I have a ๐ character, in a string that mb_detect_encoding claims is UTF-8 encoded.
I try to insert this string into a MySQL table, which is defined as (among other things) DEFAULT CHARSET=utf8
Edit: Drupal always does SET NAMES utf8 with optional COLLATE (atleast when talking to MySQL).
Edit 2: Some more details that appear to be relevant. I grab some text from a PostgreSQL database. I stick it onto an object, use mb_detect_encoding to verify that it's UTF-8, and persist the object to the database, using node_save. So while there is an HTTP request that triggers the import, the data does not come from the browser.
Edit 3: Data is denormalized over two tables:
SELECT character_set_name FROM information_schema.COLUMNS C WHERE table_schema = "[database]" AND table_name IN ("field_data_body", "field_revision_body") AND column_name = "body_value";
>+--------------------+
| character_set_name |
+--------------------+
| utf8 |
| utf8 |
+--------------------+
Edit 4: Is it possible that the character is "to new"? I'm more than a little fuzzy on the relationship between unicode and UTF-8, but this wikipedia article, implies that the character was standardized very recently.
I don't understand how that can fail with "Incorrect string value".
๐ (U+1D10E) is a character Unicode found outside the BMP (Basic Multilingual Plane) (above U+FFFF) and thus can't be represented in UTF-8 in 3 bytes. MySQL charset utf8 only accepts UTF-8 characters if they can be represented in 3 bytes. If you need to store this in MySQL, you'll need to use MySQL charset utf8mb4. You'll need MySQL 5.5.3 or later. You can use ALTER TABLE to change the character set without much problem; since it needs more space to store the characters, a couple issues show up that may require you to reduce string size. See http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-upgrading.html .
to solve this issue, first you change your database field to utf8m4b charset. For example:
ALTER TABLE `tb_name` CHANGE `field_name` `field_name` VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL;
then in your db connection, set driver_options for it to utf8mb4. For example, if you use PDO
$db = new PDO('mysql:host=localhost;dbname=testdb;charset=utf8mb4', 'username', 'password');
or in zend framework 1.2
$dbParam = array('host' => 'localhost', 'username' => 'db_user_name',
'password' => 'password', 'dbname' => 'db_name',
'driver_options' => array(
'1002' => "SET NAMES 'utf8mb4'",
'12' => 0 //this is not necessary
)
);
In your PDO connecton, set the charset.
new PDO('mysql:host=localhost;dbname=the_db;charset=utf8mb4', $user, $password);
I fixed the error:
SQLSTATE[HY000]: General error: 1366 Incorrect string value ......
with this method:
I use utf8mb4_unicode_ci for database
Set utf8mb4_unicode_ci for all tables
Set longblog datatype for column (not text, longtext.... you need big datatype to store 4 bytes of your content)
It is okay now.
If you use laravel, continue to edit config/database.php
'charset' => 'utf8mb4',
'collation' => 'utf8mb4_unicode_ci',
If you use function strtolower, replace it with mb_strtolower
Notice: you have to put <meta charset="utf-8"> on your head tag
This question already has answers here:
UTF-8 all the way through
(13 answers)
Closed 6 years ago.
My charset in the database is set to utf8_unicode_ci, all files encoded in UTF8 (without BOM).
Here is my php code:
<?php
require_once("./includes/config.php");
$article = new Article();
$fields = array(
'status' => '0',
'title' => 'ืืืื ืช"ื ืืืืคืช ืืืจืืคื ืืคืขื ื-9',
'shorttitle' => 'ืืืื ืช"ื ืืืืคืช ืืืจืืคื',
'priority' => '1',
'type' => '1',
'category' => '2',
'template' => '68',
'author' => '1',
'date' => date("Y-m-d H:i"),
'lastupdate' => date("Y-m-d H:i"),
'preview' => 'ืืื ืืื ืืื',
'content' => 'ืขืื ืงืฆืช ืืื ืืื ืืื',
'tags' => 'ืืืื ืช"ื,ืืืจืืืื,ืืืืคืืช ืืืจืืคื',
'comments' => '1'
);
$article->set($fields);
$article->save();
for some reason, the Hebrew characters appear like this in phpmyadmin:
รลพรโบรโรโข รยช"ร รรลรโขรยครยช รรโขรยจรโขรยครโ รโรยครยขร รโ-9
Database connection code:
<?php
final class Database
{
protected $fields;
protected $con;
public function __construct($host = "", $name = "", $username = "", $password = "")
{
if ($host == "")
{
global $config;
$this->fields = array(
'dbhost' => $config['Database']['host'],
'dbname' => $config['Database']['name'],
'dbusername' => $config['Database']['username'],
'dbpassword' => $config['Database']['password']
);
$this->con = new mysqli($this->fields['dbhost'], $this->fields['dbusername'], $this->fields['dbpassword'], $this->fields['dbname']);
if ($this->con->connect_errno > 0)
die("<b>Database connection error:</b> ".$this->con->connect_error);
}
else
{
$this->con = new mysqli($host, $username, $password, $name);
if ($this->con->connect_errno > 0)
die("<b>Database connection error:</b> ".$this->con->connect_error);
}
}
Any ideas why?
You have set the database's and file's character set to UTF-8, but the data transfer between PHP and the database also needs to be set correctly.
You can do this using set_charset:
Sets the default character set to be used when sending data from and to the database server.
Add the following as last statement of your Database constructor:
$this->con->set_charset("utf8");
This will not fix the issue for the data that is already in the database, but for new data written to the database you should notice the difference.
If you decide to rebuild your database, then please consider using the superior utf8mb4 character set, as described in the MySql docs:
The character set named utf8 uses a maximum of three bytes per character and contains only BMP characters. As of MySQL 5.5.3, the utf8mb4 character set uses a maximum of four bytes per character supports supplemental characters:
For a BMP character, utf8 and utf8mb4 have identical storage characteristics: same code values, same encoding, same length.
For a supplementary character, utf8 cannot store the character at all, while utf8mb4 requires four bytes to store it. Since utf8 cannot store the character at all, you do not have any supplementary characters in utf8 columns and you need not worry about converting characters or losing data when upgrading utf8 data from older versions of MySQL.
utf8mb4 is a superset of utf8
It's important that your entire line code has the same charset to avoid issues where characters displays incorrectly.
There are a few settings that needs to be properly defined and I'd strongly recommend UTF-8, as this has most letters you would need (Hebrew), but also supports a wide variety of other charsets too (Scandinavian, Greek, Arabic).
Here's a little list of things that has to be set to a specific charset.
Headers
Setting the charset in both HTML and PHP headers to UTF-8
PHP: header('Content-Type: text/html; charset=utf-8');
(PHP headers has to be placed before any kind output (echo, whitespace, HTML))
HTML: <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
(HTML-headers are placed within the <head> / </head> tag)
Connection
You also need to specify the charset in the connection itself (placed directly after creating the connection).
$this->con->set_charset("utf8");
Database and tables
Your database and all its tables has to be set to UTF-8. Note that charset is not exactly the same as collation (see this post).
You can do that by running the queries below once for each database and tables (for example in phpMyAdmin)
ALTER DATABASE yourDatabase CHARACTER SET utf8 COLLATE utf8_unicode_ci;
ALTER TABLE yourTable CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
Other
Some specific functions have the attribute of a specific charset, and if you are using such functions, it should be specified there as well
It may be that you already have values in your database that are not encoded with UTF-8. Updating them manually could be a pain and could consume a lot of time. Should this be the case, you could use something like ForceUTF8 and loop through your databases, updating the fields with that function.
Should you follow all of the pointers above, chances are your problem will be solved. If not, you can take a look at this StackOverflow post: UTF-8 all the way through.
In my database user stores data in Japanese characters. When I query those data I did not get the data in Japanese characters but I got รฅโฆยฅรฅล โบรฃยชรฃโ in stead. I don't know how to solve this. Below is my database configuration.
Yii::$app->components = [
'db' => [
'class' => 'yii\db\Connection',
'dsn' => 'mysql:host=localhost;dbname=my_database',
'username' => '****',
'password' => '****',
'charset' => 'utf8',
],
];
If I remove 'charset' => 'utf8', I got ??????? in stead of strange characters above or Japanese characters. I work with Yii framework 2 and with its ActiveRecord. Anyone knows any solutions?
I'm willing to bet that your database or your tables have the wrong charset/Collation.
Check databases:
SELECT * FROM information_schema.SCHEMATA
Check tables:
SHOW FULL COLUMNS FROM table_name;
It's possible to convert existing data (from latin1 to utf8 for example) but not super-easy. I would say you'll have an much easier time just changing the default charset "How to make MySQL handle UTF-8 properly" and create new database/tables rather than convert existing data.