I'm trying to track down a bug with some random characters appearing when saving data to our database. So far my travels have indicated that it's a character encoding issue.
I've swapped the collation on the dev to utf8_general_ci and it doesn't seem to have made a difference to the system, but I'm still unsure as to the full implications of changing collation.
I have been poking around in here, http://dev.mysql.com/doc/refman/5.0/en/charset-charsets.html and it's still not entirely clear.
I've also updated the page with the form on to include a utf-8 <meta /> tag.
The background of the issue is that posting a £ from the form, when it runs through our SQLBuilder class, it's passed through mysql_real_escape_string (deprecated I know :() and ends up in the database, and subsequently generated config files as £
As I understand it, the collation is a way for the database to compare characters, but I'm still not totally sure.
Ninja edit
Web application, posting an HTML form through a PHP class, into a MySQL DB
I usually do a mysql_query("set names utf8"); immediately after connecting to the database.
Related
I am transfering the database from one server to another server using phpmyadmin. I successfully transfered it but having issue with swedish characters. I can see the swedish characters are displaying properly within the tables but in php pages it is wrong seems like double encoded or any other problem. Can anyone help?
The problem could be lying in different parts. Welcome to the world of Unicode!
Make sure the collation for the columns in MySQL is utf8_* (I personally prefer utf8_bin).
Make sure the PHP page is telling the client that the contents are encoded with UTF8. That can/should be done in two ways:
Set the following header: header('Content-Type: text/html; charset=utf-8');
In your HTML <head> add the correct meta tag: <meta charset="utf-8">
(note: while in theory it's not strictly necessary to do both, as they're equivalent for the client, it's better to be redundant!)
Make sure the connection with MySQL uses UTF8. That can be done by executing a simple query right after the connection to the database: SET NAMES 'utf8' (e.g. mysqli_query("SET NAMES 'utf8'"); alter it accordingly if you're using PDO or the MySQLi OOP APIs).
Bonus: if you're using UTF8 in your PHP script, make sure you treat everything in an Unicode-safe way. So, prefer using mb_* functions to manipulate strings, use the u flag with preg_* functions, etc. And remember than UTF8 characters are variable in the number of bytes they use, from 1 to 4!
I have same setting for my both website only problem is with database after transfering it to an other server. Encoding of pages are same on both sites.
you can check it here
http://www.abswheels.se
http://www.dackis.se/abs/
you can see the difference. any sugguestions??
also everything is fine inside the database. I dont know why when i fetch the data with special character from database it has a problem. you can see the title bar of both website. everything is same on client side. same encoding same setting
I have a mysql database storing some fields in greek characters. In my html I have charset=utf-8 and my database columns are defined with encoding utf_general_ci. But I was not setting the connection encoding so far. As a result I have a database that doesn't display the greek characters well, but when reading back in PHP, it all shows well.
Now I try to do this the right way, so I added also in my database functions.
$mysqli->set_charset("utf8");
This works great for new entries.
But for existing entries, the problem is that when I read data in PHP, it comes garbled, since now the connection encoding has changed.
Is there a way to fix my data and make them useful again? I can continue working my old way, but I know it's wrong and can cause me more problems in the future.
I solved this issue as follows:
in a PHP script, retrieve the information as I do now, i.e without setting the connection. This way the mistake will be inverted and corrected and in your php file you will have the characters in the correct utf-8 format.
in the same PHP script, write back the information with setting the connection to utf-8
at this point the correct characters are in the database
I changed all my read/write functions of your site to use the utf-8 from now on
We imported a website from another server to our server. The code and database is 100% the same.
But the text on the website seems to have a wrong encoding.
Example:
In the database the word "Australië" is "AustraliĂŤ" while on the website its shown as Australi??.
I can fix the ?? with adding mysql_set_charset("utf8",$this->db); after the database connection.
But then its shown like in the database like "AustraliĂŤ" wich is incorrect. I tried different encodings in apache, after database and in meta tags.
The easiest way would be to change the data in the database but there is to much data in it to do this.
Anyone has a solution for this problem? Have been searching and trying a lot off things for hours.
You could try to:
set the MySQL connection collation to uft8_general_ci in the database
run SET NAMES 'utf8' and SET COLLATION_CONNECTION=utf8_unicode_ci in your PHP files
make sure all your PHP files are saved with UTF-8 encoding and do not feature a BOM
make sure the cells in your table are utf8_general_ci
make sure that MySQL charset is UTF-8 Unicode (utf8)
This is what I have. With this setup I see all characters in the database (phpMyAdmin) as they really appear on the website itself.
I have encountered a similar issue when I had a mismatch of encodings, i.e. I was saving data to a UTF-8 database by a ISO-8859-1 encoded site...
Hope this helps you.
I have built a CMS that allows HTML to be stored in a database. It all started off very simple. I displayed the HTML in a textarea using htmlspecialchars to prevent it from breaking the form. Then saved it back using html_specialchars_decode. It all seemed to work fine until someone pasted some HTML into the system instead of typing. At this point it stored fine but lost most of the whitespace which meant all the lovely indentation had to be done from scratch.
To fix it, I tried specifying everything in utf-8 encoding because any attempt to fiddle with it seemed to produce invalid characters.
I specify utf-8 in the PHP header
header('Content-Type: text/html; charset=utf-8');
I specify utf-8 in my HTML page
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
I specify utf-8 in the HTML form
<form accept-charset="utf-8"
Then I read the posted value (basically) like this:
$Val = $_POST[$SafeFieldName];
My understanding was that PHP did everything in utf-8 so I am a bit surprised at this stage that I get gobbledegook - unless I now do this:
$Val = utf8_decode($Val);
So, at this stage - it works - sort of. I loose all my lovely indentation but not all of my white space. It's as if there are some non utf8 chars being stripped out. Weirdly I'm using Chrome but in Firefox, it seems fine
I think I'm just tying myself in knots now. Any elegant suggestions? I need to get to the bottom of this as opposed to just hack it to get it to work.
The connection to the DB and the DB tables itself should support UTF-8. Make sure that your table's collation is utf8_general_ci and that all string fields within the table also have the utf8_general_ci collation.
The DB connection should be UTF-8 as well:
mysql_set_charset('utf8');
See http://akrabat.com/php/utf8-php-and-mysql/ for more info.
Update: some report that
mysql_query('SET NAMES utf8');
is required sometimes as well!
If making your tables and connection UTF-8 is not possible, you could of course save the HTML as BASE64 encoded data, and decode it back when you retrieve it from the DB again.
Check your DataBase connection encodin, and check DataBase table field encoding where you store HTML.
Maybe there encoding is different from UTF-8
If this is an issue in and out of MySQL (as you suggested in the title) then you need to make sure the columns and tables are UTF8-BIN and put mysql_set_charset('utf8'); after opening the connection to MySQL.
Sorted - and the answer is really embarrassing - but you never know, some day someone may need this :)
I noticed that it worked differently (but still fairly rubbish) in Firefox so I had a look at my style sheet and found this:
white-space: nowrap;
Someone (me) must have put that in there to try to get horizontal scrolling working in some browser. Without that, the HTML makes it all the way to the DB and back again.
My only other question was why did I need this since the whole thing should have been arriving in utf8
$Val = utf8_decode($Val);
Magically - now I don't need it.
I have had no end of problems trying to do what I thought would be relatively simple:
I need to have a form which can accept user input text in a mix of English an other languages, some multi-byte (ie Japanese, Korean, etc), and this gets processed by php and is stored (safely, avoiding SQL injection) in a mysql database. It also needs to be accessed from the database, processed, and used on-screen.
I have it set up fine for Latin chars but when I add a mix of Latin andmulti-byte chars it turns garbled.
I have tried to do my homework but just am banging my head against a wall now.
Magic quotes is off, I have tried using utf8_encode/decode, htmlentities, addslashes/stripslashes, and (in mysql) both "utf8_general_ci" and "utf8_unicode_ci" for the field in the table.
Part of the problem is that there are so many places where I could be messing it up that I'm not sure where to begin solving the problem.
Thanks very much for any and all help with this. Ideally, if someone has working php code examples and/or knows the right mysql table format, that would be fantastic.
Here is a laundry list of things to check are in UTF8 mode:
MySQL table encoding. You seem to have already done this.
MySQL connection encoding. Do SHOW STATUS LIKE 'char%' and you will see what MySQL is using. You need character_set_client, character_set_connection and character_set_results set to utf8 which can easily set in your application by doing SET NAMES 'utf8' at the start of all connections. This is the one most people forget to check, IME.
If you use them, your CLI and terminal settings. In bash, this means LANG=(something).UTF-8.
Your source code (this is not usually a problem unless you have UTF8 constant text).
The page encoding. You seem to have this one right, too, but your browsers debug tools can help a lot.
Once you get all this right, all you will need in your app is mysql_real_escape_string().
Oh and it is (sadly) possible to successfully store correctly encoded UTf8 text in a column with the wrong encoding type or from a connection with the wrong encoding type. And it can come back "correctly", too. Until you fix all the bits that aren't UTF8, at which point it breaks.
I don't think you have any practical alternatives to UTF-8. You're going to have to track down where the encoding and/or decoding breaks. Start by checking whether you can round-trip multi-language text to the data base from the mysql command line, or perhaps through phpmyadmin. Track down and eliminate problems at that level. Then move out one more level by simulating input to your php and examining the output, again dealing with any problems. Finally add browsers into the mix.
First you need to check if you can add multi-language text to your database directly. If its possible you can do it in your application
Are you serializing any data by chance? PHPs serialize function has some issue when serializing non-english characters.
Everything you do should be utf-8 encoded.
One thing you could try is to json_encode() the data when putting it into the database and json_decoding() it when it's retrieved.
The problem was caused by my not having the default char set in the php.ini file, and (possibly) not having set the char set in the mysql table (in PhpMyAdmin, via the Operations tab).
Setting the default char set to "utf-8" fixed it. Thanks for the help!!
Check your database connection settings. It also needs to support UTF-8.