Our website was developed with a meta tag set to...
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
This works fine for M-dashes and special quotes, etc. However, I have an issue when data has been entered into a CMS component that stores data in MySQL. The MySQL collation is set to UTF8_swedish_ci (I read this is ok and must have been a default when it was set up in phpMySqlAdmin).
The problem I now get is when I output info from the DB to the page, the characters are
utf8 encoded, so I run them through the uft8_decode() php function. I thought this would fix the incompatibility, but what I'm getting isn't what I expect.
When I look at the data in the DB in a text field (again through phpMySqlAdmin) it looks like this...
This – That
When I view it on the screen it looks like...
This ? That
I know I can try to find/replace a bunch of these in the DB or the text, but I'm hoping there's an easier way to do this programatically.
Thanks,
Don
Update:
Still have an issue that htmlentities() unfortunately doesn't fix.
I have text in a file like this: we’ve (special '). My MySQL collation is "latin1_swedish_ci" (the default). If I change the header or meta to either iso/utf one or the other breaks. W/ utf-8 the (’) a black diamond but the db content is fine. With iso, the inline content is ok, but the content from the db has all kinds of  and other chars. Tried changing MySQL collation to utf-8 but didn't see a difference.
I'm about resolved to changing the items manually. Thanks for any other suggestions.
If your data in the database is UTF8, you'll need to run this query after you connect to MySQL:
SET NAMES UTF8
Assuming that you were able to set the encoding properly in your database, my recommended approach here is to:
Make sure that the Content-Type header has been set properly by the
server. This can be done in php by using the header function.
header('Content-Type: text/html; charset=iso-8859-1');
Note that this takes precedence and is the easiest information to get since user agents do not have to parse it.
Set the meta tag in the HTML file.
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
For further readings, refer to:
http://www.joelonsoftware.com/articles/Unicode.html
http://www.webstandards.org/learn/articles/askw3c/dec2002/
My guess would be that despite you meta tag, the web server sends a header which sets the charset to UTF-8. However, the easiest way to fix these kinds of problems is usually to escape non-ASCII-characters to HTML entities.
Related
I have built a CMS that allows HTML to be stored in a database. It all started off very simple. I displayed the HTML in a textarea using htmlspecialchars to prevent it from breaking the form. Then saved it back using html_specialchars_decode. It all seemed to work fine until someone pasted some HTML into the system instead of typing. At this point it stored fine but lost most of the whitespace which meant all the lovely indentation had to be done from scratch.
To fix it, I tried specifying everything in utf-8 encoding because any attempt to fiddle with it seemed to produce invalid characters.
I specify utf-8 in the PHP header
header('Content-Type: text/html; charset=utf-8');
I specify utf-8 in my HTML page
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
I specify utf-8 in the HTML form
<form accept-charset="utf-8"
Then I read the posted value (basically) like this:
$Val = $_POST[$SafeFieldName];
My understanding was that PHP did everything in utf-8 so I am a bit surprised at this stage that I get gobbledegook - unless I now do this:
$Val = utf8_decode($Val);
So, at this stage - it works - sort of. I loose all my lovely indentation but not all of my white space. It's as if there are some non utf8 chars being stripped out. Weirdly I'm using Chrome but in Firefox, it seems fine
I think I'm just tying myself in knots now. Any elegant suggestions? I need to get to the bottom of this as opposed to just hack it to get it to work.
The connection to the DB and the DB tables itself should support UTF-8. Make sure that your table's collation is utf8_general_ci and that all string fields within the table also have the utf8_general_ci collation.
The DB connection should be UTF-8 as well:
mysql_set_charset('utf8');
See http://akrabat.com/php/utf8-php-and-mysql/ for more info.
Update: some report that
mysql_query('SET NAMES utf8');
is required sometimes as well!
If making your tables and connection UTF-8 is not possible, you could of course save the HTML as BASE64 encoded data, and decode it back when you retrieve it from the DB again.
Check your DataBase connection encodin, and check DataBase table field encoding where you store HTML.
Maybe there encoding is different from UTF-8
If this is an issue in and out of MySQL (as you suggested in the title) then you need to make sure the columns and tables are UTF8-BIN and put mysql_set_charset('utf8'); after opening the connection to MySQL.
Sorted - and the answer is really embarrassing - but you never know, some day someone may need this :)
I noticed that it worked differently (but still fairly rubbish) in Firefox so I had a look at my style sheet and found this:
white-space: nowrap;
Someone (me) must have put that in there to try to get horizontal scrolling working in some browser. Without that, the HTML makes it all the way to the DB and back again.
My only other question was why did I need this since the whole thing should have been arriving in utf8
$Val = utf8_decode($Val);
Magically - now I don't need it.
I created a webpage, which require to store information in non English character. I created db and table using PHPMyAdmin with charset utf8 and collate utf8_general_ci, and I enter some sample data into that table, in PHPMyAdmin browse area i can see the non english characters, but when i query those table data using php, it is showing like question mark(????? like that)
Try adding this in your header:
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
See this page for different ways of doing this on different content types.
How do you determine that the result of your query isn't properly formed utf8? IF you display it on a website, please remember to have a properly formed meta-construct in your head-section, something like this:
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
or, another way to format it in php...
<?
...somecode...
header ('Content-type: text/html; charset=utf-8');
...morecode...
Otherwise your browser might try to display it using a different character encoding.
All in all PHP handles utf8-characters pretty good. Ofc some things have to be considered, here's a good summary about this.
If the rest of your DB is not in UTF8 and just this table is in UTF8 you can either change the charset in the head of your HTML or use utf8_decode() to decode the utf8 string.
http://us.php.net/manual/en/function.utf8-decode.php
I have turkish character problem in mysql database when adding content with tinymce from admin panel.
Charset is:
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-9"" />
How can I solve this?
Thanks in advance
Make sure the table in MySQL is also defined as having charset ISO-8859-9.
There's not enough information to say what your problem is, but in general you need the same character set in your HTML page (text/html;charset), PHP's connection to the database (mysql_set_charset), and MySQL's CREATE TABLE ... DEFAULT CHARACTER SET (if you just CREATE TABLE it will end up in Latin-1 which you probably don't want. Plus you would need to make sure not to use htmlentities-without-charset-argument on output (use htmlspecialchars instead).
See eg. this answer for more detail. That's talking about using UTF-8 for the encoding, but the same applies if you substitute ISO-8859-9 all the way through. (Although unless there's a good reason not to, you should really be using UTF-8.)
well I had a similar problem with my turkish site.
My tables were in latin5_turkish_ci an the charset of the php page were latin5
there was no problem when I submitted the content via php to database, all characters were being saved correctly
but when I tried to submit the content via jquery post method then any turkish character was being saved correctly to database
and php iconv function solved my problem
Recently I worked in a project in where I need to display japanese text which are come from database. I already use
meta http-equiv="Content-Type" content="text/html; charset=utf-8"
It help to display the static text. But when it come from database it display "??????????" type text.
How can I solve this kind of problem?
Is the database charset UTF8 too? Is the connection charset UTF8? Seems like the data gets converted to ISO-8859-1 somewhere along the way.
Without more information, it is hard to find exactly what the problem is. What DBMS are you using? MySQL? PostgreSQL? Either way, I'm pretty sure either your database and/or your connection isn't using UTF8.
You can change your connection charset by using one of the following functions:
mysql_set_charset('UTF-8');
pg_set_client_encoding('UTF-8');
I'm still learning the ropes with PHP & MySQL and I know I'm doing something wrong here with how character sets are set up, but can't quite figure out from reading here and on the web what I should do.
I have a standard LAMP installation with PHP 5, MySQL 5. I set everything up with the defaults. When some of my users input comments to our database some characters show up incorrectly - mostly apostrophes and em dashes at the moment. In MySQL apostrostrophes show up as ’. They display on the page this way also (I'm using htmlentities to output user comments).
In phpMyAdmin it says my MySQL Charset is UTF8-Unicode.
In my database my tables are all set up with the default Latin1-Swedish-ci.
My web pages all have meta http-equiv="Content-Type" content="text/html; charset=utf-8"
When I look at the site's http headers I see: Content-Type: text/html
Like a newbie, I hadn't considered character sets at all until things started looking odd on some of my pages. So does it make most sense for me to convert everything to utf-8 and will this affect my PHP code? Or should I try to get it all into Latin? And do I have to go into the database and replace these odd codes, or will they magically display once I set up the charsets properly? All the fiddling I've done so far hasn't helped (I set the http headers to utf-8, and also tried latin).
If you really want to understand these issues, I would start by reading this article at mysql.com. Basically, you want every piece of the puzzle to expect UTF-8 unicode. On the PHP side, you want to do something like:
<?php header("Content-type: text/html; charset=utf-8");?>
<html>
<head>
<meta http-equiv="Content-type" value="text/html; charset=utf-8">
And when you run your insert queries you want to make sure both the table's character encoding and the encoding that you're running the queries in are UTF-8. You can accomplish the latter by running the query SET NAMES utf8 right before you run an insert query.
http://www.phpwact.org/php/i18n/charsets
That site gave me a lot of good advice on how to make everything play nice in UTF-8.
I also recomened switching from htmlentities to htmlspecialchars as it is more UTF friendly.
The main point is to make sure everything is talking the same language. Your database, your database connection, your PHP, your page is in utf8 (should have a meta tag and a header saying so).
Sorry for not understanding all of your question. But when part of the question is "UTF-8 or not?", the answer is: "UTF-8, of course!"
You definitely want to sort things out now rather than later. One of the most important programming rules is not to keep going with a bad idea - don't dig yourself in any deeper!
As latin1 and utf-8 are compatible, you can convert your tables to use utf-8 without manipulating the data contained by hand. MySQL will sort this part out for you.
It's then important to check that everything is speaking utf-8. Set the http headers in apache or use a meta tag - this says to a browser that the HTML output is utf-8.
With this in mind, you need to make sure all of the data you send really is utf-8! Configure your IDE to save php/html files as utf-8. Finally make sure that PHP is using a utf-8 connection to MySQL - issue this query after connecting:
SET NAMES 'utf-8';