I am working with a large number of foreign characters in SQL and I cannot seem to get them to save correctly;
Júlio César
Luisão
What is the best way to deal with special characters in mysql is it a setting in the data formats or do you format the data in php before you add it to the database?
Thanks
You need to set the encoding of your database (and tables) to utf8, set the connection encoding to utf8, and use utf8 as the encoding of the php files themselves.
That looks like "double encoding". It is discussed in
http://mysql.rjweb.org/doc.php/charcoll
Related
I built a PHP web page and some of the characters are like this
⇾ ....
â€....
etc
Please help me on how to eliminate this
If you're not using UTF-8 characterset, I would try to change the encoding of the database to avoid these characters:
How to convert an entire MySQL database characterset and collation to UTF-8?
I'm inserting contents into table as bulk upload format (.csv).
In my CSV some columns include special characters but server can't insert special characters included rows into db and stops execution.
How can I insert these values?
I'm trying:
$field1= addslashes(trim($data[0]));
and I'm using mysqli_real_escape_string, it's working but it omits all special characters from the fields...
$field2= mysqli_real_escape_string(trim($data[1]));
Do not call any conversion routines. Simply declare that the .csv file is encoded latin1. Do that with this clause in the LOAD DATA statement:
CHARACTER SET latin1
You have not said what charset the column/table is. Please provide SHOW CREATE TABLE. The column can be either latin1 or utf8.
Also, use mysqli_set_charset to establish which charset your client code is using. (It does not need to be the same as the column/table.)
You say "omits special characters". Do you mean that data is stored into the table(s), but with strings truncated at the special characters? See "Truncated text" in Trouble with utf8 characters; what I see is not what I stored .
You say "stops execution". That seems unlikely; please provide more details, such as the SQL that caused the termination, whether other commands come be run after it, whether you needed to start (not "restart") mysqld again.
You say "special characters"; let's see some examples. You used escaping to get $field variables, but what did you then do with those variables?
MySQL should do the conversion automatically, as long as you tell it what encoding your data is using. With the mysqli extension you use the mysqli::set_charset() function.
If you app is not using ISO-8859-1 and only your current data set does, you need to declare the application encoding and convert data yourself. You have at least three functions:
mb_convert_encoding()
iconv()
utf8_encode()
In either case, you'll get SQL errors if incompatible conversions happen. It shouldn't be the case if databases uses an encoding that contains all ISO-8859-1 characters (such as UTF-8).
Note: get rid of addslashes(), it serves no purpose at all and can only corrupt your data.
I am confused! Recently my webhotel updated php and now my old tables render special characters differently (wrongly).
Both my tables and my input/output-php-pages are set to utf-8 and since this update, also the inputs from php are treated differently; now my special characters are being utf-8-encoded as they enter the database. So since this change, when I review tables within phpMyAdmin, the old inserts have the original (non-encoded) special characters - the new posts have utf-8-encoded charcters (also special).
So what I would like to do is rewrite input and output to insert and show non-encoded characters - but I am not sure if this is possible without skipping utf-8 entirely (in php and mySQL). But is there an utf-8- way to submit non-encoded characters?
AND - perhaps more fundamentally - I need to understand what the possible downsides are. I am using Danish characters in and out and I'm not going to use any other language (for this project). So if it IS possible to insert and output non-encoded characters using utf-8 - am I then going to have unexpected/destructive issues?
I have read a lot of posts regarding php/mySQL/special characters but I haven't seen this angle on the issue yet. Hope I am not duplicating
I hope not because it has been working very nicely until the update.
Even if you are using only Danish characters, you may as well go utf8 all the way.
There are many places where the encoding needs to be stated:
The at the top of the html
The columns in the database (column CHARACTER SET defaults from table, which defaults from database)
The encoding in your PHP code.
When you CREATE TABLE, tack on DEFAULT CHARACTER SET utf8. If you have existing tables, without that, speak up; we may need to deal with them.
If you want Danish collation, the specify COLLATION utf8_danish_ci, too. Then (if I recall correctly), aa will sort after z.
(The default is utf8_general_ci, which won't do that sorting.)
Figure out what encoding you have (or can get) in your php code. If you have some text with accents in it, do this:
$hex = unpack('H*', $text);
echo implode('', $hex)
If you have utf8, å will be C3A5, for latin1 it will be E5.
Regardless of what encoding in in the tables, you must call set_charset('utf8') or set_charset('latin1') depending on what encoding is in the data in PHP. MySQL will gladly transcode between latin1 and utf8 as things are passed between PHP and MySQL. For different APIs:
⚈ mysql: mysql_set_charset('utf8');
⚈ mysqli: $mysqli_obj->set_charset('utf8');
⚈ PDO: $db = new PDO('dblib:host=host;dbname=db;charset=UTF-8', $user, $pwd);
For much more info, see http://mysql.rjweb.org/doc.php/charcoll .
I have MYSQL database collation set to latin1_swedish_ci but my site uses encoding windows-1256. This means the data inside tables is encoded with windows-1256.
What is the correct way to convert my database tables/fields and data to utf-8 using iconv or any other library?
First, you need to verify that the data in the table(s) is really latin1. Could you do SELECT HEX(col), col ... to see what it looks like.
Depending on whether it is latin1 encoding or utf8 encoding (or something else) will determine what steps to perform. (If you do these steps without knowing, you could make things worse.)
These references give you the next steps:
http://dev.mysql.com/doc/refman/5.0/en/alter-table.html and/or
http://mysql.rjweb.org/doc.php/charcoll
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
UTF-8 all the way through
okay, this is stupid that I can't figure it out.
Mysql database is set to utf8_general_ci collation. The field i'm having problems with is longtext type.
characters added to the database as é or other accented characters are returning as �.
I run the output through stripslashes and i've tried both with and without html_entity_decode but can find no change in the output. What am I doing wrong?
Cheers
What character encoding does the string have that you try to insert? If it is in ISO-8859-1 you can use the PHP function utf8_encode() to encode it to UTF-8 before inserting it into the database.
http://php.net/manual/en/function.utf8-encode.php
Getting encoding right is really tricky - there are too many layers:
Browser
Page
PHP
MySQL
The SQL command "SET CHARSET utf8" from PHP will ensure that the client side (PHP) will get the data in utf8, no matter how they are stored in the database. Of course, they need to be stored correctly first.
DDL definition vs. real data
Encoding defined for a table/column doesn't really mean that the data are in that encoding. If you happened to have a table defined as utf8 but stored as differtent encoding, then MySQL will treat them as utf8 and you're in trouble. Which means you have to fix this first.
What to check
You need to check in what encoding the data flow at each layer.
Check HTTP headers, headers.
Check what's really sent in body of the request.
Don't forget that MySQL has encoding almost everywhere:
Database
Tables
Columns
Server as a whole
Client
Make sure that there's the right one everywhere.
Conversion
If you receive data in e.g. windows-1250, and want to store in utf-8, then use this SQL before storing:
SET NAMES 'cp1250';
If you have data in DB as windows-1250 and want to retreive utf8, use:
SET CHARSET 'utf8';
Last note:
Don't rely on too "smart" tools to show the data. E.g. phpMyAdmin does (was doing when I was using it) encoding really bad. And it goes through all the layers so it's hard to find out. Also, Internet Explorer had really stupid behavior of "guessing" the encoding based on weird rules. Use simple editors where you can switch encoding. Also, I recommend MySQL Workbench.