I have an issue and I've wasted enough time on it already.
I'm writing the name of the files in a directory into my database, but some files have special characters in them, like accents, and I keep getting this error when I try to write.
SQLSTATE[ HY000 ]: General error: 1366 Incorrect string value.
I've already tried changing the charset of my database and its tables but it's still the same.
Thank you for your help.
Related
I have a $string in PHP. It doesn't matter where this comes from (it's from incoming e-mails); the important part is that sometimes, it's not valid UTF-8 according to PostgreSQL, but is valid according to PHP.
I explicitly set both mb_internal_encoding('UTF-8') and mb_regex_encoding('UTF-8'). I explicitly set client_encoding to 'UTF8' (yes, it wants it without the '-') when making the PostgreSQL database connection. I have verified over and over again that the PG database itself uses UTF8. In short: everything on my system uses UTF-8 encoding.
Details: PHP 7.4.1. PG 11.5. Windows 10. (The same thing has happened for years and years for many versions of PHP/PG/Windows.)
Before trying to INSERT a record containing $string, I make the following integrity/safety check to avoid errors:
function string_is_valid_UTF8($string)
{
if (!mb_check_encoding($string, 'UTF-8'))
return false;
else
return true;
}
if (string_is_valid_UTF8($string))
// Proceed to INSERT it into the database since PHP says it's valid UTF-8 data.
Occasionally -- NOT every time! -- PostgreSQL barks at this, even though it has been checked by PHP to be valid UTF-8. It spits out/logs this error:
pg_query_params(): Query failed: ERROR: invalid byte sequence for encoding "UTF8"
I don't get it. The only explanation I can see is that PostgreSQL and PHP have different ideas of what is valid UTF-8. This problem has bugged me for years and I just never seem to get it resolved. Again and again, sometimes with weeks or months in between, some external data coming into my system causes this issue. In spite of my check!
Is there something I can tell PostgreSQL to make it handle this differently? I don't want that error to be logged. It's really, really annoying.
At this point, I'm utterly baffled as to how this can happen. Is PHP or PostgreSQL at wrong? Considering how many times I've dealt with this and trying to solve it by a zillion different methods, it doesn't seem reasonable that it's me doing something wrong at this point.
My database deals with Chinese and Japanese characters.
When I insert rows on the phpMyAdmin command box, it works beautifully.
But problem occurs when I set up a query input on my website, the query is fetched by a php file, like:
$query = $_POST['query'];
$result = $dbc->query($query);
The non-English characters just become rubbish in database, like
ID column1
1666 ä½ å¥½å•Šå•Šå•Š
I checked that the php file receives the characters fine, the problem should come from Mysql. All charset is utf-8.
I am new to mysql, please let me know if you may need more info.
Thank you in advance.
column's collation set to utf8_general_ci
More information : https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html
Add this line into your config.inc.php in PHPMyAdmin
$cfg['DefaultCharset'] = 'utf8';
$cfg['DefaultConnectionCollation'] = 'utf8_general_ci';
Since phpMyAdmin works expected, it sounds like your application is at fault here.
Perhaps you could clarify what you mean when you wrote "php file receives the characters fine" -- how did you determine that? The way I read it, you can create the proper characters from your application and they're displayed correctly in the database through phpMyAdmin, but when you try to display it through your application it shows gibberish. In that case it seems the portion of the application that retrieves and displays the data is at fault.
Regardless, there are many answers here about having gibberish characters, the best place to start is UTF-8 all the way through and you could also read more at Special characters in PHP / MySQL or Problem with PHP and Mysql UTF-8 (Special Character). Finally, even thugh it's off the Stackoverflow network, the phpMyAdmin wiki covers this pretty well also.
As a hint, most of the time I see this issue it's caused by not issuing SET NAMES (or using an equivalent function) when setting up the MySQL connection from your PHP script.
I know there were plenty of questions like this but I am creating the new one because to my point of view it is specific to each situation.
So, my page is displayed in UTF-8 format. The data is taken from mySQL that has utf8_unicode_ci collation. The data I am displaying is the string - 1 Bröllops-Festkläder.
There are some unicode characters in here and they should display fine but they do not. On my page these are just a bunch of hieroglyphs.
Now, the interesting situation:
I am using phpMyAdmin to keep track of what is happening in the database. The website has the ability to import CSV documents containing customer data and modify each customer individually. If I import CSV document containing these characters they are written to the database, readable in phpMyAdmin and not readable on my page. If I use my script to modify the customer information and I type those characters from the browser, the it is vice versa - they are readable on the page and they are not readable in phpMyAdmin, so clearly the encoding is different. I spent ages figuring out the right combination and I could not.
UPDATE: Deceze posted a link below that I copy here to make it more noticeable. I am sure this will save hours and days to many people facing similar issues - Handling Unicode Front to Back in a Web App
There're couple of things that got involved here. If your database encoding is fine and html encoding is fine and you still see artefact, it's most likely your db connection is not using same encoding, thus leading to data corruption. If you connect by hand, you can easily enforce utf encoding, by doing query SET NAMES UTF8 as very first thing after you connect() to your database. It is sufficient to do this only once per connection.
EDIT: one important note though - depending on how you put your data to the DB, your database content may require fixing as it can be corrupted if you put it via broken connection. So, if anyone is facing the same issue - once you set all things up, ensure you are checking on fresh data set, or you may still see things incorrectly, even all is now fine.
I have designed a web form using PHP as a server-side script that should insert data into a Sybase ASE database table using ODBC Functions.
When I fill the form fields with English word and ASCII Characters it works ok and saves data in the database but when I use Arabic and extended ASCII, UTF, or Unicode characters, I get the following error message:-
Warning: odbc_exec()
[function.odbc-exec]: SQL error:
[INTERSOLV][ODBC SQL Server
driver][SQL Server]Error converting
client characters into server's
character set. Some character(s) could
not be converted. , SQL state S1000 in
SQLExecDirect in
C:\wamp\www\website1\webpage1.php on
line 111
Is this because I have the settings on my database (or server) set incorrectly? In which case what should I change? and how do I change it?
Or, do I need to some function(s) to convert the extended ASCII characters? In which case, have the necessary functions already been writeen? and where can I find them?
How I can solve that problem?
Thanks for your help.
I discovered the solution of the problem afterwards which was that I am not saving the php file in the correct encoding using Save As and choosing the right encoding.
I would like to share that so that it may be helpful for others who had the same problem.
Sometimes We face problems that seems to be very difficult but their solutions are very simple as we discovered afterwards.
I'm trying to create a script that copies data from an old legacy mysql database into my new utf-8 formatted database.
One particular field is causing me trouble, its a latin1 field - and one record has the following in it:
!-#$%'&*£¥
When the update is performed, I get the following error message:
Zend_Db_Statement_Exception: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xA3' for column 'messageContent' at row 1 in C:\xampp\htdocs\portal\
library\Zend\Db\Statement\Pdo.php on line 234
Now, I'd like to strip characters that cause a problem, but other than checking ord($character) <= 126 I don't know what to do. I'm worried that my method might lose valuable data.
I am using MySQL 5.1.43 on Windows 7 - with PHP 5.3.1
Are you just trying to convert your database? If so read this link that should give you some tips on converting it, and there is a script there that might work.
Found a problem with a function deeply nested in some classes that was the problem....