I'm trying to create a script that copies data from an old legacy mysql database into my new utf-8 formatted database.
One particular field is causing me trouble, its a latin1 field - and one record has the following in it:
!-#$%'&*£¥
When the update is performed, I get the following error message:
Zend_Db_Statement_Exception: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xA3' for column 'messageContent' at row 1 in C:\xampp\htdocs\portal\
library\Zend\Db\Statement\Pdo.php on line 234
Now, I'd like to strip characters that cause a problem, but other than checking ord($character) <= 126 I don't know what to do. I'm worried that my method might lose valuable data.
I am using MySQL 5.1.43 on Windows 7 - with PHP 5.3.1
Are you just trying to convert your database? If so read this link that should give you some tips on converting it, and there is a script there that might work.
Found a problem with a function deeply nested in some classes that was the problem....
Related
I have an issue and I've wasted enough time on it already.
I'm writing the name of the files in a directory into my database, but some files have special characters in them, like accents, and I keep getting this error when I try to write.
SQLSTATE[ HY000 ]: General error: 1366 Incorrect string value.
I've already tried changing the charset of my database and its tables but it's still the same.
Thank you for your help.
Thanks in advance for any help!
I have several different mysql columns (which are imported from a product feed) that I'm trying to convert into one to work with a custom plugin for Opencart.
The format of the MYSQL entry for the plugin is as follows...
a:11:{i:13;s:3:"964";i:14;s:9:"Automatic";i:15;s:3:"FWD";i:16;s:15:"Gas I6 4.2L/256";i:17;s:5:"Black";i:18;s:4:"Gray";i:19;s:12:"Some City, ST";i:20;s:2:"15";i:21;s:2:"21";i:22;s:6:"128365";i:23;s:17:"2XXXXG3XXXXH5XXXX";}
As you can see, the "s:X" represents the character count for each entry. I have 11 different columns with the info I'm trying to convert to this format, but without an accurate character count the plugin throws an error. So i'm trying to use a combo of sql/php to get an accurate character count that I can use to make this work. I have a handle on everything else.
SO!
My question is...how do I assign the character count to a variable? I did some googling and I found this...
SELECT CHAR_LENGTH('TABLENAME');
I realize there's more to it, just wondering how to set it as a variable.
AND
Not terribly important, but does anyone recognize the format? I've never seen this before!
Serialized PHP Array
Not sure if this will help anyone in the future since this is pretty specific, but based on the info I got from jraede (thanks alot!). I was able to load all 11 columns into a PHP array and serialize, then upload to DB. WAY easier than trying to create 11 variables for character count!
Original Question
mysql-server-6.0.10
I have this problem, I'm using the COMPRESS function to update a field with a BASE64 encoded value. The "updated" base64 string is only slightly different than the value that is currently in the database but it IS different. The only part is different is the last few bytes of the string. The problem is that mysql is seeing these values as the SAME and therefore skipping the update. It returns without error letting my program thing it did what it was told but then you go back and the data is unchagned.
For example, the first one is base64 string that was previously INSERTED to the db, the second one is what I'm trying to overwrite with:
YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO2k6MjM0MjM0MjM0MzI0MDAwO319
YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO3M6MTU6IjIzNDIzNDIzNDMyNDExMSI7fX0=
I'm running this query:
UPDATE TABLE SET fieldname=COMPRESS('YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO3M6MTU6IjIzNDIzNDIzNDMyNDExMSI7fX0=') WHERE id = 'SOMEID';
Mysql comes back with 0 rows affected.
I can reproduce this in an SQL Query window via PHPMyAdmin so I know its not a problem with code.
How can I force this to just update regardless if the strings BEGIN the same way, since it obviously inst checking the entire string? Right now my workaround is to clear the present data with one query, then run another query to update against nothing. This is inefficient though and I'm not happy with it.
First question update
I actually found out this is a result of a PHP bug with JSON_NUMERIC_CHECK!
The value was getting saved in the database properly (couldnt tell because of the compression) but when my script to go back was loading the data out of the database via JSON_ENCODE the values were getting mangled by JSON_NUMERIC_CHECK.
See the bug report I filed:
https://bugs.php.net/bug.php?id=60111
Second question update and solution
So it really turns out I was way ahead of my self on this. I was using JSON_NUMERIC_CHECK incorrectly. I was using it as a blanket solution to ensure all numeric values would be, well, numeric. The flaw in this logic is that JSON_NUMERIC_CHECK's purpose in life is to attempt to convert all numeric strings to integers, in 32-bit OS environment there is a limit to how many digits an int val can be and thus using the exponential expression as it should!
It comes down to the fact that in my system, there is a possibility of a [tracking number] could actually be an extremely long string of just numbers. JSON_NUMERIC_CHECK was doing exactly what it was suppose to do by converting this to an exponentially expressed int val.
The actual fix to this was to not use JSON_NUMERIC_CHECK and to make sure that my var types were correct where I actually needed int vals before creating the object that is later passed to the front end via JSON.
Phew, a mouthfull, but i figured if anyone else runs into this problem it might be helpful. Chao!
I'm sorry guys, nevermind, the problem is that value i'm supplying is being treated as INT instead of STRING and its over the max size of an INT so its resetting to the MAX INT and therefore being the same string.
Sometimes just thinking through it enough to post a problem on here is enough to help me figure it out :)
UPDATE:
I actually found out this is a result of a PHP bug with JSON_NUMERIC_CHECK!
The value was getting saved in the database properly (couldnt tell because of the compression) but when my script to go back was loading the data out of the database via JSON_ENCODE the values were getting mangled by JSON_NUMERIC_CHECK.
UPDATE:
I'm actually just way ahead of my self. using JSON_NUMERIC_CHECK incorrectly. My bad!
I'm unsure if this is a php-, filemaker-, mysql- or an odbc driver issue.
For security reasons the input fields of my current php webform convert special characters into hex codes, (for example: # becomes ' ) This hex code is saved in the database and will also be shown in Filemaker11 as the hex code. This is not what i want.
How can I make sure the special character will be displayed as it should be?
The other way round (from filemaker to db), no conversion will be done on inserting the special characters.
How can I make sure everything will be consistent?
Kind regards,
Jeroen
FileMaker is just showing the data stored in MySQL. If you pull up the DB in a tool like PhpMyAdmin you should see that the varchar contains the encoding as well. Since FMP is looking at it simply as a text field, it shows the encoding that was stored. If you wanted to decode in FMP you could show a calc field of the varchar that has a custom function to decode the text. (but that won't allow for updating the data..) You could also try a trigger on record load to decode the data in the fields so that you can properly view/edit.
Solved it! It appeared that I had to add an extra line to my PHP script.
after setting up the connection, php needs to tell mysql what the encoding needs to be. This can be done with the following line:
$dbh->query("SET NAMES 'utf8'");
Thanks for the effort guys!
This: ' type of encoding is not done automatically by the browser. Something is doing it. Normally you do it only on output not on input.
You can use html_entity_decode() to undo it. But I strongly suggest you figure out why it's happening in the first place.
I have designed a web form using PHP as a server-side script that should insert data into a Sybase ASE database table using ODBC Functions.
When I fill the form fields with English word and ASCII Characters it works ok and saves data in the database but when I use Arabic and extended ASCII, UTF, or Unicode characters, I get the following error message:-
Warning: odbc_exec()
[function.odbc-exec]: SQL error:
[INTERSOLV][ODBC SQL Server
driver][SQL Server]Error converting
client characters into server's
character set. Some character(s) could
not be converted. , SQL state S1000 in
SQLExecDirect in
C:\wamp\www\website1\webpage1.php on
line 111
Is this because I have the settings on my database (or server) set incorrectly? In which case what should I change? and how do I change it?
Or, do I need to some function(s) to convert the extended ASCII characters? In which case, have the necessary functions already been writeen? and where can I find them?
How I can solve that problem?
Thanks for your help.
I discovered the solution of the problem afterwards which was that I am not saving the php file in the correct encoding using Save As and choosing the right encoding.
I would like to share that so that it may be helpful for others who had the same problem.
Sometimes We face problems that seems to be very difficult but their solutions are very simple as we discovered afterwards.