I'm currently using this function to obfuscate a bit the field values in MySQL and protect it from direct dumping. It all works good and values are stored correctly, but what happens when i try to store a multibyte string?
Here's an example, let's try to encode the string álex:
<?
$v = xorencode('álex');
// step 1 - encode
echo $v."\n";
// step 2 - decode
echo xorencode($v);
?>
Works good, i see some obfuscated string first time, and then i see álex again. Now if i try to save it in a VARCHAR field in a MySQL table, and then select it - i no longer have a utf string, instead it gets returned as gllex.
Note, MySQL tables and fields collations are utf8_general_ci, files are UTF-8, and i SET NAMES utf8 after connecting. Any workaround to this?
Thanks
Related
I have one php form where i used to enter data to database(phpmyadmin), and i used SELECT query to display all values in database to view in php form.
Also i have another PHP file which i used to create JSON from the same db table.
Here when i enter foreign languages like "Experiența personală:" the value getting saved in DB is "ExperienÈ›a personală: " but when i use select query to display this in same php form it coming correctly "Experiența personală:". So the db is correct and now am using following php code to create JSON
<?php
$servername = "localhost";
$username = "root";
$password = "root";
$dbname = "aaps";
// Create connection
$con=mysqli_connect($servername,$username,$password,$dbname);
// Check connection
mysqli_set_charset($con, 'utf8');
//echo "connected";
$rslt=mysqli_query($con,"SELECT * FROM offers");
while($row=mysqli_fetch_assoc($rslt))
{
$taxi[] = array('code'=> $row["code"], 'name'=> $row["name"],'contact'=> $row["contact"], 'url'=> $row["url"], 'details'=> $row["details"]);
}
header("Content-type: application/json; charset=utf-8");
echo json_encode($taxi);
?>
and JSON looks like
[{"code":"CT1","name":"Experien\u00c8\u203aa personal\u00c4\u0192: ","contact":"4535623643","url":"images\/offers\/event-logo-8.jpg","details":"Experien\u00c8\u203aa personal\u00c4\u0192: jerhbehwgrh 234234 hjfhjerg#$%$#%#4"},{"code":"ewrw","name":"Experien\u00c8\u203aa personal\u00c4\u0192: ","contact":"ewfew","url":"","details":"eExperien\u00c8\u203aa personal\u00c4\u0192: Experien\u00c8\u203aa personal\u00c4\u0192: Experien\u00c8\u203aa personal\u00c4\u0192: "},{"code":"Experien\u00c8\u203aa personal\u00c4\u0192: ","name":"Experien\u00c8\u203aa personal\u00c4\u0192: ","contact":"","url":"","details":"Experien\u00c8\u203aa personal\u00c4\u0192: "}]
In this "\u00c8\u203aa" this is wrong it supposed to be "\u021b" (t).
So pho used to creating JSON making this issue.
But am unable to find exactly why its coming like this . please help
Avoid Unicode -- note the extra argument:
json_encode($s, JSON_UNESCAPED_UNICODE)
Don't use utf8_encode/decode.
ă turning into ă is Mojibake. It probably means that
The bytes you have in the client are correctly encoded in utf8 (good).
You connected with SET NAMES latin1 (or set_charset('latin1') or ...), probably by default. (It should have been utf8.)
The column in the tables may or may not have been CHARACTER SET utf8, but it should have been that.
If you need to fix for the data it takes a "2-step ALTER", something like
ALTER TABLE Tbl MODIFY COLUMN col VARBINARY(...) ...;
ALTER TABLE Tbl MODIFY COLUMN col VARCHAR(...) ... CHARACTER SET utf8 ...;
Before making any changes, do
SELECT col, HEX(col) FROM tbl WHERE ...
With that, ă should show hex of C483. If you see C384C692, you have "double-encoding", which is messier to fix.
Depending on the version of MySql in the database, it may not be using the full utf-8 set, as stated in the documentation:
The ucs2 and utf8 character sets do not support supplementary characters that lie outside the BMP. Characters outside the BMP compare as REPLACEMENT CHARACTER and convert to '?' when converted to a Unicode character set.
This, however, is not likely to be related to your problem. I would try a couple of different things and see if it solves your problem.
use SET NAMES utf-8
You can read more about that here
use utf8_encode() when inserting data to the database, and utf8_decode() when extracting. That way, you don't have to worry about MySql manipulating the unicode characters. Documentation
I have a site developed in codeigniter where I'd like to insert a record inside my database in a table with utf8 fields.
The problem is that when I insert something inside the table I see that:
�"�h�t�t�p�:�/�/�I�m�a�g�e�1�.�u�r�l�f�o�r�i�m�a�g�e�s�.�c�o�m�/�D�e�f�a�u�l�t�/�8�8�0�4�/�2�3�6�9�2�2�6�2�-�1�8�4�3�3�8�5�2�6�6�.�j�p�g�"�
There are many more characters. The real string is a simple path. I don't know the format of the string because it is from an external server.
This is my query to insert record. I take the string from xml and if I print it inside the page I see the correct string. The problem occurs whenever I check inside the database:
foreach($img->childNodes as $node){
$data = array(
'image'=>$node->getAttribute('path'),
);
$this->db->insert('hotel_images',$data);
}
That data is not UTF-8. It is UCS-2 or UTF-16. UCS-2 is a subset of UTF-16, so treating it as UTF-16 should do the trick.
You can convert it using iconv.
$data = iconv("UTF-16", "UTF-8", $data);
I tried to insert Hebrew a text value into a column,
But it changes the value to Gibberish.
An example of that:
mssql_query ("UPDATE TABLE SET COLUMON = N'בדיקה'");
As you can assume, It changes the value of the column, But the value changed to ????? and if I try to do it from Query Analyser it works fine.
My column's collation is HEBREW_CI_AS. How can I fix this?
You need to specify collation preperty for the string in the INSERT statement you are using. Also the string you are inserting should be of UNICODE datatype - use N prefix for that.
INSERT INTO MEMB_INFO (User, Pass, Name) VALUES ('Joni', '123456', N'גוני דף' COLLATE HEBREW_CI_AS)
Check that PHP variable can handle unicode characters. Otherwise it will be PHP that turns your string into question marks.
You may check out SQL Server drivers for PHP.
And Unicode Character Properties from PHP doicumentation.
Some resources on PHP and unicode:
http://www.sitepoint.com/bringing-unicode-to-php-with-portable-utf8/
http://php.net/manual/en/function.utf8-encode.php
http://allseeing-i.com/How-to-setup-your-PHP-site-to-use-UTF8
http://www.yiiframework.com/wiki/16/how-to-set-up-unicode/
http://pageconfig.com/post/portable-utf8
I solve this problem if someone else has this problem here is my way to fix that:
Create a new database for this specific table or else tables for your web.
Set Hebrew_CI_AS as collation (everyone to what he created).
In your PHP code use mb_convert_encoding() function for SELECT and INSERT.
I generate a mysql query from a form with a free text search field.
Something like:
SELECT ... FROM ... WHERE 'something' LIKE '%SEARCH%'
All this works fine and returns the valid rows when the search does not contain any special characters, like the danish characters ÆØÅ.
When these letters ARE used, the query returns no results, all though when i take the generated query string and plug it into phpMyAdmin i get exacly the result i want.
Thanks
add this line of code in your connection file...
mysql_set_charset("utf8", $db);
it is better for you to encode your data to UTF-8 before you pass it into query...
I'm not using Danish but Czech but I think there are the same (at least by UTF8 implementation) - you must keep in mind used encoding (original server script, data tables self and also your database connection handler).
I think you have an encoding problem, maybe phpMyAdmin is using a different client encoding than your other client. SET NAMES 'encoding' should just do what you need, I think.
Also we can use PHP variable and convert it before the select operation (supposing data base is ISOO-8859-2) .
Example:
// word with special characters
$search='kötészeti';
// conversion to ISO
$search=iconv("UTF-8","ISO-8859-2", $search);
// create search condition
$condition="SELECT ... FROM ... WHERE 'something' LIKE '%$search%'";
// apply query
mysql_query($condition);
I am trying to match a md5 has (generated through php) to its original value in a SQLExpress database.
I am using the following function in my SQL query
master.sys.fn_varbintohexsubstring(0, HASHBYTES('MD5', 'ID'), 1, 0)
Where 'ID' is the field in the database.
However they both seem to return different values for the md5 hash. I have been using '12290' as a static value to test this.
php md5() returns: 0bd81786a8ec6ae9b22cbb3cb4d88179
The following SQL Statement returns the same output:
DECLARE #password VARCHAR(255)
SET #password = master.sys.fn_varbintohexsubstring(0, HASHBYTES('MD5', '12290'), 1, 0)
SELECT #password
Yet when I run the following statement from the table:
SELECT ID, master.sys.fn_varbintohexsubstring(0, HASHBYTES('MD5', CONVERT(NVARCHAR(255), ID)), 1, 0) AS temp
FROM Clients
ORDER BY ID ASC
The 'temp' value matching to the 'ID' value of 12290 returns: 1867dce5f1ee1ddb46ff0ccd1fc58e03
Any help on the matter would be much appreciated!
Thanks
Python helped me to help you.
>>> from hashlib import md5
>>> md5('1\x002\x002\x009\x000\x00').digest().encode('hex')
'1867dce5f1ee1ddb46ff0ccd1fc58e03'
NVARCHAR is Unicode type and it seems from the above experiment that '12990' is stored as UTF-16LE in your database: '1\02\09\09\00\0'.
Assuming that the data encoding in the PHP is UTF-8 data and you don't want to change the existing data in the database, this is how you can fix your PHP script:
<?php
$password = '12290';
$hash = md5(mb_convert_encoding($password, 'UTF-16LE', 'UTF-8')) . "\n";
echo $hash;
?>
Output:
susam#swift:~$ php utf16le-hash.php
1867dce5f1ee1ddb46ff0ccd1fc58e03
In case the data in PHP is in some other encoding such as ASCII, ISO-8859-1, etc. you can change the third argument to mb_convert_encoding accordingly. The list of all supported encodings is available at: http://www.php.net/manual/en/mbstring.supported-encodings.php
Also, see http://www.php.net/manual/en/function.mb-convert-encoding.php
I don't have SQL server to test this on, but the CONVERT command might be creating the NVARCHAR with 240-odd trailing blanks (as you have specified NVARCHAR(255))
Try setting the NVARCHAR to the length of the ID to test:
ARE #password VARCHAR(255)
SET #password = master.sys.fn_varbintohexsubstring(0, HASHBYTES('MD5', CONVERT(NVARCHAR(5), '12290')), 1, 0)
SELECT #password
Try with different lengths in the CONVERT - is there any difference?
One of two things is most likely the problem:
Either the ID column in that row has a value not exactly equal to '12290' (e.g. extra whitespace)
Or the CONVERT function produces such a value
In any case, a standard debugging approach would be to use an SQL query to SELECT the string lengths of that ID field and the return value of CONVERT; if either is not equal to 5, you found the error.
Alternatively you can perform a dump of the table in question including data, and look at the generated INSERT statement to see what the database says the value in that column is.