We're running into a weird edge case where we are trying to store a json blob in a table in our database, and that blob needs to be able to contain the \ character. So a user were to enter in \test it needs to come back as exactly that, but instead its coming back as a tab followed by "est"
As far as I can tell, whats happening is that when a user enters and submits "\test" it gets evaluated into "\ \test" (remove the space, cant put two backslashes in here and have it display right?) by the client and then entered into the table. I can verify that in the SQL that gets called against the table there are two backslashes. When I look at it in the table after this step its back to "\test". When the client loads it up again it gets evaluated into a tab followed by an "est".
We are under the impression that the second backslash is necessary so that the first backslash will get escaped and not evaluated but maybe that is what is causing issues? I sort of assume when the query runs one of the backslashes gets escaped anyway but I'm not really sure what to do about that. Is there something with out our database is handling backslashes that we need to be looking out for? Is there a way to handle this that we haven't considered?
It's a Postgres database if that's helpful. I'd say I'm beginner to intermediate on this sort of thing, I'm looking through documentation but if anyone can even point me in the right direction that would be very helpful.
Postgres version as far as i can tell through Amazon Aws is 9.3
EDIT
I think ive tracked this issue down to a line in our php backend that I don't really understand. I'm looking at the documentation for that now and will mark this as answered since I've verified that its not an issue with SQL.
Blockquote A backslash as - by default - no special meaning in SQL. This might be caused by whatever code is processing those values (and sending them to the database). See here for an online example: rextester.com/QLLYG57275 – a_horse_with_no_name
I'm accepting this as the answer as I've verified that the issue is with out backend code constructing the SQL, and not how the SQL is being handled on the database end.
Related
Simple question, probably takes a rather simple answer, but I haven't found anything online about it. I'm working on an Android app that needs to access a mySQL cloud database. To this end, I'm creating a PHP webservice to interface the two (as well as provide an extra layer of protection for the mySQl credentials). The issue is that I"m using a POST method to send variables to the queries to tell them what operations to do, such as:
$db->query("DROP TABLE {$_POST[table]}");
The issue is that when I try to send values for something like an INSERT statement, I need to use quotes around the strings. Unfortunately, the UTF-8 encoding turns all the quotes into escaped quotes. What should I do to fix this?
Use stripslash(%str) to remove the slashes. I don't know where the slashes are coming from, but putting this into code to remove the slashes works.
I've got a weird scenario going on here.
On my localhost running WAMP server (Apache, MySQL, PHP), I've created a webpage that displays a list of messages from a table in my database.
Let's say the DIV container was called: #message-list
This list gets displayed correctly (when the page is launched, PHP renders the whole page).
The HTML markup that PHP echoes-out works just fine.
The MySQL database lookup therefore also works! Great.
Now...
With a bit of AJAX and jQuery magic, I've created a form to add more messages on-the-fly, by sending a POST request to a PHP scripts that uses the SAME underlying code that generated the initial #message-list DIV.
The AJAX'ed PHP script does two things:
add a record for the new message from the user;
echoes the list (which should be updated with the new message now);
When the AJAX response comes back to the browser, the JavaScript side replaces the old list with the new #message-list content.
Now this... partially works.
What goes wrong - On one given page, it seems some of my previous posts are somehow "corrupting" something inside the AJAX request on the PHP side, resulting a null response (basically no HTML code gets generated to replace the #message-list DIV tag).
On some other pages though, the AJAX response works fine.
So my question is:
Is it possible that some String data in the Database breaks the execution of my PHP script because of some invalid character, badly encoded, or a quote / double-quote?
I've tried using PHP's htmlentities() and mysql_real_escape_string() functions to solve this, but one of my pages still doesn't properly refreshes the list after the AJAX response is received.
Could it just be that I just need to cleanup / sanitize the existing content in my table?
If so, is there any easy script / query I can use to do this?
Thanks!
EDIT #1:
MySQL version = 5.5.24-log
By using mysql_client_encoding, this shows "latin1" (ah HA! That may be the issue then!)
In PHP, using the mysql_... methods (such as mysql_connect, mysql_select_db, mysql_query, etc.);
Sample of database table with possible issue:
http://pastebin.com/PjLVmXEF
By the looks of it, many developers say PDO is recommended. I'll give that a shot and see if all errors vanish. Thanks a lot for your help so far everyone!
EDIT #2:
My current solution has been this:
I've used these queries to modify my database and the table with the encoding problem:
// SQL queries:
ALTER DATABASE timegrasp charset=utf8;
ALTER TABLE tg_messages CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
Second, I noticed some characters in a specific record was not displaying correctly (an  around some double-quoted sentences). So I manually backspaced and reinserted the double-quote in MySQL Query Browser to be sure it was completely gone.
On the PHP side, I only encode the messages on the way "in" to the database, with this:
$htmlConverted = htmlentities( $pMessage, ENT_COMPAT | ENT_HTML401, "UTF-8" );
return mysql_real_escape_string( $htmlConverted );
And make sure I begin my MySQL connection with this:
mysql_set_charset("utf8", $DB_LINK);
Then, I can just read the String directly from the table without any decoding / conversion.
Finally, to test this - I copied the same message from the source (a Skype chat with my client) which had the special characters, pasted it in my web form, and now it works fine! :)
I'm not certain all the steps and parameters above were necessary, but this is what helped fix my issue.
It would be good to know for future reference though if any of this is bad practice or common "dont's" mistake when handling special characters in MySQL tables.
The PHP json_encode function refuses to process strings that are invalid in UTF-8: it returns null. If you don't set a character encoding for the database connection it's possible that some other encoding is being used -- not UTF-8 -- and the data you pass to this function is in fact not valid UTF-8.
If you mentioned details such as which database API and which connection parameters you are using I could give further advice...
One of the systems I look after receives daily csv files from a third party. Recently the integration stopped working. I managed to pinpoint root the cause - the curly apostrophe. Once replaced with a regular one import files got processed successfully.
Third party system that generates these files is one of the Microsoft products, MS Access I think. System that receive and process these files is written in PHP on MySql database.
And here are the questions I would like to ask here:
- is this PHP or MySql that does not 'like' this character?
- are there any more characters of this kind that php/mysql would have issue processing?
I am not sure what "curly apostrophe" is, but if it's usual apostrophe (like in "it's") - yes, it does, as it is used as a string delimiter in MySQL.
If it's some other charecter - then it doesn't have any special meaning in PHP.
Anyway, you have to always format SQL query parts according to their role, to avoid whatever syntax errors.
Please refer to my earlier answer on the matter: In PHP when submitting strings to the database should I take care of illegal characters
Original Question
mysql-server-6.0.10
I have this problem, I'm using the COMPRESS function to update a field with a BASE64 encoded value. The "updated" base64 string is only slightly different than the value that is currently in the database but it IS different. The only part is different is the last few bytes of the string. The problem is that mysql is seeing these values as the SAME and therefore skipping the update. It returns without error letting my program thing it did what it was told but then you go back and the data is unchagned.
For example, the first one is base64 string that was previously INSERTED to the db, the second one is what I'm trying to overwrite with:
YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO2k6MjM0MjM0MjM0MzI0MDAwO319
YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO3M6MTU6IjIzNDIzNDIzNDMyNDExMSI7fX0=
I'm running this query:
UPDATE TABLE SET fieldname=COMPRESS('YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO3M6MTU6IjIzNDIzNDIzNDMyNDExMSI7fX0=') WHERE id = 'SOMEID';
Mysql comes back with 0 rows affected.
I can reproduce this in an SQL Query window via PHPMyAdmin so I know its not a problem with code.
How can I force this to just update regardless if the strings BEGIN the same way, since it obviously inst checking the entire string? Right now my workaround is to clear the present data with one query, then run another query to update against nothing. This is inefficient though and I'm not happy with it.
First question update
I actually found out this is a result of a PHP bug with JSON_NUMERIC_CHECK!
The value was getting saved in the database properly (couldnt tell because of the compression) but when my script to go back was loading the data out of the database via JSON_ENCODE the values were getting mangled by JSON_NUMERIC_CHECK.
See the bug report I filed:
https://bugs.php.net/bug.php?id=60111
Second question update and solution
So it really turns out I was way ahead of my self on this. I was using JSON_NUMERIC_CHECK incorrectly. I was using it as a blanket solution to ensure all numeric values would be, well, numeric. The flaw in this logic is that JSON_NUMERIC_CHECK's purpose in life is to attempt to convert all numeric strings to integers, in 32-bit OS environment there is a limit to how many digits an int val can be and thus using the exponential expression as it should!
It comes down to the fact that in my system, there is a possibility of a [tracking number] could actually be an extremely long string of just numbers. JSON_NUMERIC_CHECK was doing exactly what it was suppose to do by converting this to an exponentially expressed int val.
The actual fix to this was to not use JSON_NUMERIC_CHECK and to make sure that my var types were correct where I actually needed int vals before creating the object that is later passed to the front end via JSON.
Phew, a mouthfull, but i figured if anyone else runs into this problem it might be helpful. Chao!
I'm sorry guys, nevermind, the problem is that value i'm supplying is being treated as INT instead of STRING and its over the max size of an INT so its resetting to the MAX INT and therefore being the same string.
Sometimes just thinking through it enough to post a problem on here is enough to help me figure it out :)
UPDATE:
I actually found out this is a result of a PHP bug with JSON_NUMERIC_CHECK!
The value was getting saved in the database properly (couldnt tell because of the compression) but when my script to go back was loading the data out of the database via JSON_ENCODE the values were getting mangled by JSON_NUMERIC_CHECK.
UPDATE:
I'm actually just way ahead of my self. using JSON_NUMERIC_CHECK incorrectly. My bad!
I've been building a database input form. There are three fields for notes. They were all build at the same time. They have the same logic & class system - but one of them is returning with escape marks when I update the record, e.g. I enter
1
2
3
and the updated record returns
1\r\n2\r\n3
I'm confused as the other return the text as formatted. Can you sugggest why this is happening?
As Konrad suggested your question lacks clarity. Are they both on the same page? If not, do both pages have the same DOCTYPE?
The simplest solution to this is to handle it regardless, addslashes (or mysql_real_escape_string) for INSERTs and UPDATEs and stripslashes for SELECTs.
The other scenario you have to handle is a POST to a page that has errors (e.g repopulated with post values). You must make sure that the POST value is the 'stripped' version of the slashes else you will double up on the addslashes; so only addslashes when all your post validation and verification has been completed.
As it is, your question lacks a lot of crucial information. Obviously, if everything is done identically, the data will be the same as well. Additionally, you probably don't mean that there are escape marks in your data (or do you?). In that case, you must have explicitly called a function to convert line breaks into a string containing the characters '\' and 'n'.
Lacking any code, I assume that you display your data differently; once using a normal echo and once using something like var_dump.