I'm getting an XML API response, parsing the data I need, but want to store full XML response in MySQL for add'l data use later.
I intially stored the XML in a BLOB, but found special characters sometimes in values break the INSERT.
So, I first convert XML with htmlentities into a BLOB to keep original API response data integrity. Is this a good way to do it, or is there a better method?
Properly escaped, any data can be INSERTed into a BLOB. htmlentities is not the way. For the mysqli API, use mysqli::real_escape_string. For PDO use binding.
If you are running on Windows, be sure to fetch the data in binary mode, else CR/LF may get "fixed" during the read.
In theory XML should be UTF-8, so simply putting into TEXT CHARACTER SET utf8mb4 should work instead of BLOB.
If all else fails, consider converting (in the client) the blob into Base64. It will be 4/3 bigger.
Related
I used mysql to store the picture and it is stored by longblob.
Now I need use json the transfer the data of longblob.
The json_endcode($data) returns null.
How to do it?
You could try to serialize image data to base64, but it seems to be a bad idea, since images can be really big. You better store it on ftp server and write just links to images to database
Please check this question - Binary Data in JSON String. Something better than Base64
binary data can be encoded into base64 otherwise JSON does not support it
The JSON format natively doesn't support binary data. The binary data
has to be escaped so that it can be placed into a string element (i.e.
zero or more Unicode chars in double quotes using backslash escapes)
in JSON.
I'm storing data in a MySQL database that may have some special characters. I'm wondering how to store it so that these characters are preserved if they're either output to HTML via PHP OR via JavaScript, e.g. createTextNode.
For example, the division symbol (÷) has the html code ÷, and when I store it as that it shows up fine when put directly into HTML by PHP, but when I pull it into JavaScript using $.getJSON and then insert it with createTextNode it shows up looking like ÷.
I also tried storing the symbol in the SQL directly, but my understanding is that the column would need to be changed from VARCHAR to NVARCHAR and that would cause a performance hit that doesn't seem necessary.
Given that I can modify the SQL, the PHP, or the JavaScript, is there an easy fix here? Maybe a way to unescape the HTML entity in JavaScript?
As answered by Yogesh, you should switch your collation of the DB to utf8_general_ci
So there's probably two things going on:
JSON escapes special characters.
Somewhere, something in your code flow is URL encoding the strings too.
So you just need to decode the string in your JavaScript, or you need to find what part of your code is URL encoding those strings and fix it.
I have a object with some data being posted to a php script from a javascript. This data is coming from a for, so the user will will out a form, when they hit enter an Ajax script will take the form database, put it into an object then post it to my php encoding it with JSON.
Now i'm new to stuff like JSON so im not 100% sure what its doing, i've read a bit online and my conclusion is that it encodes the data with a sort of universal encoding that all programming languages have..... Maybe not the best description of it but hey. So this isn't doing the same thing as escaping the data is it?
Any, before i process the data and put in into a database i want to escape it but im not sure of the best way to go about this? is there a way i could escape the hole object? Any tips or tricks for this sort of thing?
No, jsons are't escaped at all.
On PHP side you could use json_decode to retrive a decoded form of the data then you will access all of the original object property as a PHP array.
JSON indeed is "universal" in that it is UTF-8 by default, and multi-byte sequences are escaped in \uuuuu format.
However, if you want to store the entire JSON object in the database as-is, that doesn't take away the need to escape the entire string before you insert it into the database, using the string escaping function of your database (or parametrized queries if your library supports them).
Encoding something in JSON is no the same as escaping it. Basically JSON is a serialization format based on Javascript object literals. So on the php side you need to:
Decode the json to PHP
Validate the vales
Escape the values
Insert the values into your db
After you decode the JSON you will be left with an array (see json_decode, and pass true as the second arg to make sure its an array and not a mic of stdObject and arrays).
So then you can pull out the data you ned and escape it you normally would any array passed to you through $_POST before insertion.
I am having a little issue with storing mcrypt_module_open('rijndael-256','','ofb',''); in a MySQL db.
When it inserts the encrypted data into the MySQL db it looks like this ˜9ÏÏd‰.
It should look like this
÷`¥¶Œ"¼¦q…ËoÇ
I am wondering if I have to do something to get it to work?
Use a blob field type for storing binary data (BLOB, VARBINARY, BINARY)
If you're not doing this already: escape your values with the proper methods if you're using them directly in a SQL-statement. Or even better: use query parameters/prepared statements.
As a last resort you could just encode your data with either base64_encode or bin2hex.
If you want to display binary data on the console or in the browser (even for debugging purpose) use one of those encodings too. Otherwise you might not see the actual data because the browser might not display your binary correctly.
In general, it might be a good idea to base64 encode and decode binary data like this. See Best way to use PHP to encrypt and decrypt passwords? .
Have you tried to Collation of your table that your character supports.
The characters '÷`¥¶Œ"¼¦q…ËoÇ' looks like UTF-8 or someother charset, find charset of your characters and update table Collation based your charset
When encoding newline of textarea before storing into mysql using PHP with rawurlencode function encodes newline as %0D%0A.
For Example:
textarea text entered by user:
a
b
encoding using rawurlencode and store into database will store value as a%0D%0Ab
When retrieving from database and decoding using rawurldecode does not work and code gives error. How to overcome this situation and what is the best way to store and retrieve and display textarea values.
can you first encode this textarea string using base64_encode and then perform a base64_decode on the same, if the above does not work for you.
If the textarea does not contain URLs, you should rather use base64_encode then rawurlencode and then store as normal.
You simply should not use rawurlencode for escaping data for your database.
Each target format has it's own escaping method which in general terms makes sure it is stored/display/transferred safely from one place to another, and it doesn't need decoding at the other end.
For instance:
displaying text in HTML, use htmlentities or htmlspecialchars
storing in database, use mysqli_real_escape_string, pg_escape_string, etc...
transferring variablename, use urlencode
transferring variablecontent, use rawurlencode
etc...
You should notice that decoding these things is often done by the browser/database. So no data is actually stored escaped. And decoding doesn't need te be done by your code.
The problem is probably because you escape a sequence with rawurlencode, but your database expected the escaped format for the specific brand of database. And de-escaped it using that assumption, which was wrong, which messed up your string.
Conclusion: find out what brand database you are using, look up the specific escape function for that database, and use the proper escaping function on all your content "transferral".
P.S.: some definition may not be correct, please comment on that. I wanted to make the idea stick but am probably not using all the right terms.
First of all it is very uncommon to run textarea through urlencode()
urlencode was not designed for this purpose.
Second, if you still want to do this, then maybe the problem comes from database. First you need to tell us what database you using and what TYPE you using for storing this data: do you store it as TEXT or as BINARY data? Have you setup the correct charset in database?