I have a database that have a copy of all emails from an imap server.
So I have a column name that is the sender's address, sometimes like this:
=?utf-8?B?UmVwb3J0Z
and others without encoding, and others with others encoding
=?iso-8859-1?Q?SALUDO_Y_CO
I think it is the mime encode for emails.
How can I search that column with an already decoded value? Example
SELECT * FROM tablename WHERE decoded_column(columnname) REGEXP '".$text_to_search."'
As #tadman already mentioned there is no built-in way of doing this.
You may try to use the following solution. Create a stored function that you will use later in your WHERE clause.
CREATE FUNCTION quoted_printable(input VARCHAR(255)) RETURNS VARCHAR(255)
BEGIN
// Here the code for decoding your input
END
and then in your query:
SELECT * FROM tablename WHERE quoted_printable(columnname) REGEXP '".$text_to_search."'
Just keep in mind that stored functions have more limitations than procedures so you may need to call a procedure from you function to get what you need.
Having said that, I would still suggest to store the data in a sanitized way - as #tadman said convert them already to UTF-8 before you store them.
The solution I proposed above should be treated just as an ugly workaround rather than a viable solution.
That's quoted printable and MySQL has no built-in way of decoding that. You'll need to do it externally. I'd recommend decoding those before inserting them in the database as UTF-8. If they're stored like that now, replace them with the normalized version.
Related
I need to export data in selected tables from a MySQL database and import it into another MySQL database with a slightly different structure. IOW, I need to modify the data between the export and import (and not just the field names).
I've tried using json_encode and json_decode and it almost works, but if all the data is not pure utf8, json_encode falls over and utf8_encode doesn't solve this.
I'm considering CSV, serialize, and generating SQL in PHP. Which of those options will give me the most reliable transfer?
You could probably use PDO to query and create an array of objects than can be directly modified in code. Then use that data to import into the other database.
The better question is why UTF-8 isn't working for you. Make sure you are using a recent version of PHP.
Also make sure you are using the mb_ prefixed functions to maintain UTF-8 (or other multibyte) encoding.
List of PHP MultiByte String Functions
You can also shape the data during the export, depending on how you're dumping the data using the SELECT statement and re-ordering columns or selectively omitting some or using conditionals to change the data. The result will still be a set of rows and columns.. as long as they match the structure of your destination, you can do a blind import. Chances are, you're using mysqldump or something though, which isn't quite as easy. Depending on the changes needed, you could always use SQL to modify the data after re-import as well.
Since SQL is made for juggling data, that could possibly be the easiest way to deal with things rather than trying to parse a ton of stuff with PHP while it's in it's dumped format.
My problem was not caused by an issue with json_encode as I thought. Instead, the issue was a faulty left join.
So the answer is that all the listed methods are reliable as long as your query is formed properly.
I'm storing data in a MySQL database that may have some special characters. I'm wondering how to store it so that these characters are preserved if they're either output to HTML via PHP OR via JavaScript, e.g. createTextNode.
For example, the division symbol (÷) has the html code ÷, and when I store it as that it shows up fine when put directly into HTML by PHP, but when I pull it into JavaScript using $.getJSON and then insert it with createTextNode it shows up looking like ÷.
I also tried storing the symbol in the SQL directly, but my understanding is that the column would need to be changed from VARCHAR to NVARCHAR and that would cause a performance hit that doesn't seem necessary.
Given that I can modify the SQL, the PHP, or the JavaScript, is there an easy fix here? Maybe a way to unescape the HTML entity in JavaScript?
As answered by Yogesh, you should switch your collation of the DB to utf8_general_ci
So there's probably two things going on:
JSON escapes special characters.
Somewhere, something in your code flow is URL encoding the strings too.
So you just need to decode the string in your JavaScript, or you need to find what part of your code is URL encoding those strings and fix it.
I am having a little issue with storing mcrypt_module_open('rijndael-256','','ofb',''); in a MySQL db.
When it inserts the encrypted data into the MySQL db it looks like this ˜9ÏÏd‰.
It should look like this
÷`¥¶Œ"¼¦q…ËoÇ
I am wondering if I have to do something to get it to work?
Use a blob field type for storing binary data (BLOB, VARBINARY, BINARY)
If you're not doing this already: escape your values with the proper methods if you're using them directly in a SQL-statement. Or even better: use query parameters/prepared statements.
As a last resort you could just encode your data with either base64_encode or bin2hex.
If you want to display binary data on the console or in the browser (even for debugging purpose) use one of those encodings too. Otherwise you might not see the actual data because the browser might not display your binary correctly.
In general, it might be a good idea to base64 encode and decode binary data like this. See Best way to use PHP to encrypt and decrypt passwords? .
Have you tried to Collation of your table that your character supports.
The characters '÷`¥¶Œ"¼¦q…ËoÇ' looks like UTF-8 or someother charset, find charset of your characters and update table Collation based your charset
Ok, this subject is a hotbed I understand that. I also understand that this situation is dependent on what you are using as code. I have three situations that need to be resolved.
I have a form in where we need to allow people to make comments and statements that use commas, tildes, etc... but still remain safe from attacks.
I have people entering in dates like this: 10/13/11 mm/dd/yy in English, can this be sanitized?
How do I understand how to use htmlspecialchars(), htmlentities() and real_escape_string() correctly? I've read the php.net site and some posts here but this seems to me to be a situation in where it all depends on the person reading the question what the right answer is.
I really can't accept that... there has to be an answer wherein text formats similar to that which I am posting here can be sanitized. I'd like to know if and how it is possible.
Thanks... because it seems to me that when asking this question in other places it tends to annoy... I am learning what I need to know but I think I have hit a plateau in what I can know without an example of what it is meant to do...
Thanks in advance.
It's a very important question and it actually has a simple answer in the form of encodings. The problem you are facing it that you use a lot of languages at the same time. First you are in HTML, then in PHP and a few seconds later in SQL. All these languages have their own syntax rules.
The thing to remember is: a string should at all times be in its proper encoding.
Lets take an example. You have a HTML form and the user enters the following string into it:
I really <3 dogs & cats ;')
Upon pressing the submit button, this string is being send to your PHP script. Lets assume this is done through GET. It gets appended to the URL, which has its own syntax (the & character has special meaning for instance) so we are changing languages. This means the string must be transformed into the proper URL-encoding. In this case the browser does it, but PHP also has an urlencode function for that.
In the PHP script, the string is stored in $_GET, encoded as a PHP string. As long as you are coding PHP, this is perfectly fine. But now lets put the string to use in a SQL query. We change languages and syntax rules, therefore the string must be encoded as SQL through the mysql_real_escape_string function.
At the other end, we might want to display the string back to the users again. We retrieve the string from the database and it is returned to us as a PHP string. When we want to embed it in HTML for output, we're changing languages again so we must encode our string to HTML through the htmlspecialchars function.
Throughout the way, the string has always been in the proper encoding, which means any character the user can come up with will be dealt with accordingly. Everything should be running smooth and safe.
A thing to avoid (sometimes this is even recommended by the ignorant) is prematurely encoding your string. For instance, you could apply htmlspecialchars to the string before putting it in the database. This way, when you retrieve the string later from the database you can stick it in the HTML no problem. Sound great? Yeah, really great until you start getting support tickets of people wondering why their PDF receipts are full of & > junk.
In code:
form.html:
<form action="post.php" method="get">
<textarea name="comment">
I really <3 dogs & cats ;')
</textarea>
<input type="submit"/>
</form>
URL it generates:
http://www.example.org/form.php?comment=I%20really%20%3C3%20dogs%20&%20cats%20;')
post.php:
// Connect to database, etc....
// Place the new comment in the database
$comment = $_GET['comment']; // Comment is encoded as PHP string
// Using $comment in a SQL query, need to encode the string to SQL first!
$query = "INSERT INTO posts SET comment='". mysql_real_escape_string($comment) ."'";
mysql_query($query);
// Get list of comments from the database
$query = "SELECT comment FROM posts";
print '<html><body><h2>Posts</h2>';
print '<table>';
while($post = mysql_fetch_assoc($query)) {
// Going from PHP string to HTML, need to encode!
print '<tr><td>'. htmlspecialchars($post['comment']) .'</td></tr>';
}
print '</table>';
print '</body></html>'
The crucial thing is to understand what each sanitising function available to you is for, and when it should be used. For example, database-escaping functions are designed to make data safe to insert into the database, and should be used as such; but HTML-escaping functions are designed to neutralise malicious HTML code (like JavaScripts) and make it safe to output data for your users to view. Sanitise the right thing at the right time.*
There are two different basic approaches you can take: you can sanitise HTML when you receive it, or you can store it exactly as you received it and sanitise it only when it is time to output it to the user. Each of these methods has its proponents, but the second one is probably the least prone to problems (with the first one, what do you do if a flaw is discovered in your sanitising procedure and you find you have insufficiently sanitised content stored in your database?)
Dates can be sanitised using a date parsing function. In PHP you might look at strtotime(). Your objective is typically to take a string representation of a date and output either an object representing a date, or another string that represents the same date in a canonical way (that is: in a specific format).
Regarding the sanitization of dates, PHP has some built-in functions that can be helpful. The strtotime() function will convert just about any imaginable date/time format into a Unix timestamp, which can then be passed to the date() function to convert it to whatever formatting you like.
For example:
$date_sql = date( "Y-m-d", strtotime( $_POST["date"] ) );
I'm unsure if this is a php-, filemaker-, mysql- or an odbc driver issue.
For security reasons the input fields of my current php webform convert special characters into hex codes, (for example: # becomes ' ) This hex code is saved in the database and will also be shown in Filemaker11 as the hex code. This is not what i want.
How can I make sure the special character will be displayed as it should be?
The other way round (from filemaker to db), no conversion will be done on inserting the special characters.
How can I make sure everything will be consistent?
Kind regards,
Jeroen
FileMaker is just showing the data stored in MySQL. If you pull up the DB in a tool like PhpMyAdmin you should see that the varchar contains the encoding as well. Since FMP is looking at it simply as a text field, it shows the encoding that was stored. If you wanted to decode in FMP you could show a calc field of the varchar that has a custom function to decode the text. (but that won't allow for updating the data..) You could also try a trigger on record load to decode the data in the fields so that you can properly view/edit.
Solved it! It appeared that I had to add an extra line to my PHP script.
after setting up the connection, php needs to tell mysql what the encoding needs to be. This can be done with the following line:
$dbh->query("SET NAMES 'utf8'");
Thanks for the effort guys!
This: ' type of encoding is not done automatically by the browser. Something is doing it. Normally you do it only on output not on input.
You can use html_entity_decode() to undo it. But I strongly suggest you figure out why it's happening in the first place.