I am in the process of building a small web application which will hold around 10 pieces of information for every person inserted. Due to data protection the majority of this information must be encrypted.
Using the CodeIgniter framework and the CodeIgniter encryption class I can encode the information on the application side before storing it in the database. The CodeIgniter encryption class uses PHP's mcrypt function along with the AES_256 cipher.
The problem I have is that I need to allow the users of the application to search the information stored using a wildcard search, Possibly also via an API at a later date.
Any body come across a solution for a similar problem. I've read about the MySQL AES_ENCRYPT and AES_DECRYPT but they still require passing a key back and forth in plain text which I am reluctant to do.
I am currently at the conclusion that if I wish to continue on this route then a full table decryption is my only solution every time a search is made (obviously not good).
Well, you can't search in decrypted text without decoding it first, that is true.
However, that doesn't mean that there are no ways around this. For example, you could make an inverted index of your data and hash (sha1, md5, crc32, pick one) the keys used for searching. All you have to do then is hash the search terms you're using, look them up in the index and retrieve any record that matches, which will only be a small part of the table instead of the entire thing.
By hashing the data (use a salt!), you avoid storing the data in an unsafe way, while you can still search through the data because you made an index for it. No decryption required until you're actually sure which documents match.
Related
What would be useful solutions for hiding true database object ID in URL for security purposes? I found that one of the solutions would be:
1) Using hashids open source project
2) Using something like same old md5 on creation of the object to generate hash and store it in database, then use it in url's and querying by them, but the drawback is that querying by auto-incremented primary keys (IDs) is faster than hashes. So I believe the possibility to hash/unhash would be better?
Also as I'm on Symfony, are there maybe bundles that I could not find or built in functionalities that would help?
Please tell me what you found useful based on your experiences.
This question has been asked a lot, with different word choice (which makes it difficult to say, "Just search for it!"). This fact prompted a blog post titled, The Comprehensive Guide to URL Parameter Encryption in PHP .
What People Want To Do Here
What People Should Do Instead
Explanation
Typically, people want short random-looking URLs. This doesn't allow you much room to encrypt then authenticate the database record ID you wish to obfuscate. Doing so would require a minimum URL length of 32 bytes (for HMAC-SHA256), which is 44 characters when encoded in base64.
A simpler strategy is to generate a random string (see random_compat for a PHP5 implementation of random_bytes() and random_int() for generating these strings) and reference that column instead.
Also, hashids are broken by simple cryptanalysis. Their conclusion states:
The attack I have described is significantly better than a brute force attack, so from a cryptographic stand point the algorithm is considered to be broken, it is quite easy to recover the salt; making it possible for an attacker to run the encoding in either direction and invalidates property 2 for an ideal hash function.
Don't rely on it.
Quote from the site:
Do you have a question or comment that involves "security" and "hashids" in the same sentence? Don't use Hashids.
I'd use true encryption algorithm, like function openssl_encrypt (for example), or something like this. And encrypt ids when passing outside, decrypt when using in your code (like for db queries).
And I won't recommend storing ids in a base like any kind of encrypted "garbage", in my opinion its very inconvenient to hash your real ids. Keep it clean and pretty inside and encrypt for external display only.
Following your idea, you just need to cipher your IDs before writing the URL to HTML page and decipher them when processing those URLs.
If you want just security by obscurity, which is sufficient for, maybe 99% of curious people out there who likes to iterate over IDs in URLs, you use something simple like base64 or rot13. Of course, you can also precalculate those "public IDs" and store in the database, not encrypting each time the URL is being shown to end user.
If you want true security you have to encrypt them with some serious asymmetric cypher, storing both keys at your side, as you essentially talking with yourself and don't want a man-in-the-middle attack. This you will not be able to precalculate as at each encrypting there'll be different cyphertext, which is good for this cause.
In any case, you need something two-way, so if I were you I'd forget about word "hash", hashes are for purposes different from yours.
EDIT:
But the solution which every blog out there uses for this task for several years already is just to utilize URL rewriting, converting, in your case, URLs like http://example.com/book/5 to URLs like http://example.com/rework-by-37signals. This will completely eradicate any sign of database ID from your URL.
Ideologically, you will need something which will uniquely map the request URL to your database content anyway. If you hide MySQL database IDs behind any layer of URL rewriting, you'll just make this rewritten URL a new ID for the same content. All you gain is protection from enumeration attacks and maybe SEF URLs.
What would be useful solutions for hiding true database object ID in URL for security purposes? I found that one of the solutions would be:
1) Using hashids open source project
2) Using something like same old md5 on creation of the object to generate hash and store it in database, then use it in url's and querying by them, but the drawback is that querying by auto-incremented primary keys (IDs) is faster than hashes. So I believe the possibility to hash/unhash would be better?
Also as I'm on Symfony, are there maybe bundles that I could not find or built in functionalities that would help?
Please tell me what you found useful based on your experiences.
This question has been asked a lot, with different word choice (which makes it difficult to say, "Just search for it!"). This fact prompted a blog post titled, The Comprehensive Guide to URL Parameter Encryption in PHP .
What People Want To Do Here
What People Should Do Instead
Explanation
Typically, people want short random-looking URLs. This doesn't allow you much room to encrypt then authenticate the database record ID you wish to obfuscate. Doing so would require a minimum URL length of 32 bytes (for HMAC-SHA256), which is 44 characters when encoded in base64.
A simpler strategy is to generate a random string (see random_compat for a PHP5 implementation of random_bytes() and random_int() for generating these strings) and reference that column instead.
Also, hashids are broken by simple cryptanalysis. Their conclusion states:
The attack I have described is significantly better than a brute force attack, so from a cryptographic stand point the algorithm is considered to be broken, it is quite easy to recover the salt; making it possible for an attacker to run the encoding in either direction and invalidates property 2 for an ideal hash function.
Don't rely on it.
Quote from the site:
Do you have a question or comment that involves "security" and "hashids" in the same sentence? Don't use Hashids.
I'd use true encryption algorithm, like function openssl_encrypt (for example), or something like this. And encrypt ids when passing outside, decrypt when using in your code (like for db queries).
And I won't recommend storing ids in a base like any kind of encrypted "garbage", in my opinion its very inconvenient to hash your real ids. Keep it clean and pretty inside and encrypt for external display only.
Following your idea, you just need to cipher your IDs before writing the URL to HTML page and decipher them when processing those URLs.
If you want just security by obscurity, which is sufficient for, maybe 99% of curious people out there who likes to iterate over IDs in URLs, you use something simple like base64 or rot13. Of course, you can also precalculate those "public IDs" and store in the database, not encrypting each time the URL is being shown to end user.
If you want true security you have to encrypt them with some serious asymmetric cypher, storing both keys at your side, as you essentially talking with yourself and don't want a man-in-the-middle attack. This you will not be able to precalculate as at each encrypting there'll be different cyphertext, which is good for this cause.
In any case, you need something two-way, so if I were you I'd forget about word "hash", hashes are for purposes different from yours.
EDIT:
But the solution which every blog out there uses for this task for several years already is just to utilize URL rewriting, converting, in your case, URLs like http://example.com/book/5 to URLs like http://example.com/rework-by-37signals. This will completely eradicate any sign of database ID from your URL.
Ideologically, you will need something which will uniquely map the request URL to your database content anyway. If you hide MySQL database IDs behind any layer of URL rewriting, you'll just make this rewritten URL a new ID for the same content. All you gain is protection from enumeration attacks and maybe SEF URLs.
I am using MySQL as a back end storage.
I was asked by our risk management team to encrypt all the data prior storing it into the database.
Since then I have been doing research on how to secure the data going in and out the database.
I found couple ways one of them was MySQL Encryption Software
A second solution was to encrypt and decrypt data in MySQL using AES_ENCRYPT() AND AES_DECRYPT(). But I will need to create a 128,196 or 256 bit key in order to be able to encrypt and decrypt the data. Then every time I want to execute INSERT/UPDATE I will call AES_ENCRYPT() and supply it with a key to encrypt the data. Then when I execute SELECT then I will have to call AES_DECRYPT() and supply the same key to convert the data to a plain text.
This means that I will have to define a variable in my PHP script that have the private key so I can encrypt/decrypt by supplying the defined variable to both AES_ENCRYPT() and AES_DECRYPT()
My question is
Where/How to a store this private key to prevent a hacker from reading it. if someone hacks my server reads the key then he can simply read the data and the encryption would be mean less.
And what is the best way to go about securing my data?
Thank you
The issue you are facing is not a key issue but an issue of security of the rest of your computer. Using mysql means that mysql (if running safely) is running in it's own account. You would in fact put the keys in your mysql-owned directory. That secures much of mysql. MySql itself needs to have access to that key, so there is not much more you can do for that account. Just make sure it is readable only by the owning account.
If you hide your key and the hacker gets to your PHP code the hacker could do the following:
A. Echo/Print the KEY (it's a variable). He doesn't have to find the code where you define it, he could look for the function where you use it.
B. Forget about the KEY and use your own decrypting function to see the data and export it.
Most of the hacks are exploits to your PHP code, you should secure your computer, system and database.
Use MYSQL Stored Procedures
If only an authenticated user can get the key you could do this... Hide the KEY in a MYSQL STORED PROCEDURE and only give the key to the user when he completes the log in. All the data in the database should be encrypted and the key is not accesible as plain stored data. It should be inside the SQL Stored Procedure that a user having a password will get as query result. Users table should not be encrypted, only the password as one way.
This last solution should work only for authenticated users, like apps as example. You could protect the user profiles easily with this solution.
Users (basic data) --> Profiles (all personal details that depend upon log in)
I don't know much about stored procedures, but for sure you could apply it to the entire database and make your data accesible only through them.
Look at this article:
http://www.sitepoint.com/stored-procedures-mysql-php/
And you will find PROS and CONS here:
http://code.tutsplus.com/articles/an-introduction-to-stored-procedures-in-mysql-5--net-17843
How should 'sensitive' data be stored in MySQL Database?
1) Should I focus more on the security of the MySQL database and store the data as plain text?
I found a step by step tutorial on how to make a MySQL database more secure:
http://www.symantec.com/connect/articles/securing-mysql-step-step
2) Should I encrypt the data?
If yes, then how should the encryption be done?
Using MySQL aes_encrypt/aes_decrypt?
Using PHP AES functions/algorithm for encrypting/decrypting data?
How should the data be stored in MySQL?
BLOB
BINARY
VARBINARY
In my case the 'sensitive' data are payments done by individuals.
Thanks
It's a mixture of both. Two existing answers (at the time I wrote this https://stackoverflow.com/a/10718397/1015483 and https://stackoverflow.com/a/10718459/1015483) are valid - you need to look at about 5 methods of possible attack that I can think of
They get access to your DB server; so yes, secure that baby as much as is reasonable (Matt's answer)
Stand alone data hijacking (someone gets to your database data somehow else, could be a backup, could be they guess a password, could be MITM if you transfer data from one place to another). For this, you do encypt your data. You also may do a CSV dump for some reason and e-mail to someone. Whoops. But it happens. So encrypt (vlzvt's answer)
But three elements not mentioned:
They could gain access to your web server (if different from your DB server). If they have access to the webserver, all bets are off as they have your password, encyption keys the lot. So you need to make that even more secure than the DB server. (Matt might have meant that above - but just make it clear)
Similar to above, but not to be forgotten, is if someone gets access to phpMyAdmin or your management consule. Don't use plain text auth or config stored passwords for access.
Finally there's your application itself (and the hardest to lock down). You need to prevent against SQL injections that may reveal data. Encrypting the data would stop minimise problems if someone did gain access through an untrapped query - so for this, encryption is the solution.
For part 2 of your question:
Using MySQL encrypt/decrypt functions will stop someone who has access to the raw data, but not MITM or SQL injection or even CSV dumps taken for transport.
So, IMO (and it's only my opinion and the way I've done it) is to encrypt with PHP and sned the encrypted data over the wire, as that stops all methods of trapping the data, and a CSV dump will be "scrambled".
If you do that, you may as well use the varbinary / blob types as it stops you accidentally trying to read/edit in phpMyAdmin. Plus potentially saves a few bytes nominally (although this depends on indexes and other stuff - so that alone is not a winning argument).
And now the down side: searching and sorting. Anything you index or search on, if encrypted, will only match the entire, exact, case sensitive string padded to the correct length (normally a search will be case insensitive, and you can do part searches with LIKE). And if you want to ORDER BY then you need the original strings. So bear than in mind when designing the structure.
Hope that helps.
What's the worst possible scenario if an attacker gets access to the plain text data? Given that you have to decrypt data in order to make it useful and you therefore need the encryption key to be somewhere accessible too, any attacker who can get to the DB will likely be able to get to the key as well, unless this is for archiving rather than e.g. a live website. I'd focus on the DB server security, unless you're carting HDDs around full of data which might get lost, but it really depends on why you need to encrypt it.
if you need to secure the data in your possibly hacked database, you can encrypt it
with mcrypt
$key = "mykey";
$crypttext = mcrypt_encrypt(MCRYPT_RIJNDAEL_256,$key,$data,MCRYPT_MODE_ECB,$key);
after that you can select BLOB,TEXT,MEDIUMTEXT or anything, based on the ~expected data size.
* for VARBINARY/BINARY you might need to pack it first.
The encryption operation has additional cost.
You need evaluate if in your scenario, this additional cost will be a problem, per example, if your data grow to considerable size.
The first frontier to avoid data leaks is a strong data access policy with profiles of access an so on. This has the disadvantage that you will need to manage the mysql and configurate it.
If you want care about the managment of profiles, you can encrypt the data assuming the extra cost in CPU and (depend of encyption algorithm) a extra storage space.
The security of the a system is equals to the security of more weak component, don't focus your effort only in encryptation task, this only give you the sensation of security, if the data can be decrypted, the only thing that the intruder needs is time and bruteforce to break the encryption
Here's the story:
I have to pass some classified information from one script to another script. In the first script I need to encrypt the data first and then I have to attach the encrypted data into a GET request and send it to another script. The URL would look like this:
http://mydomain.com/mysecondscript.php?secret={encrypted stuff}
My current encrypt method is based on base64. The problem with this method is, that if I have a lot of stuff to encrypt, the encrypted result could get very long. If it's longer than 255 characters or so, the second script will not be able to decrypt it because the string will be chopped.
So I'm looking for a better encryption method, that can control the length of the encrypted result.
DANGER!
Base64 is NOT a form of encryption, but encoding. Base64 encoded strings are easy to recognize and trivial to decode. Base64 is used to encode data so they can be safely transmitted across non-binary safe medium (such as URLs and emails), but they do not hide the data itself.
What you need to do is encrypt the string using AES (see PHP's mcrypt), then base64 encode it. This of course will not solve your length problem. The question is pretty vague, but what you can do is:
Use POST instead of GET.
Store data in a database or a file which both scripts can access. Then just generate a sort of identifier and send it with the URL. The receiving script can use this identifier to retrieve the data. As an added bonus you won't have to send classified data with the URL.
EDIT: Now that I read your question more carefully, it seems like both scripts are sitting on the same server. In this case there is no reason whatsoever to pass this data via HTTP.
No matter how secure your encryption scheme is you will still need to base64 or URL-encode the result which, you have discovered, will likely exceed 255 characters. The best you can do is compress the data, then encrypt it, then encode it. It will still probably fail. You need to find an alternative to GET.
Why does it have to be transmitted in the URL at all? Save it to disk, put it in the database, add it to a message passing queue...
You can use an opaque token in the URL to identify which thing you're talking about, and then turn that token back into a useful thing on the other end by querying whatever storage mechanism you choose.
If this is sensitive information, base64 should not be used as it can be decoded easily. If you want the information to be securely encrypted, you shoule use PHP Mcrypt (link). Much more secure and can support encryption of much longer strings. Best of all, you set your own key and it cannot be decrypted without that key. It make require a tiny bit more work, but it will be safe. Also, if you are passing multiple variables that way, you can set them into an array, serialize and encrypt the array, pass the array via GET, and then decrypt/unserialize. It's as simple as that. One last thing, there are also some classes out there that will make Mcrypt a lot easier to use. May want to google to find one, it will make your life easier.
You can use MD5 to create a hash of the id so you get something like:
http://www.demo.com/displaycommittees.php?id=81dc9bdb52d04dc20036dbd8313ed055
In your db query you can do a select on the users table including a WHERE statement that also hashes the id column like:
WHERE MD5(id) = $_GET[id]
This works fine and i have always applied this algorithm. for instance assuming the actual value of the encrypted id 23, if you try to put 23 in place of the encrypted(hash) code it will not work( no result will be display).
Note: Reasons are best known to those who need a solution, so the question "why" may not come in for those who need it. they only ask for a solution and if it works fine for them, nice. But for transaction application (e.g cash transaction or transaction pins) please do avoid passing sensitive information via URL, because it can easily be cracked.
See more at: http://softideass.blogspot.com/