Is there any way to use Zend_Filter_Encrypt with large files, without rising memory limit to an unacceptable amount?
This is my code so far, but when i have to encrypt files larger than 32 MB (thats my memory limit) it fails, if I set memory to 48MB it works:
$vector = 'XX';
$algorithm = 'rijndael-192';
$options = array(
'adapter' => 'mcrypt',
'vector' => $vector,
'algorithm' => $algorithm,
'key' => $key
);
$encrypt = new Zend_Filter_File_Encrypt($options);
$result = $encrypt->filter($file);
No, there isn't. Zend_Filter_Encrypt works by encrypting/decrypting the data in one pass, thus requiring the full data in order to function.
If you need to decrypt a large file, you can do it manually in smaller chunks.
<?php
$key = 'secret key';
$iv = 'xxxxxxxxxxxxxxxx';
$cipher = mcrypt_module_open('rijndael-192', '', 'cbc', '');
mcrypt_generic_init($cipher, $key, $iv);
$fp = fopen('/tmp/encrypted.txt', 'r+b');
while (!feof($fp)) {
$data = fread($fp, 1154);
$decrypted = mdecrypt_generic($cipher, $data);
echo $decrypted;
}
fclose($fp);
mcrypt_generic_deinit($cipher);
mcrypt_module_close($cipher);
Just make sure the amount of data that you read (fread) is a multiple of the block size used by the algorithm, otherwise the results can be unexpected.
Related
thanks so much for reading.
I did so much research on this topic but did not get any further.
I have to decode data in PHP. This is the specification of the encryption (and there is no more specification to get from the encryptor):
keylength = 256
algorithm = AES/CBC/PKCS5Padding
keyspec = PBKDF2withHmacSHA1
iterations = 5000
What I also got, is a key, 49 characters long for decryption.
Each encrypted message is provided in an Array and it's base64 encoded
Example data:
$data = [
"iv" => "DoJQNS0WZRtWB...",
"salt" => "zkcHInm4ewweKG81...",
"encrypted_data" => "30MTuQEW4sVc3...",
];
I tried this:
$password = "supersecretkey";
$salt = base64_decode($data['salt']);
$iterations = 5000;
$key_length = 32;
$is_raw_output = true;
$key = hash_pbkdf2("sha1", $password, $salt, $iterations, $key_length, true);
$iv = base64_decode($data['iv']);
$encstr = base64_decode($data['encrypted_data']);
$output = openssl_decrypt($encstr, 'AES-256-CBC', $key, OPENSSL_ZERO_PADDING, $iv);
var_dump($output);
var_dump(openssl_error_string());
what I get here is
bool(false)
string(94) "error:0606508A:digital envelope routines:EVP_DecryptFinal_ex:data not multiple of block length"
I'm not sure if this is the correct approach. I tried also decoding directly without using the hash_pbkdf2 function but all I receive then is garbage.
So if I just use this
$key = "supersecretkey";
$iv = base64_decode($data['iv']);
$encstr = base64_decode($data['encrypted_data']);
$output = openssl_decrypt($encstr, 'AES-256-CBC', $key, OPENSSL_ZERO_PADDING| OPENSSL_RAW_DATA, $iv);
then I just get garbage data like
*+�l��_�y9�{(kNF7��gص��[Se!Y
I would appreciate any help on this topic as I already spent a lot of hours and read tons of documentation.
Unfortunately I do not get any more support from the entity that sends the encrypted data.
Thanks in advance!
Hi thanks all for helping me out.
The correct solution to this problem is below.
Obviously we were sent an incorrect encryption key. They generated a new one - but they insisted that the old one was correct - and then it worked like charm!
Thanks again.
$salt = base64_decode($datasend['salt']);
$iterations = 5000;
$key_length = 32;
$key = hash_pbkdf2("sha1", $e2ekey, $salt, $iterations, $key_length, true);
$iv = base64_decode($datasend['iv']);
$encstr = base64_decode($datasend['encryptedMessage']);
$datasend = openssl_decrypt($encstr, 'AES-256-CBC', $key, OPENSSL_RAW_DATA, $iv);
$datasend = json_decode($datasend,true);
I have a PHP page that loops through a CSV file and encrypts the 'email' column using the following function:
function my_encrypt($data, $key)
{
// Remove the base64 encoding from our key
$encryption_key = base64_decode($key);
// Generate an initialization vector
$iv = openssl_random_pseudo_bytes(openssl_cipher_iv_length('aes-256-cbc'));
// Encrypt the data using AES 256 encryption in CBC mode using our encryption key and initialization vector.
$encrypted = openssl_encrypt($data, 'aes-256-cbc', $encryption_key, 0, $iv);
// The $iv is just as important as the key for decrypting, so save it with our encrypted data using a unique separator (::)
return base64_encode($encrypted . '::' . $iv);
}
In another part of the app, I decrypt the returned value using:
function my_decrypt($data, $key)
{
// Remove the base64 encoding from our key
$encryption_key = base64_decode($key);
// To decrypt, split the encrypted data from our IV - our unique separator used was "::"
list($encrypted_data, $iv) = explode('::', base64_decode($data), 2);
return openssl_decrypt($encrypted_data, 'aes-256-cbc', $encryption_key, 0, $iv);
}
This all works smoothly for the most part, but every now and then, the decrypted value comes back with a few weird characters in it.
For example: rsmi3�6CTΣ%mecompany.com was returned instead of rsmith#somecompany.com.
I'm not sure if it's the input or the output that is bad, but I'm guessing it has something to do with the uploaded CSV file... encoding issue? What do those characters mean and under what conditions are they produced?
UPDATE
Here's the code I'm using to add the encrypted value to the CSV:
$file = fopen(get_stylesheet_directory() . "/emma_members.csv", "r"); //Open the old file for reading
$newFile = fopen(get_stylesheet_directory() . "/emma_members_new.csv", "w"); //Create a new file for writing
if (!$file) error_log('ERROR opening file');
if (!$newFile) error_log('ERROR creating file');
$columns = ['email', 'member_id', 'member_since', 'plaintext_preferred', 'bounce_count', 'status_name', 'last_modified_at', 'city', 'first_name', 'last_name', 'request-demo', 'job-function', 'title', 'country', 'current-ams', 'opt-in', 'address-2', 'unique-identifier', 'state', 'postal_code', 'web-address', 'address', 'phone-number', 'company', 'area-of-specialization', 'work-phone'];
while (($data = fgetcsv($file)) !== FALSE) {
$row = array_combine($columns, $data);
$email = "{$row['email']}";
$uid = my_encrypt($email, ENCRYPT_KEY_1);
$row['unique-identifier'] = $uid;
$ret = fputcsv($newFile, array_values($row));
}
UPDATE 2
So after much testing with thousands of emails, it seems the my_encrypt function returns some bad values, depending on the input of course. It didn't happen with EVERY email address, but even 1 is too many for my use case.
I even tried getting rid of the :: between the data and the iv, but that didn't work either (although it's possible I did it wrong).
Anyway, I ended up using the following function in its place, and all is well:
function encrypt_decrypt($action, $string) {
$output = false;
$encrypt_method = "AES-256-CBC";
$secret_key = PHRASE_1;
$secret_iv = PHRASE_2;
// hash
$key = hash('sha256', $secret_key);
// iv - encrypt method AES-256-CBC expects 16 bytes - else you will get a warning
$iv = substr(hash('sha256', $secret_iv), 0, 16);
if ( $action == 'encrypt' ) {
$output = openssl_encrypt($string, $encrypt_method, $key, 0, $iv);
$output = base64_encode($output);
} else if( $action == 'decrypt' ) {
$output = openssl_decrypt(base64_decode($string), $encrypt_method, $key, 0, $iv);
}
return $output;
}
I tested your encrypt and decrypt functions and they are working as expected, so the reason for the behaviour seems to be a different file encoding on your device.
Especially when reading a CSV-file sometimes a (windows) device changes the encoding and you get some curious characters like those you have shown. My recommendation is to read the files with another encoding as the default one (ISO...).
I setup a live example that "proves" the correctness on a simple string en- and decryption: https://paiza.io/projects/e/Y-1gy9Y3b-VAlXAMG4odng
The result is simple:
plaintext: rsmith#somecompany.com
ciphertext: Y0RrMWRwR1pWeGtGbFdic3dIVmFzVmp4VUFYemJGdUhzMStRSll6akIwWT06Orf+twLGopVa4083RckEw44=
decryptedtext: rsmith#somecompany.com
Here is the code:
<?php
function my_encrypt($data, $key)
{
// Remove the base64 encoding from our key
$encryption_key = base64_decode($key);
// Generate an initialization vector
$iv = openssl_random_pseudo_bytes(openssl_cipher_iv_length('aes-256-cbc'));
// Encrypt the data using AES 256 encryption in CBC mode using our encryption key and initialization vector.
$encrypted = openssl_encrypt($data, 'aes-256-cbc', $encryption_key, 0, $iv);
// The $iv is just as important as the key for decrypting, so save it with our encrypted data using a unique separator (::)
return base64_encode($encrypted . '::' . $iv);
}
function my_decrypt($data, $key)
{
// Remove the base64 encoding from our key
$encryption_key = base64_decode($key);
// To decrypt, split the encrypted data from our IV - our unique separator used was "::"
list($encrypted_data, $iv) = explode('::', base64_decode($data), 2);
return openssl_decrypt($encrypted_data, 'aes-256-cbc', $encryption_key, 0, $iv);
}
$plaintext = 'rsmith#somecompany.com';
echo 'plaintext: ' . $plaintext . PHP_EOL;
$encryptionKey = base64_encode(32);
$ciphertext = my_encrypt($plaintext, $encryptionKey);
echo 'ciphertext: ' . $ciphertext . PHP_EOL;
$decryptedtext = my_decrypt($ciphertext, $encryptionKey);
echo 'decryptedtext: ' . $decryptedtext . PHP_EOL;
?>
How do I setup my IV in codeigniter 3 encryption library?
I have encryption code below which was running smoothly in Codeigniter 2 with PHP 5.6,
function encrypt($data, $secret)
{
//Generate a key from a hash
$key = md5(utf8_encode($secret), true);
$data2 = utf8_encode($data);
$iv = utf8_encode("jvz8bUAx");
//Take first 8 bytes of $key and append them to the end of $key.
$key .= substr($key, 0, 8);
//Pad for PKCS7
$blockSize = mcrypt_get_block_size('tripledes', 'cbc');
//Encrypt data
$encData = mcrypt_encrypt('tripledes', $key, $data2, MCRYPT_MODE_CBC, $iv);
return urlencode(base64_encode($encData));
}
When I upgraded to CI 3 with PHP 7.1 mcrypt was deprecated already. So, I wanted to recreate the function in CI 3 using encryption library, but I cannot get the correct encrypted strings.
$this->load->library('encryption');
$key = md5(utf8_encode($secret), true);
$key .= substr($key, 0, 8);
$iv = utf8_encode("jvz8bUAx");
$amount = 1100;
$json = array(
'Amount' => $amount
);
$data = json_encode($json);
$params = array(
'driver' => 'mcrypt',
'cipher' => 'tripledes',
'mode' => 'cbc',
'key' => $key,
'hmac' => false
);
$ciphertext = $this->encryption->encrypt($data, $params);
$ciphertext = urlencode(base64_encode($ciphertext));
In the CI How it is works
Generate a random initialization vector (IV).
The library generates the IV for you and then prepends it for you to the resulting ciphertext. During the decryption, the IV is extracted from the ciphertext.
Since, by default, you cannot control the IV, the ciphertext will be different. If you want to really decrypt the old library encryption with the new one you have to prepend the IV as in the CI 3.
I'm trying to encrypt all files being uploaded to the server, and my method of doing it works; but I've noticed DECRYPTING files over 100kb just returns null, and I'm confused why encrypting works on these files, but decrypting doesn't. Is there something wrong with my code, or is there another approach to this? There is nothing wrong with the allowed upload sizes in php.ini, the upload.php page works perfectly fine, and uploads the files to the server. The only issue is with files over 100kb. I have a feeling it has something to do with the max variable length in PHP, but I'm not sure.
// Encrypt Function
public static function mc_encrypt($encrypt, $key)
{
$encrypt = serialize($encrypt);
$iv = mcrypt_create_iv(mcrypt_get_iv_size(MCRYPT_RIJNDAEL_256, MCRYPT_MODE_CBC), MCRYPT_DEV_URANDOM);
$key = pack('H*', $key);
$mac = hash_hmac('sha256', $encrypt, substr(bin2hex($key), -32));
$passcrypt = mcrypt_encrypt(MCRYPT_RIJNDAEL_256, $key, $encrypt.$mac, MCRYPT_MODE_CBC, $iv);
$encoded = base64_encode($passcrypt).'|'.base64_encode($iv);
return $encoded;
}
// Decrypt Function
public static function mc_decrypt($decrypt, $key)
{
$decrypt = explode('|', $decrypt.'|');
$decoded = base64_decode($decrypt[0]);
$iv = base64_decode($decrypt[1]);
if(strlen($iv)!==mcrypt_get_iv_size(MCRYPT_RIJNDAEL_256, MCRYPT_MODE_CBC)){ return false; }
$key = pack('H*', $key);
$decrypted = trim(mcrypt_decrypt(MCRYPT_RIJNDAEL_256, $key, $decoded, MCRYPT_MODE_CBC, $iv));
$mac = substr($decrypted, -64);
$decrypted = substr($decrypted, 0, -64);
$calcmac = hash_hmac('sha256', $decrypted, substr(bin2hex($key), -32));
if($calcmac!==$mac){ return false; }
$decrypted = unserialize($decrypted);
return $decrypted;
}
Where it should be decrypted:
try
{
$server = $db->prepare("SELECT * FROM `servers` WHERE `ServerIP` = :ip LIMIT 1");
$server->execute([ ":ip" => $ip ]);
$server = $server->fetch();
$sftp = new SFTPConnection($server['ServerIP'], intval($server['ServerPort']));
$sftp->login($server['ServerUser'], $server['ServerPassword']);
$fileData = $sftp->receiveFile($path);
//print $fileData;
header('Content-type: text/plain');
$fileName = $file['FileName'];
header("Content-Disposition: attachment; filename=$fileName");
//print $fileData; (returns the encrypted version)
$fileData = Encryption::mc_decrypt($fileData, $file['EncryptionKey']);
print $fileData; // (returns null on larger files)
}
catch (Exception $e)
{
echo $e->getMessage() . "\n";
}
I'm not sure what the issue is, but I do know a solution. First of all, you probably want to read in the file in chucks. You don't want to store e.g. an entire movie in RAM. So what you can do is to treat the SFTP connection as stream:
According to the sample code here:
$connection = ssh2_connect('shell.example.com', 22);
ssh2_auth_password($connection, 'username', 'password');
$sftp = ssh2_sftp($connection);
$stream = fopen("ssh2.sftp://$sftp/path/to/file", 'rb');
Note that I used 'rb' to force binary mode.
So now you can read in chunks from the stream, the only thing you need to do is to encrypt/decrypt the stream. Mcrypt does actually provide this functionality using a filter implementation.
As for the HMAC, you can stream that as well. You may want to create a filter for it - I could not find one.
So now that you can stream everything, go ahead and implement it.
Security notes:
mcrypt is an old library that should not be used anymore;
use MCRYPT_RIJNDAEL_128 instead of using MCRYPT_RIJNDAEL_256 if you want to use AES (the 256 is the block size, not the key size, the key size is determined by - wait for it - the size of the provided key);
HMAC is secure, but it should be performed over the ciphertext and the IV;
this is not a full transport protocol - but that doesn't matter much if you send the file over sftp.
Disclaimer: English isn't my mother tongue so feel free to ask if something isn't clear.
Hi there,
I have to encrypt files using AES as soon as they are uploaded on the server and send the key needed to decrypt them via mail to the client (not storing it anywhere server side). Files can be as big as 2GB and are deleted 7 days after their upload.
Here is what I'm using to encrypt/decrypt files :
function encrypt_file($source, $destination, $key) {
$iv = md5("\x1B\x3C\x58".$key, true);
$ivsize = openssl_cipher_iv_length('aes-256-cbc');
$fp = fopen($destination, 'wb') or die("Could not open file for writing.");
$handle = fopen($source, "rb");
while (!feof($handle)) {
$e = 0;
$contents = fread($handle, 4 * 1024 * 1024);
$ciphertext = openssl_encrypt($contents, 'aes-256-cbc', $key, OPENSSL_RAW_DATA, $iv);
$iv = substr($ciphertext, -$ivsize);
while (!fwrite($fp, $ciphertext)) {
$e++;
if ($e == 5) {
die("Couldn't write to file.");
break 2;
}
}
}
fclose($handle);
fclose($fp);
}
function streamdecrypt_file($source, $key) {
$iv = md5("\x1B\x3C\x58".$key, true);
$ivsize = openssl_cipher_iv_length('aes-256-cbc');
$handle = fopen($source, "rb");
while (!feof($handle)) {
$contents = fread($handle, 4 * 1024 * 1024);
$raw = openssl_decrypt($contents, 'aes-256-cbc', $key, OPENSSL_RAW_DATA, $iv);
$iv = substr($contents, -$ivsize);
print $raw; // Printing because it's directly sent to the user to download
}
fclose($handle);
}
If you're wondering why 4 * 1024 * 1024 it's just that this is the buffer size with which I got the fastest encryptions. My implementation uses the schema proposed here https://stackoverflow.com/a/30742644/3857024
I also made those 2 little functions to encrypt a string to a file using a passphrase :
function encrypt_string($source, $destination, $passphrase) {
$iv = md5("\x1B\x3C\x58".$passphrase, true);
$key = md5("\x2D\xFC\xD8".$passphrase, true);
$ciphertext = openssl_encrypt($source, 'aes-256-cbc', $key, OPENSSL_RAW_DATA, $iv);
$fp = fopen($destination, 'wb') or die("Could not open file for writing.");
fwrite($fp, $ciphertext) or die("Could not write to file.");
fclose($fp);
}
function decrypt_string($source, $passphrase) {
$iv = md5("\x1B\x3C\x58".$passphrase, true);
$key = md5("\x2D\xFC\xD8".$passphrase, true);
$contents = file_get_contents($source);
return openssl_decrypt($contents, 'aes-256-cbc', $key, OPENSSL_RAW_DATA, $iv);
}
And here is what I'm finally doing when an upload is complete :
$skey = /* 32 chars random generated string [a-Z0-9]*/
$ukey = /* 10 chars random generated string [a-Z0-9]*/
encrypt_file($originalFile, $encryptedFile, $skey);
encrypt_string($skey, $encryptedKey, $ukey);
I then delete the original file and send a link containing the $ukey to the user via mail.
When they want to decrypt the file to download it, I first decrypt the file containing the $skey using the $ukey, checking if I end up with a 32-chars 256-bits long string made of [a-Z0-9]. If the $skey doesn't match the regexp, I know the $ukey is invalid, so I stop there.
I did this so that I wouldn't have to decrypt the file to check if the key was correct or not.
Now I hope that my questions fit in SO :
Am I doing it right ?
Is there anything that could/should be improved ?
It takes about 60s to encrypt a 2GB file, is that an "ok" result ?
Is it good enough ? The goal is to prevent an attacker gaining access to the server to also gain access to the users files already stored. I know he would then be able to modify the code and access the following uploads, but that should protect the files already stored right ? Am I doing too much for nothing ?
Thank you for your answers !
For an IV, use random bytes.
For password expansion, use PBKDF2 or equivalent; the derivation needs to be slower.
Restricting a key to the characters [a-Z0-9] reduces the 256 key to essentially 36 bytes. That is not very secure. You need at least 128-bits of key material.
You need a better method to authenticate the user's password.