I'm trying to write a program that decrypts AES files on the fly with phpseclib.
Files are large, so I get an out of memory error if I use file_get_contents($f) or fread(filesize($f)) to read the input file.
For some reason, a loop like this is creating corrupted output files. WHY!? =(
For example, an input file of size 296,155,408 bytes comes out with 18,805,826 bytes. NOTE: It works if the entire file can be read in one iteration of the loop.
define('MY_BUFFER_SIZE', 128 * 1024);
$sessionKey = '....';
$filenameIn = $argv[1];
$fileIn = fopen($filenameIn, 'rb');
$filenameOut = dirname($argv[1]) . DIRECTORY_SEPARATOR . basename($argv[1], '.tar.gz') . '-decrypted.tar.gz';
$fileOut = fopen($filenameOut, 'wb');
// Setup cipher for continuous buffering and copy between files.
$aesCipher = new Crypt_AES(CRYPT_AES_MODE_CBC);
$aesCipher->setKey($sessionKey);
$aesCipher->setIV("\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00");
$aesCipher->enableContinuousBuffer();
while (!feof($fileIn)) {
$packet = fread($fileIn, MY_BUFFER_SIZE); // #TODO: Streaming not working.
fwrite($fileOut, $aesCipher->decrypt($packet));
}
fclose($fileIn);
fclose($fileOut);
Thanks to #neubert!
What was required was adding:
$aesCipher->disablePadding()
This works:
// Setup cipher for continuous buffering and copy between files.
$aesCipher = new Crypt_AES(CRYPT_AES_MODE_CBC);
$aesCipher->setKey($sessionKey);
$aesCipher->setIV("\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00");
$aesCipher->enableContinuousBuffer();
$aesCipher->disablePadding();
while (!feof($fileIn)) {
fwrite($fileOut, $aesCipher->decrypt(fread($fileIn, MY_BUFFER_SIZE)));
}
Related
I am using file_put_contents to create a video file. the problem is the speed and performance. It takes about an average of 30 to 60 minutes for an average file size of 50 mb to be created and that is just for one file alone. I am decoding a byte array to create the file. How can I improve the speed and performance?
$json_str = file_get_contents('php://input');
$json_obj = json_decode($json_str);
$Video = $json_obj->Video;
$CAF = $json_obj->CAF;
$Date = $json_obj->Date;
$CafDate = date("Y-m-d", strtotime($Date));
$video_decode = base64_decode($Video);
$video_filename = __DIR__ . '/uploads/'. $CAF . '_'.$CafDate.'_VID.mp4';
$video_dbfilename = './uploads/'. $CAF . '_'.$CafDate.'_VID.mp4';
$save_video = file_put_contents($video_filename, $video_decode);
You should not load entire files to the memory when you can't foresee the size or when it's going to handle huge files. It's better to read the file in chunks and process it chunk by chunk.
Here's a quick and dirty example of how to achieve it:
<?php
// open the handle in binary-read mode
$handle = fopen("php://input", "r");
// open handle for saving the file
$local_file = fopen("path/to/file", "w");
// create a variable to store the chunks
$chunk = '';
// loop until the end of the file
while (!feof($handle)) {
// get a chunk
$chunk = fread($handle, 8192);
// here you do whatever you want with $chunk
// (i.e. save it appending to some file)
fwrite($local_file, $chunk);
}
// close handles
fclose($handle);
fclose($local_file);
There are many problems here, with a compounding effect.
$json_str = file_get_contents('php://input');
This reads the entire input into memory, and blocks until it's done. Don't do this.
$json_obj = json_decode($json_str);
If you have 50MB of JSON, you're probably doing something wrong.
$Video = $json_obj->Video;
This is what you're doing wrong... storing a large binary blob in a small structured data serialization format. You realize that the JSON parser has to read and handle the entire thing? Don't send JSON for this!
$video_decode = base64_decode($Video);
Don't use base-64 encoding unless absolutely necessary. It adds 33% overhead to the storage size, as well as CPU for encoding/decoding. This is a complete waste, and is totally unnecessary.
$video_filename = __DIR__ . '/uploads/'. $CAF . '_'.$CafDate.'_VID.mp4';
$video_dbfilename = './uploads/'. $CAF . '_'.$CafDate.'_VID.mp4';
You may have a serious security issue with those two lines. What if someone posts a path of ../../../etc/init.d/something-evil? Don't let the user dictate the filename on disk in any way.
$save_video = file_put_contents($video_filename, $video_decode);
The comment from file_get_contents() applies here in that you should be using this method. Stream the contents to disk instead.
I am using file_put_contents to create a video file. the problem is the speed and performance. It takes about an average of 30 to 60 minutes for an average file size of 50 mb to be created and that is just for one file alone. I am decoding a byte array to create the file. How can I improve the speed and performance?
$json_str = file_get_contents('php://input');
$json_obj = json_decode($json_str);
$Video = $json_obj->Video;
$CAF = $json_obj->CAF;
$Date = $json_obj->Date;
$CafDate = date("Y-m-d", strtotime($Date));
$video_decode = base64_decode($Video);
$video_filename = __DIR__ . '/uploads/'. $CAF . '_'.$CafDate.'_VID.mp4';
$video_dbfilename = './uploads/'. $CAF . '_'.$CafDate.'_VID.mp4';
$save_video = file_put_contents($video_filename, $video_decode);
You should not load entire files to the memory when you can't foresee the size or when it's going to handle huge files. It's better to read the file in chunks and process it chunk by chunk.
Here's a quick and dirty example of how to achieve it:
<?php
// open the handle in binary-read mode
$handle = fopen("php://input", "r");
// open handle for saving the file
$local_file = fopen("path/to/file", "w");
// create a variable to store the chunks
$chunk = '';
// loop until the end of the file
while (!feof($handle)) {
// get a chunk
$chunk = fread($handle, 8192);
// here you do whatever you want with $chunk
// (i.e. save it appending to some file)
fwrite($local_file, $chunk);
}
// close handles
fclose($handle);
fclose($local_file);
There are many problems here, with a compounding effect.
$json_str = file_get_contents('php://input');
This reads the entire input into memory, and blocks until it's done. Don't do this.
$json_obj = json_decode($json_str);
If you have 50MB of JSON, you're probably doing something wrong.
$Video = $json_obj->Video;
This is what you're doing wrong... storing a large binary blob in a small structured data serialization format. You realize that the JSON parser has to read and handle the entire thing? Don't send JSON for this!
$video_decode = base64_decode($Video);
Don't use base-64 encoding unless absolutely necessary. It adds 33% overhead to the storage size, as well as CPU for encoding/decoding. This is a complete waste, and is totally unnecessary.
$video_filename = __DIR__ . '/uploads/'. $CAF . '_'.$CafDate.'_VID.mp4';
$video_dbfilename = './uploads/'. $CAF . '_'.$CafDate.'_VID.mp4';
You may have a serious security issue with those two lines. What if someone posts a path of ../../../etc/init.d/something-evil? Don't let the user dictate the filename on disk in any way.
$save_video = file_put_contents($video_filename, $video_decode);
The comment from file_get_contents() applies here in that you should be using this method. Stream the contents to disk instead.
In the code below I...
open a textfile, write four characters to it, and close it again,
re-open it, read the contents, and use filesize to report the size of the file. (It's 4, as it should be),
manipulate those contents and tack on four more characters as well. Then I write that new string to the textfile and close it again,
use filesize again to report how big the file is.
To my surprise, the answer it gives is still 4 even though the actual size of the file is 8! Examining the contents of the file proves that the write works and the length of the contents is 8.
What is going on??
By the way I have to use fread and fwrite instead of file_get_contents and file_put_contents. At least I think I do. This little program is a stepping stone to using "flock" so I can read the contents of a file and rewrite to it while making sure no other processes use the file in between. And AFAIK flock doesn't work with file_get_contents and file_put_contents.
Please help!
<?php
$filename = "blahdeeblah.txt";
// Write 4 characters
$fp = fopen($filename, "w");
fwrite($fp, "1234");
fclose($fp);
// read those characters, manipulate them, and write them back (also increasing filesize).
$fp = fopen($filename, "r+");
$size = filesize($filename);
echo "size before is: " . $size . "<br>";
$t = fread($fp, $size);
$t = $t[3] . $t[2] . $t[1] . $t[0] . "5678";
rewind($fp);
fwrite($fp, $t);
fclose($fp);
// "filesize" returns the same number as before even though the file is larger now.
$size = filesize($filename);
echo "size after is: " . $size . " ";
?>
From http://php.net/manual/en/function.filesize.php
Note: The results of this function are cached. See clearstatcache() for more details.
When you open a file by fopen() function you can obtain proper size at any time using fstat() function:
$fstat=fstat($fp);
echo 'Size: '.$fstat['size'];
Example:
$filename='blahdeeblah.txt';
$fp=fopen($filename, 'a');
$size=#filesize($filename);
echo 'Proper size (obtained by filesize): '.$size.'<br>';
$fstat=fstat($fp);
echo 'Proper size (obtained by fstat): '.$fstat['size'].'<br><br>';
fwrite($fp, '1234');
echo 'Writing 4 bytes...<br><br>';
$fstat=fstat($fp);
echo 'Proper size (obtained by fstat): '.$fstat['size'].'<br>';
fclose($fp);
$size=#filesize($filename);
echo 'Wrong size (obtained by filesize): '.$size;
Note that cached value is used only in current script. When you run the script again filesize() reads new (proper) size of file.
Example:
$filename='blahdeeblah.txt';
$fp=fopen($filename, 'a');
$size=#filesize($filename);
echo 'Proper size: '.$size.'<br>';
fwrite($fp, '1234');
fclose($fp);
$size=#filesize($filename);
echo 'Wrong size: '.$size;
I have a PHP script that occasionally needs to write large files to disk. Using file_put_contents(), if the file is large enough (in this case around 2 MB), the PHP script runs out of memory (PHP Fatal error: Allowed memory size of ######## bytes exhausted). I know I could just increase the memory limit, but that doesn't seem like a full solution to me--there has to be a better way, right?
What is the best way to write a large file to disk in PHP?
You'll need a temporary file in which you put bits of the source file plus what's to be appended:
$sp = fopen('source', 'r');
$op = fopen('tempfile', 'w');
while (!feof($sp)) {
$buffer = fread($sp, 512); // use a buffer of 512 bytes
fwrite($op, $buffer);
}
// append new data
fwrite($op, $new_data);
// close handles
fclose($op);
fclose($sp);
// make temporary file the new source
rename('tempfile', 'source');
That way, the whole contents of source aren't read into memory. When using cURL, you might omit setting CURLOPT_RETURNTRANSFER and instead, add an output buffer that writes to a temporary file:
function write_temp($buffer) {
global $handle;
fwrite($handle, $buffer);
return ''; // return EMPTY string, so nothing's internally buffered
}
$handle = fopen('tempfile', 'w');
ob_start('write_temp');
$curl_handle = curl_init('http://example.com/');
curl_setopt($curl_handle, CURLOPT_BUFFERSIZE, 512);
curl_exec($curl_handle);
ob_end_clean();
fclose($handle);
It seems as though I always miss the obvious. As pointed out by Marc, there's CURLOPT_FILE to directly write the response to disk.
Writing line by line (or packet by packet in case of binary files) using functions like fwrite()
Try this answer:
$file = fopen("file.json", "w");
$pieces = str_split($content, 1024 * 4);
foreach ($pieces as $piece) {
fwrite($file, $piece, strlen($piece));
}
fclose($file);
In my previous question on this topic, what would the implications be if I removed the dynamic variable and instead replaced it with a static one like you see below...
$source = 'http://mycentralserver.com/protected/myupdater.zip';
I've included the code below for convenience...
<?php
// TEST.PHP
$source = 'http://mycentralserver.com/protected/myupdater.zip';
$target = '.';
$out_file = fopen(basename($source), 'w');
$in_file = fopen($source, 'r');
while ($chunk = fgets($in_file)) {
fputs($out_file, $chunk);
}
fclose($in_file);
fclose($out_file);
$zip = new ZipArchive();
$result = $zip->open(basename($source));
if ($result) {
$zip->extractTo($target);
$zip->close();
}
?>
You should at least be hashing the zip with SHA-1 and checking it against a digest to ensure it hasn't changed. These digests should be extremely hard to replace.
I still think automated updates are a bit iffy.
PHP 5.2.6 and older had a vulnerability that allowed writing to arbitrary locations via Zip's extractTo() -method.
See: http://www.securityfocus.com/bid/32625
So, if the contents of the zip are untrustworthy, you must use PHP 5.2.7 or newer (or use your own Zip parser).