I'm trying to encode/decode a file that is several MBs or sometimes GBs in base64 encoding however some pieces of data gets encoded/decoded in a wrong way which results in strange characters like: � �̴.
I'm reading the file chunk by chunk encoding and saving each individually (Probably that's the problem however i cannot figure it out).
Here is what i have tried so far:
<?php
function encode_file($Ifilename, $Efilename){
$handle = fopen($Ifilename, 'rb');
$outHandle = fopen($Efilename, 'wb');
$bufferSize = 8151;
while(!feof($handle)){
$buffer = fread($handle, $bufferSize);
$ebuffer = base64_encode($buffer);
fwrite($outHandle, $ebuffer);
}
fclose($handle);
fclose($outHandle);
}
function decode_file($Ifilename, $Efilename){
$handle = fopen($Ifilename, 'rb');
$outHandle = fopen($Efilename, 'wb');
$bufferSize = 8151;
while(!feof($handle)){
$buffer = fread($handle, $bufferSize);
$dbuffer = base64_decode($buffer);
fwrite($outHandle, $dbuffer);
}
fclose($handle);
fclose($outHandle);
}
encode_file('input.txt', 'out.bin');//Big text file ~4MBs
decode_file('out.bin', 'out.txt');
After reading the whole Wikipedia article on base64, I found that every 3 characters encodes to 4 base64 characters, this is what was causing the file corruption.
The fix is to simply set the buffer to n when encoding, where n is a multiple of 3.
When decoding set the buffer to N, where N is a multiple of 4.
The working code:
<?php
function encode_file($Ifilename, $Efilename){
$handle = fopen($Ifilename, 'rb');
$outHandle = fopen($Efilename, 'wb');
$bufferSize = 3 * 256;// 3 bytes of ASCII encodes to 4 bytes of base64
while(!feof($handle)){
$buffer = fread($handle, $bufferSize);
$ebuffer = base64_encode($buffer);
fwrite($outHandle, $ebuffer);
}
fclose($handle);
fclose($outHandle);
}
function decode_file($Ifilename, $Efilename){
$handle = fopen($Ifilename, 'rb');
$outHandle = fopen($Efilename, 'wb');
$bufferSize = 4 * 256; // 4 bytes of base64 decodes to 3 bytes of ASCII
while(!feof($handle)){
$buffer = fread($handle, $bufferSize);
$dbuffer = base64_decode($buffer);
fwrite($outHandle, $dbuffer);
}
fclose($handle);
fclose($outHandle);
}
encode_file('input.txt', 'out.bin');
decode_file('out.bin', 'output.txt');
Related
I need to base64 encode big file with PHP.
file() and file_get_contents() are not options since them loads whole file into memory.
I got idea to use this:
$handle = #fopen("/tmp/inputfile.txt", "r");
if ($handle) {
while (($buffer = fgets($handle, 4096)) !== false) {
echo $buffer;
}
fclose($handle);
}
SOURCE: Read and parse contents of very large file
This working well for reading, but is it possible to do it like this:
Read line -> base64 encode -> write back to file
And then repeat for each line in file.
Would be nice if it could do it directly, without need to write to temporary file.
Base64 encodes 3 bytes of raw data into 4 bytes on 7-bit safe text. If you feed it less than 3 bytes padding will occur, and you can't have that happen in the middle of the string. However, so long as you're dealing in multiples of 3 you're golden, sooo:
$base_unit = 4096;
$handle = #fopen("/tmp/inputfile.txt", "r");
if ($handle) {
while (($buffer = fread($handle, $base_unit*3)) !== false) {
echo base64_encode($buffer);
}
fclose($handle);
}
http://en.wikipedia.org/wiki/Base64#Examples
I tried to stream an audio file using this PHP code, but always get cut around 3/4 of the file size, especially for iPad and Android agents.
ob_clean();
flush();
set_time_limit(0);
$size = intval(sprintf("%u", filesize($filename)));
$chunksize = 0.5 * (1024 * 1024);
if ($size > $chunksize) {
$handle = fopen($filename, 'rb');
$buffer = '';
while (!feof($handle)) {
$buffer = fread($handle, $chunksize);
echo $buffer;
ob_flush();
flush();
}
fclose($handle);
} else {
readfile($filename);
}
Is there a better way to do this? I have tried completely with readfile() only, the result is even worse. Thanks in advance.
I have got a large file in PHP of which I would like to replace the first 512 bytes with some other 512 bytes. Is there any PHP function that helps me with that?
If you want to optionally create a file and read and write to it (without truncating it), you need to open the file with the fopen() function in 'c+' mode:
$handle = fopen($filename, 'c+');
PHP then has the stream_get_contents() function which allows to read a chunk of bytes with a specific length (and from a specific offset in the file) into a string variable:
$buffer = stream_get_contents($handle, $length = 512, $offset = 0);
However, there is no stream_put_contents() function to write the string buffer back to the stream at a specific position/offset. A related function is file_put_contents() but it does not allow to write to a file-handle resource at a specific offset. But there is fseek() and fwrite() to do that:
$bytes_written = false;
if (0 === fseek($handle, $offset)) {
$bytes_written = fwrite($handle, $buffer, $length);
}
Here is the full picture:
$handle = fopen($filename, 'c+');
$buffer = stream_get_contents($handle, $length = 512, $offset = 0);
// ... change $buffer ...
$bytes_written = false;
if (0 === fseek($handle, $offset)) {
$bytes_written = fwrite($handle, $buffer, $length);
}
fclose($handle);
If the length of $buffer is not fixed this will not properly work. In that case it's better to work with two files and to use stream_copy_to_stream() as outlined in How to update csv column names with database table header or if the file is not large it is also possible to do that in memory:
$buffer = file_get_contents($filename);
// ... change $buffer ...
file_put_contents($filename, $buffer);
In PHP if you write to a file it will write end of that existing file.
How do we prepend a file to write in the beginning of that file?
I have tried rewind($handle) function but seems overwriting if current content is larger than existing.
Any Ideas?
$prepend = 'prepend me please';
$file = '/path/to/file';
$fileContents = file_get_contents($file);
file_put_contents($file, $prepend . $fileContents);
The file_get_contents solution is inefficient for large files. This solution may take longer, depending on the amount of data that needs to be prepended (more is actually better), but it won't eat up memory.
<?php
$cache_new = "Prepend this"; // this gets prepended
$file = "file.dat"; // the file to which $cache_new gets prepended
$handle = fopen($file, "r+");
$len = strlen($cache_new);
$final_len = filesize($file) + $len;
$cache_old = fread($handle, $len);
rewind($handle);
$i = 1;
while (ftell($handle) < $final_len) {
fwrite($handle, $cache_new);
$cache_new = $cache_old;
$cache_old = fread($handle, $len);
fseek($handle, $i * $len);
$i++;
}
?>
$filename = "log.txt";
$file_to_read = #fopen($filename, "r");
$old_text = #fread($file_to_read, 1024); // max 1024
#fclose(file_to_read);
$file_to_write = fopen($filename, "w");
fwrite($file_to_write, "new text".$old_text);
Another (rough) suggestion:
$tempFile = tempnam('/tmp/dir');
$fhandle = fopen($tempFile, 'w');
fwrite($fhandle, 'string to prepend');
$oldFhandle = fopen('/path/to/file', 'r');
while (($buffer = fread($oldFhandle, 10000)) !== false) {
fwrite($fhandle, $buffer);
}
fclose($fhandle);
fclose($oldFhandle);
rename($tempFile, '/path/to/file');
This has the drawback of using a temporary file, but is otherwise pretty efficient.
When using fopen() you can set the mode to set the pointer (ie. the begginng or end.
$afile = fopen("file.txt", "r+");
'r' Open for reading only; place
the file pointer at the beginning of
the file.
'r+' Open for reading and
writing; place the file pointer at the
beginning of the file.
$file = fopen('filepath.txt', 'r+') or die('Error');
$txt = "/n".$string;
fwrite($file, $txt);
fclose($file);
This will add a blank line in the text file, so next time you write to it you replace the blank line. with a blank line and your string.
This is the only and best trick.
I am using a variation of the familiar readfile_chunked in attempt of download for larger files:
function readfile_chunked($filename)
{
$chunk_size = 1*(1024*1024); // how many bytes per chunk
$buffer = '';
$handle = fopen($filename, 'rb');
if ($handle === false)
{
return false;
}
while (!feof($handle))
{
$buffer = fread($handle, $chunk_size);
print $buffer;
ob_flush();
flush();
sleep(1);
}
$status = fclose($handle);
return $status;
}
smaller files work fine, but this larger file is missing the last 2830 bytes.
i found out the issue to this. under the php.ini file, make sure you set implicit_flushing to On. i still have the explicit flush code after each line outputted however.