I have a process that uploads files via PHP but the resulting files end up being 2 bytes larger than the source file. I'm not sure where these 2 bytes are coming from. (the actual process is a chunked upload where I slice up a file and upload the slices, each slice winds up arriving 2 bytes longer than it started, but I've tested with a single small file and it too arrives 2 bytes larger than the source).
I'm attaching my PHP... Is this a normal feature of PHP? I'm imagining some sort of null terminator or something (there does appear to be a \n at the end of each file that wasn't there initially). Do I need to read the file into a buffer and get rid of the last two bytes before reassembling my original? I have to imagine I'm doing something wrong, but I'm confounded as to what it would be.
If I do need to manually strip off those last two bytes what's the correct way to do that (it's a binary file) and then append the rest to the overall file I'm rebuilding?
EDIT
Each uploaded file is getting a 0D0A word added to the end as PHP saves it to the server. So... I guess the question is how to prevent this from happening.
<?PHP
$target_path = $_REQUEST[ 'path' ];
$originalFileName = $_REQUEST['original_file_name'];
$target_path = $target_path . basename( $_FILES[ 'Filedata' ][ 'name' ] );
if ( move_uploaded_file( $_FILES[ 'Filedata' ][ 'tmp_name' ], $target_path ) )
{
$newFilePath = $originalFileName; //this is the overall file being re-assembled
$fh = fopen($newFilePath, 'ab') or die("can't open file");
$nextSlice = file_get_contents($target_path); //this is the slice that's 2 bytes too big each time
fputs($fh, $nextSlice);
fclose($fh);
// unlink($target_path); //normally I'd delete the slice at this point, but I'm hanging on to it while I figure out where the heck the 2 extra bytes are coming from.
fclose($fh);
echo "SUCCESS";
}
else
{
echo "FAIL:There was an error uploading the file, please try again!";
}
?>
Is the file binary? I'm thinking that file_get_contents is causing problems because it's treating it like a string. Maybe you should try fread instead?
The solution turns out to be this:
fwrite($fh, $GLOBALS["HTTP_RAW_POST_DATA"]);
I may be doing something wrong in my request that when I use the method I described in the question, the file gets written with the extra 0D0A at the end, but the above method for extracting the data has it arriving intact and exactly the right length.
Related
This is the case:
I have a 2Gb dump file called myDB.sql. It is a dump file that deletes a existing database and creates a new one, with views and triggers. So I have the string myDB_OLD spread for many lines of the code.
I would like to change these strings occurrences to myDB_NEW.
I could do this easelly using notePad++. But notepad does not opens a 2Gb file.
What I did is a PHP code that reads line by line and find and replace the string I want.
This is the code:
$myfile2 = fopen("myDB.sql", "r") or die("Unable to open file!");//Reads the file
while (!feof($myfile2)) {//Pass trough each line
$str=fgets($myfile2);//Store the line inside a variable
if (preg_match("/myDB_OLD/",$str)) {//If the string I want to change exists - replace it and conacatenate
$newStr .= str_replace("myDB_OLD","myDB_NEW",$str);
}
else {//If not concatenate it
$newStr .=$str;
}
}//End of while
fclose($myfile2);
//Save the newStr in a new file named
$myfile = fopen("newDB.sql", "w") or die("Unable to open file!");
fwrite($myfile, $newStr);
echo "finished";
This code retrieve each line of the file changes the string, concatenate in variable and creates a new file. It should works but it is not. I dont know why. I am using xdebug to figure out what is the issue, but no luck.
So I change the approach.
Instead of save each line in a variable, I save it directly in a file and that works good.
This is the new code:
$myfile = fopen("newDB.sql", "w") or die("Unable to open file!");//Creates a new file "newDB.sql"
$myfile2 = fopen("myDB.sql", "r") or die("Unable to open file!");//Reads the file
while (!feof($myfile2)) {//Pass trough each line
$str=fgets($myfile2);//Store the line inside a variable
if (preg_match("/myDB/",$str)) {//If the string I want to change exists - replace it . (Do not concatenate)
$strRep=str_replace("myDB","myDB_NEW",$str);
}
else {
$strRep =$str;
}
fwrite($myfile, $strRep);// Add the new line to the file "newDB.sql"
}//End of while
fclose($myfile);
fclose($myfile2);
echo "finished";
Ok, I solved my issue but it raises a thought. What is the issue of the first code?
I think the issue is the amount of information to be stored in a PHP variable, 2Gb.
So, is there a limit in size to a PHP variable to stores value, in this case, a string ?
If yes how can I check or change it?
Any php.ini variable?
So, is there a limit in size to a Php variable to stores value, in this case, a string ?
Yes. A string can be as large as up to 2GB (2147483647 bytes maximum). You can override this limit by increasing memory_limit directive in php.ini.
From php7 there's not that limitation in a 64 bit system:
Support for strings with length >= 2^31 bytes in 64 bit builds.
The maximum number of bytes a script is allowed to allocate is set in php.ini. Look for memory_limit. This is 16Mb after PHP 5.2.0 as a default!
This can be changed by doing a:
ini_set('memory_limit','24M');
The maximum size of a string is 2 GB in PHP, probably because of adressing limitations, if allowed by memory_limit.
Is there any way to detect if a gz file is corrupt in PHP?
I'm currently using http://www.php.net/manual/de/function.gzread.php#110078 to determine the file size and read the whole* file via
$zd = gzopen ( $file, "r" );
$contents = gzread ( $zd, $fzip_size );
gzclose ( $zd );
Unfortunately some gz files are corrupted and the last 4 bytes do not represent the real length of the gz file. As long as the number is negativ I'm able to tell that something is wrong, but sometimes it's positive (and very large) which leads to an out of memory error. How can I check in advance if the file is corrupted?
I'm reading the whole file because I found no working way to read the file line-by-line without knowing the size of the longest line - which led (in some case) to lines that were not complete.
If you can use linux gzip command it will be very simply to find if file is wrong or not. gzip -t will display no message if file is valid.
if (`gzip -t $file 2>&1`) {
echo "An error occured";
}
I'm looking for the most efficient way to write the contents of the PHP input stream to disk, without using much of the memory that is granted to the PHP script. For example, if the max file size that can be uploaded is 1 GB but PHP only has 32 MB of memory.
define('MAX_FILE_LEN', 1073741824); // 1 GB in bytes
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR.'/'.$MyTempName.'.tmp', 'w');
fwrite($hDest, fread($hSource, MAX_FILE_LEN));
fclose($hDest);
fclose($hSource);
Does fread inside an fwrite like the above code shows mean that the entire file will be loaded into memory?
For doing the opposite (writing a file to the output stream), PHP offers a function called fpassthru which I believe does not hold the contents of the file in the PHP script's memory.
I'm looking for something similar but in reverse (writing from input stream to file). Thank you for any assistance you can give.
Yep - fread used in that way would read up to 1 GB into a string first, and then write that back out via fwrite. PHP just isn't smart enough to create a memory-efficient pipe for you.
I would try something akin to the following:
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR . '/' . $MyTempName . '.tmp', 'w');
while (!feof($hSource)) {
/*
* I'm going to read in 1K chunks. You could make this
* larger, but as a rule of thumb I'd keep it to 1/4 of
* your php memory_limit.
*/
$chunk = fread($hSource, 1024);
fwrite($hDest, $chunk);
}
fclose($hSource);
fclose($hDest);
If you wanted to be really picky, you could also unset($chunk); within the loop after fwrite to absolutely ensure that PHP frees up the memory - but that shouldn't be necessary, as the next loop will overwrite whatever memory is being used by $chunk at that time.
A trivial use of PHP and frwite() to create/write to a text file.
However, is there a way to write a very large text string to a file using fwrite?()? I assume there is, and that it involves some form of buffer management. The PHP docs don't seem to have this covered.
Sample code:
$p = "Some really large string ~ 100-250K in size"
$myFile = "testp.txt";
$fh = fopen($myFile, 'w') or die("can't open file");
set_file_buffer($fh, 1000000);
fwrite($fh, $p);
fclose($fh);
Believe it or not, this simply gets a file with the name of the file inside the file.
Using a much smaller text string, it works as expected. Pointers to what I should do would be useful.
UPDATE:
Some of you are missing that I did try the above with a string of ~100K, and it didn't work. All I got in the output file was the name of the file!!!
thanks
::: 2ND UPDATE....
never mind.. the whole thing was user error... god i need a drink... or sleep!
thanks
php/fwrite works as i thought it would/should.. nothing to see here..!
There is no limit on how much data can be written to a stream (a file handle) in PHP and you do not need to fiddle with any buffers. Just write the data to the stream, done.
My PHP web application has an API that can recieve reasonably large files (up to 32 MB) which are base64 encoded. The goal is to write these files somewhere on my filesystem. Decoded of course. What would be the least resource intensive way of doing this?
Edit: Recieving the files through an API means that I have a 32MB string in my PHP app, not a 32 MB source file somewhere on disk. I need to get that string decoded an onto the filesystem.
Using PHP's own base64_decode() isn't cutting it because it uses a lot of memory so I keep running into PHP's memory limit (I know, I could raise that limit but I don't feel good about allowing PHP to use 256MB or so per process).
Any other options? Could I do it manually? Or write the file to disk encoded and call some external command? Any thought?
Even though this has an accepted answer, I have a different suggestion.
If you are pulling the data from an API, you should not store the entire payload in a variable. Using curl or other HTTP fetchers you can automatically store your data in a file.
Assuming you are fetching the data through a simple GET url:
$url = 'http://www.example.com/myfile.base64';
$target = 'localfile.data';
$rhandle = fopen($url,'r');
stream_filter_append($rhandle, 'convert.base64-decode');
$whandle = fopen($target,'w');
stream_copy_to_stream($rhandle,$whandle);
fclose($rhandle);
fclose($whandle);
Benefits:
Should be faster (less copying of huge variables)
Very little memory overhead
If you must grab the data from a temporary variable, I can suggest this approach:
$data = 'your base64 data';
$target = 'localfile.data';
$whandle = fopen($target,'w');
stream_filter_append($whandle, 'convert.base64-decode',STREAM_FILTER_WRITE);
fwrite($whandle,$data);
fclose($whandle);
Decode the data in smaller chunks. Four characters of Base64 data equal three bytes of “Base256” data.
So you could group each 1024 characters and decode them to 768 octets of binary data:
$chunkSize = 1024;
$src = fopen('base64.data', 'rb');
$dst = fopen('binary.data', 'wb');
while (!feof($src)) {
fwrite($dst, base64_decode(fread($src, $chunkSize)));
}
fclose($dst);
fclose($src);
It's not a good idea transfer 32Mb string. But I have a solution for my task, that can accept any size files from browser to app server. Algorithm:
Client
Javascript: Read file from INPUT with FileReader and readAsDataURL() to FILE var.
Cut all of data in FILE from start to first "," position Split it with array_chunks by max_upload_size/max_post_size php var.
Send chunk with UID, chunk number and chunks quantity and wait for response, then send another chunk one by one.
Server Side
Write each chunk until the last one. Then do base64 with streams:
$src = fopen($source, 'r');
$trg = fopen($target, 'w');
stream_filter_append($src, 'convert.base64-decode');
stream_copy_to_stream($src, $trg);
fclose($src);
fclose($trg);
... now you have your base64 decoded file in $target local path. Note! You can not read and write the same file, so $source and $target must be different.