I am having a problem with an echo/print that returns a large amount of data. The response is broken and is as follows:
end of data
http response header printed in body
start of data
I am running the following script to my browser to replicate the problem:
<?php
// Make a large array of strings
for($i=0;$i<10000;$i++)
{
$arr[] = "testing this string becuase it is must longer than all the rest to see if we can replicate the problem. testing this string becuase it is must longer than all the rest to see if we can replicate the problem. testing this string becuase it is must longer than all the rest to see if we can replicate the problem.";
}
// Create one large string from array
$var = implode("-",$arr);
// Set HTTP headers to ensure we are not 'chunking' response
header('Content-Length: '.strlen($var));
header('Content-type: text/html');
// Print response
echo $var;
?>
What is happening here?
Can someone else try this?
You might have automatic output buffering activated on your server. If the buffer overflows, it just starts pooping out the rest of the data instead.
Note that something like gzip compression also implicitly buffers the output. If it's the case, an ob_end_flush() call after the headers should solve it.
Browsers often limits the characters you're allowed to pass through a get variable.
To work around this, you could base 64 encode the string, and then decode it, once you recievede the respone.
I think there's javascript base 64 encode libraries available.
Like this one:
http://www.webtoolkit.info/javascript-base64.html
Related
i know those methods related to output buffering:
<?php
ob_start();
// some code
ob_end_flush();
ob_end_clean();
but i can't figure out why will i need the length of the output buffer. there is one question here is SOF about measuring the bandwidth of the user ,although that calculation will need to measure pictures and other files which is not addressed.
i did check the PHP manual and the "why" is not there.
Answering "why to use it". In order to answer this, we must first understand why we need to use ob_start.
Usually it is used for two purpose:
When we want to send HTTP headers, but we are not sure, that there is no rendering before(the first output will trigger header sending, so if you use, echo for example, before header, the later won't work)
When we want to send our output by chunks for performance. (your code can be written with a lot of echo or print, and usually they outputing only small data, so you are sending small portions of data back to browser, which is affecting performance.)
So now you can start to guess, why we need to use ob_get_length(). For example: i want to send my output by large chunks, so i am enabling output buffering and with the help of ob_get_lengthi can control how many data i can write to buffer, before sending it.
ob_get_length will return the length of the contents in the output buffer..
Here is simple example..
<?php
ob_start();
echo "Hello ";
$len1 = ob_get_length();
echo "World";
$len2 = ob_get_length();
ob_end_clean();
echo $len1 . ", ." . $len2;
?>
The above example will output: 6, 11
Updated:
ob_get_length() is helpful if you want to send a custom HTTP Content-Length header - although that is for advanced users only!
I wrote a small WebSocket library a while back, and found adding gzip support surprisingly easy. I didn't fully realize at the time that the deflate_init() / deflate_add() / inflate_init() / inflate_add() functions were actually PHP 7-only, and now I'd like to be able to run my WebSocket server under PHP 5 environments.
My problem is, deflate_add() produces output that differs slightly from gzdeflate() - by one character in the testcase below.
The deflate_add()/inflate_add()-based approach works perfectly in-browser, so the output of gzdeflate() is the incorrect one. I'm guessing gzdeflate()/gzinflate() are using zlib with different underlying options - something related to stream state, maybe? - and that's causing everything to fall apart.
Ultimately I want to know if I can convince PHP 5-era zlib functions to output "correct" deflated data.
First of all, the deflate_init()/deflate_add()-based approach I used on PHP 7:
$data = "ABC";
$ctx = deflate_init(ZLIB_ENCODING_RAW);
// unfortunately I can't find the gigantic blog post with example code
// that I learned from :(, but it contained the Ruby equivalent of the
// the substr() below. I blinked at it a bit but apparently this is how
// it's done.
$deflated = substr(deflate_add($ctx, $data, ZLIB_SYNC_FLUSH), 0, -4);
// $deflated is now "rtr\6\0"
$ictx = inflate_init(ZLIB_ENCODING_RAW);
$data2 = inflate_add($ictx, $deflated, ZLIB_NO_FLUSH);
// $data2 is now "ABC"
Here's what happens if I use gzdeflate()/gzinflate():
$data = "ABC";
$deflated = gzdeflate($data, 9, ZLIB_ENCODING_RAW);
// $deflated is now "str\6\0"
$output = gzinflate($deflated);
// $output is now "ABC"
Trying to gzinflate() the output of inflate_add() produces a data error. As a TL;DR:
print gzinflate("rtr\6\0")."\n"; // will bomb out
print gzinflate("str\6\0")."\n"; // prints "ABC"
What you are calling correct is incorrect, and what you are calling incorrect is correct.
With deflate_add you are deliberately creating an unterminated, i.e. invalid, deflate stream. Why, I have no idea. (Nor, apparently, do you, since this came from a "gigantic blog post" that you cannot find.) This is being done with the ZLIB_SYNC_FLUSH which completes the current deflate block and appends an empty stored block. The substr(,,-4) is removing most of that empty stored block at the end, leaving you with an incomplete, invalid inflate stream, prematurely ending in the middle of a stored block.
gzdeflate on the other hand is creating a properly terminated deflate stream, with a single deflate block marked as the last block. The only difference between the two streams is the first (least significant) bit, which is a 1 to mark the last block.
You do not say how the properly terminated deflate stream is "causing everything to fall apart". In any case, you can make a properly terminated deflate stream with deflate_add by using ZLIB_FINISH instead of ZLIB_SYNC_FLUSH, and forgoing the substr.
There is no way to make an invalid deflate stream with gzdeflate, if that's what you're asking. You can't just change the first bit, since for a larger string, the last block may not be the first block.
I found few articles on the Internet that are suggesting use of ob_ functions, all of these emphase benefits, and there are no downsides of using functions mentioned.
My question is what are the downsides of using ob_ functions, or setting ini_set('output_buffering', '1'); ?
The cons of using output buffering entirely depend on the context of your usage.
One of the biggest cons of output buffering is your runtime error messages or warnings may get suppressed, and you may sometimes end up with erroneous data.
Consider this example:
<?php
function render_template() {
ob_start();
// Do some processing
fetch_template_and_render();
do_render();
// end capture
$output = ob_get_clean();
return $output;
}
memchace::set( $some_key, render_template() );
?>
If either of fetch_template_and_render or do_render throw run time errors, they will get dumped into your output, and eventually in this example will end up in the database or cache.
Here are 2 snippets that demonstrate what I mean which you can try for yourself
#1
<?php
echo 1/0;
?>
outputs
Warning: Division by zero on line 1
#2
<?php
ob_start();
echo 1/0;
$var = ob_get_clean();
?>
outputs nothing.
To avoid such cases, you will need to be diligent about error checking and take precautions.
When used diligently, ob_* functions are very powerful and super useful.
There are no major drawbacks, with proper implementation, to the use of output buffering.
Output buffering can allow errors/warnings/notices (except stop errors) to appear in the output without being readily apparent. This is typically resolved with proper error checking, better configuration of the php environment and the implementation of a good error handler (such as one that converts errors to ErrorExceptions which can be caught with try/catch - see Whoops! as an example of an error handler using ErrorExceptions).
Memory can possibly be a drawback, but the output size is typically insignificant for most scripts. An exception to this may be when sending large amounts of data, such as using fpassthru to deliver file content. This can be resolved by turning off the output buffering (ob_end_clean or ob_end_flush) before writing this content to the output.
Memory consumption is the most important drawback. I've recently built a PHP script that outputs a large chunk of XML data of several megabytes. The framework this 'page' was part of used output buffering. With output buffering, you need a memory buffer large enough to contain all the data. In my case it wasn't and the script failed.
If you output the data directly to the client, you don't have this problem. This is escpecially important in cases like this and when throughputting files. In case of generating a 'normal' HTML page, you will likely not use up the whole buffer, although you still need a lot of memory if you have many simultaneous requests.
Without buffering, the data is gone and doesn't trouble your server anymore. As long as the data is buffered, it can be changed or flushed, but actually puts a load on your server.
Here's a pretty good use for ob_ functions:
ob_start("ob_gzhandler");
Provided that the zlib extension is enabled in PHP, that should ensure your output is gz-compressed. It noticably speeds up page transfers for large pages.
I faced a strange issue today.
For several months I used buffer flushing in PHP to send small string sizes to the client without problems.
Today I returned to the project and it turned out that my server won't send strings smaller than 512 bytes.
Here is my code:
<?php
echo "length:".$myUpcomingStringSize;
ob_flush();
flush();
sleep(1);
for($i = 0; $i < count($allLines); $++) {
echo $allLines[$i];
ob_flush();
flush();
}
?>
This code worked like charm the whole last year. And now it doesn't anymore. I played around a bit and added some random characters. As the string size gets equal or greater 512, the server sends the buffer content.
Can anybody imagine the issue I have to solve here? Anyone else facing this issue? Or does
someone know how to configure this minimum packet size?
If you changed neither the program nor the server, you should assume that the program never worked as intended. Especially Windows systems are known to buffer the output until a certain number of Bytes is in the output buffer. This buffering is at system-level and thus can not be affected by any PHP configuration.
If you know that 512 Bytes is the minimum required for the output buffer to send, then you could use something like
define('MIN_OUTPUT_LENGTH', 512);
echo str_pad("length: $myUpcomingStringSize", MIN_OUTPUT_LENGTH, "\0"), '\n';
// (If you run into trouble with the null-bytes, use space character instead)
Notes
If you do not use "userspace" output buffering, then ob_flush(); is redundant.
If there is no delay in your for loop, then flushing between lines is not a good idea. Especially for mobile applications where the network tries to pack as much data as possible into a single packet.
There is a syntax error in your for loop header (The expression $++ is missing a variable identifier, probably i)
I'm trying to mimick an application that sends octet streams to and from a server. The data contained in the body looks like raw bytes, and I'm fairly certain the data being sent for each command is static, so I'm hoping to map the bytes to something more readable in my application. For example, I'll have an array that does: "test" => "&^D^^&#*#dgkel" So I can call "test" and get the real bytes that need to be sent. Trouble is, PHP seems to convert these bytes. I'm not sure if it is an encoding problem or what, but what has been happing is I'll give it some bytes (for example, �ھ����#�qs��������������������X����������������������������) which has a length of 67 I believe, but PHP will say (when I do a var_dump of the HTTP request) that the headers sent contained "Content-Length: 174" or something close to that and the bytes will look like �ھ����#�qs��������������������X����������������������������
So I'm not really sure how to fix this.. Anyone have any ideas? Cheers!
Edit, a little PHP:
$request = new HttpRequest($this->GetMessageURL(), HTTP_METH_POST);
$request->addHeaders($headers);
$request->addRawPostData($buttonMapping[$button]);
$request->send();