php inserts HEX number of characters before the content - php

I'm moving a website to a new server. (Old server had php 5.3.2, the new one has php 5.5.9)
Centos, httpd Apache/2.2.26.
I've copied files and it works fine except the only weird thing:
some strange HEX number is inserted before the content of pages:
Also, in the bottom of the page, 0 is inserted after the </html> tag.
I've noticed two things:
1) In my case only two headers are sent from php script:
header("HTTP/1.1 200 OK");
header("Status: 200");
If I comment the first header than It will be ok - no strange numbers.
2) It looks like that number is the number of characters on the page (I've checked it).
And if the page is less then 8000 characters, than the number doesn't appear, but if page has 8001 characters than 1F41 appears
P.S. I was advised to remove all BOM from files. Files were OK - already without BOM. So it's not about BOM.
UPD:
I made a very simple test (index.php):
<?php header("HTTP/1.1 200 OK"); ?>
Lorem Ipsum ... 8000 characters
Everything is OK.
<?php header("HTTP/1.1 200 OK"); ?>
Lorem Ipsum ... 8001 characters
Bug happens 1f41 before Lorem Ipsum.

This is not PHP nor a BOM. You have a problem with Content-Transfer-Encoding.
The server is sending along a Chunked encoding (this is usually done when the Content-Length is unavailable) which the client apparently doesn't know how to handle, or the header combination makes the client believe that it can bypass dechunking.
Therefore, what is actually the "length of next chunk" gets interpreted by the client, and you see it as a hex bit before the content:
05
These
05
Are
03
the
1F
first characters of the senten
03
ce.
instead of
Content-Length: 48
These are the first characters of the sentence.
(I calculated the lengths by eye, they're probably wrong)
The likely cause is that you have some kind of buffering which interferes with Content Encoding. If everything stays in the buffer, all well and good, a Content-Length is available, gets sent and Bob's your uncle. But if you send more than the 8000 bytes of the buffer, the buffer is flushed and something untoward happens. Try looking in the documentation for zlib and output buffering, you might have some conflict in php.ini between what Apache does and what PHP does.
Interesting links that show how exactly 8000 is (was?) a buffer size used in some portions of apache:
https://bugs.php.net/bug.php?id=45945
https://serverfault.com/questions/366996/how-can-the-apache-2-2-deflate-module-length-limit-be-increased
This last link suggests me to suggest you to try checking whether zlib, mod_gzip, ob_gzhandler and/or mod_deflate are enabled, and possibly conflicting, or to try exchanging the one for the other.
Update (as per comment)
Adding the appropriate Transfer-Encoding header fixes the problem.
So what really happened? We know that the output was correctly chunked-encoded (i.e. the chunking itself was correct), and the client was able to interpret it after being told to. So what was missing was simply the knowledge of the content being chunked, which looks absurd and a good bit buggy: for whatever chunked the content had to know it would get chunked (d'oh!), so it had responsibility of adding the appropriate header, and yet it did not (or it did, but something else stripped it), until the OP rectified the situation adding a header himself by hand.
The problem is now solved, but I think there must be still a bug lingering somewhere in the workflow, either in the modules, the application, or in how the headers are processed.

As discussed in the PHP chat room yesterday 9F 1A is the BOM for UTF-16BE Encoding
In my case only two headers are sent from php script:
header("HTTP/1.1 200 OK"); header("Status: 200");
What's going here? Why are you setting both? One is a proper http response, the other is a CGI response. You shouldn't need to set either, as PHP will default to a 200 OK if the script completes. And setting a CGI style header is probably wrong, unless you're definitely running PHP in CGI mode through Apache.

Had this problem for serving HTML as well as plain text.
EXPLICITLY setting header Transfer-Encoding to "chunked"
header("Transfer-Encoding: chunked");
indeed solved the problem (not sure how reliably though).
That Transfer-Encoding header was being added automatically even if i didn't set it - but the problem was still there. Setting the header explicitly in PHP code solved it, but not exactly - see below. Tested using websniffer.cc, PHP 7.4.9, CentOS 7.8
UPDATE: adding the header might confuse some clients. https://securityheaders.com/ failed to connect to the site with this header explicitly set on (regardless of content length)

Related

BOM being added to any return or die response

I'm using jQuery to retrieve a json response from an endpoint
die(json_encode(array('success' => 3, 'message' => 'You must use at least 1 credit or more.')));
Whenever I check the JSON response received in chrome developer tools I'm getting a red dot showing \ufeff is being added before the json response. I've encoded the PHP file with UTF-8 in Notepad++ however it still adds the BOM character infront of any response. If I return anything or change the die it will still show the BOM character in the response.
I've tried the same file on my localhost and it works absolutely fine however on the server it adds the character.
I'm at a loss as to what's causing the issue, any help would be greatly appreciated.
This is an 13 year old issue
There are workarounds (removing BOM from all PHP files, ob_clean at script start), but the real solution is to have a PHP compiled with --enable-zend-multibyte or --enable-mbstring, or wait until it is fixed by the PHP team.
As you sometimes have no control over the PHP version and compilation flags on hosted environments, I prefer removing BOMs from all PHP files, to prevent this kind of issues. This will work on any server.
Your solution is to fix the output with JS. But for other usages, e.g. generating an image or other binary data via PHP, or sending headers, you cannot solve this way.
It seems it was specifically an issue with this server configuration as it works on other servers. For the meantime I've filtered the response to remove any BOM chracters using javascript before parsing the JSON response.

How to prevent apache from sending connection close header <s>on 304 answers</s>

I have the following setup:
Some files are dynamically generated dependent on some (only a few) session parameters. Since they have not a great diversity, i allow caching in proxys/browsers. The files get an etag on their way, and the reaction of the whole web application at first glance seems correct: Files served in correct dependence from session situations, traffic saved.
And then this erroneous behavior:
But at closer inspection, i found that in his answer in case of a 304 for those dynamically generated files, apache wrongly sends a "Connection: close" Header instead of the normally sent "Connection: KeepAlive". What he should do is: Simply do not manipulate anything concerning "connection".
I cannot find any point where to pinpoint the cause of this behavior: Nowhere in the apache config files is anything written except one single line in one single file where it is instructed to send a keepalive - which it does - as long as it does not send a 304 response for a dynamically generated file. Nowhere in PHP do i instruct that guy to send anything other than keepalives (and the latter only to try to counter the connection:close).
The apache does not do this when it serves "normal" (non-dynamic) files (with 304 answers). So in some way i assume that maybe the PHP kernel is the one who interferes here without permission or being asked. But then, an added "Header set Connection 'Keep-Alive'" in the apache config, which i too added to counter the closing of the connection, does not work, too. Normally, when you put such a header set rule (not of "early" type) in the apache config, this rules takes action AFTER finalization of any subordered work on the requested document (thus AFTER finalization of the PHP output). But in my case, nothing happens - well: in case of a 304 response. In all other cases, everything works normal and correct.
Since there do some other files go over the line at a page request, i would appreciate to get the apache rid of those connection-closures.
Is there anybody who has an idea what to do with this behavior?
P.S.: One day (and a good sleep) later, things are clearing:
The culprit in this case was a shortsightedly (on my behalf) copied example snippet, which had "HTTP/1.>>>0<<< 304" (the Null!) in it.
This protocol version number gets (correctly) post-processed by apache (after everything otherwise - including any apache modules work - got finalized), in that it decides not to send a "Connection: Keep-Alive" over the wire, since that feature didn't exist in version HTTP/1.0.
The problem in this case was to get the focus on the fact that everything inside php and apache modules worked correctly and something in the outer environment of them must have been wrong, and thereafter to shift the view to anything in the code that could possibly influence that outer environment (e.g. the protocol version).

Why does XSendfile emit intermittent garbled responses when used with Symfony Components BinaryFileResponse class?

Background
Part of my application's responsibility is handling requests for static resources (CSS, JavaScript, images) in a controlled manner. Based on some application logic, it will return one from a selection of different files that might be served on that URL at different times and to different users. These are therefore static files, but delivered in a dynamic way.
The application is based on Symfony Components and the serving of these static-ish files is handled by the BinaryFileResponse class.
The bootstrap code calls the trustXSendfileTypeHeader method:
\Symfony\Component\HttpFoundation\BinaryFileResponse::trustXSendfileTypeHeader();
The application uses some internal logic based on configuration and the detection and use of apache_get_modules() to determine availability. If XSendfile is available and the configuration says to use it, it sets the X-Sendfile-Type header:
if ($useHeader === true) {
$request->headers->set('X-Sendfile-Type', $header);
}
$response = new BinaryFileResponse($filename);
Problem
When I run this with the configuration set to never use XSendfile, or through the PHP built-in web server, which obviously does not support XSendfile, everything is perfect.
When I utilise XSendfile, it also works -- most of the time.
Every so often, typically if I press the f5 key 3-4 times in quick succession, "something" wigs out and I get a garbled response. For example, this is supposed to be a JavaScript file (copied from "Response" tab under "Net" in Firebug):
hxYîãx��HTTP/1.1 200 OK Date: Tue, 05 Feb 2013 14:49:10 GMT Server:
Apache/2.2.22 (Ubuntu) X-Powered-By: PHP/5.4.6-1ubuntu1.1
Cache-Control: public Last-Modified: Tue, 29 Jan 2013 13:33:23 GMT
Accept-Ranges: bytes Content-Transfer-Encoding: binary ETag:
"10426f-9f6-0" Vary: Accept-Encoding Content-Encoding: gzip
Content-Length: 1011 Keep-Alive: timeout=5, max=98 Connection:
Keep-Alive Content-Type: application/javascript
������­VmoÛ6þ,ÿkÀ²ãIý°~q [Üt]
XÑt¶H¤#Rv¼Àÿ}w(YSÀØ2yïå¹*¾Á>¯¥¥,è) Æ^Ât¸BaÆ\éjgäjí
Î&ð*¸Åí¸tY!³Ç$Óe"jÞ![#,n®®oï®A¨þ¸þù××Þ©¼¼ôÇêÚd¹49mv°ÔrtBÖ^;WÍÓÔg´Y¥´FéôÁR9o°35Îà^º­´N=UÐè­Eµ¢XE¸íÒ%ª°¨Úò7¬KñT¾{;£ÈrTnß³étUè{QÀçÍn·:'üJëQÍÄËZeNjOàyÕÁ:#3wö~4Òét1ù$µeN)RD|
¶FTØJ·ß½¥¨¸õGç >9TyÜxzgl-J:) b«9ûAQ½KXÉ!yÐÓ]
óÆÎ#W¡?¢vún­·7j©ÿ¢ðõÖGEÁy\ºp¤÷cKxf?ï*¼Éç0^ïîÌÇ°ñDQ¸mYJ|4t¾ñæËÛ¯Å
¨6:çøp(}þÑò|LÂ;Õ(#v¹* /[¨U|xª
æ]ÍyìjµòÛ¯p?4sI¥"v÷ôp|uQ4ò4&Ï·$eÒc¸ xo%7Ôi´2ñx;TuÙj23 áÊ%ħ¿¹lÌwÀS.&ÏØß7¸}ó
ZXzå k2'Zdùè
�¦ºû-Ù[Ó²ÿU(¯¤¥=pÃjô¾ç]]Øhhô²×ÙãÚÍ4¨[!Õ}'Òþ^Ð�ûxÿ#+ÚVÞ~áÌáy?d
aíD¹·U×ÃÚ]­ õ5íÃø¨o÷ÂAvUÆmÍaày`¦ä©A?mL[-}®(ÿË
d°öò¬}Ç¢³Çp1À^6%0 hTô^ts´ÞíWô
fO¶ö¢ÎNÜæ·HîUôÔ¶±ÌCµsxh.9åçi Û·_ÈÞØ_ÄãY_Ö}G<ì°ý2wÔ¿aw8/þù\ã±þ"0C
oÂh'tE¶À¤¥7I½éßRt.s?á^d|k/Æ)wRw÷cG¿<Þ
¼´°/^ø*ʤAVZ×y¿zÅΪ¥[²Õ1ò_Vµæï_YXÁÕö ��YXÁÕö ��
Note the presence of the headers in the response body, and the rest of it which is clearly not JavaScript. There are also some spurious characters at the start, which possibly is what leads to the headers being pushed to the body. I have tried to determine if this content is the result of gzipping, but I can't confirm that yet. (See also update below)
Question
Firstly, is BinaryFileResponse even the correct class to use for serving text (non-binary) files? The documentation for the class only says "BinaryFileResponse represents an HTTP response delivering a file." This isn't very detailed but it doesn't say anything about it being exclusively for "binary" files. However the name has its own implications, why didn't Fabien just call this class FileResponse?
Secondly, and more importantly, what could be causing this? I don't believe it is a browser issue because it is repeatable in both Firefox and Chrome. Is this a bug in the XSendfile module or in the BinaryFileResponse class perhaps? (I am likely to think it is not the former because I have used it before in a more "raw" way not via Symfony Components, with no such issues).
Has anyone else experienced this? Any idea where I should even start looking to track this down? I've looked at the BinaryResponseFile source code but it doesn't really do much with XSendfile, just sets the relevant header and prevents content in the response body, from what I can see.
Update
I've just noticed a couple of things about these garbled responses:
There are no actual headers being sent at all, i.e. on the "Headers" tab in Firebug, for the garbled responses, it only lists Request headers and doesn't even show the heading for Response headers.
Even if I set some custom header on the Response in PHP, that header does not appear at all in the garbled responses (as a header or in the response body), but the custom headers appear correctly for the responses that aren't broken.
First, let me say that I don't have any experience with this Apache module, but I'll try to guide you through a general error deduction:
You should check if you can reproduce it more reliably. While a web browser might be ok to try it out, you should go for something like curl and do the request multiple times, for example using a bash for-loop.
for i in `seq 1 5`; do curl -v http://localhost/xsendfile-url; done
The fact that the Connection: Keep-Alive header is set and that there are some weird characters before the actual HTTP header lead me to believe that you won't be able to reproduce this problem with separated curl calls, because it will open a fresh connection each time. So try this to check if that gives you the weird behavior (curl has keep alive on by default):
curl -v http://localhost/xsendfile-url http://localhost/xsendfile-url http://localhost/xsendfile-url
Using this, you could go to the projects github issue page and report your findings. Most probably they will there help you in telling you why mod_xsendfile is behaving the way it is or that you have found a bug.

Apache ignoring PHP headers when sending a 304

When I set a custom header in Apache + mod_php5, this works fine:
header('Foo: Bar');
But when I try this while also sending a 304 Not Modified response, the header appears to be removed by apache (along with X-Powered-By and other standard headers).
header('HTTP/1.1 304 No Content');
header('Foo: Bar');
Does anyone know how to solve this issue?
Does this not answer the question?
If the conditional GET used a strong cache validator (see section 13.3.3), the response SHOULD NOT include other entity-headers. Otherwise (i.e., the conditional GET used a weak validator), the response MUST NOT include other entity-headers; this prevents inconsistencies between cached entity-bodies and updated headers.
from http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5
As of Apache 2.4.23 (the latest release as of today, as far as I know), you're not going to be able to get around that problem when you send a 304 "Not Modified" response because, indeed, Apache does explicitly remove all non-whitelisted headers:
http://svn.apache.org/viewvc/httpd/httpd/tags/2.4.23/modules/http/http_filters.c?view=markup#l1331
So, whether we like it or not (because I'm on the same boat of having my CORS headers removed by Apache from the response when I send a 304), it does seem like Apache is following the RFC recommendation and it's indeed treating everything that falls outside of that list as entity headers.
One solution is to patch-up the Apache source to extend that list and turn to deploying your home-grown package to your server(s), but that's definitely not without a long list of implications of its own. On the flip side, I hear that nginx doesn't suffer from this problem.
The content that I'm delivering will be consumed, among others, by WebGL runtimes in standard browsers, so if they do complain about the lack of CORS in my 304 responses I'm going to have to turn everything to 200 OK and forego the bandwidth savings.
I do a trick to solve this issue by :
1. put all header before 304 header
2. flush these header before send 304
header('Foo: Bar');
flush();
header('HTTP/1.1 304 No Content');
Apache will not remove any header until it found 304.
We force other header send out by flush() before send 304.
That apache can not hurt me.
Try:
header('Foo: bar', true, 304);

Strange http gzip issue

Here's a strange one:
I've got nginx reverse proxying requests to apache 2 with mod_php.
A user (using firefox 3.1b3) reported that recently, he's started getting sporadic "What should firefox do with this file?" popups during normal navigation. We haven't had any other reports of this issue, and haven't been able to reproduce it ourselves.
I checked Nginx and apache's logs. Nothing in the error logs, and they both show a normal HTTP 200 for the request.
I had him send me the downloaded file, and it's generated HTML, as it should be -- except it has some trailing and leading bytes tacked on.
The opening byte sequence is the magic gzip header: 1F8B08
Here are the opening characters, C-escaped for convenience:
\x1F\x8B\x089608\r\n<!DOCTYPE HTML ...
and the file ends with:
...</html>\n\r\n0\r\n\r\n
When I fetch the same URL via wget, it starts with as expected; the mysterious opening and closing bytes are nowhere to be seen.
Has anyone ever seen anything similar to this? Could this be a FF 3.1b3 bug?
Never seen an issue exactly like that, but I did have an issue once with a transparent proxy that would claim to the web server that it could handle gzip compressed content when, in fact, it received the gzipped content from the server, stripped the gzip headers without decompressing it, and sent the result to the browser. The behavior we saw was what you describe: a save/open file dialog for what should have been a normal web page. In this case the browser in question was IE.
I'm not sure if you're problem is related, but as an experiment, you could look at the requests between the proxy and Apache and see if they are gzipped, or else turn off the gzip compression for requests in Apache and see if that fixes the issue. If so, then you probably have a problem with gzip handling in your proxy.
wget doesn't request a compressed response. Try:
curl --compressed <URL>
You could also try adding a -v to print the response headers, and check that a sensible Content-Type is being returned.

Categories