I am looking for a way to confirm if X-Sendfile is properly handling requests handed back to the webserver by a script (PHP). Images are being served correctly but I thought I would see the header in curl requests.
$ curl -I http://blog2.stageserver.net/wp-includes/ms-files.php?file=/2011/05/amos-lee-feature.jpg
HTTP/1.1 200 OK
Date: Wed, 04 Jan 2012 17:19:45 GMT
Server: Cherokee/1.2.100 (Arch Linux)
ETag: "4dd2e306=9da0"
Last-Modified: Tue, 17 May 2011 21:05:10 GMT
Content-Type: image/jpeg
Content-Length: 40352
X-Powered-By: PHP/5.3.8
Content-Disposition: inline; filename="amos-lee-feature.jpg"
Configuration
Cherokee 1.2.100 with PHP-FPM 5.3.8 in FastCGI:
cherokee.conf: vserver!20!rule!500!handler!xsendfile = 1
(Set by vServer > Behavior > Extensions php > Handler: Allow X-Sendfile [check Enabled])
Wordpress Network / WPMU 3.3.1:
define('WPMU_SENDFILE',true); is set in the wp-config.php the following just before wp-settings.php is included. This will trigger the following code to be executed in WP's wp-includes/ms-files.php:50 serves up files for a particular blog:
header( 'X-Sendfile: ' . $file );
exit;
I have confirmed that the above snippet is executing by adding an additional header for disposition right before the exit(); call. That Content-Disposition is present with curl results above and not originally in the ms-files.php code. The code that was added is:
header('Content-Disposition: inline; filename="'.basename($file).'"');
Research
I have:
Rebooted php-fpm / cherokee daemons after making configuration changes.
Tried several tricks in the comments over at php.net/readfile and replaced the simple header in ms-files.php with more complete code from examples.
php.net/manual/en/function.readfile.php
www.jasny.net/articles/how-i-php-x-sendfile/
*codeutopia.net/blog/2009/03/06/sending-files-better-apache-mod_xsendfile-and-php/*
Confirmed [cherokee support][5] and tested [with and without][6] compression even though I don't think it would apply since my images are serving correctly. I also found a suspiciously similar problem from a lighttpd post.
*cherokee-project.com/doc/other_goodies.html*
code.google.com/p/cherokee/issues/detail?id=1228
webdevrefinery.com/forums/topic/4761-x-sendfile/
Found a blurb here on SO that may indicate the header gets stripped
stackoverflow.com/questions/7296642/django-understanding-x-sendfile
Tested that the headers above are consistent from curl, wget, Firefox, Chrome, and web-sniffer.net.
Found out that I can't post more than 2 links yet due to lack of reputation.
Questions
Will X-Sendfile be present in the headers when it is working correctly or is it stripped out?
Can the access logs be used to determine if X-Sendfile is working?
I am looking for general troubleshooting tips or information here, not necessarily specific to PHP / Cherokee.
Update
I have found a suitable way to confirm X-Sendfile or X-Accel-Redirect in a test or sandbox environment: Disable X-Sendfile and check the headers.
With Allow X-Sendfile disabled in Cherokee:
$ curl -I http://blog2.stageserver.net/wp-includes/ms-files.php?file=/2011/05/amos-lee-feature.jpg
HTTP/1.1 200 OK
Date: Fri, 06 Jan 2012 15:34:49 GMT
Server: Cherokee/1.2.101 (Ubuntu)
X-Powered-By: PHP/5.3.6-13ubuntu3.3
Content-Type: image/jpeg
X-Sendfile: /srv/http/wordpress/wp-content/blogs.dir/2/files/2011/05/amos-lee-feature.jpg
Content-Length: 40352
The image will not load in browsers but you can see that the header is present. After re-enabling Allow X-Sendfile the image loads and you can be confident that X-Sendfile is working.
According to the source on github X-Sendfile headers will be stripped.
If I'm skimming the file correctly, it's only logging success if it's been compiled in debug mode.
You could check memory usage of sending large files with and without xsendfile.
They are being stripped, simply because having them present will prevent one of the reasons to use it, namely having the file served without the recepient knowing the location of the file being served.
Related
I got the following index.php on a testsite:
<?php
$r = rand(1, 1000);
$mtr=mt_rand(1, 1000);
echo "rand(1, 1000): " . $r;
echo "<br>mt_rand(1, 1000): " . $mtr;
?>
For some reason i can only get it to run once when the page loads, giving me two random numbers, as it should, and maybe once more if i reload the page with F5. But then it refuses to produce any random numbers until a couple minutes have passed.
I feel I am missing something obvious. The server is hosted by MissHosting.se, and runs php5.6. Customer support insists it is a code issue. I will be glad to provide any further information on request.
Thanks for the help!
The problem seems to come from the fact that your server employs some kind of cache. To rule out client side caching (i.e. browser cache) I requested the page several times with curl, which does not do any caching. So it is a server cache.
Now if we look at the headers with curl:
~$ curl http://sithu.net/testinggrounds/ -I
HTTP/1.1 200 OK
Date: Fri, 17 Feb 2017 16:09:36 GMT
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
X-Varnish: 6817501 6109691
Age: 9
X-Cache: HIT
X-Cache-Hits: 1
Accept-Ranges: none
Connection: keep-alive
The headers clearly indicate that the server does caching and we have hit the server cache (X-Cache: HIT and X-Cache-Hits: 1). So the next step would be to find out how/where you can change your server caching mechanism.
The X-Varnish header indicates that your server/hoster is using the Varnish HTTP Cache to do the caching.
Background
Part of my application's responsibility is handling requests for static resources (CSS, JavaScript, images) in a controlled manner. Based on some application logic, it will return one from a selection of different files that might be served on that URL at different times and to different users. These are therefore static files, but delivered in a dynamic way.
The application is based on Symfony Components and the serving of these static-ish files is handled by the BinaryFileResponse class.
The bootstrap code calls the trustXSendfileTypeHeader method:
\Symfony\Component\HttpFoundation\BinaryFileResponse::trustXSendfileTypeHeader();
The application uses some internal logic based on configuration and the detection and use of apache_get_modules() to determine availability. If XSendfile is available and the configuration says to use it, it sets the X-Sendfile-Type header:
if ($useHeader === true) {
$request->headers->set('X-Sendfile-Type', $header);
}
$response = new BinaryFileResponse($filename);
Problem
When I run this with the configuration set to never use XSendfile, or through the PHP built-in web server, which obviously does not support XSendfile, everything is perfect.
When I utilise XSendfile, it also works -- most of the time.
Every so often, typically if I press the f5 key 3-4 times in quick succession, "something" wigs out and I get a garbled response. For example, this is supposed to be a JavaScript file (copied from "Response" tab under "Net" in Firebug):
hxYîãx��HTTP/1.1 200 OK Date: Tue, 05 Feb 2013 14:49:10 GMT Server:
Apache/2.2.22 (Ubuntu) X-Powered-By: PHP/5.4.6-1ubuntu1.1
Cache-Control: public Last-Modified: Tue, 29 Jan 2013 13:33:23 GMT
Accept-Ranges: bytes Content-Transfer-Encoding: binary ETag:
"10426f-9f6-0" Vary: Accept-Encoding Content-Encoding: gzip
Content-Length: 1011 Keep-Alive: timeout=5, max=98 Connection:
Keep-Alive Content-Type: application/javascript
������VmoÛ6þ,ÿkÀ²ãIý°~q [Üt]
XÑt¶H¤#Rv¼Àÿ}w(YSÀØ2yïå¹*¾Á>¯¥¥,è) Æ^Ât¸BaÆ\éjgäjí
Î&ð*¸Åí¸tY!³Ç$Óe"jÞ![#,n®®oï®A¨þ¸þù××Þ©¼¼ôÇêÚd¹49mv°ÔrtBÖ^;WÍÓÔg´Y¥´FéôÁR9o°35Îà^º´N=UÐèEµ¢XE¸íÒ%ª°¨Úò7¬KñT¾{;£ÈrTnß³étUè{QÀçÍn·:'üJëQÍÄËZeNjOàyÕÁ:#3wö~4Òét1ù$µeN)RD|
¶FTØJ·ß½¥¨¸õGç >9TyÜxzgl-J:) b«9ûAQ½KXÉ!yÐÓ]
óÆÎ#W¡?¢vún·7j©ÿ¢ðõÖGEÁy\ºp¤÷cKxf?ï*¼Éç0^ïîÌÇ°ñDQ¸mYJ|4t¾ñæËÛ¯Å
¨6:çøp(}þÑò|LÂ;Õ(#v¹* /[¨U|xª
æ]ÍyìjµòÛ¯p?4sI¥"v÷ôp|uQ4ò4&Ï·$eÒc¸ xo%7Ôi´2ñx;TuÙj23 áÊ%ħ¿¹lÌwÀS.&ÏØß7¸}ó
ZXzå k2'Zdùè
�¦ºû-Ù[Ó²ÿU(¯¤¥=pÃjô¾ç]]Øhhô²×ÙãÚÍ4¨[!Õ}'Òþ^Ð�ûxÿ#+ÚVÞ~áÌáy?d
aíD¹·U×ÃÚ] õ5íÃø¨o÷ÂAvUÆmÍaày`¦ä©A?mL[-}®(ÿË
d°öò¬}Ç¢³Çp1À^6%0 hTô^ts´ÞíWô
fO¶ö¢ÎNÜæ·HîUôÔ¶±ÌCµsxh.9åçi Û·_ÈÞØ_ÄãY_Ö}G<ì°ý2wÔ¿aw8/þù\ã±þ"0C
oÂh'tE¶À¤¥7I½éßRt.s?á^d|k/Æ)wRw÷cG¿<Þ
¼´°/^ø*ʤAVZ×y¿zÅΪ¥[²Õ1ò_Vµæï_YXÁÕö ��YXÁÕö ��
Note the presence of the headers in the response body, and the rest of it which is clearly not JavaScript. There are also some spurious characters at the start, which possibly is what leads to the headers being pushed to the body. I have tried to determine if this content is the result of gzipping, but I can't confirm that yet. (See also update below)
Question
Firstly, is BinaryFileResponse even the correct class to use for serving text (non-binary) files? The documentation for the class only says "BinaryFileResponse represents an HTTP response delivering a file." This isn't very detailed but it doesn't say anything about it being exclusively for "binary" files. However the name has its own implications, why didn't Fabien just call this class FileResponse?
Secondly, and more importantly, what could be causing this? I don't believe it is a browser issue because it is repeatable in both Firefox and Chrome. Is this a bug in the XSendfile module or in the BinaryFileResponse class perhaps? (I am likely to think it is not the former because I have used it before in a more "raw" way not via Symfony Components, with no such issues).
Has anyone else experienced this? Any idea where I should even start looking to track this down? I've looked at the BinaryResponseFile source code but it doesn't really do much with XSendfile, just sets the relevant header and prevents content in the response body, from what I can see.
Update
I've just noticed a couple of things about these garbled responses:
There are no actual headers being sent at all, i.e. on the "Headers" tab in Firebug, for the garbled responses, it only lists Request headers and doesn't even show the heading for Response headers.
Even if I set some custom header on the Response in PHP, that header does not appear at all in the garbled responses (as a header or in the response body), but the custom headers appear correctly for the responses that aren't broken.
First, let me say that I don't have any experience with this Apache module, but I'll try to guide you through a general error deduction:
You should check if you can reproduce it more reliably. While a web browser might be ok to try it out, you should go for something like curl and do the request multiple times, for example using a bash for-loop.
for i in `seq 1 5`; do curl -v http://localhost/xsendfile-url; done
The fact that the Connection: Keep-Alive header is set and that there are some weird characters before the actual HTTP header lead me to believe that you won't be able to reproduce this problem with separated curl calls, because it will open a fresh connection each time. So try this to check if that gives you the weird behavior (curl has keep alive on by default):
curl -v http://localhost/xsendfile-url http://localhost/xsendfile-url http://localhost/xsendfile-url
Using this, you could go to the projects github issue page and report your findings. Most probably they will there help you in telling you why mod_xsendfile is behaving the way it is or that you have found a bug.
Our web application has version numbers that get served out to the client on each request so we can detect an update to the code (ie rolling updates) and displays a popup informing them to reload to take advantage of the latest update.
But I'm experiencing some weird behaviour after the update of the version number on the server, where some requests return the new version number and some return the old, so the popup keeps poping up until you have reloaded the page a few times.
Originally I suspected maybe apache was caching files it read off disk via file_get_contents so instead of storing the version number in a plain text file, I now store it in a php file that gets included with each request, but I'm experiencing the exact same issue!
Anyone have any ideas what might be causing apache or php it self to be serving out old information after i have done an update?
EDIT: I have confirmed its not browser caching as I can have the client generate unique urls to the server (that it can deal with via rewrite) and i still see the same issue where some requests return the old version number and some the new, and clearing the browser cache doesn't help
EDIT 2: The response headers as requested
HTTP/1.1 200 OK
Date: Mon, 23 Jul 2012 16:50:53 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: PHP/5.3.2-1ubuntu4.7
Cache-Control: no-cache, must-revalidate
Pragma: no-cache
Expires: Sat, 26 Jul 1997 05:00:00 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 500
Connection: close
Content-Type: text/html
EDIT 3: So trying to reproduce to get the response headers I found I could only make it happen going through our full deploy process which involves creating versioned folders storing the code and symlinking the relavant folder into the webroot. Just changing the version number wasn't enough to cause it to happen! So seems to be somehow related to the symlinks i create!
I have the same problem when there is a change in the symlink. Have a look at https://bugs.php.net/bug.php?id=36555 it's maybe what you are looking for.
Try (as said in this bug report) setting realpath_cache_size is 0.
EDIT #2
Ok, the problem is different to what I originally thought, so I'm putting this 'edit' up the top. I've also updated the question title (the old one was 'Streaming wav audio from a mysql blob').
The problem now seems to be related to CodeIgniter sessions. The above script will only run if the user is logged in. For some reason, when I manually set the response headers (either
with php's header() or codeigniters output class) I can see from my logs that everything gets reloaded and reinitilised -- and that the session data is lost, so the user is no longer logged in, so the script is actually outputting an error.
Once I removed any requirement or reference to session data the audio plays fine... but this isn't really an option unless I can manage to authenticate the user some other way. Very frustrating.
.
.
.
** Original Text **
I have a mysql blob which contains audio data in wav format. I'm trying to output that to the browser to stream in whatever audio player plugin the browser wants to use.
The audio player works fine if I point it at a physical .wav file, so that doesn't seem to be a problem. However, if I point it at my PHP script I get a 'Search for suitable plugin?' popup in Firefox (which fails to find anything), and just an empty inactive player in Chrome.
Here's the PHP, the $aud object contains information retrieved from the database:
header("Content-length: $aud->size");
header("Content-type: audio/x-wav");
echo $aud->recording;
exit();
If I add header("Content-Disposition: attachment; filename=$name"); to the above I get to download the file, which then plays fine in an external audio player, but that's not what I want.
This snippit is part of a CodeIgniter application, if that would make a difference. I have routing set up so that /audio/$id.wav will grab the appropriate data and output it with the code above.
Can anyone see or think of any reason the above might not be working?
EDIT
These are the headers returned by the php script:
Date: Tue, 22 Mar 2011 22:12:06 GMT
Server: Apache/2.2.16 (Ubuntu)
X-Powered-By: PHP/5.3.3-1ubuntu9.3
Set-Cookie: ci_session=<long encrypted session string>; expires=Wed, 23-Mar-2011 00:12:06 GMT; path=/
Content-Length: 12345
Keep-Alive: timeout=15, max=98
Connection: Keep-Alive
Content-Type: audio/x-wav
200 OK
And for comparison, these headers are returned when I force a download of the above audio and open that wav file directly in the browser:
Date: Tue, 22 Mar 2011 22:10:53 GMT
Server: Apache/2.2.16 (Ubuntu)
Last-Modified: Tue, 22 Mar 2011 22:08:30 GMT
Etag: "200a83-3039-49f197bfcb380"
Accept-Ranges: bytes
Content-Length: 12345
Content-Type: audio/x-wav
200 OK
Saving and then opening the file directly does work. Having the PHP script output to the browser does not.
I would start investigating this problem by comparing the HTTP content-length, content-type, and all the other headers sent by the web server when pointing to the physical .wav file, with the headers sent when trying to open the PHP script. I think the actual content body is correct, according to your post stating that if you download the file as an attachment, it can be played with an audio player application.
Try adding this header in there:
header('Content-Transfer-Encoding: binary');
See if that helps it out.
I believe this would be a more CPU friendly method, can it be implemented with php ?, instead of gzipping content for every request, I compress the files once and serve those instead =).
Yes, this is quite easy to do with Apache.
Store the uncompressed and compressed files side by side. E.g.:
\-htdocs
|-index.php
|-javascript.js
\-javascript.js.gz
Enable content negotiation in Apache. Use:
Options +MultiViews
Now when "/javascript" is requested, Apache will serve the gzipped version if the client declares it accepts it (through Accept-encoding).
Example of two HTTP requests (some headers omitted):
Client claims to accept gzip
GET /EP/Exames/2006-2007/exame2B HTTP/1.1
Host: lebm.geleia.net
Accept-Encoding: gzip, identity
HTTP/1.1 200 OK
Date: Fri, 13 Aug 2010 16:22:59 GMT
Content-Location: exame2B.nb.gz
Vary: negotiate,accept-encoding
TCN: choice
Last-Modified: Sun, 04 Feb 2007 15:33:53 GMT
ETag: "0-c9d-428a84de03a40;48db6d490abee"
Accept-Ranges: bytes
Content-Length: 3229
Content-Type: application/mathematica
Content-Encoding: gzip
‹áüÅE
(response continues)
Client does not claim to accept gzip
GET /EP/Exames/2006-2007/exame2B HTTP/1.1
Host: lebm.geleia.net
Accept-Encoding: identity
HTTP/1.1 200 OK
Date: Fri, 13 Aug 2010 16:23:14 GMT
Content-Location: exame2B.nb
Vary: negotiate,accept-encoding
TCN: choice
Last-Modified: Sun, 04 Feb 2007 15:33:53 GMT
ETag: "0-257f-428a84de03a40;48db6d490abee"
Accept-Ranges: bytes
Content-Length: 9599
Content-Type: application/mathematica
(************** Content-type: application/mathematica **************
CreatedBy='Mathematica 5.2'
(response continues)
See a more complete version here http://pastebin.com/TAwxpngX
Yes, this is a sensible approach to save both bandwidth and connections. (You can enable gzip compression within Apache if so desired, but it's potentially worth doing this anyway as you've save connections.)
In essence, use a PHP function to check if the browser supports gzip compression. (If if doesn't you'll need to fetch the JavaScript/CSS as per normal.) If it does, you can simply point the JavaScript or CSS source location at a PHP script which is responsible for:
Checking to see if there's a compressed version in place. (Simply output the existing 'on disk' if there is.)
Creating a compressed version of the required files.
You'll also probably want to enable/disable this from a define/top level config (for testing purposes, etc.) As a suggestion, you could store the required CSS/JavaScript files paths in a set of arrays which could be used as a basis for creating the cache file or including the files in the traditional manner as a fallback.
I've written a solution along these lines in the past that created a file based on a hash of the required filenames. As such, the cache was automatically rebuilt if a different/additional file was included. (It also re-built the cache after 'n' hours, but that's only to keep things fresh if the filenames didn't change, but the content did.)