I have to distribute a huge file to some people (pictures of a prom) via my Apache2/PHP server which is giving me some headaches: Chrome and Firefox both show a filesize of 2GB but the file is actually >4GB, so I started to track things down.
I am doing the following thing in my php script:
header("Content-Length: ".filesize_large($fn));
header("Actual-File-Size: ".filesize_large($fn)); //Debug
readfile($fn);
filesize_large() is returning the correct filesize for >4gb files as a string (yes, even on 32-bit PHP).
Now the interesting part; the actual HTTP header:
Content-Length: 2147483647
Actual-File-Size: 4236525700
So the filesize_large() method is working totally fine, but PHP or Apache somehow limit the value of Content-Length?! Why that?
Apache/2.2.22 x86, PHP 5.3.10 x86, I am using SSL over https
Just so you guys believe me when I say filesize_large() is correct:
function filesize_large($filename)
{
return trim(shell_exec('stat -c %s '.escapeshellarg($filename)));
}
Edit:
Seems like PHP casts the content length to an integer when communicating with apache2 over the sapi interface on 32-bit systems. No workaround sadly except not including the Content-Size in case of files >2GB
Workaround (and actually a far better solution in the first place): Use mod_xsendfile
You have to use 64-bit operation system in order to support long integers for Content-length header.
I would recommend to use Vagrant for development.
Header based on strings, but content length based on int. If take a look here https://books.google.com/books?id=HTo_AmTpQPMC&pg=PA130&lpg=PA130&dq=ap_set_content_length%28r,+r-%3Efinfo.size%29;&source=bl&ots=uNqmcTbKYy&sig=-Wth33sukeEiSnUUwVJPtyHSpXU&hl=en&sa=X&ei=GP0SVdSlFM_jsATWvoGwBQ&ved=0CDEQ6AEwAw#v=onepage&q=ap_set_content_length%28r%2C%20r-%3Efinfo.size%29%3B&f=false
you will see example of ap_set_content_length(); function which was used to serve content-length response. It accepts file length from system function. Try to call php filesize() and you'll probably see the same result.
If you take a look into ap_set_content_length declaration http://ci.apache.org/projects/httpd/trunk/doxygen/group__APACHE__CORE__PROTO.html#ga7ab393c56cf073ce7aadc3b7ca3db7b2
you will see that length declared as apr_off_t.
And here http://svn.haxx.se/dev/archive-2004-01/0871.shtml you can read, that this type depends from compiler options which is 32bit in your case.
I would recommend you to read source code of Apache and PHP projects.
Related
I'm trying to determine the CPU type of the server my PHP is running on. I'm not need the CPU usage.
The determination of the CPU type should work independently of the system (Win/Linux..).
The solutions I found evaluate the /proc/cpuinfo file.
This solution only works for linux systems and then only if the rights for access are available.
Example: PHP Script - Get Server Processor
My approach was to use php function php_uname with parameter 'm'.
<?php
var_dump(php_uname('m'));
//string(6) "x86_64"
Unfortunately, this solution fails on some host systems.
An answer is then generated there which is identical to the parameter 'a'.
Example:
'Linux localhost 3.10.0-1160.36.2.el7.x86_64 #1 SMP'
The solution should only require PHP and must be able to run without additional installations. I am thankful for all hints.
Try phpinfo(). phpinfo(int $flags = INFO_ALL): bool.
Outputs a large amount of information about the current state of PHP.
This includes information about PHP compilation options and
extensions, the PHP version, server information and environment
I don't know if it's a bad installation or a PHP bug when php_uname('m') returns an incorrect response on some host systems. These are all GNU/Linux systems. Access to /proc/cpuinfo is forbidden. However, I can run the uname command there. This is my current solution:
function getCPU(){
$name = php_uname('m');
//workaround bug
if(strlen($name) > 20 AND stripos($name,'linux') !== false){
$name = `uname -m`;
}
return trim($name);
}
The function returns a short string like "AMD64", "x86_64" or "aarch64" on success.
Demo on 3v4l.org.
I'm using php wget to download mp4 files from another server
exec("wget -P files/ $http_url");
but I didn't find any option to check if file downloaded correctly, or not yet.
I tried to get duration file using getID3(), but it always return good value, even if file not downloaded correctly
// Check file duration
$file = $getID3->analyze($filepath);
echo $file['playtime_string']; // 15:00 always good value
there is any function to check that?
Thanks
First off I would try https instead. If the server(s) you're connecting to happen to support it, you get around this entire issue because lost bytes are usually caused by flaky hardware or bad MTU settings on a router on their network. The http connections gracefully degrade to giving you as much of the file as it could manage, whereas https connections just plain fail when they lose bytes because you can't decrypt non-intact packets.
Lazy IT people tend to get prodded to fix complete failures of https, but they get less pressure to diagnose and fix corner cases like missing bytes that only occur larger transactions over http.
If https is not available, keep reading.
An HTTP server response may include a Content-Length header indicating the number of bytes in a particular transaction.
If the header is there, you should be able to see it by running wget directly, adding the -v flag.
If it's not there, I believe wget will report Length: unspecified followed by the content-type header's value.
If it tells you (and assuming the byte count is accurate) then you can just compare the byte count of the file you got and the one in the transaction.
If the server(s) you're contacting don't provide this header, you're left with less exact methods, like finding some player that will basically play the mp3 until it ends and then see how long it took and then compare that to the length listed in the ID3 tag (which is in the very beginning of the file). You're not going to be able to get an exact match though, because the time in the tag (if it's there) is only accurate to the second, meaning half a second could be gone from the end of the file and you wouldn't know.
(Backstory: My PHP script is executing another PHP script through the command line (PHP's "exec()" command) so that the cURL session the target script creates doesn't cause the original PHP script to hang. I'm doing this so that I can send transactional email without slow page loads for my user if the email provider's servers are laggy.)
I'm calling a PHP script like this:
exec("[php] [target script] [querystring]"), where [querystring] is a typical HTTP GET querystring (variable=value&variable2=value2). The reason I'm passing the email data to the target script via a querystring is that my PHP host has a flag disabled in the PHP.ini that disallows PHP scripts from detecting the $argv's they're called with via the command line. (For whatever reason, appending a querytring works though.)
So, the question is: What would be the character limit on this querystring? Would there even be one? There is no HTTP or web browser in the mix -- would this mean Apache's and browsers' GET character limits wouldn't be a constraint? Would there naturally be a command line constraint, though?
I don't think there is any character limit, if it is, it's probably determined by the operating system that you use or the language which in this case would translate to your memory.
I hope that answers your question.
Please note that PHP setups with the suhosin patch installed will have a default limit of
512 characters for get parameters. Although bad practice, most browsers (including IE)
supports URLs up to around 2000 characters, while Apache has a default of 8000. To add
support for long parameters with suhosin, add suhosin.get.max_value_length = in php.ini
I'm using passthru("cat filepath") in my download script. My concern is that it might use a lot of server resource.
What is the difference between directly link a file in a public directory and download a file using passthru("cat filepath") in php?
What is the difference between directly link a file in a public directory and download a file using passthru("cat filepath") in php?
The difference is that linking directly to a file does not invoke PHP, while running a PHP script which in turn runs cat causes, well, both PHP and cat to be invoked. This will take up a moderate amount of extra memory, but won't cause server load under most circumstances.
I was using readfile(), but this function can't be used for files larger than 2gb
You might want to find a better solution than passing all of the file contents through PHP, in that case. Look into X-Sendfile support in your web server software of choice.
Don't use passthru() for that, you're opening yourself to CLI Injection and performance is terrible. readfile() exists just for that.
readfile($filepath);
There is a small overhead when passing through PHP compared to a direct link but we are usually talking of milliseconds. However, the browser will not be able to request a 206 Partial when using readfile() unless you code support for it or use something like PEAR::HTTP_Download.
EDIT: Seems you are using passthru() because apparently readfile() doesn't handle >2GB files properly (I never had that problem with readfile(), in fact I just tested it with a 7.2 GB file and it worked fine). In which case, at least escape your parameters.
function readfile_ext($filepath) {
if(!file_exists($filepath))
return false;
passthru('cat ' . escapeshellarg($filepath));
return true;
}
Instead of passthru('cat filepath'), use the PHP native readfile('filepath'), which has better performance.
Both methods will be slower than simply directly linking to the file though, since PHP has a certain overhead.
We have a web app using Andrew Valums ajax file uploader, if we kick off 5 - 10 image uploads at once, more often then not at least 2 or 3 will result in the same gd error "Corrupt JPEG data"
Warning: imagecreatefromjpeg() [function.imagecreatefromjpeg]:
gd-jpeg, libjpeg: recoverable error: Corrupt JPEG data:
47 extraneous bytes before marker 0xd9 in ....
However this did not happen on our old test server, or local development box's, only on our new production server.
The file size on the server is the same as the original on my local machine, so it completes the upload but I think the data is being corrupted by the server.
I can "fix" the broken files by deleting them and uploading again, or manually uploading via FTP
We had a shared host on Godaddy and just have started to have this issue on a new box (that I set up, so probably explains a lot :) CentOS 5.5+, Apache 2.2.3, PHP 5.2.10
You can see some example good and bad picture here. http://174.127.115.220/temp/pics.zip
When I BinDiffed them I see a consistent pattern the corruption is always 64 byte blocks, and while the distance between corrupted blocks is not constant the number 4356 comes up a lot.
I really think we can rule out the Internet as error checking and retransmission with TCP is pretty reliable, further there seems to be no difference between browser versions, or if I turn anti-virus and firewalls off.
So I'm picking configuration of Apache / PHP?
Some cameras will append some data inside the file that will get interpreted incorrectly (most likely do to character encoding with in the headers).
A solution I found was to read the file in binary mode like so
$fh = fopen('test.jpg', 'rb');
$str = '';
while($fh !== false && !feof($fh)){
$str .= fread($fh, 1024);
}
$test = #imagecreatefromstring($str);
imagepng($test,'save.png');
Well, i think the problem is jpeg-header data, and as far as i know there is nothing to do with it by PHP, i think the problem is your fileuploader, maybe there are some configuration for it that you are missing.
Hmm - a 64 byte corruption?...or did you mean 64 bit?
I'm going to suggest that the issue is in fact as a result of the PHP script. the problem that regularly comes up here is that the script inserts CRLFs into the data stream being uploaded, and is caused by differences between the Window/*nix standards.
Solution is to force the php script to upload in binary mode (use the +b switch for ALL fopen() commands in the php upload). It is safe to upload a text file in binary mode as at least you can still see the data.
Read here for more information on this issue:
http://us2.php.net/manual/en/function.fopen.php
This can be solved with:
ini_set ('gd.jpeg_ignore_warning', 1);
I had this problem with GoDaddy hosting.
I had created the database on GoDaddy using their cPanel interface. It was created as "latin collation" (or something like that). The database on the development server was UTF8. I've tried all solutions on this page, to no avail. Then I converted the database to UTF8, and it worked.
Database encoding shouldn't affect BLOB data (or so I would think). BLOB stands for BINARY Large Object (something...), to my knowledge!
Also, strangely, the data was copied from the dev to production server while the database was still "latin", and it was not corrupted at all. It's only when inserting new images that the problem appeared. So I guess the image data was being fed to MySQL as text data, and I think there is a way (when using SQL) of inserting binary data, and I did not follow it.
Edit: just took a look at the MySQL export script, here it is:
INSERT INTO ... VALUES (..., _binary 0xFFD8FF ...
Anyway, hope this will help someone. The OP did not indicate what solved his problem...