Is there any way to log (record) the time it takes to download a file from a web server via the browser? The file is written on the HDD and the environment is LAMP.
Thanks.
Sure there is. You will have to either use mod_rewrite to let Apache know the file is to be served by PHP (ask for more instructions if this is your case) or just use a PHP script to fetch the file like this:
http://youserver.com/download.php?filename=mypicture.jpeg
And then you can can have download.php like this:
<?php
// gets the starting time
$time_start = microtime(true);
// WATCHOUT! THIS IS NOT SECURE! EXAMPLE ONLY.
#$filename = $_GET['filename'];
// gets the intro.mp3 file and outputs it to the user
$filename = "intro.mp3";
header('Content-type: audio/mpeg');
header('Content-Length: '.filesize($filename));
header('Content-Disposition: attachment; filename="intro.mp3"');
readfile($filename);
// gets the end time and duration
$time_end = microtime(true);
// write time to hdd, database, whatever
// ...
error_log("Processing time: ". sprintf("%.4f", ($time_end-$time_start))." seconds");
?>
Please do remember that the $filename = $_GET['filename'] is example only and should be properly escaped so that people can't hack into your server.
EDIT:
Altered to check if it really worked - Mark made me question it! ;) Needed minor tweaking (especially on microtime) but yes, it does work!
One way would be to proxy the download via a script (can be achieved easily with mod_rewrite). This means that you need to do all the work like mapping extensions into their corresponding mime-types (for the Content-Type header). Try downloading a file from your browser, and take a look at the headers it sends; you need to mimic those. You can use eg. web-sniffer.net to inspect the headers.
I don't think PHP would have access to this kind of information. I'm not sure if Apache will tell you either, or whether the method below will also time just until the file got dumped into the transmit buffer, but I would suggest you try this:
Define a custom log in which you include the "%D" directive, this gives you "The time taken to serve the request, in microseconds."
Related
I'm working on file storage and my design is to display data from my database about the file along with a button fitted with an a href tag to download my file. I successfully do all of this when I keep my storage in my website root - but upon suggestions, i've made changes.
I now have my project root in my C:/ drive but my storage I want outside of the root but I keep my storage in my d:/ drive, per suggestions due to security - along with the fact my d:/ drive is my cloud storage anyway. From what I see online, that means I can't use <a href="d:/storage/Username/file download> anymore to download my file!
If that is true, how do you download from outside your website root?
And if not, how do I get past the security restrictions on downloading outside the root using my href method?
Thanks!
you can use readfile()
either link to downloader.php?f=somefile.jpg
of use rewriterules.
in downloader.php do something like
<?php
$path = 'D:/map/';
if( login_check($user) && is_file($path . $_GET['f']) ) {
header('Content-Type: image/jpeg');
readfile($path . $_GET['f']);
}
?>
I added the login_check() because you mentioned something about security.
Further it could be advisable to check the parameter for illegal characters (especially slashes)
And the content type may vary with chosen file
Got it to work, thanks for the start from Ivo P.
I started by sending to download.php my file name, file extension, and file path all separated by * (The * is not an allowed file name). An example is:
download.php?f=myTextFile*txt*d:/storage/test/myTextFile.txt
Then, I explode $_GET['f'] by *, make sure the array is size 3, then set content headers:
$file_info = explode("*", $_GET['f'], 3);
if(is_file($file_info[2]) && count($file_info) == 3) {
header('Content-Description: File Transfer');
header("Content-Type: " . mime_content_type($file_info[2]));
header('Content-Disposition: attachment; filename="'.$file_info[0] . '.' .$file_info[1].'"');
header('Expires: 0');
header('Pragma: public');
readfile($file_info[2]);
}
And that was the working solution for me! Tested on txt, jpg, and exe - works perfectly. I have other personalized for my project tests to make sure that 'f' is valid (so users can't screw it up by going to address bar and making a mess of things, see real_escape_string) but that's the gist of it!
Edit thanks to Bjorn: Keep in mind this allows full access of server, make sure you personalize your own download.php to accept inputs following your own format (otherwise anyone could steal your php files, js files, ect!)
I face a case I never did, and I dont know how to properly do it.
I have a php script which generate files for clients. At the end of the script, I echo the path for them to download the file, simply.
How can I do to provide the file - or the path or any what - for downloading it, and be sure to delete the file once downloaded.
Widely, I'd like to make the file available for one/unique download only. How to ?
EDIT
I cannot use headers
There are a few components to getting this to work. Without knowing which framework you use, I'll use comments as placeholders.
There is no way to do it without using the header function, though.
Here is the source for a file that outlines the process:
<?php
$fileid = $_GET['fileid'];
$key = $_GET['key'];
// find the file in the database, and store it in $file
if ($keyMatches) {
// it is important for security to only use file paths from the database
$actualPath = $file->getPathOnDisk();
$fileInfo = finfo_open(FILEINFO_MIME_TYPE);
$mime = finfo_file($fileInfo, $actualPath);
$fp = fopen($actualPath, 'rb');
header("Content-Type: " . $mime);
header("Content-Length: " . filesize($actualPath));
fpassthru($fp);
}
else
{
http_response_code(403); // forbidden
}
You'll use this by linking to download.php?fileid=1234&key=foobar, and generating the URL at the same time you generate the key and store it in the database.
For security, you'll keep the files outside of the web root, meaning they cannot be accessed through the web server without going through a script.
fpassthru is reasonably fast, and will not likely have a performance impact.
You must do a download file gateway, like download.php?id=XXX
Where XXX is the unique ID of each file you will store in DB. And of course, the file to be downloaded.
Then, each time a user will visit the page, you can :
- Check if he has already downloaded the file
- If no, redirect it to the real path of file
- If yes, display 403 message.
When a user download a file, update the DB, generate or copy the file to a new name, you play with headers, and delete file upon download or after a small timeout.
I'm having the following problem with my VPS server.
I have a long-running PHP script that sends big files to the browser. It does something like this:
<?php
header("Content-type: application/octet-stream");
readfile("really-big-file.zip");
exit();
?>
This basically reads the file from the server's file system and sends it to the browser. I can't just use direct links(and let Apache serve the file) because there is business logic in the application that needs to be applied.
The problem is that while such download is running, the site doesn't respond to other requests.
The problem you are experiencing is related to the fact that you are using sessions. When a script has a running session, it locks the session file to prevent concurrent writes which may corrupt the session data. This means that multiple requests from the same client - using the same session ID - will not be executed concurrently, they will be queued and can only execute one at a time.
Multiple users will not experience this issue, as they will use different session IDs. This does not mean that you don't have a problem, because you may conceivably want to access the site whilst a file is downloading, or set multiple files downloading at once.
The solution is actually very simple: call session_write_close() before you start to output the file. This will close the session file, release the lock and allow further concurrent requests to execute.
Your server setup is probably not the only place you should be checking.
Try doing a request from your browser as usual and then do another from some other client.
Either wget from the same machine or another browser on a different machine.
In what way doesn't the server respond to other requests? Is it "Waiting for example.com..." or does it give an error of any kind?
I do something similar, but I serve the file chunked, which gives the file system a break while the client accepts and downloads a chunk, which is better than offering up the entire thing at once, which is pretty demanding on the file system and the entire server.
EDIT: While not the answer to this question, asker asked about reading a file chunked. Here's the function that I use. Supply it the full path to the file.
function readfile_chunked($file_path, $retbytes = true)
{
$buffer = '';
$cnt = 0;
$chunksize = 1 * (1024 * 1024); // 1 = 1MB chunk size
$handle = fopen($file_path, 'rb');
if ($handle === false) {
return false;
}
while (!feof($handle)) {
$buffer = fread($handle, $chunksize);
echo $buffer;
ob_flush();
flush();
if ($retbytes) {
$cnt += strlen($buffer);
}
}
$status = fclose($handle);
if ($retbytes && $status) {
return $cnt; // return num. bytes delivered like readfile() does.
}
return $status;
}
I have tried different approaches (reading and sending the files in small chunks [see comments on readfile in PHP doc], using PEARs HTTP_Download) but I always ran into performance problems when the files are getting big.
There is an Apache mod X-Sendfile where you can do your business logic and then delegate the download to Apache. The download will not be publicly available. I think, this is the most elegant solution for the problem.
More Info:
http://tn123.org/mod_xsendfile/
http://www.brighterlamp.com/2010/10/send-files-faster-better-with-php-mod_xsendfile/
The same happens go to me and i'm not using sessions.
session.auto_start is set to 0
My example script only runs "sleep(5)", and adding "session_write_close()" at the beginning doesn't solve the problem.
Check your httpd.conf file. Maybe you have "KeepAlive On" and that is why your second request hangs until the first is completed. In general your PHP script should not allow the visitors to wait for long time. If you need to download something big, do it in a separate internal request that user have no direct control of. Until its done, return some "executing" status to the end user and when its done, process the actual results.
I have a strange problem here: If I try to download more than one file with the same download script (I've tried 5 different scripts found on php.net), the first goes well but the second has a delay of about 60 seconds from the time of its request. If I cancel the first download, then the second starts suddenly.
I've tested direct file download from apache and everything is ok.
This is the last script I've tried:
<?php
$filename= $_GET['file'];
header("Content-Length: " . filesize($filename));
header('Content-Type: application/zip');
header('Content-Disposition: attachment; filename=writeToFile.zip');
$file_contents = file_get_contents($filename);
print($file_contents);
?>
Could it be that the underlying web server has a limit on concurrent connections from the same IP? Can you try from two different IPs at the same time?
no, no no... problem is somewhere else
you have started session (manualy or automatically)
and session are usually stored in files...
so when open first script then open session and LOCK FILE ... other request must wait for unlocking session file...
existing two solution...
- use self class for session storing ... without locking of file session (bug scripts may overwrite sessions data)
- or before file_get_content call session_write_close();
You might try readfile($filename) instead of $file_contents = file_get_contents($filename);print($file_contents); Since readfile() doesn't store the contents in a string, it doesn't take up memory the way file_get_contents() does.
#Pekka Gaiser has a good point about concurrent connections. Also take a look at what kind of memory limits your PHP is using.
This first script gets called several times for each user via an AJAX request. It calls another script on a different server to get the last line of a text file. It works fine, but I think there is a lot of room for improvement but I am not a very good PHP coder, so I am hoping with the help of the community I can optimize this for speed and efficiency:
AJAX POST Request made to this script
<?php session_start();
$fileName = $_POST['textFile'];
$result = file_get_contents($_SESSION['serverURL']."fileReader.php?textFile=$fileName");
echo $result;
?>
It makes a GET request to this external script which reads a text file
<?php
$fileName = $_GET['textFile'];
if (file_exists('text/'.$fileName.'.txt')) {
$lines = file('text/'.$fileName.'.txt');
echo $lines[sizeof($lines)-1];
}
else{
echo 0;
}
?>
I would appreciate any help. I think there is more improvement that can be made in the first script. It makes an expensive function call (file_get_contents), well at least I think its expensive!
This script should limit the locations and file types that it's going to return.
Think of somebody trying this:
http://www.yoursite.com/yourscript.php?textFile=../../../etc/passwd (or something similar)
Try to find out where delays occur.. does the HTTP request take long, or is the file so large that reading it takes long.
If the request is slow, try caching results locally.
If the file is huge, then you could set up a cron job that extracts the last line of the file at regular intervals (or at every change), and save that to a file that your other script can access directly.
readfile is your friend here
it reads a file on disk and streams it to the client.
script 1:
<?php
session_start();
// added basic argument filtering
$fileName = preg_replace('/[^A-Za-z0-9_]/', '', $_POST['textFile']);
$fileName = $_SESSION['serverURL'].'text/'.$fileName.'.txt';
if (file_exists($fileName)) {
// script 2 could be pasted here
//for the entire file
//readfile($fileName);
//for just the last line
$lines = file($fileName);
echo $lines[count($lines)-1];
exit(0);
}
echo 0;
?>
This script could further be improved by adding caching to it. But that is more complicated.
The very basic caching could be.
script 2:
<?php
$lastModifiedTimeStamp filemtime($fileName);
if (isset($_SERVER['HTTP_IF_MODIFIED_SINCE'])) {
$browserCachedCopyTimestamp = strtotime(preg_replace('/;.*$/', '', $_SERVER['HTTP_IF_MODIFIED_SINCE']));
if ($browserCachedCopyTimestamp >= $lastModifiedTimeStamp) {
header("HTTP/1.0 304 Not Modified");
exit(0);
}
}
header('Content-Length: '.filesize($fileName));
header('Expires: '.gmdate('D, d M Y H:i:s \G\M\T', time() + 604800)); // (3600 * 24 * 7)
header('Last-Modified: '.date('D, d M Y H:i:s \G\M\T', $lastModifiedTimeStamp));
?>
First things first: Do you really need to optimize that? Is that the slowest part in your use case? Have you used xdebug to verify that? If you've done that, read on:
You cannot really optimize the first script usefully: If you need a http-request, you need a http-request. Skipping the http request could be a performance gain, though, if it is possible (i.e. if the first script can access the same files the second script would operate on).
As for the second script: Reading the whole file into memory does look like some overhead, but that is neglibable, if the files are small. The code looks very readable, I would leave it as is in that case.
If your files are big, however, you might want to use fopen() and its friends fseek() and fread()
# Do not forget to sanitize the file name here!
# An attacker could demand the last line of your password
# file or similar! ($fileName = '../../passwords.txt')
$filePointer = fopen($fileName, 'r');
$i = 1;
$chunkSize = 200;
# Read 200 byte chunks from the file and check if the chunk
# contains a newline
do {
fseek($filePointer, -($i * $chunkSize), SEEK_END);
$line = fread($filePointer, $i++ * $chunkSize);
} while (($pos = strrpos($line, "\n")) === false);
return substr($line, $pos + 1);
If the files are unchanging, you should cache the last line.
If the files are changing and you control the way they are produced, it might or might not be an improvement to reverse the order lines are written, depending on how often a line is read over its lifetime.
Edit:
Your server could figure out what it wants to write to its log, put it in memcache, and then write it to the log. The request for the last line could be fulfulled from memcache instead of file read.
The most probable source of delay is that cross-server HTTP request. If the files are small, the cost of fopen/fread/fclose is nothing compared to the whole HTTP request.
(Not long ago I used HTTP to retrieve images to dinamically generate image-based menus. Replacing the HTTP request by a local file read reduced the delay from seconds to tenths of a second.)
I assume that the obvious solution of accessing the file server filesystem directly is out of the question. If not, then it's the best and simplest option.
If not, you could use caching. Instead of getting the whole file, you just issue a HEAD request and compare the timestamp to a local copy.
Also, if you are ajax-updating a lot of clients based on the same files, you might consider looking at using comet (meteor, for example). It's used for things like chats, where a single change has to be broadcasted to several clients.