i am running multiple websites with high traffic , as a requirement , all images are downloaded via image.php?id=IMAGE_ID_HERE .
If you ever done that before , you know that that file will be reading the file image and echoing it to the browser with special headers .
My problem is , the load on the server is very high (150-200) and TOP command shows multiple instances of image.php , so image.php is running slow !
the problem probably is fopen loading the image to the memory before sending it to the client. How to read a file and pass it through directly?
Thank you guys
UPDATE
After you optimized the code, used caching wherever possible, do create a CDN . couple of servers, sync methods, load balancers and no need to worry about requests anymore :)
fopen and file_get_contents are nearly equivalent
to speed up with consistence the page load you can use
http://www.php.net/fpassthru
or, even better
http://www.php.net/readfile
with those functions, content of file is printed directly, byte per byte
as opposed to file_get_contents, for example, where you store the whole data inside a variable
$var = file_get_contents();
so, to make these work correctly you will need to disable output buffering (otherwise it would make readfile() pointless) in the page that serves the images
hope this helps!
Why dont you cache the image content with apc ?
if(!apc_exists('img_'.$id)){
apc_store('img_'.$id,file_get_content(...));
}
echo apc_fetch('img_'.$id);
this way image content will not be read from your disk more than once.
Related
Best practices to export CSV in PHP: output buffer vs temporary file
Scenario
I execute a SELECT on a database that returns any number of rows, may be few or many (one million+), those rows need to go inside a .csv file with the first row beeing header.
Doubt
I know two ways of exporting CSV files with PHP: using output buffer php://output or creating a temporary file, serve it to user, than delete it.
Which way is better, knowing it may be a small file or a very big one? Consider PHP memory limit (in php.ini), request time out, etc.
Using the temporary file in case you have large file is the only good option.
you can redirect second request(if file exist) directly to your file and let web server to serve it without executing php.
if client has disconnected, while download a file through api, - in most cases he will start downloading again;
more of that, you will got access logs on your web server, to check who and how many times access this file.
It depends on the situation.
Use an output buffer when you know the file is not ridiculously large and when it is a download that doesn't occur to often.
When you have something large, that will be downloaded a large number of times (simultaneous), writing it to a file might be better to lighten the load on your database and site.
I'd think the answer is pretty obvious: write directly to php://output. It's the same as echo ..; the output will be send to the client more or less directly. It may or may not get buffered for a bit, but unless you have explicit output buffering activated or your web server has a ridiculously large buffer, it'll send it right through. "Sending a file" (presumably via readfile) would pass the data though the same output buffer, but would be much more complicated and error prone.
I am creating an server-side api for my app. One of the steps requires the app do download a very large zip file from my server, but it can't be done by just downloading http://mydomain.com/file.zip. My app passes some authentication headers and other safety stuff which must be processed by a php script (e.g. http://mydomain.com/download.php?auth=foo&key=value&etc)
I have done this kind of thing earlier using images:
<?php
$can_download = /* some complicated auth stuff */
if($can_download) {
header('Content-Type: image/jpeg');
echo file_get_contents('image.jpg');
}
?>
My question is: can it be done with 200+ MB zip file? I know I have to modify headers somehow and probably use some advanced php functions, but I couldn't find any tutorial neither here, on Stack Overflow, nor anywhere else.
Edit: I also have to be able to resume my downloads, because it's very likely that user would quit the app, though it should be able to resume the download (e.g. from 47%). Can it be done?
For those looking for the answer: the correct function is readfile and 'resuming' problem can be solved using this custom function.
The correct function to use in this instance is readfile.
The given code would encounter problems because it unnecessarily loads the file contents into memory before forwarding them to the browser. This does not help performance, and can easily result in running out of memory.
Yes, however you want to use something like readfile instead. The reason being is that file_get_contents reads the entire file into memory before passing it to the client.
Is it bad practise to retrieve images this way? I have a page to call this script about 100 times (there are 100 images). Can i cause server overload or too many http requests or something? I have problems with the server and i dont know if this is causing it :(
// SET THE CONTENT TYPE HEADER
header('Content-type: image/jpeg');
// GET THE IMAGE TO DISPLAY
$image = imagecreatefromjpeg( '../path/to/image/' . $_SESSION[ID] . '/thumbnail/' . $_GET[image]);
// OUTPUT IMAGE AND FREE MEMORY
imagejpeg($image);
imagedestroy($image);
I call the script from regular tags. The reason I call them through PHP is that the images are private to the user.
All help greatly appreciated!!
With this, you are :
Reading the content of a file
Evaluating that content to an in-memory image
Re-rendering that image
If you just want to send an image (that you have on disk) to your users, why not just use readfile(), like this :
header('Content-type: image/jpeg');
readfile('../path/to/image/' . $_SESSION[ID] . '/thumbnail/' . $_GET[image]);
With that, you'll just :
Read the file
and send its content
Without evaluating it to an image -- eliminating some useless computations in the process.
As a sidenote : you should not use $_GET[image] like that in your path : you must make sure no malicious data is injected via that parameter !
Else, anyone will potentially be able to access any possible file on your server... they just have to specify some relative path in the image parameter...
Yes, it's very bad. You're decoding a .jpg into a memory-based bitmap (which is "huge" compared to the original binary .jpg. You then recompress the bitmap into a jpeg.
So you're wasting a ton of
a) memory
b) CPU time
c) losing even more image quality, because jpg is a lossy format.
why not just do:
<?php
header('Content-type: text/jpeg');
readfile('/path/to/your/image.jpg');
instead?
To answer two particular questions from your question
Can i cause server overload or too many http requests or something?
yes, of course.
by both numerous HTTP requests and image processing.
You have to reduce number of images and implement some pagination to show images in smaller packs.
You may also implement some Conditional GET functionality to reduce bandwidth and load.
If things continue getting bad, and you have some resources to dispose, consider to install some content distribution proxy. nginx with X-Accel-Redirect header is a common example
I have problems with the server and i dont know if this is causing it :(
You shouldn't shoot in the dark then. Profile your site first.
Hi,
I download a large amount of files for data mining. I used to use PHP for this purpose but I am finding it to be too slow. Also I just want a small part of the web page. I want to achieve two things
Curl should be able to utilize all my download bandwidth
Is there any way to download only a part of the web page where my data resides.
I am not confined to PHP. If curl works better in terminal I would use that.
Yes, you can download only a part of the page by using the CURLOPT_RANGE option, and you can also provide a write callback function that simply returns an error when you've received "enough" data and you want to stop and move on.
Are you downloading HTML? Your comment leads me to believe that you are. If that's the case, simply load up the html with Simple PHP DOM and get only the part that you want. Although, I find it hard to believe that grabbing just the HTML is slowing you down. Are you downloading any files or media as well?
Link : http://simplehtmldom.sourceforge.net/
There is no way to download only part of a page. When you request a URL, the server response is what it is.
Utilize more of your bandwidth by using cURL's ability to make multiple connections at once.
Similar to my last question, I'd like to have a PHP function that can take a local path and tell me (a) how much the total file size is for HTML, CSS, JS and images, and (b) the total load time for this page. Like YSlow I think, but as a PHP function.
Any thoughts? I looked around and was wondering can I use CURL for this? Even though I need to check paths that are on my own server? thanks!
Update:
After reading the comments, realizing I'm off base. Instead wondering is there a way programatically get a YSlow score for a page (or similar performance score). I assume it would need to hit a third-party site that would act as the client. I'm basically trying to loop through a group of pages and get some sort of performance metric. Thanks!
For the filesize.
Create a loop to read all files in a specific directory with dir.
Then for each file use filesize.
Loadtime
Loadtime depends on the connection speed and the filesize. And I see that you specify that you are reading locally the files. You can detect how much time it take you to read those files but this will not be the loadtime for the page for an external user.