My question is whether or not using multiple PHP includes() is a bad idea. The only reason I'm asking is because I always hear having too many stylesheets or scripts on a site creates more HTTP requests and slows page loading. I was wondering the same about PHP.
The detailed answer:
Every CSS or JS file referenced in a web page is actually fetched over the network by the browser, which involves often 100s of milliseconds or more of network latency. Requests to the same server are (by convention, though not mandated) serialized one or two at a time, so these delays stack up.
PHP include files, on the other hand, are all processed on the server itself. Instead of 100s of milliseconds, the local disk access will be 10s of milliseconds or less, and if cached, will be direct memory accesses which is even faster.
If you use something like http://eaccelerator.net/ or http://php.net/manual/en/book.apc.php then your PHP code will all be precompiled on the server once, and it doesn't even matter if you're including files or dumping them all in one place.
The short answer:
Don't worry about it. 99 times out of 100 with this issue, the benefits of better code organization outweigh the performance increases.
The use of includes helps with code organization, and is no hindrance in itself. If you're loading up a bunch of things you don't need, that will slow things down -- but that's another problem. Clarification: As you include pages, be aware what you're adding to the load; don't carelessly include unneeded resources.
As already said, the use of multiple PHP includes helps to keep your code organized, so it is not a bad idea. It will become a problem when too many includes are used, because the web server will have to perform an I/O operetation for each include you have.
If you have a large web application, you can boost it using a PHP accelerator, which caches data and compiled code from the PHP bytecode compiler in shared memory. If you have lots of PHP includes in a specific file they will be performed just once. Further calls to that file will hit the cache, so no PHP require will be performed.
It really depends on what you want to do, I mean if you have a piece of code that is used all the time it is really convenient to include it instead of copying and pasting all the time and that will make your code more clear and not slower, but if you include all the functions or classes you have written in your files without using them of course thats not a good practice...I would suggest using a framework (like codeigniter or something else you find convinient) because it really helps clearing this things out... good luck!
The only reason I'm asking is because
I always hear having too many
stylesheets or scripts on a site
creates more HTTP requests and slows
page loading. I was wondering the same
about PHP.
Do notice that an HTTP request is several orders of magnitude slower that a PHP include.
Several HTTP requests -> Client has to request and accept over the wire several files
Several PHP includes -> Client has to request and accept only one file
The includes, obviously, will have a server penalty. But by your question... don't sweat it. Only on really large-scale PHP projects will you face such problems.
Related
As per my knowledge I know that PHP file can not serve two clients at same time, so in Wordpress index.php will be the file handling all requests, so how it is all the way efficient and faster? is there any logical clustering? or any programming techniques in PHP that Wordpress follows? I have a website built on index.php where all requests comes to index.php and seems it lags in performance for fewer requests, dont know how it is going well for Wordpress or to some other CMS?
There is no built-in limit such as "one client per file". You can customize your limits on your webserver and/or your fastCGI pool if you're using one. It will be more accurate to think in terms of "one php thread per request", but even that could be misleading depending on your scenario.
Wordpress's index.php is just a router that in turn picks a theme template and renders it replacing each variable according to the request. No magic there, just basic templating logic.
You index.php lag might be caused by several reasons, including but not limited to:
You are referencing external js script in the header, which will
temporarily block page rendering.
You are trying to establish an early DB connection to a slow or high latency DB server
You are making heavy SQL queries on each request
You are not caching sections of the page that are subject to little or no modification from one request to the next
You are using file based sessions on a slow storage machine
You are not using an opcode cache AND you're doing an expensive PHP calculation on each request.
You have a poorly tuned webserver that allocates too much resources for each request, even for static assets.
I'm coding a site in PHP and getting "pretty urls" (also hiding my directories) by directing all requests to one index.php file (using .htaccess). The index file then parses the uri and includes the requested files. These files also have more than a couple of includes in them, and each may open up a MySQL connection. And then those files have includes too, which open sql connections. It goes down to about 3-4 levels.
Is this process CPU and memory intensive, both from the PHP includes and opening (and closing) MySQL connections in each included file?
Also, would pretty urls using purely htaccess use less resources?
PHP Overheads
The answer re the logical decomposition of your app into a source hierarchy depends on how your solution is being hosted.
If you use a dedicated host/VM then you will probably have mod_php+Xcache or equiv and the answer will be: no, it doesn't really hit the runtime since everything gets cached in-memory at the PHP Opcode level.
If you use a shared hosting service then it will impact performance since any PHP scripts will be loaded through PHP-cgi probably via suPHP and the entire source hierarchy that is included will need to be read in and compiled per request. Worse, on a shared solution, if this request is the first in say 1 min, then the servers file cache will have been flushed and marshalling this source will involve a lot of physical I/O = seconds time delay.
I administer a few phpBB forums and have found that by aggregating common include hierarchies for shared hosting implementations, I can half the user response time. Here are some are articles which describe this in more detail (Terry Ellison [phpBB]). And to quote one article:
Let me quantify my views with some ballpark figures. I need to emphasise that the figures below are indicative. I have included the benchmarks as attachments to this article, just in case you want to validate them on your own service.
20–40. The number of files that you can open and read per second, if the file system cache is not primed.
1,500–2,500. The number of files that you can open and read per second, if the file system cache is primed with their contents.
300,000–400,000. The number of lines per second that the PHP interpreter can compile.
20,000,000. The number of PHP instructions per second that the PHP interpreter can interpret.
500-1,000. The number of MySQL statements per second that the PHP interpreter can call, if the database cache is primed with your table contents.
For more see More on optimising PHP applications in a Webfusion shared service where you can copy the benchmarks to run yourself.
MySQL connection
The easiest thing to do here is to pool the connection. I use my own mysqli class extension which uses a standard single-object-per-class template. In my case any module can issue a:
$db = AppDB::get();
to return this object. This is cheap as it is an internal call involve half a dozen PHP opcodes.
An alternative but traditional method is to use a global to hold the object and just do a
global $db;
in any function that need to use it.
Footnote for Small Applications
You suggested combining all includes into a single include file. This is OK for stable production, but a pain during testing. Can I suggest a simple compromise? Keeps them separate for testing but allow loading of a single composite. You do this in two parts (i) I assume each include defines a function or class, so use a standard template for each include, e.g.
if( !function_exists( 'fred' ) ) {
require "include/module1.php";
}
Before any loads in the master script simple do:
#include "include/_all_modules.php";
This way, when you are test you delete _all_modules.php and the script falls back to loading individual modules. When you're happy you can recreate the _all_modules.php. You can event do this server side by a simple "release" script which does a
system( 'cp include/[a-z]*.php include/_all_modules.php' );
This way, you get the best of both worlds
It depends on the MySQL client code, I know for one that connections often get reused when opening a MySQL connection with the same parameters.
Personally I wouldd only initialize the database connection in the front controller (your index.php file), because everything should come through there anyway.
You could use the include_once() or require_once() methods to ensure that PHP only parses them once, thus saving processing time. This would be particularly valuable if you suspect that your code might attempt to include files more than once per script execute.
http://php.net/manual/en/function.include-once.php
I would imagine that using .htaccess to parse URLs would always use more resources than any other method, purely because those rules would be activated upon every single .php file request your server encountered.
I've heard of two caching techniques for the PHP code:
When a PHP script generates output it stores it into local files. When the script is called again it check whether the file with previous output exists and if true returns the content of this file. It's mostly done with playing around the "output buffer". Somthing like this is described in this article.
Using a kind of opcode caching plugin, where the compiled PHP code is stored in memory. The most popular of this one is APC, also eAccelerator.
Now the question is whether it make any sense to use both of the techniques or just use one of them. I think that the first method is a bit complicated and time consuming in the implementation, when the second one seem to be a simple one where you just need to install the module.
I use PHP 5.3 (PHP-FPM) on Ubuntu/Debian.
BTW, are there any other methods to cache PHP code or output, which I didn't mention here? Are they worth considering?
You should always have an opcode cache like APC. Its purpose is to speed up the parsing of your code, and will be bundled into PHP in a future version. For now, it's a simple install on any server and doesn't require you write or change any code.
However, caching opcodes doesn't do anything to speed up the actual execution of your code. Your bottlenecks are usually time spent talking to databases or reading to/from disk. Caching the output of your program avoids unnecessary resource usage and can speed up responses by orders of magnitude.
You can do output caching many different ways at many different places along your stack. The first place you can do it is in your own code, as you suggested, by buffering output, writing it to a file, and reading from that file on subsequent requests.
That still requires executing your PHP code on each request, though. You can cache output at the web server level to skip that as well. Crafting a set of mod_rewrite rules will allow Apache to serve the static files instead of the PHP code when they exist, but you'll have to regenerate the cached versions manually or with a scheduled task, since your PHP code won't be running on each request to do so.
You can also stick a proxy in front of your web server and use that to cache output. Varnish is a popular choice these days and can serve hundreds of times more request per second with caching than Apache running your PHP script on the same server. The cache is created and configured at the proxy level, so when it expires, the request passes through to your script which runs as it normally would to generate the new version of the page.
You know, for me, optcache , filecache .. etc only use for reduce database calls.
They can't speed up your code. However, they improve the page load by using cache to serve your visitors.
With me, APC is good enough for VPS or Dedicated Server when I need to cache widgets, $object to save my mySQL Server.
If I have more than 2 Servers, I like to used Memcache , they are good on using memory to cache. However it is up to you, not everyone like memcached, and not everyone like APC.
For caching whole web page, I ran a lot of wordpress, and I used APC, Memcache, Filecache on some Cache Plugins like W3Total Cache. And I see ( my own exp ): Filecache is good for caching whole website, memory cache is good for caching $object
Filecache will increase your CPU if your hard drive is slow, and Memory cache is terrible if you don't have enough memory on your VPS.
An SSD HDD will be super good speed to read / write file, but Memory is always faster. However, Human can't see what is difference between these speed. You only pick one method base on your project and your server ( RAM, HDD ) or are you on a shared web hosting?
If I am on a shared hosting, without root permission, without php.ini, I like to use phpFastCache, it a simple file cache method with set, get, stats, delete only.
In Addition, I like to use .htaccess to cache static files like images, js, css or by html headers. They will help visitors speed up your page, and save your server bandwidth.
And If you can use .htaccess to redirect to static .html cache if you cache whole page is a great thing.
In future, APC or some Optcache will be bundle into PHP version, but I am sure all the cache can't speed up your code, they use to:
Reduce Database / Query calls.
Improve the speed of page load by use cache to serve.
Save your API Transactions ( like Bing ) or cURL request...
etc...
A lot of times, when it comes to PHP web applications, the database is the bottleneck. As such, one of the best things you can do is to use memcached to cache results in memory. You can also use something like xhprof to profile your code, and really dial in on what's taking the most time.
Yes, those are two different cache-techniques, and you've understood them correctly.
but beware on 1):
1.) Caching script generated output to files or proxies may render problems
if content change rapidly.
2.) x-cache exists too and is easy to install on ubuntu.
regards,
/t
I don't know if this really would work, but I came across a performance problem with a PHP script that I had. I have a plain text file that stores data as a title and a URL tab separated with each record separated by a new line. My script grabs the file at each URL and saves it to its own folder.
Then I have another page that actually displays the local files (in this case, pictures) and I use a preg_replace() to change the output of each line from the remote url to a relative one so that it can be displayed by the server. My tab separated file is now over 1 MB and it takes a few SECONDS to do the preg_replace(), so I decided to look into output caching. I couldn't find anything definitive, so I figured I would try my own hand at it and here's what I came up with:
When I request the page to view stuff locally, I try to read it from a variable in a global scope. If this is empty, it might be that this application hasn't run yet and this global needs populated. If it was empty, read from an output file (plain html file that literally shows everything to output) and save the contents to the global variable and then display the output from the global.
Now, when the script runs to update the tab separated file, it updates the output file and the global variable. This way, the portion of the script that actually does the stuff that runs slowly only runs when the data is being updated.
Now I haven't tried this yet, but theoretically, this should improve my performance a lot, although it does actually still run the script, but the data would never be out of date and I should get a much better load time.
Hope this helps.
What's the best way (ways?) to speed up a php web site and how much faster it can using this or that way?
PHP isn't really the kind of language where you can do micro-optimizations, or just work on the code alone. There's really no point. Although PHP isn't particularly fast, PHP itself is rarely the bottleneck in a given web site.
You need to work out where that bottleneck is before you can fix it. There are a lot of common bottlenecks, with common solutions. It's difficult to generalize, given so few details, but there are a lot of performance hints that apply to most web sites.
The first good place to look is actually on the client side, rather than the server side. How large are your pages (including images, CSS, JavaScript and the like)? How many HTTP requests does a single page view require? Use something like Firebug (and the YSlow add-on for Firebug) to see how long your page actually takes to load, and which bits of your page cause the problem. Some general hints:
Work out ways to shrink the CSS and JavaScript - remove anything you don't need, and run the rest through a tool like YUI Compressor.
If you have multiple CSS and JavaScript files, try to combine them into a single file.
Optimize all of your images as much as possible, and see if you can combine any of those into a single file using CSS sprites or similar. PunyPNG is good for lossless images. A decent JPEG encoder (NOT Photoshop) is good for photos.
Move the CSS to the top of the page, and the JavaScript to the bottom, so the browser can render the page before the JavaScript has finished downloading.
Make sure that all of your CSS, JavaScript and HTML are being served compressed.
Make sure that you're using appropriate caching - if a file hasn't changed, there's no point in re-downloading it.
Once you've got the client side out of the way, you might have to turn your attention to the server side.
Install an opcode cache, like APC, XCache, or Zend Optimizer. It's very easy to do, and will always provide some improvement. Once you've done that, profile your pages, to find out where the time is actually being spent.
More likely than not, you'll be spending most of your time waiting for the database to return results. So, at a bare minimum:
Work out which queries are taking the longest, and work on them first. Use your head though - a query that takes five seconds on an admin page that nobody looks at is not as important as a query that takes one second on the front page.
Make sure that your query uses appropriate indexes. No common query should ever need to do a full table scan. Certain kinds of sorting or grouping may be unable to use indexes - try to avoid them, or modify the query so that it can use indexes.
Make sure that your queries aren't using temporary tables.
Use the EXPLAIN keyword - it's very useful.
Tune the database server itself. MySQL is generally not optimized for performance.
Once you've done that, it's usually best to start working out how to use caching. The best way to speed PHP code up is to reduce the amount of work it has to do.
Make sure your database's query cache is working properly.
Use something like Memcached to store frequently used results, instead of getting them from the database.
If you have enough memory, try to keep everything in Memcached, resorting to the database only when something isn't present in the cache.
If you have chunks of pages that are dynamic, but the same for all users, try caching those chunks. For example, if two users are looking at an article, the article itself is going to be exactly the same for each user, even if the rest of the page isn't. Generate the HTML for the article, and chuck it in the cache.
If you have lots of non-authenticated users, it's entirely possible that they'll all be seeing the exact same page. Two non-authenticated users looking at the above article won't just see an identical article - they'll see an identical page, right down to the login links. Set your PHP scripts up so you can use HTTP caching headers (check the last modified date, and return a 304 Not Modified if it's not been changed). Once you've done that, stick a Squid reverse-proxy in front of the webserver, and let Squid serve pages out of it's cache.
After that point, the general approach is to start using more servers, and the problem becomes one of scaling, rather than raw speed. The general plan is to make sure that your website has a shared-nothing architecture - all persistent data is stored in the database. Then, you install multiple webservers, move the database server to a separate machine, and run the entire thing behind a caching reverse proxy. To add more capacity, you add more machines.
One way: php accelerators, e.g. APC.
Another; read blog articles, e.g. performance tuning overview.
A general question i would say. Try looking for optimazation tips online...
Several parameters are involved:
I/O access (using it a lot - file_exists, is_file overheads)
Database access (optimize queries, use stored procedures, check your db cache)
Using an opcode cache (like APC)
Compressing output
Serving js/css minified and compressed (and using subdomains to deliver them to the browser)
Using memcache to cache data into memory for faster access
You can use benchmarking tools to test your environment before and after the optimizations.
Try apache bench for example.
Filesize.
A file of 500 KB takes longer to download then a file of 300 KB. So optimize and crop as much as you can.
Accelators
Self explainable: List of PHP accelerators
Server upgrade
Though this costs money, when dealing with a lot of traffic, it will have impact on how fast the .php files gets processes and how fast data will be send to the user.
I don't recommend this though since there are other (free) ways to improve speed.
Don't user external resources
If you are linking some images trough other sites, the speed of the downloading will not be in your control. Instead, if you plan on using images from others download them to your own server first (or upload them to your own provider) and load them that way.
Review and improve your code
Find short cuts, remove unnecessary code, delete unused variables, reuse others etc.
There are other ways but I believe the above information has the most impact on your speed
You should probably do some search for existing answers to this question, however...
APC for opcode caching
Memcached for object storing (to reduce the number of database queries)
Check for / optimize slow SQL queries
Measure and find bottlenecks
Don't rely on (slow) web services on each page load, etc.
Yahoo has got some good basic advice on speeding up web pages, much of it very easy to implement. You may also want to download yslow + firebug for firefox; they will help indicate possible basic bottlenecks from a client request perspective.
The rest of the advice here is good, so I wont add much else other than; don't bother optimising any code until you're 100% sure that you've found a bottleneck. I can't stress that enough. Don't waste time tweaking code or implementing new things (ie caching) because you "feel" will make things quicker, act only on real evidence (ie performance profiling).
I have a PHP client that requests an XML file over HTTP (i.e. loads an XML file via URL). As of now, the XML file is only several KB in size. A problem I can foresee is that the XML becomes several MBs or Gbs in size. I know that this is a huge question and that there are probably a myriad of solutions, but What ideas do you have to transport this data to the client?
Thanks!
based on your use case i'd definitely suggest zipping up the data first. in addition, you may want to md5 hash the file and compare it before initiating the download (no need to update if the file has no changes), this will help with point #2.
also, would it be possible to just send a segment of XML that has been instead of the whole file?
Ignoring how well a browser may or may-not handle a GB-sized XML file, the only real concern I can think of off the top of my head is if the execution time to generate all the XML is greater than any execution time thresholds that are set in your environment.
PHP's max_execution_time setting
PHP's set_time_limit() function
Apache's TimeOut Directive
Given that the XML is created dynamically with your PHP, the simplest thing I can think of is to ensure that the file is gzipped automatically by the webserver, like described here, it offers a general PHP approach and an Apache httpd-specific solution.
Besides that, having a browser (what else can be a PHP-client?) do such a job every night for some data synchonizing sounds like there must be a far simpler solution somewhere else.
And, of course, at some point, transferring "a lot" of data is going to take "a lot" of time...
The problem is that he's syncing up two datasets. The problem is completely misstated.
You need to either a) keep a differential log of changes to dataset A to that you can send that log to dataset B, or b) keep two copies of the dataset (last nights and the current dataset), and then compare them so you can then send the differential log from A to B.
Welcome to the world of replication.
The problem with (a) is that it's potentially invasive to all of your code, though if you're using an RDBMS you could do some logging perchance via database triggers to keep track of inserts/updates/deletes, and write the information in to a table, then export the relevant rows as your differential log. But, that can be nasty too.
The problem with (b) is the whole "comparing the database" all at once. Fine for 100 rows. Bad for 10^9 rows. Nasty nasty.
In fact, it can all be nasty. Replication is nasty.
A better plan is to look into a "real" replication system designed for the particular databases that you're running (assuming you're running a database). Something that perhaps sends database log records over for synchronization rather than trying to roll your own.
Most of the modern DBMS systems have replication systems.
Gallery2, which allows you to upload photos over http, makes you set up a couple of php parameters, post_max_size and upload_max_filesize, to allow larger uploads. You might want to look into that.
It seems to me that posting large files has problems with browser time-outs and the like, but on the plus side it works with proxy servers and firewalls better than trying a different file upload protocol.
Thanks for the responses. I failed to mention that transferring the file should be relatively fast (few mintues max, is this even possible?). The XML that is requested will be parsed and inserted into a database every night. The XML may be the same from the night before, or it may be different. One solution that was proposed is to zip the xml file and then transfer it. So there are basically two requirements: 1. it has to relatively fast 2. it should minimize the number of writes to the database.
One solution that was proposed is to zip the xml file and then transfer it. but that only satisfies (1)
Any other ideas?
Are there any algorithms that I could apply to compress the XML? How are large files such as MP3s being downloaded in a matter of seconds?
PHP receiving GB's of data will take long and is overhead.
Even more perceptible to flaws.
I would - dispatch the assignment to a shellscript (wget with simple error catching) that is not bothered by execution time and on failure could perhaps even retry on its own merit.
Am not experienced with this, but though one could use exec() or alike, these sadly run modal.
Calling a script with **./test.sh &** makes it run in background and solves that problem / i guess. The script could easily let your PHP pick it back up via a wget `http://yoursite.com/continue-xml-stuff.php?id=1049381023&status=0´. The id could be a filename, if you don't need to backtrack lost requests. The status would indicate how the script ended up handling the request.
Have you thought about using some sort of version control system to handle this? You could leverage its ability to calculate and send just the differences in the files, plus you get the added benefits of maintaining a version history of your file.
Since I don't know the details of your situation I'll throw question out there. Just for sake of argument does it have to be HTTP? FTP is much better suited for large data transfer and can be automated easily via PHP or Perl.
If you are using Apache, you might also consider Apache mod_gzip. This should allow you to compress the file automatically and the decompression should also happen automatically, as long as both sides accept gzip compression.