I have a site that gets around 1000 page views a minute when traffic is high
and the page has a js code which stores the browser details of the
visitor into mysql using a php file for connecting
to db.
For eg browser.js calls storebrowserdata.php
Is there a way for me to cache the php file and js file without
affecting my stats data that is stored in the db??
When the traffic is high during the day the site slows down and cpu
utilization also goes up.
PHP is interpreted. If you want to grab each visitor's browser/user agent information on each page load then you kinda need to run that script each time; you can't cache it.
If this functionality is slowing down your site, either use an alternative solution like Google Analytics, or investigate a NoSQL solution like Mongo DB that offers atomic updates and in general runs faster than MySQL.
you can cache js file as well as php one.
but it will do nothing good for you.
it's database update that most likely slows down your site.
what certain operation consumes most CPU? Did you profile your application? Is it the only page using PHP mysql on your site?
If you need to control the flow of process executed, you can use a Queue and control the rate of execution.
About this system, this is a great open source project: http://www.rabbitmq.com/
Related
I want to use session values in my script which are stored at the server using php can any one kindly explain the process to achieve this.
I want to build a chat app for this am planning to use those session values.
Assume usera and userb are logged in and their userid is sessioned based on this scenario i want to do a chat app.
Now i had done the app but i had used setinterval function of Javascript and am calling the chats i want to avoid the database hits on every 3 mill sec.
Kindly Help me out
Thanks In Advance
You're basically attempting to use PHP session files as a file cache.
Instead, you should use an object caching system such as Memcached or Redis. If memory caching isn't an option (shared hosting, etc), then you could implement your own file cache (or you could use something like PHPFastCache, which supports file caching).
Note: File caching for a chat app may or may not speed up your application. It depends on how you implement it and a number of other factors.
Hi put the session value in input box,
<input type='hidden' id='session_value' value='<?php $_SESSION['value']?>'>
Using the id fetch the session value in script,
<script>
var session_value = document.getElementById ( "session_value" );
</script>
3ms is insanely short delay to run a polling chat system. I suggest increasing it to at least 200ms but preferably around 1000ms.
$_SESSION values are per user and not recommended for viewing a chat stream for a number of reasons. Instead it sounds like you are looking more to just update the chat feed.
The database unless it is hosted on another server and $_SESSION will be the equivalent, since the database is effectively files as well. The database will actually generally be faster than reading raw file storage since Queries are normally cached and Indexing helps lookup records quicker. In addition you won't have to worry about concurrent connections to the files either.
If anything enable OPCache and install APCu for your PHP installation, to help aid the serving of requests. OPCache will cache your compiled OP code into memory so that subsequent requests to the file won't need to be recompiled.
APCu will act as your file cache, again storing your rendered data in memory.
Additionally many Database Frameworks such as Doctrine can also utilize APC caching for query and result caching.
Instead of using a InnoDB or MyISAM storage engines for your chat messages I suggest trying the MEMORY storage engine.
So instead of accessing the File System I/O your database would instead be utilizing the Memory I/O. The general concept is few writes, many reads. Since one person writes to the database, requires everyone to read the data. Just keep in mind that the Memory storage engine is temporary and is lost if the server restarts or power is lost.
For more information see: https://dev.mysql.com/doc/refman/5.6/en/memory-storage-engine.html
Overall if you are able, I would suggest look at using Socket IO (Websockets) instead of either database or file based caching. This puts the load on the clients instead of the server, and everything occurs in real-time instead of polling for changes.
For some examples see:
Ratchet http://socketo.me/
React http://reactphp.org/
Node.js http://tutorialzine.com/2014/03/nodejs-private-webchat/
Are they looping scripts or do they have some code on watch? How do they do it?
I'm looking for a tutorial to write a script so I can watch a database table for timestamp change, and when it changes, the code automatically grabs that row and echoes it.
Facebook depends on the user's browser (javascript timer) to pull the newsfeed every X seconds (think ajax). They also keep a cache of each user's feed.
Regarding performance, big Internet companies have huge horsepower to host their apps (think in several hundres of thousands of servers), but also they optimize, optimize and optimize again. Google created it's own webserver optimized to their needs, in Facebook's case, they developed their own database, called Cassandra, and then donated the source code to the Apache Foundation. Facebook also developed their own php compiler to convert php code into faster C++ native code called "hiphop for php", wich is also freely available.
On the other hand, if your queries are optimized, your database indexes are fine tuned, and your dbms is configured to perform well there should not be a great performance impact for what you are requiring.
I'm writing a website using PHP / MySQL. The website should allow users browsers to be automatically updated when a table is updated in a database (which is caused by someone inserting something though the PHP site). Basically, if the state of the table changes other users on the website should immediately see the changes.
I'm relatively new to PHP / AJAX and other web technologies. The only way I can think of doing this is to have the clients manually recheck the database every second or two via a timer (but this seems like a lot of wasted bandwidth). Is there some way to have the users automatically notified when the database changes?
Thank you!
You already had the solution in mind. You can use ajax to automatically check for changes.
It's not possible for the server to contact you when something has changed. It doesn't have to be bandwidth intensive when you lower the retrieval intervals. You can have Ajax check for a blank page with as little as possible text. That way the bandwidth usage wont be too bad.
Facebook does it this way too.
1) AJAX Polling.
As you say, use ajax to call the PHP on the server side to get
updates.
2) HTML5 Websockets.
Websockets are a new feature of HTML5 which allow the browser to keep a bi-directional TCP connection open to the server, where data can be transmitted both ways without polling. Currently this isn't as widely supported as ajax polling is.
3) Java Applet/ActiveX control.
Generally the most undesirable solution because of the user having to install third party software.
I have a dynamic php (Yii framework based) site. User has to login to do anything on the site. I am trying to understand how caching and CDN work; and I am a bit confused.
Caching (memcache):
My site has a good amount of css, js, and images. I've been given to understand that enabling caching ("memcache"?) will GREATLY speed up my site. But this has me confused. How does caching help? I mean, how can you cache something that's coming out of DB for each user separately? For instance, user-1 logs-in, he sees his control panel. User-2 logs-in, user 2 will see their control panel.
How do I determine what to cache? Plus, how do I enable caching (memcaching)?
CDN:
I have been told to use a content delivery network like CloudFlare. It is suppose to automatically cache my site. So, when my user-1 logs in, what will it cache? Will it cache only the homepage CSS, JS, and homepage images? Because everything else requires login? What happens when user logs-out? I mean, do "sessions" interfere with working of a CDN?
Does serving up images via CDN reduce significant load on my server? I don't have much cash for getting a clustered-server configuration. So, I just want my (shared) server to be able to devote all its resources in processing PHP code. So, how much load can I save by using "caching" (something like memcache) and/or "CDN" (something like CloudFlare)?
Finally,
What would be general strategy to implement in this scenario for caching, cdn, and basic performance optimization? do I need to make any changes to my php-code to enable CDN like CloudFlare and to enable/implement/configure caching? What can I do that would take least amount of developer/coding time and will make my site run much much faster?
Oh wait, some of my pages like "about us" page etc. are going to be static html too. But they won't get as many hits. Except for maybe the iFrame page that will be used for my Facebook Page.
I actually work for CloudFlare & thought I would hop in to address some of the concerns.
"do I need to make any changes to my php-code to enable CDN like
CloudFlare and to enable/implement/configure caching? What can I do
that would take least amount of developer/coding time and will make my
site run much much faster?"
No, nothing like a need to re-write urls, etc. We automatically cache static content by file extension. This does require changing your DNS to point to us, however.
Does serving up images via CDN reduce significant load on my server?
Yes, and it should also help most visitors access the site faster and save you a fair amount on bandwidth.
"Oh wait, some of my pages like "about us" page etc. are going to be
static html too."
CloudFlare doesn't cache HTML by default. You use PageRules to setup more advanced caching options for things like static HTML.
Caching helps because instead of performing disk io for each user the data is stored in the memory, ie memcached. This provides a SIGNIFICANT increase in performance.
Memcache is generally used for cacheing data ie query results.
http://pureform.wordpress.com/2008/05/21/using-memcache-with-mysql-and-php/
There are lots of tutorials.
I have only ever used amazon s3 which is is not quite a cdn. It is more of a storage platform but still it helps to take the load off of my own servers when serving media.
I would put all of your static resources on a CDN so your own server would not have to serve these. It would not require any modifcation to your php code. This includes JS, and CSS.
For your static pages (your about page) I'd make sure that php isn't processing that since there is no reason for it. Your web server should serve it directly.
Cacheing will require changes to your code. For cacheing a normal flow is:
1) user makes a request
2) check if data is in cache
3) if it is not in cache do the DB query and put it in cache
4) if it is in cache retrieve it
5) return data.
You can cache anything that requires disk io and you should see a speed up.
Memcached works by storing database information (usually from a remote server or even a database engine on the same server) in a flat file format in the filesystem of the web server. Accessing a flat file directly to retrieve data stored in a regulated format is much much muuuuuch faster than accessing that data from a remote query each time. This is typically useful when you have data that can be safely stored for certain periods of time as it is not subject to regular changes.
The way this works is that if you want to store a user's account information in a cache to speed up loading pages where that user is logged in. You would load the information and cache it locally. On any subsequent requests for that data, it will load in a fraction of the time it normally would take to load that information from the database itself. Obviously you will need to make sure that you update/recache that information if the user changes it while logged in, but you will greatly reduce the time it takes to serve up pages if you implement a caching system that can minimize the time spent waiting on the database.
I'm personally not familiar with CloudFlare so I can't offer any advice to that effect, but in terms of implementing caching in your application, you should check out:
http://code.google.com/p/memcached/wiki/NewOverview
And read the rest of the Wiki entries there which cover installation/implementation/etc. That should get you started on the right track.
I've heard of two caching techniques for the PHP code:
When a PHP script generates output it stores it into local files. When the script is called again it check whether the file with previous output exists and if true returns the content of this file. It's mostly done with playing around the "output buffer". Somthing like this is described in this article.
Using a kind of opcode caching plugin, where the compiled PHP code is stored in memory. The most popular of this one is APC, also eAccelerator.
Now the question is whether it make any sense to use both of the techniques or just use one of them. I think that the first method is a bit complicated and time consuming in the implementation, when the second one seem to be a simple one where you just need to install the module.
I use PHP 5.3 (PHP-FPM) on Ubuntu/Debian.
BTW, are there any other methods to cache PHP code or output, which I didn't mention here? Are they worth considering?
You should always have an opcode cache like APC. Its purpose is to speed up the parsing of your code, and will be bundled into PHP in a future version. For now, it's a simple install on any server and doesn't require you write or change any code.
However, caching opcodes doesn't do anything to speed up the actual execution of your code. Your bottlenecks are usually time spent talking to databases or reading to/from disk. Caching the output of your program avoids unnecessary resource usage and can speed up responses by orders of magnitude.
You can do output caching many different ways at many different places along your stack. The first place you can do it is in your own code, as you suggested, by buffering output, writing it to a file, and reading from that file on subsequent requests.
That still requires executing your PHP code on each request, though. You can cache output at the web server level to skip that as well. Crafting a set of mod_rewrite rules will allow Apache to serve the static files instead of the PHP code when they exist, but you'll have to regenerate the cached versions manually or with a scheduled task, since your PHP code won't be running on each request to do so.
You can also stick a proxy in front of your web server and use that to cache output. Varnish is a popular choice these days and can serve hundreds of times more request per second with caching than Apache running your PHP script on the same server. The cache is created and configured at the proxy level, so when it expires, the request passes through to your script which runs as it normally would to generate the new version of the page.
You know, for me, optcache , filecache .. etc only use for reduce database calls.
They can't speed up your code. However, they improve the page load by using cache to serve your visitors.
With me, APC is good enough for VPS or Dedicated Server when I need to cache widgets, $object to save my mySQL Server.
If I have more than 2 Servers, I like to used Memcache , they are good on using memory to cache. However it is up to you, not everyone like memcached, and not everyone like APC.
For caching whole web page, I ran a lot of wordpress, and I used APC, Memcache, Filecache on some Cache Plugins like W3Total Cache. And I see ( my own exp ): Filecache is good for caching whole website, memory cache is good for caching $object
Filecache will increase your CPU if your hard drive is slow, and Memory cache is terrible if you don't have enough memory on your VPS.
An SSD HDD will be super good speed to read / write file, but Memory is always faster. However, Human can't see what is difference between these speed. You only pick one method base on your project and your server ( RAM, HDD ) or are you on a shared web hosting?
If I am on a shared hosting, without root permission, without php.ini, I like to use phpFastCache, it a simple file cache method with set, get, stats, delete only.
In Addition, I like to use .htaccess to cache static files like images, js, css or by html headers. They will help visitors speed up your page, and save your server bandwidth.
And If you can use .htaccess to redirect to static .html cache if you cache whole page is a great thing.
In future, APC or some Optcache will be bundle into PHP version, but I am sure all the cache can't speed up your code, they use to:
Reduce Database / Query calls.
Improve the speed of page load by use cache to serve.
Save your API Transactions ( like Bing ) or cURL request...
etc...
A lot of times, when it comes to PHP web applications, the database is the bottleneck. As such, one of the best things you can do is to use memcached to cache results in memory. You can also use something like xhprof to profile your code, and really dial in on what's taking the most time.
Yes, those are two different cache-techniques, and you've understood them correctly.
but beware on 1):
1.) Caching script generated output to files or proxies may render problems
if content change rapidly.
2.) x-cache exists too and is easy to install on ubuntu.
regards,
/t
I don't know if this really would work, but I came across a performance problem with a PHP script that I had. I have a plain text file that stores data as a title and a URL tab separated with each record separated by a new line. My script grabs the file at each URL and saves it to its own folder.
Then I have another page that actually displays the local files (in this case, pictures) and I use a preg_replace() to change the output of each line from the remote url to a relative one so that it can be displayed by the server. My tab separated file is now over 1 MB and it takes a few SECONDS to do the preg_replace(), so I decided to look into output caching. I couldn't find anything definitive, so I figured I would try my own hand at it and here's what I came up with:
When I request the page to view stuff locally, I try to read it from a variable in a global scope. If this is empty, it might be that this application hasn't run yet and this global needs populated. If it was empty, read from an output file (plain html file that literally shows everything to output) and save the contents to the global variable and then display the output from the global.
Now, when the script runs to update the tab separated file, it updates the output file and the global variable. This way, the portion of the script that actually does the stuff that runs slowly only runs when the data is being updated.
Now I haven't tried this yet, but theoretically, this should improve my performance a lot, although it does actually still run the script, but the data would never be out of date and I should get a much better load time.
Hope this helps.