I have an internal website using LAMP architecture. My main page takes around 10 secs to load the data. There isn't lot of data, around 4-5k records. I dont have any complex MYSQL queries, but have a lot of them i.e. around 10-15 queries. Basically I'm extracting meta-data to display on the page. These are very simple queries. I have lot of PHP and javascript logic which is of medium complexity. I can't remove any of that. I have around 1800 lines of code in that page and I'm using datatables to display data.
The datatable contains 25 columns and lot of html select elements.
So how will I know what is causing performance bottleneck in this page? I tried to be as clear as possible, but please let me know if you have any questions.
Appreciate your time and help.
use any php profiler tool like zend profiler to see what section is taking much time
http://erichogue.ca/2011/03/linux/profiling-a-php-application/
Try to get all the data at once. I've had issues in the past where using multiple selects actually takes longer than more efficiently querying (also make sure you use a sql connection for as long as possible). Other than that profiling is the way to go in case your other code needs optimization.
Related
I have an application that is fetching several e-commerce websites using Curl, looking for the best price.
This process returns a table comparing the prices of all searched websites.
But now we have a problem, the number of stores are starting to increase, and the loading time actually is unacceptable at the user experience side. (actually 10s pageload)
So, we decided to create a database, and start to inject all Curl filtered result inside this database, in order to reduce the DNS calls, and increase Pageload.
I want to know, despite of all our efforts, is still an advantage implement a Memcache module?
I mean, will it help even more or it is just a waste of time?
The Memcache idea was inspired by this topic, of a guy that had a similar problem: Memcache to deal with high latency web services APIs - good idea?
Memcache could be helpful, but (in my opinion) it's kind of a weird way to approach the issue. If it was me, I'd go about it this way:
Firstly, I would indeed cache everything I could in my database. When the user searches, or whatever interaction triggers this, I'd show them a "searching" page with whatever results the server currently has, and a progress bar that fills up as the asynchronous searches complete.
I'd use AJAX to add additional results as they become available. I'm imagining that the search takes about ten seconds - it might take longer, and that's fine. As long as you've got a progress bar, your users will appreciate and understand that Stuff Is Going On.
Obviously, the more searches go through your system, the more up-to-date data you'll have in your database. I'd use cached results that are under a half-hour old, and I'd also record search terms and make sure I kept the top 100 (or so) searches cached at all times.
Know your customers and have what they want available. This doesn't have much to do with any specific technology, but it is all about your ability to predict what they want (or write software that predicts for you!)
Oh, and there's absolutely no reason why PHP can't handle the job. Tying together a bunch of unrelated interfaces is one of the things PHP is best at.
Your result is found outside the bounds of only PHP. Do not bother hacking together a result in PHP when a cronjob could easily be used to populate your database and your PHP script can simply query your database.
If you plan to only stick with PHP then I suggest you change your script to index your database from the results you have populated it with. To populate the results, have a cronjob ping a PHP script that is not accessible to the users which will perform all of your curl functionality.
I'm currently developing a Zend Framework project, using Doctrine as ORM.
I ran into the typical situation where you have to show a list of items (around 400) in a table, and of course, I don't want to show them all at once.
I've already used Zend_Paginator before (only some basic usage), but i always used to get all the items from the DB, and then paginate them, but now it doesn't feel quite right.
My question is this: is it better to get all items from DB first and then "paginate" them, or to get "pages" of items as they are requested? which would have a larger impact on performance?
For me, it is better to get a part of the data and then paginate through them.
If you get All the data from a DB you paginate with the help of JavaScript.
The first opening of the page will take a long time (for 400 rec. is OK).
Browser has a limited memory. If a user opens up a lot of tabs in the browser
and you take a lot of memory (with your data)
this will slow down the speed of the browser and the speed of your application.
You have only 400 records but the increase of the data happens very often.
At worst, the whole browser may break when the page is opened.
What if browser doesn't support JS ...
If you get part of the data from DB, the only defect is if
a user has a very slow Internet speed(but this is the defect in the first option - in the first refresh of the page).
If someone wants to get to another page, it will take a little bit longer than JavaScript.
The second option is better(for me) in the long run, because if it works it will work for years.
The database engines are usually best suited to do the retrieval for you. So, in general, if you can delegate a data-retrieval task to the DB engine instead of doing it in-memory and using your programming language, the best bet for performance is to let the DB engine do it for you.
But also remember that if you don't configure the indices correctly or don't run a good query, you won't get the best result out of your DB engine.
However, most DB engines nowadays are capable of optimizing your queries for you and running them in their most normal form.
I have a Wordpress Plug-in with users requesting a feature that is the view counter.
I know only a few approaches on making a view counter, and the problem is that I want to optimize for performance and memory issues.
I have done a small amount of research and it seems, "mod_log_mysql", may be a great approach, but I do not have any prior knowledge on how this mod works, nor could have any ideas on how to connect it with the Wordpress Plug-in.
Or I could use a database. When the page is viewed, an update or a insert (which is said to be faster than a update) event occurs.
Thus my following option are:
Update/Insert on server side when
the page is called for.
Research more on mod_log_mysql and
find a way to connect it with the
plug-in.
Find a Premade view counter.
If there is a better approach, I would like to hear them in hoping this will solve my problem.
It really depends on what you want to achieve and how much time you want to spend on it.
If you need more than a simple view counter per page/event, go for a premade one.
If you need something simple, I would go for option #1.
If you're worried about performance, use a memory table for staging the 'counts', and then have a php script move that into a regular table periodically (i.e. using a cronjob). I wouldn't expect updating a view counter in a memory table to have any significant performance impact.
Option #2 could easily fall into premature optimization category.
Option 1 by far seems the easiest and probably most efficient. There is little overhead associated with making a single call to a database whose connection is already open from other operations done on the page previously.
So i have a site that I am working on that has been touched by many developers over time and as new features arised the developers at the time felt it necessary to just add another query to get the data that is needed. Which leaves me with a php page that is slow and runs maybe 70 queries. Some of the queries are in the actual PHP file for this page and some are scattered throughout many different function. Now i have the responsibility of trying to speed up the page to meet certain requirements. I am seeking the best course of action.
Is there a way to print all the queries that are running other then going through the file and finding each and every one?
should I cache the queries that are slow using memcached?
Is there an idea that anyone has had to help me speed up the page?
Is there a plugin or tool to analyze the queries, I am using YSlow and there is nothing there to look at queries?
Something I do is to have a my_mysql_query(...) function that behaves as mysql_query(...) but which I can then tailor to log out the execution time together with the text of the query. MySQL can log slow queries with very little fiddling - see here.
If there is not a central query method that is called to run each query, then the only options is to look for each query and find where it is in the code. Otherwise you could go to that query function and print each query that runs through it.
Using cache will depend on how often the data changes. If it changes frequently, it may not give you any performance boost to cache it.
One idea to help you speed up the page is to do the following:
group like queries into the same query and use the data in multiple parts
consider breaking the page into multiple locations
Depending on the database you are using. There are analyze functions in some databases that will help you optimize your queries. For example, you can use EXPLAIN with mysql. (http://dev.mysql.com/doc/refman/5.0/en/explain.html) You may need to consider consulting with a DBA on the issue.
Good luck.
Okay, so I'm sure plenty of you have built crazy database intensive pages...
I am building a page that I'd like to pull all sorts of unrelated database information from. Here are some sample different queries for this one page:
article content and info
IF the author is a registered user, their info
UPDATE the article's view counter
retrieve comments on the article
retrieve information for the authors of the comments
if the reader of the article is signed in, query for info on them
etc...
I know these are basically going to be pretty lightning quick, and that I could combine some; but I wanted to make sure that this isn't abnormal?
How many fairly normal and un-heavy queries would you limit yourself to on a page?
As many as needed, but not more.
Really: don't worry about optimization (right now). Build it first, measure performance second, and IFF there is a performance problem somewhere, then start with optimization.
Otherwise, you risk spending a lot of time on optimizing something that doesn't need optimization.
I've had pages with 50 queries on them without a problem. A fast query to a non-large (ie, fits in main memory) table can happen in 1 millisecond or less, so you can do quite a few of those.
If a page loads in less than 200 ms, you will have a snappy site. A big chunk of that is being used by latency between your server and the browser, so I like to aim for < 100ms of time spent on the server. Do as many queries as you want in that time period.
The big bottleneck is probably going to be the amount of time you have to spend on the project, so optimize for that first :) Optimize the code later, if you have to. That being said, if you are going to write any code related to this problem, write something that makes it obvious how long your queries are taking. That way you can at least find out you have a problem.
I don't think there is any one correct answer to this. I'd say as long as the queries are fast, and the page follows a logical flow, there shouldn't be any arbitrary cap imposed on them. I've seen pages fly with a dozen queries, and I've seen them crawl with one.
Every query requires a round-trip to your database server, so the cost of many queries grows larger with the latency to it.
If it runs on the same host there will still be a slight speed penalty, not only because a socket is between your application but also because the server has to parse your query, build the response, check access and whatever else overhead you got with SQL servers.
So in general it's better to have less queries.
You should try to do as much as possible in SQL, though: don't get stuff as input for some algorithm in your client language when the same algorithm could be implemented without hassle in SQL itself. This will not only reduce the number of your queries but also help a great deal in selecting only the rows you need.
Piskvor's answer still applies in any case.
Wordpress, for instance, can pull up to 30 queries a page. There are several things you can use to stop MySQL pull down - one of them being memchache - but right now and, as you say, if it will be straightforward just make sure all data you pull is properly indexed in MySQL and don't worry much about the number of queries.
If you're using a Framework (CodeIgniter for example) you can generally pull data for the page creation times and check whats pulling your site down.
As other have said, there is no single number. Whenever possible please use SQL for what it was built for and retrieve sets of data together.
Generally an indication that you may be doing something wrong is when you have a SQL inside a loop.
When possible Use joins to retrieve data that belongs together versus sending several statements.
Always try to make sure your statements retrieve exactly what you need with no extra fields/rows.
If you need the queries, you should just use them.
What I always try to do, is to have them executed all at once at the same place, so that there is no need for different parts (if they're separated...) of the page to make database connections. I figure it´s more efficient to store everything in variables than have every part of a page connect to the database.
In my experience, it is better to make two queries and post-process the results than to make one that takes ten times longer to run that you don't have to post-process. That said, it is also better to not repeat queries if you already have the result, and there are many different ways this can be achieved.
But all of that is oriented around performance optimization. So unless you really know what you're doing (hint: most people in this situation don't), just make the queries you need for the data you need and refactor it later.
I think that you should be limiting yourself to as few queries as possible. Try and combine queries to mutlitask and save time.
Premature optimisation is a problem like people have mentioned before, but that's where you're crapping up your code to make it run 'fast'. But people take this 'maxim' too far.
If you want to design with scalability in mind, just make sure whatever you do to load data is sufficiently abstracted and calls are centralized, this will make it easier when you need to implement a shared memory cache, as you'll only have to change a few things in a few places.