I have select which is counting number of rows from 3 tables, I'm using wp function $wpdb->get_var( $sql ), there are about 10 000 rows in tables. Sometimes this select takes <1 second to load sometimes more than 15. If I run this sql in phpmyadmin it always returns number of rows in less than 1 second, where could be problem?
There are a couple of things you can do.
First, do an analysis of your query. Putting EXPLAIN before the query will output data about the query and you may be able to find any problems with that.
Read more about EXPLAIN here
Also, WordPress may not have indexed the most commonly used columns.
Try indexing some of the columns which you most commonly use within your query and see if it helps.
For example:
ALTER TABLE wp_posts ADD INDEX (post_author,post_status)
Plugin
You could try a plugin such as Debug Queries which prints the queries onto the front-end and it helps debug where things are taking along time. This is recommended to run only in the dev area and not on a live website
I would also recommending hooking up something like New Relic and trying to profile what's happening on the application side. If New Relic is not an option, you might be able to use xhprof (http://pecl.php.net/package/xhprof) and/or IfP (https://code.google.com/p/instrumentation-for-php/). Very few queries will perform the same in production in an application as they do in direct SQL queries. You may have contention, read locks, or any other number of things that cause a query from php to effectively stall on its way over to MySQL. In which case you might literally see the query running very fast, but the time it takes to actually begin executing that query from PHP would be very slow. You'll definitely need to profile what's happening on the way from WordPress to MySQL and back, based on what you're saying. The tools I mentioned should all be very useful for helping you accomplish that.
Related
Okay, I know this i kinda a vague question. But I have a requirement where I need to list all the tables and the columns which are not being used in the application. I can do manual code search but that would take so much time. The number of tables to check are 140 and maximum number of fields in a table are 90.
Right now I have started searching the code for table names, and I have created an excel sheet with all the table names, and when I found a table in code I highlight that in green. So tables are bit easier.
My question really is, is there a way to speed up this process? or some methods / techniques can be applied?
Thank you?
All depends on your app size and the coding technique.
If I had a large application, I would enable full mysql log (or hack into any database wrapper I may have, to log the queries), run the application, and then extract the information from the log. However, doing so you are just moving the problem. Instead of worrying to capture all the queries, you now need to ensure to run each and every line of your application code (so you are sure that nothing escaped and that you have analyzed all the possibilities).
This is in fact called "code coverage analysis" and there are tools which will help you with that.
This said, I believe that the manual analysis may be quicker for small applications.
I suggest you to build a script that perform the job. For example, you can optain the table list in a database with a query like this:
show tables from YOUR_DATABASE;
More info: http://dev.mysql.com/doc/refman/5.0/en/show-tables.html
And then you can loop your tables and check for fields using:
show columns from YOUR_TABLE;
More info: http://dev.mysql.com/doc/refman/5.0/en/show-columns.html
Finally you can search (grep for example) your tables and fields in your code and write a log or something similar.
I'm using a table to do login, the table has a record of 2500. when i try to login it takes nearly 30-40 seconds to load. I'm using PHP and MySQL.I need to make the sql query faster to check the data. Solutions are welcomed thanks in advance pls help me out
When locating the causes of problems of performance issues, there are many things to consider in the your application stack:
Database: Indexes, Joins, Query Formation
Network in between: Routing issues, Bandwith, connectivity speed
Code: Check if your code structure is not creating unecessary delays, forexample some people have their validations span over client and server both in a single method which increases method lifetime longer. Try to put validation's core logic on database side like in stored procedures. Try to use methods with lesser overheads.
You should have included your query so we can examine it.
Don't pull all your records (e.g. don't use Select * from users) and then loop through to find a match.
Instead, use WHERE clause to limit your records to (i.e. one row). That is
SELECT col1,col2,.. FROM Users
WHERE username='username' AND password='password'
You can try that and see the performance...
So i have a site that I am working on that has been touched by many developers over time and as new features arised the developers at the time felt it necessary to just add another query to get the data that is needed. Which leaves me with a php page that is slow and runs maybe 70 queries. Some of the queries are in the actual PHP file for this page and some are scattered throughout many different function. Now i have the responsibility of trying to speed up the page to meet certain requirements. I am seeking the best course of action.
Is there a way to print all the queries that are running other then going through the file and finding each and every one?
should I cache the queries that are slow using memcached?
Is there an idea that anyone has had to help me speed up the page?
Is there a plugin or tool to analyze the queries, I am using YSlow and there is nothing there to look at queries?
Something I do is to have a my_mysql_query(...) function that behaves as mysql_query(...) but which I can then tailor to log out the execution time together with the text of the query. MySQL can log slow queries with very little fiddling - see here.
If there is not a central query method that is called to run each query, then the only options is to look for each query and find where it is in the code. Otherwise you could go to that query function and print each query that runs through it.
Using cache will depend on how often the data changes. If it changes frequently, it may not give you any performance boost to cache it.
One idea to help you speed up the page is to do the following:
group like queries into the same query and use the data in multiple parts
consider breaking the page into multiple locations
Depending on the database you are using. There are analyze functions in some databases that will help you optimize your queries. For example, you can use EXPLAIN with mysql. (http://dev.mysql.com/doc/refman/5.0/en/explain.html) You may need to consider consulting with a DBA on the issue.
Good luck.
I am developing a script in my localhst using PHP and mysql and I am dealing with large data (about 2 millions of records for scintific research)
some queries I need to call once in a life (to analyse the data and prepare some data); however it takes very long time for example: now my script is analysing some data for more than 4 hours
I knew I might have some problems in the optimization of my database I am not an expert
for example I just figured out that "indexing" can be useful to speed up the queries
however even with indexing some columns my script is still very slow
any idea how to speed up my script (in PHP and mysql)
I am using XAMPP as a server package
Thanks a lot for help
best regards
update 1:
part of my slow script which takes more than 4 hours to process
$sql = "select * from urls";//10,000 record of cached HTML documents
$result = $DB->query($sql);
while($row = $DB->fetch_array($result)){
$url_id = $row["id"];
$content = $row["content"];
$dom = new DOMDocument();
#$dom->loadHTML($content);
$xpath = new DOMXPath($dom);
$row = $xpath->evaluate("/html/body//a");
for($i = 0; $i < $row->length; $i++) {
// lots of the code here to deal with the HTML documents and some update and insert and select queries which query another table which has 1 million record
}
update 2:
I do not have "JOIN" in my quires or even "IN"
they are very simple queries
and don't know! and I don't know how to know which causes the slowness?
is it the PHP or the MYSQL?
First of all, to be able to optimize efficiently, you need to know what it taking time :
is PHP doing too much calculations ?
do you have too many SQL queries ?
do you have SQL queries that take too much time ?
if yes, which ones ?
where is your script spending time ?
With those informations, you can then try to figure out :
if you can diminish the number of SQL queries
for instance, if you are doing the exact same query over and over again, you are obviously wasting time
another idea is to "regroup" queries, if that is possible ; for instance, use only one query to get 10 lines, instead of 10 queries which all get one line back.
if you can optimize queries that take too long
either using indexes -- those which are usefull generally depend on the joins and conditions you are using
or re-writing the queries, if they are "bad"
About optimization of select statements, you can take a look at : 7.2. Optimizing SELECT and Other Statements
if PHP is doing too much calculations, can you have it make less calculations ?
Maybe not recalculating similar stuff again and again ?
Or using more efficient queries ?
if PHP is taking time, and the SQL server is not over-loaded, using parallelism (launching several calculations at the same time) might also help speed up the whole thing.
Still : this is quite a specific question, and the answers will be probably be pretty specific too -- which means more informations might be necessary if you want more than general answer...
Edit after your edits
As you only have simple queries, things might be a bit easier... Maybe.
First of all : you need to identify the kind of queries you are doing.
I'm guessing, of all all your queries, you can identify some "types" of queries.
for instance : "select * from a where x = 12" and "select * from a where x = 14" are of the same type : same select, same table, same where clause -- only the value changes
once you know which queries are used the most, you'll need to check if they are optimized : using EXPLAIN will help
(if needed, I'm sure some people will be able to help you understand its output, if you provider it alongside the schema of you DB (tables + indexes))
If needed : create the right indexes -- that's kind of the hard/specific part ^^
it is also for those queries that reducing the number of queries might prove useful...
when you're done with queries often used, it's time to go with queries that take too long ; using microtime from PHP will help you find out which ones those are
another solution is to use the 5.2.4. The Slow Query Log
when you have identified those queries, same as before : optimize.
Before that, to find out if PHP is working too much, or if it's MySQL, a simple way is to use the "top" command on Linux, or the "process manager" (I'm not on windows, and don't use it in english -- the real name might be something else).
If PHP is eating 100% of CPU, you have your culprit. If MySQL is eating all CPU, you have your culprit too.
When you know which one of those is working too much, it's a first step : you know what to optimize first.
I see from your portion of code that your are :
going through 10,000 elements one by one -- it should be easy to split those in 2 or more slices
using DOM and XPath, which might eat some CPU on the PHP-side
If you have a multi-core CPU, an idea (that I would try if I see that PHP is eating lots of CPU) would to to parallelize.
For instance, you could have two instances of the PHP script running at the same time :
one that will deal with the first half of the URLs
the SQL query for this one will be like "select * from urls where id < 5000"
and the other one that will deal with the second half of the URLs
its query will be like "select * from urls where id >= 5000"
You will get a bit more concurrency on the network (probably not a problem) and on the database (a database knows how to deal with concurrency, and 2 scripts using it will generally not be too much), but you'll be able to process almost twice the same amount of documents in the same time.
If you have 4 CPU, splitting the urls-list in 4 (or even more ; find out by trial and error) parts would do too.
Since your query is on one table and has no grouping or ordering, it is unlikely that the query is slow. I expect the issue is the size and number of the content fields. It appears that you are storing the entire HTML of a webpage in your database and then pulling it out every time you want to change a couple of values on the page. This is a situation to be avoided if at all possible.
Most scientific webapps (like BLAST for example) have the option to export the data as a delimited text file like a csv. If this is the case for you, you might consider restructuring your url table so that you have one column per data field in the csv. Then your update queries will be significantly faster as you will be able to do them entirely in SQL instead of pulling the entire url table into PHP, accessing and pulling one or more other records for each url record and then updating your table.
Assumably you have stored your data as webpages so you can dump the content easily to a browser. If you change your database schema as I've suggested, you'll need to write a webpage template that you can plug the data into when you wish to output it.
Knowing queries and table structures it would be easier.
If you cant give them out check if you have IN operator. MySQL tends to slow too much in there. Also try to run
EXPLAIN yourquery;
and see how it is executed. Sometimes sorting takes too much time. Try to avoid sorting on non-index columns.
inner joins are quicker than left or right joins
Has always sped up my queries going through after and thinking specifically about the joins.
have a look in your mysql config for settings you can turn off etc
If you are not using indexes it can be the main problem. There are many more optimization hints and tricks. Better will be to show i.e. your slowest query. It's not possible to help without any input data. Indexes and correct joins can speed this up really much.
If the queries will return same data you can store them in file or in memory and do them just once.
2 millions of records is not much.
Before you can optimise, you need to find out where the bottleneck is. Can you run the script on a smaller dataset, for testing purposes?
In that case, you should set such a test up, and then profile the code. You can either use a dedicated profiler such as Xdebug, or if you find it too daunting to configure (Not that complicated really, but you sound like you're a bit in the deep end already), you may feel more comfortable with a manual approach. This means starting a timer before parts of your code and stopping it after, then printing the result out. You can then narrow down which part is slowest.
Once you got that, we can give more specific answers, or perhaps it will be apparent to you what to do.
Is it possible to do a simple count(*) query in a PHP script while another PHP script is doing insert...select... query?
The situation is that I need to create a table with ~1M or more rows from another table, and while inserting, I do not want the user feel the page is freezing, so I am trying to keep update the counting, but by using a select count(\*) from table when background in inserting, I got only 0 until the insert is completed.
So is there any way to ask MySQL returns partial result first? Or is there a fast way to do a series of insert with data fetched from a previous select query while having about the same performance as insert...select... query?
The environment is php4.3 and MySQL4.1.
Without reducing performance? Not likely. With a little performance loss, maybe...
But why are you regularily creating tables and inserting millions of row? If you do this only very seldom, can't you just warn the admin (presumably the only one allowed to do such a thing) that this takes a long time. If you're doing this all the time, are you really sure you're not doing it wrong?
I agree with Stein's comment that this is a red flag if you're copying 1 million rows at a time during a PHP request.
I believe that in a majority of cases where people are trying to micro-optimize SQL, they could get much greater performance and throughput by approaching the problem in a different way. SQL shouldn't be your bottleneck.
If you're doing a single INSERT...SELECT, then no, you won't be able to get intermediate results. In fact this would be a Bad Thing, as users should never see a database in an intermediate state showing only a partial result of a statement or transaction. For more information, read up on ACID compliance.
That said, the MyISAM engine may play fast and loose with this. I'm pretty sure I've seen MyISAM commit some but not all of the rows from an INSERT...SELECT when I've aborted it part of the way through. You haven't said which engine your table is using, though.
The other users can't see the insertion until it's committed. That's normally a good thing, since it makes sure they can't see half-done data. However, if you want them to see intermediate data, you could throw in an occassional call to "commit" while you're inserting.
By the way - don't let anybody tell you to turn autocommit on. That a HUGE time waster. I have a "delete and re-insert" job on my database that takes 1/3rd as long when I turn off autocommit.
Just to be clear, MySQL 4 isn't configured by default to use transactions. It uses the MyISAM table type which locks the entire table for each insert, if I remember correctly.
Your best bet would be to use one of the MySQL bulk insertion functions, such as LOAD DATA INFILE, as these are dramatically faster at inserting large amounts of data. As for the counting, well, you could break the inserts into N groups of 1000 (or Y) then divide your progress meter into N sections and just update it on each group's request.
Edit: Another thing to consider is, if this is static data for a template, then you could use a "select into" to create a new table with the same data. Not sure what your application is, or the intended functionality, but that could work as well.
If you can get to the console, you can ask various status questions that will give you the information you are looking for. There's a command that goes something like "SHOW processlist".