SELECT * FROM articles t LEFT OUTER JOIN category_type category ON (t.category_id=category.id)
WHERE (t.status = 6 AND t.publish_on <= '2014-02-14' AND t.id NOT IN (13112,9490,9386,6045,1581,1034,991,933,879,758) AND t.category_id IN (14)) ORDER BY t.id DESC LIMIT 7;
It take more then 1.5 second to execute this query.
Can you give me some idea ? How can I improve this query and minimum execution time ?
First thing => use where instead of inner join. Because where is faster than inner join query.
Second thing => use indexes for the frequently searched columns. As in your example you search on the basis of status, publish_on besides id as primary index.
If you are using mysql then you can try propose table structure option in the phpmyadmin which can help you to decide valid data types for your column names. This could help you to optimize your query processing.
query processing time depends on many things like: database server load, amount of data in the table and the data types used for the column names too.
why join it with category table?, the category table is not in the where clause nor in the select column clause, so why add it in the query?
oops, * was used, so it "is" in the category table
apologies
Related
SELECT a.ts, b.barcodenumber, a.remarks, c.department
FROM documentlog a
INNER JOIN (select docid, max(logid) as logid from documentlog GROUP BY docid) d ON d.docid=a.docid AND d.logid=a.logid
INNER JOIN user c ON c.uid=a.user
INNER JOIN document b ON b.id=a.docid
WHERE c.department = 'PTO' AND b.end = 0
My problem is When I execute this query it's slow like 2sec+ execution but the data is only 9 , How can I speed up the execution of my query?
Old SS for EXPLAIN RESULT
UPDATED SS for EXPLAIN RESULT (Add INDEX logid,docid)
Check out your EXPLAIN result. Notice that MySQL does not use any kind of key when querying the documentlog table i.e., the documentlog table does not have a key defined on it. More than 2 million records are processed at this point in your query. This could be the most likely source of the slowness of your query.
Add an index on the docid, and logid fields in your documentlog table and check if it improves the queries' execution time.
Update!!
The output of the updated EXPLAIN query is saying that it is using a full table scan!! (i.e., type=ALL) to produce the output of the main outer query. Why? This is caused by the fact that there are no indices defined on the attributes used in the Where clause i.e., (department and end).
In general, if you want to speed up queries, then one has to make sure that appropriate indices are defined for the attributes used in the queries' WHERE condition.
By the way, you can learn more about the meaning of MySQL's EXPLAIN result by reading its documentation.
I have two tables tableOne = 90K data and tableTwo = 100k data, i will look for the duplicate numbers on both tables with the given conditions and the matching must be 1:1 if multiple match are on the other table only one will be tagged as match (given that the data on both tables has match data).
I have this select statement below, but when i run it on my local xampp and even on CMD the screen freezes after i press enter then it takes hours before it returns an error out of memory. Hope you can help me with this.
SELECT rNum,
cDate,
cTime,
aNumber,
bNumber,
duration,
tag,
aNumber2,
bNumber2,
'hasMatch',
concatDate,
timeMinutes
FROM tableOne a
LEFT JOIN
tableTwo b ON a.aNumber2 = b.aNumber2
AND a.bNumber2 = b.bNumber2
WHERE a.hasMatch = 'valid'
AND (a.duration - b.duration) <= 3
AND (a.duration - b.duration) >= -3
AND TIMEDIFF(a.concatDate,b.concatDate) <= 3
AND TIMEDIFF(a.concatDate,b.concatDate) >= -3
Thank you In advance.
If you're doing 1:1 relationship with two tables then I think you should probably go with INNER JOIN rather than LEFT JOIN
Secondly, your query doesn't seem to be indexed properly. So, better would be using EXPLAIN SELECT ... to see the profile of SQL and create INDEXES for Filters.
in your SELECT you have aNumber2 and based on your join rule both table a and table b have aNumber2 column. it's a problem. if two table have a column with the same name, on select you should specify the table.
for example like this
SELECT a.aNumber2 as a_number2,....
in your query the same problem exists for other columns like duration and concatDate
another thing is you should use INNER JOIN in your case instead of LEFT JOIN.
if you final result have many rows(thousands), take them step by step... add LIMIT to your example and take 100 result each time.
The current implementation is a single complex query with multiple joins and temporary tables, but is putting too much stress on my MySQL and is taking upwards of 30+ seconds to load the table. The data is retrieved by PHP via a JavaScript Ajax call and displayed on a webpage. Here is the tables involved:
Table: table_companies
Columns: company_id, ...
Table: table_manufacture_line
Columns: line_id, line_name, ...
Table: table_product_stereo
Columns: product_id, line_id, company_id, assembly_datetime, serial_number, ...
Table: table_product_television
Columns: product_id, line_id, company_id, assembly_datetime, serial_number, warranty_expiry, ...
A single company can have 100k+ items split between the two product tables. The product tables are unioned and filtered by the line_name, then ordered by assembly_datetime and limited depending on the paging. The datetime value is also reliant on timezone and this is applied as part of the query (another JOIN + temp table). line_name is also one of the returned columns.
I was thinking of splitting the line_name filter out from the product union query. Essentially I'd determine the ids of the lines that correspond to the filter, then do a UNION query with a WHERE condition WHERE line_id IN (<results from previous query>). This would cut out the need for joins and temp tables, and I can apply the line_name to line_id and timezone modification in PHP, but I'm not sure this is the best way to go about things.
I have also looked at potentially using Redis, but the large number of individual products is leading to a similarly long wait time when pushing all of the data to Redis via PHP (20-30 seconds), even if it is just pulled in directly from the product tables.
Is it possible to tweak the existing queries to increase the efficiency?
Can I push some of the handling to PHP to decrease the load on the SQL server? What about Redis?
Is there a way to architect the tables better?
What other solution(s) would you suggest?
I appreciate any input you can provide.
Edit:
Existing query:
SELECT line_name,CONVERT_TZ(datetime,'UTC',timezone) datetime,... FROM (SELECT line_name,datetime,... FROM ((SELECT line_id,assembly_datetime datetime,... FROM table_product_stereos WHERE company_id=# ) UNION (SELECT line_id,assembly_datetime datetime,... FROM table_product_televisions WHERE company_id=# )) AS union_products INNER JOIN table_manufacture_line USING (line_id)) AS products INNER JOIN (SELECT timezone FROM table_companies WHERE company_id=# ) AS tz ORDER BY datetime DESC LIMIT 0,100
Here it is formatted for some readability.
SELECT line_name,CONVERT_TZ(datetime,'UTC',tz.timezone) datetime,...
FROM (SELECT line_name,datetime,...
FROM (SELECT line_id,assembly_datetime datetime,...
FROM table_product_stereos WHERE company_id=#
UNION
SELECT line_id,assembly_datetime datetime,...
FROM table_product_televisions
WHERE company_id=#
) AS union_products
INNER JOIN table_manufacture_line USING (line_id)
) AS products
INNER JOIN (SELECT timezone
FROM table_companies
WHERE company_id=#
) AS tz
ORDER BY datetime DESC LIMIT 0,100
IDs are indexed; Primary keys are the first key for each column.
Let's build this query up from its component parts to see what we can optimize.
Observation: you're fetching the 100 most recent rows from the union of two large product tables.
So, let's start by trying to optimize the subqueries fetching stuff from the product tables. Here is one of them.
SELECT line_id,assembly_datetime datetime,...
FROM table_product_stereos
WHERE company_id=#
But look, you only need the 100 newest entries here. So, let's add
ORDER BY assembly_datetime DESC
LIMIT 100
to this query. Also, you should put a compound index on this table as follows. This will allow both the WHERE and ORDER BY lookups to be satisfied by the index.
CREATE INDEX id_date ON table_product_stereos (company_id, assembly_datetime)
All the same considerations apply to the query from table_product_televisions. Order it by the time, limit it to 100, and index it.
If you need to apply other selection criteria, you can put them in these inner queries. For example, in a comment you mentioned a selection based on a substring search. You could do this as follows
SELECT t.line_id,t.assembly_datetime datetime,...
FROM table_product_stereos AS t
JOIN table_manufacture_line AS m ON m.line_id = t.line_id
AND m.line_name LIKE '%test'
WHERE company_id=#
ORDER BY assembly_datetime DESC
LIMIT 100
Next, you are using UNION to combine those two query result sets into one. UNION has the function of eliminating duplicates, which is time-consuming. (You know you don't have duplicates, but MySQL doesn't.) Use UNION ALL instead.
Putting this all together, the innermost sub query becomes this. We have to wrap up the subqueries because SQL is confused by UNION and ORDER BY clauses at the same query level.
SELECT * FROM (
SELECT line_id,assembly_datetime datetime,...
FROM table_product_stereos
WHERE company_id=#
ORDER BY assembly_datetime DESC
LIMIT 100
) AS st
UNION ALL
SELECT * FROM (
SELECT line_id,assembly_datetime datetime,...
FROM table_product_televisions
WHERE company_id=#
ORDER BY assembly_datetime DESC
LIMIT 100
) AS tv
That gets you 200 rows. It should get those rows fairly quickly.
200 rows are guaranteed to be enough to give you the 100 most recent items later on after you do your outer ORDER BY ... LIMIT operation. But that operation only has to crunch 200 rows, not 100K+, so it will be far faster.
Finally wrap up this query in your outer query material. Join the table_manufacture_line information, and fix up the timezone.
If you do the indexing and the ORDER BY ... LIMIT operation earlier, this query should become very fast.
The comment dialog in your question indicates to me that you may have multiple product types, not just two, and that you have complex selection criteria for your paged display. Using UNION ALL on large numbers of rows slams performance: it converts multiple indexed tables into an internal list of rows that simply can't be searched efficiently.
You really should consider putting your two kinds of product data in a single table instead of having to UNION ALL multiple product tables. The setup you have now is inflexible and won't scale up easily. If you structure your schema with a master product table and perhaps some attribute tables for product-specific information, you will find yourself much happier two years from now. Seriously. Please consider making the change.
Remember: Index fast, data slow. Use joins over nested queries. Nested queries return all of the data fields whereas joins just consider the filters (which should all be indexed - make sure there's a unique index on table_product_*.line_id). It's been a while but I'm pretty sure you can join "ON company_id=#" which should cut down the results early on.
In this case, all of the results refer to the same company (or a much smaller subset) so it makes sense to run that query separately (and it makes the query more maintainable).
So your data source would be:
(table_product_stereos as prod
INNER JOIN table_manufacture_line AS ml ON prod.line_id = ml.line_id and prod.company_id=#
UNION
table_product_televisions as prod
INNER JOIN table_manufacture_line as ml on prod.line_id = ml.line_id and prod.company_id=#)
From which you can select prod. or ml. fields as required.
PHP is not a solution at all...
Redis can be a solution.
But the main thing I would change is the index creation for the tables (add missing indexe)...If you're running into temp tables you didn't create indexes well for the tables. And 100k rows in not much at all.
But I cant help you without any table creation statements as well as queries you run.
Make sure your "where part" is part of youf btree index from left to right.
I have two tables: "users" and "posts." The posts table has a 'post' column and a 'poster_id' column. I'm working on a PHP page that shows the latest posts by everyone, like this:
SELECT * FROM posts WHERE id < '$whatever' LIMIT 10
This way, I can print each result like this:
id: 43, poster_id:'4', post: hello, world
id: 44, poster_id:'4', post: hello, ward
id: 45, poster_id:'5', post: oh hi!
etc...
Instead of the id, I would like to display the NAME of the poster (there's a column for it in the 'users' table)
I've tried the following:
SELECT *
FROM posts
WHERE id < '$whatever'
INNER JOIN users
ON posts.poster_id = users.id LIMIT 10
Is this the correct type of join for this task? Before learning about joins, I would query the users table for each post result. The result should end up looking similar to this:
id: 43, poster_id:'4', name:'foo', post: hello, world
id: 44, poster_id:'4', name:'foo', post: hello, ward
id: 45, poster_id:'5', name:'fee', post: oh hi!
etc...
Thanks for helping in advance.
WHERE clause must come after the FROM clause.
SELECT posts.*, users.* // select your desired columns
FROM posts
INNER JOIN users ON posts.poster_id = users.id
WHERE id < '$whatever'
LIMIT 10
the SQL Order of Operation is as follows:
FROM clause
WHERE clause
GROUP BY clause
HAVING clause
SELECT clause
ORDER BY clause
UPDATE 1
For those column names that exists on both tables, add an ALIAS on them so it can be uniquely identified. example,
SELECT post.colName as PostCol,
users.colName as UserCol, ....
FROM ....
on the example above, both tables has column name colName. In order to get them both, you need to add alias on them so in your front end, use PostCol and UserCol to get their values.
Try:
SELECT *
FROM posts
INNER JOIN users ON posts.poster_id = users.id
WHERE posts.id < '$whatever'
LIMIT 10
Got the syntax a little incorrect.
Should be
SELECT * FROM posts
INNER JOIN users ON posts.poster_id = users.id
WHERE id < '$whatever' LIMIT 10
The answers already given tell you the main reason for your query not working at all (ie the WHERE clause should come after the JOIN clauses), however, I'd like to make a couple of additional points:
I would suggest using an OUTER JOIN for this. It probably won't make much difference, but in the event of a post record having an invalid poster_id, an INNER JOIN will mean the record is dropped from the results, whereas an OUTER JOIN will mean that the record is included, but the values from the users table will be null. I imagine you don't want to ever have an invalid poster_id on the posts table, but broken data does happen even in the best regulated system, and it is helpful in these cases to still get the data from the query.
I would strongly suggest not doing SELECT *, and instead itemising the fields you want to get back from the query. SELECT * has a number of problems, but it's particularly bad when you have multiple tables in the query, because if you have fields with the same name on both tables, (eg id), then it becomes very hard to distinguish which one you're working with, as your PHP recordset won't include the table reference. Itemising the fields may make your query string longer, but it won't make it any slower - if anything it'll be quicker - and it will be easier to work with in the long run.
Neither of these points are essential; the query will work without them (as long as you switch the WHERE clause to after the JOIN), but they may improve your query and hopefully also improve your understanding of SQL.
I am trying to query 2 tables in a database, each query having nothing to do with each other, other then being on the same page.
Query 1 - The first query on the page will retrieve text and images that are found throughout the page from Table A.
Query 2 - The second query will retrieve several products with a image, description and title for each product from Table B.
I know that putting the second query inside the first query's while loop would work but of course is very inefficient.
How can I and what is the best way to retrieve all the data I need through 1 query?
Thanks,
Dane
So all you want to know is if its ok to have 2 queries on the same webpage? Its A-OK. Go right ahead. Its completelly normal. No one expects a join between table news and table products. Its normal to usetwo queries to fetch data from two unrelated tables.
Use LEFT or INNER JOIN (depends on whether you want to display records from TableA that have no correspondent records in TableB)
SELECT a.*, b.*
FROM TableA a
[LEFT or INNER] JOIN TableB b ON (b.a_id = a.id)
If there's no way to relate the two tables to each other, then you can't use a JOIN to grab records from both. You COULD use a UNION query, but that presumes that you can match up fields from each table, as a UNION requires you to select the same number/type of fields from each table.
SELECT 'pageinfo' AS sourcetable, page.id, page.images, page.this, page.that
WHERE page.id = $id
UNION
SELECT 'product' AS sourcetable, products.id, products.image, product.other, product.stuff
But this is highly ugly. You're still forcing the DB server to do two queries in the background plus the extra work of combining them into a single result set, and then you have to do extra work to dis-entangle in your code to boot.
It's MUCH easier, conceptually and maintenance-wise, to do two seperate queries instead.