I have to run this Mysql query on my website to fetch huge amount of data: (3 tables , each with 100,000 + records)
SELECT on_resume.*, on_users.subscribed, on_users.user_avatar, on_resume_page.*
FROM on_resume
LEFT JOIN on_users ON (on_resume.resume_userid = on_users.user_id )
LEFT JOIN on_resume_page ON ( on_resume.resume_userid = on_resume_page.resume_userid)
WHERE on_resume.active= '1'
GROUP BY on_resume.rid
ORDER BY on_resume.rid DESC
LIMIT 0,18
The time I run this at Phpmyadmin sql section, the whole mysqld service will be down and needs to be restarted.
Now I was testing this query and I found out if I don't use Group by and Order by conditions the query will be fine.
SELECT on_resume.*, on_users.subscribed, on_users.user_avatar, on_resume_page.*
FROM on_resume
LEFT JOIN on_users ON (on_resume.resume_userid = on_users.user_id )
LEFT JOIN on_resume_page ON ( on_resume.resume_userid = on_resume_page.resume_userid)
WHERE on_resume.active= '1'
LIMIT 0,18
Showing rows 0 - 17 ( 18 total, Query took 0.4248 sec)
Why is it like this and how can I fix it?...
NOTE : I have tested the SQL query with group by or Order by alone in either case , even with one of them still the query fails and hangs the server.
EDIT : This problem is solved by making column on_resume_page.resume_userid indexed.
This is what i was told, took a while to figure it out:
Look at #jer in Chicago comment
Remember, when there is a GROUP BY clause, there are certain rules that apply for grouping columns. One of those rules is "The Single-Value Rule" -- every column named in the SELECT list must also be a grouping column unless it is an argument for one of the set functions. MySQL extends standard SQL by allowing you to use columns or calculations in a SELECT list that don't appear in a GROUP BY clause. However, we are warned not to use this feature unless the columns you omit from the GROUP BY clause are not unique in the group because you will get unpredictable results.
Related
I want to perform a query like to get the last record of any type in DB, at my localhost, I use Maria Db and the query is as follow:
SELECT *
FROM table_a
WHERE column_a=999
OR column_b=999
GROUP
BY `group`;
group is a column which I save type in it, for instance: marketing, blog, order, etc
This query works fine on local, but on the server I get the following error:
SQLSTATE[42000]:
Syntax error or access violation:
1055 Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column
'db_name.table_a.id' which is not functionally dependent on columns in GROUP BY clause;
this is incompatible with sql_mode=only_full_group_by\n
The SQL being executed was:
SELECT * FROM `table_a` WHERE (`column_a`=999) OR (`column_b`=999) GROUP BY `group`"
According to MySQL document I can use the following command to make this possible:
SET GLOBAL sql_mode=(SELECT REPLACE(##sql_mode,'ONLY_FULL_GROUP_BY',''));
But I don't have the sufficient privilege on Db and get the following error:
#1227 - Access denied; you need (at least one of) the SUPER privilege(s) for this operation
I asked the hosting to do this for me, they replied that they don't want to do this action
I use the YII2 framework, and now I want a way to add this on the option of database_config.php of the framework or change the query to something else with the same result, and not losing performance
ONLY_FULL_GROUP_BY is a good thing, which enforces basic ANSI SQL rules. Don't change it, fix your code instead.
From there one: you want entire records, so you should not think aggregation, but filtering.
Then: in a database table, records are unordered; for your question to just make sense, you need a column that defines the ordering of rows in each group, so it unambiguous what "last" record mean. Let me assume that you have such column, called id.
Here is a typical approach at this top-1-per-group problem, using a correlated subquery for filtering:
SELECT *
FROM table_a a
WHERE
999 IN (column_a, column_b)
AND id = (
SELECT MAX(a1.id)
FROM table_a a1
WHERE 999 IN (a1.column_a, a1.column_b) AND a1.grp = a.grp
)
Alternatively, if you are running MySQL 8.0, you can use window functions:
SELECT *
FROM (
SELECT a.*,
ROW_NUMBER() OVER(PARTITION BY grp ORDER BY id DESC) rn
FROM table_a a
WHERE 999 IN (column_a, column_b)
) a
WHERE rn = 1
Side note: group is a language keyword, hence a poor choice for a column name. I renamed it to grp in the queries.
There are a few ways to "bypass" the sql_mode but be aware that the result you get might not be correct.
First you can use ANY_VALUE(). Example like this:
SELECT any_value(column_a), any_value(column_b), `group` FROM table_a
WHERE (column_a=999) OR (column_b=999) GROUP BY `group`;
When using ANY_VALUE() function you have to write all the columns in SELECT from the table and append with ANY_VALUE() except for the column that you use in the GROUP BY.
Using MAX() or MIN() can return result but still it might not be the correct
result, especially for any row(s) that have more than 1 count:
SELECT MAX(column_a), MAX(column_b), `group`
FROM table_a
WHERE (column_a=999) OR (column_b=999) GROUP BY `group`;
Using GROUP_CONCAT will give you a view at what are the values in non-grouped columns. Compare the results with the other queries above and you can see on row(s) that returns more than one count, does the other queries returning according to what you want?
SELECT group_concat(column_a), group_concat(column_b), group_concat(`group`)
FROM table_a
WHERE (column_a=999) OR (column_b=999) GROUP BY `group`;
I'm not sure if you can do this but you can set the sql_mode off temporarily then you should be able to run your query:
SET sql_mode=""; -- you don't need to set global privilege.
SELECT * FROM table_a
WHERE (column_a=999) OR (column_b=999) GROUP BY `group`;
Demo here.
Still, the best option is to retain the sql_mode as it is and construct the query according to the requirement.
P/S: GROUP is a reserved word in both MySQL & MariaDB. You can use it as column name but you have to always add back-ticks to define the column or else, running the query will return you an error like
Query: select * from table_a group by group
Error Code: 1064
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'group' at line 1
I've been working with Mysql for a while, but this is the first time I've encountered this problem.
The thing is that I have a select query...
SELECT
transactions.inventoryid,
inventoryName,
inventoryBarcode,
inventoryControlNumber,
users.nombre,
users.apellido,
transactionid,
transactionNumber,
originalQTY,
updateQTY,
finalQTY,
transactionDate,
transactionState,
transactions.observaciones
FROM
transactions
LEFT JOIN
inventory ON inventory.inventoryid = transactions.inventoryid
LEFT JOIN
users ON transactions.userid = users.userid
GROUP BY
transactions.transactionNumber
ORDER BY
transactions.inventoryid
But the GROUP BY is eliminating 2 values from the QUERY.
In this case, when I output:
foreach($inventory->inventory as $values){
$transactionid[] = $values['inventoryid'];
}
It returns:
2,3,5
If I eliminate the GROUP BY Statement it returns
2,3,4,5,6
Which is the output I need for this particular case.
The question is:
Is there a reason for this to happen?
If I'm grouping by a transaction and that was supposed to affect the query, wouldn't it then return only 1 value?
Maybe I'm over thinking this one or been working too long on the code that I don't see the obvious flaw in my logic. But if someone can lend me a hand I would appreciate it.
In standard SQL you can only SELECT colums which are
contained in GROUP BY clause
(or) aggregate "colums", like MAX() or COUNT().
You need to consult the MySQL description of the interpretation they use for columns which are not contained by GROUP BY (and which are no aggregated column) MySQL Handling of GROUP BY to find out what happens here.
Do you need more information?
I have a database that is already in use and I have to improve the performance of the system that's using this database.
There are 2 major queries running about 1000 times in a loop and this queries have inner joins to 3 other tables each. This in turn is making the system very slow.
I tried actually to remove the query from the loop and fetch all the data only once and process it in PHP. But this is putting to much load on the memory (RAM) and the system is hanging if 2 or more clients try to use the system.
There is a lot of data in the tables even after removing the expired data .
I have attached the query below.
Can anyone help me with this issue ?
select * from inventory
where (region_id = 38 or region_id = -1)
and (tour_opp_id = 410 or tour_opp_id = -1)
and room_plan_id = 141 and meal_plan_id = 1 and bed_type_id = 1 and hotel_id = 1059
and FIND_IN_SET(supplier_code, 'QOA,QTE,QM,TEST,TEST1,MQE1,MQE3,PERR,QKT')
and ( ('2014-11-14' between from_date and to_date) )
order by hotel_id desc ,supplier_code desc, region_id desc,tour_opp_id desc,inventory.inventory_id desc
SELECT * ,pinfo.fri as pi_day_fri,pinfoadd.fri as pa_day_fri,pinfochld.fri as pc_day_fri
FROM `profit_markup`
inner join profit_markup_info as pinfo on pinfo.profit_id = profit_markup.profit_markup_id
inner join profit_markup_add_info as pinfoadd on pinfoadd.profit_id = profit_markup.profit_markup_id
inner join profit_markup_child_info as pinfochld on pinfochld.profit_id = profit_markup.profit_markup_id
where profit_markup.hotel_id = 1059 and (`booking_channel` = 1 or `booking_channel` = 2)
and (`rate_region` = -1 or `rate_region` = 128)
and ( ( period_from <= '2014-11-14' and period_to >= '2014-11-14' ) )
ORDER BY profit_markup.hotel_id DESC,supplier_code desc, rate_region desc,operators_list desc, profit_markup_id DESC
Since we have not seen your SHOW CREATE TABLES; and EXPLAIN EXTENDED plan it is hard to give you 1 answer
But generally speaking in regard to your query "BTW I re-wrote below"
SELECT
hotel_id, supplier_code, region_id, tour_opp_id, inventory_id
FROM
inventory
WHERE
region_id IN (38, -1)
AND tour_opp_id IN (410, -1)
AND room_plan_id IN (141, 1)
AND bed_type_id IN (1, 1059)
AND supplier_code IN ('QOA', 'QTE', 'QM', 'TEST', 'TEST1', 'MQE1', 'MQE3', 'PERR', 'QKT')
AND ('2014-11-14' BETWEEN from_date AND to_date )
ORDER BY
hotel_id DESC, supplier_code DESC, region_id DESC, tour_opp_id DESC, inventory_id DESC
Do not use * to get all the columns. You should list the column that you really need. Using * is just a lazy way of writing a query. limiting the columns will limit the data size that is being selected.
How often is the records in the inventory are being updates/inserted/delete? If not too often then you can use consider using SQL_CACHE. However, caching a query will cause you problems if you use it and the inventory table is updated very often. In addition, to use query cache you must check the value of query_cache_type on your server. SHOW GLOBAL VARIABLES LIKE 'query_cache_type';. If this is set to "0" then the cache feature is disabled and SQL_CACHE will be ignored. If it is set to 1 then the server will cache all queries unless you tell it not too using NO_SQL_CACHE. If the option is set to 2 then MySQL will cache the query only where SQL_CACHE clause is used. here is documentation about query_cache_type
If you have an index on those following column in this order it will help you (hotel_id, supplier_code, region_id, tour_opp_id, inventory_id)
ALTER TABLE inventory
ADD INDEX (hotel_id, supplier_code, region_id, tour_opp_id, inventory_id);
If possible increase sort_buffer_size on your server as most likely you issue here is that your are doing too much sorting.
As for the second query "BTW I re-wrote below"
SELECT
*, pinfo.fri as pi_day_fri,
pinfoadd.fri as pa_day_fri,
pinfochld.fri as pc_day_fri
FROM
profit_markup
INNER JOIN
profit_markup_info AS pinfo ON pinfo.profit_id = profit_markup.profit_markup_id
INNER JOIN
profit_markup_add_info AS pinfoadd ON pinfoadd.profit_id = profit_markup.profit_markup_id
INNER JOIN
profit_markup_child_info AS pinfochld ON pinfochld.profit_id = profit_markup.profit_markup_id
WHERE
profit_markup.hotel_id = 1059
AND booking_channel IN (1, 2)
AND rate_region IN (-1, 128)
AND period_from <= '2014-11-14'
AND period_to >= '2014-11-14'
ORDER BY
profit_markup.hotel_id DESC, supplier_code DESC, rate_region DESC,
operators_list DESC, profit_markup_id DESC
Again eliminate the use of * from your query
Make sure that the following columns have the same type/collation and same size. pinfo.profit_id, profit_markup.profit_markup_id, pinfoadd.profit_id, pinfochld.profit_id and each one have to have an index on every table. If the columns have different types then MySQL will have to convert the data every time to join the records. Even if you have index it will be slower. Also, if those column are characters type (ie. VARCHAR()) make sure they are of the CHAR() with a collation of latin1_general_ci as this will be faster for finding ID, but if you are using INT() even better.
Use the 3rd and 4th trick I listed for the previous query
Try using STRAIGHT_JOIN "you must know what your doing here or it will bite you!" Here is a good thread about this When to use STRAIGHT_JOIN with MySQL
I hope this helps.
For the first query, I am not sure if you can do much (assuming you have already indexed the fields you are ordering by) apart from replacing the * with column names (Don't expect this to increase the performance drastically).
For the second query, before you go through the loop and put in selection arguments, you could create a view with all the tables joined and ordered then make a prepared statement to select from the view and bind arguments in the loop.
Also, if your php server and the database server are in two different places, it is better if you did the selection through a stored procedure in the database.
(If nothing works out, then memcache is the way to go... Although I have personally never done this)
Here you have increase query performance not an database performance.
For both queries first check index is available on WHERE and ON(Join) clause columns, if index is missing then you have to add index to improve query performance.
Check explain plane before create index.
If possible show me the explain plane of both query that will help us.
It's been a while since I needed help, but today I'm here to basically get assistance from your knowledge. I'm currently quite stuck on a very annoying SQL problem, which is the following.
I have two tables. Painteditems, and specialitems. Both tables have unique column names (painteditemid, specialitemid etc), yet both tables share similar values. I want to get results from both tables.
Let's say this is my setup:
PaintedItems
paintedItemName
paintedItemColor
visible
SpecialItems
specialItemName
specialItemColor
visible
I used this query:
SELECT *
FROM `painteditems` AS pa,
`specialitems` AS sp
WHERE (pa.`visible` = 1
OR sp.`visible` = 1)
AND (pa.`painteditemname` = 'itemname1'
OR sp.`specialitemname` = 'itemname1')
AND (pa.`painteditemcolor` = 'black'
OR sp.`specialitemcolor` = 'black')
That resulted in:
Showing rows 0 - 29 ( 259,040 total, Query took 39.4352 sec)
even though both tables contain only 10.000 rows altogether. Adding this did nothing:
GROUP BY pa.`painteditemid`, sp.`specialitemid`
Still 260k rows. How should I approach this?
Thank you in advance.
edit: fixed spacing, code blocks
Sure sounds like you want a UNION between the two tables. Right now, you are getting a cartesian product which is why the results are so large:
select *, 'painted' Source
from painteditems
where visible = 1
and painteditemname = 'itemname1'
and painteditemcolor = 'black'
union all
select *, 'special' Source
from specialitems
where visible = 1
and specialitemname = 'itemname1'
and specialitemcolor = 'black'
You will need to replace the SELECT * with your column names. Also the number of columns and datatypes must match in both queries.
UNION ALL will return all rows from both tables, if you only want DISTINCT rows then you will want to use UNION
The UNION operator is used to combine the result-set of two or more SELECT statements. Defiantly You can make use of UNION as shown in the #bluefeet's answer If you meet below conditions.
SELECT statement within the UNION must have the same number of
columns
The columns must also have similar data type
The columns in each SELECT statement must be in the same order.
I would do this with a union all in the subquery:
select *
from ((select paintedItemName as ItemName, paintedItemColor as ItemColor, visible, 'Painted' as which
from painteditems
) union all
(select specialItemName, SpecialItemColor, visible, 'Special' as which
from specialitems
)
) t
where visible = 1 and itemname = 'itemname1' and itemcolor = 'black'
This allows you to have only one set of results. In a union, the column names come from the first subquery, which this renames to more generic names. The reason I prefer this approach is because the where clause does not need to be repeated multiple times -- which can lead to errors and maintenance problems.
I have the following query:
SELECT vBrowser,iconBrowser, count(iconBrowser) as 'N'
FROM user_ip_tmp WHERE code='9m9g9tsv2y'
GROUP BY iconBrowser
ORDER BY N DESC
LIMIT 40
And this works properly. But the delirious cause query took a long time.
Showing rows 0 - 17 ( 18 total, Query took 4.4189 sec)
Things that are in WHERE statement, should be indexed.
Try to use EXPLAIN statement before your SELECT to see what and how is used to retrief your requested results.
And if the column code is not an unique value, i would recommend to put it in some other table, where it is unique. Then build the query using JOIN though the FOREIGN KEY.