Only in a sql query (used in a php script), I need to use the value of a column into the query to optimize the speed but it doesn't work well.
I have mutiple tables so I use JOIN
If I do that :
LEFT JOIN `gestdetails` gd ON gd.id IN (205467,205468,205469)
I works well !
But if I use the value of a column (gc.details = 205467,205468,205469) :
LEFT JOIN `gestdetails` gd ON gd.id IN (gc.details)
It doesn't work, it only use the first value '205467'.
I'm trying to do that to go faster with the query : if I do the classic JOIN on id = id, it also works but it's way too long...
LEFT JOIN `gestdetails` gd ON (gd.idcommande = gc.id)
gestcommandes = table of unique orders
gestdetails = table of multiple products linked to idcommande of gestcommandes
Here is the full query :
SELECT
gc.*,
count(gd.idcommande),
GROUP_CONCAT(CONCAT(gd.id,"#",gd.idcommande,"#",gd.ref,"#",gd.poids,"#",gd.numcolis,"#",gd.designation,"#",gd.prix) SEPARATOR "|"),
ad.*
FROM
`gestcommandes` gc
LEFT JOIN `gestdetails` gd ON gd.id IN (gc.details)
LEFT JOIN `adresses` ad ON ad.id_commande = gc.id
WHERE
`mp` LIKE '%-eb' AND (`statut` = 5 OR `statut` = 6) AND `date_expe` >= '2020-04-07 00:00:00' AND `date_expe` <= '2020-04-07 23:59:59'
AND
ad.is_delivery = 1
GROUP BY
gc.id
Executing this query, it only uses the first number into the gc.details column (i.e '205467' from '205467,205468,205469' and of course, I've got warnings :
Warning: #1292 Truncated incorrect INTEGER value: '205467,205468,205469'
gc.details can count 1 to 100 numbers...
I tryed so many things, I don't manage to do it, so if anybody can help me (without creating procedure)... thank you !
ps : the "GROUP_CONCAT" part works well... if I manage to get more than 1 result for the details...
I have almost thousands of data to display for my reports and it makes my browser lags due to the heavy data. I think that my query is the real problem. How can I optimized my query? is there something that I should add in my query?
I am using Xampp which supports PHP7.
SELECT
`payroll_billed_units`.`allotment_code`,
`payroll_billed_units`.`category_name`,
`payroll_billed_units`.`ntp_number`,
`payroll_billed_units`.`activity`,
`payroll_billed_units`.`regular_labor`,
`payroll_sub`.`block_number`,
(SELECT
GROUP_CONCAT(DISTINCT `lot_number` SEPARATOR ', ')
FROM
`payroll_billed_units` `lot_numbers`
WHERE
`lot_numbers`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `lot_numbers`.`category_name` = `payroll_billed_units`.`category_name`
AND `lot_numbers`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `lot_numbers`.`activity` = `payroll_billed_units`.`activity`) AS `lot_numbers`,
(SELECT
COUNT(`billed`.`ntp_id`)
FROM
`regular_ntp` `billed`
WHERE
`billed`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `billed`.`category_name` = `payroll_billed_units`.`category_name`
AND `billed`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `billed`.`activity` = `payroll_billed_units`.`activity`) AS `billed`,
(SELECT
COUNT(`approved`.`id`)
FROM
`payroll_billed_units` `approved`
WHERE
`approved`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `approved`.`category_name` = `payroll_billed_units`.`category_name`
AND `approved`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `approved`.`activity` = `payroll_billed_units`.`activity`) AS `approved`
FROM
`payroll_billed_units`
JOIN payroll_transaction ON payroll_billed_units.billing_number =
payroll_transaction.billing_number
JOIN payroll_sub ON payroll_transaction.billing_number =
payroll_sub.billing_number
WHERE payroll_billed_units.billing_date = '2019-02-13'
AND payroll_transaction.contractor_name = 'Roy Codal' GROUP BY allotment_code, category_name, activity
I was expecting that it will load or display all my data.
The biggest problem are the dependendt sub-selects, they are responsible for a bad performance. A sub-select will be executed for EVERY ROW of the outer query. And if you cascade subs-selects, you'll quickly have a query run forever.
If any of the parts would yield only 5 resultsets, 3 sub-select would mean that the database has to run 625 queries (5^4)!
Use JOINs.
Several of your tables need this 'composite' index:
INDEX(allotment_code, category_name, ntp_number, activity) -- in any order
payroll_transaction needs INDEX(contractor_name), though it may not get used.
payroll_billed_units needs INDEX(billing_date), though it may not get used.
For further discussion, please provide SHOW CREATE TABLE for each table and EXPLAIN SELECT ...
Use simply COUNT(*) instead of COUNT(foo). The latter checks the column for being not-NULL before including it. This is usually not needed. The reader is confused by thinking that there might be NULLs.
Your GROUP BY is improper because it is missing ntp_number. Read about the sql_mode of ONLY_FULL_GROUP_BY. I bring this up because you can almost get rid of some of those subqueries.
Another issue... Because of the "inflate-deflate" nature of JOIN with GROUP BY, the numbers may be inflated. I recommend you manually check the values of the COUNTs.
I'm making alot of unnecessary querys into my database what causes slow time to first Byte. I'm using PHP PDO Class Wrapper Version 1.2 (Beta).
$row = $this->baseClass->db->pdoQuery('SELECT SUM(user_have_group.user_group = 2 ) AS boosters , count(user.id ) AS created_accounts FROM `user` LEFT JOIN `user_have_group` ON user_have_group.user_id = user.id ')->result();
I manage to get counts from related tables using LEFT JOIN, but I need one more result from not related table, and I have no idea how to do it. Any advice?
I managed to make it in the following way
$row = $this->baseClass->db->pdoQuery('SELECT a.boosters,a.created_accounts,b.happy_clients from (SELECT SUM(user_have_group.user_group = 2 ) AS boosters , count(user.id ) AS created_accounts FROM user LEFT JOIN user_have_group ON user_have_group.user_id = user.id ) AS a,(SELECT SUM(boost_order.status="complete") AS happy_clients FROM boost_order) AS b')->result();
Thanks chris85 for help
Me and our team are experiencing a strange problem with MySQL queries. We use a SELECT statement with a COUNT and for some reason it's quite 'fast' in the client we use (SQLyog), but it's realy slow when we use PHP.
We have tried to use the ancient mysql_query(), the mysqli extension and we also tried to use PDO but all didn't make a difference.
On other posts here on Stackoverflow we found that it might be a DNS issue and that it could be fixed using 'skip_name_resolve' in the my.ini but we already had this in our configuration.
Timed results:
Client: 2.092 sec
PHP: 9.1071 sec
This is the query we use:
SELECT SQL_NO_CACHE
COUNT(m.mm_id) AS total
FROM
db.media_multimedia m
WHERE m.cat_id IN
(SELECT
mc.cat_id
FROM
db.media_multimedia_category mc
WHERE mc.cat_active = 1)
AND m.mm_published = 1
AND (
m.mm_title LIKE "%denniy%"
OR m.mm_text LIKE "%denniy%"
OR m.mm_id IN
(SELECT
a.mm_id
FROM
db.`media_tag_multimedia` a
LEFT JOIN media.`media_tag` b
ON a.`tag_id` = b.tag_id
WHERE b.tag_name LIKE "%denniy%")
)
AND m.mm_publishing_date >= "2012-04-24 00:00:00"
AND m.mm_publishing_date <= "2013-04-24 23:59:59" ;
*NOTE: for this testcase we added SQL_NO_CACHE to the query to make sure we always fetch a new result set.*
We are using the following PHP and MYSQL versions:
MySQL: 5.1.61
PHP: 5.3.3
Any suggestions to fix this problem?
I can't explain the difference in performance, although if I had to guess I'd say one or some of the following things are in play:
The server is optimising your query differently because of the different client connections,
The different location of your SQLyog and PHP clients might be a factor
Like I said, just guesses.
But regardless, I've attempted to tidy up your query as follows. I wonder if this might perform better (and more consistently)?
SELECT SQL_NO_CACHE
COUNT(m.mm_id) AS total
FROM
db.media_multimedia m
INNER JOIN db.media_multimedia_category mc
ON m.cat_id = mc.cat_id
AND mc.cat_active = 1
LEFT JOIN db.media_tag_multimedia a
ON m.mm_id = a.mm_id
INNER JOIN media.media_tag b
ON a.tag_id = b.tag_id
WHERE
m.mm_published = 1
AND
(
m.mm_title LIKE "%denniy%"
OR
m.mm_text LIKE "%denniy%"
OR
b.tag_name LIKE "%denniy%"
)
AND m.mm_publishing_date >= "2012-04-24 00:00:00"
AND m.mm_publishing_date <= "2013-04-24 23:59:59" ;
I have a situation where lets say i'm trying to get the information about some food. Then I need to display all the information plus all the ingredients in that food.
With my query, i'm getting all the information in an array but only the first ingredient...
myFoodsArr =
[0]
foodDescription = "the description text will be here"
ratingAverage = 0
foodId = 4
ingredient = 1
ingAmount = 2
foodName = "Awesome Food name"
typeOfFood = 6
votes = 0
I would like to get something back like this...
myFoodsArr =
[0]
foodDescription = "the description text will be here"
ratingAverage = 0
foodId = 4
ingArr = {ingredient: 1, ingAmount: 4}, {ingredient: 3, ingAmount: 2}, {ingredient: 5, ingAmount: 1}
foodName = "Awesome Food name"
typeOfFood = 6
votes = 0
This is the query im working with right now. How can I adjust this to return the food ID 4 and then also get ALL the ingredients for that food? All while at the same time doing other things like getting the average rating of that food?
Thanks!
SELECT a.foodId, a.foodName, a.foodDescription, a.typeOfFood, c.ingredient, c.ingAmount, AVG(b.foodRating) AS ratingAverage, COUNT(b.foodId) as tvotes
FROM `foods` a
LEFT JOIN `foods_ratings` b
ON a.foodId = b.foodId
LEFT JOIN `foods_ing` c
ON a.foodId=c.foodId
WHERE a.foodId=4
EDIT:
Catcall introduced this concept of "sub queries" I never heard of, so I'm trying to make that work to see if i can do this in 1 query easily. But i just keep getting a return false. This is what I was trying with no luck..
//I changed some of the column names to help them be more distinct in this example
SELECT a.foodId, a.foodName, a.foodDescription, a.typeOfFood, AVG(b.foodRating) AS ratingAverage, COUNT(b.foodId) as tvotes
FROM foods a
LEFT JOIN foods_ratings b ON a.foodId = b.foodId
LEFT JOIN (SELECT fId, ingredientId, ingAmount
FROM foods_ing
WHERE fId = 4
GROUP BY fId) c ON a.foodId = c.fId
WHERE a.foodId = 4";
EDIT 1 more thing related to ROLANDS GROUP_CONCAT/JSON Idea as a solution 4 this
I'm trying to make sure the JSON string im sending back to my Flash project is ready to be properly parsed Invalid JSON parse input. keeps popping up..
so im thinking i need to properly have all the double quotes in the right places.
But in my MySQL query string, im trying to escape the double quotes, but then it makes my mySQL vars not work, for example...
If i do this..
GROUP_CONCAT('{\"ingredient\":', \"c.ingredient\", ',\"ingAmount\":', \"c.ingAmount\", '}')`
I get this...
{"ingredient":c.ingredient,"ingAmount":c.ingAmount},{"ingredient":c.ingredient,"ingAmount":c.ingAmount},{"ingredient":c.ingredient,"ingAmount":c.ingAmount}
How can i use all the double quotes to make the JSON properly formed without breaking the mysql?
This should do the trick:
SELECT food_ingredients.foodId
, food_ingredients.foodName
, food_ingredients.foodDescription
, food_ingredients.typeOfFood
, food_ingredients.ingredients
, AVG(food_ratings.food_rating) food_rating
, COUNT(food_ratings.foodId) number_of_votes
FROM (
SELECT a.foodId
, a.foodName
, a.foodDescription
, a.typeOfFood
, GROUP_CONCAT(
'{ingredient:', c.ingredient,
, ',ingAmount:', c.ingAmount, '}'
) ingredients
FROM foods a
LEFT JOIN foods_ing c
ON a.foodsId = c.foodsId
WHERE a.foodsId=4
GROUP BY a.foodId
) food_ingredients
LEFT JOIN food_ratings
ON food_ingredients.foodId = food_ratings.foodId
GROUP BY food_ingredients.foodId
Note that the type of query you want to do is not trivial in any SQL-based database.
The main problem is that you have one master (food) with two details (ingredients and ratings). Because those details are not related to each other (other than to the master) they form a cartesian product with each other (bound only by their relationship to the master).
The query above solves that by doing it in 2 steps: first, join to the first detail (ingredients) and aggregate the detail (using group_concat to make one single row of all related ingredient rows), then join that result to the second detail (ratings) and aggregate again.
In the example above, the ingredients are returned in a structured string, exactly like it appeared in your example. If you want to access the data inside PHP, you might consider adding a bit more syntax to make it a valid JSON string so you can decode it into an array using the php function json_decode(): http://www.php.net/manual/en/function.json-decode.php
To do that, simply change the line to:
CONCAT(
'['
, GROUP_CONCAT(
'{"ingredient":', c.ingredient
, ',"ingAmount":', c.ingAmount, '}'
)
, ']'
)
(this assumes ingredient and ingAmount are numeric; if they are strings, you should double quote them, and escape any double quotes that appear within the string values)
The concatenation of ingredients with GROUP_CONCAT can lead to problems if you keep a default setting for the group_concat_max_len server variable. A trivial way to mitigate that problem is to set it to the maximum theoretical size of any result:
SET group_concat_max_len = ##max_allowed_packet;
You can either execute this once after you open the connection to mysql, and it will then be in effect for the duration of that session. Alternatively, if you have the super privilege, you can change the value across the board for the entire MySQL instance:
SET GLOBAL group_concat_max_len = ##max_allowed_packet;
You can also add a line to your my.cnf or my.ini to set group_concat_max_lenght to some arbitrary large enough static value. See http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_group_concat_max_len
One obvious solution is to actually perform two queries:
1) get the food
SELECT a.foodId, a.foodName, a.foodDescription, a.typeOfFood
FROM `foods` a
WHERE a.foodsId=4
2) get all of its ingredients
SELECT c.ingredient, c.ingAmount
FROM `foods_ing` c
WHERE c.foodsId=4
This approach has the advantage that you don't duplicate data from the "foods" table into the result. The disadvantage is that you have to perform two queries. Actually you have to perform one extra query for each "food", so if you want to have a listing of foods with all their ingredients, you would have to do a query for each of the food record.
Other solutions usually have many disadvantages, one of them is using GROUP_CONCAT function, but it has a tough limit on the length of the returned string.
When you compare MySQL's aggregate functions and GROUP BY behavior to SQL standards, you have to conclude that they're simply broken. You can do what you want in a single query, but instead of joining directly to the table of ratings, you need to join on a query that returns the results of the aggregate functions. Something along these lines should work.
select a.foodId, a.foodName, a.foodDescription, a.typeOfFood,
c.ingredient, c.ingAmount,
b.numRatings, b.avgRating
from foods a
left join (select foodId, count(foodId) numRatings, avg(foodRating) avgRating
from foods_ratings
group by foodId) b on a.foodId = b.foodId
left join foods_ing c on a.foodId = c.foodId
order by a.foodId