Me and our team are experiencing a strange problem with MySQL queries. We use a SELECT statement with a COUNT and for some reason it's quite 'fast' in the client we use (SQLyog), but it's realy slow when we use PHP.
We have tried to use the ancient mysql_query(), the mysqli extension and we also tried to use PDO but all didn't make a difference.
On other posts here on Stackoverflow we found that it might be a DNS issue and that it could be fixed using 'skip_name_resolve' in the my.ini but we already had this in our configuration.
Timed results:
Client: 2.092 sec
PHP: 9.1071 sec
This is the query we use:
SELECT SQL_NO_CACHE
COUNT(m.mm_id) AS total
FROM
db.media_multimedia m
WHERE m.cat_id IN
(SELECT
mc.cat_id
FROM
db.media_multimedia_category mc
WHERE mc.cat_active = 1)
AND m.mm_published = 1
AND (
m.mm_title LIKE "%denniy%"
OR m.mm_text LIKE "%denniy%"
OR m.mm_id IN
(SELECT
a.mm_id
FROM
db.`media_tag_multimedia` a
LEFT JOIN media.`media_tag` b
ON a.`tag_id` = b.tag_id
WHERE b.tag_name LIKE "%denniy%")
)
AND m.mm_publishing_date >= "2012-04-24 00:00:00"
AND m.mm_publishing_date <= "2013-04-24 23:59:59" ;
*NOTE: for this testcase we added SQL_NO_CACHE to the query to make sure we always fetch a new result set.*
We are using the following PHP and MYSQL versions:
MySQL: 5.1.61
PHP: 5.3.3
Any suggestions to fix this problem?
I can't explain the difference in performance, although if I had to guess I'd say one or some of the following things are in play:
The server is optimising your query differently because of the different client connections,
The different location of your SQLyog and PHP clients might be a factor
Like I said, just guesses.
But regardless, I've attempted to tidy up your query as follows. I wonder if this might perform better (and more consistently)?
SELECT SQL_NO_CACHE
COUNT(m.mm_id) AS total
FROM
db.media_multimedia m
INNER JOIN db.media_multimedia_category mc
ON m.cat_id = mc.cat_id
AND mc.cat_active = 1
LEFT JOIN db.media_tag_multimedia a
ON m.mm_id = a.mm_id
INNER JOIN media.media_tag b
ON a.tag_id = b.tag_id
WHERE
m.mm_published = 1
AND
(
m.mm_title LIKE "%denniy%"
OR
m.mm_text LIKE "%denniy%"
OR
b.tag_name LIKE "%denniy%"
)
AND m.mm_publishing_date >= "2012-04-24 00:00:00"
AND m.mm_publishing_date <= "2013-04-24 23:59:59" ;
Related
Only in a sql query (used in a php script), I need to use the value of a column into the query to optimize the speed but it doesn't work well.
I have mutiple tables so I use JOIN
If I do that :
LEFT JOIN `gestdetails` gd ON gd.id IN (205467,205468,205469)
I works well !
But if I use the value of a column (gc.details = 205467,205468,205469) :
LEFT JOIN `gestdetails` gd ON gd.id IN (gc.details)
It doesn't work, it only use the first value '205467'.
I'm trying to do that to go faster with the query : if I do the classic JOIN on id = id, it also works but it's way too long...
LEFT JOIN `gestdetails` gd ON (gd.idcommande = gc.id)
gestcommandes = table of unique orders
gestdetails = table of multiple products linked to idcommande of gestcommandes
Here is the full query :
SELECT
gc.*,
count(gd.idcommande),
GROUP_CONCAT(CONCAT(gd.id,"#",gd.idcommande,"#",gd.ref,"#",gd.poids,"#",gd.numcolis,"#",gd.designation,"#",gd.prix) SEPARATOR "|"),
ad.*
FROM
`gestcommandes` gc
LEFT JOIN `gestdetails` gd ON gd.id IN (gc.details)
LEFT JOIN `adresses` ad ON ad.id_commande = gc.id
WHERE
`mp` LIKE '%-eb' AND (`statut` = 5 OR `statut` = 6) AND `date_expe` >= '2020-04-07 00:00:00' AND `date_expe` <= '2020-04-07 23:59:59'
AND
ad.is_delivery = 1
GROUP BY
gc.id
Executing this query, it only uses the first number into the gc.details column (i.e '205467' from '205467,205468,205469' and of course, I've got warnings :
Warning: #1292 Truncated incorrect INTEGER value: '205467,205468,205469'
gc.details can count 1 to 100 numbers...
I tryed so many things, I don't manage to do it, so if anybody can help me (without creating procedure)... thank you !
ps : the "GROUP_CONCAT" part works well... if I manage to get more than 1 result for the details...
I have almost thousands of data to display for my reports and it makes my browser lags due to the heavy data. I think that my query is the real problem. How can I optimized my query? is there something that I should add in my query?
I am using Xampp which supports PHP7.
SELECT
`payroll_billed_units`.`allotment_code`,
`payroll_billed_units`.`category_name`,
`payroll_billed_units`.`ntp_number`,
`payroll_billed_units`.`activity`,
`payroll_billed_units`.`regular_labor`,
`payroll_sub`.`block_number`,
(SELECT
GROUP_CONCAT(DISTINCT `lot_number` SEPARATOR ', ')
FROM
`payroll_billed_units` `lot_numbers`
WHERE
`lot_numbers`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `lot_numbers`.`category_name` = `payroll_billed_units`.`category_name`
AND `lot_numbers`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `lot_numbers`.`activity` = `payroll_billed_units`.`activity`) AS `lot_numbers`,
(SELECT
COUNT(`billed`.`ntp_id`)
FROM
`regular_ntp` `billed`
WHERE
`billed`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `billed`.`category_name` = `payroll_billed_units`.`category_name`
AND `billed`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `billed`.`activity` = `payroll_billed_units`.`activity`) AS `billed`,
(SELECT
COUNT(`approved`.`id`)
FROM
`payroll_billed_units` `approved`
WHERE
`approved`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `approved`.`category_name` = `payroll_billed_units`.`category_name`
AND `approved`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `approved`.`activity` = `payroll_billed_units`.`activity`) AS `approved`
FROM
`payroll_billed_units`
JOIN payroll_transaction ON payroll_billed_units.billing_number =
payroll_transaction.billing_number
JOIN payroll_sub ON payroll_transaction.billing_number =
payroll_sub.billing_number
WHERE payroll_billed_units.billing_date = '2019-02-13'
AND payroll_transaction.contractor_name = 'Roy Codal' GROUP BY allotment_code, category_name, activity
I was expecting that it will load or display all my data.
The biggest problem are the dependendt sub-selects, they are responsible for a bad performance. A sub-select will be executed for EVERY ROW of the outer query. And if you cascade subs-selects, you'll quickly have a query run forever.
If any of the parts would yield only 5 resultsets, 3 sub-select would mean that the database has to run 625 queries (5^4)!
Use JOINs.
Several of your tables need this 'composite' index:
INDEX(allotment_code, category_name, ntp_number, activity) -- in any order
payroll_transaction needs INDEX(contractor_name), though it may not get used.
payroll_billed_units needs INDEX(billing_date), though it may not get used.
For further discussion, please provide SHOW CREATE TABLE for each table and EXPLAIN SELECT ...
Use simply COUNT(*) instead of COUNT(foo). The latter checks the column for being not-NULL before including it. This is usually not needed. The reader is confused by thinking that there might be NULLs.
Your GROUP BY is improper because it is missing ntp_number. Read about the sql_mode of ONLY_FULL_GROUP_BY. I bring this up because you can almost get rid of some of those subqueries.
Another issue... Because of the "inflate-deflate" nature of JOIN with GROUP BY, the numbers may be inflated. I recommend you manually check the values of the COUNTs.
I'm using Laravel 5.2 and PHP 7.0.6 on Debian. MySQL is 5.5.30 on a remote server.
When I run the query through HeidiSQL or from the command line client it executes in about 0.2 seconds. Executing the exact same SQL in Laravel (debug mode on or off) takes nearly 500 seconds. I tried pulling the Laravel query out of the MySQL query log and running it, and it ran fast. It's only when executing the statement through Laravel that it's slow. The laravel code is below. My timer code is before and after this line, so the slowdown is definitely here. The query only returns one row.
$results = \DB::select($sql);
An anonymized version of the query is below (business-specific data would otherwise be visible). Hopefully I haven't mangled anything in the process.
SELECT ll.id,
ll.other_id,
ll.third_id,
ll.created_at,
ll.h_id,
ll.sub,
ll.sub2,
ll.status,
px.created_at,
b.abbr,
ldl.transaction_id,
ldl.purchase_price,
CONCAT(ld.first_name, ' ', ld.last_name) as fullname,
lcase(ll.email_address) as email_address,
ll.total_revenue
FROM table1 ll
INNER JOIN table2 ld on ll.id = ld.fid
INNER JOIN table3 ldl on ll.id = ldl.fid
LEFT OUTER JOIN table4 px on ll.id = px.fid
INNER JOIN table5 b on ldl.bid = b.id
WHERE ll.created_at > '2016-05-04 00:00:00'
AND ll.created_at < '2016-05-04 23:59:59'
AND v_id IN (41,42,43,45,46)
AND ldl.b_id IN (131)
AND lcase(ll.email_address) in ('example#email.com')
AND ll.status = 'ACCEPTED'
I experienced this issue and finally found a solution. I had a query that when I ran on the command line it executed in about 40ms. When I put the same exact query into Laravel using:
DB::select( $myQuery ); // Where $myQuery is the exact string I ran on the command line
...the query took about 1 minute. Something was clearly wrong.
In the end, it had everything to do with parameter binding. By simply changing this:
DB::select("SELECT * FROM users WHERE username='Andrew'");
...to this:
DB::select("SELECT * FROM users WHERE username = :name",['name' => 'Andrew']);
My problem was solved.
For any else struggling with this, we experienced a similar issue when querying a db view, but the solution was the opposite of the above.
When running this query in the DB directly, the response time was a fraction of a second.:
select * from `db_view` where `id` = 12345;
But when using fluent builder for the same query, response was > 12 seconds (!):
DB::table('db_view')->where('id', $this->id)->first();
Seems from mysql show full processlist that the db was creating a sort index every time the query was called, but only when called from Laravel, for some reason.
We tried this, as per the answer above, but it didn't help either:
DB::select("select * from `db_view` where `id` = :id", ['id' => $this->id])[0];
In the end, we tried excluding the binding entirely and it finally worked as expected, without building a sort index for each query:
DB::select("select * from `db_view` where `id` = {$this->id}")[0];
I'm having a performace problem with the execution of a select in PHP PDO.
Using a script available here at stackoverflow (Simplest way to profile a PHP script), I identified where the problem IS, but I have not found a solution.
My select that is the problem is:
SELECT REDACAO.ID_REDACAO AS ID_REDACAO,
DATE_FORMAT(REDACAO.DATA,'%d/%m/%Y') AS DATAE,
ALUNO.ID_ALUNO AS ID_ALUNO,
(SELECT IFNULL((DATEDIFF(DATE_ADD((SELECT MAX(DATA) FROM REDACAO WHERE ID_ALUNO = ALUNO.ID_ALUNO AND ID_REDACAO NOT IN (SELECT ID_REDACAO FROM CORRECAO)), INTERVAL 7 DAY), now())),NULL) as DATA FROM REDACAO LIMIT 1) AS ULTIMA,
ALUNO.NOME as ALUNO,
REDACAO.ID_TEMA AS ID_TEMA,
TEMA.TITULO as TEMA,
TEMA.MOTIVACIONAIS AS MOTIVACIONAIS,
REDACAO.TEXTO AS TEXTO,
REDACAO.ID_STATUS AS STATUS,
B.NOTA as NOTA,
B.RCORRIGIDA AS CORRIGIDA,
B.NOTA1,
B.COMENTARIO1,
B.NOTA2,
B.COMENTARIO2,
B.NOTA3,
B.COMENTARIO3,
B.NOTA4,
B.COMENTARIO4,
B.NOTA5,
B.COMENTARIO5,
B.COMENTARIO6,
C.COMENTARIO AS COMENTARIO
FROM REDACAO
LEFT OUTER JOIN (SELECT SUM(CORRECAO.C1+CORRECAO.C2+CORRECAO.C3+CORRECAO.C4+CORRECAO.C5) AS NOTA, RCORRIGIDA AS RCORRIGIDA, CORRECAO.C1 as NOTA1, CORRECAO.COM1 as COMENTARIO1, CORRECAO.C2 as NOTA2, CORRECAO.COM2 as COMENTARIO2, CORRECAO.C3 as NOTA3, CORRECAO.COM3 as COMENTARIO3, CORRECAO.C4 as NOTA4, CORRECAO.COM4 as COMENTARIO4, CORRECAO.C5 as NOTA5, CORRECAO.COM5 as COMENTARIO5, CORRECAO.COMGERAL AS COMENTARIO6, CORRECAO.ID_REDACAO FROM CORRECAO GROUP BY CORRECAO.ID_REDACAO) B
ON B.ID_REDACAO = REDACAO.ID_REDACAO
JOIN ALUNO ON ALUNO.ID_ALUNO = REDACAO.ID_ALUNO
JOIN TEMA ON TEMA.ID_TEMA = REDACAO.ID_TEMA
LEFT OUTER JOIN (SELECT (COUNT(COMENTARIO.ID_COMENTARIO)) AS COMENTARIO, COMENTARIO.ID_REDACAO FROM COMENTARIO GROUP BY COMENTARIO.ID_REDACAO) C
ON C.ID_REDACAO = REDACAO.ID_REDACAO
WHERE REDACAO.ID_PROFESSOR = $CodProfessor
and REDACAO.ID_STATUS != 6
ORDER BY (CASE WHEN REDACAO.ID_STATUS = 4 THEN 1 ELSE 0 END) DESC
I'm using (PDO :: FETCH_ASSOC) to get the data. Some columns respond in less than 1 second and others in more than 20 seconds.
Any idea what could be the problem and how to solve it?
Your query contains following that will slow it down:
many joins
many subselects
select without where
functions like COUNT, isnull, datediff, sum.(some of these may cancel an index)
case when
order by
group by
Depending on your indexes, on how the tables are joined, and on how big are the tables, this will eventually get very slower.
Try using 'explain' command, and simplify the query if possible.
explain output
a good video about explain
PDO is not at fault; the query is complex. And there may be missing indexes.
Turn this into a LEFT JOIN (because IN (SELECT...) optimizes poorly.
AND ID_REDACAO NOT IN ( SELECT ID_REDACAO FROM CORRECAO)
Upgrade to 5.6; it has some improvements.
You have two JOIN ( SELECT ... ). Before 5.6 that would be optimized terribly. Move one of them out into a temp table, to which you add a suitable index.
In one of the subqueries, GROUP BY CORRECAO.ID_REDACAO seems to be unnecessary.
These indexes (or PRIMARY KEYs) are needed:
CORRECAO: (ID_REDACAO)
REDACAO: (ID_REDACAO), (ID_PROFESSOR)
ALUNO: (ID_ALUNO)
TEMA: (ID_TEMA)
COMENTARIO: (ID_REDACAO, ID_COMENTARIO) ("compound index")
If those suggestions do not help enough, come back with SHOW CREATE TABLE for each table.
I have a complex MSSQL query with multiple joins. When I query using SQL Query Analyzer, it returns results after consuming about 5 minutes. However, when I query using PHP mssql_query() function, it returns an error mssql_query() failed with no useful information.
Followings are the analysis of this problem so far.
Response time limit of php server is not a problem. Time limit is set to infinite. In fact, this same query returns results after about 30 seconds on different production servers.
I tried using mssql_get_last_message() to get more details on why it failed. But so far, no luck. All I get is "Changed database context to 'dbsdb'", where 'dbsdb' is the name of database I am querying.
This is the query I am using, if that matters.
SELECT L.[strStation], L.[nEntitlement1], L.[nEntitlement2], O.nOrders,
L.[nEntitlement3], L.[nEntitlement4], S.[strFirm], S.[nStatus],
S.[nFeatures], S.[nFeatures2], S.[nFeatures3], S.[strDescription]
FROM [tblLogin] AS L
LEFT JOIN [tblStation] AS S ON L.[strStation] = S.[strStation]
LEFT JOIN (
SELECT [strStation], COUNT(DISTINCT(nRecordId)) as nOrders
FROM [tblOrder]
WHERE [timeOrderDate] >= '2013-06-04' AND
[timeOrderDate] <= '2013-06-28'
GROUP BY strStation
) AS O ON O.[strStation] = S.[strStation]
WHERE L.[timeLoginTime] >= '2013-06-04 00:00:00' AND
L.[timeLoginTime] <= '2013-06-28 23:59:59' AND
S.[strFirm] in ('FIRMA','FIRMB')
This is the code that I use to query.
$link = mssql_connect('192.168.251.90', 'sa', '');
if (!mssql_select_db('dbsdb', $link))
var_dump(mssql_get_last_message());
$query = mssql_query($sql, $link);
EDIT I
I narrowed down to one spot. When I remove O.nOrders part from the SELECT clause, the query works perfectly fine.
But WHY???
EDIT II
This is driving me crazy. Query works sometimes even with O.nOrders part in SELECT clause. This seems totally random to me, but I know there is no such thing as real random in this world...there should be a reason...