We have a registration system Database and basically what this query does is check the students that are in the class so that they can be selected to be marked as absent if they are absent. For some reason, it takes 30 seconds. Does anybody know why?
FROM Stdts
LEFT JOIN StdtReg ON StdtReg.StdtID = Stdts.ID
LEFT JOIN usrs ON StdtReg.userID = usrs.ID
WHERE (SELECT ID FROM ClssInstncEnrol cie WHERE cie.status = 0 AND classInstanceID={$_GET['ci']} AND StdtID = Stdts.ID LIMIT 1) IS NOT NULL
OR (SELECT ID FROM DropIns di WHERE di.type <> -1 AND classInstanceID= {$_GET['ci']} AND StdtID = Stdts.ID LIMIT 1) IS NOT NULL
AND (CONCAT(Stdts.firstName, ' ', Stdts.lastName) OR CONCAT(usrs.firstName,' ', usrs.lastName))
ORDER BY firstName, lastName
Run the query with "EXPLAIN " before it and it will tell you how each table is being joined and where you might be missing an index.
Also, you have an SQL injection waiting to happen with queries of this form with HTTP params interpolated directly in the query.
Finally, you've left off some of the query and the schema, but this strikes me as something that could be done with joins rather than subselects, or even as separate queries to generate the list of student ids more efficiently before you even run the main query.
Try to check the execution plan of your query to see what could be wrong (if you have huge tables and do not use appropriate index it can be long)
Maybe this:
AND (CONCAT(Stdts.firstName, ' ', Stdts.lastName) OR CONCAT(usrs.firstName,' ', usrs.lastName))
should be:
AND (CONCAT(Stdts.firstName, ' ', Stdts.lastName) = CONCAT(usrs.firstName,' ', usrs.lastName))
Related
This question already has answers here:
Search Form with One or More (Multiple) Parameters
(2 answers)
Closed 4 years ago.
I have search field in which user can specify a lot of values to search like price, surface, year, garden, balcony etc.
In my search there is not even one field required every one is optional so user can provide 0 inputs filled or all.
Basically all this info's are saved in my database but I don't really know how to structure my code.
At the moment I have PHP file which I call from front and in this file I'm checking which field was filled and I'm executing method from class which do select to db and return data. This is working fine for every input separately but when I combain for example 2 different fields like price and surface then none of methods will be executed.
Im basically asking about an idea for architecture of search where user can fullfill many different fields. Im not using any PHP framework.
I could do something like:
if(a & b & c & d & e & f) then execute method a
if(a & b & c & d & e) then execute method b
if(a & b & c & d) then execute method c
and so on.. where this letters(a, b, c etc...) are $_POST['something'] but I would have a lots of if's to check which POST (which inputs) user fullfill and sent. Later on I would need to create a lot of methods in class with different SELECTs to db basing on which POST we have... I don't think that's best solution because I would basically repeat my code.
Something like this
$sql = 'SELECT * FROM sometable';
$where = [];
$params = [];
if($a){
$where[] = 'a = :a';
$params[':a'] = $a;
}
if($b){
$where[] = 'b = :b';
$params[':b'] = $b;
}
if(!empty($where)){
$sql .= ' WHERE '.implode(' AND ', $where);
}
$stmt = $PDO->prepare($sql);
$res = $stmt->execute($params);
And so On.
It almost always preferable to use and array and implode for things like this instead of concatenation. Often concatenation will leave you with a hanging "separator" in this case " AND ". For example if we tried this with concatenation:
//if we put WHERE here and then nothing passes our conditions we wind up with:
//"SELECT * FROM sometable WHERE" which wont work
$sql = 'SELECT * FROM sometable ';
//we still need something like an array if we want to prepare our query.
//which is something we should always do
$params = [];
if($a){
//if we put WHERE here, then what if this condition doesn't pass
//do we put it in the next condition? How do we tell. .
$sql .= 'WHERE a = :a AND ';
$params[':a'] = $a;
}
if($b){
//again if the first condition didn't pass how do we know to put "WHERE" here.
//"SELECT * FROM sometable b = :b AND" which wont work
$sql .= 'b = :b AND ';
$params[':b'] = $b;
}
if($c){
//lets say the first 2 conditions passes but this last one failed
//"SELECT * FROM sometable WHERE a = :a AND b = :b AND" which wont work
$sql .= 'c = :c';
$params[':c'] = $c;
}
//we would need to do something like this to trim the last "AND" off
$sql = preg_replace('/\sAND\s$/', '', $sql);
//--------------------
//now if we were prepending "AND" instead of appending it, we're no better off.
//--------------------
//we can fix the where issue by using a string variable (and testing it latter)
$where = '';
if($a){
$where .= 'a = :a';
$params[':a'] = $a;
}
if($b){
//However lets say the first condition failed, we get this:
//"SELECT * FROM sometable WHERE AND b = :b" which wont work
$where .= ' AND b = :b';
$params[':b'] = $b;
//--------------------------
//so in every condition following we would have to test $where
//and if its not empty then we can prepend "AND"
if(!empty($where)) $where .= ' AND ';
$where .= 'b = :b';
$params[':b'] = $b;
}
if($c){
if(!empty($where)) $where .= ' AND ';
$where .= 'c = :c';
$params[':c'] = $c;
}
//finally to fix the "WHERE" issue we need to do something like this:
if(empty($where)) $sql .= ' WHERE '.$where;
//we could also try something like this in every condition:
if($d){
if(empty($where)) $where .= ' WHERE ';
//However, this breaks our fix for prepending "AND", because
//$where will never be empty when we test it.
//if(!empty($where)) $where .= ' AND ';
$where .= 'd = :d';
$params[':d'] = $d;
}
Hopefully that all makes sense. It's just so much easier to use an array and implode it later.
I just wanted to show that to help visualize the issues with concatenation. We wind writing more code, using the same number of variables and double the conditional logic. Or we can get into complicated things like Regex to trim the hanging AND off etc.
Hope that helps!
BECAUSE I mentioned it in the comments.
If you are using "OR" you can of course do the same thing with that, but typically "OR" will cause a full scan of the DB. It's just the way OR works. When we use "AND" the DB (basically) takes the return set and applies the next condition to that, because both have to pass. However, with "OR" rows that failed the first condition could still pass if the second condition passes. So the DB must scan the full record set for each or, as well as keep track of all the rows that passed in the previous conditions. It's just the way the logic works for "OR".
Now for improved "OR" performance we can use a sub-query that is a union. Like this:
$sql = 'SELECT * FROM sometable AS t';
$union = [];
$params = [];
if($a){
$union[] = 'SELECT id FROM sometable WHERE a = a:';
$params[':a'] = $a;
}
if($b){
$union[] = 'SELECT id FROM sometable WHERE b = b:';
$params[':b'] = $b;
}
if(!empty($union)){
$sql .= '
JOIN( '.
implode(' UNION ', $union).
' ) AS u ON t.id = u.id
}
What we wind up with is something like this query:
SELECT
*
FROM
sometable AS t
JOIN (
SELECT id FROM sometable WHERE a = a:
UNION
SELECT id FROM sometable WHERE b = b:
) AS u ON t.id = u.id
When we use "OR" as our dataset grows the DB must store these results in temp table as well as search the entire dataset. Because we are pulling all the columns in the table, this dataset will quickly grow. Once it hits a certian size it will get swapped to Disc and our performance will take a big hit for that.
With the Union query, we also create a temp table. But because we are only concerned with pulling out the ids this temp table will be very small. Union unlike Union ALL will also automatically remove duplicate records further reducing our dataset. So we want to use Union and not Union ALL.
Then we join this back on the table in the outer query and use that to pull the all the columns from just the rows that we need.
Basically we are accepting the fact that we need a temp table and minimizing the impact of that.
This might not seem like it would be much faster, and in some cases it might not be (when no swapping happens). But for me, using a query like you describe where users can search on multiple fields, I was able to reduce the time it took from about 15 seconds to under 1 second. My query had several joins in it such as if a user put in a state, I had to join on participant then participants_addresses (junction table) and then addresses and then finally on states. But if they put in a phone I had to join participant > participants_phones > phone etc.
I can't guarantee this will work in every case and you should use Explain and SQL_NO_CACHE when benchmarking your queries. For example EXPLAIN SELECT SQL_NO_CACHE * FROM .... Explain will tell you how the indexes are working and No Cache prevents the DB from caching the query if you run it multiple times. Caching will make it look like it is fast when it's really not.
You can do something similar when sorting, which also kills performance.
SELECT
*
FROM
sometable AS t
JOIN (
SELECT id FROM sometable WHERE a = a: ORDER BY date DESC
) AS u ON t.id = u.id
This has a similar effect of only sorting the id's in the temp table (instead of the whole dataset), and then when we join it, it actually keeps the order the ids are in. I forget if the order of the subquery vs the outer query matter.
For fun you can even combine the two with 2 nested sub-queries, with the Union as the deepest query (it's something like this).
SELECT
*
FROM
sometable AS t
JOIN (
SELECT id FROM sometable AS t0 JOIN (
SELECT id FROM sometable WHERE a = a:
UNION
SELECT id FROM sometable WHERE b = b:
) AS u ON t0.id = u.id
ORDER BY t0.date DESC
) AS t1 ON t.id = t1.id
It can get pretty complicated though ... lol.
Anyway, I was bored and maybe, just maybe, it will work for someone like it did for me. (this is what happens when I don't get sleep) :)
UPDATE
IF you have problems with the parameters you can output the SQL with the values filled in by doing this:
echo str_replace(array_keys($params), $params, $sql)."\n";
But use this only for Debugging, not for putting the data into the query because that would defeat the purpose of using prepared statements and open you up to SQLInjection attacks. That said, it can make it easier to see if you are missing anything or have any spelling errors. I also use this when I just want to test the query in PHPMyAdmin, but am to lazy to cut an paste the data into it. Then I just copy the output put it in PHPMyAdmin and then I can rule out any issues with PHP or tweak the query if need be.
You can also have issues if you have to many elements in the array, AKA extra placeholders that are not in the query.
For that you can do
//count the number of : in the query
$num_placeholders = substr_count(':', $sql);
//count the elements in the array
$num_params = count($params);
if($num_placeholders > $num_params ) echo "to many placeholders\n";
else if($num_placeholders < $num_params ) echo "to many params\n";
One last thing to be mindful of when mixing "AND" and "OR" is stuff like this
SELECT * FROM foo WHERE arg1 = :arg1 OR arg2 = :arg2 AND arg3 = :arg3
The way it executes this is like this
SELECT * FROM foo WHERE arg1 = :arg1 OR (arg2 = :arg2 AND arg3 = :arg3)
This will return all rows that match arg1 regardless of the rest of the query.
Most of the time this would not be what you want. You would actually want it to do it this way:
SELECT * FROM foo WHERE (arg1 = :arg1 OR arg2 = :arg2) AND arg3 = :arg3
Which is called an "Exclusive OR". This will return all rows that match arg1 OR arg2 AND arg3
Hope that helps.
You could also create an wanted list of nesseccary items and Check If each Item is Set by the PHP function isset().
OK.. so I had a post here:
MySQL/PHP/PDO + How to get a row for each (duplicate) entry in IN() clasue?
Apparently... there is no solution. (or so I'm told)..
So is there an alternative solution to using the IN() clause? One that DOES in fact return a row for each item passed in...regardless if its a duplicate entry or not?
I have suggestions about using a (self) JOIN.. or possibly even EXISTS... but I am not clear on how I can go about adjusting my current dynamic query using such suggestions?
$qMarks = str_repeat('?,', count($brandlist) - 1) . '?'; //create '?' mark placeholders for query, remove last comma and replace with '?'
//preserve IN() order
$displayList_sql = "SELECT * FROM $tablename WHERE CONCAT(brandname, ' ', dosage) IN ($qMarks) ORDER BY FIELD(CONCAT(brandname, ' ', dosage),'". trim(implode("','", $brandlist))."')";
$displayList_stmt = $conn->prepare($displayList_sql);
$displayList_stmt->execute($brandlist);//make note of passing in array as param to execute() call
Can this be altered to use a JOIN or EXISTS (anything) so that it returns a row for each item? (which is a dynamically posted array?)
It's not exactly pretty, but you can "convert" your list to a subquery and join that against the actual table you want to query.
SELECT t.stuff
FROM (SELECT 'in item 1' AS item
UNION ALL SELECT 'in item 2'
UNION ALL ...
) AS inList
INNER JOIN $tablename AS t ON inList.item = CONCAT(t.brandname, ' ', t.dosage)
ORDER BY ...
My guess is there is probably some php methods you can use to explode/split the variable you had used to populate the IN list to create the SELECT ... UNION ALL ... subquery.
Worst, most primitive case (in pseudo code), assuming a simple comma-separated list:
theList = "SELECT " + REPLACE(theList, ',', ' AS item UNION ALL SELECT')
If query length becomes an issue, another option is to create a temp table to store the IN list in, and then join against that. (This technique can also sometimes be used to make the query faster; since the temp table can be indexed to help with the join operation.)
I have a script that goes through all order history. It takes several minutes to print the results, but I noticed I perform several SQL statements that are similar enough I wonder if you could do another query on an existing SQL result.
For example:
-- first SQL request
SELECT * FROM orders
WHERE status = 'shipped'
Then, in a foreach loop, I want to find information from this result. My naive approach is to perform these three queries. Note the similarity to the query above.
-- grabs customer's LTD sales
SELECT SUM(total) FROM orders
WHERE user = :user
AND status = 'shipped'
-- grabs number of orders customer has made
SELECT COUNT(*) FROM orders
WHERE user = :user
AND status = 'shipped'
AND total != 0
-- grabs number of giveaways user has won
SELECT COUNT(*) FROM orders
WHERE user = :user
AND status = 'shipped'
AND total = 0
I end up querying the same table several times when the results I seek are subsets of the first query. I'd like to get information from the first query without performing more SQL calls. Some pseudocode:
$stmt1 = $db->prepare("
SELECT * FROM orders
WHERE status = 'shipped'
");
$stmt1->execute();
foreach($stmt1 as $var) {
$username = $var['username'];
$stmt2 = $stmt1->workOn("
SELECT SUM(total) FROM this
WHERE user = :user
");
$stmt2->execute(array(
':user' => $username
));
$lifesales = $stmt2->fetchColumn();
$stmt3 = $stmt1->workOn("
SELECT COUNT(*) FROM this
WHERE user = :user
AND total != 0
");
$stmt3->execute(array(
':user' => $username
));
$totalorders = $stmt3->fetchColumn();
$stmt4 = $stmt1->workOn("
SELECT COUNT(*) FROM this
WHERE user = :user
AND total = 0
");
$stmt4->execute(array(
':user' => $username
));
$totalgaws = $stmt4->fetchColumn();
echo "Username: ".$username;
echo "<br/>Lifetime Sales: ".$lifesales;
echo "<br/>Total Orders: ".$totalorders;
echo "<br/>Total Giveaways: ".$totalgaws;
echo "<br/><br/>";
}
Is something like this possible? Is it faster? My existing method is slow and ugly, I'd like a quicker way to do this.
We could do one pass through the table to get all three aggregates for all users:
SELECT s.user
, SUM(s.total) AS `ltd_sales`
, SUM(s.total <> 0) AS `cnt_prior_sales`
, SUM(s.total = 0) AS `cnt_giveaways`
FROM orders s
WHERE s.status = 'shipped'
GROUP
BY s.user
That's going to be expensive on large sets. But if we are needing that for all orders, for all users, that's likely going to be faster than doing separate correlated subqueries.
An index with leading column of user is going to allow MySQL to use the index for the GROUP BY operation. Including the status and total columns in the index will allow the query to be satisfied entirely from the index. (With the equality predicate on status column, we could also try an index with status as the leading column, followed by user column, then followed by total.
If we only need this result for a small subset of users e.g. we are fetching only the first 10 rows from the first query, then running a separate query is likely going to be faster. We'd just incorporate the condition WHERE s.user = :user into the query, as in the original code. But run just the one query rather than three separate queries.
We can combine that with the first query by making it into an inline view, wrapping it in parens and putting into the FROM clause as a row source
SELECT o.*
, t.ltd_sales
, t.cnt_prior_sale
, t.cnt_giveaways
FROM orders o
JOIN (
SELECT s.user
, SUM(s.total) AS `ltd_sales`
, SUM(s.total <> 0) AS `cnt_prior_sales`
, SUM(s.total = 0) AS `cnt_giveaways`
FROM orders s
WHERE s.status = 'shipped'
GROUP
BY s.user
) t
ON t.user = o.user
WHERE o.status = 'shipped'
I'm not sure about that column named "prior" sales... this is returning all shipped orders, without regard to comparing any dates (order date, fulfillment date, shipment date), which we would typically associate with a concept of what "prior" means.
FOLLOWUP
noticing that the question is modified, removing the condition "status = 'shipped'" from the count of all orders by the user...
I will note that we can move conditions from the WHERE clause into the conditional aggregates.
Not that all these results are needed by OP, but as a demonstration...
SELECT s.user
, SUM(IF(s.status='shipped',s.total,0)) AS `ltd_sales_shipped`
, SUM(IF(s.status<>'shipped',s.total,0)) AS `ltd_sales_not_shipped`
, SUM(s.status='shipped' AND s.total <> 0) AS `cnt_shipped_orders`
, SUM(s.status='canceled') AS `cnt_canceled`
, SUM(s.status='shipped' AND s.total = 0) AS `cnt_shipped_giveaways`
FROM orders s
GROUP
BY s.user
Once the results are returned from the database, you can not run an SQL on top of them. However you can store them in a temporary table, to reuse them.
https://dev.mysql.com/doc/refman/8.0/en/create-temporary-table.html
https://dev.mysql.com/doc/refman/8.0/en/create-table-select.html
https://dev.mysql.com/doc/refman/8.0/en/insert-select.html
You need to create a temporary table, and insert all the data from the select statement, and then you can run queries on that table. Not sure if it would help much in your case.
For your particular case you can do something like:
select user, (total = 0) as is_total_zero, count(*), sum(total)
from orders
where status = 'shipped'
group by user, total = 0
However you would have to do some additional summing to get the results of the second query which gives you the sums per user, as they would be divided into two different groups with a different is_total_zero value.
I have two different tables where I need to fetch the list of store ids from one table and then find the list of coupons for those store ids.
Currently,
SELECT `storename` FROM `stores` where `brandname` = 21
This will return something like
Store 1
Store 2
Store 3
Store 4
And I need to to run another query like
SELECT * FROM `coupons` where `storename` = {{All these stores}}
I can't use while loops because, the number of stores comes from first query can't be determined and the the output I want was not coming as expected while using while loop as I am trying to do something like
while(first query output get storename)
{
do query here
while(second query output get all coupons per store)
{
// All coupons display here.
}
}
This is making quite complicated as well, is there anyway that I can tweak my SQL query and get results easily?
Thanks
you can use this query:
SELECT * FROM `coupons`
where `storename` IN (
SELECT `storename` FROM `stores` where `brandname` = 21);
$query = 'SELECT t.storename, h.couponid AS couponsid, h.coupon AS couponvalue'
. ' FROM #__stores AS t'
. ' LEFT JOIN #__coupons AS h ON h.storename = t.storename'
. ' where `brandname` = 21'
. ' ORDER BY anything you like'
;
i had to check if a value (string) is in my database.
at the moment i do a
select a.email,b.vuid from user a, verteiler_user b where a.email=\''.$email.'\' and a.kid=' . $kid.' and b.vid=' . $vid . ' and a.uid = b.uid
as query with a mysql_num_rows, then check if >=1
But is it faster to do a query with limit 1; ? and check if a row is coming back ?
Yes. It would be faster to run a limit 1 query. And if all you're doing is checking for the existence of a row, why bother returning all those columns? Simply select 1. That will be (negligibly) faster. BTW: Your code looks vulnerable to SQL injection attacks. Consider sanitizing the dynamic parts of your query with mysql_real_escape_string() or a similar function.
LIMIT 1 will have better performance because it will stop matching when the LIMIT has been satisfied.
If you only ever expect 1 row, add LIMIT 1 to your query.
Also, if you are only checking for the presence of that string in the query, there is no need to use column names with SELECT in the query. Just use SELECT 1....
you could try:
select count(*) from user a, verteiler_user b
where a.email=\''.$email.'\' and a.kid=' . $kid.' and b.vid=' . $vid . ' and a.uid = b.uid
and get the count by:
$row=mysql_fetch_array(...);
if ($row[0] > 0) // is there a hit?
...