Taking long time executing complex mySql query - php

Question

My query looks like this:
$result_portlist=mysql_query("
SELECT
portfolio AS the_port,
SUM(balance * course) AS the_balance,
SUM(insurance * course) AS the_insurance
FROM banks_all
WHERE CONCAT(balance, insurance)!='0'
GROUP BY portfolio
",$db);
$myrow_portlist=mysql_fetch_array($result_portlist);
if(mysql_num_rows($result_portlist)>0) {
do {
echo '<span> value1, vaule2, value3...</span>';
$result_portlist1=mysql_query("
SELECT
department AS the_department,
SUM(balance * course) AS the_balance,
SUM(insurance * course) AS the_insurance
FROM banks_all
WHERE CONCAT(balance, insurance)!='0'
AND portfolio='$myrow_portlist[the_port]'
GROUP BY department
",$db);
$myrow_portlist1=mysql_fetch_array($result_portlist1);
if (mysql_num_rows($result_portlist1)>0) {
do {
echo '<span> value1, vaule2, value3...</span>'
$result_portlist2=mysql_query("
SELECT
manager_name AS the_manager,
SUM(balance * course) AS the_balance,
SUM(insurance * course) AS the_insurance
FROM banks_all
WHERE CONCAT(balance, insurance)!='0'
AND portfolio='$myrow_portlist[the_port]'
AND department='$myrow_portlist1[the_department]'
GROUP BY manager_name
",$db);
{
do {
echo '<span> value1, vaule2, value3...</span>';
} while ($myrow_portlist2=mysql_fetch_array($result_portlist2));
}
} while ($myrow_portlist1=mysql_fetch_array($result_portlist1));
}
} while ($myrow_portlist1=mysql_fetch_array($result_portlist1));
}
So this query takes hours to execute for table with 40,000+ rows with hundreds of combinations of portfolio+department+manager_name. What I thought was to make one query at start that would group values by manager_name and then let php group portfolio and department. After creating this query, my mind has just blown up and has no free memory left to think :)
Please advice me how to rearrange this query or how to simplify it to take less time to execute.
Thanks in advance.
EDIT: here is image, how my table (with nested tables) look like:
IMAGE

Answer 1

The reason why it takes a lot of time is because you are executing a massive amount of queries... Let's say you have 10 portfolios, in 10 departments, you'd make 100 queries. This amount is insane, and reducing the number of queries is pretty simple in your case.
I would simply grab all the data in one query, and use a 3 level group by with a rollup:
$result=mysql_query("
SELECT
portfolio AS the_port,
department AS the_department,
manager_name AS the_manager,
SUM(balance * course) AS the_balance,
SUM(insurance * course) AS the_insurance
FROM banks_all
WHERE
CONCAT(balance, insurance)!='0'
GROUP BY
portfolio,
department,
manager_name
with ROLLUP
ORDER BY the_port,the_department,the_manager;
",$db);
echo '<table>';
while($row = mysql_fetch_assoc($result)){
if($row['the_port'] === null){
//This is the big total row... Let's skip it for now shall we?
continue;
}
echo '<tr>';
if($row['the_department'] === null){
echo '<td>Portfolio: '.$row['the_port'].'</td>';
}else{
if($row['the_manager'] === null){
echo '<td>Department: '.$row['the_port'].'</td>';
}else{
echo '<td>Manager: '.$row['the_port'].'</td>';
}
}
echo '<td>'.$row['the_balance'].'</td>';
echo '<td>'.$row['the_insurance'].'</td>';
echo '</tr>';
}
echo '</table>';
This query will return all lines grouped by portfolio then grouped by department, then grouped by manager. This would return all rows from the finest point (your third while) but with ROLLUP you will also get a row for the other group by levels. This is exactly as if you would have done GROUP BY portfolio,department and GROUP BY portfolio inside the same query.
In the rolled up rows, the value of the keys omitted from the group by will be NULL. So you can easily check on the_manager, when it's null it means you have the full department, when the_department is null you have the full portfolio.