Querying efficiently - php

Question

I have two tables: Exam (ExamID, Date, Modality) and CT(ctdivol, ExamID(FK)) with the attributes in brackets.
Note: CT table has about 100 000 entries.
I want to calculate the average of ctdivol in a specific interval of dates.
I have this code that works but is too slow:
function get_CTDIvolAVG($min, $max) {
$values = 0;
$number = 0;
$query = "SELECT (unix_timestamp(date)*1000), examID
from exam use index(dates)
where modality = 'CT'
AND (unix_timestamp(date)*1000) between '" . $min . "' AND '" . $max . "';";
$result = mysql_query($query) or die('Query failed: ' . mysql_error());
while($line = mysql_fetch_array($result, MYSQL_ASSOC)) {
$avg = "SELECT SUM(ctdivol_mGy), count(ctdivol_mGy)
from ct use index(ctd)
where examID ='" . $line["examID"] ."'
AND ctdivol_mGy>0;";
$result1 = mysql_query($avg) or die('Query failed: ' . mysql_error());
while ($ct = mysql_fetch_array($result1, MYSQL_ASSOC)) {
$values = $values + floatval($ct["SUM(ctdivol_mGy)"]);
$number = $number + floatval($ct["count(ctdivol_mGy)"]);
}
}
if ($number!=0) {
echo $values/$number;
}
}
How can I make it faster?

Answer 1

Use EXPLAIN to see the query execution plan.
For that first query, MySQL can't make effective use of a index range scan operation. That expression in the WHERE clause has to be evaluated for every row in the table. We get better performance when we do the comparison to a bare column. Do the manipulation on the literal side... get those values converted to the datatype of the column you're comparing to.
WHERE e.date BETWEEN expr1 AND expr2
For expr1, you need an expression that converts your $min value into a datetime. Just be careful of timezone conversions. I think this might do what you need for expr1:
FROM_UNIXTIME( $min /1000)
Something like:
WHERE e.date BETWEEN FROM_UNIXTIME( $min /1000) AND FROM_UNIXTIME( $max /1000)
Then we should see MySQL able to make effective use of an index with leading column of date. The EXPLAIN output should show range for the access type.
If the number of columns being returned is a small subset, consider a covering index. Then the EXPLAIN will show "Using index", which means the query can be satisfied entirely from the index, with no lookups to pages in the underlying table.
Secondly, avoid running queries multiple times in a loop. It is usually more efficient to run a single query that returns a single resultset, because of the overhead of sending the SQL to the database, that database parsing the SQL text, for valid syntax (keywords in the right places), valid semantics (identifiers reference valid objects), considering possible access paths and determining which is lowest cost, then executing the query plan, obtaining metadata locks, generating the resultset, returning that to the client, and then cleaning up. It's not noticeable for a single statement, but when you start running a lot of statements in a tight loop, it starts to add up. Couple that with an inefficient query, and it starts to get really noticeable.
IF examID column in exam is unique and not null (or its the PRIMARY KEY of exam, then it looks like you could use a single query, like this:
SELECT UNIX_TIMESTAMP(e.date)*1000 AS `date_ts`
, e.examID AS `examID`
, SUM(ct.ctdivol_mGy) AS `SUM(ctdivol_mGy)`
, COUNT(ct.ctdivol_mGy) AS `count(ctdivol_mGy)`
FROM exam e
LEFT
JOIN ct
ON ct.examid = e.examID
AND ct.ctdivol_mGy > 0
WHERE e.modality = 'CT'
AND e.date >= FROM_UNIXTIME( $min /1000)
AND e.date <= FROM_UNIXTIME( $max /1000)
GROUP
BY e.modality
, e.date
, e.examID
ORDER
BY e.modality
, e.date
, e.examID
For best performance of that, you'd want covering indexes:
... ON exam (modality, date, examID)
... ON ct (examID, ctdivol_mGy)
We'd want to see the EXPLAIN output; we'd expect that MySQL could make use of the index on exam to do the GROUP BY (and avoiding a "Using filesort" operation), and also make use of a ref operation on the index to ct.
To reiterate... that query requires that examID be the PRIMARY KEY of the exam table (or at least be guaranteed to be unique and non-null). Otherwise, the result from that can be different than the original code. Absent that gurantee, we could use either an inline view, or subqueries in the SELECT list. But in terms of performance, we don't want to go there without good reason to.
That's just some general ideas, not a hard and fast "this will be faster".

Answer 2

You can write a join on the first table to a subquery table by exam_id:
$query = "SELECT (unix_timestamp(date)*1000) as time_calculation, ed.examID, inner_ct.inner_sum, inner_ct.inner_count "
" FROM exam ed,"
. " ( SELECT SUM(ctdivol_mGy) as inner_sum, count(ctdivol_mGy) as inner_count, examID"
. " FROM ct"
. " WHERE ctdivol_mGy>0 ) inner_ct"
. " WHERE ed.modality = 'CT' AND time_calculation between"
. " '$min' and '$max'"
. " AND ed.examId = inner_ct.examID";
The ( SELECT . . .) inner_ct creates an in memory table you can join from. Useful if you're selecting composed data (sums in your case) across a join.
Conversely, you can use the following syntax:
$query = "SELECT (unix_timestamp(date)*1000) as time_calculation, ed.examID, inner_ct.inner_sum, inner_ct.inner_count "
" FROM exam ed,"
. " LEFT JOIN ( SELECT SUM(ctdivol_mGy) as inner_sum, count(ctdivol_mGy) as inner_count, examID"
. " FROM ct"
. " WHERE ctdivol_mGy>0 ) inner_ct"
. " ON ed.examID = inner_ct.examID"
. " WHERE ed.modality = 'CT' AND time_calculation between"
. " '$min' and '$max'";

Answer 3

You have not provided sample data in the question so we resort to assumptions in an attempt to answer. If there is only one exam row for many rows in ct - but an exam row can exist that has no ct rows at all - then this single query should provide the results required.
SELECT
exam.examID
, (unix_timestamp(exam.date) * 1000
, SUM(ct.ctdivol_mGy)
, COUNT(ct.ctdivol_mGy)
FROM exam
LEFT OUTER JOIN ct on exam.examID = ct.examID AND ct.ctdivol_mGy > 0
WHERE exam.modality = 'CT'
AND exam.date >= #min AND exam.date < #max
GROUP BY
exam.examID
, (unix_timestamp(exam.date) * 1000)
;
Note I am not attempting the PHP code, just concentrating on the SQL. I have used #min and #max to indicate the 2 dates required in the where clause. These should be of the same data type as the column exam.date so do those calculations in PHP before adding into the query string.
I want to calculate the average of ctdivol in a specific interval of
dates.
If you are trying to return a single figure, then this should help:
SELECT
AVG(ct.ctdivol_mGy)
FROM exam
INNER JOIN ct on exam.examID = ct.examID AND ct.ctdivol_mGy > 0
WHERE exam.modality = 'CT'
AND exam.date >= #min AND exam.date < #max
;
Note for this variant we probably don't need a left join (but again due to a lack of sample data and expected result that is an assumption).