Speeding up my queries in PHP - php

I'm working on trying to speed up a webpage I have created. I know the issue is that I have a query within a query. I feel like there has to be a quicker way to accomplish the same results, but I'm running out of ideas. (My first attempt at this took 45 seconds for the page to load, now I'm down to about 6)
What I'm trying to do is pull run rate information from tables. I need to pull the correct startup and end of run rates from the runrate table, but all I have to go off of initially is the workcenter ID.
I feel like if the tables were set up a little bit better then it probably would've have been so difficult, but it's what I inherited and as a result I'm a bit stuck. I need to pull a month worth of data from each workcenter (about 15) where there can be as many as 4-5 runs each day... Quite a bit of data to process.
Here's the PHP code:
$qtotalStartup = mysql_query("
SELECT startup.recordID, startup.date, startup.time, runrate.rate AS temRate, runrate.formID
FROM jos_a_inproc_startup startup JOIN jos_a_runrate runrate ON startup.recordID = runrate.recordID
WHERE startup.workcenterId = $id AND runrate.rate > 0 AND runrate.formID = 1 AND startup.date > DATE_SUB(NOW(), INTERVAL 1 MONTH)") or die(mysql_error());
$totalStartCtr = mysql_num_rows($qtotalStartup);
if ($totalStartCtr > 0) {
while($rtotalStartup = mysql_fetch_assoc($qtotalStartup)) {
$hours = 0;
$goalRate = 0;
$sumHrRR = 0;
$startDate = 0;
$startTime = 0;
$startupNum = $rtotalStartup['recordID'];
$goalRate = $rtotalStartup['temRate'];
$startDate = $rtotalStartup['date'];
$startTime = $rtotalStartup['time'];
$startTime = strtotime($startDate . ' ' . $startTime);
//now that we have all of the startup form info, we can move to the end of run information
//this query will retrieve the correct date, time, and ending run rate for us to use with our calculations.
$qtotalEOR = mysql_query("
SELECT eor.recordID AS eorRec, eor.date, eor.time, eor.startupid, runrate1.rate AS tempRate, runrate1.formID
FROM jos_a_inproc_eor eor JOIN jos_a_runrate runrate1 ON eor.recordID = runrate1.recordID
WHERE eor.startupid = $startupNum AND runrate1.rate > 0 AND runrate1.formID = 3") or die(mysql_error());
$totalEORCtr = mysql_num_rows($qtotalEOR);
if ($totalEORCtr > 0) {
while($rtotalEOR = mysql_fetch_assoc($qtotalEOR)) {
//reset the accumulator to 0 so we don't get extra 'bad' data.
$sumHrRR = 0;
$newGoalRate = 0;
$lastestDate = 0;
$latestTime = 0;
$eorNum = $rtotalEOR['eorRec'];
$latestDate = $rtotalEOR['date'];
$latestTime = $rtotalEOR['time'];
$latestTime = strtotime($latestDate . ' ' . $latestTime);
$sumHrRR= $rtotalEOR['tempRate'];
Any ideas would be greatly appreciated. I know it may be difficult to understand what I'm trying to get at without much more information, so let me know if you need to know anything else. Thanks.

Maby try using multiple JOINS like this one:
SELECT startup.recordID, startup.date, startup.time,
runrate.rate AS temRate, runrate.formID
-- stuff from second query
eor.recordID AS eorRec, eor.date AS eor_date,
eor.time AS eor_time, eor.startupid AS eor_startupid,
runrate1.rate AS eor_tempRate,
runrate1.formID AS runrate1_formID
FROM jos_a_inproc_startup startup
JOIN jos_a_runrate runrate ON startup.recordID = runrate.recordID
-- second query LEFT JOIN
LEFT JOIN jos_a_inproc_eor eor
ON eor.startupid = startup.recordID
LEFT JOIN jos_a_runrate runrate1
ON eor.recordID = runrate1.recordID
AND runrate1.rate > 0
AND runrate1.formID = 3
WHERE startup.workcenterId = $id
AND runrate.rate > 0
AND runrate.formID = 1
AND startup.date > DATE_SUB(NOW(), INTERVAL 1 MONTH)
I don't know if I'm right but I think that you are also doing some aggregation work with results inside PHP. You could do it inside database using like sum() or avg() and GROUP BY. You will save some time when transfering smaller result set from database to server and time for looping and aggregating inside PHP. Also most of the time using JOIN is much faster than using queries in loop or even subqueries inside query.
You should also check if indexes are set on columns you search in. Also use EXPLAIN to check how query is executed.

you can use Mem-Cache techniques to make it much faster ,and try to make your queries the simpler that u can .. dont retrieve values that you dont use in your scripts ..

How many records are you typically dealing with as output? How big are the tables? Have you reviewed the indexes? Have you analyzed them recently (rebuilt them)?
Also, are you sending the data back to the browser using deflate? See:
http://httpd.apache.org/docs/2.2/mod/mod_deflate.html

Well, you could try using multiple INNER JOINs (see) and have only one query instead of one query inside a query, which greatly impacts on performance. You could try something like this, and tweaking it a little:
SELECT
startup.recordID AS startupRecordID,
startup.date AS startupDate,
startup.time AS startupTime,
runrate.rate,
runrate.formID,
eor.recordID AS eorRecordID,
eor.date AS eorDate,
eor.time AS eorTime,
eor.startupid AS eorStartupID
FROM jos_a_inproc_startup startup
INNER JOIN jos_a_runrate runrate
ON startup.recordID = runrate.recordID
INNER JOIN jos_a_inproc_eor eor
ON startup.recordID = eor.startupid
WHERE
startup.workcenterId = $id
AND runrate.rate > 0
AND runrate.formID = 1
AND startup.date > DATE_SUB(NOW(), INTERVAL 1 MONTH)

Related

PHP/MYSQL query execution time optimisation

I am having a great deal of difficulty with this set of queries. I cannot find a way to speed up the second query at all. I've tried joining the the first query as a subquery on the second one, running through the first query results one by one and the current setup, all of which have proven extremely slow.
I would like to simply add a limit to the second query instead of this hocky stuff but our mysql version is too old to support it. For some reason it is also treating EXPLAINas a syntax error, which is unhelpful.
How can I reduce the execution time of this?
$limitQuery = $pdo->prepare("
SELECT r.supplier_option_code FROM third_party_raw_stock_price AS r ORDER BY r.id LIMIT 100
");
$limitQuery->execute();
$limitIds = $limitQuery->fetchAll();
$limitIds = implode("', '",array_column($limitIds, 'supplier_option_code'));
$limitQuery = null;
$linkColumn = 'supplier_code';
$thirdPartyId = 'FS';
$migrateQuery = $pdo->prepare("
UPDATE third_party_raw_stock_price AS r
JOIN options_new AS o
ON o.".$linkColumn." = r.supplier_option_code AND r.supplier_prefix = '".$thirdPartyId."'
JOIN third_party_config AS t
ON t.code = '".$thirdPartyId.""'
SET o.price = '989.99', o.cost_price_variation = '3.33', o.stock = '7'
WHERE r.supplier_option_code IN ('$limitIds')
");
$migrateQuery->execute([':config' => $thirdPartyId]);

Pull number of rows from a SQL query and put it in PHP as a variable?

This is 4 queries put into one. This is really old code and once I can make this work we can update it later to PDO for security. What I am trying to do is count rows from
select count(*) from dialogue_employees d_e,
dialogue_leaders d_l where
d_l.leader_group_id = d_e.leader_group_id
and use it in a formula where I also count how many rows from dialogue.status = 1.
The formula is on the bottom to create a percentage total from the results. This is PHP and MySQL and I wasn't sure the best way to count the rows and put them as a variable in php to be used in the formula on the bottom?
function calculate_site_score($start_date, $end_date, $status){
while($rows=mysql_fetch_array($sqls)){
$query = "
SELECT
dialogue.cycle_id,
$completecount = sum(dialogue.status) AS calculation,
$total_employees = count(dialogue_employees AND dialogue_leaders), dialogue_list.*,
FROM dialogue,
(SELECT * FROM dialogue_list WHERE status =1) AS status,
dialogue_employees d_e,
u.fname, u.lname, d_e.*
user u,
dialogue_list,
dialogue_leaders d_l
LEFT JOIN dialogue_list d_list
ON d_e.employee_id = d_list.employee_id,
WHERE
d_l.leader_group_id = d_e.leader_group_id
AND d_l.cycle_id = dialogue.cycle_id
AND u.userID = d_e.employee_id
AND dialogue_list.employee_id
AND site_id='$_SESSION[siteID]'
AND start_date >= '$start_date'
AND start_date <= '$end_date'";
$sqls=mysql_query($query) or die(mysql_error());
}
$sitescore=($completecount/$total_employees)*100;
return round($sitescore,2);
}
If you separate out your queries you will gain more control over your data. You have to be careful what your counting. It's pretty crowded in there.
If you just wanted to clean up your function you can stack your queries like this so they make more sense, that function is very crowded.
function calculate_site_score($start_date, $end_date, $status){
$query="select * from dialogue;";
if ($result = $mysqli->query($query))) {
//iterate your result
$neededElem = $result['elem'];
$query="select * from dialogue_list where status =1 and otherElem = " . $neededElem . ";";
//give it a name other than $sqls, something that makes sense.
$list = $mysqli->query($query);
//iterate list, and parse results for what you need
foreach($list as $k => $v){
//go a level deeper, or calculate, rinse and repeat
}
}
Then do your counts separately.
So it would help if you separate queries each on their own.
Here is a count example How do I count columns of a table

Have PHP count rows with a query that is already using count(*)?

I have a subquery and I want to count the rows in PHP for MySQL. I am trying to fix old code and know that PDO is better and more secure and we will eventually rewrite all this code, but for now I need to just make it work. My problem is figuring out the command for the $total_employees to count the rows. This number will be used in a formula later. Is there a way to do it as 2 subqueries or rewriting it in the SQL statement other than just using php and mysql_fetch_row? I am trying to avoid multiple while loops. This is condensed from a bigger query for easier viewing.
while($rows=mysql_fetch_array($sqls)){
$cycle_id = $rows[cycle_id];
$sqls=("select subb.sqlcal AS sqlcalemp from
(select count(*) as sqlcal from dialogue_employees d_e,
dialogue_leaders d_l where
d_l.leader_group_id = d_e.leader_group_id and
d_l.cycle_id = $cycle_id) as subb");
$total_employees += $rows[sqlcalsemp];
This was the older code that worked before trying to update it:
while($rows=mysql_fetch_array($sqls)){
$cycle_id = $rows[cycle_id];
$sqlcalcemp=mysql_query("select count(*) from dialogue_employees d_e,
dialogue_leaders d_l where
d_l.leader_group_id = d_e.leader_group_id and
d_l.cycle_id = $cycle_id") or die(mysql_error());
$rowtotal = mysql_fetch_row($sqlcalcemp);
$total_employees += $rowtotal[0];
your looping through and looking at each cycle_id...
maybe try something like this to grab all counts for each cycle_id at once
select SUM(d_e.leader_group_id IS NOT NULL) as sqlcalemp,d_l.cycle_id
from dialogue_leaders d_l
left join dialogue_employees d_e
on d_l.leader_group_id = d_e.leader_group_id
group by d_l.cycle_id
http://sqlfiddle.com/#!9/4995b/4

More efficient way to do SQL queries

I've been using the below php and sql for loading schedule information and real time information for passenger trains in the UK. Essentially you have to find the relevant schedules, and then load the realtime information for each schedule which is in a different table relating to todays trains.
The query is taking a little longer than is really idea and using lots of CPU% which again isn''t ideal. I'm pretty weak when it comes to sql programming so any pointers as to what is inefficient would be great.
This is for an android app and so i've tried to all with one call over http. The prints(*) and > is for splitting the string at the other end.
Here is the code:
<?
//Connect to the database
mysql_connect("localhost","XXXX","XXXX")
or die ("No connection could be made to the OpenRail Database");
mysql_select_db("autotrain");
//Set todays date from system and get HTTP parameters for the station,time to find trains and todays locations table.
$date = date('Y-m-d');
$test = $_GET['station'];
$time = $_GET['time'];
$table = $_GET['table'];
//Find the tiploc associated with the station being searched.
$tiplocQuery = "SELECT tiploc_code FROM allstations WHERE c LIKE '$test';";
$tiplocResult =mysql_query($tiplocQuery);
$tiplocRow = mysql_fetch_assoc($tiplocResult);
$tiploc=$tiplocRow['tiploc_code'];
//Now find the timetabled trains for the station where there exists no departure information. Goes back two hours to account for any late running.
$timeTableQuery = "SELECT tiplocs.tps_description AS 'C', locations$table.public_departure, locations$table.id,schedules.stp_indicator
,schedules.train_uid
FROM locations$table, tiplocs, schedules_cache, schedules,activations
WHERE locations$table.id = schedules_cache.id
AND schedules_cache.id = schedules.id
AND schedules.id =activations.id
AND '$date'
BETWEEN schedules.date_from
AND schedules.date_to
AND locations$table.tiploc_code = '$tiploc'
AND locations$table.real_departure LIKE '0'
AND locations$table.public_departure NOT LIKE '0'
AND locations$table.public_departure >='$time'-300
AND locations$table.public_departure <='$time'+300
AND schedules.runs_th LIKE '1'
AND schedules_cache.destination = tiplocs.tiploc
ORDER BY locations$table.public_departure ASC
LIMIT 0,30;";
$timeTableResult=mysql_query($timeTableQuery);
while($timeTablerow = mysql_fetch_assoc($timeTableResult)){
$output[] = $timeTablerow;
}
//Now for each id returned in the timetable, get the locations and departure times so the app may calculate expected arrival times.
foreach ($output as $value) {
$id = $value['id'];
$realTimeQuery ="SELECT locations$table.id,locations$table.location_order,locations$table.arrival,locations$table.public_arrival,
locations$table.real_arrival,locations$table.pass,locations$table.departure,locations$ table.public_departure,locations$table.real_departure,locations$table.location_cancelled,
tiplocs.tps_description FROM locations$table,tiplocs WHERE id =$id AND locations$table.tiploc_code=tiplocs.tiploc;";
$realTimeResult =mysql_query($realTimeQuery);
while($row3 = mysql_fetch_assoc($realTimeResult)){
$output3[] = $row3;
}
print json_encode($output3);
print("*");
unset($output3);
unset($id);
}
print('>');
print json_encode($output);
?>
Many Thanks
Matt
The biggest issue with your setup is this foreach loop because it is unnecessary and results in n number of round trips to the database to execute a query, fetch and analyze the results.
foreach ($output as $value) {
Rewrite the initial query to include all of the fields you will need to do your later calculations.
Something like this would work.
SELECT tl.tps_description AS 'C', lc.public_departure, lc.id, s.stp_indicator, s.train_uid,
lc.id, lc.location_order, lc.arrival, lc.public_arrival, lc.real_arrival, lc.pass, lc.departure, lc.real_departure, lc.location_cancelled
FROM locations$table lc INNER JOIN schedules_cache sc ON lc.id = sc.id
INNER JOIN schedules s ON s.id = sc.id
INNER JOIN activations a ON s.id = a.id
INNER JOIN tiplocs tl ON sc.destination = tl.tiploc
WHERE '$date' BETWEEN schedules.date_from AND schedules.date_to
AND lc.tiploc_code = '$tiploc'
AND lc.real_departure LIKE '0'
AND lc.public_departure NOT LIKE '0'
AND lc.public_departure >='$time'-300
AND lc.public_departure <='$time'+300
AND s.runs_th LIKE '1'
ORDER BY lc.public_departure ASC
LIMIT 0,30;
Eliminating n query executions from your page load should dramatically increase response time.
Ignoring the problems with the code, in order to speed up your query, use the EXPLAIN command to evaluate where you need to add indexes to your query.
At a guess, you probably will want to create an index on whatever locations$table.public_departure evaluates to.
http://dev.mysql.com/doc/refman/5.0/en/using-explain.html
A few things I noticed.
First, you are joining tables in the where clause, like this
from table1, table2
where table1.something - table2.something
Joining in the from clause is faster
from table1 join table2 on table1.something - table2.something
Next, I'm not a php programmer, but it looks like you are running similar queries inside a loop. If that's true, look for a way to run just one query.
Edit starts here
This is in response to gazarsgo's that I back up by claim about joins in the where clause being faster. He is right, I was wrong. This is what I did. The programming language is ColdFusion:
<cfsetting showdebugoutput="no">
<cfscript>
fromtimes = ArrayNew(1);
wheretimes = ArrayNew(1);
</cfscript>
<cfloop from="1" to="1000" index="idx">
<cfquery datasource="burns" name="fromclause" result="fromresult">
select count(distinct hscnumber)
from burns_patient p join burns_case c on p.patientid = c.patientid
</cfquery>
<cfset ArrayAppend(fromtimes, fromresult.executiontime)>
<cfquery datasource="burns" name="whereclause" result="whereresult">
select count(distinct hscnumber)
from burns_patient p, burns_case c
where p.patientid = c.patientid
</cfquery>
<cfset ArrayAppend(wheretimes, whereresult.executiontime)>
</cfloop>
<cfdump var="#ArrayAvg(fromtimes)#" metainfo="no" label="from">
<cfdump var="#ArrayAvg(wheretimes)#" metainfo="no" label="where">
I did ran it 5 times. The results, in milliseconds, follow.
9.563 9.611
9.498 9.584
9.625 9.548
9.831 9.769
9.792 9.813
The first number represents joining in the from clause, the second joining in the where clause. The first number is lower only 60% of the time. Had it been lower 100% percent of the time, it would have shown that joining in the from clause is faster, but that' not the case.

while (mysql_fetch_array) in a while loop

i have this code:
while ($sum<16 || $sum>18){
$totala = 0;
$totalb = 0;
$totalc = 0;
$ranka = mysql_query("SELECT duration FROM table WHERE rank=1 ORDER BY rand() LIMIT 1");
$rankb = mysql_query("SELECT duration FROM table WHERE rank=2 ORDER BY rand() LIMIT 1");
$rankc = mysql_query("SELECT duration FROM table WHERE rank=3 ORDER BY rand() LIMIT 1");
while ($rowa = mysql_fetch_array($ranka)) {
echo $rowa['duration'] . "<br/>";
$totala = $totala + $rowa['duration'];
}
while ($rowb = mysql_fetch_array($rankb)) {
$totalb = $totalb + $rowb['duration'];
}
while ($rowc = mysql_fetch_array($rankc)) {
$totalc = $totalc + $rowc['duration'];
}
$sum=$totala+$totalb+$totalc;
}
echo $sum;
It works fine, But the problem is until "$sum=16" the "echo $rowa['duration']" executes, the question is, is there a away to "echo" only the latest executed code in the "while ($rowa = mysql_fetch_array($ranka))" i this while loop?
Because most of the times returns all the numbers until the "$sum=16"
You are explicitly echoing the $rowa['duration'] in the first inner while loop. If you only want to print the last duration from the $ranka set, simple change the echo to $rowa_duration = $rowa['duration'] then echo it outside the loop.
while ($rowa = mysql_fetch_array($ranka)) {
$rowa_duration = $rowa['duration'];
$totala = $totala + $rowa['duration'];
}
echo $rowa_duration . '<br/>';
What you are doing there is bad on multiple levels. And your english horrid. Well .. practice makes perfect. You could try joining ##php chat room on FreeNode server. That would improve both your english and php skills .. it sure helped me a lot. Anyway ..
The SQL
First of all, to use ORDER BY RAND() is extremely ignorant (at best). As your tables begin the get larger, this operation will make your queries slower. It has n * log2(n) complexity, which means that selecting querying table with 1000 entries will take ~3000 times longer then querying table with 10 entries.
To learn more about it , you should read this blog post, but as for your current queries , the solution would look like:
SELECT duration
FROM table
JOIN (SELECT CEIL(RAND()*(SELECT MAX(id) FROM table)) AS id) as choice
WHERE
table.id >= choice.id
rank = 1
LIMIT 1
This would select random duration from the table.
But since you you are actually selecting data with 3 different ranks ( 1, 2 and 3 ), it would make sense to create a UNION of three queries :
SELECT duration
FROM table
JOIN (SELECT CEIL(RAND()*(SELECT MAX(id) FROM table)) AS id) as choice
WHERE
table.id >= choice.id
rank = 1
LIMIT 1
UNION ALL
SELECT duration
FROM table
JOIN (SELECT CEIL(RAND()*(SELECT MAX(id) FROM table)) AS id) as choice
WHERE
table.id >= choice.id
rank = 2
LIMIT 1
UNION ALL
SELECT duration
FROM table
JOIN (SELECT CEIL(RAND()*(SELECT MAX(id) FROM table)) AS id) as choice
WHERE
table.id >= choice.id
rank = 3
LIMIT 1
Look scary, but it actually will be faster then what you are currently using, and the result will be three entries from duration column.
PHP with SQL
You are still using the old mysql_* functions to access database. This form of API is more then 10 years old and should not be used, when writing new code. The old functions are not maintained (fixed and/or improved ) anymore and even community has begun the process of deprecating said functions.
Instead you should be using either PDO or MySQLi. Which one to use depends on your personal preferences and what is actually available to you. I prefer PDO (because of named parameters and support for other RDBMS), but that's somewhat subjective choice.
Other issue with you php/mysql code is that you seem to pointlessly loop thought items. Your queries have LIMIT 1, which means that there will be only one row. No point in making a loop.
There is potential for endless loop if maximum value for duration is 1. At the start of loop you will have $sum === 15 which fits the first while condition. And at the end that loop you can have $sum === 18 , which satisfies the second loop condition ... and then it is off to the infinity and your SQL server chokes.
And if you are using fractions for duration, then the total value of 3 new results needs to be even smaller. Just over 2. Start with 15.99 , ends with 18.01 (that's additional 2.02 in duration or less the 0.7 per each). Again .. endless loop.
Suggestion
Here is how i would do it:
$pdo = new PDO('mysql:dbname=my_db;host=localhost', 'username', 'password');
$pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
$sum = 0;
while ( $sum < 16 )
{
$query = 'that LARGE query above';
$statement = $pdo->prepare( $query );
if ( $statement->execute() )
{
$data = $statement->fetchAll( PDO::FETCH_ASSOC );
$sum += $data[0]['duration']+$data[1]['duration']+$data[2]['duration'];
}
}
echo $data[0]['duration'];
This should do what your code did .. or at least, what i assume, was your intentions.

Categories