I'm sure there is a better way I could be doing this. I have three main tables with large amounts of data to run through:
records_main, sales, and appointments. Each with close to 20,000 records.
I need to join these three tables as well as a few others that arn't two large.
$SQL = "SELECT DISTINCT appointments.id AS aid, appointments.date, appointments.estimate_price, appointments.next_action, appointments.next_action_date, appointments.result, appointments.result_type, appointments.notes,
customer.id AS cid, customer.homeowner1_fname, customer.homeowner1_lname, customer.address, customer.city, customer.state, customer.zipcode, customer.phone1, customer.phone1_type, customer.phone2, customer.phone2_type, customer.phone3, customer.phone3_type, customer.phone4, customer.phone4_type, customer.phone5, customer.phone5_type, customer.lead_source, customer.lead_category, customer.primary_interest, customer.secondary_interest, customer.additional_interest1, customer.additional_interest2,
originator.employee_id AS originator_employee_id,originator.fname AS originator_fname,originator.lname AS originator_lname,
setter.employee_id AS setter_employee_id,setter.fname AS setter_fname,setter.lname AS setter_lname,
resetter.employee_id AS resetter_employee_id, resetter.fname AS resetter_fname, resetter.lname AS resetter_lname,
salesrep.employee_id AS salesrep_employee_id, salesrep.fname AS salesrep_fname, salesrep.lname AS salesrep_lname,
salesrep2.employee_id AS salesrep2_employee_id, salesrep2.fname AS salesrep2_fname, salesrep2.lname AS salesrep2_lname
FROM
core_records_appointments as appointments
INNER JOIN core_records_main as customer ON appointments.customer = customer.id
LEFT JOIN core_employees_main as originator ON appointments.originator = originator.id
LEFT JOIN core_employees_main as setter ON appointments.setter = setter.id
LEFT JOIN core_employees_main as resetter ON appointments.resetter = resetter.id
LEFT JOIN core_employees_main as salesrep ON appointments.sales_representative = salesrep.id
LEFT JOIN core_employees_main as salesrep2 ON appointments.sales_representative2 = salesrep2.id
";
This takes a few seconds but finally does load. The last join I put though seems to break the query together:
LEFT JOIN core_records_sales as sales ON appointments.day = sales.day_sold AND appointments.customer = sales.customer
After this I have a limit set and a group by
Anything I can do to improve this? I am using this with jqgrid, not sure if that helps
As a first approach, try to execute the query on the database using the EXPLAIN syntax. This will give you an analysis of the statement and tells you, if it uses indexes at all. If it doesn't use indexes or not enough indexes, try adding them to the tables.
Have you tried giving index on all the columns you are LEFT JOINing on?
TRY giving index to all these if they are not already declared as index
Related
1. Summarise the problem
Our company has been using this system for a long time and people feel huge system slowdown. I can't rewrite code because its super hardcoded and too many files to do so.
Few months ago we saw that one query takes about 170 s to execute.
My workaround was add LIMITS to each query and from 170s it went down to 2-3s. But last week i saw that code now takes about 5-25 sec.
Also this system is on php 5.2. Its basically impossible to upgrade code to work for 5.6
3 people was rewriting this code and after 8 hour we managed to make to login page.
Indexing is limited because most of database info have strings rather than integer.
2. Provide background including what you've already tried
What i tried:
- Adding limits by default 300, also users can change this limit.
- Migrating server to better hardware.
- Giving more RAM (10gb now) and two virtual CPU.
- Migrating from local storage to network (better performance at this moment).
3. Show some code
SELECT pastabos_apskaitininkui,
uzdaviniai.pakr_salis,
uzdaviniai.iskr_salis,
uzdaviniai.id AS uzdavinio_id,
salys.salis AS muitines_salis,
uzdaviniai.pakr_regionas,
uzdaviniai.iskr_regionas,
uzdaviniai.pakr_miestas,
uzdaviniai.iskr_miestas,
uzdavinio_priekaba,
priekabos.tipas AS priekabos_tipas,
uzdaviniai.kas_sukure_skyrius,
uzdaviniai.kam_sukure_skyrius,
uzdaviniai.kas_sukure_vadyb,
uzdaviniai.kam_sukure_vadyb,
uzdav_users.vardas_sutr AS uzdav_user_vardas,
pakrov_salis.sutrumpinimas AS uzdav_pakr_salis,
iskrov_salis.sutrumpinimas AS uzdav_iskr_salis,
uzdaviniai.pasikrovimo_data AS uzd_pasikrovimo_data,
uzdaviniai.pristatymo_data AS uzd_pristatymo_data,
uzdaviniai.pastabos,
suma_vykdytojui_valiuta,
vykdytojui_valiuta,
uzdaviniai.eil_nr,
uzdaviniai.statusas,
uzdaviniai.statusas_baigtas_ranka,
keliones_lapas.reiso_nr,
keliones_lapas.masina,
keliones_lapas.masinos_id,
keliones_lapas.priekabos_nr,
keliones_lapas.priekabos_id,
keliones_lapas.priekabos2_nr,
keliones_lapas.reiso_pavadinimas,
keliones_lapas.vykdytojo_tipas,
keliones_lapas.vykdytojo_imone,
keliones_lapas.atvykimo_data,
keliones_lapas.vairuotojas,
keliones_lapas.vairuotojas2,
keliones_lapas.vairuotojas3,
keliones_lapas.vairuotojas4,
keliones_lapas.marsrutas,
klientai.firmos_pavadinimas AS uzsakovas,
uzsakovo_uzs_nr,
valiutos.valiuta AS frachto_valiuta,
uzsakymas.vadybininkas,
uzsakymas.skyrius,
uzsakymas.frachtas_suma,
uzsakymas.pavadinimas,
uzsakymas.uzsakymo_id,
uzsakymas.reg_data,
uzsakymas.pastabos AS uzs_pastabos,
uzsakymas.statusas AS uzsakymo_statusas,
sask_ist.serija,
sask_ist.sask_nr,
sask_ist.id AS sask_id,
sask_ist.suma,
sask_ist.valiuta,
sask_ist.kursas AS sask_ist_kursas,
sask_ist.apmoketi_iki,
sask_ist.atidejimas,
issiuntimo_data,
sask_ist_apmokejimas.suma AS ist_apm_suma,
sask_ist_apmokejimas.valiuta AS ist_apm_valiuta,
sask_ist_apmokejimas.kursas AS ist_apm_kursas,
sask_ist_apmokejimas.id AS ist_apm_id,
klientu_pakr_vietos.pavadinimas AS muitines_pavadinimas,
klientu_pakr_vietos.miestas AS muitines_miestas,
klientu_pakr_vietos.adresas AS muitines_adresas,
kokio_krovinio_dalis,
krovinys_konsoliduotas,
kroviniu_tipai.pavadinimas AS krovinio_pavadinimas,
kroviniai.pasikrovimo_data,
kroviniai.pristatymo_data,
kroviniai.uzsakovo_krovinio_nr,
kroviniai.id AS krovinio_id,
svoris_t,
ldm,
turis,
paleciu_skaicius,
vnt,
temperature,
kroviniai.konteinerio_nr,
kroviniai.pakr_vietos_id AS kr_pakr_vietos_id,
kroviniai.iskr_vietos_id AS kr_iskr_vietos_id,
kroviniai.pakr_salis AS kr_pakr_salis,
kroviniai.iskr_salis AS kr_iskr_salis,
kroviniai.pakr_miestas AS kr_pakr_miestas,
kroviniai.pakr_regionas AS kr_pakr_regionas,
kroviniai.iskr_regionas AS kr_iskr_regionas,
kroviniai.iskr_miestas AS kr_iskr_miestas,
kroviniai.ismuitinimo_data AS ismuitinimo_data,
kroviniai.uzmuitinimo_data AS uzmuitinimo_data,
kroviniai.pasienio_postas AS pasienio_postas,
kroviniai.ismuitinimo_vieta AS ismuitinimo_vieta,
statusai.pavadinimas AS st_pavadinimas,
st_uzsakymai.id AS st_id,
st_salys.salis AS st_salis,
st_uzsakymai.data AS st_data,
st_uzsakymai.laikas AS st_laikas,
st_uzsakymai.ivykio_pastabos_sutr AS st_pastabos_sutr,
st_uzsakymai.ivykio_pastabos AS st_pastabos,
st_keliones_lapas.reiso_pavadinimas AS st_reiso_pav,
vairuotojai.vardas AS sq_vardas,
priekabos.masinos_nr AS sq_priekaba
FROM uzdaviniai
LEFT JOIN priekabos
ON priekabos.id = uzdaviniai.uzdavinio_priekaba
LEFT JOIN users AS uzdav_users
ON uzdav_users.id = uzdaviniai.kam_sukure_vadyb
LEFT JOIN salys AS pakrov_salis
ON pakrov_salis.id = uzdaviniai.pakr_salis
LEFT JOIN salys AS iskrov_salis
ON iskrov_salis.id = uzdaviniai.iskr_salis
LEFT JOIN uzsakymas
ON uzdaviniai.uzsakymo_id = uzsakymas.uzsakymo_id
LEFT JOIN uzd_kur_vykdytas
ON uzdaviniai.id = uzd_kur_vykdytas.uzdavinio_id
LEFT JOIN keliones_lapas
ON keliones_lapas.reiso_nr = uzd_kur_vykdytas.reiso_nr
LEFT JOIN st_uzsakymai
ON st_uzsakymai.uzdavinio_id = uzdaviniai.id
LEFT JOIN statusai
ON statusai.id = st_uzsakymai.statuso_id
LEFT JOIN salys AS st_salys
ON st_salys.id = st_uzsakymai.salis
LEFT JOIN keliones_lapas AS st_keliones_lapas
ON st_keliones_lapas.reiso_nr = st_uzsakymai.reiso_nr
LEFT JOIN vairuotojai
ON vairuotojai.id = keliones_lapas.vairuotojas
LEFT JOIN sask_ist_uz_ka
ON sask_ist_uz_ka.uzsakymo_id = uzsakymas.uzsakymo_id
LEFT JOIN sask_ist
ON sask_ist.id = sask_ist_uz_ka.sask_id
LEFT JOIN sask_ist_apmokejimas
ON sask_ist.id = sask_ist_apmokejimas.sask_id
LEFT JOIN uzsakymo_kroviniai
ON uzsakymas.uzsakymo_id = uzsakymo_kroviniai.uzsakymo_id
LEFT JOIN kroviniai
ON kroviniai.id = uzsakymo_kroviniai.krovinio_id
LEFT JOIN klientu_pakr_vietos
ON klientu_pakr_vietos.id = kroviniai.ismuitinimo_vieta
LEFT JOIN klientu_pakr_vietos AS iskr_vietos_uzd
ON iskr_vietos_uzd.id = uzdaviniai.pakr_vietos_id
LEFT JOIN salys
ON salys.id = klientu_pakr_vietos.salis
LEFT JOIN cmr_kroviniai
ON kroviniai.id = cmr_kroviniai.krovinio_id
LEFT JOIN cmr
ON cmr.id = cmr_kroviniai.cmr_id
LEFT JOIN cmr_panaudojimas
ON cmr_panaudojimas.cmr_id = cmr_kroviniai.cmr_id
AND keliones_lapas.reiso_nr = cmr_panaudojimas.reiso_nr
LEFT JOIN kroviniu_tipai
ON kroviniu_tipai.id = kroviniai.krovinio_pavadinimas
LEFT JOIN klientai
ON klientai.id = uzsakymas.kliento_id
LEFT JOIN valiutos
ON valiutos.id = uzsakymas.frachtas_valiuta
WHERE 1 = 1
AND ( `uzdaviniai`.`pasikrovimo_data` BETWEEN
'2019-07-08' AND '2019-07-08 23:59:59'
)
ORDER BY uzdaviniai.pasikrovimo_data DESC
LIMIT 300
As you can see there is many left join, statements.
4. Result
What i want is some help what can i do with this piece of crap or some recommendation what can i change. As i expect these query will take more and more time to execute.
If the query is fast when executed directly in mysql console, then the query has no problem at all, and what is slow is the php code that uses the results of the query, php code that you don't show and that you should revise/post here.
If the query is slow even in mysql console, then the problem can be the size of the master/joined tables (but this may be resolved by relaxing the memory limits of the mysql server and/or adding ram to the server), or could be non correct indexing of the joined tables, since any LEFT JOIN needs the joined table to be indexed on the field you use to join it (and this could be resolved by creating the correct indexes on the joined tables).
Consider changing your
WHERE 1 = 1
AND ( `uzdaviniai`.`pasikrovimo_data` BETWEEN
'2019-07-08' AND '2019-07-08 23:59:59'
)
TO
WHERE uzdaviniai.pasikrovimo_data BETWEEN
'2019-07-08' AND '2019-07-08 23:59:59'
and let us know results (with the backticks), please.
I have a dhtmlx-grid created from multiple tables. And I am able to edit the main table but not the joined ones.
The dhtmlx guides are rather vague on creating a grid from multiple tables and making it fully editable.
I followed the steps here:
http://docs.dhtmlx.com/connector__php__complex_queries.html#complexqueries
http://docs.dhtmlx.com/connector__php__basis.html#loadingfromdatabase
and tried to create a connection, but failed.
I'm using their php-connector and the code that I tried is as follows:
$jpnx_regs = new GridConnector($conn,'MySQL'); // connector initialization
$jpnx_regs->sql->attach("Update","
UPDATE wp_japanextion_attendance_data
LEFT JOIN wp_buyer_employee
ON wp_buyer_employee.ID = wp_japanextion_attendance_data.ID_employee
LEFT JOIN wp_buyer_company
ON wp_buyer_company.ID = wp_buyer_employee.ID_company
SET
reg_user='{reg_user}',
reservation_date='{reservation_date}',
reservation_hour='{reservation_hour}',
copy_company_name='{copy_company_name}',
copy_company_TEL='{copy_company_TEL}',
copy_name='{copy_name}',
copy_mobile_TEL='{copy_mobile_TEL}',
copy_email='{copy_email}',
rec_creation_date='{rec_creation_date}',
iploc='{iploc}',
trackvia_inserted='{trackvia_inserted}'
WHERE id='{id}'
");
$jpnx_regs->render_complex_sql("
SELECT
*, wp_japanextion_attendance_data.ID AS jpnx_ID,
wp_buyer_employee.ID AS employee_ID,
wp_buyer_company.ID AS buyer_ID,
FROM
wp_japanextion_attendance_data
LEFT JOIN wp_buyer_employee
ON wp_buyer_employee.ID = wp_japanextion_attendance_data.ID_employee
LEFT JOIN wp_buyer_company
ON wp_buyer_company.ID = wp_buyer_employee.ID_company
WHERE
reservation_date LIKE '%November%' OR
reservation_date LIKE '%December%' OR
reservation_date LIKE '%January%'
", "jpnx_ID","
reg_user,reservation_date,reservation_hour,company_name,company_TEL,name,mobile_TEL,email,rec_creation_date,iploc,trackvia_inserted,ID
");
↑ Just as their guide says. I first initialize the grid like this:
$jpnx_regs = new GridConnector($conn,'MySQL');
Then I attach the update action:
$jpnx_regs->sql->attach("Update"," ... sql code here
Then I render the grid with the render_complex_sql.
$jpnx_regs->render_complex_sql("
BTW, just in case, here down below is my working code for my grid from multiple tables but I can only update the main table:
$jpnx_regs = new GridConnector($conn,'MySQL');
if ($jpnx_regs->is_select_mode())
$jpnx_regs->render_sql("
SELECT
*, wp_japanextion_attendance_data.ID AS jpnx_ID,
wp_buyer_employee.ID AS employee_ID,
wp_buyer_company.ID AS buyer_ID,
FROM
wp_japanextion_attendance_data
LEFT JOIN wp_buyer_employee
ON wp_buyer_employee.ID = wp_japanextion_attendance_data.ID_employee
LEFT JOIN wp_buyer_company
ON wp_buyer_company.ID = wp_buyer_employee.ID_company
", "jpnx_ID","reg_user,reservation_date,reservation_hour,company_name,company_TEL,name,mobile_TEL,email,rec_creation_date,iploc,trackvia_inserted,ID
");
else //code for other operations - i.e. update/insert/delete
$jpnx_regs->render_table("wp_japanextion_attendance_data","id","
reg_user,reservation_date,reservation_hour,copy_company_name,copy_company_TEL,copy_name,copy_mobile_TEL,copy_email,rec_creation_date,iploc,trackvia_inserted");
I am trying to return all the results from table one, AKA ship_skill_tree, while matching up the rows found in table two, AKA character_sheet_skills, even if the rows do not exist in table two.
SELECT c.`level` , t.`skillLevel` AS levelNeeded, i.`typeName`
FROM `ship_skill_tree` t
LEFT JOIN `character_sheet_skills` c ON t.`skillTypeID` = c.`typeID`
LEFT JOIN `invTypes` i ON i.`typeID` = t.`skillTypeID`
WHERE t.`shipTypeID` = 11176 AND c.`character_id` = 1;
Table One Data:
|shipTypeID|shipGroupID|skillTypeID|skillLevel
______________________________________________
|11011|26|3332|1
|11129|31|3327|1
|11132|31|3327|1
|11134|31|3327|1
|11172|830|3328|5
|11172|830|12093|1
|11174|893|3328|5
|11174|893|28615|1
|11176|831|3330|5
|11176|831|12092|1
Table Two Data:
|character_id|typeID|skillpoints|level|published
______________________________________________
|1|3300|1415|2|1
|1|3301|8000|3|1
|1|3327|256000|5|1
|1|3330|2829|2|1
|1|3340|181020|4|1
|1|3341|1024000|5|1
|1|3342|32000|3|1
|1|3343|32202|3|1
|1|3380|256000|5|1
|1|3385|256000|5|1
|1|3386|256000|5|1
|1|3392|256000|5|1
|1|3394|90514|4|1
|1|3402|256000|5|1
|1|3410|768000|5|1
|1|3411|135765|4|1
|1|3412|750|1|1
|1|3413|256000|5|1
|1|3416|45255|4|1
|1|3417|0|0|1
|1|3418|0|0|1
|1|3419|135765|4|1
|1|3420|181020|4|1
|1|3423|0|0|1
|1|3425|90510|4|1
|1|3426|45255|4|1
|1|3428|500|1|1
|1|3429|8000|3|1
|1|3436|45255|4|1
|1|3437|45255|4|1
|1|3438|500|1|1
|1|3449|256000|5|1
|1|3453|0|0|1
|1|3455|256000|5|1
|1|3456|226275|4|1
|1|11579|271530|4|1
|1|12186|0|0|1
|1|12187|0|0|1
|1|12188|0|0|1
|1|12190|22547|3|1
|1|12191|45255|4|1
|1|12192|45255|4|1
|1|12193|45255|4|1
|1|12195|45255|4|1
|1|16281|256000|5|1
|1|17940|1024000|5|1
|1|20342|1280000|5|1
|1|22551|40000|3|1
|1|22578|181020|4|1
|1|25739|0|0|1
|1|26252|16000|3|1
|1|26253|750|1|1
|1|26261|750|1|1
|1|32918|16000|3|1
invTypes table:
|typeID|typeName
________________
|3327|Spaceship Command
|3328|Gallente Frigate
|3330|Caldari Frigate
|3332|Gallente Cruiser
|12092|Interceptors
|12093|Covert Ops
|28615|Electronic Attack Ships
In the above query shipTypeID will always, or should always, be valid and match a record in table one, however, in table two, the rows that match may not exist. What I need is to output as follows:
|level|levelNeeded|typeName
___________________________
|2|5|Caldari Frigate
|NULL|1|Interceptors
Currently this is what is returned:
|level|levelNeeded|typeName
___________________________
|2|5|Caldari Frigate
EDIT: Solution!
SELECT c.`level` , t.`skillLevel` AS levelNeeded, i.`typeName`
FROM `ship_skill_tree` t
LEFT JOIN `character_sheet_skills` c ON t.`skillTypeID` = c.`typeID` AND c.`character_id` = 1
INNER JOIN `invTypes` i ON i.`typeID` = t.`skillTypeID`
WHERE t.`shipTypeID` = 11176
You need to put any restrictions on the table being joined in the ON clause. If you put them in the WHERE clause it doesn't work, because the rows that don't have any matches will produce NULL for those columns, and the WHERE clause will filter them out.
SELECT c.`level` , t.`skillLevel` AS levelNeeded, i.`typeName`
FROM `ship_skill_tree` t
LEFT JOIN `character_sheet_skills` c ON t.`skillTypeID` = c.`typeID` AND c.`character_id` = 1
LEFT JOIN `invTypes` i ON i.`typeID` = t.`skillTypeID`
WHERE t.`shipTypeID` = 11176
DEMO
You need to use a right join or an outer join rather than a left join. Have a look through the Visual Representation of SQL Joins for a good overview
I've been using the below php and sql for loading schedule information and real time information for passenger trains in the UK. Essentially you have to find the relevant schedules, and then load the realtime information for each schedule which is in a different table relating to todays trains.
The query is taking a little longer than is really idea and using lots of CPU% which again isn''t ideal. I'm pretty weak when it comes to sql programming so any pointers as to what is inefficient would be great.
This is for an android app and so i've tried to all with one call over http. The prints(*) and > is for splitting the string at the other end.
Here is the code:
<?
//Connect to the database
mysql_connect("localhost","XXXX","XXXX")
or die ("No connection could be made to the OpenRail Database");
mysql_select_db("autotrain");
//Set todays date from system and get HTTP parameters for the station,time to find trains and todays locations table.
$date = date('Y-m-d');
$test = $_GET['station'];
$time = $_GET['time'];
$table = $_GET['table'];
//Find the tiploc associated with the station being searched.
$tiplocQuery = "SELECT tiploc_code FROM allstations WHERE c LIKE '$test';";
$tiplocResult =mysql_query($tiplocQuery);
$tiplocRow = mysql_fetch_assoc($tiplocResult);
$tiploc=$tiplocRow['tiploc_code'];
//Now find the timetabled trains for the station where there exists no departure information. Goes back two hours to account for any late running.
$timeTableQuery = "SELECT tiplocs.tps_description AS 'C', locations$table.public_departure, locations$table.id,schedules.stp_indicator
,schedules.train_uid
FROM locations$table, tiplocs, schedules_cache, schedules,activations
WHERE locations$table.id = schedules_cache.id
AND schedules_cache.id = schedules.id
AND schedules.id =activations.id
AND '$date'
BETWEEN schedules.date_from
AND schedules.date_to
AND locations$table.tiploc_code = '$tiploc'
AND locations$table.real_departure LIKE '0'
AND locations$table.public_departure NOT LIKE '0'
AND locations$table.public_departure >='$time'-300
AND locations$table.public_departure <='$time'+300
AND schedules.runs_th LIKE '1'
AND schedules_cache.destination = tiplocs.tiploc
ORDER BY locations$table.public_departure ASC
LIMIT 0,30;";
$timeTableResult=mysql_query($timeTableQuery);
while($timeTablerow = mysql_fetch_assoc($timeTableResult)){
$output[] = $timeTablerow;
}
//Now for each id returned in the timetable, get the locations and departure times so the app may calculate expected arrival times.
foreach ($output as $value) {
$id = $value['id'];
$realTimeQuery ="SELECT locations$table.id,locations$table.location_order,locations$table.arrival,locations$table.public_arrival,
locations$table.real_arrival,locations$table.pass,locations$table.departure,locations$ table.public_departure,locations$table.real_departure,locations$table.location_cancelled,
tiplocs.tps_description FROM locations$table,tiplocs WHERE id =$id AND locations$table.tiploc_code=tiplocs.tiploc;";
$realTimeResult =mysql_query($realTimeQuery);
while($row3 = mysql_fetch_assoc($realTimeResult)){
$output3[] = $row3;
}
print json_encode($output3);
print("*");
unset($output3);
unset($id);
}
print('>');
print json_encode($output);
?>
Many Thanks
Matt
The biggest issue with your setup is this foreach loop because it is unnecessary and results in n number of round trips to the database to execute a query, fetch and analyze the results.
foreach ($output as $value) {
Rewrite the initial query to include all of the fields you will need to do your later calculations.
Something like this would work.
SELECT tl.tps_description AS 'C', lc.public_departure, lc.id, s.stp_indicator, s.train_uid,
lc.id, lc.location_order, lc.arrival, lc.public_arrival, lc.real_arrival, lc.pass, lc.departure, lc.real_departure, lc.location_cancelled
FROM locations$table lc INNER JOIN schedules_cache sc ON lc.id = sc.id
INNER JOIN schedules s ON s.id = sc.id
INNER JOIN activations a ON s.id = a.id
INNER JOIN tiplocs tl ON sc.destination = tl.tiploc
WHERE '$date' BETWEEN schedules.date_from AND schedules.date_to
AND lc.tiploc_code = '$tiploc'
AND lc.real_departure LIKE '0'
AND lc.public_departure NOT LIKE '0'
AND lc.public_departure >='$time'-300
AND lc.public_departure <='$time'+300
AND s.runs_th LIKE '1'
ORDER BY lc.public_departure ASC
LIMIT 0,30;
Eliminating n query executions from your page load should dramatically increase response time.
Ignoring the problems with the code, in order to speed up your query, use the EXPLAIN command to evaluate where you need to add indexes to your query.
At a guess, you probably will want to create an index on whatever locations$table.public_departure evaluates to.
http://dev.mysql.com/doc/refman/5.0/en/using-explain.html
A few things I noticed.
First, you are joining tables in the where clause, like this
from table1, table2
where table1.something - table2.something
Joining in the from clause is faster
from table1 join table2 on table1.something - table2.something
Next, I'm not a php programmer, but it looks like you are running similar queries inside a loop. If that's true, look for a way to run just one query.
Edit starts here
This is in response to gazarsgo's that I back up by claim about joins in the where clause being faster. He is right, I was wrong. This is what I did. The programming language is ColdFusion:
<cfsetting showdebugoutput="no">
<cfscript>
fromtimes = ArrayNew(1);
wheretimes = ArrayNew(1);
</cfscript>
<cfloop from="1" to="1000" index="idx">
<cfquery datasource="burns" name="fromclause" result="fromresult">
select count(distinct hscnumber)
from burns_patient p join burns_case c on p.patientid = c.patientid
</cfquery>
<cfset ArrayAppend(fromtimes, fromresult.executiontime)>
<cfquery datasource="burns" name="whereclause" result="whereresult">
select count(distinct hscnumber)
from burns_patient p, burns_case c
where p.patientid = c.patientid
</cfquery>
<cfset ArrayAppend(wheretimes, whereresult.executiontime)>
</cfloop>
<cfdump var="#ArrayAvg(fromtimes)#" metainfo="no" label="from">
<cfdump var="#ArrayAvg(wheretimes)#" metainfo="no" label="where">
I did ran it 5 times. The results, in milliseconds, follow.
9.563 9.611
9.498 9.584
9.625 9.548
9.831 9.769
9.792 9.813
The first number represents joining in the from clause, the second joining in the where clause. The first number is lower only 60% of the time. Had it been lower 100% percent of the time, it would have shown that joining in the from clause is faster, but that' not the case.
I need to output the following query as csv.
I can easily write php logic to transpose the rows to columns from my group_concat column
However I am keen to keep as much of the data part in the database and minimize the manipulations on the php side.
I am experimenting with the two columns below the group_concat in the query.
The problem is the abundance value also returns for life_stage column. If there is no way around this other than manipulating the group_concat key values then that's fine, I just wanted to double check. Thanks in advance
SELECT
`tr`.`tr_id_pk` as 'RecordKey',
`t`.`tax_name` as `TaxonName`,
`tr`.`tr_date` as 'Date',
`s`.`si_name` as 'SiteName',
`tr`.`tr_grid_reference` as 'GridReference',
`tr`.`tr_is_site_grid` as 'IsSiteGrid',
`r`.`rec_name` as 'Recorder',
`r`.`rec_email` as 'RecorderEmail',
`tr`.`tr_comment` as 'RecordComment',
`tr`.`tr_last_update` as 'LastUpdated',
`tr`.`tr_form_key` as 'FormKey',
`c`.`co_name` as 'County',
`vc`.`vc_name` as 'ViceCounty',
`h`.`hab_name` as 'Habitat',
GROUP_CONCAT(DISTINCT CONCAT_WS('=', `ra`.`ra_name`, `rad`.`rad_value`)) as 'RecordAttributeKeyValuePairs',
`rad`.`rad_value` AS `abundance`,
`rad`.`rad_value` AS `life_stage`
FROM
`taxon_record`as `tr`
INNER JOIN
`taxon`as `t` ON `tr`.`tax_id_fk` = `t`.`tax_id_pk`
INNER JOIN
`recorder`as `r` ON `tr`.`rec_id_fk` = `r`.`rec_id_pk`
INNER JOIN
`site`as `s` ON `tr`.`si_id_fk` = `s`.`si_id_pk`
LEFT JOIN
`county`as `c` ON `tr`.`co_id_fk` = `c`.`co_id_pk`
LEFT JOIN
`vice_county`as `vc` ON `tr`.`vc_id_fk` = `vc`.`vc_id_pk`
LEFT JOIN
`habitat`as `h` ON `tr`.`hab_id_fk` = `h`.`hab_id_pk`
LEFT JOIN
(`record_attribute_data`as `rad`
INNER JOIN `record_attribute`as `ra` ON (`rad`.`ra_id_fk` = `ra`.`ra_id_pk`)) ON (`tr`.`tr_id_pk` = `rad`.`tr_id_fk`)
WHERE
`r`.`rec_email` = 'some_email#somewhere.com'
GROUP BY `tr`.`tr_id_pk`;
What you wish to do is known as "pivoting" your data and is something for which some other RDBMS have native support, but MySQL does not (by design, as the developers feel that such manipulations belong in the presentation layer).
However, as you've hinted at, you can construct a rather horrible MySQL query to perform the pivoting operation manually (one needs to join the attributes tables to the query once for each output column):
SELECT tr.tr_id_pk, abundance, life_stage -- etc.
FROM taxon_record AS tr
LEFT JOIN (
SELECT rad.tr_id_fk, rad.rad_value AS abundance
FROM
record_attribute_data ra JOIN record_attribute rad
ON rad.ra_id_fk = ra.ra_id_pk
WHERE ra.ra_name = 'abundance'
) AS tAbundance ON tAbundance.tr_id_fk = tr.tr_id_pk
LEFT JOIN (
SELECT rad.tr_id_fk, rad.rad_value AS life_stage
FROM
record_attribute_data ra JOIN record_attribute rad
ON rad.ra_id_fk = ra.ra_id_pk
WHERE ra.ra_name = 'life_stage'
) AS tLife_Stage ON tLife_Stage.tr_id_fk = tr.tr_id_pk
-- etc.
If you choose to go down this path, you can make your life slightly easier by generating this query using either a looping construct in PHP or a prepared statement in MySQL.
Is the problem that you are fetching the same column twice under different names?:
`rad`.`rad_value` AS `abundance`,
`rad`.`rad_value` AS `life_stage`
Looks like you'll always get the same thing for abundance and life_stage that way.