MYSQL Inner Join speed issue - php

I have been having mega issues with this query.
It's a file that's run by cron every 24 hours and runs conditional checks against all members in the database.
Apparently it crashes the MYSQL server (takes 2 hours to execute?).
Every table has primary keys set on rows foo_id & foo_uid so the joins are indexed under PRIMARY and as such should be good for speed.
please help, this is killing me.
$members = new WA_MySQLi_RS("members", $alpha, 1);
$members->setQuery("SELECT
registration.*,
child_base_survey.*,
child_base_scas.*,
child_base_smqf.*,
parent_base_survey.*,
parent_base_ippa.*,
parent_base_eac.*,
parent_base_scas.*,
parent_base_smqf.*,
parent_base_eval.*,
user_access_level.*,
parent_one_month_survey.*,
parent_one_month_ippa.*,
parent_one_month_eac.*,
parent_one_month_eval.*,
child_three_month_survey.*,
child_three_month_scas.*,
child_three_month_smqf.*,
parent_three_month_survey.*,
parent_three_month_scas.*,
parent_three_month_smqf.*,
parent_three_month_eval.*,
cron.*
FROM registration
INNER JOIN child_base_survey ON registration.rego_parent_uid = child_base_survey.child_base_survey_uid
INNER JOIN child_base_scas ON child_base_survey.child_base_survey_uid = child_base_scas.child_base_scas_uid
INNER JOIN child_base_smqf ON child_base_scas.child_base_scas_uid = child_base_smqf.child_base_smqf_uid
INNER JOIN parent_base_survey ON child_base_smqf.child_base_smqf_uid = parent_base_survey.parent_base_survey_uid
INNER JOIN parent_base_ippa ON parent_base_survey.parent_base_survey_uid = parent_base_ippa.parent_base_ippa_uid
INNER JOIN parent_base_eac ON parent_base_ippa.parent_base_ippa_uid = parent_base_eac.parent_base_eac_uid
INNER JOIN parent_base_scas ON parent_base_eac.parent_base_eac_uid = parent_base_scas.parent_base_scas_uid
INNER JOIN parent_base_smqf ON parent_base_scas.parent_base_scas_uid = parent_base_smqf.parent_base_smqf_uid
INNER JOIN parent_base_eval ON parent_base_smqf.parent_base_smqf_uid = parent_base_eval.parent_base_eval_uid
INNER JOIN user_access_level ON parent_base_eval.parent_base_eval_uid = user_access_level.user_access_level_uid
INNER JOIN parent_one_month_survey ON user_access_level.user_access_level_uid = parent_one_month_survey.parent_one_month_survey_uid
INNER JOIN parent_one_month_ippa ON parent_one_month_survey.parent_one_month_survey_uid = parent_one_month_ippa.parent_one_month_ippa_uid
INNER JOIN parent_one_month_eac ON parent_one_month_ippa.parent_one_month_ippa_uid = parent_one_month_eac.parent_one_month_eac_uid
INNER JOIN parent_one_month_eval ON parent_one_month_eac.parent_one_month_eac_uid = parent_one_month_eval.parent_one_month_eval_uid
INNER JOIN child_three_month_survey ON parent_one_month_eval.parent_one_month_eval_uid = child_three_month_survey.child_three_month_survey_uid
INNER JOIN child_three_month_scas ON child_three_month_survey.child_three_month_survey_uid = child_three_month_scas.child_three_month_scas_uid
INNER JOIN child_three_month_smqf ON child_three_month_scas.child_three_month_scas_uid = child_three_month_smqf.child_three_month_smqf_uid
INNER JOIN parent_three_month_survey ON child_three_month_smqf.child_three_month_smqf_uid = parent_three_month_survey.parent_three_month_survey_uid
INNER JOIN parent_three_month_scas ON parent_three_month_survey.parent_three_month_survey_uid = parent_three_month_scas.parent_three_month_scas_uid
INNER JOIN parent_three_month_smqf ON parent_three_month_scas.parent_three_month_scas_uid = parent_three_month_smqf.parent_three_month_smqf_uid
INNER JOIN parent_three_month_eval ON parent_three_month_smqf.parent_three_month_smqf_uid = parent_three_month_eval.parent_three_month_eval_uid
INNER JOIN cron ON parent_three_month_eval.parent_three_month_eval_uid = cron.cron_uid WHERE registration.rego_parent_uid = ?");
$members->bindParam("s", "" . ((isset($_SESSION["rego_parent_uid"])) ? $_SESSION["rego_parent_uid"] : "") . "", "-1"); //WAQB_Param1
$members->execute();
?>

Run them as separate queries, from a cursory glance, it looks like you are just joining every possible table with data associated with a reference to an id value. Your results are going to be the cross product of every matching row in each table. For N tables, each row in table X will be repeated r0*r1*r2*....rX-1*rX+1*...rN. With the 23 tables there, if each only had 2 rows, you'd have nearly 8.4 million rows in your results.

Related

INNER JOIN with multiple tabels

I Have PHP Script That SELECT FROM 7 Tabels and client_id is the Common Column
And Return Single Tebel With all the clients by as there client_id row after row, now The problem is the if there is in one Tabel tow rows with 2 duplicate id That's print duplicate rows like 100 times
maybe i don't do INNER JOIN right or something if someone has an idea how to prevent this
$stmt = $pdo->query("SELECT * FROM client_form cf
INNER JOIN client_form_2 cf2 ON cf.client_id = cf2.client_id
INNER JOIN client_form_3 cf3 ON cf.client_id = cf3.client_id
INNER JOIN client_form_4 cf4 ON cf.client_id = cf4.client_id
INNER JOIN client_form_5 cf5 ON cf.client_id = cf5.client_id
INNER JOIN client_form_6 cf6 ON cf.client_id = cf6.client_id
INNER JOIN client_form_7 cf7 ON cf.client_id = cf7.client_id
");

Why Huge Join Fails After Specific Table?

I have a huge select query where i have to join more than 85 tables. I keep getting an error when running the query, if I re-run the query when shrinking the overall statement it runs fine.
See a portion of the join below, it does that all the way to table 85:
select $imploded_tables from $apps a
left join $mobile_c0 $m_c0 on $apps.id = $m_c0.id
left join $mobile_c1 $m_c1 on $m_c0.id = $m_c1.id
left join $mobile_c2 $m_c2 on $m_c1.id = $m_c2.id
left join $mobile_c3 $m_c3 on $m_c2.id = $m_c3.id
left join $mobile_c4 $m_c4 on $m_c3.id = $m_c4.id
left join $mobile_c5 $m_c5 on $m_c4.id = $m_c5.id
left join $mobile_c6 $m_c6 on $m_c5.id = $m_c6.id
left join $mobile_c7 $m_c7 on $m_c6.id = $m_c7.id
left join $mobile_c8 $m_c8 on $m_c7.id = $m_c8.id
left join $mobile_c9 $m_c9 on $m_c8.id = $m_c9.id
left join $mobile_c10 $m_c10 on $m_c9.id = $m_c10.id
...
...
Mysql Documentation
The maximum number of tables that can be referenced in a single join is 61.
As per #sf_admin answer the maximum no of join is 61 , so you can do something like below
select imploded_tables from apps a
JOIN
(
SELECT * from mobile_c0
UNION
SELECT * from mobile_c1
....
)tmp
where(tmp.id=a.id)
It might not 100% percent answer your question but this is some work around I did

How I can merge repeated rows in MySQL?

I'm working on a project that each orgnizations have barriers, each organization can contain 1 or more barriers, and then I need generate an XML file that show each barrier into the organization .
But executing this sql code:
SELECT organizations.*, barriers.barrierName AS bname, barriers.type AS btype, geographs.name AS geo, rpd.rpdname AS rpdn, rpd.meaning AS rpdmean FROM organizations
left join orgbarriers on orgbarriers.idOrg = organizations.id
left join barriers on orgbarriers.idBarrier = barriers.id
left join orggeographs on organizations.id = orggeographs.idOrg
left join geographs on geographs.id = orggeographs.idGeo
left join orgrpds on orgrpds.idOrg = organizations.id
left join rpd on rpd.id = orgrpds.idRPD
I get repeated rows like this image:
Use the keyword "DISTINCT"
SELECT DISTINCT rest of your query

Get data from all tables in database

Ok I have 5 tables in my database they are as follows
officelocations_tbl
state_tbl
city_tbl
staff_tbl
titles_tbl
The titles table is only associated with the staff table but the others are all inner joined. I have tried various mysql statements but none are allowing me to bring in the titles_tbl.
here is the latest version of the sql statement I am attempting to use:
SELECT officelocations_tbl.*,city_tbl.*, state_tbl.* , titles_tbl.*,
contact1.firstName AS c1Firstname, contact1.lastName AS c1lastName,
contact1.middleInitial AS c1middleInitial, contact1.suffix AS c1suffix,
contact1.accredations AS c1accredations, contact1.phone AS c1Phone,
contact1.faxNumber AS c1FaxNumber, contact1.mobilePhone AS c1Mobile,
contact1.email AS c1Email, contact1.titleID AS c1Title,
contact2.firstName AS c2Firstname, contact2.lastName AS c2lastName,
contact2.middleInitial AS c2middleInitial, contact2.suffix AS c2suffix,
contact2.accredations AS c2accredations, contact2.phone AS c2Phone,
contact2.faxNumber AS c2FaxNumber, contact2.mobilePhone AS c2Mobile,
contact2.email AS c2Email, contact2.titleID AS c2Title,
partner.firstName AS c3Firstname, partner.lastName AS c3lastName,
partner.middleInitial AS c3middleInitial, partner.suffix AS c3suffix,
partner.accredations AS c3accredations, partner.phone AS c3Phone,
partner.faxNumber AS c3FaxNumber, partner.mobilePhone AS c3Mobile,
partner.email AS c3Email, partner.titleID AS c3Title
FROM officelocations_tbl
JOIN city_tbl ON (officelocations_tbl.cityID = city_tbl.cityID)
INNER JOIN titles_tbl ON titles_tbl.titleID = staff_tbl.titleID
LEFT OUTER JOIN state_tbl ON (officelocations_tbl.stateID = state_tbl.stateID)
LEFT OUTER JOIN staff_tbl contact1 ON (contact1.staffID = officelocations_tbl.contact1)
LEFT OUTER JOIN staff_tbl contact2 ON (contact2.staffID = officelocations_tbl.contact2)
LEFT OUTER JOIN staff_tbl partner ON (partner.staffID = officelocations_tbl.partner)
However this gives me an error [Err] 1054 - Unknown column 'staff_tbl.titleID' in 'on clause'. If I remove both the lines:
INNER JOIN titles_tbl ON titles_tbl.titleID = staff_tbl.titleID
titles_tbl.*,
it works but doesn't pull in the title. I have tried doing it this way as well but then it only pulls in the title once and not for all three contacts.
SELECT
staff_tbl.staffID,
staff_tbl.staffID_C2,
staff_tbl.staffID_P,
staff_tbl.firstName,
staff_tbl.middleInitial,
staff_tbl.lastName,
staff_tbl.suffix,
staff_tbl.accredations,
staff_tbl.email,
staff_tbl.phone,
staff_tbl.mobilePhone,
staff_tbl.officePhone,
staff_tbl.faxNumber,
staff_tbl.address1,
staff_tbl.address2,
staff_tbl.cityID,
staff_tbl.stateID,
staff_tbl.zipCode,
staff_tbl.titleID,
staff_tbl.locationID,
staff_tbl.photoURL,
staff_tbl.vCardURL,
staff_tbl.qRCodeURL,
staff_tbl.resumeURL,
staff_tbl.biography,
staff_tbl.dateCreated,
officelocations_tbl.locationID,
officelocations_tbl.officeName,
officelocations_tbl.address1,
officelocations_tbl.address2,
officelocations_tbl.cityID,
officelocations_tbl.stateID,
officelocations_tbl.zipCode,
officelocations_tbl.officePhone,
officelocations_tbl.contact1,
officelocations_tbl.contact2,
officelocations_tbl.partner,
city_tbl.cityID,
city_tbl.cityName,
state_tbl.stateID,
state_tbl.state_abreviation,
state_tbl.state_name,
titles_tbl.titleID,
titles_tbl.titleName,
contact1.firstName AS c1Firstname, contact1.lastName AS c1lastName,
contact1.middleInitial AS c1middleInitial, contact1.suffix AS c1suffix,
contact1.accredations AS c1accredations, contact1.phone AS c1Phone,
contact1.faxNumber AS c1FaxNumber, contact1.mobilePhone AS c1Mobile,
contact1.email AS c1Email, contact1.titleID AS c1Title,
contact2.firstName AS c2Firstname, contact2.lastName AS c2lastName,
contact2.middleInitial AS c2middleInitial, contact2.suffix AS c2suffix,
contact2.accredations AS c2accredations, contact2.phone AS c2Phone,
contact2.faxNumber AS c2FaxNumber, contact2.mobilePhone AS c2Mobile,
contact2.email AS c2Email, contact2.titleID AS c2Title,
partner.firstName AS c3Firstname, partner.lastName AS c3lastName,
partner.middleInitial AS c3middleInitial, partner.suffix AS c3suffix,
partner.accredations AS c3accredations, partner.phone AS c3Phone,
partner.faxNumber AS c3FaxNumber, partner.mobilePhone AS c3Mobile,
partner.email AS c3Email, partner.titleID AS c3Title
FROM officelocations_tbl
INNER JOIN staff_tbl ON staff_tbl.staffID = officelocations_tbl.contact1
INNER JOIN state_tbl ON state_tbl.stateID = officelocations_tbl.stateID
INNER JOIN titles_tbl ON titles_tbl.titleID = staff_tbl.titleID
INNER JOIN city_tbl ON city_tbl.cityID = officelocations_tbl.cityID
LEFT OUTER JOIN staff_tbl contact1 ON (contact1.staffID = officelocations_tbl.contact1)
LEFT OUTER JOIN staff_tbl contact2 ON (contact2.staffID = officelocations_tbl.contact2)
LEFT OUTER JOIN staff_tbl partner ON (partner.staffID = officelocations_tbl.partner)
This will only pull for the first association of staff_tbl.staffID = officelocations_tbl.contact1. I am stumped as to what to try next. Is there anyone who would know how to get it to pull all 5 tables?
You just need to move your INNER JOIN down lower, e.g. from
INNER JOIN titles_tbl ON titles_tbl.titleID = staff_tbl.titleID
LEFT OUTER JOIN state_tbl ON (officelocations_tbl.stateID = state_tbl.stateID)
to
LEFT OUTER JOIN state_tbl ON (officelocations_tbl.stateID = state_tbl.stateID)
INNER JOIN titles_tbl ON titles_tbl.titleID = staff_tbl.titleID
At the time the parser reaches the INNER JOIN, staff_tbl has not yet been joined, and the parser will not "look ahead" to see if it's joined later. So it immediately bails with a "no such table/field".
Switching the order in which this occurs allows taff_table to be joined in first, and then you can use it in further joins.

Doctrine 2 QueryBuilder vs Handcoded DQL - different results

I have Doctrine2 DQL query but I want to build it with QueryBuilder, I have noticed that produced DQL is somewhat different from the handcrafted one, and I'm wondering what am I missing here - maybe I'm not aware of something or doing things wrong way?
Ok, some details:
My handcrafted query looks like this:
select count(fi.id)
from Entities\Content\FolderLookup fl
join fl.site fls
join fl.folder flf,
Entities\Content\FolderItem fi
join fi.site fis
join fi.folder fif
join fi.item it
join it.type tp
join it.content ic
where fl.namePath = ?1
and tp.name = ?2
and fls.id = fis.id
and flf.id = fif.id
Now, I'm trying to reproduce it like this with QueryBuilder:
$qb->select("count(fi.id)")->from("Entities\Content\FolderLookup", "fl")->join("fl.site","fls")->join("fl.folder", "flf");
$qb->from("Entities\Content\FolderItem","fi")->join("fi.site","fis")->join("fi.folder","fif");
$qb->join("fi.item","it")->join("it.type","tp")->join("it.content","ic");
$wherePart = $qb->expr()->andx();
$wherePart->add($qb->expr()->eq("fl.namePath","?1"));
$wherePart->add($qb->expr()->eq("tp.name","?2"));
$wherePart->add($qb->expr()->eq("fls.id","fis.id"));
$wherePart->add($qb->expr()->eq("flf.id","fif.id"));
$qb->where($wherePart);
This however is producing this DQL query:
SELECT count(fi.id) FROM Entities\Content\FolderLookup fl,
Entities\Content\FolderItem fi
INNER JOIN fl.site fls
INNER JOIN fl.folder flf
INNER JOIN fi.site fis
INNER JOIN fi.folder fif
INNER JOIN fi.item it
INNER JOIN it.type tp
INNER JOIN it.content ic
WHERE (fl.namePath = ?1)
AND (tp.name = ?2)
AND (fls.id = fis.id)
AND (flf.id = fif.id)
As you can see there is part of this missing comapring to handcrafted one (First line):
fl join fl.site fls join fl.folder flf
I'm not sure why these joins are missing as I am defining them here:
$qb->select("count(fi.id)")->from("Entities\Content\FolderLookup", "fl")->join("fl.site","fls")->join("fl.folder", "flf");
Update:
The fun part starts, when DQL gets translated into SQL - in this case MySQL:
Handcrafted one becomes:
SELECT count(f0_.id) AS sclr0 FROM FolderLookup f1_ INNER JOIN Site s2_ ON f1_.site_id = s2_.id INNER JOIN Folder f3_ ON f1_.folder_id = f3_.id, FolderItem f0_ INNER JOIN Site s4_ ON f0_.site_id = s4_.id INNER JOIN Folder f5_ ON f0_.folder_id = f5_.id INNER JOIN Item i6_ ON f0_.item_id = i6_.id INNER JOIN ItemType i7_ ON i6_.type_id = i7_.id INNER JOIN ItemContent i8_ ON i6_.content_id = i8_.id WHERE f1_.namePath = ? AND i7_.name = ? AND s2_.id = s4_.id AND f3_.id = f5_.id
Where generated one looks like this:
SELECT count(f0_.id) AS sclr0 FROM FolderLookup f1_, FolderItem f0_ INNER JOIN Site s2_ ON f1_.site_id = s2_.id INNER JOIN Folder f3_ ON f1_.folder_id = f3_.id INNER JOIN Site s4_ ON f0_.site_id = s4_.id INNER JOIN Folder f5_ ON f0_.folder_id = f5_.id INNER JOIN Item i6_ ON f0_.item_id = i6_.id INNER JOIN ItemType i7_ ON i6_.type_id = i7_.id INNER JOIN ItemContent i8_ ON i6_.content_id = i8_.id WHERE (f1_.namePath = ?) AND (i7_.name = ?) AND (s2_.id = s4_.id) AND (f3_.id = f5_.id)
And this is invalid statement, as database returns with:
Column not found: 1054 Unknown column 'f1_.site_id' in 'on clause'
Any ideas welcome.
It seems the DQL parser is wrongly positioning the joins to the wrong from.
My initial suggestion is to try to make only 1 FROM item and a subselect.
Also, I'd love if you add the same content you asked here in our bug tracking: http://www.doctrine-project.org/jira/browse/DDC
Thanks a lot!
Guilherme Blanco
Doctirne Core Developer
they are not missing. just reordered
INNER JOIN fl.site fls
INNER JOIN fl.folder flf

Categories