Performance, sql heavy join vs multiple small request - php

I have the following Mysql database structure
[Table - Category1]
[Table Category1 -> Category2 ] (One to N relation)
[Table - Category2]
[Table Category2 -> Item ] (One to N relation)
[Table - Item]
and I want to get everything into an array in PHP with the following structure
$arr[$i]['name'] = 'name of something in category1';
$arr[$i]['data'][$j]['name'] = 'name of something in category2';
$arr[$i]['data'][$j]['data'][$k]['name'] = 'name of something in item';
So basically I don't know if I should use one "heavy" sql request with JOIN like the following one or use an iterative method
The join request
SELECT c1.name as c1name, c2.name as c2name, i.name
FROM category1 c1
LEFT JOIN category1_to_category2 c1tc2 ON c1.id = c1tc2.id_category1
LEFT JOIN category2 c2 ON c1tc2.id_category2 = c2.id
LEFT JOIN category2_to_item c2ti ON c2.id = c2ti.id_category2
LEFT JOIN item i ON c2ti.id_item = i.id
The iterative method
$sql = 'SELECT id, name FROM category1';
$result = $mysqli->query($sql);
$arr = array();
$i = 0;
while ($arr[$i] = $result->fetch_assoc()) {
$join = $mysqli->query('SELECT c2.id, c2.name FROM category2 c2 LEFT JOIN category1_to_category2 c1tc2 ON c2.id = c1tc2.id_category 2 WHERE c1tc2.id_category1 = '.$arr[$i]['id']);
$j = 0;
while ($arr[$i]['data'][$j] = $join->fetch_assoc())
/* same request as above but with items */
$i++;
}
The iterative solution will make around 10 * 20 request which seems a lot to me that's why I would choose the first solution (4 JOIN single request).
However, with the single request solution, my array will look like that
$arr[0]['c1name'];
$arr[0]['c2name'];
$arr[0]['iname'];
And it will require some PHP traitement to obtain the desired array which I require to display in tabs in an HTML page. So my question is, is it better to have one big SQL request with some PHP array manipulation or to have multiple small request without the PHP array manipulation ? I know that in most case, getting all the data from SQL is a better solution but in this case I'm not sure. By the way, my only consideration is the loading time of my web page.
Thanks in advance for your help =).

It is typically better, and your example is no exception, to have the SQL server do as much of the data formatting and iteration as possible as SQL servers are typically more efficient at the task than common programming languages.
Add to this that you are cutting down on query load of the server and you have a very good reason for using complex joins.
The only downside is complex SQL queries can be hard to format and debug, if not already using a 3rd party SQL tool I would recommend getting one.

To go with the answer by Wobbles (that I agree with), I would suggest that you do a single query but you store the last key for each of c1name, c2name and iname. When these change you increment the relevant array subscript and initialise the lower level ones again to build up your array.
Something like this:-
<?php
$sql = "SELECT c1.name AS c1name, c2.name AS c2name, i.name AS iname
FROM category1 c1
LEFT JOIN category1_to_category2 c1tc2 ON c1.id = c1tc2.id_category1
LEFT JOIN category2 c2 ON c1tc2.id_category2 = c2.id
LEFT JOIN category2_to_item c2ti ON c2.id = c2ti.id_category2
LEFT JOIN item i ON c2ti.id_item = i.id"
$result = $mysqli->query($sql);
$arr = array();
$i = 0;
$j = 0;
$k = 0;
$c1name = '';
$c2name = '';
$iname = '';
while ($row = $result->fetch_assoc())
{
switch(true)
{
case $row['c1name'] != $c1name :
$i++;
$j = 0;
$k = 0;
$arr[$i]['name'] = $row['c1name'];
$arr[$i]['data'][$j]['name'] = $row['c2name'];
$arr[$i]['data'][$j]['data'][$k]['name'] = $row['iname'];
break;
case $row['c2name'] != $c2name :
$j++;
$k = 0;
$arr[$i]['data'][$j]['name'] = $row['c2name'];
$arr[$i]['data'][$j]['data'][$k]['name'] = $row['iname'];
break;
default :
$k++;
$arr[$i]['data'][$j]['data'][$k]['name'] = $row['iname'];
break;
}
$c1name = $row['c1name'];
$c2name = $row['c2name'];
$iname = $row['iname'];
}
As an aside there is some code at work that is used to generate a menu. Just 2 levels, and it was originally coded as one query for the first level and then one query for each of the records in the first level to get all the items below it. Not complex (there are only ~16 items in the first level, and on average under 10 items below each of those). I rewrote that to a single joined query. Typical time to generate that menu dropped from 0.25 seconds down to 0.004 seconds. It is easy for the time taken sending queries to the database to rapidly become excessive.

Related

How to group my results and have all rows shown?

SELECT i.itemsname
, i.itemsprice
, i.itemsdescrip
, c.catname
, c.catdes
, c.status
, c.collapse
, c.catid
FROM items i
LEFT
JOIN categories c
ON c.catid = i.catid
WHERE i.restid
AND c.restid =12
GROUP
BY c.catid
that is my query at the moment but I would like to have something like this....
but this is what I'm getting:
Ok, I lied in the comments, so With PDO (haven't tested it)
$stmt = $PDO->prepare('SELECT
categories.catname,
items.itemsname,
items.itemsprice,
items.itemsdescrip,
categories.catdes,
categories.status,
categories.collapse,
categories.catid
FROM items
LEFT JOIN categories ON items.catid=categories.catid
WHERE items.restid AND categories.restid = :restid');
$stmt->execute([':restid' => 12]);
$data = $stmt->fetchAll(\PDO::FETCH_GROUP);
foreach($data as $catname => $rows){
//echo group html stuff
//echo "<dl>";
//echo "<dt>$catname</dt>".;
foreach($rows as $row){
//echo row data stuff
// echo "<dd> {stuff} </dd>";
}
//echo "</dl>";
}
I'll leave the html up to you. But as I said you want a data structure like this
[
'BREAKFASTS' => [
0 => [ name => "wimpy hamburger", description => "bla bla", price => "$100,000"],
1 => [ ... ]
],
'SINGLE BURGERS' => [ ...]
]
note that the first field after "SELECT" is by default the field used by FETCH_GROUP
See in this way, the first foreach can output the title of the category, which is BREAKFASTS for example. Then the inner foreach can do the individual rows in the table.
Personally I would use a dl, dt, dd tag setup as my structure (hinted in the comments, i really am to lazy today to code all the html, <sigh>)
https://www.w3schools.com/tags/tag_dt.asp
UPDATE
You may want to check your query
...
WHERE
items.restid AND ...
Seems to be flawed, just saying. I saw this while optomizing the query for sorting.
SELECT
c.catname,
i.itemsname,
i.itemsprice,
i.itemsdescrip,
c.catdes,
c.status,
c.collapse,
c.catid
FROM
(
SELECT c0.catid FROM categories AS c0 WHERE c0.restid = :restid SORT BY c0.catname
) AS t
JOIN
categories AS c ON t.catid=c.catid
LEFT JOIN
items AS i ON items.catid=categories.catid
WHERE
items.restid = ? //< this is the error/omission/strangeness i pointed out above.
So a few things to note, first you should base the query off the categories, as an empty category should be shown, while an item without a category will blow it all to bits ( basically, ie how can you group them by the category if they have none ) You'll wind up with some hodgepoge of items with no category at the end, of course based on your example I'm assuming a Many to One relationship. For example One category can have Many items, and Many items can belong to a category. (it's probably more ideal to do a Many to Many, but that's another story for another day)
The reason the above query is more optimized is the inner query, creates only a small temp table using the catid, And sorts on just the data from the cat table and only the data that is pulled by the where.
Then as we move to the outer query, they basically inherent the sort from the join, and we can pull the rest of the data from that. It's typically about 2-10x faster this way (of course I haven't test this particular query) in theory. Of course this is a bit more complex/advanced query and is optional, but it should improve sort performance if my mind is in the right place tonight... lol
Also I abbreviated your table names (alias), as I said I am lazy like that. Sadly my answers are always so long, dont ask me how I see all these issues, it's just experience or how my dyslexic brain works?
Lastly, if you really must use mysqli, you can manually group them with something like this.
$data = [];
while(false !== ($row = $res->fetch_assoc())){
$key = $row['catname'];
if(!isset($data[$key])) $data[$key] = [];
$data[$key][] = $row;
}
It's all so prosaic (common place, non-poetic) at this point for me.
Good luck.
$cat = mysqli_query($connect, "SELECT
categories.catname,
items.itemsname,
items.itemsprice,
items.itemsdescrip,
categories.catdes,
categories.catid
FROM items
LEFT JOIN categories ON items.catid=categories.catid
WHERE items.restid AND categories.restid = 12");
if($cat === FALSE) {
die(mysqli_error());
}
$data = [];
while ($rowb = mysqli_fetch_array($cat)) {
$key = $rowb['catname'];
if(!isset($data[$key])) $data[$key] = [];
$data[$key][] = $rowb;
foreach($data as $catname => $rowbs){
echo "
<dl><button class='accordiontry'><dt>$catname</dt></button>";
<div class='panel1'>
foreach($rowbs as $rowb){
echo"<div class='rmenu'>
<dd><span class='item'>{$rowb['itemsname']}</span>
<span class='price'>£{$rowb['itemsprice']}</span><br>
<span class='des'>{$rowb['itemsdescrip']}</span> ";
}
echo"</div></dd>
</div></dl>";
}
}
}

Optimize sql query PHP Phalcon

I Have this code:
if(($banners = $this->app->cache5min->get($key_cache)) === NULL) {
$find_items = array();
$banners = $this->app->db->fetchAll("
SELECT b.id,b.active, b.richmedia, b.rich_position, b.ip_limit,
b.cookie_limit, b.cookie_interval, b.day_limit, b.limit_interval,
b.frequency, b.close_btn, b.use_geo,
bpl.s_x, bpl.s_y
FROM bs_items_places AS bp
LEFT JOIN bs_items AS b ON b.id = bp.item_id
LEFT JOIN bs_places AS bpl ON bpl.id = bp.place_id
WHERE b.active = 1
AND b.date_start < '".$dateNow."'
AND b.date_stop > '".$dateNow."'
AND bpl.active = 1
AND bp.place_id = {$idpl}
AND (IF((time_from!='00:00:00' AND time_to!='00:00:00'),
(time_from<='".$dateTimeNow."'
AND time_to >='".$dateTimeNow."'), 1)
)
GROUP BY b.id
ORDER BY b.frequency DESC, b.day_limit DESC
");
foreach($banners AS $bnr) {
$find_items[$bnr['id']] = $bnr;
}
$banners = $find_items;
$this->app->cache5min->save($key_cache,$banners);
}
//echo '<pre>'; print_r($banners); exit;
if(!$banners || !count($banners)) { return $this->getDefaultBanner($idpl,$x); }
When I remove first statement, my CPU is good, when I return first statement, my CPU 100% and this gives mysql proccess, how I Can Optimize select query?
First step is to get rid of LEFT, unless you need it. This may open up more optimization options.
This construct is essentially un-optimizable:
WHERE start < ...
AND end > ...
There is an "explode-implode" problem. First, the number of rows explodes due to the JOINs, then it implodes do to the GROUP BY. But... Meanwhile, there are no "aggregates" such as COUNT() or SUM(). So this smells like a poorly formed query -- Why do you need to do the GROUP BY? If you are getting 'duplicate' rows, then you have a worse problem -- what happens in the columns that are not duplicated? You will get random values. See what happens without the GROUP BY.
This composite index on bp may help:
INDEX(place_id, item_id)

PDO advanced search from multiple tables

I am trying to create a person search, based on multiple select boxes, radio buttons and dropdowns.
Some of the select boxes are arrays so I need to consider all of the selected options to show the results.
There are different tables for services, languages, about, work experience etc.
But something is wrong with my logic. At the moment, when I am trying the search it goes like this: I select English language from the language dropdow, but there is nobody who speaks English so there are no results which is correct. Then if I select one of the services (let's sat cleaning), then there will be one result because the same person has listed cleaning as their service, but this is wrong because he doesn't speak English.
I am using union all and LIKE in my query, but can somebody tell me why it still showing results? If i am doing something wrong, could you please point me in the right direction?
This is my query and PHP code so far:
if (isset($_POST['userServices']) && !empty($_POST['userServices'])){
$userServices = $_POST['userServices'];
for ($i = 0; $i < count($userServices); $i++) {
$services = $userServices[$i];
}
}
if (isset($_POST['languagesArray']) && !empty($_POST['languagesArray'])){
$languagesArray = $_POST['languagesArray'];
for ($i = 0; $i < count($languagesArray); $i++) {
$languages = $languagesArray[$i];
}
}
$search = $user_home->runQuery("
SELECT nanny_services.user_id, userFirstName
FROM hoidja.nanny_services
JOIN services on nanny_services.service_id = services.service_id
JOIN tbl_users ON tbl_users.user_id = nanny_services.user_id
WHERE (services.service_id LIKE :service_id)
GROUP BY nanny_services.user_id;
UNION ALL
SELECT user_language.user_id, userFirstName
FROM hoidja.user_language
JOIN languages on user_language.user_language_id = languages.user_language_id
JOIN tbl_users ON tbl_users.user_id = user_language.user_id
WHERE (languages.user_language_id LIKE :language_id)
GROUP BY user_language.user_id
");
$search->execute(array(
':service_id' => $services,
':language_id' => $languages,
));
$search_results = $search->fetchAll(PDO::FETCH_ASSOC);
echo json_encode($search_results);
Something like this
SELECT nanny_services.user_id, userFirstName
FROM hoidja.nanny_services
JOIN services on nanny_services.service_id = services.service_id
JOIN tbl_users ON tbl_users.user_id = nanny_services.user_id
WHERE (services.service_id = :service_id)
AND (languages.user_language_id = :language_id)
GROUP BY nanny_services.user_id;
without UNION

MySQL Selecting million records to generate urls

I currently getting a 2 million records from different tables to generate a url to create a sitemap. The script eat too much resources and use 100% of the servers performance
query
SELECT CONCAT("/url/profile/id/",u.id,"/",nickname) as url FROM users AS u
UNION ALL
Select CONCAT("url/city/", c.id, "/paramId/",p.id,"/",Replace(p.title, " ", "+"),"/",r.region_Name,"/",c.city_Name) AS url
From city c
Join region r On r.id = c.id_region
Join country country On country.id = c.id_country
cross join param p
Where country.used = 1
And p.active = 1
//i store it on an array $url_list then process for creating a sitemap..but it takes time and to much resources
//i tried to get the data by batch using LIMIT 0,50000
but getting the maxrow for paging takes time. also the code doesn't look good for i have to run a two query that has a large data
$url_list = array();
$maxrow = SELECT COUNT(*) AS max from (
SELECT CONCAT("/url/profile/id/",u.id,"/",nickname) as url FROM users AS u
UNION ALL
Select CONCAT("url/city/", c.id, "/paramId/",p.id,"/",Replace(p.title, " ", "+"),"/",r.region_Name,"/",c.city_Name) AS url
From city c
Join region r On r.id = c.id_region
Join country country On country.id = c.id_country
cross join param p
Where country.used = 1
And p.active = 1) as tmp
$limit = 50,000;
$bybatch = ceil($maxrow/$limit);
$start = 0;
for($i = 0;$i < $bybatch; $i++){
// run query and store to $result
(SELECT CONCAT("/url/profile/id/",u.id,"/",nickname) as url FROM users AS u
UNION ALL
Select CONCAT("url/city/", c.id, "/paramId/",p.id,"/",Replace(p.title, " ", "+"),"/",r.region_Name,"/",c.city_Name) AS url
From city c
Join region r On r.id = c.id_region
Join country country On country.id = c.id_country
cross join param p
Where country.used = 1
And p.active = 1 LIMIT $start,$limit);
$start += $limit;
//push to $url_list
$url_list = array_push($result);
}
//when finish i use this to create a site map
$linkCount = 1;
$fileNomb = 1;
$i = 0;
foreach ($url_list as $ul) {
$i += 1;
if ($linkCount == 1) {
$doc = new DOMDocument('1.0', 'utf-8');
$doc->formatOutput = true;
$root = $doc->createElementNS('http://www.sitemaps.org/schemas/sitemap/0.9', 'urlset');
$doc->appendChild($root);
}
$url= $doc->createElement("url");
$loc= $doc->createElement("loc", $ul['url']);
$url->appendChild($loc);
$priority= $doc->createElement("priority",1);
$url->appendChild($priority);
$root->appendChild($url);
$linkCount += 1;
if ($linkCount == 49999) {
$f = fopen($this->siteMapMulti . $fileNomb .'.xml', "w");
fwrite($f,$doc->saveXML());
fclose($f);
$linkCount = 1;
$fileNomb += 1;
}
}
Any better way to do this? or to speed up the performance?
Added
Why is this faster than sql query but consumes 1 hundred percent of the servers resources and performance
$this->db->query('SELECT c.id, c.city_name, r.region_name, cr.country_name FROM city AS c, region AS r, country AS cr WHERE r.id = c.id_region AND cr.id = c.id_country AND cr.id IN (SELECT id FROM country WHERE use = 1)');
$arrayCity = $this->db->recordsArray(MYSQL_ASSOC);
$this->db->query('SELECT id, title FROM param WHERE active = 1');
$arrayParam = $this->db->recordsArray(MYSQL_ASSOC);
foreach ($arrayCity as $city) {
foreach ($arrayParam as $param) {
$paramTitle = str_replace(' ', '+', $param['title']);
$url = 'url/city/'. $city['id'] .'/paramId/'. $param['id'] .'/'. $paramTitle .'/'. $city['region_name'] .'/'. $city['city_name'];
$this->addChild($url);
}
}
I suggest you not to use UNION and just issue two separated queries. It will speed up a query itself.
Also as you mentioned above it's good idea to get data by batches.
And finally, don't collect all data in memory. Immediately write it to file in your loop.
Just open file in beginning, write each URL entry in loop and close file in end.
— open file for writing
— count query users table
— do several selects with LIMIT in loop (as you already done)
— right here in loop while ($row = mysql_fetch_array()) write each row to file
and than repeat such algorithm for another table.
It would be useful to implement a function for writing data to file, so you can call that function and adhere to the DRY principle.

MYSQL Joins - Where Unique ID lies in 1 table

I have 2 tables I want to connect in a strange / dangerous / non-dynamic way. I don't have control over these tables. I'm trying to pull the summary from one table that contains event id but not category id but I need to reference another table to make sure that said event is in said category. This table contains both event id and cat id. I'm trying to join them but I keep getting returned nothing.
I know this is dangerous, but I also have control over the categories so I know that my category ID's will not change unless I specify. Since it auto-increments - my categories will be 1, 2, 3.
The Gist of my Tables
*events_vevent*
- ev_id
- catid
---
*events_vevdetail*
- evdet_id
- summary
My Code
$data = array();
for($i = 0; $i < 3; $i++){
$summary = array();
$query_summary = mysql_query("SELECT events_vevdetail.summary FROM
events_vevdetail, events_vevent
WHERE 'events_vevent.evdet_id = $i' LIMIT 5")
or die(mysql_error());
while(($row = mysql_fetch_array($query_summary)))
$summary[] = $row[0];
switch($i){
case 0:
$data['cat1'] = $summary;
break;
case 1:
$data['cat2'] = $summary;
break;
case 2:
$data['cat3'] = $summary;
break;
}
}
echo json_encode($data);
Explanation
so what I'm trying to do is: Since I know category 1 will always have an ID of 0, I want to pull the most recent 5 posts, but only posts in category ID 0. Same for cateogry2 and 3. Right now I'm getting empty arrays. I feel like I need 2 mysql queries (one for each table) and then compare but I'm not 100% sure and would rather do this the right way than the long way.
tl;dr is my MYSQL right?
This query will return top most 5 records from each category.
SELECT e1 . *
FROM events_vevent e1
LEFT OUTER JOIN events_vevent e2 ON
( e1.catid = e2.catid AND e1.ev_id < e2.ev_id )
GROUP BY e1.ev_id
HAVING COUNT( * ) < 5
ORDER BY catid, ev_id DESC

Categories