I need to optimise SQL queries in my PHP code used by hugely over headed web service. I'am getting list of words.
1) get word identifiers
2) foreach identifier get word as php object
3) print word details in xml
Now I have a code that takes this identifier of Word in constructor.
Then when user accesses properties is lazily loading given properties.
Web service is returning all word details so it takes all word properties.
Making many 5-10 simple sql queries each time like get native word, get foreign word, get transcription. It was done as such assuming that I one time need less info about word and the other time I need more information.
Now when my hosting provider deactivated my website as impacting too much overhead on resources of shared hosting I need to optimize it.
1) I will provide INDEX, UNIQUE where it is possible and it has't been used yet.
2) I think about replacing many simple sql queries lazily retrieving word's properties with longer joining query:
EXPLAIN SELECT nt.deutsch_id, nt.article, n.deutsch_word, ft.french_id, ft.article, f.french_word, p.part, fd.transcription, fd.definition
FROM translation_enfr ft
INNER JOIN translation_ende nt ON nt.translation_id = ft.translation_id
INNER JOIN deutsch n ON n.deutsch_id = nt.deutsch_id
INNER JOIN french f ON f.french_id = ft.french_id
INNER JOIN parts p ON p.part_id = ft.part_id
LEFT JOIN french_details fd ON fd.translation_id = ft.translation_id
WHERE ft.translation_id =2
Do you think it will be better/faster than using:
public function getNativeWord($withArticle = true) {
if(is_null($this->nativeWord)) {
$q = "SELECT {$langLabel}_word FROM {$langLabel}
WHERE {$langLabel}_id = :native_id";
}
}
And other current queries are similar:
"SELECT {$langLabel}_word FROM {$langLabel} WHERE {$langLabel}_id = :foreign_id"
"SELECT article FROM translation_en{$this->nativeLang} WHERE translation_id = :translation_id";
"SELECT parts.part FROM parts INNER JOIN translation ON parts.part_id = translation.part_id WHERE translation_id = :trans_id"
"SELECT transcription FROM {$langLabel}_details WHERE translation_id = :trans_id";
"SELECT definition FROM {$langLabel}_details WHERE translation_id = :trans_id";
I think to preload each properties of this Word object only remaining to load easily images, sentences, comments as here I have 1toMany relationship!
UPDATED 1
Output of EXPLAIN SELECT...*
Related
I have almost thousands of data to display for my reports and it makes my browser lags due to the heavy data. I think that my query is the real problem. How can I optimized my query? is there something that I should add in my query?
I am using Xampp which supports PHP7.
SELECT
`payroll_billed_units`.`allotment_code`,
`payroll_billed_units`.`category_name`,
`payroll_billed_units`.`ntp_number`,
`payroll_billed_units`.`activity`,
`payroll_billed_units`.`regular_labor`,
`payroll_sub`.`block_number`,
(SELECT
GROUP_CONCAT(DISTINCT `lot_number` SEPARATOR ', ')
FROM
`payroll_billed_units` `lot_numbers`
WHERE
`lot_numbers`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `lot_numbers`.`category_name` = `payroll_billed_units`.`category_name`
AND `lot_numbers`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `lot_numbers`.`activity` = `payroll_billed_units`.`activity`) AS `lot_numbers`,
(SELECT
COUNT(`billed`.`ntp_id`)
FROM
`regular_ntp` `billed`
WHERE
`billed`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `billed`.`category_name` = `payroll_billed_units`.`category_name`
AND `billed`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `billed`.`activity` = `payroll_billed_units`.`activity`) AS `billed`,
(SELECT
COUNT(`approved`.`id`)
FROM
`payroll_billed_units` `approved`
WHERE
`approved`.`allotment_code` = `payroll_billed_units`.`allotment_code`
AND `approved`.`category_name` = `payroll_billed_units`.`category_name`
AND `approved`.`ntp_number` = `payroll_billed_units`.`ntp_number`
AND `approved`.`activity` = `payroll_billed_units`.`activity`) AS `approved`
FROM
`payroll_billed_units`
JOIN payroll_transaction ON payroll_billed_units.billing_number =
payroll_transaction.billing_number
JOIN payroll_sub ON payroll_transaction.billing_number =
payroll_sub.billing_number
WHERE payroll_billed_units.billing_date = '2019-02-13'
AND payroll_transaction.contractor_name = 'Roy Codal' GROUP BY allotment_code, category_name, activity
I was expecting that it will load or display all my data.
The biggest problem are the dependendt sub-selects, they are responsible for a bad performance. A sub-select will be executed for EVERY ROW of the outer query. And if you cascade subs-selects, you'll quickly have a query run forever.
If any of the parts would yield only 5 resultsets, 3 sub-select would mean that the database has to run 625 queries (5^4)!
Use JOINs.
Several of your tables need this 'composite' index:
INDEX(allotment_code, category_name, ntp_number, activity) -- in any order
payroll_transaction needs INDEX(contractor_name), though it may not get used.
payroll_billed_units needs INDEX(billing_date), though it may not get used.
For further discussion, please provide SHOW CREATE TABLE for each table and EXPLAIN SELECT ...
Use simply COUNT(*) instead of COUNT(foo). The latter checks the column for being not-NULL before including it. This is usually not needed. The reader is confused by thinking that there might be NULLs.
Your GROUP BY is improper because it is missing ntp_number. Read about the sql_mode of ONLY_FULL_GROUP_BY. I bring this up because you can almost get rid of some of those subqueries.
Another issue... Because of the "inflate-deflate" nature of JOIN with GROUP BY, the numbers may be inflated. I recommend you manually check the values of the COUNTs.
Im programming a search with ZF3 and the DB module.
Everytime i use more than 1 short keyword - like "49" and "am" or "1" and "is" i get this error:
Statement could not be executed (HY000 - 2006 - MySQL server has gone away)
Using longer keywords works perfectly fine as long as i dont use 2 or more short keywords.
The problem only occurs on the live server its working fine on the local test server.
The project table has ~2200 rows with all kind of data the project_search table has 17000 rows with multiple entries for each project , each looking like:
id, projectid, searchtext
The searchtext Column is fulltext. Here the relevant part of the php code:
$sql = new Sql($this->db);
$select = $sql->select(['p'=>'projects']);
if(isset($filter['search'])) {
$keywords = preg_split('/\s+/', trim($filter['search']));
$join = $sql->select('project_search');
$join->columns(['projectid' => new Expression('DISTINCT(projectid)')]);
$join->group("projectid");
foreach($keywords as $keyword) {
$join->having(["LOCATE('$keyword', GROUP_CONCAT(searchtext))"]);
}
$select->join(
["m" => $join],
"m.projectid = p.id",
['projectid'],
\Zend\Db\Sql\Select::JOIN_RIGHT
);
}
Here the resulting Query:
SELECT p.*, m.projectid
FROM projects AS p
INNER JOIN (
SELECT projectid
FROM project_search
GROUP BY projectid
HAVING LOCATE('am', GROUP_CONCAT(searchtext))
AND LOCATE('49', GROUP_CONCAT(searchtext))
) AS m
ON m.projectid = p.id
GROUP BY p.id
ORDER BY createdAt DESC
I rewrote the query using "MATCH(searchtext) AGAINST('$keyword)" and "searchtext LIKE '%keyword%' with the same result.
The problem seems to be with the live mysql server how can i debug this ?
[EDIT]
After noticing that the error only occured in a special view which had other search related queries - each using multiple joins (1 join / keyword) - i merged those queries and the error was gone. The amount of queries seemed to kill the server.
Try refactoring your inner query like so.
SELECT a.projectid
FROM (
SELECT DISTINCT projectid
FROM projectsearch
WHERE searchtext LIKE '%am%'
) a
JOIN (
SELECT DISTINCT projectid
FROM projectsearch
WHERE searchtext LIKE '%49%'
) b ON a.projectid = b.projectid
It should give you back the same set of projectid values as your inner query. It gives each projectid value that has matching searchtext for both search terms, even if those terms show up in different rows of project_search. That's what your query does by searching GROUP_CONCAT() output.
Try creating an index on (searchtext, projectid). The use of column LIKE '%sample' means you won't be able to random-access that index, but the two queries in the join may still be able to scan the index, which is faster than scanning the table. To add that index use this command.
ALTER TABLE project_search ADD INDEX project_search_text (searchtext, projectid);
Try to do this in a MySQL client program (phpmyadmin for example) rather than directly from your php program.
Then, using the MySQL client, test the inner query. See how long it takes. Use EXPLAIN SELECT .... to get an explanation of how MySQL is handling the query.
It's possible your short keywords are returning a ridiculously high number of matches, and somehow overwhelming your system. In that case you can put a LIMIT 1000 clause or some such thing at the end of your inner query. That's not likely, though. 17 kilorows is not a large number.
If that doesn't help your production MySQL server is likely misconfigured or corrupt. If I were you I would call your hosting service tech support, somehow get past the front-line support agent (who won't know anything except "reboot your computer" and other such foolishness), and tell them the exact times you got the "gone away" message. They'll be able to check the logs.
Pro tip: I'm sure you know the pitfalls of using LIKE '%text%' as a search term. It's not scalable because it's not sargable: it can't random access an index. If you can possibly redesign your system, it's worth your time and effort.
You could TRY / CATCH to check if you get a more concrete error:
BEGIN TRY
BEGIN TRANSACTION
--Insert Your Queries Here--
COMMIT
END TRY
BEGIN CATCH
DECLARE #ErrorMessage NVARCHAR(4000);
DECLARE #ErrorSeverity INT;
DECLARE #ErrorState INT;
SELECT
#ErrorMessage = ERROR_MESSAGE(),
#ErrorSeverity = ERROR_SEVERITY(),
#ErrorState = ERROR_STATE();
IF ##TRANCOUNT > 0
ROLLBACK
RAISERROR (#ErrorMessage, -- Message text.
#ErrorSeverity, -- Severity.
#ErrorState -- State.
);
END CATCH
Although because you are talking about short words and fulltext it seems to me it must be related to StopWords.
Try running this query from both your dev server and production server and check if there are any differences:
SELECT * FROM INFORMATION_SCHEMA.INNODB_FT_DEFAULT_STOPWORD;
Also check in my.ini (if that is the config file) text file if these are set to:
ft_stopword_file = ""
ft_min_word_len = 1
As stated in my EDIT the problem wasnt the query from the original Question, but some other queries using the search - parameter as well. Every query had a part like follows :
if(isset($filter['search'])) {
$keywords = preg_split('/\s+/', trim($filter['search']));
$field = 1;
foreach($keywords as $keyword) {
$join = $sql->select('project_search');
$join->columns(["pid$field" => 'projectid']);
$join->where(["LOCATE('$keyword', searchtext)"]);
$join->group("projectid");
$select->join(
["m$field" => $join],
"m$field.pid$field = p.id"
);
$field++;
}
}
This resulted in alot of queries with alot of resultrows killing the mysql server eventually. I merged those Queries into the first and the error was gone.
I've got a bit of a problem with my code. I'm sure that it is something simple, but I just can't figure it out! I have been on tons of forums and have read several books... but every answer that I have worked to has failed. I almost guarantee that it's the way that I am using my syntax (and yes I know... procedural PHP is not really used anymore) but I am really a bit of a newbie to this and I am just trying to pick up the basics before moving onto OOP and PDO connections.
Could you please help me? At the moment I can get the user to select their date from the date picker and the results specifically from that date only will return... only problem is that the event is displaying the event_id as opposed to the name of the event that it relates to (1 = 5km run) for example.
Somehow I need to access the events table and pull the row that relates to that specific event_id.
I have normalized my database, and according to my tutor it looks ok. To give you an idea what it looks like - logins table (all user logins details), results table (a history of submitted events) events table (the events themselves).
On the results table the foreign keys are logins_id and the event_id. The primary key is the results_id in the results table and the only data stored here is the time and data (individual columns).
<?php // -----Stage 1. On submission of the form run the following -----//
if (isset($_POST['submit_d'])) {
$mydate = $_POST ['MyDate'];
$my = preg_replace('/[^a-zA-Z0-9]+/', ' ', $mydate);
if ($mydate) {
$result = mysql_query("SELECT * FROM logins WHERE username = '$username' LIMIT 1");
//This function will take the above query and create an array...
while($row = mysql_fetch_array($result))
{
//With the array created above, I can create variables (left) with the outputted array (right)
$logins_id3 = $row['logins_id'];
}
$sql = "SELECT * FROM results where $logins_id3 AND date = $mydate ";
/* ----- Here is the code that I want to use in conjunction with the above statement --->
$query = "SELECT logins.username,events.event,results.time,results.date,logins.age,logins.gender
FROM logins INNER JOIN results ON logins.logins_id=results.logins_id INNER JOIN events ON results.event_id=events.event_id
ORDER BY time ASC LIMIT 10";
*/
$resultz = mysql_query($sql);
if( mysql_num_rows($resultz) ) {
while ($row = mysql_fetch_array($resultz)) {
echo "<table><tr><th>Username</th><th>Event</th><th>Time (HH:MM:SS)</th><th>Date (YY/MM/DD)</th><th>Age</th><th>Gender</th>
</tr><tr><td>".$username."</td>"
."<td>".$row['event_id']."</td>"."<td>".$row['time']."</td>"." <td>".$row['date']."</td>"."<td>".$row['age']."</td>".
"<td>" $row['gender']."</td></tr></table>";
}
}
}
}
?>
The other thing I would like to do.. although this is not crucial, is to strip special characters from the input. Basically I'm using a jquery calendar picker and I want the user to be able to select their date in the 2014-05-26 format and the php to remove the - before it is submitted to the database, that way it doesn't effect the users experience but it will work with my current code.
Anyways sorry to waffle on, any help on either of these matters would be much appreciated!
Yours Sincerely:
Peter Scales.
You can use a join to get the data that relates to the event ID:
SELECT * FROM results r LEFT JOIN events e ON r.event_id = e.event_id WHERE ...
You can then select where "e.event_id = $event_id"; and the rest of your query logic.
You can also filter out any unwanted characters by using preg_replace: http://ar2.php.net/preg_replace
I'm using the following query to join two tables. Table gz_topics features basic information, such as page Title and Subtitle, while the articles for each page are stored in table gz_articles_topics_intro. GZ.URL is actually a variable that matches each page's URL, but I'm using "Birds" here as an example.
$Zext = mysql_fetch_assoc(mysql_query("SELECT GZ.N, GZ.URL, GZ.Title, GZ.Live,
AI.URL, AI.Article, AI.Pagedex
FROM gz_topics GZ
LEFT JOIN gz_articles_topics_intro AI ON AI.URL = GZ.URL
WHERE GZ.URL LIKE 'Birds' AND GZ.Live = 1"));
It works fine. The problems begin when I put my articles in three separate tables and try to join them...
$Zext = mysql_fetch_assoc(mysql_query("SELECT GZ.N, GZ.URL, GZ.Title, GZ.Live,
Art.URL, Art.Article, Art.Pagedex,
AI.URL, AI.Article, AI.Pagedex, AN.URL, AN.Article, AN.Pagedex
FROM gz_topics GZ
LEFT JOIN gz_articles_topics Art ON Art.URL = GZ.URL
LEFT JOIN gz_articles_topics_intro AI ON AI.URL = GZ.URL
LEFT JOIN gz_articles_topics_names AN ON AN.URL = GZ.URL
WHERE GZ.URL LIKE '$MyURL' AND GZ.Live = 1"));
When I paste it into SQL, it works just fine, displaying all the data, including the article. But no values display on my page.
This is the code I use to display the article:
$Article = $Zext['Article'];
echo $Article;
I've tried inner joins and outer joins, while loops, etc., but nothing seems to work. My PHP/MySQL skills are intermediate, and I don't have a clue what the problem is because, as I said, it works fine when I paste it into SQL, and I don't see any error messages.
The problem must be right under my nose, because this looks pretty simple - even for me. ;)
Thanks for any tips.
On edit: I fixed one mistake, and my query is now displaying items from the table gz_topics, like the page title. However, it still doesn't display the article.
Problem 1) You have three columns with the same name "Article"
Art.Article, AI.Article, AN.Article
You may rename them like this:
Art.Article AS Article1, AI.Article AS Article2, AN.Article AS Article3
And then you can use the appropriate column (1, 2 or 3):
$Article = $Zext['Article1'];
Problem 2) If the contents are plain text (not HTML), you should escape the output:
echo htmlentities($Article);
I have a situation where lets say i'm trying to get the information about some food. Then I need to display all the information plus all the ingredients in that food.
With my query, i'm getting all the information in an array but only the first ingredient...
myFoodsArr =
[0]
foodDescription = "the description text will be here"
ratingAverage = 0
foodId = 4
ingredient = 1
ingAmount = 2
foodName = "Awesome Food name"
typeOfFood = 6
votes = 0
I would like to get something back like this...
myFoodsArr =
[0]
foodDescription = "the description text will be here"
ratingAverage = 0
foodId = 4
ingArr = {ingredient: 1, ingAmount: 4}, {ingredient: 3, ingAmount: 2}, {ingredient: 5, ingAmount: 1}
foodName = "Awesome Food name"
typeOfFood = 6
votes = 0
This is the query im working with right now. How can I adjust this to return the food ID 4 and then also get ALL the ingredients for that food? All while at the same time doing other things like getting the average rating of that food?
Thanks!
SELECT a.foodId, a.foodName, a.foodDescription, a.typeOfFood, c.ingredient, c.ingAmount, AVG(b.foodRating) AS ratingAverage, COUNT(b.foodId) as tvotes
FROM `foods` a
LEFT JOIN `foods_ratings` b
ON a.foodId = b.foodId
LEFT JOIN `foods_ing` c
ON a.foodId=c.foodId
WHERE a.foodId=4
EDIT:
Catcall introduced this concept of "sub queries" I never heard of, so I'm trying to make that work to see if i can do this in 1 query easily. But i just keep getting a return false. This is what I was trying with no luck..
//I changed some of the column names to help them be more distinct in this example
SELECT a.foodId, a.foodName, a.foodDescription, a.typeOfFood, AVG(b.foodRating) AS ratingAverage, COUNT(b.foodId) as tvotes
FROM foods a
LEFT JOIN foods_ratings b ON a.foodId = b.foodId
LEFT JOIN (SELECT fId, ingredientId, ingAmount
FROM foods_ing
WHERE fId = 4
GROUP BY fId) c ON a.foodId = c.fId
WHERE a.foodId = 4";
EDIT 1 more thing related to ROLANDS GROUP_CONCAT/JSON Idea as a solution 4 this
I'm trying to make sure the JSON string im sending back to my Flash project is ready to be properly parsed Invalid JSON parse input. keeps popping up..
so im thinking i need to properly have all the double quotes in the right places.
But in my MySQL query string, im trying to escape the double quotes, but then it makes my mySQL vars not work, for example...
If i do this..
GROUP_CONCAT('{\"ingredient\":', \"c.ingredient\", ',\"ingAmount\":', \"c.ingAmount\", '}')`
I get this...
{"ingredient":c.ingredient,"ingAmount":c.ingAmount},{"ingredient":c.ingredient,"ingAmount":c.ingAmount},{"ingredient":c.ingredient,"ingAmount":c.ingAmount}
How can i use all the double quotes to make the JSON properly formed without breaking the mysql?
This should do the trick:
SELECT food_ingredients.foodId
, food_ingredients.foodName
, food_ingredients.foodDescription
, food_ingredients.typeOfFood
, food_ingredients.ingredients
, AVG(food_ratings.food_rating) food_rating
, COUNT(food_ratings.foodId) number_of_votes
FROM (
SELECT a.foodId
, a.foodName
, a.foodDescription
, a.typeOfFood
, GROUP_CONCAT(
'{ingredient:', c.ingredient,
, ',ingAmount:', c.ingAmount, '}'
) ingredients
FROM foods a
LEFT JOIN foods_ing c
ON a.foodsId = c.foodsId
WHERE a.foodsId=4
GROUP BY a.foodId
) food_ingredients
LEFT JOIN food_ratings
ON food_ingredients.foodId = food_ratings.foodId
GROUP BY food_ingredients.foodId
Note that the type of query you want to do is not trivial in any SQL-based database.
The main problem is that you have one master (food) with two details (ingredients and ratings). Because those details are not related to each other (other than to the master) they form a cartesian product with each other (bound only by their relationship to the master).
The query above solves that by doing it in 2 steps: first, join to the first detail (ingredients) and aggregate the detail (using group_concat to make one single row of all related ingredient rows), then join that result to the second detail (ratings) and aggregate again.
In the example above, the ingredients are returned in a structured string, exactly like it appeared in your example. If you want to access the data inside PHP, you might consider adding a bit more syntax to make it a valid JSON string so you can decode it into an array using the php function json_decode(): http://www.php.net/manual/en/function.json-decode.php
To do that, simply change the line to:
CONCAT(
'['
, GROUP_CONCAT(
'{"ingredient":', c.ingredient
, ',"ingAmount":', c.ingAmount, '}'
)
, ']'
)
(this assumes ingredient and ingAmount are numeric; if they are strings, you should double quote them, and escape any double quotes that appear within the string values)
The concatenation of ingredients with GROUP_CONCAT can lead to problems if you keep a default setting for the group_concat_max_len server variable. A trivial way to mitigate that problem is to set it to the maximum theoretical size of any result:
SET group_concat_max_len = ##max_allowed_packet;
You can either execute this once after you open the connection to mysql, and it will then be in effect for the duration of that session. Alternatively, if you have the super privilege, you can change the value across the board for the entire MySQL instance:
SET GLOBAL group_concat_max_len = ##max_allowed_packet;
You can also add a line to your my.cnf or my.ini to set group_concat_max_lenght to some arbitrary large enough static value. See http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_group_concat_max_len
One obvious solution is to actually perform two queries:
1) get the food
SELECT a.foodId, a.foodName, a.foodDescription, a.typeOfFood
FROM `foods` a
WHERE a.foodsId=4
2) get all of its ingredients
SELECT c.ingredient, c.ingAmount
FROM `foods_ing` c
WHERE c.foodsId=4
This approach has the advantage that you don't duplicate data from the "foods" table into the result. The disadvantage is that you have to perform two queries. Actually you have to perform one extra query for each "food", so if you want to have a listing of foods with all their ingredients, you would have to do a query for each of the food record.
Other solutions usually have many disadvantages, one of them is using GROUP_CONCAT function, but it has a tough limit on the length of the returned string.
When you compare MySQL's aggregate functions and GROUP BY behavior to SQL standards, you have to conclude that they're simply broken. You can do what you want in a single query, but instead of joining directly to the table of ratings, you need to join on a query that returns the results of the aggregate functions. Something along these lines should work.
select a.foodId, a.foodName, a.foodDescription, a.typeOfFood,
c.ingredient, c.ingAmount,
b.numRatings, b.avgRating
from foods a
left join (select foodId, count(foodId) numRatings, avg(foodRating) avgRating
from foods_ratings
group by foodId) b on a.foodId = b.foodId
left join foods_ing c on a.foodId = c.foodId
order by a.foodId