I'm trying to learn best practices with multiple SQL queries in PHP and mySQL. Not sure if I should have two separate queries or if I should join (or UNION?) my queries into one.
Can I do the following or is there another way to accomplish the same thing?
It's important to note that my first query is pulling a list of questions and answers for a quiz. The second query is pulling a list of score profiles that I'll be using to assign to users based on their quiz score. The only relationship between these two tables is that both use the same foreign key linked to an ID in my "quiz" table. I wouldn't need to JOIN these tables side by side, but I think I would probably need to UNION them so my second query results appear after the first.
Now if I should union my queries into one, I have no idea how I would loop through the query results to create my array. When I loop through each result, I would need some conditional logic that does something with my first query results and something different with my second query results. How would I write those conditions?
Here is what I'm doing now that seems to work fine as two separate queries...
...query the database for the first time to get my quiz questions and answers:
$result_quiz = mysql_query("
SELECT quiz_question.question_text, quiz_question.id, quiz_answer.answer_text, quiz_answer.points
FROM quiz
JOIN quiz_question ON (quiz.id = quiz_question.quiz_id)
JOIN quiz_answer ON (quiz_question.id = quiz_answer.quiz_question_id)
WHERE quiz.id = $id;
");
...then based on the above query, build my array of questions and answers:
$quiz = array();
while ($row = mysql_fetch_assoc($result_quiz)) {
if (!isset($quiz['questions']['q'.$row['id'].''])) {
$quiz['questions']['q'.$row['id'].''] = array(
'question' => $row['question_text'],
'answers' => array()
);
}
$quiz['questions']['q'.$row['id'].'']['answers'][] = array('answer' => $row['answer_text'],'points' => $row['points']);
}
...then query the database for the second time to get a list of score profiles:
$result_profile = mysql_query("
SELECT quiz_profile.name, quiz_profile.description, quiz_profile.min_points, quiz_profile.max_points
FROM quiz
JOIN quiz_profile ON (quiz.id = quiz_profile.quiz_id)
WHERE quiz.id = $id;
");
...then loop through my second query so I can append the score profiles to my array:
while ($row = mysql_fetch_assoc($result_profile)) {
$quiz['profiles'][] = array('name' => $row['name'],'description' => $row['description'],'min_points' => $row['min_points'],'max_points' => $row['max_points']);
}
Is this the best approach?
Or should I combine my queries into one using UNION?
If so, how can I write my conditions to know what results I'm looping through?
Thanks SO much!
I suggest sticking with the current approach, unless there is a serious problem with performance - combining the two queries might improve total throughput (slightly), but would significantly increase the complexity of the code required to display it.
I recommend to take a step back at the design of your app, not the code itself: If you go down the UNION road, you have to sacrifice all isolation between your questions-code and your scoreprofiles-code.
I say, this is not a good thing: What if one day the score profiles are in some way recalculated or trabsformed between db and user? Is it intuitive, that for that you have to touch the function getting the questions from the db?
In PHP, there is a rule of thumb, which (as all rules of thumb) is not 100% perfect, but quite a start: It should be possible to have all code relating to one field of functionality be located in one file, the includes and requires mirroring the dependencies.
Related
I'm not very experienced with more advanced MySQL query stuff.. (mostly basic queries, return and parse response..etc)
However.. I am not clear on the correct approach when I need multiple things (responses) from the database.. Is there a way to get these things from the single query? or do I need to do a new query for each time?
Background:
I use PDO to do a SELECT statement
ie:
$getAllVideos_sql = "SELECT * as FROM $tableName WHERE active IS NOT NULL OR active != 'no' ORDER BY topic, speaker_last, title;";
$getAllVideos_stmt = $conn->prepare($getAllVideos_sql);
$getAllVideos_stmt->execute();
$getAllVideos_stmt->setFetchMode(PDO::FETCH_ASSOC);
$results = $getAllVideos_stmt->fetch(PDO::FETCH_ASSOC);
//parse as I see fit
This gives me my 'chunk of data' that I can pick apart and display as I want.
However.. I want to also be able to give some stats (totals)
For the total (distinct) 'topics'.. as well as total count for the 'titles' (should all be unique by default)
Do I need to do another query, prepare, execute, setFetchMode, fetch all over again?
Is this the proper way to do this? Or is there a way to crib off the initial commands that are already in play?
To be clear, I'm not really looking for a query... I'm looking to understand the proper way one does this.. when they need several pieces of data like I do? multiple queries and executions..etc?
Or maybe it can and -should- be done in one snippet? With an adjustment to the query itself to return sub select/queries info?
this isnt the correct syntax, because it only returns 1 record..(but the total topic count seems to be correct, even though I only get 1 record returned)
SELECT *, count(DISTINCT topic)as totalTopics, count(DISTINCT title)as totalTitles FROM $tableName;
Maybe this the more proper approach? Try to include these totals/details in the main query to pick out?
Hope this makes sense.
Thanks
I don't think you're going to get anything very clean that'll do this, however something like this might work:
SELECT * from $Table t
INNER JOIN (
SELECT COUNT(DISTINCT Topic) as TotalTopics FROM $Table
) s ON 1 = 1
INNER JOIN (
SELECT COUNT(DISTINCT Title) as TotalTitles FROM $Table
) f ON 1 = 1
WHERE ( Active IS NOT NULL ) AND Active != 'no'
Especially with web applications, many people are regularly doing counts or other aggregations somewhere along the way. Sometimes if it is a global context such as all topics for all users, having some stored aggregates helps rather than requerying all record counts every time.
Example. If you have a table with a list of "topics", have a column in there for uniqueTitleCount. Then, based on a trigger, when a new title is added to a topic, the count is automatically updated by adding 1. You can pre-populate this column by doing a correlated update to said "topics" table, then once the trigger is set, you can just have that column.
This also works as I see many times that people want "the most recent". If your system has auto-increment IDs in the tables, similarly, have the most recent ID created for a given topic, or even most recent for a given title/document/thread so you don't have to keep doing something like.
select documentID, other_stuff
from sometable
where documentID in ( select max( documentID )
from sometable
where the title = 'something' )
Use where these make sense then your optimization pull-downs get easier to handle. You could even have a counter per document "title" and even a most recent posting date so they can quickly be sorted based on interest, frequency of activity, whatever.
In a site I maintain I have a need to query the same table (articles) twice, once for each category of article. AFAIT there are basically two ways of doing this (maybe someone can suggest a better, third way?):
Perform the db query twice, meaning the db server has to sort through the entire table twice. After each query, I iterate over the cursor to generate html for a list entry on the page.
Perform the query just once and pull out all the records, then sort them into two separate arrays. After this, I have to iterate over each array separately in order to generate the HTML.
So it's this:
$newsQuery = $mysqli->query("SELECT * FROM articles WHERE type='news' ");
while($newRow = $newsQuery->fetch_assoc()){
// generate article summary in html
}
// repeat for informational articles
vs this:
$query = $mysqli->query("SELECT * FROM articles ");
$news = Array();
$info = Array();
while($row = $query->fetch_assoc()){
if($row['type'] == "news"){
$news[] = $row;
}else{
$info[] = $row;
}
}
// iterate over each array separate to generate article summaries
The recordset is not very large, current <200 and will probably grow to 1000-2000. Is there a significant different in the times between the two approaches, and if so, which one is faster?
(I know this whole thing seems awfully inefficient, but it's a poorly coded site I inherited and have to take care of without a budget for refactoring the whole thing...)
I'm writing in PHP, no framework :( , on a MySql db.
Edit
I just realized I left out one major detail. On a given page in the site, we will display (and thus retrieve from the db) no more than 30 records at once - but here's the catch: 15 info articles, and 15 news articles. On each page we pull the next 15 of each kind.
You know you can sort in the DB right?
SELECT * FROM articles ORDER BY type
EDIT
Due to the change made to the question, I'm updating my answer to address the newly revealed requirement: 15 rows for 'news' and 15 rows for not-'news'.
The gist of the question is the same "which is faster... one query to two separate queries". The gist of the answer remains the same: each database roundtrip incurs overhead (extra time, especially over a network connection to a separate database server), so with all else being equal, reducing the number database roundtrips can improve performance.
The new requirement really doesn't impact that. What the newly revealed requirement really impacts is the actual query to return the specified resultset.
For example:
( SELECT n.*
FROM articles n
WHERE n.type='news'
LIMIT 15
)
UNION ALL
( SELECT o.*
FROM articles o
WHERE NOT (o.type<=>'news')
LIMIT 15
)
Running that statement as a single query is going to require fewer database resources, and be faster than running two separate statements, and retrieving two disparate resultsets.
We weren't provided any indication of what the other values for type can be, so the statement offered here simply addresses two general categories of rows: rows that have type='news', and all other rows that have some other value for type.
That query assumes that type allows for NULL values, and we want to return rows that have a NULL for type. If that's not the case, we can adjust the predicate to be just
WHERE o.type <> 'news'
Or, if there are specific values for type we're interested in, we can specify that in the predicate instead
WHERE o.type IN ('alert','info','weather')
If "paging" is a requirement... "next 15", the typical pattern we see applied, LIMIT 30,15 can be inefficient. But this question isn't asking about improving efficiency of "paging" queries, it's asking whether running a single statement or running two separate statements is faster.
And the answer to that question is still the same.
ORIGINAL ANSWER below
There's overhead for every database roundtrip. In terms of database performance, for small sets (like you describe) you're better off with a single database query.
The downside is that you're fetching all of those rows and materializing an array. (But, that looks like that's the approach you're using in either case.)
Given the choice between the two options you've shown, go with the single query. That's going to be faster.
As far as a different approach, it really depends on what you are doing with those arrays.
You could actually have the database return the rows in a specified sequence, using an ORDER BY clause.
To get all of the 'news' rows first, followed by everything that isn't 'news', you could
ORDER BY type<=>'news' DESC
That's MySQL short hand for the more ANSI standards compliant:
ORDER BY CASE WHEN t.type = 'news' THEN 1 ELSE 0 END DESC
Rather than fetch every single row and store it in an array, you could just fetch from the cursor as you output each row, e.g.
while($row = $query->fetch_assoc()) {
echo "<br>Title: " . htmlspecialchars($row['title']);
echo "<br>byline: " . htmlspecialchars($row['byline']);
echo "<hr>";
}
Best way of dealing with a situation like this is to test this for yourself. Doesn't matter how many records do you have at the moment. You can simulate whatever amount you'd like, that's never a problem. Also, 1000-2000 is really a small set of data.
I somewhat don't understand why you'd have to iterate over all the records twice. You should never retrieve all the records in a query either way, but only a small subset you need to be working with. In a typical site where you manage articles it's usually about 10 records per page MAX. No user will ever go through 2000 articles in a way you'd have to pull all the records at once. Utilize paging and smart querying.
// iterate over each array separate to generate article summaries
Not really what you mean by this, but something tells me this data should be stored in the database as well. I really hope you're not generating article excerpts on the fly for every page hit.
It all sounds to me more like a bad architecture design than anything else...
PS: I believe sorting/ordering/filtering of a database data should be done on the database server, not in the application itself. You may save some traffic by doing a single query, but it won't help much if you transfer too much data at once, that you won't be using anyway.
I am looking for some input on querying multiple tables,
I currently have a list which contains the day, (each day the reports where made.)
$list = mysql_query("SELECT * FROM list ORDER BY id");
while ($row = mysql_fetch_assoc($list)){
$drop_list[] = $row['day'];
}
My end goal is to create a query which checks a unique row from each table,
I was thinking arround the lines of something like this.
foreach ($drop_list as $v) {
$daily = mysql_query("SELECT * FROM $v WHERE ID = 1");
while ($row = mysql_fetch_assoc($daily)){
$id = $row['id'];
$name = $row['name'];
$age = $row['age'];
$day = $row['day'];
}
echo "<tr><td>$id</td><td>$name</td><td>$age</td><td>$day</td></tr>";
}
Then put that into a function and echo it out in between the table tag.
I am sure the code works, (Have not tested yet Typing this from tablet) but was curious if using foreach item in array query the data from DB and echo it out to give me the daily results for the id in array?
Also curious if other have different method to accomplish this?
The short answer
Running several SQL queries inside a foreach loop will hurt performance. There is properly a better solution where everything is fetched in one query. Database queries are expensive and should be optimized as much as possible to reduce loading times of the webpage and save resources for other simultaneous requests. You can download the MySQL Workbench to help write queries as well as optimize them using the analyzer tool. There are plenty of tutorials on how to use this program around the web.
A possible solution
I assume you know the tables which you want to query and that the list of them stays the same for long periods of time. I would then fetch everything inside one query using multiple SELECT statements and the UNION keyword. This assumes the columns inside the different tables are the same. By looking at the code it seems they all declare the required columns.
SELECT * FROM table1 WHERE id = 1
UNION ALL
SELECT * FROM table2 WHERE id = 1
UNION ALL
SELECT * FROM table3 WHERE id = 1
This will fetch every single row from each of the listed tables where the id equals 1. By appending the ALL keyword to the union statement we assure duplicate rows from all the tables are also returned.
One big disadvantage of this solution is that we have no reference to from which table each row originates from. If this is required some more complex SQL queries are properly necessary, but I would still recommend combining the queries into one.
Important!
Please note that the mysql_* functions are deprecated. They are not supported anymore and security holes are not patched. I strongly recommend switching to the PDO or MySQLi extensions. They provide the better solutions in security and performance for PHP.
Side note
By looking at your code I really do not understand, why you have several tables all declaring the same columns? This seems redundant to me, but maybe I lack some more insight. It would be more effective to have only one table to maintain.
I hope this can help guide you, happy coding!
I have a database design here that looks this in simplified version:
Table building:
id
attribute1
attribute2
Data in there is like:
(1, 1, 1)
(2, 1, 2)
(3, 5, 4)
And the tables, attribute1_values and attribute2_values, structured as:
id
value
Which contains information like:
(1, "Textual description of option 1")
(2, "Textual description of option 2")
...
(6, "Textual description of option 6")
I am unsure whether this is the best setup or not, but it is done as such per requirements of my project manager. It definitely has some truth in it as you can modify the text easily now without messing op the id's.
However now I have come to a page where I need to list the attributes, so how do I go about there? I see two major options:
1) Make one big query which gathers all values from building and at the same time picks the correct textual representation from the attribute{x}_values table.
2) Make a small query that gathers all values from the building table. Then after that get the textual representation of each attribute one at a time.
What is the best option to pick? Is option 1 even faster as option 2 at all? If so, is it worth the extra trouble concerning maintenance?
Another suggestion would be to create a view on the server with only the data you need and query from that. That would keep the work on the server end, and you can pull just what you need each time.
If you have a small number of rows in attributes table, then I suggest to fetch them first, fetch all of them! store them into some array using id as index key in array.
Then you can proceed with building data, now you just have to use respective array to look for attribute value
I would recommend something in-between. Parse the result from the first table in php, and figure out how many attributes you need to select from each attribute[x]_values table.
You can then select attributes in bulk using one query per table, rather than one query per attribute, or one query per building.
Here is a PHP solution:
$query = "SELECT * FROM building";
$result = mysqli_query(connection,$query);
$query = "SELECT * FROM attribute1_values";
$result2 = mysqli_query(connection,$query);
$query = "SELECT * FROM attribute2_values";
$result3 = mysqli_query(connection,$query);
$n = mysqli_num_rows($result);
for($i = 1; $n <= $i; $i++) {
$row = mysqli_fetch_array($result);
mysqli_data_seek($result2,$row['attribute1']-1);
$row2 = mysqli_fetch_array($result2);
$row2['value'] //Use this as the value for attribute one of this object.
mysqli_data_seek($result3,$row['attribute2']-1);
$row3 = mysqli_fetch_array($result3);
$row3['value'] //Use this as the value for attribute one of this object.
}
Keep in mind that this solution requires that the tables attribute1_values and attribute2_values start at 1 and increase by 1 every single row.
Oracle / Postgres / MySql DBA here:
Running a query many times has quite a bit of overhead. There are multiple round trips to the db, and if it's on a remote server, this can add up. The DB will likely have to parse the same query multiple times in MySql which will be terribly inefficient if there are tons of rows. Now, one thing that your PHP method (multiple queries) has as an advantage is that it'll use less memory as it'll release the results as they're no longer needed (if you run the query as a nested loop that is, but if you query all the results up front, you'll have a lot of memory overhead, depending on the table sizes).
The optimal result would be to run it as 1 query, and fetch the results 1 at a time, displaying each one as needed and discarding it, which can reek havoc with MVC frameworks unless you're either comfortable running model code in your view, or run small view fragments.
Your question is very generic and i think that to get an answer you should give more hints to how this page will look like and how big the dataset is.
You will get all the buildings with theyr attributes or just one at time?
Cause your data structure look like very simple and anything more than a raspberrypi can handle it very good.
If you need one record at time you don't need any special technique, just JOIN the tables.
If you need to list all buildings and you want to save db time you have to measure your data.
If you have more attribute than buildings you have to choose one way, if you have 8 attributes and 2000 buildings you can think of caching attributes in an array with a select for each table and then just print them using the array. I don't think you will see any speed drop or improvement with so simple tables on a modern computer.
$att1[1]='description1'
$att1[2]='description2'
....
Never do one at a time queries, try to combine them into a single one.
MySQL will cache your query and it will run much faster. PhP loops are faster than doing many requests to the database.
The query cache stores the text of a SELECT statement together with the corresponding result that was sent to the client. If an identical statement is received later, the server retrieves the results from the query cache rather than parsing and executing the statement again.
http://dev.mysql.com/doc/refman/5.1/en/query-cache.html
So, basically, I have a MySQL table called 'topics' and another one called 'replies', for example. In table 'topics', there's a field called 'relforum' which relates this topic to a forum section. And in the table 'replies', there's a field called 'reltopic', which relates the reply to a topic. Both tables have an id field, auto_increment primary key.
Now, I want to select all replies from a certain forum. Since 'replies' has no 'relforum' field, my way would be:
Select all topics with 'relforum' equal to that certain forum and loop through them
While in the loop, select all replies from the topic that is being 'processed' right now
Merge all fetch_array results in one multidimensional array, then loop through them.
That's something like this:
$query = mysql_query("SELECT * FROM `topics` WHERE `relforum` = '1'");
while($array = mysql_fetch_array($query)) {
$temp = mysql_query("SELECT * FROM `replies` WHERE `reltopic` = {$array['id']}");
$results[] = mysql_fetch_array($temp);
}
Is there a way to merge all that into fewer queries? Because this process would basically run one query per topic in that forum plus one. That would be too much :P
Adding the relforum field to the replies table is a solution (I'm still designing the DB Part so it's not a problem to add it), but I would like to see if there's a solution.
I'm really not good at SQL things, I only know the basic SELECT/INSERT/UPDATE, and I usually generate the last two ones using PHPMyAdmin, so... I guess I need some help.
Thanks for reading!
You need to learn to use joins. The link below is for SQL server but the theory for mySQl is pretty much the same for basic joins. Please do not use comma-based joins as they are 18 years outdated and are a porr pracitce. Learn to use the ANSII standard joins.
http://www.tek-tips.com/faqs.cfm?fid=4785
In accessing a database, you almost never want to use any looping. Databases are designed to perform best when asked to operate on sets of data not individual rows. So you need to stop thinking about looping and start thinking about the set of the data you need.
SELECT
r.*
FROM
replies r
INNER JOIN
topics t
ON
r.reltopic = t.id
WHERE
t.relforum = 1;
You basically need a join of two tables.
SELECT * FROM `replies`, `topics` WHERE `replies`.`reltopic` = `topics`.`id` AND `topics`.`relforum` = '1';
SELECT r.* FROM replies r, topics t
WHERE t.relforum = 1 AND r.reltopic = t.id
get rid of the backquotes. they're nonstandard and clutter the code
Yes, you should use a join here. However you will need to take greater care processing your result set.
Joins are the essential query in a relational database schema. Add them to your arsenal of knowledge :)