I've spent far too long trying to come up with a good solution to this problem on my own. Haven't found any good answers to help, though I've tried meshing solutions from different answers without luck.
Goal: Present a paginated Projects page containing data from multiple
tables.
I'm using several joins in a single MYSQL query, because I wanted to avoid running queries in a PHP loop. That alone has introduced a fetch_assoc problem, where I had to build my own results array from the while(fetch_assoc) loop because of the repetitive rows resulting from the query. However, the next problem is that the query runs through a pagination class which I use to generate pagination links and get query data like num_rows and such.
So right now, I have a page showing the data I want, but the pagination class is showing me the wrong amount of rows and pagination links. Not sure what the best way is around this problem. Hoping to get some ideas from one of you brilliant experts!
Code is basically:
$sql = "...."
$projects = new pager($sql)
echo $projects->num_rows
while($projects->fetch_assoc){
build new array from data
$project_array[][][]
}
foreach($project_array){
display data from all tables...
}
$projects->showPageLinks
This SQLFiddle provides an example of the data I get:
Notice that there are 7 result rows, but for only 4 projects. So the pagination class shows 7 num_rows (rightfully) but there's only 4 projects.
Any ideas welcome!
I though of maybe concatenating sub query results from other tables into string value in own columns and parsing that with php explode() and such, which would then produce single project rows with all the data, but I haven't been able to accomplish that yet.
Thank you in advance for your help!
The only solution I was able to come up with was to concatenate results from sub queries / joins into single columns in the primary resultset. That allows me to get the results I need without having to modify the pagination class.
See example of concatenated results in this SQLFiddle
Example Query:
SELECT p.id pid, p.name pname, c.name cname,
GROUP_CONCAT(DISTINCT CONCAT_WS('|',w.hours,w.notes,t.name) SEPARATOR ';') as works
FROM projects p
LEFT JOIN (SELECT * FROM clients) AS c ON c.id=p.client_id
LEFT JOIN (SELECT * FROM work) AS w ON w.project_id=p.id
LEFT JOIN (SELECT * FROM tools) AS t ON t.id=w.tool_id
GROUP BY p.id
GROUP_CONCAT(DISTINCT ... is what concatenates the individual rows from the joined results, and the CONCAT_WS('',col,col,..) concatenates the individual result columns. Both allow to you to use separators you define, just like php implode() would. Ultimately, you end up with a formatted string of results in a single column/row on which you can use php to explode() rows and then their columns.
Now I get the right result count, and simply explode the rows and fields by the defined separators.
Example Code:
$wkeys = array('hours','notes','toolname'); // to use as array key (column name)
while($r=$projects->fetch_assoc()){
if(strpos($r['works'],';')!==false){
$wrows = explode(";",$r['works']); // individual rows
foreach($wrows as $k=>$v) {
$wv = explode("|",$v); // individual col=val pairs
$works[] = array_combine($wkeys,$wv);
}
}
}
Hope this helps someone else who may be facing the same situation.
EDIT: While this solution does work quite well for me, I did end up running into a problem with the default length limit for GROUP_CONCAT which is 1024. Any value that was over 1024 was cut down to 1024. See this answer to resolve that. I'd advise you be proactive and change it now, unless you're absolutely sure your values will never be longer than 1024.
Related
I'm not very experienced with more advanced MySQL query stuff.. (mostly basic queries, return and parse response..etc)
However.. I am not clear on the correct approach when I need multiple things (responses) from the database.. Is there a way to get these things from the single query? or do I need to do a new query for each time?
Background:
I use PDO to do a SELECT statement
ie:
$getAllVideos_sql = "SELECT * as FROM $tableName WHERE active IS NOT NULL OR active != 'no' ORDER BY topic, speaker_last, title;";
$getAllVideos_stmt = $conn->prepare($getAllVideos_sql);
$getAllVideos_stmt->execute();
$getAllVideos_stmt->setFetchMode(PDO::FETCH_ASSOC);
$results = $getAllVideos_stmt->fetch(PDO::FETCH_ASSOC);
//parse as I see fit
This gives me my 'chunk of data' that I can pick apart and display as I want.
However.. I want to also be able to give some stats (totals)
For the total (distinct) 'topics'.. as well as total count for the 'titles' (should all be unique by default)
Do I need to do another query, prepare, execute, setFetchMode, fetch all over again?
Is this the proper way to do this? Or is there a way to crib off the initial commands that are already in play?
To be clear, I'm not really looking for a query... I'm looking to understand the proper way one does this.. when they need several pieces of data like I do? multiple queries and executions..etc?
Or maybe it can and -should- be done in one snippet? With an adjustment to the query itself to return sub select/queries info?
this isnt the correct syntax, because it only returns 1 record..(but the total topic count seems to be correct, even though I only get 1 record returned)
SELECT *, count(DISTINCT topic)as totalTopics, count(DISTINCT title)as totalTitles FROM $tableName;
Maybe this the more proper approach? Try to include these totals/details in the main query to pick out?
Hope this makes sense.
Thanks
I don't think you're going to get anything very clean that'll do this, however something like this might work:
SELECT * from $Table t
INNER JOIN (
SELECT COUNT(DISTINCT Topic) as TotalTopics FROM $Table
) s ON 1 = 1
INNER JOIN (
SELECT COUNT(DISTINCT Title) as TotalTitles FROM $Table
) f ON 1 = 1
WHERE ( Active IS NOT NULL ) AND Active != 'no'
Especially with web applications, many people are regularly doing counts or other aggregations somewhere along the way. Sometimes if it is a global context such as all topics for all users, having some stored aggregates helps rather than requerying all record counts every time.
Example. If you have a table with a list of "topics", have a column in there for uniqueTitleCount. Then, based on a trigger, when a new title is added to a topic, the count is automatically updated by adding 1. You can pre-populate this column by doing a correlated update to said "topics" table, then once the trigger is set, you can just have that column.
This also works as I see many times that people want "the most recent". If your system has auto-increment IDs in the tables, similarly, have the most recent ID created for a given topic, or even most recent for a given title/document/thread so you don't have to keep doing something like.
select documentID, other_stuff
from sometable
where documentID in ( select max( documentID )
from sometable
where the title = 'something' )
Use where these make sense then your optimization pull-downs get easier to handle. You could even have a counter per document "title" and even a most recent posting date so they can quickly be sorted based on interest, frequency of activity, whatever.
I have been trying to create a database for fun to get a better understanding of databases and using PHP to query them for a website I'm messing around with. Pretty much I have one database with 4 tables when a user enters a search term in a PHP search box my code searches the database for any entries containing the search term. Now I can easily get my code to search individual tables, but I cannot seem to get it to search all 4 tables and display the results on the same page.
info: making a database for skyrim
Table names: classes, powers, skills, shouts
column names: name, information
Here is a snippet of the code I have that works so far:
$raw_results = mysql_query("
SELECT *
FROM `xaviorin_skyrim`.`shouts` , `xaviorin_skyrim`.`classes`
WHERE (CONVERT(`UID` USING utf8) LIKE '%".$query."%' OR
CONVERT(`Name` USING utf8) LIKE '%".$query."%' OR
CONVERT(`Information` USING utf8) LIKE '%".$query."%')
") or die(mysql_error());`
Literally all I thought I would need to do is change the table name from "shouts" to say "classes" in a new raw_results line of code but that didn't work. I have attempted unions and joins and either keep screwing them up or just don't understand how to properly format them.
echo "<p><h3>".$results['Name']."</h3>".$results['Information']."</p>";
The code above this text is what displays the results on the page on my website... it works but I don't know how to combine the information from all 4 tables into one page. If I'm going about this in the wrong way and anyone can point me in the right direction I would GREATLY appreciate it... I've been trying to research the problem without finding a proper answer for near a month now.
The problem with your approach is that relational databases do a cross join when there are several query results from two different tables. So basically every match in one table will be combined with every match from the second table. When you have 3 entries in the first and 4 in the second table, you will get 3 * 4 = 12 entries in your query result. If you add more tables, you get even more results. You want to do a full text search in several tables that are totally unrelated, thus creating some kind of non-existing relation via cross joining them will not be useful.
What you actually want to do is a UNION ALL (UNION is slower because it prunes duplicates) of several queries:
SELECT name, information, 'shouts' AS tablename FROM shouts WHERE ...
UNION ALL
SELECT name, information, 'classes' AS tablename FROM classes WHERE ...
This will do search queries on every single table and then place the results in a single result. Also note that I added a third column to each query to ensure that the originating table is not lost after merging the results.
Unless you need to do some sorting afterwards, I would suggest that you do all statements separately. Combining them this way will most likely make the post-processing more complex. And several single queries will also be faster than one big query with UNION statements.
And as I mentioned in the comments: Don't use mysql_* functions!
I am new to php and mysql and I am trying for 2 day to create a web application which can store specific data about the user from the form and than allow users to search based on similar slightly complicated form. I got the forms working and also input to the database but I have the problem with receiving the data from mysql and maybe in the future I will have with the search function too but i did not think much about that yet. Right now I am focusing on getting that specific data from db.
SELECT players.*,
brackets.name AS interested_in,
classes.name AS looking_for,
tags.name AS tagged_with
FROM players JOIN player_brackets ON (players.id = player_brackets.player_id)
JOIN brackets ON (player_brackets.bracket_id = brackets.id)
JOIN player_classes ON (player_brackets.player_id = player_classes.player_id)
JOIN classes ON (player_classes.class_id = classes.id)
JOIN player_tags ON (player_classes.player_id = player_tags.player_id)
JOIN tags ON (player_tags.tag_id = tags.id)
WHERE players.id = '18'
I have this query which works fine but the data i get from it is every combination of the data in the db which is right now about more than 100 results which are all almost the same. Its multiplying the results by number of fields in the db for example players (1) * brackets (3) * classes (11) * tags (5) which will output 165 rows of almost the same data with the difference in fields brackets, classes and tags.
If I group that results by players.id I get just one row but the data from other fields are lost. I want to get 1 row which will contains all of it (col with multiple data in in an array probably).
DB got 7 tables - players, tags, classes, brackets and 3 tables which contains player id and tag/class/bracket id combination.
I have been searching web and SO but nothing really helped me. It would be very nice if someone can help me with this one. Thanks for reply and I am sorry for my poor english its not my primary language.
When you join, MySQL is going to give you every combination.
As you've seen, you can GROUP BY, but then it will only give you one combination.
I think you're looking for a series of queries ("get all brackets for player 1" then "get all classes for player 1 in bracket 12", etc).
Or you can parse the results intelligently in your app.
There's also GROUP_CONCAT, but that rarely meets anyone's needs (since it comes back as one string).
I have a query that works, but it's taking at least 3 seconds to run so I think it can probably be faster. It's used to populate a list of new threads and show how many unread posts there are in each thread. I generate the query string before throwing it into $db->query_read(). In order to only grab results from valid forums, $ids is string with up to 50 values separated by commas.
The userthreadviews table has existed for 1 week and there are roughly 9,500 rows in it. I'm not sure if I need to set up a cron job to regularly clear out thread views more than a week old, or if I will be fine letting it grow.
Here's the query as it currently stands:
SELECT
`thread`.`title` AS 'r_title',
`thread`.`threadid` AS 'r_threadid',
`thread`.`forumid` AS 'r_forumid',
`thread`.`lastposter` AS 'r_lastposter',
`thread`.`lastposterid` AS 'r_lastposterid',
`forum`.`title` AS 'f_title',
`thread`.`replycount` AS 'r_replycount',
`thread`.`lastpost` AS 'r_lastpost',
`userthreadviews`.`replycount` AS 'u_replycount',
`userthreadviews`.`id` AS 'u_id',
`thread`.`postusername` AS 'r_postusername',
`thread`.`postuserid` AS 'r_postuserid'
FROM
`thread`
INNER JOIN
`forum`
ON (`thread`.`forumid` = `forum`.`forumid`)
LEFT JOIN
(`userthreadviews`)
ON (`thread`.`threadid` = `userthreadviews`.`threadid`
AND `userthreadviews`.`userid`=$userid)
WHERE
`thread`.`forumid` IN($ids)
AND `thread`.`visible`=1
AND `thread`.`lastpost`> time() - 604800
ORDER BY `thread`.`lastpost` DESC LIMIT 0, 30
An alternate query that joins the post table (to only show threads where user has posted) is actually twice as fast, so I think there's got to be something in here that could be changed to speed it up. Could someone provide some advice?
Edit: Sorry, I had put the EXPLAIN in front of the alternate query. Here is the correct output:
As Requested, here is the output generated by EXPLAIN SELECT:
Have a look at the mysql explain statement. It gives you a execution plan of your query.
Once you know the plan, you can check if you have got a index on the fields involved in the plan. If not, create them.
Perhaps the plan reveals details about how the query can be written in another way, such that the query will be more optimized.
To have no indexes on joins / where (used key = NULL on explain), this is the reason why your queries are slow. You should index them in such a way :
CREATE INDEX thread_forumid_index ON thread(forumid);
CREATE INDEX userthreadviews_forumid_index ON userthreadviews(forumid);
Documentation here
Try to index the table forumid if it is not indexed
Suggestions:
move the conditions from the WHERE clause to the JOIN clause
put the JOIN with the conditions before the other JOIN
make sure you have proper indexes and that they are being used in the query (create the ones you'll need... too much indexes can be as bad as too few)
Here is my suggestion for the query:
SELECT
`thread`.`title` AS 'r_title',
`thread`.`threadid` AS 'r_threadid',
`thread`.`forumid` AS 'r_forumid',
`thread`.`lastposter` AS 'r_lastposter',
`thread`.`lastposterid` AS 'r_lastposterid',
`forum`.`title` AS 'f_title',
`thread`.`replycount` AS 'r_replycount',
`thread`.`lastpost` AS 'r_lastpost',
`userthreadviews`.`replycount` AS 'u_replycount',
`userthreadviews`.`id` AS 'u_id',
`thread`.`postusername` AS 'r_postusername',
`thread`.`postuserid` AS 'r_postuserid'
FROM
`thread`
INNER JOIN (`forum`)
ON ((`thread`.`visible` = 1)
AND (`thread`.`lastpost` > $time)
AND (`thread`.`forumid` IN ($ids))
AND (`thread`.`forumid` = `forum`.`forumid`))
LEFT JOIN (`userthreadviews`)
ON ((`thread`.`threadid` = `userthreadviews`.`threadid`)
AND (`userthreadviews`.`userid` = $userid))
ORDER BY
`thread`.`lastpost` DESC
LIMIT
0, 30
These are good candidates to be indexed:
- `forum`.`forumid`
- `userthreadviews`.`threadid`
- `userthreadviews`.`userid`
- `thread`.`forumid`
- `thread`.`threadid`
- `thread`.`visible`
- `thread`.`lastpost`
It seems you already have lots of indexes... so, make sure you keep the ones you really need and remove the useless ones.
So, basically, I have a MySQL table called 'topics' and another one called 'replies', for example. In table 'topics', there's a field called 'relforum' which relates this topic to a forum section. And in the table 'replies', there's a field called 'reltopic', which relates the reply to a topic. Both tables have an id field, auto_increment primary key.
Now, I want to select all replies from a certain forum. Since 'replies' has no 'relforum' field, my way would be:
Select all topics with 'relforum' equal to that certain forum and loop through them
While in the loop, select all replies from the topic that is being 'processed' right now
Merge all fetch_array results in one multidimensional array, then loop through them.
That's something like this:
$query = mysql_query("SELECT * FROM `topics` WHERE `relforum` = '1'");
while($array = mysql_fetch_array($query)) {
$temp = mysql_query("SELECT * FROM `replies` WHERE `reltopic` = {$array['id']}");
$results[] = mysql_fetch_array($temp);
}
Is there a way to merge all that into fewer queries? Because this process would basically run one query per topic in that forum plus one. That would be too much :P
Adding the relforum field to the replies table is a solution (I'm still designing the DB Part so it's not a problem to add it), but I would like to see if there's a solution.
I'm really not good at SQL things, I only know the basic SELECT/INSERT/UPDATE, and I usually generate the last two ones using PHPMyAdmin, so... I guess I need some help.
Thanks for reading!
You need to learn to use joins. The link below is for SQL server but the theory for mySQl is pretty much the same for basic joins. Please do not use comma-based joins as they are 18 years outdated and are a porr pracitce. Learn to use the ANSII standard joins.
http://www.tek-tips.com/faqs.cfm?fid=4785
In accessing a database, you almost never want to use any looping. Databases are designed to perform best when asked to operate on sets of data not individual rows. So you need to stop thinking about looping and start thinking about the set of the data you need.
SELECT
r.*
FROM
replies r
INNER JOIN
topics t
ON
r.reltopic = t.id
WHERE
t.relforum = 1;
You basically need a join of two tables.
SELECT * FROM `replies`, `topics` WHERE `replies`.`reltopic` = `topics`.`id` AND `topics`.`relforum` = '1';
SELECT r.* FROM replies r, topics t
WHERE t.relforum = 1 AND r.reltopic = t.id
get rid of the backquotes. they're nonstandard and clutter the code
Yes, you should use a join here. However you will need to take greater care processing your result set.
Joins are the essential query in a relational database schema. Add them to your arsenal of knowledge :)