I'm not very experienced with more advanced MySQL query stuff.. (mostly basic queries, return and parse response..etc)
However.. I am not clear on the correct approach when I need multiple things (responses) from the database.. Is there a way to get these things from the single query? or do I need to do a new query for each time?
Background:
I use PDO to do a SELECT statement
ie:
$getAllVideos_sql = "SELECT * as FROM $tableName WHERE active IS NOT NULL OR active != 'no' ORDER BY topic, speaker_last, title;";
$getAllVideos_stmt = $conn->prepare($getAllVideos_sql);
$getAllVideos_stmt->execute();
$getAllVideos_stmt->setFetchMode(PDO::FETCH_ASSOC);
$results = $getAllVideos_stmt->fetch(PDO::FETCH_ASSOC);
//parse as I see fit
This gives me my 'chunk of data' that I can pick apart and display as I want.
However.. I want to also be able to give some stats (totals)
For the total (distinct) 'topics'.. as well as total count for the 'titles' (should all be unique by default)
Do I need to do another query, prepare, execute, setFetchMode, fetch all over again?
Is this the proper way to do this? Or is there a way to crib off the initial commands that are already in play?
To be clear, I'm not really looking for a query... I'm looking to understand the proper way one does this.. when they need several pieces of data like I do? multiple queries and executions..etc?
Or maybe it can and -should- be done in one snippet? With an adjustment to the query itself to return sub select/queries info?
this isnt the correct syntax, because it only returns 1 record..(but the total topic count seems to be correct, even though I only get 1 record returned)
SELECT *, count(DISTINCT topic)as totalTopics, count(DISTINCT title)as totalTitles FROM $tableName;
Maybe this the more proper approach? Try to include these totals/details in the main query to pick out?
Hope this makes sense.
Thanks
I don't think you're going to get anything very clean that'll do this, however something like this might work:
SELECT * from $Table t
INNER JOIN (
SELECT COUNT(DISTINCT Topic) as TotalTopics FROM $Table
) s ON 1 = 1
INNER JOIN (
SELECT COUNT(DISTINCT Title) as TotalTitles FROM $Table
) f ON 1 = 1
WHERE ( Active IS NOT NULL ) AND Active != 'no'
Especially with web applications, many people are regularly doing counts or other aggregations somewhere along the way. Sometimes if it is a global context such as all topics for all users, having some stored aggregates helps rather than requerying all record counts every time.
Example. If you have a table with a list of "topics", have a column in there for uniqueTitleCount. Then, based on a trigger, when a new title is added to a topic, the count is automatically updated by adding 1. You can pre-populate this column by doing a correlated update to said "topics" table, then once the trigger is set, you can just have that column.
This also works as I see many times that people want "the most recent". If your system has auto-increment IDs in the tables, similarly, have the most recent ID created for a given topic, or even most recent for a given title/document/thread so you don't have to keep doing something like.
select documentID, other_stuff
from sometable
where documentID in ( select max( documentID )
from sometable
where the title = 'something' )
Use where these make sense then your optimization pull-downs get easier to handle. You could even have a counter per document "title" and even a most recent posting date so they can quickly be sorted based on interest, frequency of activity, whatever.
Related
I have a custom php blog where I have categories and posts.
In a side widget I have listed all my category titles and next to each of them I'd like to show how many posts they contain.
How can I do that?
My categories table has 2 columns: cat_id and cat_title
My posts table has a post_category_id column.
As your question is extremely simple I won't give you the code, but give you the answer so you have to implement it. For solving your problem you have two options:
1. To use WHERE statement
You can read about it here.
2. To use Inner Join
You can read about it here.
The first one is easier, and as far as I know, it's also faster.
Counting the results:
After filtering the results and getting it through PHP, you can get the rows count with int $mysqli_result->num_rows; (OO style) or int mysqli_num_rows ( mysqli_result $result ) (procedural style) (documentation).
Why not use:
$count = count($mysqli_result_array);
It's trivial enough to count the number of posts per category in SQL.
select post_category_id, count(*) from posts group by post_category_id;
HOWEVER, this query requires a full table scan of your posts, so it is much too slow to do every time your page is accessed. There are several solutions to this problem.
One solution is to add a post_count column to your categories table, and periodically run a query to update your post-counts.
update categories
set categories.post_count = i.cnt
from (
select post_category_id, count(*) cnt
from posts
group by post_category_id
) i
where categories.cat_id = i.post_category_id
Another solution is to add a post_count column to your categories table, and create a trigger for your posts table which automatically updates the counts every time a post is added (or removed). This solution has advantages (the counts are always up to date, and are incrementally updated) and disadvantages (every posting now creates lock contention over the post counts). Trigger code can vary depending on exactly which database you are using, but the insert trigger should look something like this:
CREATE TRIGGER post_count_trigger ON posts
AFTER INSERT
BEGIN
update categories set post_count=post_count+1 where cat_id=post_categori_id
END
My PHP application has the function product_fetch([parameters]) which returns 'Product' objects which information are stored in database.
In my admin area, there is a page called "Featured Products" which allows me to select 10 products to be displayed in the main page.
Now comes the problem: I made 10 select/combobox, each allows me to select one product, out of 400. So in order to make all the options, a query has to be made: SELECT * FROM products
Question: Is it correct to make such a query, even though there's hundreds of rows?
The solution you proposed is certainly do-able, and 400 rows is really meek compared to the upper limits of what MySQL is capable of handling. What is more concerning is the user experience here. Granted this will only affect you, but I would design myself something a little nicer than a bunch of <select>s. My idea is to start with just one textbox that autocompletes the names of your products. This can be accomplished if the product title has a fulltext index. Then your autocomplete script could use this query:
SELECT * FROM Products WHERE MATCH(title) AGAINST ('contents of textbox' IN BOOLEAN MODE);
There are plenty of jQuery plugins like Autocomplete that will handle the JS side (querying the server for autocomplete results). The one I just mentioned adds a term GET parameter which you could easily grab and throw into the query:
// You might want to only select the relevant columns
$stmt = $pdo->prepare('SELECT * FROM Products WHERE MATCH(title) AGAINST (:search IN BOOLEAN MODE)');
$stmt->execute(array(':search' => $_GET['term']);
// Output JSON results
echo json_encode($stmt->fetchall());
Once you type in (or click on an autocomplete result) in the one textbox, another should appear below it and the focus should go to it. From there you can type another product, and continue until you reach 10. Each textbox (unless there is only one) should have a delete link next to it that removes that product from the featured listing.
I'll look for a site that implements this kind of functionality so you can better understand what I'm talking about. Basically here, what you're searching for is achieved. You don't have to have 10 select boxes with 400 options and you don't need a SELECT * FROM Products.
You would be much better of specifying which produts you want in the query and only returning those if you have no intention of using any of the others at all.
You can do this using many methods. A simple one would be using an ID field in an in statement like this:
select col1, col2 from products where id in(1,4,12,5)
It might seem to make little difference, but what if your procuts table had a hundred thousand rows in it?
You could also have a flag in the table to say that the items are featured which would let you use something like this:
select col1, col2 from products where featured='Y'
Or you could even have a table that only has featured items (even just their ID) and join it to your main listing like this:
select
a.col1,
a.col2
from
products a
join featured b
on a.id=b.id
If you want to pick through your whole table, you can even use a simple limit clause that picks up a certain number of rows from the table and can be reused to get the next set:
select col1, col2 from products limit 10;
// Will pick the first 10 rows of data.
select col1, col2 from procuts limit 30,10;
// Will pick rows 31-40 (skipping first 30, then returning next 10)
The short version is though, no matter how you do it, pulling back a whole table of data to pick through it within PHP is a bad thing and to be avoided at all costs. It makes the database work much harder, uses more network between the database and PHP (even if they are on the same machine, it is transferring a lot more data between the two of them) and it will by default make PHP use a lot more resources to process the information.
SELECT * FROM tbl LIMIT $page,10;
The above query will select 10 entries from offset $page
Fetching all rows is a bad idea as you are using 10 only anyway. When your table expands to millions of rows, you will see a noticeable difference.
There is nothing fundamentally wrong with selecting all rows if that's what you mean to do. Hundreds of rows also shouldn't be a problem for the query in terms of performance. However, thousands or millions might if your database grows that big.
Think about the user - can they scroll through hundreds of products? Probably. If not, then maybe it's the UI design at fault, not the query.
I've trying to create some stats for my table but it has over 3 million rows so it is really slow.
I'm trying to find the most popular value for column name and also showing how many times it pops up.
I'm using this at the momment but it doesn't work cause its too slow and I just get errors.
$total = mysql_query("SELECT `name`, COUNT(*) as b FROM `people` GROUP BY `name` ORDER BY `b` DESC LIMIT 0,5;")or die(mysql_error());
As you may see I'm trying to get all the names and how many times that name has been used but only show the top 5 to hopefully speed it up.
I would like to be able to then do get the values like
while($row = mysql_fetch_array($result)){
echo $row['name'].': '.$row['b']."\r\n";
}
And it will show things like this;
Bob: 215
Steve: 120
Sophie: 118
RandomGuy: 50
RandomGirl: 50
I don't care much about ordering the names afterwards like RandomGirl and RandomGuy been the wrong way round.
I think I've have provided enough information. :) I would like the names to be case-insensitive if possible though. Bob should be the same as BoB, bOb, BOB and so on.
Thank-you for your time
Paul
Limiting results on the top 5 won't give you a lot of speed-up, you'll gain time in the result retrieval, but in mySQL side the whole table still needs to be parsed (to count).
You will speed-up your count query having index on name column, of course as only the index will be parsed and not the table.
Now if you really want to speed up the result and avoid parsing the name index when you need this result (which will still be quite slow if you really have millions of rows), then the only other solution is computing the stats when inserting, deleting or updating rows on this table. That is using triggers on this table to maintain a statistics table near this one. Then you will really only have a simple select query on this statistics table, with only 5 rows parsed. But you will slow down your inserts, delete and update operations (which are already quite slow, especially if you maintain indexes, so if the stats are important you should study this solution).
Do you have an index on name? It might help.
Since you are doing the counting/grouping and then sorting an index on name doesn't help at all MySql should go through all rows every time, there is no way to optimize this. You need to have a separate stats table like this:
CREATE TABLE name_stats( name VARCHAR(n), cnt INT, UNIQUE( name ), INDEX( cnt ) )
and you should update this table whenever you add a new row to 'people' table like this:
INSERT INTO name_stats VALUES( 'Bob', 1 ) ON DUPLICATE KEY UPDATE cnt = cnt + 1;
Querying this table for the list of top names should give you the results instantaneously.
I know i am writing query's wrong and when we get a lot of traffic, our database gets hit HARD and the page slows to a grind...
I think I need to write queries based on CREATE VIEW from the last 30 days from the CURDATE ?? But not sure where to begin or if this will be MORE efficient query for the database?
Anyways, here is a sample query I have written..
$query_Recordset6 = "SELECT `date`, title, category, url, comments
FROM cute_news
WHERE category LIKE '%45%'
ORDER BY `date` DESC";
Any help or suggestions would be great! I have about 11 queries like this, but I am confident if I could get help on one of these, then I can implement them to the rest!!
Putting a wildcard on the left side of a value comparison:
LIKE '%xyz'
...means that an index can not be used, even if one exists. Might want to consider using Full Text Searching (FTS), which means adding full text indexing.
Normalizing the data would be another step to consider - categories should likely be in a separate table.
SELECT `date`, title, category, url, comments
FROM cute_news
WHERE category LIKE '%45%'
ORDER BY `date` DESC
The LIKE '%45%' means a full table scan will need to be performed. Are you perhaps storing a list of categories in the column? If so creating a new table storing category and news_article_id will allow an index to be used to retrieve the matching records much more efficiently.
OK, time for psychic debugging.
In my mind's eye, I see that query performance would be improved considerably through database normalization, specifically by splitting the category multi-valued column into a a separate table that has two columns: the primary key for cute_news and the category ID.
This would also allow you to directly link said table to the categories table without having to parse it first.
Or, as Chris Date said: "Every row-and-column intersection contains exactly one value from the applicable domain (and nothing else)."
Anything with LIKE '%XXX%' is going to be slow. Its a slow operation.
For something like categories, you might want to separate categories out into another table and use a foreign key in the cute_news table. That way you can have category_id, and use that in the query which will be MUCH faster.
Also, I'm not quite sure why you're talking about using CREATE VIEW. Views will not really help you for speed. Not unless its a materialized view, which MySQL doesn't suppose natively.
If your database is getting hit hard, the solution isn't to make a view (the view is still basically the same amount of work for the database to do), the solution is to cache the results.
This is especially applicable since, from what it sounds like, your data only needs to be refreshed once every 30 days.
I'd guess that your category column is a list of category values like "12,34,45,78" ?
This is not good relational database design. One reason it's not good is as you've discovered: it's incredibly slow to search for a substring that might appear in the middle of that list.
Some people have suggested using fulltext search instead of the LIKE predicate with wildcards, but in this case it's simpler to create another table so you can list one category value per row, with a reference back to your cute_news table:
CREATE TABLE cute_news_category (
news_id INT NOT NULL,
category INT NOT NULL,
PRIMARY KEY (news_id, category),
FOREIGN KEY (news_id) REFERENCES cute_news(news_id)
) ENGINE=InnoDB;
Then you can query and it'll go a lot faster:
SELECT n.`date`, n.title, c.category, n.url, n.comments
FROM cute_news n
JOIN cute_news_category c ON (n.news_id = c.news_id)
WHERE c.category = 45
ORDER BY n.`date` DESC
Any answer is a guess, show:
- the relevant SHOW CREATE TABLE outputs
- the EXPLAIN output from your common queries.
And Bill Karwin's comment certainly applies.
After all this & optimizing, sampling the data into a table with only the last 30 days could still be desired, in which case you're better of running a daily cronjob to do just that.
So, basically, I have a MySQL table called 'topics' and another one called 'replies', for example. In table 'topics', there's a field called 'relforum' which relates this topic to a forum section. And in the table 'replies', there's a field called 'reltopic', which relates the reply to a topic. Both tables have an id field, auto_increment primary key.
Now, I want to select all replies from a certain forum. Since 'replies' has no 'relforum' field, my way would be:
Select all topics with 'relforum' equal to that certain forum and loop through them
While in the loop, select all replies from the topic that is being 'processed' right now
Merge all fetch_array results in one multidimensional array, then loop through them.
That's something like this:
$query = mysql_query("SELECT * FROM `topics` WHERE `relforum` = '1'");
while($array = mysql_fetch_array($query)) {
$temp = mysql_query("SELECT * FROM `replies` WHERE `reltopic` = {$array['id']}");
$results[] = mysql_fetch_array($temp);
}
Is there a way to merge all that into fewer queries? Because this process would basically run one query per topic in that forum plus one. That would be too much :P
Adding the relforum field to the replies table is a solution (I'm still designing the DB Part so it's not a problem to add it), but I would like to see if there's a solution.
I'm really not good at SQL things, I only know the basic SELECT/INSERT/UPDATE, and I usually generate the last two ones using PHPMyAdmin, so... I guess I need some help.
Thanks for reading!
You need to learn to use joins. The link below is for SQL server but the theory for mySQl is pretty much the same for basic joins. Please do not use comma-based joins as they are 18 years outdated and are a porr pracitce. Learn to use the ANSII standard joins.
http://www.tek-tips.com/faqs.cfm?fid=4785
In accessing a database, you almost never want to use any looping. Databases are designed to perform best when asked to operate on sets of data not individual rows. So you need to stop thinking about looping and start thinking about the set of the data you need.
SELECT
r.*
FROM
replies r
INNER JOIN
topics t
ON
r.reltopic = t.id
WHERE
t.relforum = 1;
You basically need a join of two tables.
SELECT * FROM `replies`, `topics` WHERE `replies`.`reltopic` = `topics`.`id` AND `topics`.`relforum` = '1';
SELECT r.* FROM replies r, topics t
WHERE t.relforum = 1 AND r.reltopic = t.id
get rid of the backquotes. they're nonstandard and clutter the code
Yes, you should use a join here. However you will need to take greater care processing your result set.
Joins are the essential query in a relational database schema. Add them to your arsenal of knowledge :)