I'm new to this, sorry if the title is confusing. I am building a simple php/mysql gallery of sorts. It will show the newest 25 entries when a user first goes to it, and also allows off-site linking to individual items in the list. If the URL contains an ID, javascript will scroll to it. But if there are 25+ entries, it's possible that my query will fetch the newest results, but omit an older entry that happens to be in the URL as an ID.
That means I need to do something like this...
SELECT * FROM `submissions` WHERE uid='$sid'
But after that has successfully found the submission with the special ID, also do
SELECT * FROM `submissions` ORDER BY `id` DESC LIMIT 0, 25`
So that I can populate the rest of the gallery.
I could query that database twice, but I am assuming there's some nifty way to avoid that. MySQL is also ordering everything (based on newest, views, and other vars) and using two queries would break that.
You could limit across a UNION like this:
(SELECT * FROM submissions WHERE uid = '$uid')
UNION
(SELECT * FROM submissions WHERE uid <> '$uid' ORDER BY `id` LIMIT 25)
LIMIT 25
Note LIMIT is listed twice as in the case that the first query returns a result, we would have 26 results in the union set. This will also place the "searched for" item first in the returned sort result set (with the other 24 results displayed in sort order). If this is not desirable, you could place an ORDER BY across the union, but your searched for result would be truncated if it happened to be the 26th record.
If you need 25 rows with all of them being sorted, my guess is that you would need to do the two query approach (limiting second query to either 24 or 25 records depending on whether the first query matched), and then simply insert the uid-matched result into the sorted records in the appropriate place before display.
I think the better solution is:
SELECT *
FROM `submissions`
order by (case when usid = $sid then 0 else 1 end),
id desc
limit 25
I don't think the union is guaranteed to return results in the order of the union (there is no guarantee in the standard or in other databases).
Related
Lets start by saying that I cant use INDEXING as I need the INSERT, DELETE and UPDATE for this table to be super fast, which they are.
I have a page that displays a summary of order units collected in a database table. To populate the table an order number is created and then individual units associated with that order are scanned into the table to recored which units are associated with each order.
For the purposes of this example the table has the following columns.
id, UID, order, originator, receiver, datetime
The individual unit quantities can be in the 1000's per order and the entire table is growing to hundreds of thousands of units.
The summary page displays the number of units per order and the first and last unit number for each order. I limit the number of orders to be displayed to the last 30 order numbers.
For example:
Order 10 has 200 units. first UID 1510 last UID 1756
Order 11 has 300 units. first UID 1922 last UID 2831
..........
..........
Currently the response time for the query is about 3 seconds as the code performs the following:
Look up the last 30 orders by by id and sort by order number
While looking at each order number in the array
-- Count the number of database rows that have that order number
-- Select the first UID from all the rows as first
-- Select the last UID from all the rows as last
Display the result
I've determined the majority of the time is taken by the Count of the number of units in each order ~1.8 seconds and then determining the first and last numbers in each order ~1 second.
I am really interested in if there is a way to speed up these queries without INDEXING. Here is the code with the queries.
First request selects the last 30 orders processed selected by id and grouped by order number. This gives the last 30 unique order numbers.
$result = mysqli_query($con, "SELECT order, ANY_VALUE(receiver) AS receiver, ANY_VALUE(originator) AS originator, ANY_VALUE(id) AS id
FROM scandb
GROUP BY order
ORDER BY id
DESC LIMIT 30");
While fetching the last 30 order numbers count the number of units and the first and last UID for each order.
while($row=mysqli_fetch_array($result)){
$count = mysqli_fetch_array(mysqli_query($con, "SELECT order, COUNT(*) as count FROM scandb WHERE order ='".$row['order']."' "));
$firstLast = mysqli_fetch_array(mysqli_query($con, "SELECT (SELECT UID FROM scandb WHERE orderNumber ='".$row['order']."' ORDER BY UID LIMIT 1) as 'first', (SELECT UID FROM barcode WHERE order ='".$row['order']."' ORDER BY UID DESC LIMIT 1) as 'last'"));
echo "<td align= center>".$count['count']."</td>";
echo "<td align= center>".$firstLast['first']."</td>";
echo "<td align= center>".$firstLast['last']."</td>";
}
With 100K lines in the database this whole query is taking about 3 seconds. The majority of the time is in the $count and $firstlast queries. I'd like to know if there is a more efficient way to get this same data in a faster time without Indexing the table. Any special tricks that anyone has would be greatly appreciated.
Design your database with caution
This first tip may seems obvious, but the fact is that most database problems come from badly-designed table structure.
For example, I have seen people storing information such as client info and payment info in the same database column. For both the database system and developers who will have to work on it, this is not a good thing.
When creating a database, always put information on various tables, use clear naming standards and make use of primary keys.
Know what you should optimize
If you want to optimize a specific query, it is extremely useful to be able to get an in-depth look at the result of a query. Using the EXPLAIN statement, you will get lots of useful info on the result produced by a specific query, as shown in the example below:
EXPLAIN SELECT * FROM ref_table,other_table WHERE ref_table.key_column=other_table.column;
Don’t select what you don’t need
A very common way to get the desired data is to use the * symbol, which will get all fields from the desired table:
SELECT * FROM wp_posts;
Instead, you should definitely select only the desired fields as shown in the example below. On a very small site with, let’s say, one visitor per minute, that wouldn’t make a difference. But on a site such as Cats Who Code, it saves a lot of work for the database.
SELECT title, excerpt, author FROM wp_posts;
Avoid queries in loops
When using SQL along with a programming language such as PHP, it can be tempting to use SQL queries inside a loop. But doing so is like hammering your database with queries.
This example illustrates the whole “queries in loops” problem:
foreach ($display_order as $id => $ordinal) {
$sql = "UPDATE categories SET display_order = $ordinal WHERE id = $id";
mysql_query($sql);
}
Here is what you should do instead:
UPDATE categories
SET display_order = CASE id
WHEN 1 THEN 3
WHEN 2 THEN 4
WHEN 3 THEN 5
END
WHERE id IN (1,2,3)
Use join instead of subqueries
As a programmer, subqueries are something that you can be tempted to use and abuse. Subqueries, as show below, can be very useful:
SELECT a.id,
(SELECT MAX(created)
FROM posts
WHERE author_id = a.id)
AS latest_post FROM authors a
Although subqueries are useful, they often can be replaced by a join, which is definitely faster to execute.
SELECT a.id, MAX(p.created) AS latest_post
FROM authors a
INNER JOIN posts p
ON (a.id = p.author_id)
GROUP BY a.id
Source: http://20bits.com/articles/10-tips-for-optimizing-mysql-queries-that-dont-suck/
Im currently working on a project that requires MySql database and im having a hard time constructing the query that i want get.
i want to get the previous 10 rows from the specific WHERE condition on my mysql query.
for example
My where is date='December';
i want the last 10 months to as a result.
Feb,march,april,may,june,july,aug,sept,oct,nov like that.
Another example is.
if i have a 17 strings stored in my database. and in my where clause i specify that WHERE strings='eyt' limit 3
Test
one
twi
thre
for
payb
six
seven
eyt
nayn
ten
eleven
twelve
tertin
fortin
fiftin
sixtin
the result must be
payb
six
seven
Thanks in advance for your suggestions or answers
If you are using PDO this is the right syntax:
$objStmt = $objDatabase->prepare('SELECT * FROM calendar ORDER BY id DESC LIMIT 10');
You can change ASC to DESC in order to get either the first or the last 10.
Here's a solution:
select t.*
from mytable t
inner join (select id from mytable where strings = 'eyt' order by id limit 1) x
on t.id < x.id
order by t.id desc
limit 3
Demo: http://sqlfiddle.com/#!9/7ffc4/2
It outputs the rows in descending order, but you can either live with that, or else put that query in a subquery and reverse the order.
Re your comment:
x in the above query is called a "correlation name" so we can refer to columns of the subquery as if they were columns of a table. It's required when you use a subquery as a table.
I chose the letter x arbitrarily. You can use anything you like as a correlation name, following the same rules you would use for any identifier.
You can also optionally define a correlation name for any simple table in the query (like mytable t above), so you can refer to columns of that table using a convenient abbreviated name. For example in t.id < x.id
Some people use the term "table alias" but the technical term is "correlation name".
I have a sql statement:
$feed=$conn->prepare("SELECT * FROM posts WHERE post_by=? OR id=? ORDER BY id DESC LIMIT 10");
$feed->bind_param("ii",$friend['id'],$like[id]);
$feed->execute();
The $friend['id'] is the id of a user's friend, $like[id] is the id of a like by the user's friend.
The posts fetched with this query appear in a page.
What I want is I want to know which all posts have been posted by the user's friends (Which all posts have been fetched using $friends['id']) and which all posts have been liked by the user's friends and appear in the feed(Which all posts have been fetched using $like['id'])
I want to know all possibilities I can try to achieve what I want.
I have tried varying my query with UNION ALL but it shows errors and I could'nt achieve what I want.
Currently there are no errors but I want the user to know how this post appeared in the newsfeed.
Hope you all get a good idea about my question and all types of hacks are also accepted as I want in someway to achieve the result I would also agree to change mt query nature.
Please comment for more info.
Thanks in advance.
SELECT *, post_by = ?postId AS post_by_friend
FROM posts
WHERE post_by = ?postId OR
id = ?friendId
ORDER BY id DESC
LIMIT 10
post_by_friend will be 1 if it matched the first condition, otherwise 0. I haven't benchmarked it, but this method should be faster than StuartLC's UNION suggestion.
What you can do is break the query up on its 'OR' clause into a UNION of two separate queries, and add a marker column to indicate whether the row was found by friend or by like:
SELECT *
FROM
(
SELECT *, 'Friend' AS HowFound
FROM posts
WHERE post_by= ?postId
UNION
SELECT *, 'Like' AS HowFound
FROM posts
WHERE id= ?friendId AND post_by <> ?postId
) x
ORDER BY id DESC
LIMIT 10;
You'll want to exclude rows which match both friend and post classifications from one of the selects, otherwise it will be reported twice (or, otherwise your app will need to combine them).
I'm no PHP guru, but I'm sure there is a way to name the parameters to allow the above exclusion.
The derived table is needed to order and restrict the overall result.
I have a website where visitors can create "battles" and upload videos to compete in these battles.
The following mySQL query is trying to retrieve details of every "battle" in a database.
The problem I have is that if a "battle creator" has uploaded two "videos" to the same battle, a duplicate battle prints out.
How can I make the query only print out one value for each battle, even if the videos table has two entries under the same battle_id?
Thanks!
SELECT * from Battles, Player, Video
WHERE (Battles.battle_creator = Player.player_id
AND Battles.battle_id = Video.battle_id
AND Video.player_id = Battles.battle_creator)
ORDER BY Battles.battle_date DESC;
There's no way to get the information you're asking for from a single query once multiple videos have been assigned to a battle by a single user.
The best way to get all the data for the battle is to separate your query into two subqueries:
SELECT * from Battles, Player
WHERE Battles.battle_creator = Player.player_id
ORDER BY Battles.battle_date DESC;
...and then:
SELECT * from Video
ORDER BY Battles.battle_date DESC, Player.player_id;
The first query will give you one row per battle; the second will give you all videos for all battles, which you can iterate over.
From a scaling perspective, you'll do better to avoid JOINs altogether, so the extra work will be well worth it.
You can either add LIMIT 1 clause to your query to only get first result, or use DISTINCT clause like
SELECT DISTINCT *
FROM ...
That said, you should not use "SELECT * " when querying for more than one table -
use "SELECT table.*" or "SELECT table.field1, table.field2, ..." to be more specific.
You can't do "exactly" that, because your query:
SELECT * from Battles, Player, Video ...
is implicitly asking for all the videos. So you need to ask yourself first, how do I select that one video I want?
If you just want one video, whatever, then add LIMIT 1 to the query and be done with that. ORDER BY video_date ASC or DESC before the LIMIT to retrieve the earliest or latest video.
Otherwise, you have to do something like:
SELECT * from Battles
JOIN Player ON (Battles.battle_creator = Player.player_id)
JOIN Video ON (Battles.battle_id = Video.battle_id
AND Video.player_id = Battles.battle_creator)
WHERE Video.video_id = (SELECT MIN(video_id) FROM Video AS Video2 WHERE
Battles.battle_id = Video2.battle_id
AND Video2.player_id = Battles.battle_creator)
ORDER BY Battles.battle_date DESC;
In the example above I used, as "video choice criterion", "the video with smallest video_id". You will want to have an index on (Video.video_id), something like
CREATE INDEX video_ndx ON Video(player_id, battle_id, video_id);
As Ninsuo's comment points out, the proper way to control this is, after your ORDER BY clause, specify LIMIT 1.
This won't work if you want the entire table, just without duplicates. Consider running some comparison checks on your returned data, or using SELECT DISTINCT.
I'm trying to get 4 random results from a table that holds approx 7 million records. Additionally, I also want to get 4 random records from the same table that are filtered by category.
Now, as you would imagine doing random sorting on a table this large causes the queries to take a few seconds, which is not ideal.
One other method I thought of for the non-filtered result set would be to just get PHP to select some random numbers between 1 - 7,000,000 or so and then do an IN(...) with the query to only grab those rows - and yes, I know that this method has a caveat in that you may get less than 4 if a record with that id no longer exists.
However, the above method obviously will not work with the category filtering as PHP doesn't know which record numbers belong to which category and hence cannot select the record numbers to select from.
Are there any better ways I can do this? Only way I can think of would be to store the record id's for each category in another table and then select random results from that and then select only those record ID's from the main table in a secondary query; but I'm sure there is a better way!?
You could of course use the RAND() function on a query using a LIMIT and WHERE (for the category). That however as you pointed out, entails a scan of the database which takes time, especially in your case due to the volume of data.
Your other alternative, again as you pointed out, to store id/category_id in another table might prove a bit faster but again there has to be a LIMIT and WHERE on that table which will also contain the same amount of records as the master table.
A different approach (if applicable) would be to have a table per category and store in that the IDs. If your categories are fixed or do not change that often, then you should be able to use that approach. In that case you will effectively remove the WHERE from the clause and getting a RAND() with a LIMIT on each category table would be faster since each category table will contain a subset of records from your main table.
Some other alternatives would be to use a key/value pair database just for that operation. MongoDb or Google AppEngine can help with that and are really fast.
You could also go towards the approach of a Master/Slave in your MySQL. The slave replicates content in real time but when you need to perform the expensive query you query the slave instead of the master, thus passing the load to a different machine.
Finally you could go with Sphinx which is a lot easier to install and maintain. You can then treat each of those category queries as a document search and let Sphinx randomize the results. This way you offset this expensive operation to a different layer and let MySQL continue with other operations.
Just some issues to consider.
Working off your random number approach
Get the max id in the database.
Create a temp table to store your matches.
Loop n times doing the following
Generate a random number between 1 and maxId
Get the first record with a record Id greater than the random number and insert it into your temp table
Your temp table now contains your random results.
Or you could dynamically generate sql with a union to do the query in one step.
SELECT * FROM myTable WHERE ID > RAND() AND Category = zzz LIMIT 1
UNION
SELECT * FROM myTable WHERE ID > RAND() AND Category = zzz LIMIT 1
UNION
SELECT * FROM myTable WHERE ID > RAND() AND Category = zzz LIMIT 1
UNION
SELECT * FROM myTable WHERE ID > RAND() AND Category = zzz LIMIT 1
Note: my sql may not be valid, as I'm not a mySql guy, but the theory should be sound
First you need to get number of rows ... something like this
select count(1) from tbl where category = ?
then select a random number
$offset = rand(1,$rowsNum);
and select a row with offset
select * FROM tbl LIMIT $offset, 1
in this way you avoid missing ids. The only problem is you need to run second query several times. Union may help in this case.
For MySQl you can use
RAND()
SELECT column FROM table
ORDER BY RAND()
LIMIT 4