Basically I have a rating website where users rate other people from different schools. Right now I'm trying to do the leaderboard script but I cannot properly get the rating to show. My query is only returning one result. ($helpMe) Essentially I'm declaring the average total since my db doesn't store that, it only stores total number and total votes and divides accordingly.
https://pastebin.com/fVf4B8En here is the main code
I suspect that the culprit is here but not sure how
$helpMe = mysql_query("SELECT `rating_number`, FORMAT( (`total_points` / `rating_number`), 1 ) AS `average_rating` FROM `view_rating` WHERE `status` = 1 ORDER BY `average_rating` DESC");
while ($ratingRow = mysql_fetch_assoc($helpMe)) {
Related
There are more than 10m records in a table. Assume I display 15 items per page, this query is the simplest one (offset = 1,500,000)
select x,y,z
from table_name
order by x
limit 15
offset 1500000
this query takes about 5 seconds. Experts in database suggest the following approach:
I changed the query to this
select x,y,z
from table_name
where id > 3649168
order by x
limit 15
where 3649168 is the id of last record in page 1,499,999 and it takes about 250ms which is a super fast approach.
To get 3649168, I set a fields last_record_id in each page i and in page i+1 I use this to fetch information. This approach works perfect for the next page but I don't know how to get the previous page.
I have a database with more than 600 rows but I can only retrieve/display 100 every hour. So I use
select * from table ORDER BY id DESC LIMIT 100
to retrieve the first 100. How do I write a script that will retrieve the data in batches of 100 every 1hr so that I can use it in a cron job?
Possible solution.
Add a field for to mark the record was already shown.
ALTER TABLE tablename
ADD COLUMN shown TINYINT NULL DEFAULT NULL;
NULL will mean that the record was not selected, 1 - that record is marked for selection, 0 - that record was already selected.
When you need to select up to 100 records you
2.1. Mark records to be shown
UPDATE tablename
SET shown = 1
WHERE shown = 1
OR shown IS NULL
ORDER BY shown = 1 DESC, id ASC
LIMIT 100;
shown = 1 condition in WHERE considered the fact that some records were marked but were not selected due to some error. shown = 1 DESC re-marks such records before non-marked.
If there is 100 or less records which were not selected all of them will be marked, else only 100 records with lower id (most ancient) will be marked.
2.2. Select marked records.
SELECT *
FROM tablename
WHERE shown = 1
ORDER BY id
LIMIT 100;
2.3. Mark selected records.
UPDATE tablename
SET shown = 0
WHERE shown = 1
ORDER BY id
LIMIT 100;
This is applicable when only one client selects the records.
If a lot of clients may work in parallel, and only one cliens must select a record, then use some cliens number (unique over all clients) for to mark a record for selection instead of 1.
Of course if there is only one client, and you guarantee that selection will not fail, you may simply store last shown ID somewhere (on the client side, or in some service table on the MySQL side) and simply select "next 100" starting from this stored ID:
SELECT *
FROM tablename
WHERE id > #stored_id
ORDER BY id
LIMIT 100;
and
SELECT MAX(id)
FROM tablename
WHERE id > #stored_id
ORDER BY id
LIMIT 100;
for to store instead of previous #stored_id.
Thank you #Akina and #Vivek_23 for your contributions. I was able to figure out an easier way to go about it.
Add a new field to table, eg shownstatus
Create a cronjob to display 100 (LIMIT 100) records with their shownstatus not marked as shown from table every hour and then update each record's shownstatus to shown NB. If I create a cronjob to run every hour for the whole day, I can get all records displayed and their shownstatus updated to shown by close of day.
Create a second cronjob to update all record's shownstatus to notshown
The downside to this is that, you can only display a total of 2,400 records a day. ie. 100 records every hour times 24hrs. So if your record grows to about 10,000. You will need to set your cronjob to run for atleast 5 days to display all records.
Still open to a better approach if there's any, but till then, I will have to just stick to this for now.
Let's say you made a cron that hits a URL something like
http://yourdomain.com/fetch-rows
or a script for instance, like
your_project_folder/fetch-rows.php
Let's say you have a DB table in place that looks something like this:
| id | offset | created_at |
|----|--------|---------------------|
| 1 | 100 | 2019-01-08 03:15:00 |
| 2 | 200 | 2019-01-08 04:15:00 |
Your script:
<?php
define('FETCH_LIMIT',100);
$conn = mysqli_connect(....); // connect to DB
$result = mysqli_query($conn,"select * from cron_hit_table where id = (select max(id) from cron_hit_table)")); // select the last record to get the latest offset
$offset = 0; // initial default offset
if(mysqli_num_rows($result) > 0){
$offset = intval(mysqli_fetch_assoc($result)['offset']);
}
// Now, hit your query with $offset included
$result = mysqli_query($conn,"select * from table ORDER BY id DESC LIMIT $offset,100");
while($row = mysqli_fetch_assoc($result)){
// your data processing
}
// insert new row to store next offset for next cron hit
$offset += FETCH_LIMIT; // increment current offset
mysqli_query($conn,"insert into cron_hit_table(offset) values($offset)"); // because ID would be auto increment and created_at would have default value as current_timestamp
mysqli_close($conn);
Whenever cron hits, you fetch last row from your hit table to get the offset. Hit the query with that offset and store the next offset for next hit in your table.
Update:
As pointed out by #Dharman in the comments, you can use PDO for more abstracted way of dealing with different types of database(but make sure you have appropriate driver for it, see checklist of drivers PDO supports to be sure) along with minor checks of query syntaxes.
Lets start by saying that I cant use INDEXING as I need the INSERT, DELETE and UPDATE for this table to be super fast, which they are.
I have a page that displays a summary of order units collected in a database table. To populate the table an order number is created and then individual units associated with that order are scanned into the table to recored which units are associated with each order.
For the purposes of this example the table has the following columns.
id, UID, order, originator, receiver, datetime
The individual unit quantities can be in the 1000's per order and the entire table is growing to hundreds of thousands of units.
The summary page displays the number of units per order and the first and last unit number for each order. I limit the number of orders to be displayed to the last 30 order numbers.
For example:
Order 10 has 200 units. first UID 1510 last UID 1756
Order 11 has 300 units. first UID 1922 last UID 2831
..........
..........
Currently the response time for the query is about 3 seconds as the code performs the following:
Look up the last 30 orders by by id and sort by order number
While looking at each order number in the array
-- Count the number of database rows that have that order number
-- Select the first UID from all the rows as first
-- Select the last UID from all the rows as last
Display the result
I've determined the majority of the time is taken by the Count of the number of units in each order ~1.8 seconds and then determining the first and last numbers in each order ~1 second.
I am really interested in if there is a way to speed up these queries without INDEXING. Here is the code with the queries.
First request selects the last 30 orders processed selected by id and grouped by order number. This gives the last 30 unique order numbers.
$result = mysqli_query($con, "SELECT order, ANY_VALUE(receiver) AS receiver, ANY_VALUE(originator) AS originator, ANY_VALUE(id) AS id
FROM scandb
GROUP BY order
ORDER BY id
DESC LIMIT 30");
While fetching the last 30 order numbers count the number of units and the first and last UID for each order.
while($row=mysqli_fetch_array($result)){
$count = mysqli_fetch_array(mysqli_query($con, "SELECT order, COUNT(*) as count FROM scandb WHERE order ='".$row['order']."' "));
$firstLast = mysqli_fetch_array(mysqli_query($con, "SELECT (SELECT UID FROM scandb WHERE orderNumber ='".$row['order']."' ORDER BY UID LIMIT 1) as 'first', (SELECT UID FROM barcode WHERE order ='".$row['order']."' ORDER BY UID DESC LIMIT 1) as 'last'"));
echo "<td align= center>".$count['count']."</td>";
echo "<td align= center>".$firstLast['first']."</td>";
echo "<td align= center>".$firstLast['last']."</td>";
}
With 100K lines in the database this whole query is taking about 3 seconds. The majority of the time is in the $count and $firstlast queries. I'd like to know if there is a more efficient way to get this same data in a faster time without Indexing the table. Any special tricks that anyone has would be greatly appreciated.
Design your database with caution
This first tip may seems obvious, but the fact is that most database problems come from badly-designed table structure.
For example, I have seen people storing information such as client info and payment info in the same database column. For both the database system and developers who will have to work on it, this is not a good thing.
When creating a database, always put information on various tables, use clear naming standards and make use of primary keys.
Know what you should optimize
If you want to optimize a specific query, it is extremely useful to be able to get an in-depth look at the result of a query. Using the EXPLAIN statement, you will get lots of useful info on the result produced by a specific query, as shown in the example below:
EXPLAIN SELECT * FROM ref_table,other_table WHERE ref_table.key_column=other_table.column;
Don’t select what you don’t need
A very common way to get the desired data is to use the * symbol, which will get all fields from the desired table:
SELECT * FROM wp_posts;
Instead, you should definitely select only the desired fields as shown in the example below. On a very small site with, let’s say, one visitor per minute, that wouldn’t make a difference. But on a site such as Cats Who Code, it saves a lot of work for the database.
SELECT title, excerpt, author FROM wp_posts;
Avoid queries in loops
When using SQL along with a programming language such as PHP, it can be tempting to use SQL queries inside a loop. But doing so is like hammering your database with queries.
This example illustrates the whole “queries in loops” problem:
foreach ($display_order as $id => $ordinal) {
$sql = "UPDATE categories SET display_order = $ordinal WHERE id = $id";
mysql_query($sql);
}
Here is what you should do instead:
UPDATE categories
SET display_order = CASE id
WHEN 1 THEN 3
WHEN 2 THEN 4
WHEN 3 THEN 5
END
WHERE id IN (1,2,3)
Use join instead of subqueries
As a programmer, subqueries are something that you can be tempted to use and abuse. Subqueries, as show below, can be very useful:
SELECT a.id,
(SELECT MAX(created)
FROM posts
WHERE author_id = a.id)
AS latest_post FROM authors a
Although subqueries are useful, they often can be replaced by a join, which is definitely faster to execute.
SELECT a.id, MAX(p.created) AS latest_post
FROM authors a
INNER JOIN posts p
ON (a.id = p.author_id)
GROUP BY a.id
Source: http://20bits.com/articles/10-tips-for-optimizing-mysql-queries-that-dont-suck/
I have a pretty simple query,
$query3 = $db->query("SELECT * FROM mybb_ranks WHERE id='1' ORDER by points DESC");
And what it'll return is a database of people who are registered and ranked. Since players points can be randomly changing due to matches, we determine the rank # by assigning it once the value is fetched in a way like this:
$i = 1;
while($row = mysqli_fetch_array($query5))
{
echo "$row[player]'s rank is $i";
$i++;
}
As you can see, it orders the player's by their points, but determines the rank # from a variable that adds after displaying every rank. However, in each of the user profiles, I would like to display their rank as well. This is a bit more difficult, because I need a certain way to count the amount of rows the query has to go through in order to get to the certain player. So for instance, if PlayerB is ranked at #5, I would need a way to display that on his own profile. For this to happen, I imagine the query would need to be altered to be able to count each individual row (4 rows) before it reaches the certain player on the 5th row. I was wondering, how would I go about this?
Try this:
UPDATE mybb_ranks
LEFT JOIN (
SELECT
player,
#rank:=#rank+1 as current_rank
FROM mybb_ranks
WHERE id='1'
ORDER BY points DESC
) AS t
ON mybb_ranks.player = t.player
SET mybb_ranks.rank = t.current_rank
That means you have to create additional column rank in your mybb_ranks table.
This query will update the rank of user each time you generate your ranks list.
So when you need to show user's rank in his profile page you just request it from the table:
SELECT rank
FROM mybb_ranks
WHERE player = :player_id
if you want it more dynamic, you can run this UPDATE query every time when you generate your player profile page, right before SELECT rank.
I'm building my own advertisement platform, and I have a little problem. How can I show my ads an equal number of times?
So for example:
Name | Views
Ads 1 | 100
Ads 2 | 98
Ads 3 | 99
So my system need to show the ads with the least views, in this case "Ads 2 or Ads 3".
So all my ads follow each others views. So when my 3 ads have 3.000 views total, there should be 1.000 views on every view.
I'm coding in PHP, and I don't have an example, because I need inspiration how to fix my problem.
Select your least viewed add like this:
SELECT * FROM ads ORDERBY views ASC LIMIT 0, 1
This way, all the ads with less views will slowly catch up.
-- Edit, using your next requirement
probabilityForHighestScore = 30;
random = rand(0, 100);
if (random > probabilityForHighestScore)
SELECT * FROM ads ORDERBY views ASC LIMIT 0, 1
else
SELECT * FROM ads ORDERBY score DESC LIMIT 0, 1
If you need something else, you'd better explain you whole requirement first. Because if it is not clear for you, it won't be clear in your question, and the answers won't do what you want.
I'm assuming you have a database table containing information about these ads. You could add, if you don't already have it, a views field to that table. Then, whenever you need to display an ad, you just grab the one with the lowest view count, add 1 to it's view counter, and display the ad.
Edit:
The problem with #MarvinLabs solution, as I explain in the comments, is that it's giving a huge bonus to a single record.
Let's say you have 50 separate ads in your system. Let's also say that your highest scoring record has a score of 9.9/10, and your second highest scoring record has a score of 9.8/10. Both of these are very high scoring items, but if you use #MarvinLabs code, the highest scoring record will get 30% of all views, while the second highest scoring record will get 1.4% of all views (70 percent of all views divided across the 49 non-highest scoring ads).
What you might want to consider is allowing for a larger range of high scoring ads to be included. You can do this in any one of three ways:
First, you can set a threshold, or multiple thresholds, which divide a certain percentage of views to certain ranges of scores. For example, you could have it so that ads which score more than 9/10 get 30% of all views. You would do that like this:
$random = rand(1,100);
if ($random > 30) {
$sql = "SELECT * FROM ads WHERE score >= 9 ORDER BY views ASC";
} else {
$sql = "SELECT * FROM ads WHERE score < 9 ORDER BY views ASC";
}
The problem with this is that if you don't have any ads with a score above 9, you won't get anything back. For that reason, you probably don't want to use this method.
Second, you could spread your 30% of views across the top 5 or 10 ads:
SELECT *
FROM ads
WHERE id IN
(SELECT id
FROM ads
ORDER BY score DESC
LIMIT 10)
ORDER BY views ASC;
This solves the problem of "what if I don't have any records above the threshold" while still spreading the "high score bonus" across more than just a single record. The problem with this, if you consider it a problem that is, is that it doesn't scale with the volume of ads you have on record. Whether you have 10 records or 10,000 records, you'll still give the bonus to just 10 (or 20, or 50.. whatever you set) records.
If you want to scale, you'll want the third solution.
The third solution is to set your limit based on a percentage of the total number of records in the table. Since MySQL doesn't have a built-in way of handling this, you'll need to workaround this in one of two ways:
One way to do it the lazy way and run two queries - one to get the current record count, and another to create a query based on it. For example:
$query1 = "SELECT COUNT(*) FROM ads";
//store result in $count
$percentage = round($count * 0.10); //get 10% of records
$query2 = "SELECT * FROM ads WHERE id IN " .
"(SELECT id " .
" FROM ads " .
" ORDER BY score DESC " .
" LIMIT " . $percentage . ") " .
"ORDER BY views ASC"
A better way would be to avoid the second round-trip to the database and use a prepared statement:
SELECT #percentage := ROUND(COUNT(*) * 10/100) FROM ads;
PREPARE PERCENTAGE FROM
SELECT *
FROM ads
WHERE id IN
(SELECT id FROM ads
ORDER BY score DESC
LIMIT ?)
ORDER BY views ASC;
EXECUTE PERCENTAGE USING #percentage;