Limit number of MySQL results per user - php

I have a site where users can list items for sale. The front of the site displays random items on sale using the following SQL.
SELECT * FROM auctions
WHERE closed = 0 AND suspended = 0 AND starts <= 1390990443
ORDER BY RAND() LIMIT 30
The problem is that one users has a lot of sales to the extent that the front of the site is mostly that one user. I would like to limit the items displayed by the user using the auction.user column to five items per user.
I am hoping that there is SQL to do this in one line, if not then advice or a link on who to build up results on multiple queries would be appreciated. Coded language is PHP.
Edit: database structure is here. http://pastebin.com/3ua18k4h

Personally I would try and form some solution pulling results into PHP and filtering them there, but that's because I feel more comfortable in PHP than MySQL. However you could do something like this without the PHP filtering and just via your MySQL query, no idea how it would do scaling up to lots of users though (not sure on your full table schema, so I improvised)
SELECT ID, USERNAME, AUCTION_ID
FROM
(
SELECT *, #row:=IF(username=#username,#row,0)+1 AS auctioncount, #username:=username FROM
(SELECT *,RAND() AS trand FROM table1) t1,
(SELECT #row:=0,#username:='') tm2
ORDER BY username,trand
) t2
WHERE auctioncount<=5 LIMIT 30
SQL fiddle: http://sqlfiddle.com/#!2/9bd47/1
You can change auctioncount to the maximum number of listings you want per user.

You can get more results with MYSQL with a bigger LIMIT and then create an array for every items for sale by user and finally take only 5 items per user, so you will only use 1 request to Mysql and more CPU usage, could be a good idea if you have a lot of traffic on your web site.
1) get more results :
SELECT * FROM auctions
WHERE closed = 0 AND suspended = 0 AND starts <= 1390990443
ORDER BY RAND() LIMIT 1000
2) Loop and store per items per user :
foreach($results as $item) {
array_push($itemsPerUser[$item['userId']], $item);
}
3) Filter only 5 items per user:
foreach($itemsPerUser as $user => $items) {
$fiveItemsPerUser = array_slice($items, 0, 5);
}
P.S: this is pseudo-code, you should add more check on array length etc ...

Not quite a direct answer to your problem, but perhaps following is enough:
Select 30 random users with running auctions, and display the oldest auction of each of them:
SELECT * FROM auctions WHERE id IN (
SELECT MIN(id) FROM auctions
WHERE closed = 0 AND suspended = 0 AND starts <= 1390990443
GROUP BY userid ORDER BY RAND() LIMIT 30
)
At least you can remove the newlines and have it all in a single line sql query, which can't be done with the exact query you requested.

Here's something to think about, although I appreciate that the intellectual leap from this to a (fair and) working solution might be a step too far!...
SELECT RAND(#i:=RAND()*1000);
+-----------------------+
| RAND(#i:=RAND()*1000) |
+-----------------------+
| 0.7903550134881911 |
+-----------------------+
SELECT RAND(#i);
+--------------------+
| RAND(#i) |
+--------------------+
| 0.7903550134881911 |
+--------------------+
SELECT RAND(#i:=RAND()*1000);
+-----------------------+
| RAND(#i:=RAND()*1000) |
+-----------------------+
| 0.9555754568014065 |
+-----------------------+

Related

display huge data in batches of 100 every hour in mysql/php

I have a database with more than 600 rows but I can only retrieve/display 100 every hour. So I use
select * from table ORDER BY id DESC LIMIT 100
to retrieve the first 100. How do I write a script that will retrieve the data in batches of 100 every 1hr so that I can use it in a cron job?
Possible solution.
Add a field for to mark the record was already shown.
ALTER TABLE tablename
ADD COLUMN shown TINYINT NULL DEFAULT NULL;
NULL will mean that the record was not selected, 1 - that record is marked for selection, 0 - that record was already selected.
When you need to select up to 100 records you
2.1. Mark records to be shown
UPDATE tablename
SET shown = 1
WHERE shown = 1
OR shown IS NULL
ORDER BY shown = 1 DESC, id ASC
LIMIT 100;
shown = 1 condition in WHERE considered the fact that some records were marked but were not selected due to some error. shown = 1 DESC re-marks such records before non-marked.
If there is 100 or less records which were not selected all of them will be marked, else only 100 records with lower id (most ancient) will be marked.
2.2. Select marked records.
SELECT *
FROM tablename
WHERE shown = 1
ORDER BY id
LIMIT 100;
2.3. Mark selected records.
UPDATE tablename
SET shown = 0
WHERE shown = 1
ORDER BY id
LIMIT 100;
This is applicable when only one client selects the records.
If a lot of clients may work in parallel, and only one cliens must select a record, then use some cliens number (unique over all clients) for to mark a record for selection instead of 1.
Of course if there is only one client, and you guarantee that selection will not fail, you may simply store last shown ID somewhere (on the client side, or in some service table on the MySQL side) and simply select "next 100" starting from this stored ID:
SELECT *
FROM tablename
WHERE id > #stored_id
ORDER BY id
LIMIT 100;
and
SELECT MAX(id)
FROM tablename
WHERE id > #stored_id
ORDER BY id
LIMIT 100;
for to store instead of previous #stored_id.
Thank you #Akina and #Vivek_23 for your contributions. I was able to figure out an easier way to go about it.
Add a new field to table, eg shownstatus
Create a cronjob to display 100 (LIMIT 100) records with their shownstatus not marked as shown from table every hour and then update each record's shownstatus to shown NB. If I create a cronjob to run every hour for the whole day, I can get all records displayed and their shownstatus updated to shown by close of day.
Create a second cronjob to update all record's shownstatus to notshown
The downside to this is that, you can only display a total of 2,400 records a day. ie. 100 records every hour times 24hrs. So if your record grows to about 10,000. You will need to set your cronjob to run for atleast 5 days to display all records.
Still open to a better approach if there's any, but till then, I will have to just stick to this for now.
Let's say you made a cron that hits a URL something like
http://yourdomain.com/fetch-rows
or a script for instance, like
your_project_folder/fetch-rows.php
Let's say you have a DB table in place that looks something like this:
| id | offset | created_at |
|----|--------|---------------------|
| 1 | 100 | 2019-01-08 03:15:00 |
| 2 | 200 | 2019-01-08 04:15:00 |
Your script:
<?php
define('FETCH_LIMIT',100);
$conn = mysqli_connect(....); // connect to DB
$result = mysqli_query($conn,"select * from cron_hit_table where id = (select max(id) from cron_hit_table)")); // select the last record to get the latest offset
$offset = 0; // initial default offset
if(mysqli_num_rows($result) > 0){
$offset = intval(mysqli_fetch_assoc($result)['offset']);
}
// Now, hit your query with $offset included
$result = mysqli_query($conn,"select * from table ORDER BY id DESC LIMIT $offset,100");
while($row = mysqli_fetch_assoc($result)){
// your data processing
}
// insert new row to store next offset for next cron hit
$offset += FETCH_LIMIT; // increment current offset
mysqli_query($conn,"insert into cron_hit_table(offset) values($offset)"); // because ID would be auto increment and created_at would have default value as current_timestamp
mysqli_close($conn);
Whenever cron hits, you fetch last row from your hit table to get the offset. Hit the query with that offset and store the next offset for next hit in your table.
Update:
As pointed out by #Dharman in the comments, you can use PDO for more abstracted way of dealing with different types of database(but make sure you have appropriate driver for it, see checklist of drivers PDO supports to be sure) along with minor checks of query syntaxes.

Select random row per distinct field value?

I have a MySQL query that results in something like this:
person | some_info
==================
bob | pphsmbf24
bob | rz72nixdy
bob | rbqqarywk
john | kif9adxxn
john | 77tp431p4
john | hx4t0e76j
john | 4yiomqv4i
alex | n25pz8z83
alex | orq9w7c24
alex | beuz1p133
etc...
(This is just a simplified example. In reality there are about 5000 rows in my results).
What I need to do is go through each person in the list (bob, john, alex, etc...) and pull out a row from their set of results. The row I pull out is sort of random but sort of also based on a loose set of conditions. It's not really important to specify the conditions here so I'll just say it's a random row for the example.
Anyways, using PHP, this solution is pretty simple. I make my query and get 5000 rows back and iterate through them pulling out my random row for each person. Easy.
However, I'm wondering if it's possible to get what I would from only a MySQL query so that I don't have to use PHP to iterate through the results and pull out my random rows.
I have a feeling it might involve a BUNCH of subselects, like one for each person, in which case that solution would be more time, resource and bandwidth intensive than my current solution.
Is there a clever query that can accomplish this all in one command?
Here is an SQLFiddle that you can play with.
To get a random value for a distinct name use
SELECT r.name,
(SELECT r1.some_info FROM test AS r1 WHERE r.name=r1.name ORDER BY rand() LIMIT 1) AS 'some_info'
FROM test AS r
GROUP BY r.name ;
Put this query as it stands in your sqlfiddle and it will work
Im using r and r1 as table alias names. This will also use a subquery to select a random some_info for the name
SQL Fiddle is here
My first response would be to use php to generate a random number:
$randId = rand($min, $max);
Then run a SQL query that only gets the record where your index equals $randID.
Here is the solution:
select person, acting from personel where id in (
select lim from
(select count(person) c, min(id) i, cast(rand()*(count(person)-1) +min(id)
as unsigned) lim from personel group by person order by i) t1
)
The table used in the example is below:
create table personel (
id int(11) not null auto_increment,
person char(16),
acting char(19),
primary key(id)
);
insert into personel (person,acting) values
('john','abd'),('john','aabd'),('john','adbd'),('john','abfd'),
('alex','ab2d'),('alex','abd3'),('alex','ab4d'),('alex','a6bd'),
('max','ab2d'),('max','abd3'),('max','ab4d'),('max','a6bd'),
('jimmy','ab2d'),('jimmy','abd3'),('jimmy','ab4d'),('jimmy','a6bd');
You can limit the number of queries, and order by "rand()" to get your desired result.
Perhaps if you tried something like this:
SELECT name, some_info
FROM test
WHERE name = 'tara'
ORDER BY rand()
LIMIT 1

SQLite queries running slowly, need optimization help

I have a SQLite DB with about 24k records in one of the tables, 15 in the other. The table with 15 records holds information about forms that need to be completed by users (roughly 1k users). The table with 24k records holds information about which forms have been completed by who, and when. When a user logs in, there is about a ~3/4 second wait time while the queries run to determine what the user has finished so far. Too long for my client. I know I can't be doing my queries in the best way, because they are contained within a loop. But I cannot seem to figure out how to optimize my query.
The queries run as follows:
1) Select all of the forms and information
$result = $db->query("SELECT * FROM tbl_forms");
while($row = $result->fetchArray()){
//Run other query 2 here
}
2) For each form/row, run a query that figures out what is the most recent completion information about that form for the user.
$complete = $db->querySingle("SELECT * FROM tbl_completion AS forms1
WHERE userid='{$_SESSION['userid']}' AND form_id='{$row['id']}' AND forms1.id IN
(SELECT MAX(id) FROM tbl_completion
GROUP BY tbl_completion.userid, tbl_completion.form_id)", true);
There are 15 forms, so there is a total of 16 queries running. However, with my table structure, I'm unsure as how to get the "most recent" (aka max form id) form information using 1 joined query instead.
My table structure looks like so:
tbl_forms:
id | form_name | deadline | required | type | quicklink
tbl_completion:
id | userid | form_id | form_completion | form_path | timestamp | accept | reject
Edit: Index on tbl_forms (id), Index on tbl_forms (id, form_name), Index on tbl_complete (id)
I've tried using a query that is like:
SELECT * FROM tbl_completion AS forms1
LEFT OUTER JOIN tbl_forms ON forms1.form_id = tbl_forms.id
WHERE forms1.userid='testuser' AND forms1.id IN
(SELECT MAX(id) FROM tbl_completion GROUP BY tbl_completion.userid, tbl_completion.form_id)
Which will give me the most up-to-date information about the forms completed, as well as the form information, but the only problem with this is I need to output all the forms in a table (like: Form 1-Incomplete, Form 2-Completed, etc) I cannot seem to figure out how to get it to work with the left table being tbl_forms and getting all form info, as well as "latest" form tbl_completion info. I also tried doing a 3 LOJ with the last "table" as a temp table holding the maxid, but it was very slow AND didn't give me what I wanted.
Can anybody help?? Is there a better optimized query I can run once, or can I do something else on the DB side to speed this up? Thank you in advance.
You're missing indexes. See:
DOs and DONTs for Indexes
Also, the SELECT MAX(id) FROM tbl_completion GROUP BY tbl_completion.userid, tbl_completion.form_id could presumably discard unneeded rows if you toss in your userid in a where clause.
It sounds like you might be running into the concurrency limitations of SQLite. SQLite does not support concurrent writes, so if you have a lot of users, you end up having a lot of contention. You should consider migrating to another DBMS in order to satisfy your scaling needs.

count the number of comments (php/mysql)

i am using this code so i can count the number of comments for each article
SELECT *, COUNT(comment_id) as count
FROM article_comments
WHERE article_id =colname
GROUP BY article_id
this is what my comment table look like
http://i54.tinypic.com/2cdu3dk.png
i want to save these number in another table (the articles table.. each number next to it's article ) like this
http://i54.tinypic.com/2dgm82u.png
and when the user enter a comment..the number change automatically
someone help me with the code
or if there is another way to do this
i know it's a long question
but i have been trying to solve this for like..forever
thanx
You could set a TRIGGER that updates the comment count table every time a comment is added. Or you could simply add the UPDATE query right after the INSERT query in your comment page.
You probably do not need a lookup table. 1 article has many comments. Therefore, structure your comments table something like this (add an article field);
id | article | content
-------------------------
1 | 1 | Comment 1 for article 1.
2 | 1 | Comment 2 for article 1.
3 | 2 | Comment 3 for article 2.
When displaying your article, list comments using the following query;
SELECT a.id, a.content FROM articles a WHERE a.article = :myArticleId
When creating a new comment:
INSERT INTO comments (article, content) VALUES (:currentArticleId, :content)
UPDATE article SET commentCount = commentCount + 1 WHERE article = :currentArticleId
The articles table will look something like this;
id | commentCount | content
------------------------------
1 | 0 | Article with 0 comments.
2 | 3 | Article with 3 comments.
This requires some work on your part, but it has more benefits than drawbacks.
Your proposed solution has 2 large drawbacks;
COUNT() in SQL does not scale very well and can be slow, normally it can be avoided.
The lookup table adds unnecessary complexity to your application.
Triggers should also always be avoided. They create "magic" conditions - your database can be changed without you knowing about it. Triggers are often more difficult to change than code too.
$query = mysql_query("SELECT * FROM article_comments WHERE article_id =".$youarticleId);
//the number of comments is :
$number_Of_Comments = mysql_num_rows($query);
//save it to another table
$query2 = mysql_query("UPDATE yourTable set numberOfComments =".$number_Of_Comments);
on saving comments, try to:
update table_where_you_count_the_comments set number_of_comments = number_of_comments +1 where article_id = theID limit 1;
or look for mysql triggers.
you're asking the sql server to select everything and the count id at the same time, use one of them and give it a where close, and Bingo!

Trouble combining Two sql queries into one

I have a table which contains due dates for individual member records. Each row contains four fields:
ID | Next_Due | Next_Due_Missed | Amount
=============================================================
123 | 2010-12-05 | NULL | 41.32
456 | 2010-12-10 | 2010-12-05 | 21.44
789 | 2010-12-20 | 2010-12-10 | 39.99
ID is the unique id of each MEMBER
Next Due - is the next due day of their regular subscription period
Next_Due_Missed is populated ONLY if there was an error collecting the first round of subscription payment.
Amount is amount owned for subscription.
My goal is to create a sql query that checks if next_due_missed exists and is not null. If it does, use that value as the '$date'. If not, set $date = value of next_due
this is done easily enough except my results are grouped by Next_Due in normal circumstances and will omit next_due_missed if I combine the way I currently am.
Every payment period, there may be 600+ records with next_due equal to the desired date (and 10-15 equal to next_due_missed).
My current query is:
$stmt = $db->prepare("SELECT next_due, next_due_missed FROM table_name WHERE (next_due > CURDATE() OR next_due_missed > CURDATE()) GROUP BY next_due ASC");
This only returns results for next_due however. Omitting the GROUP BY clause returns hundreds of results (while I need to group in this stage).
Similarly at a later point, I will need to break out those individual records and actually create payment records based on the 'next_due' and 'next_due_missed' values.
Any ideas what I am missing?
I am not sure the purpose of your GROUP BY other than to get DISTINCT values, but left it in in case you provided a partial query:
SELECT coalesce(next_due_missed, next_due) as EffectiveNextDue
FROM table_name
WHERE coalesce(next_due_missed, next_due) > CURDATE()
GROUP BY coalesce(next_due_missed, next_due)

Categories