PHP / MySQL nth row with DATE_SUB, etc - php
So I have a table, I use to graph some data.
id, agcid, ooscount, date
Data is samples every 5 minutes for 18 objects
ID is auto increment, agcid represents the ID of the object, date is self explanitory and ooscount is an integer column.
I'm currently using this:
$sql2 = "SELECT ooscount, date
FROM ooscount
where agcid = '".$agcid."' AND date >= DATE_SUB(NOW( ),INTERVAL 7 HOUR)
order by date DESC";
Simple, yet effective, however I have an extremely large dataset which complicates the graphs.
I tried adding AND ooscount.id mod 3 = 0 to this however, the data sets on each agcid seemed very random, some had only a couple, others had 5x more than they should.
I then tried an approach like this:
$sql2 = "set #row:=-1;
SELECT ooscount.*
FROM
ooscount
INNER JOIN
(
SELECT id
FROM
(
SELECT #row:=#row+1 AS rownum, id
FROM
(
SELECT id, date, ooscount FROM ooscount where agcid = '".$agcid."' and date >= DATE_SUB(NOW( ),INTERVAL 12 HOUR) order by date DESC
) AS sorted
) as ranked
WHERE rownum % 3 = 0
) AS subset
ON subset.id = ooscount.id;";
It's commented out, forgive the //'s. However this approach while it works in phpMyAdmin, gives me this:
PHP Fatal error: Call to a member function fetch_assoc()
on a non-object in /var/www/khamlin/oosagc.php on line 145
Lastly, the portion of code to display this:
$result2 = $conn->query($sql2);
$rowsarray = "";
//var_dump($result2);
while ($row2 = $result2->fetch_assoc()) {
I believe the problem is that examples I've found don't expect multiple data sets on a single table, and thus ID would be incremental (ie: 1,2,3,4,5,6, etc), rather than ( 168006, 168023, 168041, 168060 ).
How would I go about changing this to suit my needs here?
Structure
CREATE TABLE IF NOT EXISTS `ooscount` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`agcid` int(3) NOT NULL,
`ooscount` int(10) NOT NULL,
`date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=168580 ;
````
Sample Data: https://jpst.it/1bfFt
Current Output: a data point of ooscount and date every 5 minutes (ie:
data: [{"ooscount":"4907","date":"2018-02-21 09:40:01"},{"ooscount":"4905","date":"2018-02-21 09:35:02"},{"ooscount":"4910","date":"2018-02-21 09:30:02"},{"ooscount":"4904","date":"2018-02-21 09:25:01"},{"ooscount":"4900","date":"2018-02-21 09:20:02"},{"ooscount":"4898","date":"2018-02-21 09:15:02"},{"ooscount":"4989","date":"2018-02-21 09:10:01"},{"ooscount":"5008","date":"2018-02-21 09:05:02"},{"ooscount":"4933","date":"2018-02-21 09:00:01"},{"ooscount":"4915","date":"2018-02-21 08:55:01"},{"ooscount":"4903","date":"2018-02-21 08:50:02"},{"ooscount":"4898","date":"2018-02-21 08:45:01"},{"ooscount":"4899","date":"2018-02-21 08:40:01"},{"ooscount":"4909","date":"2018-02-21 08:35:01"},{"ooscount":"4918","date":"2018-02-21 08:30:01"},{"ooscount":"4906","date":"2018-02-21 08:25:02"},{"ooscount":"4906","date":"2018-02-21 08:20:01"},{"ooscount":"4900","date":"2018-02-21 08:15:02"},{"ooscount":"4895","date":"2018-02-21 08:10:01"},{"ooscount":"4883","date":"2018-02-21 08:05:02"},{"ooscount":"4881","date":"2018-02-21 08:00:01"},{"ooscount":"4879","date":"2018-02-21 07:55:02"},{"ooscount":"4883","date":"2018-02-21 07:50:02"},{"ooscount":"4875","date":"2018-02-21 07:45:02"},{"ooscount":"4875","date":"2018-02-21 07:40:01"},{"ooscount":"4879","date":"2018-02-21 07:35:02"},{"ooscount":"4882","date":"2018-02-21 07:30:01"},{"ooscount":"4888","date":"2018-02-21 07:25:01"},{"ooscount":"4877","date":"2018-02-21 07:20:01"},{"ooscount":"4879","date":"2018-02-21 07:15:02"},{"ooscount":"4884","date":"2018-02-21 07:10:02"},{"ooscount":"4886","date":"2018-02-21 07:05:02"},{"ooscount":"4904","date":"2018-02-21 07:00:02"},{"ooscount":"4906","date":"2018-02-21 06:55:01"},{"ooscount":"4904","date":"2018-02-21 06:53:32"},{"ooscount":"4904","date":"2018-02-21 06:50:02"},{"ooscount":"4892","date":"2018-02-21 06:45:01"},{"ooscount":"4795","date":"2018-02-21 06:40:02"},{"ooscount":"4793","date":"2018-02-21 06:35:01"},{"ooscount":"4799","date":"2018-02-21 06:30:02"},{"ooscount":"4797","date":"2018-02-21 06:25:02"},{"ooscount":"4802","date":"2018-02-21 06:20:01"},{"ooscount":"4801","date":"2018-02-21 06:15:01"},{"ooscount":"4792","date":"2018-02-21 06:10:02"},{"ooscount":"4798","date":"2018-02-21 06:05:02"},{"ooscount":"4797","date":"2018-02-21 06:00:01"},{"ooscount":"4798","date":"2018-02-21 05:55:02"},{"ooscount":"4794","date":"2018-02-21 05:50:01"},{"ooscount":"4796","date":"2018-02-21 05:45:02"},{"ooscount":"4804","date":"2018-02-21 05:40:02"},{"ooscount":"4803","date":"2018-02-21 05:35:01"},{"ooscount":"4809","date":"2018-02-21 05:30:02"},{"ooscount":"4811","date":"2018-02-21 05:25:01"},{"ooscount":"4810","date":"2018-02-21 05:20:01"},{"ooscount":"4831","date":"2018-02-21 05:15:02"},{"ooscount":"4839","date":"2018-02-21 05:10:02"},{"ooscount":"4859","date":"2018-02-21 05:05:01"},{"ooscount":"4859","date":"2018-02-21 05:00:01"},{"ooscount":"4858","date":"2018-02-21 04:55:02"},{"ooscount":"4858","date":"2018-02-21 04:50:02"},{"ooscount":"4863","date":"2018-02-21 04:45:02"},{"ooscount":"4868","date":"2018-02-21 04:40:01"},{"ooscount":"4872","date":"2018-02-21 04:35:01"},{"ooscount":"4868","date":"2018-02-21 04:30:02"},{"ooscount":"4867","date":"2018-02-21 04:25:02"},{"ooscount":"4870","date":"2018-02-21 04:20:01"},{"ooscount":"4866","date":"2018-02-21 04:15:02"},{"ooscount":"4864","date":"2018-02-21 04:10:02"},{"ooscount":"4863","date":"2018-02-21 04:05:01"},{"ooscount":"4874","date":"2018-02-21 04:00:02"},{"ooscount":"4881","date":"2018-02-21 03:55:01"},{"ooscount":"4850","date":"2018-02-21 03:50:02"},{"ooscount":"4846","date":"2018-02-21 03:45:01"},{"ooscount":"4847","date":"2018-02-21 03:40:02"},{"ooscount":"4850","date":"2018-02-21 03:35:02"},{"ooscount":"4845","date":"2018-02-21 03:30:01"},{"ooscount":"4847","date":"2018-02-21 03:25:02"},{"ooscount":"4848","date":"2018-02-21 03:20:01"},{"ooscount":"4847","date":"2018-02-21 03:15:02"},{"ooscount":"4852","date":"2018-02-21 03:10:03"},{"ooscount":"4854","date":"2018-02-21 03:05:01"},{"ooscount":"4864","date":"2018-02-21 03:00:01"},{"ooscount":"4867","date":"2018-02-21 02:55:02"},{"ooscount":"4868","date":"2018-02-21 02:50:01"},{"ooscount":"4862","date":"2018-02-21 02:45:02"}],
The goal is to have less data points over the same amount of time.
I found a solution that works for me.
$sql2 = "SELECT *
FROM (
SELECT #row := #row +1 AS rownum, ooscount.*
FROM (
SELECT #row :=0) r, ooscount where agcid='".$agcid."' and date >= DATE_SUB(NOW( ),INTERVAL 12 HOUR)
) ranked
WHERE rownum %3 =1";
Related
SELECT rows that are referenced less than x times in another table
In MySQL I have two tables, my reservable "weekend":
id bigint(20) unsigned,
label varchar(64),
date_start date,
max_attendees smallint(5) unsigned
And my attendees:
id bigint(20) unsigned,
name varchar(64),
email varchar(255),
weekend bigint(20) unsigned
I want to select all weekends that have attendees less than their max_attendees. This includes weekends that have 0 attendees.
Note: I also need to ignore weekend with id "1";
Currently, this works fine with PHP (I'm using Wordpress for mysql access), like so:
$weekends = $wpdb->get_results("SELECT * FROM $weekends_table
WHERE id <> 1", ARRAY_A);
$open_weekends = array();
foreach ($weekends as $weekend) {
$id = $weekend['id'];
$attendees = $wpdb->get_row("SELECT COUNT(id) as attendees
FROM $attendees_table
WHERE weekend = $id", ARRAY_A);
if ( $attendees['attendees'] < $weekend['max_attendees'] ) {
$weekend['attendees'] = $attendees['attendees'];
$open_weekends[] = $weekend;
}
}
Shouldn't I be able to do this in MySQL without the PHP? My knowledge of MySQL doesn't extend that far. Can you suggest a query?
use the HAVING clause This is untested, so you may have to play with it, but here's the gist: SELECT w.*, COUNT(a.name) FROM weekend w LEFT JOIN attendees a ON w.id = a.weekend WHERE w.id <> 1 GROUP BY w.id HAVING (COUNT(a.name) < w.max_attendees) OR (COUNT(a.name) IS NULL)
A very simple approach would be this: SELECT COUNT($attendees_table.id) as attendees attendees_table.max_attendees as maximum FROM weekends_table, attendees_table WHERE attendees_table.weekend = weekends_table.id GROUP BY weekends_table.id You could use a JOIN ON attendees_table. This should be possible as well: SELECT COUNT(attendees_table.id) as attendees weekends_table.max_attendees as maximum FROM weekends_table, attendees_table WHERE attendees_table.weekend = weekends_table.id GROUP BY weekends_table.id HAVING attendees < maximum This is all untested. I don't have your tables or data, but it might get you going? Ah, it didn't get what you wanted. To include zero attendees you can use a subselect: SELECT weekends_table.id AS weekend_id FROM weekends_table WHERE weekends_table.max_attendees > (SELECT COUNT(*) FROM attendees_table WHERE attendees_table.weekend = weekends_table.id) It should return weekend id's where there's at least room for one more attendee. Again, completely untested, but perhaps it works?
Better way to find the latest change in sql value
I have 2 tables, one of which lists items, the other of which keeps track of when items changed in price.
price_table:
price_id int
item_id int
price_amount float
change_date date // this date is stored as Y-m-d, without a time
and
item_table:
item_id int
item_name char(40)
item_description char(40)
I know that the following finds the item_ids changed at the current date:
SELECT item_id
FROM price_table
WHERE change_date = "$today"
when
$today = date('Y-m-d');
I then run another query for EACH item_id returned (this is why I think my method is inefficient)
SELECT card_price
FROM price_table
WHERE item_id
ORDER BY change_date ASC
And get the last 2 values from the PHP array, comparing them to get the difference in price. My test had an inner join to return the item_name as well, but I've removed that to simplify the question. Is there a better way to do this? And if so, can it be expanded to use the last 2 or 3 days as the change criteria instead?
This is what triggers are for, I'm also going to talk about DATE_SUB and date intervals Using triggers will allow you to keep the current price in the price table and keep historical prices in a separate table. For instance, consider creating a new table called price_table_old price_table_old: price_id int item_id int price_amount float change_date date On the price_table, create an 'update' trigger... insert into price_table_old ( `item_id`, `price_amount` , `change_date ` ) VALUES ( OLD.`item_id` , OLD.`price_amount` , NOW() ) To help you with trigger syntax, I copied the trigger I used for a staff table below When ever someone changes a price, the price_table_old table is automatically updated. From here, things are a little more logical to get the current price and comparisons of previous prices. You could run a query like.... SELECT price_table.item_id , price_table.price_amount , (price_table.price_amount - price_table_old.price_amount ) AS pricediff , price_table_old.change_date FROM price_table LEFT JOIN price_table_old ON price_table_old.item_id = price_table.item_id ORDER BY price_table.item_id , price_table_old.change_date DESC or stick a where in there to zone in on a specific item and/or date range and/or a limit to get the last (say) 3 price changes. For instance, to get a list of price changes for the last 3 days on all items, you could use a statement like this SELECT price_table.item_id , price_table.price_amount , (price_table.price_amount - price_table_old.price_amount ) AS pricediff , price_table_old.change_date FROM price_table LEFT JOIN price_table_old ON price_table_old.item_id = price_table.item_id WHERE price_table_old.change_date > ( DATE_SUB( NOW() , INTERVAL 3 DAY )) ORDER BY price_table.item_id , price_table_old.change_date DESC Here is my staff table trigger... CREATE TRIGGER `Staff-CopyOnUpdate` BEFORE UPDATE ON `staff` FOR EACH ROW insert into staff_old ( `sid` , `title`, `firstname` , `surname` , `type`, `email` , `notify`, `actiondate`, `action` ) VALUES ( OLD.`sid` , OLD.`title`, OLD.`firstname` , OLD.`surname` , OLD.`type`, OLD.`email` , OLD.`notify`, NOW() , 'updated' )
MYSQL / PHP Help - Selecting next available dates from a table either side of original booking
I have made this table with all the dates of the years going upto 2063, to keep it simple, it contains 3 columns which have been pre-popluated, example as follows...
AutoInc | date | status
1 | 2013-05-05 | available
2 | 2013-05-06 | available
3 | 2013-05-07 | booked
4 | 2013-05-08 | booked
5 | 2013-05-09 | booked
6 | 2013-05-10 | available
7 | 2013-05-11 | booked
8 | 2013-05-12 | available
Now, checking if the dates are available is easy enough for me but the difference is this calendar is going to check if there are dates available in the block they require (4 days in a row for example) either side of the dates they originally entered.
Cutting to the chase, I have established what the next available dates are in either direction but what I need is for it to check in blocks, i.e, where the next 4 days in a row are available in either direction on the table. Would some alterations to this code be possible?
$previousDate = mysql_query("SELECT * FROM calendar WHERE autoInc < $requestedDate AND status='available' ORDER BY autoInc DESC");
If you need more please let me know,
OK, The actual story so far...
Actual table
AutoIncNo | BookingDate | Status
The calendar is in 2 parts.
Calendar to select a date
A list menu to select the amount of nights they wish to stay.
So I take the original date (the one they select) and rearrange the format to suit the table...
$CalendarDate = str_replace("/", "-", "$CalendarDate");
$QueryDate = date("Y-m-d", strtotime($CalendarDate));
Connect to the database...
include_once('../connect/connectdatabase.php');
Run the first query to check if the dates they require are available.
$QueryDate is the date they select
$NightsForQuery is the amount of nights they want to stay
$CalendarQuery = mysql_query("SELECT * FROM BookingsCalendar WHERE BookingDate='$QueryDate' LIMIT 1");
while($row = mysql_fetch_array($CalendarQuery)) {$AutoInc = $row["AutoIncNo"];}
$AutoInc2 = $AutoInc + $NightsForQuery - 2;
$SelectDates = mysql_query("SELECT * FROM BookingsCalendar WHERE AutoIncNo BETWEEN $AutoInc AND $AutoInc2");
while($row = mysql_fetch_array($SelectDates)) {
$AutoIncNo = $row["AutoIncNo"];
$BookingDate = $row["BookingDate"];
$Status = $row["Status"];
if ($Status == 'booked') {
$LastBookedDate = $BookingDate;
$LastAutoIncNo = $AutoIncNo;
$Handle = 1;
}
} // End - while($row = mysql_fetch_array($SelectDates)) {
if ($Handle !== 1) {echo 'DATES AVAILABLE >> WRITE BOOKING CODE';}
So if the handle is not equal to 1 its fine and they can book, but, if the dates arn't available (i.e, $Handle == 1) I need to check the closest available dates either side (before and after) the date they wanted where the Status is 'available' for the amount of nights they wish to stay...
So I set out to establish the first available date in either direction and thts where I got stuck. Looking at it I'm sure you could run a while loop to find the next available block inside the code above, but not sure how.
if ($Handle == 1) {
$FirstDateQuery = mysql_query("SELECT * FROM BookingsCalendar WHERE AutoIncNo < $LastAutoIncNo AND Status='available' ORDER BY AutoIncNo DESC LIMIT 1");
while($row = mysql_fetch_array($FirstDateQuery)) {
$AutoIncNo = $row["AutoIncNo"];
$BookingDate = $row["BookingDate"];
$Status = $row["Status"];
} // End - while($row = mysql_fetch_array($SelectDates)) {
$FirstBookedDate = $BookingDate;
echo $FirstBookedDate . ' - ' . $LastBookedDate;
} // End - if ($Handle == 1) {
If you have a better more efficient way or can adapt what is already here, that would be grand... My brain hurts lol.
Continuation...
I split it down to this one...
SELECT * FROM BookingsCalendar WHERE Status='available' AND AutoIncNo < $LastAutoIncNo ORDER BY AutoIncNo DESC LIMIT 4
It works mate, but, it selects the previous 4 rows individually.
So for example, if someone tries to book from 2013.06.01 but cant because its 'booked' for the next 4 days, the above script runs and brings up 2013.05.31 - 2013.05.30 - 2013.05.29 - 2013.05.28 as a result.
But if one of those dates are booked it will skip it and give me the next one (selecting as it is the next 4 that meet the condition 'available')
So if say 2013.05.29 was booked it would show 2013.05.31 - 2013.05.30 - 2013.05.28 - 2013.05.27 missing out the day which is booked.
Now the thing is that we need the next 4 rows together (undivided/continuous/without breaks in the dates) which are 'available'.
Any ideas?
UPDATE:
The thing is that it will still select the next 4 rows but not together as they appear by date. So, if someone searches for 4 days before or after the date they will get a result but they will be spread out between dates which are booked. When they book it will need it be 1 entire period, see what I mean bud?
Maybe I'm not explaining it very well so sorry about that, but the result must be in a block/together without any 'booked dates in between'.
ENTIRE QUERY:
$FirstDateQuery = mysql_query("SELECT * FROM ((SELECT * FROM BookingsCalendar WHERE Status='available' AND AutoIncNo < $LastAutoIncNo ORDER BY AutoIncNo DESC LIMIT 4)UNION (SELECT * FROM BookingsCalendar WHERE Status='available' AND AutoIncNo > $LastAutoIncNo ORDER BY AutoIncNo ASC LIMIT 4)) as avail_4 ORDER BY avail_4.AutoIncNo DESC")or die(mysql_error());
while($row = mysql_fetch_array($FirstDateQuery)) {
$AutoIncNo = $row["AutoIncNo"];
$BookingDate = $row["BookingDate"];
$Status = $row["Status"];
$avail_4 = $row["avail_4"];
echo $BookingDate . ' ' . $avail_4 . ' ' . ' ';
} // End - while($row = mysql_fetch_array($FirstDateQuery)) {
This works, but doesn't quite achieve the goal.
DATA TO PLAY WITH:
These scripts will setup the table the way I have it...
CREATE TABLE `table_setup` (
`number` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`number`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
INSERT INTO table_setup VALUES(NULL);
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
INSERT INTO table_setup SELECT NULL FROM table_setup;
DELETE FROM table_setup WHERE NUMBER > 18264;
CREATE TABLE `BookingsCalendar` (
`AutoIncNo` SMALLINT(2) NOT NULL AUTO_INCREMENT,
`BookingDate` DATE NOT NULL,
`Status` VARCHAR(10) NOT NULL DEFAULT 'available'
`InvoiceRefference` varchar(40) NOT NULL,
`CustomerName` varchar(40) NOT NULL,
`CustomerEmail` varchar(40) NOT NULL,
`CustomerPhone` varchar(30) NOT NULL,
`Address1` varchar(100) NOT NULL,
`County` varchar(40) NOT NULL,
`Country` varchar(30) NOT NULL,
`PostCode` varchar(10) NOT NULL,
PRIMARY KEY (`AutoIncNo`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
INSERT INTO BookingsCalendar (BookingDate)
SELECT DATE_ADD('2013-05-05', INTERVAL number-1 DAY)
FROM table_setup
WHERE NUMBER < 18264;
DROP TABLE `table_setup`
Just typing from the top of my head. SELECT * FROM calendar WHERE DATE(date) BETWEEN DATE_SUB($requestedDate, INTERVAL 4 DAY) AND DATE_ADD($requestedDate, INTERVAL 4 DAY) AND status='available' ORDER BY autoInc DESC HTH EDIT: New answer based on revised question SELECT * FROM ( (SELECT * FROM BookingsCalendar WHERE Status='available' AND AutoIncNo < $LastAutoIncNo ORDER BY AutoIncNo DESC LIMIT 4) UNION (SELECT * FROM BookingsCalendar WHERE Status='available' AND AutoIncNo > $LastAutoIncNo ORDER BY AutoIncNo ASC LIMIT 4) ) as avail_4 ORDER BY avail_4.AutoIncNo DESC This should work. My bad, forgot to add parentheses around inner selects.
MySQL inclusion/exclusion of posts
This post is taking a substantial amount of time to type because I'm trying to be as clear as possible, so please bear with me if it is still unclear. Basically, what I have are a table of posts in the database which users can add privacy settings to. ID | owner_id | post | other_info | privacy_level (int value) From there, users can add their privacy details, allowing it to be viewable by all [privacy_level = 0), friends (privacy_level = 1), no one (privacy_level = 3), or specific people or filters (privacy_level = 4). For privacy levels specifying specific people (4), the query will reference the table "post_privacy_includes_for" in a subquery to see if the user (or a filter the user belongs to) exists in a row in the table. ID | post_id | user_id | list_id Also, the user has the ability to prevent some people from viewing their post in within a larger group by excluding them (e.g., Having it set for everyone to view but hiding it from a stalker user). For this, another reference table is added, "post_privacy_exclude_from" - it looks identical to the setup as "post_privacy_includes_for". My problem is that this does not scale. At all. At the moment, there are about 1-2 million posts, the majority of them set to be viewable by everyone. For each post on the page it must check to see if there is a row that is excluding the post from being shown to the user - this moves really slow on a page that can be filled with 100-200 posts. It can take up to 2-4 seconds, especially when additional constraints are added to the query. This also creates extremely large and complex queries that are just... awkward. SELECT t.* FROM posts t WHERE ( (t.privacy_level = 3 AND t.owner_id = ?) OR (t.privacy_level = 4 AND EXISTS ( SELECT i.id FROM PostPrivacyIncludeFor i WHERE i.user_id = ? AND i.thought_id = t.id) OR t.privacy_level = 4 AND t.owner_id = ?) OR (t.privacy_level = 4 AND EXISTS (SELECT i2.id FROM PostPrivacyIncludeFor i2 WHERE i2.thought_id = t.id AND EXISTS (SELECT r.id FROM FriendFilterIds r WHERE r.list_id = i2.list_id AND r.friend_id = ?)) OR t.privacy_level = 4 AND t.owner_id = ?) OR (t.privacy_level = 1 AND EXISTS (SELECT G.id FROM Following G WHERE follower_id = t.owner_id AND following_id = ? AND friend = 1) OR t.privacy_level = 1 AND t.owner_id = ?) OR (NOT EXISTS (SELECT e.id FROM PostPrivacyExcludeFrom e WHERE e.thought_id = t.id AND e.user_id = ? AND NOT EXISTS (SELECT e2.id FROM PostPrivacyExcludeFrom e2 WHERE e2.thought_id = t.id AND EXISTS (SELECT l.id FROM FriendFilterIds l WHERE l.list_id = e2.list_id AND l.friend_id = ?))) AND t.privacy_level IN (0, 1, 4)) AND t.owner_id = ? ORDER BY t.created_at LIMIT 100 (mock up query, similar to the query I use now in Doctrine ORM. It's a mess, but you get what I am saying.) I guess my question is, how would you approach this situation to optimize it? Is there a better way to set up my database? I'm willing to completely scrap the method I have currently built up, but I wouldn't know what to move onto. Thanks guys. Updated: Fix the query to reflect the values I defined for privacy level above (I forgot to update it because I simplified the values)
Your query is too long to give a definitive solution for, but the approach I would follow is to simply the data lookups by converting the sub-queries into joins, and then build the logic into the where clause and column list of the select statement: select t.*, i.*, r.*, G.*, e.* from posts t left join PostPrivacyIncludeFor i on i.user_id = ? and i.thought_id = t.id left join FriendFilterIds r on r.list_id = i.list_id and r.friend_id = ? left join Following G on follower_id = t.owner_id and G.following_id = ? and G.friend=1 left join PostPrivacyExcludeFrom e on e.thought_id = t.id and e.user_id = ? (This might need expanding: I couldn't follow the logic of the final clause.) If you can get the simple select working fast AND including all the information needed, then all you need to do is build up the logic in the select list and where clause.
Had a quick stab at simplifying this without re-working your original design too much.
Using this solution your web page can now simply call the following stored procedure to get a list of filtered posts for a given user within a specified period.
call list_user_filtered_posts( <user_id>, <day_interval> );
The whole script can be found here : http://pastie.org/1212812
I haven't fully tested all of this and you may find this solution isn't performant enough for your needs but it may help you in fine tuning/modifying your existing design.
Tables
Dropped your post_privacy_exclude_from table and added a user_stalkers table which works pretty much like the inverse of user_friends. Kept the original post_privacy_includes_for table as per your design as this allows a user restrict a specific post to a subset of people.
drop table if exists users;
create table users
(
user_id int unsigned not null auto_increment primary key,
username varbinary(32) unique not null
)
engine=innodb;
drop table if exists user_friends;
create table user_friends
(
user_id int unsigned not null,
friend_user_id int unsigned not null,
primary key (user_id, friend_user_id)
)
engine=innodb;
drop table if exists user_stalkers;
create table user_stalkers
(
user_id int unsigned not null,
stalker_user_id int unsigned not null,
primary key (user_id, stalker_user_id)
)
engine=innodb;
drop table if exists posts;
create table posts
(
post_id int unsigned not null auto_increment primary key,
user_id int unsigned not null,
privacy_level tinyint unsigned not null default 0,
post_date datetime not null,
key user_idx(user_id),
key post_date_user_idx(post_date, user_id)
)
engine=innodb;
drop table if exists post_privacy_includes_for;
create table post_privacy_includes_for
(
post_id int unsigned not null,
user_id int unsigned not null,
primary key (post_id, user_id)
)
engine=innodb;
Stored Procedures
The stored procedure is relatively simple - it initially selects ALL posts within the specified period and then filters out posts as per your original requirements. I have not performance tested this sproc with large volumes but as the initial selection is relatively small it should be performant enough as well as simplifying your application/middle tier code.
drop procedure if exists list_user_filtered_posts;
delimiter #
create procedure list_user_filtered_posts
(
in p_user_id int unsigned,
in p_day_interval tinyint unsigned
)
proc_main:begin
drop temporary table if exists tmp_posts;
drop temporary table if exists tmp_priv_posts;
-- select ALL posts in the required date range (or whatever selection criteria you require)
create temporary table tmp_posts engine=memory
select
p.post_id, p.user_id, p.privacy_level, 0 as deleted
from
posts p
where
p.post_date between now() - interval p_day_interval day and now()
order by
p.user_id;
-- purge stalker posts (0,1,3,4)
update tmp_posts
inner join user_stalkers us on us.user_id = tmp_posts.user_id and us.stalker_user_id = p_user_id
set
tmp_posts.deleted = 1
where
tmp_posts.user_id != p_user_id;
-- purge other users private posts (3)
update tmp_posts set deleted = 1 where user_id != p_user_id and privacy_level = 3;
-- purge friend only posts (1) i.e where p_user_id is not a friend of the poster
/*
requires another temp table due to mysql temp table problem/bug
http://dev.mysql.com/doc/refman/5.0/en/temporary-table-problems.html
*/
-- the private posts (1) this user can see
create temporary table tmp_priv_posts engine=memory
select
tp.post_id
from
tmp_posts tp
inner join user_friends uf on uf.user_id = tp.user_id and uf.friend_user_id = p_user_id
where
tp.user_id != p_user_id and tp.privacy_level = 1;
-- remove private posts this user cant see
update tmp_posts
left outer join tmp_priv_posts tpp on tmp_posts.post_id = tpp.post_id
set
tmp_posts.deleted = 1
where
tpp.post_id is null and tmp_posts.privacy_level = 1;
-- purge filtered (4)
truncate table tmp_priv_posts; -- reuse tmp table
insert into tmp_priv_posts
select
tp.post_id
from
tmp_posts tp
inner join post_privacy_includes_for ppif on tp.post_id = ppif.post_id and ppif.user_id = p_user_id
where
tp.user_id != p_user_id and tp.privacy_level = 4;
-- remove private posts this user cant see
update tmp_posts
left outer join tmp_priv_posts tpp on tmp_posts.post_id = tpp.post_id
set
tmp_posts.deleted = 1
where
tpp.post_id is null and tmp_posts.privacy_level = 4;
drop temporary table if exists tmp_priv_posts;
-- output filtered posts (display ALL of these on web page)
select
p.*
from
posts p
inner join tmp_posts tp on p.post_id = tp.post_id
where
tp.deleted = 0
order by
p.post_id desc;
-- clean up
drop temporary table if exists tmp_posts;
end proc_main #
delimiter ;
Test Data
Some basic test data.
insert into users (username) values ('f00'),('bar'),('alpha'),('beta'),('gamma'),('omega');
insert into user_friends values
(1,2),(1,3),(1,5),
(2,1),(2,3),(2,4),
(3,1),(3,2),
(4,5),
(5,1),(5,4);
insert into user_stalkers values (4,1);
insert into posts (user_id, privacy_level, post_date) values
-- public (0)
(1,0,now() - interval 8 day),
(1,0,now() - interval 8 day),
(2,0,now() - interval 7 day),
(2,0,now() - interval 7 day),
(3,0,now() - interval 6 day),
(4,0,now() - interval 6 day),
(5,0,now() - interval 5 day),
-- friends only (1)
(1,1,now() - interval 5 day),
(2,1,now() - interval 4 day),
(4,1,now() - interval 4 day),
(5,1,now() - interval 3 day),
-- private (3)
(1,3,now() - interval 3 day),
(2,3,now() - interval 2 day),
(4,3,now() - interval 2 day),
-- filtered (4)
(1,4,now() - interval 1 day),
(4,4,now() - interval 1 day),
(5,4,now());
insert into post_privacy_includes_for values (15,4), (16,1), (17,6);
Testing
As I mentioned before I've not fully tested this but on the surface it seems to be working.
select * from posts;
call list_user_filtered_posts(1,14);
call list_user_filtered_posts(6,14);
call list_user_filtered_posts(1,7);
call list_user_filtered_posts(6,7);
Hope you find some of this of use.
Select *, max(date) works in phpMyAdmin but not in my code
OK, my statement executes well in phpMyAdmin, but not how I expect it in my php page. This is my statement: SELECT `egid`, `group_name` , `limit`, MAX( `date` ) FROM employee_groups GROUP BY `egid` ORDER BY `egid` DESC ; This is may table: CREATE TABLE `employee_groups` ( `egid` int(10) unsigned NOT NULL, `date` date NOT NULL, `group_name` varchar(50) NOT NULL, `limit` smallint(5) unsigned NOT NULL, PRIMARY KEY (`egid`,`date`) ) ENGINE=MyISAM DEFAULT CHARSET=cp1251; I want to extract the most recent list of groups, e.g. if a group has been changed I want to have only the last change. And I need it as a list (all groups).
Your query might be broken. You should not select fields that aren't in the group by unless one of the following two conditions apply: You use an aggregate function. The value is functionally dependant on the grouped by columns. The two fields group_name and limit appear to break these rules. This means that you will get indeterminate results for these columns. If you are trying to select the max per group then you should use a slightly different technique. See Quassnoi's article MYSQL: Selecting records holding a groupwise maximum for a variety of methods you could use. Here's one way to do it: SELECT di.* FROM ( SELECT egid, MAX(date) AS date FROM employee_groups d GROUP BY egid ) dd JOIN employee_groups di ON di.egid = dd.egid AND di.date = dd.date
aggregate functions will work in mysql, different to the sql standard. to access the value of max(date) from php, you have to alias it:
SELECT `egid`, `group_name` , `limit`, MAX( `date` ) as maxdate
FROM …
you can then select it like any other colum from php with
while($row = mysqli_fetch_row($result)) {
echo htmlspecialchars($row['maxdate']);
}
hope that helps