Suppose a table videos
id | name | views
----------+--------------+-----------
1 | Video1 | 52
2 | Video2 | 150
...
For getting the video which is popular/most viewed this week, I could create another table: videoviews
id | foreign_key | viewed_on
----------+--------------+-----------
1 | 1 | 10/12/2018
2 | 1 | 09/12/2018
...
From this table, I can easily get the data for last week/last month etc. That's not an issue.
Problem:
Suppose I have 1000 Videos and Each video gets 100 Views per day.
My videoviews table will have 100000 records each day.
I know this is not the best way to achieve this functionality. Just wondering what is?
I found these on SO but..
How to get most visited posts of the week?
Popular Today, This Week, This Month - Design Pattern
Problem: Suppose I have 1000 Videos and Each video gets 100 Views per day. My videoviews table will have 100000 records each day.
Do you need a complete record of each individual view?
You could, instead, use a counter approach, where you store one row per video per day, and simply increment its value when a new row comes in. This is granular enough to provide useful per-day analytics, without having to store a million rows for a million video views.
Add extra columns called views and start_date on your videoviews table.
On hitting the page with the video, fetch the views, increment, and update where the week starts with start_date.
Only one row is required per week. You can also remove old weeks if you like.
so i have a database that have a set of events that are suppose to happen and end at certain time. So say my table is (the event here is an actual event, not a code thing)
Event | TimeStart | TimeEnd | Day
A | 0800 | 1400 | 2
B | 2000 | 2300 | 3
C | 1200 | 1900 | 4
What i want is that i want the event to occur IF its between the TimeEnd and TimeEnd + 3 hours. The problem that i encounter is what if its 2300? I used the days with date(N) that correspond to monday (1) - Sunday (7)
So if its Event B, i need to have a code that take the current time, reduce it by 3 hours (so that i doesn't go to day 4) then i get the TimeEnd and add on 3 hours to it.
The problem now that i tested is that How do i make Event B to be on Day 3? even if i used strtotime() Event B still shows day 4 after it passed 2400 hour.
To make the matter clearer, i am creating a voting poll that only starts after the event ended and it only last for 3 hours.
Edit found the answer, didn't know using "Monday this week" made sense for date() :D
Your representation of times in the table is kinda weird. If each event will happen only one time (doesn't repeat for instance every monday), I suggest you use unix timestamp to represent the times. Then you will have no such problems.
I am developing a personal finance tracker (for fun!) and I have a table of categories. Each category is an entry in the table and at the end of the month they are all duplicated with their relevant balances reset to the start of the month reading for the new month.
Among others, these categories can be of type 'savings' and so have a running total. If I want to retrieve a category or update it then I used the category_id field and this works fine for the current working month but linking months together is breaking my brain. For the savings categories I want to show how the running_total has increased over the previous six months but in my current DB design, categories don't "know" about their previous months as they are created new at the start of each month.
The only way I could currently retrieve the last 6 months of a savings running_total is to search by the category name but this is potentially unreliable.
I have considered adding a field to the table which is "previous_month_category_id" which would work as a way to link the categories together but would be expensive to implement as it would require 6 MSQL operations each time grabbing the "previous_month_category_id" from the result and then re running the query.
If MYSQL can do some kind of recursion then maybe this could work but I feel like there is a more obvious answer staring me in the face.
I'm using Codeigniter and MYSQL but not scared of vanilla PHP if required.
Help on how to do this would be great.
UPDATE 1:
Below is a sample from what the savings category might look like mixed in amongst other categories. At the end of each month the entry is duplicated with the same category_name, type, buget, year, and users_id but the category_id auto increments, the month updates to the new month number and the running total is the previous running_total + the budget. How would I do one database query to retrieve these without using the category_name? As this could change is the user decided to caller it "Bigger TV" at the end of July
+-------------+--------------+------+--------+---------------+------+-------+----------+
| category_id |category_name | type | budget | running_total | year | month | users_id |
+-------------+--------------+------+--------+---------------+------+-------+----------+
| 44 | Big TV | sav | 20 | 240 | 2012 | 8 | 77 |
+-------------+--------------+------+--------+---------------+------+-------+----------+
| 32 | Big TV | sav | 20 | 220 | 2012 | 7 | 77 |
+-------------+--------------+------+--------+---------------+------+-------+----------+
| 24 | Big TV | sav | 20 | 200 | 2012 | 6 | 77 |
UPDATE 2:
I'm not sure I'm explaining myself very well So I'll put some more detail around how the app works and see if that helps.
I have tables called "categories", "transactions" and "users". A category can be one of three types, 1: Cash, 2: Regular Payment, 3: Savings. Think of cash and regular payment types as buckets, at the start of each month each bucket is full and the aim is to take money out of it and make sure there is still a bit left at the end of the month (or at least not negative).
This is fine on a month by month basis and works very well (for me, I have used this system for 2 years now I think). The trip up comes with Savings as they are linked month by month and are more like a big bucket that is added to each month (with a set increment called budget) until it overspills and is then drained (like Big TV would be when you buy it), or taken from a little bit here and there and the aim is to build up an emergency fund (like "When my car breaks down" type thing).
When the relevant information is displayed for each category only the current month is shown for cash and regular as that is all that is important, for the savings however the current amount is also shown but it would be nice to show a small history graph of how it had built up (or depleted) over time. To do this I need some way of searching for the previous end of month states of these categories so that the graph can be plotted but currently I can't work out how to link them all by anything other than the category_name.
I have tried to implement a bit of DB normalisation but this is the first schema I've implemented having known about normalisation so I've probably missed some aspects of it and possibly avoided any over normalisation where it didn't feel right.
Below are my tables:
categories
+-------------+--------------+------+--------+---------------+------+-------+----------+
| category_id |category_name | type | budget | running_total | year | month | users_id |
+-------------+--------------+------+--------+---------------+------+-------+----------+
transactions
+----------------+--------------+--------+------+----------+------------------------+
| transaction_id | description | amount | date | users_id | categories_category_id |
+----------------+--------------+--------+------+----------+------+-------+---------+
they are joined on categories_category_id which is a foreign key
I have always worked off the premise that each category needs an new entry for each month but it seems from the comments and answers below that I would be better off having just one category entry regardless of month and then just calculating everything on the fly?
Although, the budgets can be changed by the user and so for record keeping I'm not sure if this would work also the "deposits" never really happen it is just the category being duplicated at the end of the month so I guess that would need to dealt with.....
The aim of this app has always been to decouple financial tracking from the physical transaction that occur in a bank account and provide a layer over someones finances thus allowing the user to avoid hard to explain transactions etc and just focus on over all cash position. There is no concept of an "income" in this system, or a bank account.
It seems to me like your database design could use some work. I'm still not completely familiar with what you're really trying to do, but my initial thoughts would be to store each transaction as a single row in a table, and then query that table in different ways to generate different types of reports on it. Something like this:
transactions:
+----+---------+--------+---------------+-----------+-------------+
| id | user_id | amount | running_total | datestamp | category_id |
+----+---------+--------+---------------+-----------+-------------+
categories:
+----+------+------+
| id | name | type |
+----+------+------+
Don't increment the categories based on time. Add an entry to the categories table when you actually have a new category. If a transaction could possibly belong to multiple categories, then use a third (relational) table that relates transactions (based on transaction ID) to categories (based on category ID).
When you have a deposit, the amount field will be positive and for withdrawals, it will be negative. You can get your current running total by doing something like:
SELECT running_total FROM transactions
WHERE id = (SELECT MAX(id) FROM transactions WHERE user_id = '$userID');
You can find your total difference for a particular month by doing this:
SELECT SUM(amount) FROM transactions WHERE DATE('%c', datestamp) = '$monthNumber';
You can find the total spending for a particular category by doing this:
SELECT SUM(t.amount) FROM transactions t
INNER JOIN categories c ON t.category_id = c.id WHERE c.name = 'Big TV';
There are plenty of other possibilities, but the purpose here is just to demonstrate a possibly better way to store your data.
I'm going to make a Mysql based calendar system where you can have repeating pattern for lets say every monday forever and ever. It must also cover static/once-only events. What I'm wondering about, is which solution would be most logical (and best) for me to use. I have four methods which I'm wondering to chose between.
Method #1
Make a function which accepts parameters from and to. This function would create a temporary table table which imports existing static schedule through INSERT ... SELECT. Afterward it would read of the pattern table and populate the temporary table through the peroid based on from and to.
This solution seems nice from the point of view that queries will be simplier to fetch data with and it works into infinity since you can just repopulate the table depending of which month you're loading. What I'm curious about is whenever this might be a laggy way to do it or not.
Method #2
Create and join given patterns through a subquery and JOIN with static calendar.
This seems to be rather annoying since the queries would be a lot more bigger and would probably not be good at all(?).
Method #3
Basicly just INSERT pattern for lets say one year ahead. Then I guess a cron job would repopulate to make it one year ahead always.
This is a simple way to do it, but it feels like a lot of unneeded data stored and it doesn't really give the infinity which I'm after.
Method #4 (Suggested by Veger)
If I understand correctly, this method would fetch the pattern from another query and creates events upon execution. It's similar to my thoughts regarding Method #1 in that way that I consider simple pattern to create several rows.
However if this would be implemented outside Mysql, I would loose some database functionality which I'm after.
I hope you guys understood my situation, and if you could suggest either given and argue why it's the best or give another solution.
Personally I like the Method #1 the most, but I'm curious if it's laggy to repopulate the calendar table each and every call.
I have built this kind of calendar before. I found the best way to do it is to approach it the way that crons are scheduled. So in the database, make a field for minute, hour, day of month, month, and day of week.
For an event every Friday in June and August at 10:00pm your entry would look like
Minute Hour DayOfMonth Month DayOfWeek
0 22 * 6,8 5
You could then have a field that flags it as a one time event which will ignore this information and just use the start date and duration. For events that repeat that end eventually (say every weekend for 3 months) you just need to add an end date field.
This will allow you to select it back easily and reduce the amount of data that needs to be stored. It simplifies your queries as well.
I don't think there is a need to create temporary tables. To select back the relevant events you would select them by the calendar view. If your calendar view is by the month, your select would look something like:
SELECT Events.*
FROM Events
WHERE (Month LIKE '%,'.$current_month.',%' OR Month = '*')
AND DATE(StartDate) >= "'.date('Y-m-d', $firstDayOfCurrentMonth).'"
AND DATE(EndDate) <= "'.date('Y-m-d', $lastDayOfCurrentMonth).'"
Obviously this should be in a prepared statement. It also assumes that you have a comma before and after the first and last value in the comma separated list of months (ie. ,2,4,6,). You could also create a Month table and a join table between the two if you would like. The rest can be parsed out by php when rendering your calendar.
If you show a weekly view of your calendar you could select in this way:
SELECT Events.*
FROM Events
WHERE (DayOfMonth IN ('.implode(',', $days_this_week).','*')
AND (Month LIKE '%,'.$current_month.',%' OR Month = '*'))
AND DATE(StartDate) >= "'.date('Y-m-d', $firstDayOfCurrentMonth).'"
AND DATE(EndDate) <= "'.date('Y-m-d', $lastDayOfCurrentMonth).'"
I haven't tested those queries so there maybe some messed up brackets or something. But that would be the general idea.
So you could either run a select for each day that you are displaying or you could select back everything for the view (month, week, etc) and loop over the events for each day.
I like Veger's solution best .. instead of populating multiple rows you can just populate the pattern. I suggest the crontab format .. it works so well anyway.
You can query all patterns for a given customer when they load the calendar and fill in events based on the pattern. Unless you have like thousands of patterns for a single user this should not be all that slow. It should also be faster than storing a large number of row events for long periods. You will have to select all patterns at once and do some preprocessing but once again, how many patterns do you expect per user? Even 1000 or so should be pretty fast.
I've had this idea since I was still programming in GW Basic ;-) though, back then, I took option #3 and that was it. Looking back at it, and also some of the other responses, this would be my current solution.
table structure
start (datetime)
stop (datetime, nullable)
interval_unit ([hour, day, week, month, year?])
interval_every (1 = every <unit>, 2 every two <units>, etc.)
type ([positive (default), negative]) - will explain later
Optional fields:
title
duration
The type field determines how the event is treated:
positive; normal treatment, it shows up in the calendar
negative; this event cancels out another (e.g. every Monday but not on the 14th)
helper query
This query will narrow down the events to show:
SELECT * FROM `events`
WHERE `start` >= :start AND (`stop` IS NULL OR `stop` < :stop)
Assuming you query a range by dates alone (no time component), the the value of :stop should be one day ahead of your range.
Now for the various events you wish to handle.
single event
start = '2012-06-15 09:00:00'
stop = '2012-06-15 09:00:00'
type = 'positive'
Event occurs once on 2012-06-15 at 9am
bounded repeating event
start = '2012-06-15 05:00:00'
interval_unit = 'day'
interval_every = 1
stop = '2012-06-22 05:00:00'
type = 'positive'
Events occur every day at 5am, starting on 2012-06-15; last event is on the 22nd
unbounded repeating event
start = '2012-06-15 13:00:00'
interval_unit = 'week'
interval_every = 2
stop = null
type = 'positive'
Events occur every two weeks at 1pm, starting on 2012-06-15
repeating event with exceptions
start = '2012-06-15 16:00:00'
interval_unit = 'week'
interval_every = 1
type = 'positive'
stop = null
start = '2012-06-22 16:00:00'
type = 'negative'
stop = '2012-06-22 16:00:00'
Events occur every week at 4pm, starting on 2012-06-22; but not on the 22nd
I would suggest something around the lines of this:
Split your Events table into 2 because there are clearly 2 different types recurring events and static events and depending on the type they will have different attributes.
Then for a given Event look-up you would run 2 queries, one against each Event Type. For the static events table you would defiantly need (at least) one datetime field so the lookup for a given month would simply use that feild in the conditions (where event_date > FirstDayOfTheMonth and event_date < LastDayOfTheMonth ). Same logic for a weekly/yearly view.
This result set would be combined with a second result set from the recurring events table. Possible attributes could be similar to crontab entries, using day of week/day of month as the 2 main variables. If you're looking at a monthly view,
select * from recurring_events where DayOfWeek in (1,2,3,4,5,6,7) or (DayOfMonth > 0 and DayOfMonth < #NumberOfDaysInThisMonth )
Again similar if for a weekly/yearly view. To make this even simpler to interface, use stored procedures with all the logic for determining 'which days of the week are found between date A and date B'.
Once you have both result sets, you could aggregate them together in the client then display them together. The adavantage to this is there will be no need for "mock/empty records" nor async cronjobs which pre-fill, the queries could easily happen on the fly and if performance actually degrades, add a caching layer, especially for a system of this nature a cache makes perfect sense.
I'm actually looking for something similar to this and my solution so far (on paper I didn't start to structure or code yet) stores in 2 tables:
the "events" would get the date of the first occurrence, the title and description (plus the auto-increment ID).
the "events_recursion" table would cross reference to the previous table (with an event_id field for instance) and could work in 2 possible ways:
2.A: store all the occurrences by date (i.e. one entry for every occurrence so 4 if yo want to save "every friday of this month" or 12 for "the 1st of every month in 2012")
2.B: or saving the interval (I would save it in seconds) from the date of the first event in a field + the date of the last occurrence (or end of recursion) in another field such as
ID: 2
EVENT_ID: 1
INTERVAL: 604800 (a week if I'm not mistaken)
END: 1356912000 (should be the end of this year)
Then when you open the php that shows the schedule it would check for the event still active in that month with a joint between the two tables.
The reason why I would use 2 tables cross-referenced instead of saving all in one tables just comes from the facts that my projects sees very crazy events such as "every fridays AND the 3rd monday of every month" (that in this case would be 1 entry in the events tables and 2 with same "event_id" field in the second table. BTW my projects is for music teachers that here got small work on strict schedules decided 3 or 6 months at a time and are a real mess).
But as I have said i haven't started yet so I'm looking forward to seeing your solution.
PS: please forgive (and forget) my english, first isn't my language and second it is pretty late night and I'm sleepy
Maybe check out some great ideas from MySQL Events
and some more:
http://phpmaster.com/working-with-mysql-events/?utm_source=rss&utm_medium=rss&utm_campaign=phpmaster-working-with-mysql-events
http://dev.mysql.com/doc/refman/5.5/en/create-event.html
the best solution depends on whether you want to favor standard compliance (RFC5545) or working exclusively within MySQL.
depend on on flexible your recurrence rule engine needs to be. If you want simple rules (every 1st of month or every January, ...) then the solutions offered above have been detailed at length.
However should you want your application to offer compatibility with existing standards (RFC5545) which involves much more complex rules you should have a look at this SO post when building a calendar app, should i store dates or recurrence rules in my database?
I would do it as I explained here. It will create an infinite calander:
PHP/MySQL: Model repeating events in a database but query for date ranges
The downside is that there will be some calculation during the query. If you need a high performance website, preloading the data will be the way to go. You dont even have to preload all the events in the calendar, to make it possible for easy changing the values in a single event. But it would be wise to store all dates from now till ....
Now using cached values does make it less infinite, but it will increase speed.
Copy of awnser for easy access:
I would create a tally table with just one col called id and fill that table with numbers from 0 to 500. Now we easily use that to make selections instead of using a while loop.
Id
-------------------------------------
0
1
2
etc...
Then i'd store the events in a table with Name as varchar, startdate as datetime and repeats as int
Name | StartDate | Repeats
-------------------------------------
Meeting | 2012-12-10 00:00:00 | 7
Lunch | 2012-12-10 00:00:00 | 1
Now we can use the tally table to select all dates between two dates by using:
SELECT DATE_ADD('2012-12-09 00:00:00',INTERVAL Id DAY) as showdate
FROM `tally`
WHERE (DATE_ADD('2012-12-09 00:00:00',INTERVAL Id DAY)<='2012-12-20 00:00:00')
ORDER BY Id ASC
ShowDate
-------------------------------------
2012-12-09 00:00:00
2012-12-10 00:00:00
2012-12-11 00:00:00
2012-12-12 00:00:00
2012-12-13 00:00:00
2012-12-14 00:00:00
2012-12-15 00:00:00
2012-12-16 00:00:00
2012-12-17 00:00:00
2012-12-18 00:00:00
2012-12-19 00:00:00
2012-12-20 00:00:00
Then we join this on the events table to calculate the difference between the startdate and the showdate. We devided the results of this by the repeats column and if the remainder is 0, we have match.
All combined becomes:
SELECT E.Id, E.Name, E.StartDate, E.Repeats, A.ShowDate, DATEDIFF(E.StartDate, A.ShowDate) AS diff
FROM events AS E, (
SELECT DATE_ADD('2012-12-09 00:00:00',INTERVAL Id DAY) as showdate
FROM `tally`
WHERE (DATE_ADD('2012-12-09 00:00:00',INTERVAL Id DAY)<='2012-12-20 00:00:00')
ORDER BY Id ASC
) a
WHERE MOD(DATEDIFF(E.StartDate, A.ShowDate), E.Repeats)=0
AND A.ShowDate>=E.StartDate
Which results in
Id | Name |StartDate | Repeats | ShowDate | diff
---------------------------------------------------------------------------------
1 | Meeting | 2012-12-10 00:00:00 | 7 | 2012-12-10 00:00:00 | 0
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-10 00:00:00 | 0
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-11 00:00:00 | -1
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-12 00:00:00 | -2
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-13 00:00:00 | -3
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-14 00:00:00 | -4
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-15 00:00:00 | -5
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-16 00:00:00 | -6
1 | Meeting | 2012-12-10 00:00:00 | 7 | 2012-12-17 00:00:00 | -7
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-17 00:00:00 | -7
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-18 00:00:00 | -8
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-19 00:00:00 | -9
2 | Lunch | 2012-12-10 00:00:00 | 1 | 2012-12-20 00:00:00 | -10
Now you could (and should!) speed things up. For instance by directly storing dates in a table so you can just select all dates directly instead of using a tally table with dateadd. Every thing you can cache and dont have to calculate again is good.
I am trying to write ONE SQL query, which gives always gives three rows of results. Database is as follows:
uid | program_date | program_time | program_name
------------------------------------------------
1 | 2012-04-16 | 21:00 | Some movie
2 | 2012-04-16 | 23:00 | Program end
3 | 2012-04-17 | 10:00 | Animation
4 | 2012-04-17 | 11:00 | Some other movie
5 | 2012-04-17 | 12:00 | Some show
All I need - always have three rows - what is on air now, next and upcomming. So if today is 2012-04-16 21:00 it should output Some movie, Program end, Animation.
At 2012-04-17 00:00 it should output Program end, Animation, Some other movie.
Problem is that I need to "navigate" back in one day if there is no records WHERE program_date = date("Y-m-d") AND program_time <= date("H:i:s");
There is another problem - database does not have Unix timestamp field, only Uid, program_date (date field) and program_time (time field) and program_name.
Also, there might be, that Uid's are not inserted into table in sequence, as some program entry might be inserted in between into existing program schedule.
I am trying various approaches, but want to do everything in one SQL query, without looping in PHP.
Can anyone help me here?
As TV-people count and show time in rather strange manner, MySQL function may be created to handle their non-human ;-) logic easier:
CREATE FUNCTION TV_DATE(d CHAR(10), t CHAR(5))
RETURNS CHAR(16) DETERMINISTIC
RETURN CONCAT(d, IF (t < "06:00", "N", " "), t);
User-defined functions are declared per-database and this may be done just once. DETERMINISTIC tells that function always return the same result for the same input and internal MySQL optimizer may rely on that. N is just a letter which is larger (in string comparison) than whitespace. Consider it as mnemonics for next or night.
note: Hours should be always formatted with 2 digits!
Then using this function we may select what we need even simpler:
-- what is on air now
(SELECT `program_name`, TV_DATE(`program_date`, `program_time`) AS `tv_time`
FROM `table`
WHERE (`tv_time` <= TV_DATE(date("Y-m-d"), date("H:i"))
ORDER BY `tv_time` DESC
LIMIT 1)
UNION
-- next and upcomming
(SELECT `program_name`, TV_DATE(`program_date`, `program_time`) AS `tv_time`
FROM `table`
WHERE (`tv_time` > TV_DATE(date("Y-m-d"), date("H:i"))
ORDER BY `tv_time` ASC
LIMIT 0, 2)
Keep in mind, that if all records in DB are in future you'll get only 2 of them.
The same for situation, when the next program is the last one in DB.
You may add different constant values into queries in order to distinguish those 2 situations.