We have a database of news posts on our site that has some 3000 rows (one for each post, dating back to 2001). Each has a unix timestamp indicating it's post date.
I want to have an archive page that shows a summary of how many posts were on that given month. So grouped by year, then month
2011
January (13 posts)
February (25 posts)
2012
March (30 posts)
etc.
What's the most efficient way of doing this? I had a mess of code before that had two for loops (year, then month) which would query the database for each month and count how many posts were made then. It was, not surprisingly, incredibly slow.
Any tips on how to group the data and avoid hitting the database for every single month.
This is best handled with a aggregated table in the database for keeping statistics on a monthly basis. You can have a simple table with 3 columns.
Year | Month | Count
on a daily basis you run a simple aggregated query counting the number of posts for that month (using a filter) and update the table above. It will be fast and clean.
Related
Im trying to figure out the most efficient way of calculating statistics using data from MySQL database with dates.
Currently, I use the following syntax
Example:
SELECT sum(Precipitation) from DataTable GROUP BY YEAR(Datetime)
This works perfectly fine, I get the total rainfall for each year. However, now I would like to implement the option to set the beginning of the rain season. In some places, the rain season might begin for example in September. In such case I would need the same calculation, i.e. also grouped by "years", but always since Sep to Aug.
I was thinking about how to do this and the only way I can think of would be somehow calculating the monthly sums and the using PHP try to add them up. But the problem is that that would probably be much slower given there is lots of data and the original script uses just this one line above.
Is there any more efficient way of then getting something like
2014 - xyz inches, 2015 - xyz inches, but where the 2014 would correspond for example to season 2014/2015 etc.
The data in the table is like this: column 1 is always the Datetime and then the actual value, data in 5 minute intervals. I need to maintain the table structure, so I cannot create a different table where the values would be organized differently.
Use this query:
SELECT SUM(Precipitation)
FROM DataTable
GROUP BY YEAR(DATE_SUB(Datetime, INTERVAL 8 MONTH))
This query shifts every date backwards by 8 months, with the result that September 1, 2016 would appear to be the first day of 2016, and August, 2016, would appear to be the last month of 2015.
I have a query that counts the "Xp" difference per day from my database, this all works as it should however it groups from midnight-midnight, what I would like to do is group 3am to 3am.
However another issue I think I may have is that my query may not always have the rows being the exact second at 3am due to the fact that it has to run a huge query and retrieve data from another website per user profile, so it should get all data after 3am, but before maybe 4am or something, so it has enough time to get all of the rows.
my current mysql is:
SELECT FROM_UNIXTIME(date, '%Y%m%d') AS YYYYMMDD, MAX(xp)-MIN(xp) AS xp_gain
FROM skills
WHERE userID = '$checkID'
AND skill = '$skill'
AND date >= '$date'
GROUP BY YYYYMMDD
ORDER BY date ASC
The best way to handle this is to add (if you can) another column that is just a DATE (not a DATETIME) and have this field rollover from one day to the next at 3am, (you can to this by subtracting 3 hours from the current time when doing the INSERT).
This gives you a couple of benefits, especially with a large number of rows:
It is much faster to query or group by a DATE than a range of
DATETIME
It will always query the rows at the exact second of 3am,
regardless of how long the query takes.
I'm looking for an efficient way to collate all blog posts into a menu of the following format:
2012
August(6)
September(4)
October(2)
Month representing the month(obviously), and the value inside the brackets representing the number of posts in that month. Once clicked, a search will then be made for all posts in that month, in that year.
I need it to be dynamic, picking up November automatically when a post is created in that month, and carrying on into December, into 2013 etc etc...
All I have is a UNIX timestamp for each post. I would really like to avoid using seperate functions to gather endless comlex arrays etc.
Any help much appreciated.
From your question, I understand you're trying to come up with a query to group a number of elements by month and year. The following should do the trick:
SELECT
YEAR(dateField) AS YEAR,
MONTH(dateField) AS MONTH,
COUNT(*) AS TOTAL
FROM table
GROUP BY YEAR, MONTH
Obviously, "dateField" being the name of your datetime/timestamp column and "table" being the name of your table.
More information on the GROUP BY clause and aggregate functions (such as the COUNT(*) function used above) here.
I have a problem that I just cannot seem to get my head around, and hope someone can help give me some advice.
Ever since getting solar PV cells fitted on my house roof, I have been generating electricity and in accordance to some (rather generous) incentives to do this kind of thing, have been making money for every kWh of electricity I generate. Seeing this as being a bit of a database project, I set about writing some PHP/MySQL to track daily generation, and now have nearly a year's worth of daily kWh readings, which are nicely presented to me in graphical form, both in a month-by-month view, and as a yearly (grouped into months) graph.
I'm now wanting to expand the system to show revenue in monetary terms, rather than kWh of electricity. Currently, the figure is £0.454 per kWh, though this figure changes every year on the April 1st (it was £0.433 previously).
This is my current MySQL structure:
Table feedin:
year (year4) rate (float)
2010 0.433
2011 0.433
2012 0.454
Table generation:
day (DATE) reading (float)
2011-12-01 7.682
2011-12-02 5.747
2011-12-03 4.982
... ...
2012-08-13 8.022
2012-08-14 19.449
2012-08-15 5.484
My first attempt at this was all rather cumbersome with a very mixed mess of PHP and MySQL queries, with the bulk of the logic being done in PHP (my MySQL skills are "limited", at best). However, as time is going on, I see that it would be ideal if the whole thing were done in MySQL.
I've no real idea how to tackle this. My initial thoughts are that we need to select yearly chunks of data (well, date-ranges from April 1st in one year, to March 31st the next), and multiply it by the appropriate year rate. And that "appropriate year rate is the rate applicable at the start of that date range, ie, as of April 1st).
Ideally, I'd like the query to be able to cope with multiple yearly boundaries, so, for example, several years down the road, I'd like to be able to query the absolute total revenue produced to date. Ultimately, I would just like to pass the query the start and end dates, and it returns the correct figure.
Link the year of the generation date to the year of the feedin tariff
SELECT *, generation.reading*feedin.rate AS profit
FROM generation, feedin
WHERE YEAR(generation.day)=feedin.year
BUT as this must relate to year start of APRIL 1st
SELECT *, generation.reading*feedin.rate AS profit
FROM generation, feedin
WHERE YEAR(DATE_SUB(generation.day, INTERVAL 3 MONTH))=feedin.year
This will move the recorded dates back 3 months too, making them Jan-Dec instead of Apr-Mar wich will then match the feedin year
something along these lines:
select year, sum(reading) as total_generation, (total_generation*feedin.rate)
FROM feedin
LEFT JOIN generation on feedin.year = YEAR(generation.day)
GROUP BY year
Hope this does what you want (tested and working)
SELECT (a.rate*b.reading),a.year as amount from generation as b, feedin as a where Year(b.day)=a.year
I'm creating a calendar that displays a timetable of events for a month. Each day has several parameters that determine if more events can be scheduled for this day (how many staff are available, how many times are available etc).
My database is set up using three tables:
Regular Schedule - this is used to create an array for each day of the week that outlines how many staff are available, what hours they are available etc
Schedule Variations - If there are variations for a date, this overrides the information from the regular schedule array.
Events - Existing events, referenced by the date.
At this stage, the code loops through the days in the month and checks two to three things for each day.
Are there any variations in the schedule (public holiday, shorter hours etc)?
What hours/number of staff are available for this day?
(If staff are available) How many events have already been scheduled for this day?
Step 1 and step 3 require a database query - assuming 30 days a month, that's 60 queries per page view.
I'm worried about how this could scale, for a few users I don't imagine that it would be much of a problem, but if 20 people try and load the page at the same time, then it jumps to 1200 queries...
Any ideas or suggestions on how to do this more efficiently would be greatly appreciated!
Thanks!
I can't think of a good reason you'd need to limit each query to one day. Surely you can just select all the values between a pair of dates.
Similarly, you could use a join to get the number of events scheduled events for a given day.
Then do the loop (for each day) on the array returned by the database query.
Create a table:
t_month (day INT)
INSERT
INTO t_month
VALUES
(1),
(2),
...
(31)
Then query:
SELECT *
FROM t_month, t_schedule
WHERE schedule_date = '2009-03-01' + INTERVAL t_month.day DAY
AND schedule_date < '2009-03-01' + INTERVAL 1 MONTH
AND ...
Instead of 30 queries you get just one with a JOIN.
Other RDBMS's allow you to generate rowsets on the fly, but MySQL doesn't.
You, though, can replace t_month with ugly
SELECT 1 AS month_day
UNION ALL
SELECT 2
UNION ALL
...
SELECT 31
I faced the same sort of issue with http://rosterus.com and we just load most of the data into arrays at the top of the page, and then query the array for the relevant data. Pages loaded 10x faster after that.
So run one or two wide queries that gather all the data you need, choose appropriate keys and store each result into an array. Then access the array instead of the database. PHP is very flexible with array indexing, you can using all sorts of things as keys... or several indexes.