Hi I have a MySQL table of Facebook pages (fbpagesfancount) that has the total fan count by day since 01 Jan 2016.
The structure is like this:-
Pageid, Pagename, Updated_Date, Fan_Count
There are a number of specific days that are missing and do not therefore have fancount values due to Facebook API issues.
The days that are missing are usually single days, for example, there is a value for the day before and the day after.
I'd like to create a new table that has a record for every day since 01/01/2016 for each page (750 pages) and then update the days that are missing by averaging the day before and the day after the missing date.
Is this possible using MySQL only or should I write a script in PHP that performs this task and if so, any suggestions on the logic would be helpful.
Any other suggestions on how to tackle this issue would be welcome.
Thanks
Jonathan
Yes, it is possible in SQL only.
No, you should not attempt it as it is more complicated and for a single shot there's no need.
Yes, write a script in any language you know, for instance PHP.
I'm not sure why you even want to create a new table? You could add a flag to your current table saying its an origional count vs an average, and just find the missing numbers and add them in a script.
Related
I'm really struggling for days with (what i think) is a pretty advanced operation that i plan to schedule to run in my database every week.
this is the structure of my table (unit_uptime_daily):
What i need to do is run a script every week that, for every unit_id that exists in that table, gets all the rows of that unit_id thats timestamp is that present day < 6 days (so all the unit_ids with a timestamp of the previous week) add up the total_uptime column of the 7 rows and insert the new row into a weekly table.
Effectively, i am grabbing the 7 latest rows for each unit_id, adding up the total_uptime and then inserting unit_id, result of added total_uptime and timestamp into a new table.
Thanks in advance, if this is even possible to do!
Use cron jobs. Most of hosting providers provides this facility. Google cron jobs to find more about it and here's an answer that could help you.
Run a PHP file in a cron job using CPanel
I have solved this by using PHP, I got the list of unique ids, and did all the maths in a loop, inserting each result into the new table. I had already done this but would have liked it to be possible in SQL.
I am making an auction website, where users can add products, and set the end date.
My table enddate is like: 2015-2-25 14:01:23
and to determine if the item is ended I have a row called ended which can be 0 or 1. While I am writing this, I am thinking maybe I should check if time(system) is after or before enddate row to determine if the item is ended...
Anyway How would I go about changing the row ended to 1 on the exact time and date (even if someone's not viewing the page to run an ajax script) Would I have to do chron, or would it be better if I done what I mentioned above regarding checking the times?
Which one would you recommend? If using row ended how would I accomplish updating this on the enddate time.
Well, that's not the way to use a database. Your own suggestion is the way to go: Check if the current time is after or before enddate row to determine if the item has ended...
I am developing a website which will have 200.000 pages. There is also a browse section, which shows most popular, highest rated etc. documents. However this section will become almost static couple of weeks later, after launch. So I also would like to implement a filtering system which will show today's, this week's, this month's most popular items, just like youtube.
Just like this:
http://www.youtube.com/videos?c=2
How should I implement this function? Do I need another table, which will have a new entry for every document each day?
docid, date, view_count, rating
So I will get today's row for filtering by using a day, or calculate a week (7 rows) for filtering by using week? It seems not efficient. Do you have any suggestions?
I am using LAMP stack by the way.
Thanks,
Assuming you timestamp the records in your table, you should be able to put a where clause that limits the timestamp to whatever timeframe you want.
You can cache the result, especially the longer ones, for long enough to make the request inconsequential.
EDIT
But perhaps you mean most popular today, not most popular that was added today?
In which case, I don't have an answer.
The most direct approach is to save the timestamp and the resource id each time the resource is shown in recent_views(what, when). Daily/weekly/monthly charts can be created with appropriate WHERE clauses like WHERE when > $beginOfPeriod AND when < $endOfPeriod.
For performance reasons you can aggregate the values each night, save the sums in separate tables like daily_views(what, sum) and truncate the source table.
I guess I would calculate the date's in code and then pass them as arguments, to the SQL you are using.
I would do it using a compiler. Youtube probably does that too, considering the amount of traffic and the response times.
The principle is easy to understand. You log every every view or rating in a page_view table. You define periods at which the compilation occurs (hourly, daily, weekly, monthly). Every time you hit the good time (e.g.: end of the day), you execute the compiler, which essentially execute a query à-la...
SELECT * FROM page_view WHERE date > $from_date AND date < $to_date
... and store the result. This probably works better in a cron job.
The next time you need to display the information, you can just fetch the stored result and display it without re-computation. There are a variety of storage methods you can use: a MySQL table (e.g.: page_view_compiled), memcached, etc.
I have an online dictionary. All inputs are saved in table "id-word-unixtime". Now I want to make statistics of how many words were searched in an day, and then make graph or table with that data.
What is a best way to archive this? Should I make another table for dates or write a PHP script?
I just need basic direction.
Many questions there, but the main thing you seem concerned about is getting dates from unixtime.
MySQL has FROM_UNIXTIME().
Something like this should get you started:
SELECT FROM_UNIXTIME(unixtime,'%Y-%m-%d') as the_date,
count(word) as word_count
FROM table
-- add conditions here
GROUP BY 1;
If you have PHP-specific questions regarding data presentation I suggest you open another question.
You probably want to compute the answer once and then cache the result, to reduce the load on your server. Once the day is over, you only have to calculate all the statistics for it once.
Something important to thing about is when a day begins and ends. If your site has visitors form all over, you should probably use UTC. If 95% of your visitors are from the US, Eastern Time might make more sense.
SELECT COUNT(*), FROM_UNIXTIME(unixtime, '%Y-%M-%D')
FROM table
WHERE unixtime BETWEEN $start AND $end
GROUP BY FROM_UNIXTIME(unixtime, '%Y-%M-%D')
This should give you each day with searches per day. It's quite an expensive search, so you may want to cache it.
You're right, it's a very basic SQL query. Something like
SELECT word, count(word) FROM table
WHERE unixtime BETWEEN (starttime AND endtime) GROUP BY word
You can calculate starttime and endtime in either PHP or MySQL.
And sure, you will need to write a PHP script to draw a graph, but it's another question.
So I've got a simple query in MySQL that sets a new member's expiration date once they pay their dues:
UPDATE members SET joined=CURDATE(), expires=DATE_ADD(CURDATE(), INTERVAL 1 YEAR), active='1' WHERE id=1000
this query has run 200+ times, normally with the correct result - the current date is put in the joined field, and a year from that date in the expires field. However, in about 10 instances, the expires date has been set to 00-00-0000 with no obvious explanation. I started writing the query to a text file every time to make sure the syntax was correct and I hadn't missed anything - and I didn't - it's exactly that query (with only the ID varying) for every query, those that work, and those that don't.
The only thing I can think here is that there must be an issue with MySQL's DATE_ADD function. Has anyone else experienced anything like this?
UPDATE:
I should add that the joined field is correct with the current date in the cases where the expires date is incorrect.
I'm using MySQL 5.0.81.
There are no triggers.
The table is using MyISAM.
IMPORTANT UPDATE:
I'm an idiot - when I say 11-30-1999 that's not actually what's in the database. I absent-mindedly wrote that, but in fact the database contains the value 00-00-0000 - 11-30-1999 is just how it gets rendered by PHP onto my page. Sorry about that, hopefully that will make this problem less difficult to figure out.
Just a thought.... those "wrong" dates didn't happen to be leap year dates did they.... Feb. 29th for example?
It shouldn't matter, but it may be a bug.