I have an online dictionary. All inputs are saved in table "id-word-unixtime". Now I want to make statistics of how many words were searched in an day, and then make graph or table with that data.
What is a best way to archive this? Should I make another table for dates or write a PHP script?
I just need basic direction.
Many questions there, but the main thing you seem concerned about is getting dates from unixtime.
MySQL has FROM_UNIXTIME().
Something like this should get you started:
SELECT FROM_UNIXTIME(unixtime,'%Y-%m-%d') as the_date,
count(word) as word_count
FROM table
-- add conditions here
GROUP BY 1;
If you have PHP-specific questions regarding data presentation I suggest you open another question.
You probably want to compute the answer once and then cache the result, to reduce the load on your server. Once the day is over, you only have to calculate all the statistics for it once.
Something important to thing about is when a day begins and ends. If your site has visitors form all over, you should probably use UTC. If 95% of your visitors are from the US, Eastern Time might make more sense.
SELECT COUNT(*), FROM_UNIXTIME(unixtime, '%Y-%M-%D')
FROM table
WHERE unixtime BETWEEN $start AND $end
GROUP BY FROM_UNIXTIME(unixtime, '%Y-%M-%D')
This should give you each day with searches per day. It's quite an expensive search, so you may want to cache it.
You're right, it's a very basic SQL query. Something like
SELECT word, count(word) FROM table
WHERE unixtime BETWEEN (starttime AND endtime) GROUP BY word
You can calculate starttime and endtime in either PHP or MySQL.
And sure, you will need to write a PHP script to draw a graph, but it's another question.
Related
Hi I have a MySQL table of Facebook pages (fbpagesfancount) that has the total fan count by day since 01 Jan 2016.
The structure is like this:-
Pageid, Pagename, Updated_Date, Fan_Count
There are a number of specific days that are missing and do not therefore have fancount values due to Facebook API issues.
The days that are missing are usually single days, for example, there is a value for the day before and the day after.
I'd like to create a new table that has a record for every day since 01/01/2016 for each page (750 pages) and then update the days that are missing by averaging the day before and the day after the missing date.
Is this possible using MySQL only or should I write a script in PHP that performs this task and if so, any suggestions on the logic would be helpful.
Any other suggestions on how to tackle this issue would be welcome.
Thanks
Jonathan
Yes, it is possible in SQL only.
No, you should not attempt it as it is more complicated and for a single shot there's no need.
Yes, write a script in any language you know, for instance PHP.
I'm not sure why you even want to create a new table? You could add a flag to your current table saying its an origional count vs an average, and just find the missing numbers and add them in a script.
Is there a way to add articles (data) in mysql/php but make them
auto-publish during the day when I´m not available.
So lets say, if I have a news site but I´ll be busy tomorrow the whole day so I could pre-write articles the day before with timestamp and they would appear when I want
Is this possible?
How would the script be like:
SELECT FROM articles WHEN TIME is 2011-12-01 12:15
Thanks
As simple as:
SELECT * FROM articles WHERE timestamp <= NOW()
Though I never worked with them to me the easiest solutions seems to be Cronjobs combined with an extra waiting table and a script linking both.
You pre-write your article and store them in table together with the time stamp you want to publish them.
Your cron will invoke a script every 2,3, 5 hours (twice a day, whatever).
This script checks the time stamps in the table against the actual time and if it is about time to realise the article it will do so (or hand the information to the realise script).
So what I'm doing is storing data from a website every 4hrs. I want to have a line graph of the last two days, the y-axis would be number of players and that value can be anywhere from 0-30,000, the value is dependent on the scrape of the website.
What is the best way to store the data in mysql and where is a easy to use graphing solution?
Has anyone used Raphaël?
Hey, Google Charts is exactly what you're looking for. It can create any type of chart from a data set, and is very customizable.
As for the actual data retrieving, the answers above will help you. :)
You could have a table structure like
Player_Stats
players int
hour int
Then each hour you could write something like:
insert into Player_stats (players, hour) values(NUMBEROFPLAYERS, HOUR#);
Where HOUR# is a value from 1 to X number of possible hours (if you want to only store things in a running log, otherwise, change hour to a timestamp)....the insert would be more like
insert into Player_Stats (players, timestamp) values(NUMBEROFPLAYERS, NOW());
Then you'd retreive your data with:
select players, hour from Player_Stats;
Or if you kept things in perpetuity and wanted to grab a range from now to 2 days ago:
select players, timestamp from Player_Stats where timestamp between now() and date_sub(timestamp, interval 2 day);
Then you could use a charting library like Google Visualizations...they have good documentation on formatting the data specifically for their different charts.
I've done something similar. I stored the number of players along with a time stamp in a table, then used jquery and jqplot to display the data.
I would set up a cron to run a SELECT TO OUTFILE myFile statement regularly. Note that myFile cannot be an existing file for security purposes (docs), so you'd have to have the cron also delete the file after the plot is created.
I have found ploticus to be very easy to work with, and can make some very complex plots without too much difficulty.
I am developing a website which will have 200.000 pages. There is also a browse section, which shows most popular, highest rated etc. documents. However this section will become almost static couple of weeks later, after launch. So I also would like to implement a filtering system which will show today's, this week's, this month's most popular items, just like youtube.
Just like this:
http://www.youtube.com/videos?c=2
How should I implement this function? Do I need another table, which will have a new entry for every document each day?
docid, date, view_count, rating
So I will get today's row for filtering by using a day, or calculate a week (7 rows) for filtering by using week? It seems not efficient. Do you have any suggestions?
I am using LAMP stack by the way.
Thanks,
Assuming you timestamp the records in your table, you should be able to put a where clause that limits the timestamp to whatever timeframe you want.
You can cache the result, especially the longer ones, for long enough to make the request inconsequential.
EDIT
But perhaps you mean most popular today, not most popular that was added today?
In which case, I don't have an answer.
The most direct approach is to save the timestamp and the resource id each time the resource is shown in recent_views(what, when). Daily/weekly/monthly charts can be created with appropriate WHERE clauses like WHERE when > $beginOfPeriod AND when < $endOfPeriod.
For performance reasons you can aggregate the values each night, save the sums in separate tables like daily_views(what, sum) and truncate the source table.
I guess I would calculate the date's in code and then pass them as arguments, to the SQL you are using.
I would do it using a compiler. Youtube probably does that too, considering the amount of traffic and the response times.
The principle is easy to understand. You log every every view or rating in a page_view table. You define periods at which the compilation occurs (hourly, daily, weekly, monthly). Every time you hit the good time (e.g.: end of the day), you execute the compiler, which essentially execute a query à-la...
SELECT * FROM page_view WHERE date > $from_date AND date < $to_date
... and store the result. This probably works better in a cron job.
The next time you need to display the information, you can just fetch the stored result and display it without re-computation. There are a variety of storage methods you can use: a MySQL table (e.g.: page_view_compiled), memcached, etc.
I have this SQL query about getting the 5 events today:
SELECT n.nid, n.type, n.title, nr.body, nr.teaser, FROM_UNIXTIME(e.event_start) start_date, FROM_UNIXTIME(e.event_end) end_date
FROM node n
LEFT JOIN event e ON n.nid = e.nid
LEFT JOIN node_revisions nr ON nr.nid = e.nid
WHERE n.`type` = 'event'
AND NOW() BETWEEN FROM_UNIXTIME(e.event_start) AND FROM_UNIXTIME(e.event_end)
ORDER BY n.`created` DESC
LIMIT 5
Then I need to get the "this week's event" using "a week that includes "today" and starts on a Sunday".
How can I do that in MySQL?
Any help would greatly be appreciated. Thanks in advance :)
Cheers,
Mark
You need to define "this week" better -- do you mean a 7-days sliding window centered on today, or a week (the one that includes "today") starting e.g. on a Sunday? That's entirely dependent on the semantics of "this week" and it's impossible for us to decide what you meant by said ambiguous expression. Of the two approaches you mention, one or the other (or a variant thereon) will be appropriate depending on your meaning.
Edit: the OP has clarified in a comment that he means "a week that includes "today" and starts on a Sunday" -- and I deduce from his use of FROM_UNIXTIME that the specific SQL dialect he's targeting is MySQL. Then, WEEK(somedate, 0) is the MySQL function that should give him exactly what he wants, see mysql's docs.
Specifically,
AND WEEK(CURDATE, 0) BETWEEN WEEK(FROM_UNIXTIME(e.event_start), 0)
AND WEEK(FROM_UNIXTIME(e.event_end), 0)
should be the WHERE clause the OP is looking for.
I'm not sure if this is for SQL Server or MySQL, but in MySQL you could get the current weekday of today and then use date_add to subtract that many days from the current date (start date) then using start date, use date_add again to add 7 days (end date).
Hopefully that helps, let me know if you need help with the syntax.
Based on the table/column names, it appears you're working with Drupal. Have you considered using a View to achieve your goal? I can't tell from the context whether this is part of a module you're writing, in which case keep plugging away, or whether you just want a list of events to display in a block, in which case a View should be able to do all this for you without messing around with PHP/SQL.
I don't know if you have that option, but for performance reasons it could be better to do the date calculations in your program code. If you use a function on a column in a WHERE clause, MySQL cannot use indexes. A simple example: http://netfactory.dk/2004/12/13/mysql-date-functions-and-indexes/
Most languages should have decent functions/libraries for date/time manipulation.