Algorithms for processing visit logs - php

Question

Suppose I have a MySQL table that looks like the following, where I keep track of when (Date) a user (User.id) read an article on my website (Article.id):
------------------------------------------
Article_Impressions
------------------------------------------
date | user_id | article_id
--------------------+---------+-----------
2013-04-02 15:33:23 | 815 | 2342
2013-04-02 15:38:21 | 815 | 108
2013-04-02 15:39:33 | 161 | 4815
...
I'm trying to determine how many session I had, as well as average session duration per user on a given day. A session ends when an article was not read within 30 minutes after another article.
Question
How can I efficiently determine how many session I had on a given day? I'm using PHP and MySQL.
My first idea is to query all that data for a given day, sorted by user. Then I iterate through each user, check if an impression was within 30 minutes of the last impression, and tally up a total count of session each user had that day.
Since we have around 2 million impressions a day on our site, I'm trying to optimize this report generator.

Answer 1

Try this query
Query 1:
select
#sessionId:=if(#prevUser=user_id AND diff <= 1800 , #sessionId, #sessionId+1) as sessionId,
#prevUser:=user_id AS user_id,
article_id,
date,
diff
from
(select #sessionId:=0, #prevUser:=0) b
join
(select
TIME_TO_SEC(if(#prevU=user_id, TIMEDIFF(date, #prevD), '00:00')) as diff,
#prevU:=user_id as user_id,
#prevD:=date as date,
article_id
from
tbl
join
(select #prev:=0, #prevU=0)a
order by
user_id,
date) a
[Results]:
| SESSIONID | USER_ID | ARTICLE_ID | DATE | DIFF |
-----------------------------------------------------------------
| 1 | 161 | 4815 | 2013-04-02 15:39:33 | 0 |
| 2 | 815 | 2342 | 2013-04-02 15:33:23 | 0 |
| 2 | 815 | 108 | 2013-04-02 15:38:21 | 298 |
| 3 | 815 | 108 | 2013-04-02 16:38:21 | 3600 |
This query will return a unique session for every new user and also for same user if the next article read is after 30 mins as per your requirement mentioned in your question. The diff column returns the seconds difference between the 2 articles by the same user which helps us count the sessionId. Now using this result it will be easy for you to count the average time per user and also total time per session.
Hope this helps you...
SQL Fiddle

Answer 2

If the concept of the user "session" is important to your analytics, then I would start logging data in your table to make querying of session-related data not such a painful process. A simple approach would be to log your PHP session ID. If your PHP session id is set to have the same 30 minute expiry, and you log the PHP session ID to this table then you would basically have exactly what you are looking for.
Of course that won't help you with your existing records. I would probably go ahead and create the session field and then back-populate it with randomly generated "session" id's. I wouldn't look for a fully SQL solution for this, as it may not do what you want in terms of handling edge cases (sessions spanning across days, etc.). I would write a script to perform this backfill, which would contain all the logic you need.
My general approach would be to SELECT all the records like this:
SELECT user_id, date /* plus any other fields like unique id that you would need for insert */
FROM Article_Impressions
WHERE session_id IS NULL
ORDER BY user_id ASC, date ASC
Note: make sure you have index on both user_id and date fields.
I would then loop through the result set, building a temp array of each user_id, and loop through that array for all date values assigning a randomly generated session id which would change each time the date change was greater than 30 minutes. Once the user value increments, I would make inserts for that previous user to update the session_id values and then reset the temp array to empty and continue that process with the next user.
Note that it is probably important to take the approach of keeping a relatively small temp/working array like this, as with the number of records you are talking about, you are likely not going to be able to read the entire result set into an array in memory.
Once your data is populated, the query becomes trivial:
Unique sessions for each day:
SELECT DATE(date) as `day`, COUNT(DISTINCT session_id) AS `unique_sessions`
FROM Article_Impressions
GROUP BY `day`
ORDER BY `day` DESC /* or ASC depending on how you want to view it */
Average sessions per day:
SELECT AVG(sessions_per_day.`unique_sessions`) AS `average_sessions_per_day`
FROM
(
SELECT DATE(date) as `day`, COUNT(DISTINCT session_id) AS `unique_sessions`
FROM Article_Impressions
GROUP BY `day`
) AS sessions_per_day
GROUP BY sessions_per_day.`day`
Note: you need an index on the new session_id field.