I'm going to be building a page counter for a site to determine views of different pages. Basically, I have users that I'd like to track with respect to page views of about 120 specific pages.
I'm having trouble determining the most efficient way to do this as well as the logic behind it. I was trying 3 tables, but this would end up in many unnecessary rows. I thought storing an array in a single field then updating the array, but I'm not sure this will grow or if it is even possible. Below is my way of limiting the number of requests. Basically every 100 will count 100. Any ideas for structuring this?
$sample_rate = 100;
if(mt_rand(1,$sample_rate) == 1) {
//update set query
}
Based on our conversation in the comments I would recommend the following simple design:
TblPages
--------
Page_Id (pk)
Page_URL (unique)
TblViews
--------
View_Page_Id
View_User_Id
View_Count
(pk is View_Page_Id + View_User_Id)
Whenever a user views a page, you execute a stored procedure that will either insert a record to the TblViews table if there is no record for that user and page, or update the View_Count of the existing record.
To get all the views for a page you use SUM(View_Count) and group by View_Page_Id, to get all the page views by a user you group by View_User_Id, etc.
Related
I have a screen that looks very much like facebook timeline
users can view posts of other users etc.
to get these posts i do something like
select user.id,user.name,posts.title,posts.body from posts left join users;
now data i need to collect is "Who saw this post" .
is there any elegant way to do it ?
right now all what i can think of is every time i fetch posts. i loop over them, then collect all ids of posts that the query returned and then push in another table
user_views [pk:user_id+postId]
userId,postId
1 , 1
Then when i'm fetching posts next time i can do count of user_views.
select *,count(user_views.id) from posts join user_views on post_id = post.id
but this sound like a lot of work for each VIEW, specially that most probably user will see a most multiple times,
is there any known patterns for such need ?
This is a design question and the answer really depends on your needs.
If you want to know exactly who viewed what post and how many times, then you need to collect the data on user - post level.
However, you may decide that you do not really care who viewed which post how many times, you just want to know how many times a post was viewed. In this case you may only have a table with post id and view count fields and you just increment the view count every time a post is being viewed.
Obviously, you can apply a mixed approach and have a detailed user - post table (perhaps even with timestamp) and have an aggregate table with post id and view count fields. The detailed table can be used to analyse your user's behaviour in a greater detail, or present them a track of their own activities, while your aggretage table can be used to quickly fetch overall view counts for a post. The aggregate table can be updated by a trigger.
The question is how to structure a huge mysql database with for example 10.000 profiles.
Example1
1 Database
3 Database fields (field_profile, field_images, field_posts etc.)
field_profile 10.000 profiles with an id and the rest of the information.
field_images 150.000 images related to the id in database_profile.
field_images 350.000 posts related to the id in database_profile.
Slow searching but when i want to change something really easy.
Example2
1 Database
30.000 Database fields (field_profile_profile1, field_images_profile1, field_posts_profile1, field_profile_profile2, field_images_profile2, field_posts_profile2 etc..
field_profile_profile1 1 profile with information.
field_images_profile1 50
field_posts_profile1 3500
Fast searching but when i want to change something really difficult?!
Which example is the best or is there a better option?
I would go for normalization, according that your number of records are not even close to huge:
[table_profile]
profile_id
[table_image]
image_id
profile_id
[table_post]
post_id
profile_id
You should simply come up with table that if Field profile and field image is same for different post then
table profile
field_id field_profile field_image
table post
field_id field_post
Second option if field_profile , field_image is different every time then come up with single table .
table field
field profile field_image field_post
Try to store repeating data in different table and use id's whenever needed , it will decrease your data storage and increase speed of database access .
I have a MySQL database with a growing number of users and each user has a list of items they want and of items they have - and each user has a specific ID
The current database was created some time ago and it currently has each users with a specific row in a WANT or HAVE table with 50 columns per row with the user id as the primary key and each item WANT or HAVE has a specific id number.
this currently limits the addition of 50 items per user and greatly complicates searches and other functions with the databases
When redoing the database - would it be viable to instead simply create a 2 column WANT and HAVE table with each row having the user ID and the Item ID. That way there is no 'theoretical' limit to items per user.
Each time a member loads the profile page - a list of their want and have items will then be compiled using a simple SELECT WHERE ID = ##### statement from the have or want table
Furthermore i would need to make comparisons of user to user item lists, most common items, user with most items, complete user searches for items that one user wants and the other user has... - blah blah
The amount of users will range from 5000 - 20000
and each user averages about 15 - 20 items
will this be a viable MySQL structure or do i have to rethink my strategy?
Thanks alot for your help!
This will certainly be a viable structure in mysql. It can handle very large amounts of data. When you build it though, make sure that you put proper indexes on the user/item IDs so that the queries will return nice and quick.
This is called a one to many relationship in database terms.
Table1 holds:
userName | ID
Table2 holds:
userID | ItemID
You simply put as many rows into the second table as you want.
In your case, I would probably structure the tables as this:
users
id | userName | otherFieldsAsNeeded
items
userID | itemID | needWantID
This way, you can either have a simple lookup for needWantID - for example 1 for Need, 2 for Want. But later down the track, you can add 3 for wishlist for example.
Edit: just make sure that you aren't storing your item information in table items just store the user relationship to the item. Have all the item information in a table (itemDetails for example) which holds your descriptions, prices and whatever else you want.
I would recommend 2 tables, a Wants table and a Have table. Each table would have a user_id and product_id. I think this is the most normalized and gives you "unlimited" items per user.
Or, you could have one table with a user_id, product_id, and type ('WANT' or 'HAVE'). I would probably go with option 1.
As you mentioned in your question, yes, it would make much more sense to have a separate tables for WANTs and HAVEs. These tables could have an Id column which would relate the row to the user, and a column that actually dictates what the WANT or HAVE item is. This method would allow for much more room to expand.
It should be noted that if you have a lot of of these rows, you may need to increase the capacity of your server in order to maintain quick queries. If you have millions of rows, they will have a great deal of strain on the server (depending on your setup).
What you're theorizing is a very legitimate database structure. For a many to many relationship (which is what you want), the only way I've seen this done is to, like you say, have a relationships table with user_id and item_it as the columns. You could expand on it, but that's the basic idea.
This design is much more flexible and allows for the infinite items per user that you want.
In order to handle wants and have, you could create two tables or you could just use one and have a third column which would hold just one byte, indicating whether the user/item match is a want or a need. Depending on the specifics of your projects, either would be a viable option.
So, what you would end up with is at least the following tables:
Table: users
Cols:
user_id
any other user info
Table: relationships
Cols:
user_id
item_id
type (1 byte/boolean)
Table: items
Cols:
item_id
any other item info
Hope that helps!
Im building a classifieds website of adverts where I want to store a count of the number of views of each advert which I want to be able to display in a graph at a later date by day and month etc.. for each user and each of their adverts. Im just struggling with deciding how best to implement the mysql database to store potentially a large amount of data for each advert.
I am going to create a table for the page views as follows which would store a record for each view for each advert, for example if advert (id 1) has 200 views the table will store 200 records:
Advert_id (unique id of advert)
date_time (date and time of view)
ip_address ( unique ip address of person viewing advert)
page_referrer (url of referrer page)
As mentioned I am going to create the functionality for each member of the site to view a graph for the view statistics for each of their adverts so they can see how many total views each of their adverts have had, and also how many views their advert has had each day (between 2 given dates) and also how many views per month each advert has had. I'll do this by grouping by the date_time field.
If my site grows quite large and for example has 40,000 adverts and each advert has on average 3,000 page views, that would mean the table has 120 Million records. Is this too large ? and would the mysql queries to produce the graphs be very slow?
Do you think the table and method above is the best way to store these advert view statistics or is there a better way to do this?
Unless you really need to store all that data it would probably be better to just increment the count when the advert is viewed. So you just have one row for each advert (or even a column in the row for the advert).
Another option is to save this into a text file and then process it offline but it's generally better to process data as you get it and incorporate that into your applications process.
If you really need to save all of that data then rotating the log table weekly maybe (after processing it) would reduce the overhead of storing all of that information indefinitely.
I was working with website with 50.000 unique visitors per day, and i had same table as you.
Table was growthing ~200-500 MB/day, but i was able to clean table every day.
Best option is make second table, count visitors every day, add result to 2nd table, and flush 1st table.
first table example:
advert_id
date & time
ip address
page referrer
second table example (for graph):
advert_id
date
visitors
unique visitors
Example SQL query to count unqiue visitors:
SELECT
advert_id,
Count(DISTINCT ip_address),
SUBSTRING(Date,1,10) as Date
FROM
adverts
GROUP BY
advert_id,
Date
Problem is not even perfomance (MySQL ISAM Engine is quite smart and fast), problem is storage such big data.
90% statistics tools (even google analytics or webalyzer) is making graphs only once per day, not in real-time.
And quite good idea is store IP as INT using function ip2long()
What is the best way to count page views for dynamic pages like the url example below? I'm using PHP and MySQL. A brief explanation would help. Thanks!
http://www.example.com/posts/post.php?id=3
Usually the table structure looks like this:
table pages:
id | name | ...
==========================
1 Some Page
2 Some Other Page
table pages_views:
page_id | views
================
1 1234
2 80
where pages_views has a unique index on page_id
The MySQL statement to increment the views then looks as follows:
INSERT INTO `pages_views` SET views=1 WHERE page_id=?
ON DUPLICATE KEY UPDATE views=views+1 ;
Since pages_views.page_id is unique, the row for the page will get created if it doesn't exist; if it exists (that's the "duplicate key" clause), the counter will be incremented.
I chose two separate tables here, as CMS pages usually aren't updated too often (and therefore, their load is mostly reads), whereas page views are read and updated, well, with each page view.
This is my code and it's working properly when I open the page or When I refresh the page, page views is incrementing by 1. If the page_id doesn't exist it will insert a record with views = 1, if page_id exists it will increment the views
`INSERT INTO pages_views ( pages_id, views) VALUES ( $page_id, 1) ON DUPLICATE KEY UPDATE views=views+1`
With PDO you will have something like this
$sql = "INSERT INTO pages_views ( pages_id, views) VALUES ( :pageId, 1) ON DUPLICATE KEY UPDATE views=views+1";
$q = $conn->prepare($sql);
$q->execute(array(':pageId'=>$pageId));
Well, you can simply add a field pageviews to your pages table and do UPDATE pageviews = pageviews +1 WHERE id = 1 query on each page load
For the sake of viewing statistics by day/week/month/year, I have made two tables. The first archives all visits to the site with my page and id saved on the same row. The second table records tallys, such as Piskvor describes.
The benefit is that I can view stats for any page and ID I want over time (but that'll be a lot of rows over time...) or I can simply view total pageviews. For the visitors of my site, I serve information from this second table, but my admin panel makes full use of the first table.
statsEach
- statID
- page (example: page 100 is index.php, or 210 is news.php)
- id (example: 1 is news story 1, 2 is news story 2,...)
- date
- time
- user
and
statsTotal
- statTotalID
- page
- id
- total
I don't know what you need/want to do, or even if my table structure is best, but this works for me.
Just increase an integer on the post you currently serve.
A simple example can be found at http://www.configure-all.com/page_view_counter.php