i need to show the number of online visitors, but there is a problem with selecting algoritm to do it!
maybe i must create a table in DB, where i'll store ip addresses of visitors and time of visit! by so i can show the count of ip addresses, which's time >= NOW() - 10 minutes, for example...("now()-10 minutes" is just to show the logic, i know that this is not a function:)
is this goog way to go?
please give me an idea.
Thanks
This is good tutorial. Note that mysql (i believe youll use it) online users table should be typed as MEMORY.
I'm not really sure how you would use AJAX to store the data...
I personally use the database solution.
I store user_id, last_seen, IP and location in the site (but that's not necessary to just get the count).
When the user requests a page refresh the last_seen column and delete all the entries with NOW()-last_seen greater than x minutes.
Keeping track of "visitors" (as opposed to raw page requests, which the web server should track on its own) is a complex art.
You could store IP addresses, as you described, but what about a visitor who's using a proxy that rotates their IP as frequently as every page load? What about a set of visitors all using the same proxy that uses the same IP for all of them?
My recommendation: don't bother doing any of it yourself, and use Google's free Analytics service. It tracks visitors, browsers, traffic sources, and just about anything else you could possibly want to know about who's looking at your site.
Yes, the algorithm is o.k. in general but with some corrections
May be you want to delete all outdated records first, and then just count the rest.
10 minutes is too much. 1-3 is the average time user spend on the page.
AJAX project is funny, but it has nothing to do with storage. You need more education of terms of client-server application. AJAX is the transport, not storage.
If you expect the website will have a lot of visitors, the query could make the page for one user every 10 minutes pretty slow...
If so, I would suggest to write an CLI script that will clean the old entries and run it in a cronjob. That way the user wouldn't notice any delay, as the parsetime will be spent on the CLI.
Related
I just want a approach on how to build a database with live records, so don't just downvote. I don't expect any code.
At the moment I have a MySql database with about 2 thousand users, they are are getting more though. Each player/user has several points, which are increasing or decreasing by certain actions.
My goal is that this database gets refreshed about every second and the user with more points move up and others move down... and so on
My question is, what is the best approach for this "live database" where records have to be updated every second. In MySql I can run time based actions which are executing a SQL command but this isn't the greatest way I think. Can someone suggest a good way to handle this? E.g. other Database providers like MongoDB or anything else?
EDIT
This doesn't work client side, so I can't simply push/post it into the databse due some time based events. For explanation: A user is training his character in the application. This training (to get 1 level up) takes 12 hours. After the time is elapsed the record should be updated in the database AUTOMATICALLY also if the user doesn't send a post request by his self (if the user is not logged in) other users should see the updated data in his profile.
You need to accept the fact that rankings will be stale to some extent. Your predicament is no different than any other gaming platform (or SO rankings for that matter). Business decisions were put in place and constantly get reviewed for the level of staleness. Take the leaderboards on tags here, for instance. Or the recent change that has profile pages updated a lot more frequently, versus around 4AM GMT.
Consider the use of MySQL Events. It is built-in functionality that replaces the need for cron tasks. I have 3 event-related links off my profile page if interested. You could calculate ranks on a timed schedule (your tolerance for staleness) and the users' requests for them would be fast (faster than the below from Gordon). On the con-side, they are stale.
Consider not saving (writing) rank info but rather focus just on filling in the slots of your other data. And get your rankings on the fly. As an example, see this rankings answer here from Gordon. It is dynamic, runs upon request with at least at that moment non-staleness, and would not require Events.
Know that only you should decide what is tolerable for the UX.
Basicaly, I have an PHP webservice which will be available from:
website
mobile phone clients - (android/iphone)
Data is retreived in JSON format. Request is sent using GET method over HTTPS protocol. Data in database is kept in mongo database.
Here comes my main questions:
I want to get the statistics for each user - how many calls he is doing per-minute/hour - how to save this data? Will it be okay to save everything into mongo database? (1 day statistics = 43 millions rows) Is there a solution maybe which will keep data only for X days and then auto-trunc everything automatically?
I also want to get statistics for each IP - how many calls were made from it per-minute/hour/day - how to save this data? Will it be okay to save everything into mongo database? Will it not become too large?
Is it always possible to see an IP of an user who is making a call to a webservice? what about IPv6?
These 3 questions are the most interesting to me currently. I am planning on letting users use basic services without the need of loggin-in! My main concern is the database/file system performance.
Next comes the description about - what measures I am planning to use. How did I come to this solution and why these 3 questions above are essential. Feel free to ignore the text below if you are not interested in details :)
I want to protect my webservice against crawling I.e. somebody can pass parameters (which are not hard to guess) to get entire data off my databse :)
Here is an usage example: https://mydomain/api.php?action=map&long=1.23&lat=2.45
As you can see - I am already using a secure https protocol, in order to prevent accidental catching entire GET request. It also protects agains 'man in the middle' attacks. However, it doesn't stop attackers from getting into website and going through JS AJAX calls to get an actual request structure. Or decompiling entire android's .APK file.
After reading a lot of questions through out the internet - I came to the conclusion that there is no way of protecting my data entirely, but I think I have found an approach of making the life of a crawlers a lot harder!
And I need your advice on either - if this whole thing is worth implementing and what technologies shall be used in my case (see next).
Next comes the security measures against non-website (mobile device) use of a service for users which are not logged-in.
<?php
/*
Case: "App first started". User: "not logged in"
0. generate UNIQUE_APP_ID for APP when first started
1. send UNIQUE_APP_ID to server (or request new UNIQUE_APP_ID from server to write it on device)
1.1. Verify how many UNIQUE_APP_IDs were created from this IP
if more than X unique values in last Y minutes ->
ban IP temporary
ban all UNIQUE_APP_IDs created by IP during Y minutes (use delay to link them together).
but exclude UNIQUE_APP_IDs from ban if these are not showing strange behaviour ( mainly - no calls to API)
force affected users to log-in to continue to use the service or ask to retry later when ban wears off
else register UNIQUE_APP_ID on server
Note: this is a 'hard' case, as the IP might belong to some public Wi-Fi AP. Precautions:
* temporary instead of permanent ban
* activity check for each UNIQUE_APP_ID belonging to IP. There could be legit users who use the service from long time thus will not be affected by this check. Created Z time ago from this IP.
* users will be not be ever banned - rather forced to log-in, where more restrictive actions will be performed individually!
Now that the application is registered and all validations are passed:
2. Case: "call to API is made". User: "not logged-in"
2.1. get IP from which call is made
2.2. get unique app ID of client
2.3. verity ID against DB on server
if not exists -> reject call
2.4. check how many calls this particular ID did in the last X minutes
if more than X calls -> ban only this unique ID
2.5 check how many Ids were banned from this IP in the last X minutes. If more than Y then ban new calls for whole IP temporary
check if all banned IDs were created from the same IP, if yes then also ban that IP, if it is different from new IP
*/
?>
As you can see - my whole solution is based on the idea that I can store the data about webservice and retrieve it for analysis easily.. for each single webservice call.. Or maybe each X'th call. I have no idea about - what kind of database shall be used. I was thinking that mongo might not be the best choice. Maybe MySQL? Keeping data safe from wrong users is one reason. Another reason is that abusal usage of database will result in a huge load on a database.(DDos?) So, i think this might be a good idea to count webservice calls.
On the other side. A bit of calculations.
If there are 1000 users working simultaniusly. Each generating 30 calls to a webservice per minute. So it's 30000 disc writes in a minute. In hour it's 60 times that i.e. 1.800.000 disc writes in an hour. If I am planning to keep statistics about daily usage then it's 24 times that i.e. in
average there will be 43.200.000 records for tracking purposes kept on a server.
Each record contains information about: time + IP + user unique ID
I was also thinking about not storing any data at all. Use redis instead. I know that there is some kind of counter exists. For for each individual IP I can create a separate key and start counting calls. In this case everything will be kept in server's RAM. There is also an expire date parameter which is possible to set for each key. And for separate users I can store their IDs instead of network IP. This solution only came to my mind after I finished writing this whole essay, so let me hear your ideas about questions above.
I'm developing a project of mine with scalability in mind and I've come to a crossroad. On my website I would like to detect if a user is online or not. And I can't quite think of the best way to handle this. The way I was thinking would be something along these lines(in psuedocode):
// SQL user table:
user {
"name": "blah blah",
"email": "derpy#derpyderp.com",
"online": false
}
So whenever the user logs in I could update his online column to true. However that would eventually lead to SQL queries happening every time a user logs in and if it happens that I get say, 10 logins per second, well, that's a lot of queries happening. Another way I figured I could do the same thing but in a different table:
// Activity table:
activity {
"user_id": 2,
"online": true
}
For some reason I believe that would lead to less memory consumption because of the separation from the user table. However I'm not sure if it would have any actual effect on performance.
So if you could bless me with your insight I would be more then grateful, thank you.
Generally speaking it's a common practice to add a column to the users table to store the lastActivity time. Anytime the user logs in, or accesses a page, store the current time in that field. If you want to know whether or not they are online, see if the last recorded time is within a certain window - say, five minutes. You can query all rows to see how many users are currently online as a result.
I wouldn't be too worried about running queries every few seconds - your server can handle it (assuming these are well-written and not very verbose).
you can use datetime for field type and don't forget to record user IP so you can track time o
Depending on how you want it to work you basically have two options:
Define a timeout after which you consider a user logged out
Use ajax/websockets/whatever to poll user
1: Timeout
This is the simpler use case. Every time the user requests a page, you update a timestamp in your database.
To find out how many users are online, you would do a query against this database and do a COUNT of users who have been active in the last N minutes.
This way you will get a relatively accurate idea of how many people are actively using the site at the moment.
2: Constant polling
This is a bit more complex to implement due to having to update the server with Ajax. Otherwise it works in a similar fashion to #1.
Whenever a user is on a page, you can keep a websocket open or do ajax requests every N seconds to the server.
This way you can get a pretty good idea of how many people have pages open on your site currently, but if a user leaves the page open in their browser and doesn't do anything, it would still count them as being online.
A slight modification to the idea would be to use a script on the client to monitor mouse movement. If the user doesn't move the mouse on your page for say 10 minutes, you would stop the polling or disconnect the websocket. This would fix the problem of showing users who are idle as being online.
It could be a really easy thing to do but I am getting pretty confused by it.
Basicly, I want to get the amount of people currently browsing my website. The problem is, I can't count sessions. Many of them are created for different purposes. (e.g $_SESSION['age'] for the users who logged in.)
Also, there are "logged_in" and "not_logged_in" people.
In short, how can I count the users in my website in the most professional way?
Ps. Please note that I will run the function every 1-5 minutes, get the amount, store it to be used as cache. After 5 minutes, cache will be replaced again, and so on. Hence, db connections will be a possibility if you think that would be the most professional way.
One of the issues at hand is, your PHP doesn't know when your user closes the browser.
I've done something similar with a combination of a "last_used"-field in the user database (timestamp/datetime) and a "heartbeat"-function (jscript), which regularly sends an "I'm still alive"-command to the server (every x seconds), where then this last_used-field is updated. It ain't a high traffic site, but it does the job very reliable.
Then just count the amount of users which a very current heartbeat in the database:
SELECT COUNT(*)
FROM your_user_table
WHERE last_used > TIMESTAMPADD(NOW(), INTERVAL -30 SECOND)
You don't need a user table for this, you can use a session table, too. Such a mechanism can then be used for other features, too, like giving the users new messages on the fly and similar.
I would like to make a web-based game which is Travian-like (or Ikariam-like). The game will be in PHP & MySQL-based. I wonder how can I achieve the live updating of game attributes.
For frontend, I can achieve by using AJAX calls (fetch the latest values from database), or even fake update of values (not communicated with server).
For backend, is this done by a PHP cron job (which runs every few seconds)? If so, can anyone provide me some sample codes?
by the way, I know it would be a trouble if I use IIS + FastCGI.
=== Version Information ===
PHP : 5.2.3
IIS : 6.0 with FastCGI
OS : Windows Server 2003 Standard R2
The correct answer depends on your exact needs.
Does everyone always get resources at the same rate? If so, a simple solution is to track how long their user has existed, calculate the amount of resources based on the rate they're getting, and subtract the number of resources they've spent in total. That's going to be a bit of a problem if the rate can ever change, though, so if you use this solution, you're pretty much stuck with the rate you pick unless you rewrite the handling entirely (for example to the one below).
If it varies how quickly people can get resources, you'll need to update the data periodically. A cronjob/scheduled task would work well to make sure everyone is updated, but in some situations, it might be better to simply measure how long it's been since you've updated each user's resources, and then update them on every page load they make while logged in by multiplying the time they've been away by the rate at which they gain resources - that way, you avoid updating until you actually need the new value.
For a Travian like resource management you need to keep track when you updated the users resources for the last time. If you read the resource values (for a page refresh or something), you need to add the amount of resources gained since the 'last update time' (depending on the amount of resources fields and boni the user gets) and send that value to the browser. You could also the let browser script calculate these amounts.
You might to consider caching all resource amounts somehow, since these values are required a lot, improving the communication with your database.
If a user finishes building a resource field, uses the market, builds a structure, etc you need to update the amount of resources (and the 'last update time'), because you cannot keep track on these kind of events simply.
By calculating the resources the database load is reduced, since you do not need to write the new values every time when the user refreshes the browser page. It is also more accurate since you have less rounding errors.
To keep the resources increasing between page refreshes you need a method as Frank Farmer described. Just embed the resource amount and the 'gain frequency' in some javascript and increase the resource amount every 'gain frequency' by one.
You can also calculate the ressources each time a page or the javascript asks. You'd need to store the last updated time.
It may be an old post but it comes up right away in Google so here's another option which is how the game I've been developing does it.
I use a client side JavaScript that uses a flash socket to get live updates from a dedicated game server running on the host.
I use the xmlsocket kit from http://devpro.it/xmlsocket/