I just want a approach on how to build a database with live records, so don't just downvote. I don't expect any code.
At the moment I have a MySql database with about 2 thousand users, they are are getting more though. Each player/user has several points, which are increasing or decreasing by certain actions.
My goal is that this database gets refreshed about every second and the user with more points move up and others move down... and so on
My question is, what is the best approach for this "live database" where records have to be updated every second. In MySql I can run time based actions which are executing a SQL command but this isn't the greatest way I think. Can someone suggest a good way to handle this? E.g. other Database providers like MongoDB or anything else?
EDIT
This doesn't work client side, so I can't simply push/post it into the databse due some time based events. For explanation: A user is training his character in the application. This training (to get 1 level up) takes 12 hours. After the time is elapsed the record should be updated in the database AUTOMATICALLY also if the user doesn't send a post request by his self (if the user is not logged in) other users should see the updated data in his profile.
You need to accept the fact that rankings will be stale to some extent. Your predicament is no different than any other gaming platform (or SO rankings for that matter). Business decisions were put in place and constantly get reviewed for the level of staleness. Take the leaderboards on tags here, for instance. Or the recent change that has profile pages updated a lot more frequently, versus around 4AM GMT.
Consider the use of MySQL Events. It is built-in functionality that replaces the need for cron tasks. I have 3 event-related links off my profile page if interested. You could calculate ranks on a timed schedule (your tolerance for staleness) and the users' requests for them would be fast (faster than the below from Gordon). On the con-side, they are stale.
Consider not saving (writing) rank info but rather focus just on filling in the slots of your other data. And get your rankings on the fly. As an example, see this rankings answer here from Gordon. It is dynamic, runs upon request with at least at that moment non-staleness, and would not require Events.
Know that only you should decide what is tolerable for the UX.
Related
I'm working on a basic lamp(willing to change) website , and I currently need a way to run some function on the server that runs for several hours per user, and every X hours it needs to query the mysql database to see if the value for that user has been updated, if it hasn't it need it to insert a new record in the database...I also should mention that the 'every X hours' can change per user too, and the total runtime of the function per user can also vary.
So basically I need a function that runs continuously on the server for few hours per user. What is the best way to do this? I want the site to be able to support many users (like 10000 +).
I'm willing to try new technologies for every aspect of the site, I'm still in the design phase and I was looking for some input.
I've looked at cron but not really sure how well it would work when dealing with so many users...
edit: Here is a typical scenario of events;
User presses button on the website and closes the browser.
Server starts a timer from when they pressed the button, now
the server will check if that user has pressed a different button within a given time frame (time frame can change per user), say within 30 minutes. If they didn't press the other button then the server needs to automatically insert a new record in the database.
The script will need to continue running, checking every 30 mins for say the next 5 hours.
Thank you!
Cron would work as well as you can code the page it will run. It's not a cron limitation.
The question is ambiguous btw. Maybe explaining your full scenario would help.
Meanwhile, my suggestion would be to set up a scrip that allows you to manually check what you need to check.
You definitely need the DB to be InnoDB optimized with proper indexes to be able to support 1000 plus users.
To alleviate the number of calls to the database, a common practice is to run scripts only on what you are interested (so in the case of users you would only select those who have logged on in say the past 3 hours)
That's achievable in 2 ways, a simple select statement, or by adding entries to a specific table on the login page, and remove them after the automated script has finished running.
All of this is pure theory without understanding exactly what you need to do though.
You are telling what/how you want to do, but not why you want to do it. Maybe letting us know why could lead to a different how ;)
However, what you can do is still use cron (or anything similar). The trick is to have
a last_interaction timestamp column
a maximum_interval column
a daily_runtime column
in your users database. Not optimized but you are in the design phase so you shouldn't pay too much attention to the performance aspect (except is explicitly required).
I am a newbie with PHP and therefore this is more of a conceptual question or maybe even a question about 'best practices'.
Often, I see websites with stats drawn from their database. For example, let's say it is a sales lead website. It may have stats at the top of the page like:
NEW SALES LEADS YESTERDAY: 123
NEW SALES LEADS THIS MONTH: 556
NEW SALES LEADS THIS YEAR: 3870
Obviously, this should not be calculated everytime the page is displayed, right? That would potentially be a large burden on the server? How do people cache this type of data. Any best practices? I thought I writing a CRON jobs that would calculate it on a daily basis and insert to a database. What are your ideas? Thank you!
You can calculate it once and then store it in a xcache. Here, however there doesn't seem to be a need for a cron. The query can run one time and store the result in xcache. Important thing here would be to set the expiration time of the stored value according to your use case. For eg. if you need to store daily stats like above, set the expiration time to be a few hours. In case of data which gets updated every minute, you can set the expiration time to be a few minutes.
Something like this.
$newSalesLeadYest;
if(xcache_isset("newSalesLeadYest")){
$newSalesLeadYest = xcache_get("newSalesLeadYest");
} else{
$newSalesLeadYest = runQueryToFetchStat();
//Cache set for X secs
xcache_set("newSalesLeadYest", $newSalesLeadYest, X);
}
What you need is to come up with a caching strategy.
Some factors to help you decide:
How frequent does the data change?
How important is the current values - is it ok if it's 1min, 1hr, 1day old?
How expensive, time wise, is loading fresh data?
How much traffic are you getting? 10s, 100s, millions?
There are a few ways you can achieve the result.
You can use something like memcached to persist the data to avoid it being generated each request.
You can use http caching and load the data client side using javascript from an api.
You can have a background worker (eg. run by cron), which generates the latest figures and persists to a lookup database table.
You could improve the queries and indexes so that getting live data is fast enough to do every request
You could alter you database schema so that you have more static data
From the 3 examples you gave, 3 simple counts should not be expensive enough to warrant complex caching systems. If you can paste the sql queries, we can help optimise them.
The data sounds like it will only get updated once per day, so a simple nightly cron "flatten" query would be a nice fit.
Ok I know the title doesn't really tell you what my problem is but I'll try it now.
I am developing a game. People can subscribe their animals for a race. That race starts at a specific time. It is a race for which ALL users can subscribe. So the calculation of which animal is first, second etc. happens in an php file that is executed, every 2mins there is a new calculation for about 1h. So there are 30 calculations. But ofc. this code is not connected to the logged in user. The logged in user can click on the LIVE button to see the current result.
Example: There is a race at 17.00 later today. 15 animals subscribed, from 4 players and they can all check how their animals are doing.
I do not want someone to post me the full code but I want to know how I should let a php code run for about 1 hour (so execute code, sleep 2min, new calculation, sleep 2min and so on) on my server or so. So it is not connected to the user.
I thought about cron jobs but that is really not the solution for this I believe.
Thank you for reading :p
Two approaches:
You use an algorithm which will always come to the same conclusion, regardless of when it is run and who runs it. You just define the starting parameters, then at any time you can calculate the result (or the intermediate result at any point in time between start and finish) when needed. So any user can at any time visit your site and the algorithm will calculate the current standings on the fly from some fixed starting condition.
Alternatively, you keep all data in a central data store and actually update the data in certain intervals; any user can request the current standings at any time and the latest data from the datastore will be used. You will still need an algorithm that has traits of the one described above, since you're likely explicitly not actually running the simulation in real time. Just every x seconds, you run your calculations again, calculating what is supposed to have changed from the last time you ran them.
In essence, any algorithm you use needs this approach. Even a "realtime" program simply keeps looping, changing values little by little from their previous state. The interval between theses changes can be arbitrarily stretched out, to the point where you calculate nothing until it becomes necessary. In the meantime, you just store all the data you need in a database.
Cron jobs are the wright way i think. Check this out when you are not so good with algorithm:How To: PHP with Cron job Maybe you have to use different cron jobs.
I'm developing a project of mine with scalability in mind and I've come to a crossroad. On my website I would like to detect if a user is online or not. And I can't quite think of the best way to handle this. The way I was thinking would be something along these lines(in psuedocode):
// SQL user table:
user {
"name": "blah blah",
"email": "derpy#derpyderp.com",
"online": false
}
So whenever the user logs in I could update his online column to true. However that would eventually lead to SQL queries happening every time a user logs in and if it happens that I get say, 10 logins per second, well, that's a lot of queries happening. Another way I figured I could do the same thing but in a different table:
// Activity table:
activity {
"user_id": 2,
"online": true
}
For some reason I believe that would lead to less memory consumption because of the separation from the user table. However I'm not sure if it would have any actual effect on performance.
So if you could bless me with your insight I would be more then grateful, thank you.
Generally speaking it's a common practice to add a column to the users table to store the lastActivity time. Anytime the user logs in, or accesses a page, store the current time in that field. If you want to know whether or not they are online, see if the last recorded time is within a certain window - say, five minutes. You can query all rows to see how many users are currently online as a result.
I wouldn't be too worried about running queries every few seconds - your server can handle it (assuming these are well-written and not very verbose).
you can use datetime for field type and don't forget to record user IP so you can track time o
Depending on how you want it to work you basically have two options:
Define a timeout after which you consider a user logged out
Use ajax/websockets/whatever to poll user
1: Timeout
This is the simpler use case. Every time the user requests a page, you update a timestamp in your database.
To find out how many users are online, you would do a query against this database and do a COUNT of users who have been active in the last N minutes.
This way you will get a relatively accurate idea of how many people are actively using the site at the moment.
2: Constant polling
This is a bit more complex to implement due to having to update the server with Ajax. Otherwise it works in a similar fashion to #1.
Whenever a user is on a page, you can keep a websocket open or do ajax requests every N seconds to the server.
This way you can get a pretty good idea of how many people have pages open on your site currently, but if a user leaves the page open in their browser and doesn't do anything, it would still count them as being online.
A slight modification to the idea would be to use a script on the client to monitor mouse movement. If the user doesn't move the mouse on your page for say 10 minutes, you would stop the polling or disconnect the websocket. This would fix the problem of showing users who are idle as being online.
I would like to make a web-based game which is Travian-like (or Ikariam-like). The game will be in PHP & MySQL-based. I wonder how can I achieve the live updating of game attributes.
For frontend, I can achieve by using AJAX calls (fetch the latest values from database), or even fake update of values (not communicated with server).
For backend, is this done by a PHP cron job (which runs every few seconds)? If so, can anyone provide me some sample codes?
by the way, I know it would be a trouble if I use IIS + FastCGI.
=== Version Information ===
PHP : 5.2.3
IIS : 6.0 with FastCGI
OS : Windows Server 2003 Standard R2
The correct answer depends on your exact needs.
Does everyone always get resources at the same rate? If so, a simple solution is to track how long their user has existed, calculate the amount of resources based on the rate they're getting, and subtract the number of resources they've spent in total. That's going to be a bit of a problem if the rate can ever change, though, so if you use this solution, you're pretty much stuck with the rate you pick unless you rewrite the handling entirely (for example to the one below).
If it varies how quickly people can get resources, you'll need to update the data periodically. A cronjob/scheduled task would work well to make sure everyone is updated, but in some situations, it might be better to simply measure how long it's been since you've updated each user's resources, and then update them on every page load they make while logged in by multiplying the time they've been away by the rate at which they gain resources - that way, you avoid updating until you actually need the new value.
For a Travian like resource management you need to keep track when you updated the users resources for the last time. If you read the resource values (for a page refresh or something), you need to add the amount of resources gained since the 'last update time' (depending on the amount of resources fields and boni the user gets) and send that value to the browser. You could also the let browser script calculate these amounts.
You might to consider caching all resource amounts somehow, since these values are required a lot, improving the communication with your database.
If a user finishes building a resource field, uses the market, builds a structure, etc you need to update the amount of resources (and the 'last update time'), because you cannot keep track on these kind of events simply.
By calculating the resources the database load is reduced, since you do not need to write the new values every time when the user refreshes the browser page. It is also more accurate since you have less rounding errors.
To keep the resources increasing between page refreshes you need a method as Frank Farmer described. Just embed the resource amount and the 'gain frequency' in some javascript and increase the resource amount every 'gain frequency' by one.
You can also calculate the ressources each time a page or the javascript asks. You'd need to store the last updated time.
It may be an old post but it comes up right away in Google so here's another option which is how the game I've been developing does it.
I use a client side JavaScript that uses a flash socket to get live updates from a dedicated game server running on the host.
I use the xmlsocket kit from http://devpro.it/xmlsocket/