I'm working on a basic lamp(willing to change) website , and I currently need a way to run some function on the server that runs for several hours per user, and every X hours it needs to query the mysql database to see if the value for that user has been updated, if it hasn't it need it to insert a new record in the database...I also should mention that the 'every X hours' can change per user too, and the total runtime of the function per user can also vary.
So basically I need a function that runs continuously on the server for few hours per user. What is the best way to do this? I want the site to be able to support many users (like 10000 +).
I'm willing to try new technologies for every aspect of the site, I'm still in the design phase and I was looking for some input.
I've looked at cron but not really sure how well it would work when dealing with so many users...
edit: Here is a typical scenario of events;
User presses button on the website and closes the browser.
Server starts a timer from when they pressed the button, now
the server will check if that user has pressed a different button within a given time frame (time frame can change per user), say within 30 minutes. If they didn't press the other button then the server needs to automatically insert a new record in the database.
The script will need to continue running, checking every 30 mins for say the next 5 hours.
Thank you!
Cron would work as well as you can code the page it will run. It's not a cron limitation.
The question is ambiguous btw. Maybe explaining your full scenario would help.
Meanwhile, my suggestion would be to set up a scrip that allows you to manually check what you need to check.
You definitely need the DB to be InnoDB optimized with proper indexes to be able to support 1000 plus users.
To alleviate the number of calls to the database, a common practice is to run scripts only on what you are interested (so in the case of users you would only select those who have logged on in say the past 3 hours)
That's achievable in 2 ways, a simple select statement, or by adding entries to a specific table on the login page, and remove them after the automated script has finished running.
All of this is pure theory without understanding exactly what you need to do though.
You are telling what/how you want to do, but not why you want to do it. Maybe letting us know why could lead to a different how ;)
However, what you can do is still use cron (or anything similar). The trick is to have
a last_interaction timestamp column
a maximum_interval column
a daily_runtime column
in your users database. Not optimized but you are in the design phase so you shouldn't pay too much attention to the performance aspect (except is explicitly required).
Related
I am currently working on a project that runs online tournaments. Normally admins of the site will generate brackets when it is time for the tournaments to start, but we have run into inconsistent start times, etc. and i am looking into proper ways to automate this process.
I have looked into running cronjobs every x min to check if a bracket needs to be generated but i am worried about issue when it comes to overlapping cronjobs, having to create/manage cronjobs through cpanel etc.
I was thinking about other solutions and thought it would be great if a user could load a page, the backend checks timestamps and determines if the bracket should be generated. An event is then fired/set to begin the auto-generation process elsewhere so it does not impact user load times. I just do not know the best route of going about this.
PS: I just need an idea of the direction i should be looking into so i can learn how to solve this issue i am not looking to copy and paste code. I just haven't been able to find anything. All of my search results provide cronjob examples.
EDIT
After thinking about things could using this work?
$(document).ready(function() {
$.ajax('Full Url Path Here');
})
I don't need to pass user input, or return any data i simply need a way to fire an event, it would be easy to include this only when needed via a helper class. Also i won't necessarily have to worry about users attempting to access i can restrict the route to ajax only requests and since nothing is needed/used on input or returned as output what can happen?
You could do it everytime a user loads a page (idea not tested, but theoretically possible):
1) Create a file and store the timestamp of the last time you updated the database.
2) Everytime a user loads a page, read that timestamp and check if 15 minutes passed.
3) If 15 minutes passed: Run a background script (with shell_exec?) that will do what you want and update the timestamp when it's done executing.
One obvious flaw with this system is that if you have no visitors in let's say a 30 minute frame, you will miss 2 updates. Though I guess that if you have no visitors you also have no point in generating brackets?
Ok I know the title doesn't really tell you what my problem is but I'll try it now.
I am developing a game. People can subscribe their animals for a race. That race starts at a specific time. It is a race for which ALL users can subscribe. So the calculation of which animal is first, second etc. happens in an php file that is executed, every 2mins there is a new calculation for about 1h. So there are 30 calculations. But ofc. this code is not connected to the logged in user. The logged in user can click on the LIVE button to see the current result.
Example: There is a race at 17.00 later today. 15 animals subscribed, from 4 players and they can all check how their animals are doing.
I do not want someone to post me the full code but I want to know how I should let a php code run for about 1 hour (so execute code, sleep 2min, new calculation, sleep 2min and so on) on my server or so. So it is not connected to the user.
I thought about cron jobs but that is really not the solution for this I believe.
Thank you for reading :p
Two approaches:
You use an algorithm which will always come to the same conclusion, regardless of when it is run and who runs it. You just define the starting parameters, then at any time you can calculate the result (or the intermediate result at any point in time between start and finish) when needed. So any user can at any time visit your site and the algorithm will calculate the current standings on the fly from some fixed starting condition.
Alternatively, you keep all data in a central data store and actually update the data in certain intervals; any user can request the current standings at any time and the latest data from the datastore will be used. You will still need an algorithm that has traits of the one described above, since you're likely explicitly not actually running the simulation in real time. Just every x seconds, you run your calculations again, calculating what is supposed to have changed from the last time you ran them.
In essence, any algorithm you use needs this approach. Even a "realtime" program simply keeps looping, changing values little by little from their previous state. The interval between theses changes can be arbitrarily stretched out, to the point where you calculate nothing until it becomes necessary. In the meantime, you just store all the data you need in a database.
Cron jobs are the wright way i think. Check this out when you are not so good with algorithm:How To: PHP with Cron job Maybe you have to use different cron jobs.
I'm developing a web game (js php mysql) in which one clicks a button to start an action that takes time to complete (let's say 10 hours) and when it finishes some points are added to that player's total.. The problem is that I need those points to be added even if the player is not online at the time the action finishes.. for example I need to have the rankings updated, or an email sent to the player..
I thought about a cron job checking constantly for ending actions, but I think that would kill the resources (contantly checking against actions of thousands of players..).
Is there a better solution to this problem?
Thanks for your attention!!
You can just write into your database when it's finished and when the user logs in you add the earned points to his account. You can also check with a cronjob. Even if you have millions of user this will not kill your server.
Cron is perfect for this. You could write your tasks in stored procedures, then have cron run an SQL script to call the stored procedure that would update the records of your players.
Databases are designed to work with thousands and millions of pieces of information efficiently, so I don't think the idea that it will kill system resources is a valid one unless you hosting system is really constrained already.
If you want to be safe against cheating you need to do the checking on the server anyway. If the "waiting" will happen within a Javascript on the client, one could easily decrease the remaing time.
So you need to send the job to the server (which is assumed to be safe against clock modifications) and the server will determine the end timestamp. You could store your jobs in a queue.
If you only need this information for the user himself you can just look at the queue when the user logs in. Otherwise run a cron job every minute (or so). This job will mark all jobs finished when their timestamp is in the past (and remove them from the database).
If you need more precise checking you will need to come up with an alternative server side solution that is doing this more often (e.g. a simple program polling the database every few seconds).
I'm developing a project of mine with scalability in mind and I've come to a crossroad. On my website I would like to detect if a user is online or not. And I can't quite think of the best way to handle this. The way I was thinking would be something along these lines(in psuedocode):
// SQL user table:
user {
"name": "blah blah",
"email": "derpy#derpyderp.com",
"online": false
}
So whenever the user logs in I could update his online column to true. However that would eventually lead to SQL queries happening every time a user logs in and if it happens that I get say, 10 logins per second, well, that's a lot of queries happening. Another way I figured I could do the same thing but in a different table:
// Activity table:
activity {
"user_id": 2,
"online": true
}
For some reason I believe that would lead to less memory consumption because of the separation from the user table. However I'm not sure if it would have any actual effect on performance.
So if you could bless me with your insight I would be more then grateful, thank you.
Generally speaking it's a common practice to add a column to the users table to store the lastActivity time. Anytime the user logs in, or accesses a page, store the current time in that field. If you want to know whether or not they are online, see if the last recorded time is within a certain window - say, five minutes. You can query all rows to see how many users are currently online as a result.
I wouldn't be too worried about running queries every few seconds - your server can handle it (assuming these are well-written and not very verbose).
you can use datetime for field type and don't forget to record user IP so you can track time o
Depending on how you want it to work you basically have two options:
Define a timeout after which you consider a user logged out
Use ajax/websockets/whatever to poll user
1: Timeout
This is the simpler use case. Every time the user requests a page, you update a timestamp in your database.
To find out how many users are online, you would do a query against this database and do a COUNT of users who have been active in the last N minutes.
This way you will get a relatively accurate idea of how many people are actively using the site at the moment.
2: Constant polling
This is a bit more complex to implement due to having to update the server with Ajax. Otherwise it works in a similar fashion to #1.
Whenever a user is on a page, you can keep a websocket open or do ajax requests every N seconds to the server.
This way you can get a pretty good idea of how many people have pages open on your site currently, but if a user leaves the page open in their browser and doesn't do anything, it would still count them as being online.
A slight modification to the idea would be to use a script on the client to monitor mouse movement. If the user doesn't move the mouse on your page for say 10 minutes, you would stop the polling or disconnect the websocket. This would fix the problem of showing users who are idle as being online.
I'm writing a realtime wep application, something similar to auction site. The problem is that I need a daemon script, preferrably php, that runs in background and constantly launches queries to mysql db and basing on some of criterias (time and conditions from resultsets) updates other tables. Performance of the daemon is crucial. Sample use case: we have a deal that is going to expire in 2:37 minutes. Even if nobody is watching/bidding it we need to expire it exactly in 2:37 since the time it started.
Can anybody advise a programming technology/software that performs this kind of task the best?
Thanks in advance
UPDATED: need to perform a query when a deal expires, no matter if it has ever been accessed by a user or not.
Why do you need to fire queries at time intervals? Can't you just change how your frontend works?
For example, in the "Deals" page, just show only deals that haven't expired - simplified example:
SELECT * FROM Deal WHERE NOW() <= DateTimeToExpire
Accordingly for the "Orders" page, a deal can become a placed order only if time hasn't expired yet.
Does your daemon need to trigger actions instantaneously? If you need a table containing the expired state as a column you could just compute the expire value on the fly or define a view? You could then use a daemon/cron job querying the view every 10 minutes or so if you have to send out emails or do some cleanup work etc.