What I currently have is a website where admins can create polls. On a poll, they can set a date on when it should end
What I'm aiming to do is on that time that the poll ends, the winner is found (alternative vote method, so a little more work than just tallying up votes) and the status of the poll is changed (this is just a bool)
My current thoughts are to create a cron task when submitting a poll, to be run at the time given. this will just run something like php app/console Poll:FindWinner --my_argument=pollGUID, and my php will find the winner of the given poll and change the status. There could be multiple polls running at once, and possibly multiple dates down the line (eg a date to end the poll on, and a date to display the winner until)
Should a poll be deleted or edited, the cron task would be changed respectively
The only other alternative that I can think of is creating a mysql scheduled event, but I don't think that's a good idea
Is there a better solution for what I'm trying to accomplish? This isn't something that will be run periodically, it must only be done at the datetime provided
I just want a approach on how to build a database with live records, so don't just downvote. I don't expect any code.
At the moment I have a MySql database with about 2 thousand users, they are are getting more though. Each player/user has several points, which are increasing or decreasing by certain actions.
My goal is that this database gets refreshed about every second and the user with more points move up and others move down... and so on
My question is, what is the best approach for this "live database" where records have to be updated every second. In MySql I can run time based actions which are executing a SQL command but this isn't the greatest way I think. Can someone suggest a good way to handle this? E.g. other Database providers like MongoDB or anything else?
EDIT
This doesn't work client side, so I can't simply push/post it into the databse due some time based events. For explanation: A user is training his character in the application. This training (to get 1 level up) takes 12 hours. After the time is elapsed the record should be updated in the database AUTOMATICALLY also if the user doesn't send a post request by his self (if the user is not logged in) other users should see the updated data in his profile.
You need to accept the fact that rankings will be stale to some extent. Your predicament is no different than any other gaming platform (or SO rankings for that matter). Business decisions were put in place and constantly get reviewed for the level of staleness. Take the leaderboards on tags here, for instance. Or the recent change that has profile pages updated a lot more frequently, versus around 4AM GMT.
Consider the use of MySQL Events. It is built-in functionality that replaces the need for cron tasks. I have 3 event-related links off my profile page if interested. You could calculate ranks on a timed schedule (your tolerance for staleness) and the users' requests for them would be fast (faster than the below from Gordon). On the con-side, they are stale.
Consider not saving (writing) rank info but rather focus just on filling in the slots of your other data. And get your rankings on the fly. As an example, see this rankings answer here from Gordon. It is dynamic, runs upon request with at least at that moment non-staleness, and would not require Events.
Know that only you should decide what is tolerable for the UX.
My application has a VIEW of another table that displays a list of all recorded high scores for the last calendar week. Each week, I'd like to increment a value in the rows of every user who is in the top X of this list to indicate that they've been in said top X however many weeks in the past. Since I'm only using a date filter and there's no sort of server side "event" firing off each week, I'm not sure of the most efficient way to register this information on a weekly basis.
One way I can think of doing so is to simply run some sort of "check" every time a user logs on, and the first user to log on each calendar week shoulders the burden of telling the server to increment the top X users of last week. While this seems a bit hokey to me, I will gladly do it with enough Internet Approval.
The main languages I'm working with are MySQL and PHP, in case it's of relevance.
You could write an update script that 1) Checks when it run last, and if that was this week, then abort. 2) Update the last-run timestamp. 3) Process the previous week. Now you can schedule this script with your operating system (either by running it with php, or by using e.g. wget to "download" the page). E.g. on linux you'd normally use cron, and task scheduler on windows. This is much better than using the first user because that can lead to race-conditions where it is hard to ensure that the processing cannot be run twice (in parallel).
You should consider running a cron job. You can run a check each time a user logs in but this really isn't ideal, especially if your system is going to scale up.
I'm looking at a system where people RSVP to a free web conference, these web conferences can get really busy with a lot of interest and there are limited places, so there has to be some way to ensure it doesn't get over-subscribed... So, we need to have a countdown from the time they book their seat to the time that their order is completed and confirmed. This way it's the first X amount of people to actually click 'Attending' that get the seats for the conference. If the user does not complete their booking their allocated seat is put back in the pool for someone else to grab...
An overview of what happens (how I see this best working)
1) User clicks "RSVP" on event, this makes an AJAX request to /rsvp/{event_id}/
This goes and stores a unique id (refered to as token in the rest of this question) along with a timestamp. It stores that token in a session too.
The user is then notified they have X amount of time (let's say 5 mins) to complete the rest of their details (name/email/d.o.b, etc)
However: before anything is put in the database, it checks if the amount of pending orders is less than the total seats available, if not (there are no seats left for the conference) then it returns saying "sorry, no seats available any more, keep checking as bookings aren't completed and more seats become available"
2) If the user fills this out in time, it stores their details in database as "attending"....
If however they fail to fill the form out in time, there is a cron job running every second and go through and delete any tokens that have a timestamp of over 5 minutes ago so they will loose their chance to attend, this 'seat' on the conference is then put back in to the pool. (They'll be notified they were unsuccessful and taken back to the first step)
All easy enough to write with simply putting records in a database, doing a COUNT(*) FROM pending_bookings WHERE conference_id = {x} and working out how many seats are either confirmed or orders pending, then subtracting from the total seats available for the conference.
But I don't feel that using MySQL would be very scalable on this - these have (and I'm sure will have again) in excess of 200k people trying to grab around 200 seats, doing a COUNT(*) for each one of those people will get pretty expensive, and we can't do any decent caching as it needs to be checking in real time how many people are in the process.
I've looked into using Amazon SimpleDB for this, just for it's deploy-and-go scalability, but I've used it before and seen that COUNT() isn't necessarily accurate (nature of it's scalability I guess) - for obvious reasons that COUNT needs to be 100% accurate, I need to be able to add records along with a timestamp to it and be able to delete records from it older than five minutes.
Seems to me that all 200k people could get a "token" to start with but only 200 can complete the token.
So two things come to mind;
1) why not keep track of "Seats Remaining" on the event itself that way you're not doing a count and over bookings cannot occur because the lock needed to update the event would prevent it ever going below zero.
2) At anytime during the persons subscription if the seats remaining drops to zero all the remaining tokens become invalid and users are "kicked" out of the signup process (being nice about it and apologizing but all seats are now full etc)
doing a COUNT(*) for each one of those people will get pretty expensive
Have you found that out, or is it a gut feeling? I'd rather benchmark that (and also whether your web servers can carry that load) than assume it on beforehand. Indexing (and properly selecting a storage engine) might help a lot here.
Furthermore, wouldn't it be a lot more user-friendly to let people register and enter their account data before registering for an event? Because now you seem to favor people who can type real quick, or who use a decent browser.
You could cache the count available in the database, and update that whenever you issue (registration started), confirm (registration finished), or revoke (over 5 minutes) a token. But if you need that kind of performance, this really isn't what a SQL database is for.
You could implement a fairly simple token broker using a heap (based on expiring-soonest). Every time a token is requested, it'd check the soonest-expiring token and see if it has expired. If so, revoke it from whoever it was assigned to, and give it to the new person. If not, tell them to try again later.
When a registration is completed, you'd have to remove that token from the heap (a relatively expensive operation, but you're only doing that 200 times). So your heap would always have tokens equal to the number of free+pending slots.
Handling even thousands-per-second of find-soonest-expiring requests against e.g., a Fibonacci heap is trivial on even modest hardware.
I didn't check, but I suspect that someone else has already solved most if not all of this problem, you may be able to download said token broker.
Sorry for misreading the question. In this case I would suggest having a cache table (yes, you can), that will be updated using a trigger (ON INSERT / DELETE) on the table where the transactions are.
When you allow a user to go into the transaction you insert his hash and expiration timestamp into that table. It triggers a trigger, that will update the cached value (i.e. value in the cache table) according to that - +1 for insert / -1 for delete.
When checking wheteher there are free seats, you check the cache table.
Would this sort of caching work? :)
Redis http://redis.io is a good fit for what you're wanting to do. You can keep a counter of available seats, store the temporary data and have it auto-expire... Super versatile.
I agree with xQbert, with each successful attendance fill out completion you just decrement the total in the database row for the event (or an adjoining table).
If your worried about reservation collisions then just decrement it at the beginning of filling out the attendance info; Then add a row to a temp table saying the person with this session id is filling out the attendance form for this event and they have 5 minutes from this timestamp to complete it. When they complete the form their entry is removed from this temp table.
Then run a cron job on this table that queries for timestamps that are less than or equal to the current time - 5 min. Any that come up will be removed from the table and their events will get a seat incremented back on it. Then when they submit the form past the 5 min timeframe it will check to make sure they still have an entry in the table before saving their seat permanently.
If they don't then check to see if there are any seats remaining. If there are then decrement the seat count and insert their reservation record. If not then give them an error that says that they didn't complete the form in the allotted time, lost their position in line and you ran out of seats.
Its simple enough and can be done all on the PHP side. You will probably want to put a JS timer on their page so that they know how much time they have left though.
It's probably worth to re-visit your design:
The remaining seats are the total number of seats minus booked seats.
This number is only valid in a certain moment of time as the number of booked seats change.
To book a seat, you're doing a transaction: A user signals that she wants to book one or multiple seats, processes the registration and then finally books (or not).
As long as this transaction has not been finished, the remaining seats number that you can calculate is probably wrong.
You have two options here:
Permissive Locking
Optimistic Locking
With the first one, each time a user starts a transaction and tells the application that X seats should be booked, these X seats are locked. It does not mean that those seats are already booked, they are just locked, so the user has enough time to complete the booking transaction. If the transaction finishes, those locked seats are booked by the user.
With the second one, Optimistic Locking the user just starts to register but nothing is locked. At the end of the booking transaction, the transaction may fail if not enough seats are available any longer.
Permissive Locking might prevent some users from subscribing before they start a transaction - even if they could a day later - Optimistic Locking might prevent some users from subscribing at the end of the transaction.
You need to find out what's working best for your case. Normally optimistic locking is more nice to users (as only some fail at the end), however permissive locking will help you to not frustrate users at the end of a transaction. If there is always a run on the tickets, maybe permissive locking is best.
You can think about how to make things less frustrating for users by bringing usability into play. For example with optimistic locking, each page in the transaction could have a AJAX remaining counter at the top giving the current numbers of remaining seats, so you can inform users early if seats have run out. So even they might already have put something into forms, they can see how lucky (or fast enough) they are with their doing.
I would not time-limit users to sign-up btw.. That puts stress on the user. With optimistic locking and the AJAX bar users will be stressed enough if seats run out. It's just that your system does not need to care about sign-ups.
If you want to allow sign-ups with a peace of mind, you need to choose permissive locking. Then you need a timeout, but I would do this per user action, so if the user is active, the timeout will be extended with another 15 minutes until completed. I would choose a high value here to not frustrate the user of getting a timeout.
For those users who want to sign-up while all available seats are locked by signup transactions, you should offer a back-list and tell the users that currently X seats are locked with sign-ups but they might be lucky to catch another seat later. Or you allow to over-book the number of locked seats so those users are queued in case another user does not finish her transaction successfully.
BTW, the database design should reflect the procedure you define upfront, I don't think you run into real problems here as long as you know what you try to achieve. As everything is done in buisiness transactions you can even keep a simple count per each event of total, booked and locked seats. That's one simple query, no need for aggregation like with COUNT(*). Triggers and stored procedures can be helpful, too.
I think COUNT(*) applied to 200k records could be retrieved fairly quickly. I just tried a query on 243k records and it took well under 1 second.
But I don't think you even need COUNT, except at the final confirmation step.
I don't think you need any "cron jobs", either. That would certainly be more intensive than just querying the database.
Not to mention the fact that you don't even have to let users check how many people are attending if the event is already fully booked. Once you hit 200 attendees, you can just change another db value. The front page should not even display a clickable button if you don't have seats left.
Also, if there are plenty of seats left, why limit a user to 5 minutes? Why clean out their session with a "cron job" and force them to start all over? What if there are still 100 seats left? Instead, when 200 seats are reserved, but not all confirmed, you have the newest clickers crowd out the older ones one-by-one.
On your front page, where you have the "sign up button", it doesn't need to perform the COUNT(*) query.
You can just have it do this:
SELECT `event_status` from `events` WHERE `event` = 'this_event';
The event_status can be updated as soon as all the seats are confirmed.
Then using your PHP or whatever, you either display the button, or display "Sorry, Event is Full".
if($event_status == 'full') { echo 'Sorry, it's full; }
else { echo $press_this_button; }
Once all 200 seats are reserved, but not all are confirmed, you start kicking people out. When a new person clicks the button, you do this.
SELECT MIN(time_initiated) as time_initiated, id, sessID FROM testtime WHERE `time_initiated` <= DATE_SUB(NOW(), INTERVAL 5 MINUTE);
If that gives you any results, then you have your guy to kick off the waiting list. Just flag his session as "Expired" and if he ever does click submit, 1st check to see if the event is fully confirmed. If so, tell him sorry he was too late. If not, then repeat the above query, kick someone else off the bottom of the list, and confirm him, unless all people on the waiting list are less than 5 minutes old, in which case you can ask him to wait a few minutes and see if any spots free up.
After every successful confirmation, you can query the COUNT of confirmed guests and if the number is 200+, you change the event_status to "full" and prevent anyone else from clicking that button. They will just see the message that the even is full. You should be able to wrap those 2 queries in a transaction to ensure that multiple people cannot book at the exact same time and you end up overbooked.
That's just my 2 cents.
I'm looking for the most elegant and secure method to do the following.
I have a calendar, and groups of users.
Users can add events to specific days on the calendar, and specify how long each event lasts for.
I've had a few requests from users to add the ability for them to define that events of a specific length include a break, of a certain amount of time, or require that a specific amount of time be left between events.
For example, if event is >2 hours, include a 20min break. for each event, require 30 minutes before start of next event.
The same group that has asked for an event of >2 hours to include a 20 min break, could also require that an event >3 hours include a 30 minute break.
In the end, what the users are trying to get is an elapsed time excluding breaks calculated for them. Currently I provide them a total elapsed time, but they are looking for a running time.
However, each of these requests is different for each group. Where one group may want a 30 minute break during a 2 hour event, and another may want only 10 minutes for each 3 hour event.
I was kinda thinking I could write the functions into a php file per group, and then include that file and do the calculations via php and then return a calculated total to the user, but something about that doesn't sit right with me.
Another option is to output the groups functions to javascript, and have it run client-side, as I'm already returning the duration of the event, but where the user is part of more than one group with different rules, this seems like it could get rather messy.
I currently store the start and end time in the database, but no 'durations', and I don't think I should be storing the calculated totals in the db, because if a group decides to change their calculations, I'd need to change it throughout the db.
Is there a better way of doing this?
I would just store the variables in mysql, but I don't see how I can then say to mysql to calculate based on those variables.
I'm REALLY lost here. Any suggestions? I'm hoping somebody has done something similar and can provide some insight into the best direction.
If it helps, my table contains
eventid, user, group, startDate, startTime, endDate, endTime, type
The json for the event which I return to the user is
{"eventid":"'.$eventId.'", "user":"'.$userId.'","group":"'.$groupId.'","type":"'.$type.'","startDate":".$startDate.'","startTime":"'.$startTime.'","endDate":"'.$endDate.'","endTime":"'.$endTime.'","durationLength":"'.$duration.'", "durationHrs":"'.$durationHrs.'"}
where for example, duration length is 2.5 and duration hours is 2:30.
Store only the start time and end time for the event, and a BLOB field named notes.
I've worked on several systems that suffered from feature creep of these sorts of requirements until the code and data modeling became nothing but an unmaintainable collection of exception cases. It was a lot of work to add new permutations to the code, and typically these cases were used only once.
If you need enforcement of the rules and conditions described in the notes field, it's actually more cost-effective to hire an event coordinator instead of trying to automate everything in software. A detail-oriented human can adapt to the exception cases much more rapidly than you can adapt the code to handle them.