On the surface it looks very simple problem that I am facing. In an enterprise web app (LAMP stack) we need to add some time based & schedule based tasks. Some examples are
when a user logs in and has stayed active for more than 30 minutes, send them a lucky coupon.
send a newsletter to subscribers every Monday. [easily handled by a cron job]
If a user does not login for 3 days, start stalking her. [doable by cron job but ...]
deduct phone bill amount from user account on 1st working day of every month at 9.
repeat failed deduction every subsequent work day at 9 for a max of 15 retries.
I hope that give you an idea of what is going on that needs to be handled.
At the moment we have cron jobs of almost every possible situation and they are kind of working but as you can see with the above scenarios, we are forced to run those crons almost every second (bit exaggeration but almost).
To handle the issue more elegantly and better implement the ddd concepts, we are thinking to make clock ticking as first class citizen of the application.
We would like to make a simple central clock ticker class, that will emit ticks as time events every second.
The ticks will be published to the central event bus.
And all the classes that are interested to act on the tick, will subscribe to the event bus.
What I am unable to figure out yet is that this will result in making a lot of subscriber/registrant classes code to run on every tick. As this is already the case with cron, could there be a better way to handle the subscription part so that a specific subscription is notified only when it needs to be notified?
And before we even get into solving this problem the way I am proposing, is there is a better way to handle this kind of problems? The key point in this whole scenario seems to be how to trigger something X based on how much time has passed since something Y happened in the domain. I believe I am not the first one to face this issue and this problem must have been long solved already but I am unable to stumble upon any road sign pointing me to the right direction.
The way I have handled this in the past is to queue commands as soon as I know something should happen and then the scheduler will fire off the commands when the time has come.
The scheduler is simply a process that runs as a service and wakes every N milliseconds to find any commands that have passed their ScheduledTime.
For example:
The user has logged in. Queue a command for 30 minutes hence to give them a coupon. After 30 minutes, the scheduler will send the command. If the session is still active, then the command is accepted and a coupon is presented. Otherwise, it simply does nothing.
You also mention several examples that are best handled by a traditional scheduler (cron as you mentioned) and will fire off a batch command. Depending on how knowledgeable your domain is about things like newsletters, you would either issue individual commands to your domain objects or simply pull a report and run a job to send emails.
If you do handle these types of processes in your domain, then your domain should also queue the next command. A saga or process manager would be most suitable for this type of operation. E.g.
CreateNewsletter (This is the batch) -> NewsletterCreated
Accounts.Each(SendAccountNewsletter) -> AccountNewsletterSent
NewsletterCompleted (This is the batch) ->
Queue(command: CreateNewsletter, when: NextMondayAt9) (This is the next batch)
Hope that helps.
P.S. If you publish ticks on your bus, you will have a ton of noise to filter through.
Related
I came across a situation i want to trigger some code at specific time, i.e when user does booking, the freelancer must accept/reject the booking request, if he doesnt, after x duration (15* mins lets say) it would be rejected and user would get push notification. All code is done but currently im running a cronjob after each 1 minute which checks for any unresponded bookings and checks when their time (15mins, dynamic) passed so then I execute my code for afterward, it is not good i guess as its running db queuries over and over each minute.
I'm aware with laravel queue jobs as well but didnt see anything for me to run that job for a specific time only (i.e execute this job after 15mins, if it isnt responded, reject it)
have you looked at Queue delay?
https://laravel.com/docs/9.x/queues#delayed-dispatching
This sounds like what you are looking for, I would just trigger the queue and delay when they make a booking so it executes 15 minutes after.
Use scheduled tasks.
use App\Console\Commands\SendEmailsCommand;
$schedule->command('emails:send Taylor --force')->daily();
$schedule->command(SendEmailsCommand::class, ['Taylor', '--force'])->daily();
https://laravel.com/docs/9.x/scheduling#scheduling-artisan-commands
I want to setup a system for a privileged user to create a new task to run from date/time X to date/time Y saved in MySQL or SQLite? The task will send out a request to remote server via SSH and when the end date/time is up another SSH request would be sent.
What I'm not sure about is how to actually trigger the event at the start time and howto trigger the other at the end time?
Should I be polling the server somehow every 1min (sounds like a performance hit) or setup jobs in Iron.io/Amazon SQS or something else?
I noticed Amazon SQS only allows messages to queue for up to 14 days, how would that work for events weeks or months in the future?
Im not looking for code, just the idea on how it should work.
Basically there are two solutions, but maybe a hybrid version suits your problem best...
Use a queue (build into Laravel) and set up delayed jobs in the queue to be fired later on. You already mention that this might not be the best solution when a task takes months/weeks.
Use a cron job. Downside with this is that you can check once every day but that could mean a delay of 23h59m or you can check every minute but that might give you performance issues (in most cases it kind of works, but definitely not perfect).
Combining 1 & 2 might be the best solution; check in de beginning of a day whether there are tasks going to end in the coming day. If so, schedule a job in the queue to end the task at the exact time at which it should end. This gives you scalability and the possibility to create tasks that end a year after they where created.
I am working on a web-based lease management application that needs to be able to generate various reports, reminders and alerts every day, based on information in the database at any given time. Some examples of the kind of reports, reminders and alerts include:
Send a transactional email letting a set of users know that their next invoice is due in 15 days.
Send a transactional email letting a set of users know that they have 1 or more past-due invoices
Alert a property manager that a particular property is X days past due and offer to print a set of eviction documents
etc.
It seems like the simplest approach is to define a collection of scripts that execute via a set of cron jobs every morning. Each script would check for the criteria needed to trigger a specific response from the system. For instance I may have a collection of scripts like, SendInvoiceDueIn15DaysEmail.php, SendInvoicePastDue30DaysEmail.php, etc.
My primary question is, given a database filled with test data, what is the best way to simulate the passage of time, say 90 days, to ensure that the data triggers the correct set of responses each day? Some of my daily tasks need to interact with third party APIs like Mandrill, Mail Chimp, some industry-specific accounting packages, etc.
My secondary question is, if anybody has experience developing applications that center around scheduled, recurring events that happen in the future, am I on the right track here? I've already built most of the core system (user management, property management, lease management), now it's time to test the automated side of things.
For what it's worth the core of application is using Laravel 4, but I don't think is strictly a PHP question.
TL;DR How do I go about simulating the passage of time in order to check that over a 90 day period the system correctly detects a set of events and triggers an appropriate response which completed successfully?
I think the answer here is the same as in everything else you wish to test - a Clock mock. Abstract the way you check what time it is right now. Then you could create a mock clock implementation that would work much faster (or report whatever you'd like) during tests and another implementation that would simply return the true time in production.
This way you could also test other scenarios like time changes on the server or Daylight saving time.
While thinking about making browser games i met a disturbing trouble. Many games has something called 'event queue' that works like this. I have an "army" and send it to my opponent. I takes some amount of time. When my "army" gets oponent "army's" location there is a fight calculated so my "army" looses or wins. Everything is fine when me or my oponent is in the game. Some kind of ajax work can manage php script to be called and calculate fight - then save it's result to database. But how can it work if neither me, nor opponent is logged in when battle occurs? The good example of such game is Ogame when you are sending ships to another planet and it can meet it's destination when you are logged out, perform fight, get sources of your enemy and comeback, so it caluculates not only starships on planet but also amount of resources in that particular moment.
I've been thinking that i can save time of attack both attacker and his opponent in database rows and calculate the fight while any of player logs into the game, but this solution has many disadvantages. For example if there is third player that will attack before i will attack my oponent it will change hist army statistics (because he can kill part of his army) and i has many consequences. Also if none of the players would logout for a long time they would appear in game as they were not fighting and that would not be true..
Do you have any idea how to make such event queue? I guess using cron is also not good idea - it can be sometimes used to rarely and if there is thousands of players thousands of cron instances could be a killer for performance. Any concept would be appreciated - hope i described my problem good enough :)
In your database, create a "job" table (whatever contents you need, plus an EXPIRE date - which is when the job is finished) and everytime a user causes a job creation (e.g. moving ships from one cluster to another), make a new entry in this table with EXPIRE set to the time when the ships have arrived in the target cluster.
Then, everytime anyone (!) loads any site (even the startpage), the script checks for jobs in the queue which should have been done by now. If there are any jobs where EXPIRE is lower than NOW, then it does the job (or, for larger jobs, forks off a php cli process to do the job).
Assuming the case that it is 2012-12-28 12:00, the game has two players A and B, where A launches an attack on B that takes 6 hours. A job is created with EXPIRE=2012-12-28 18:00 and the content "Fleet foobar attacks B".
Case A:
Now, A logs out and nothing happens until 19:00. B logs in, and before anything other happens (this is important to ensure that "notifications" or inboxes don't appear out of sync), the script sees that a job is in the table and overdue. The script executes the jobs and creates the notification to B that he got overrun by a huge fleet from A. Job done, all is fine (except the mood of B).
Case B:
A logs out. At 18:30, an anonymous user visits the startpage. The script again looks in the jobtable and executes the job. B logs in at 19:00 and sees the notification of his loss. Job done.
The idea behind CRON-less event queue in my project is simply following:
All events have "end" time. I'm using raw UNIX time() timestamps for easy compare operations.
When user "clicks" (open any game page) queue engine fetches all events which are "ended" <= current time(), according to priority.
Events which are happened at same second are executed from high priority to low.
If there are no users in game - no one need events. Just that simple :) Game sleeps until someone login the game and roll-back event queue, as described in second step.
You can add the same functionality on any web-site, which has periodic events and decent daily site traffic.
(Actually I also made cron.php backend by demand)
What I think I want:
A language that would run persistently on a server to run recurring events that would be able to respond to html requests and give them tailored data. The scheduled events would be dynamically timed and occur frequently. The events would almost always be changes to database information, and the results would be viewed through a web application.
Why I think PHP isn't going to work for me:
Since I can only get my PHP to run when a user requests a page, or through a fixed rigid schedule, I'm having to look at the request time versus the last time it ran and fill in the gaps that should have happened. For example, if I had something that needed to occur every 15 minutes and no PHP scripts ran for several hours; then it would have to check how many were missed a run catch up to simulate what I believe should have happened in that time.
Real world examples and specifics:
I'm going to point to a really stupid example in this section because it's actually really pretty close in real-world concept to what I'm trying to explain. The online game neopets has shops that restock every so often with randomly generated goods. It restocks more if the store is emptier during every ~15 minute restock but the old items remain.
This isn't what I want exactly, but I recognized that at it's basic concept was extremely similar. I'll be extrapolating some odd additions to this system as I think it makes more sense than what I'm actually doing.
What should I do to recreate this shop but with the following requirements:
Restock times are dynamically derived from other events occurring (set time since another item sells perhaps), therefore the schedule isn't a rigid "every quarter hour" that I can run through scheduled PHP scripts. Instead it could be exactly at 5:46:23PM
The "restocking" is derived from other time specific information. I am unable to just simulate all restocks that should have occurred once a new customer checks the page.
What direction should I go?
I have no idea what I'm talking about so assume the following is shit:
I suspect I am wanting a persistent running program on the server that performs these tasks and puts the results into a database which is then accessed and delivered through php to the end user. I'm really just getting started though and I know I'm very likely making a very stupid assumption. Is there something that is both the persistent program and PHP together that I should learn?
I'm doing this as a hobby and just want to learn more so I can make more fun toys for myself.
I think you are mixing up two concerns:
A language that would run persistently on a server to run recurring events that would be able to respond to html requests and give them tailored data
You are not looking for a language to run persistently, but for a system, e.g. a JobQueue (something like Gearman). This JobQueue will do your main scheduled works. You can use PHP to get messages from that queue, trigger new jobs and inspect running jobs, etc. in addition to building the website people will interact with in PHP.
You could always use CRON and call a PHP file to do the dirty work, for the logic side.
(And by that I mean, use the PHP "system" function Documented Here to do the rescheduling commands in CRON itself)