I'm working on a service where the user schedules his tweets so for example i want to post a tweet tomorrow at 12:30 PM. How can that be done ? is cron jobs the best way to do so ? like running a cronjob every 5 minutes and see if there are tweets to post in this interval ? Are there any alternatives ?
Running a cron job is definitely the easiest solution, however there are other approaches available, one such approach would be to use a queue like Amazons SQS
This lets you simply throw things onto a queue to be processed later, by default they are immediately added to the queue in a state ready to be processed immediately, however you can add items to the queue with a timestamp they should remain dormant until. This would be your users predefined tweet time.
You could then have a script running that is constantly listening to the queue for any new tweets that need to be sent, as soon as a queue item becomes available, it will be processed.
The downside to this is of course that it's more effort, the upsides though are that you can scale easier since you can have multiple machines processing tweets and they wont ever send out the same tweet twice (whereas if two machines are running the same cron, there's the chance they'll both send out the tweet)
Related
I came across a situation i want to trigger some code at specific time, i.e when user does booking, the freelancer must accept/reject the booking request, if he doesnt, after x duration (15* mins lets say) it would be rejected and user would get push notification. All code is done but currently im running a cronjob after each 1 minute which checks for any unresponded bookings and checks when their time (15mins, dynamic) passed so then I execute my code for afterward, it is not good i guess as its running db queuries over and over each minute.
I'm aware with laravel queue jobs as well but didnt see anything for me to run that job for a specific time only (i.e execute this job after 15mins, if it isnt responded, reject it)
have you looked at Queue delay?
https://laravel.com/docs/9.x/queues#delayed-dispatching
This sounds like what you are looking for, I would just trigger the queue and delay when they make a booking so it executes 15 minutes after.
Use scheduled tasks.
use App\Console\Commands\SendEmailsCommand;
$schedule->command('emails:send Taylor --force')->daily();
$schedule->command(SendEmailsCommand::class, ['Taylor', '--force'])->daily();
https://laravel.com/docs/9.x/scheduling#scheduling-artisan-commands
I want to setup a system for a privileged user to create a new task to run from date/time X to date/time Y saved in MySQL or SQLite? The task will send out a request to remote server via SSH and when the end date/time is up another SSH request would be sent.
What I'm not sure about is how to actually trigger the event at the start time and howto trigger the other at the end time?
Should I be polling the server somehow every 1min (sounds like a performance hit) or setup jobs in Iron.io/Amazon SQS or something else?
I noticed Amazon SQS only allows messages to queue for up to 14 days, how would that work for events weeks or months in the future?
Im not looking for code, just the idea on how it should work.
Basically there are two solutions, but maybe a hybrid version suits your problem best...
Use a queue (build into Laravel) and set up delayed jobs in the queue to be fired later on. You already mention that this might not be the best solution when a task takes months/weeks.
Use a cron job. Downside with this is that you can check once every day but that could mean a delay of 23h59m or you can check every minute but that might give you performance issues (in most cases it kind of works, but definitely not perfect).
Combining 1 & 2 might be the best solution; check in de beginning of a day whether there are tasks going to end in the coming day. If so, schedule a job in the queue to end the task at the exact time at which it should end. This gives you scalability and the possibility to create tasks that end a year after they where created.
We have a web app that uses IMAP to conditionally insert messages into users' mailboxes at user-defined times.
Each of these 'jobs' are stored in a MySQL DB with a timestamp for when the job should be run (may be months into the future). Jobs can be cancelled at anytime by the user.
The problem is that making IMAP connections is a slow process, and before we insert the message we often have to conditionally check whether there is a reply from someone in the inbox (or similar), which adds considerable processing overhead to each job.
We currently have a system where we have cron script running every minute or so that gets all the jobs from the DB that need delivering in the next X minutes. It then splits them up into batches of Z jobs, and for each batch performs an asynchronous POST request back to the same server with all the data for those Z jobs (in order to achieve 'fake' multithreading). The server then processes each batch of Z jobs that come in via HTTP.
The reason we use an async HTTP POST for multithreading and not something like pnctl_fork is so that we can add other servers and have them POST the data to those instead, and have them run the jobs rather than the current server.
So my question is - is there a better way to do this?
I appreciate work queues like beanstalkd are available to use, but do they fit with the model of having to run jobs at specific times?
Also, because we need to keep the jobs in the DB anyway (because we need to provide the users with a UI for managing the jobs), would adding a work queue in there somewhere actually be adding more overhead rather than reducing it?
I'm sure there are better ways to achieve what we need - any suggestions would be much appreciated!
We're using PHP for all this so a PHP-based/compatible solution is really what we are looking for.
Beanstalkd would be a reasonable way to do this. It has the concept of put-with-delay, so you can regularly fill the queue from your primary store with a message that will be able to be reserved, and run, in X seconds (time you want it to run - the time now).
The workers would then run as normal, connecting to the beanstalkd daemon and waiting for a new job to be reserved. It would also be a lot more efficient without the overhead of a HTTP connection. As an example, I used to post messages to Amazon SQS (by http). This could barely do 20 QPS at very most, but Beanstalkd accepted over a thousand per second with barely any effort.
Edited to add: You can't delete a job without knowing it's ID, though you could store that outside. OTOH, do users have to be able to delete jobs at any time up to the last minute? You don't have to put a job into the queue weeks or months in advance, and so you would still only have one DB-reader that ran every, say, 1 to 5 mins to put the next few jobs into the queue, and still have as many workers as you would need, with the efficiencies they can bring.
Ultimately, it depends on the number of DB read/writes that you are doing, and how the database server is able to handle them.
If what you are doing is not a problem now, and won't become so with additional load, then carry on.
I'm new to PHP, so I need some guidance as to which would be the simplest and/or elegant solution to the following problem:
I'm working on a project which has a table with as many as 500,000 records, at user specified periods, a background task must be started which will invoke a command line application on the server that does the magic, the problem is, at each 1 minute or so, I need to check on all 500,000 records(and counting) if something needs to be done.
As the title says, it is time-critical, this means that a maximum of 1 minute delay can be allowed between the time expected by the user and the time that the task is executed, of course the less delay, the better.
Thus far, I can only think of a very dirty option, have a simple utility app that runs on the server, that at each minute, will make multiple requests to the server, example:
check records between 1 and 100,000;
check records between 100,000 and 200,000;
etc. you get the point;
and the server basically starts a task for each bulk of 100,000 records or less, but it seems to me that there must be a faster approach, something similar to facebook's notification.
Additional info:
server is Windows 2008
using apache + php
EDIT 1
users have an average of 3 tasks per day at about 6-8 hours interval
more than half of the tasks can be at least 1 time per day executed at the same time[!]
Any suggestion is highly appreciated!
The easiest approach would be using a persistent task that runs the whole time and receives notification about records that need to be processed. Then it could process them immediately or, in case it needs to be processed at a certain time, it could sleep until either that time is reached or another notification arrives.
I think I gave this question more than enough time, I will stick to a utility application(that sits on the server) that will make requests to a URL accessible only from the server's IP which will spawn a new thread for each task if multiple tasks needs to be executed at the same time, it's not really scalable but it will have to do for now.
I have a few ideas about this but here is what I need to do and just wanted some second opinions really.
I am writing a small auction site in PHP/SQL, but I have come up against a hurdle.
When an item finishes, much like eBay, I need to be able to tell that it's finished and send out the emails to who has won it and who has sold it.
The only way I can think of is to schedule a piece of code to keep checking what auctions have ended but surely there is a better way?
The solution can be in multiple parts :
A script that is launched via Cron (every 5 minutes could be good, even less...). It detects the finished auction and put them in a queue.
A script, that pretty much runs continuously, and that processes items in the queue.
Note that :
You have to ensure that an auction is still open before displaying the page ! (a simple test) That way people can't join in after it closes.
For each script, you can use PHP, or any other language
Advantages :
The cron job is very fast, low on resources, and if there are a lot of auction to process, there is no risk it will be run in parallel (and then conflicts)
The queue system ensure that your system won't crash because there is too much going on... It will process the queue as fast as possible, but if it is not fast enough, the website will continue to run. You can however end up with emails being sent hours or days after the auction is closed. But the "limit" is way more predictible, and won't crash your system.
You can extend it in the future with multithreading processing of the queue, distributed processing... This is a scalable architecture.
This architecture is fun.
Additionnal informations :
Regarding the daemon script, I doesn't have to run continuously. What you can do is : at the end of the cron job, if there are items in the queue, then it checks if the other script (processing) is running. If yes then exit. If the other script is not running, it launches it...
The daemon script gets an item out of the queue and process it. At the end, if there are still items in the queue, it processes it, else, it exits.
With this system, everything is optimal and everyone loves each other !
To check if the other script is running, you can use a file and write in it "1" or "0" (= running / not running). The first script reads it, the second writes it... You can also use the database to do it. Or you can maybe use system tools or shell command...
Please be kind to share the SQL script that query the highest bidder based on the bidding end date (how to know the bidding is over) and award the product to the highest bidder
I would setup a cron job to run every 10-20-30-60 minutes etc to send out emails and update the auction details.
If you're script is fast, running it every minute or so may be alright.
Be aware that many shared hosting will only allow you to send out a certain number of emails per hour.
Do these emails need to be sent out instantly?,
I can see 2 possible problems and goals you are trying to achive:
Visual: You want that when a user browse your website, without updating or refreshing the page, it keeps updating the page so that if an audition ends, it appears something like "Audition ended, the item goes to...".
Solution: You should use Javascript and AJAX. (I assume you are already using it for countdowns or something). Make an AJAX call every 5 seconds (could be enough) and update the content.
Pratical: You want that if an audition is ended an user cannot join it. Solution: You can do it just with PHP and mysql. You could create a fields where you store the audition start timestamp and then make a simple if (time() >= ($timestamp + $duration)) {} (where $timestamp is the start of the audition and $duration is the duration of the audition) to block possible bad users trying to do it.