I am working on a web application that requires php code to run at a specific date/time.
Some hypothetical examples would be sending a user an email at 09:00 on their birthday or modifying a database entry (mySQL) at a predetermined date and time.
What would be the conventional way to implement this kind of scheduling feature?
I've seen cron-jobs been used for similar requirements but would this be feasible for a large amount of scheduled tasks?
Depends on the size of your project, but usually we don't have one script do all the cron jobs, because things need to happen at different times. Some emails might send at 9pm, some database cleanup might happen at midnight, and sessions might expire every few minutes. The best thing to do is set up separate cron jobs for each thing. There are a couple exceptions for this:
Some tasks need to run very frequently, like every 30 seconds. For that we set up a cron task which checks "ps" to see whether its own process is already running, and have it just wait 30 seconds and loop.
Some tasks make more sense to run when a queue is full. For that we trigger the task when a certain user-based script executes the 100th or 200th or 300th time. For example when a user pings 100 times without doing anything else, we log them out without using a cron task, but there is a separate cron task which checks if they've been inactive for 10 minutes.
Related
Currently, I have done the following:
I created one scheduled task which runs daily to get the Scheduled time from Mysql DB for the currentdate and store it into the .txt file
SELECT workflow_id, DATE_FORMAT(schedule_datetime,'%H:%i')TIMEONLY FROM scheduling_event
where DATE(schedule_datetime) = CURDATE()
Created one more scheduled task that runs each 5mins to check if the scheduled time present in the .txt file matches the CURRENT TIME if yes then it calls the scheduled_program.php file.
The issue here is - this is not an efficient way if nothing is scheduled on the current date. So Is there any way to create/update a dynamic scheduled task instead of running each 5mins? ie: the first scheduled task will run and take the scheduled time on the current date then it will create a task based on the scheduled time. if the day ends delete all the scheduled tasks for the day.
Note: Number of the scheduled task is not fixed. imusing Windows 10, php7.
I am trying to achieve, run a scheduled_program php file on schedule Date and TIME
It looks like you're trying to solve a problem that's not serious. I guess you don't want to waste your computer's time running a cron job every five minutes if it has nothing to do.
But here's the thing:
a cron job
that runs a php program
that does a single query
to retrieve a list of workflows
and run them
has negligible cost if you run the cron job every five minutes, or even every single minute, and there are no workflows to run.
On the other hand, debugging and troubleshooting cronjobs is hard, especially in production.
So, I respectfully suggest you keep this system as simple as you possibly can. You will have to explain it over the telephone to somebody in the middle of the night at least once. That's the unfortunate truth of scheduled tasks. The dynamic scheduled task system you propose is not simple.
I'm using laravel 5 and i have about 2000 instagram account's. And i need to check their current followers/following status every day and save them in my database.
The cron job was a perfect choice because i can say at what time of the day i want to run it, and it worked fine when i had less number of accounts.
But now my cron job often closes after i have download data for around 600 usernames. I read somewhere on internet that cron jobs are good for short running tasks that needs to be scheduled at specific times.
My tasks will grow with number of account's and currently it needs around 2 or 3 hours to finish. What is the best way to do this ?
Put some column such as 'in_process' check into your DB regarding Instagram account for which you are processing followers/status.
Run another job if one is killed. Now your job should process accounts which haven't been processed yet.
I am using Cakephp.
I want to run scheduled jobs.
Here's the situation:
User set time for some task on UI (say 1 week).
When the time is over, I want to execute some specific task. Meanwhile, the user can also change the time, and the task should be executed at updated time.
What is the best way (and reliable) to achieve this?
PS: Complexity is not the issue. But the task must always run after specific time under all circumstances.
Set the execution date in your table in a field
Set a status (pending) as well
Run a cron job that runs a CakePHP shell every X seconds or minutes, whatever you need OR create a shell that keeps running all time and check the records every X seconds in a loop.
The shell will process tasks that are configure for an execution date lower than the current date
Set the status to success or failed depending on the outcome
It's up to you how you want to handle failed tasks and if it's OK if a task executes 10secs later than configured or 10 minutes. There are multiple factors that play into this: Your interval of the cron job / query against the table. Do they have to processed in parallel? Is it OK to process them after each other? Your information is to vague.
The way to do that in CakePHP is to create a shell and run it with Cronjobs.
I need to run some tasks continuously. These tasks consist, mainly, of retrieving specific records from the DB, analyzing and saving them. This a non-trivial analysis, which might take several seconds (more than a minute, perhaps).
I do not know how frequently will new records be saved in the DB waiting for analysis (there's another cronjob for that).
Should I retrieve records one by one calling the same analysis function again once it finishes (recursively) and try to keep the cronjob running until there are no more unanalyzed records?
Or should I retrieve a fixed amount of new records on each cronjob run and call the cronjob every certain amount of minutes?
A job queue server may work well for this scenario (See ActiveMQ or MemcacheQ for example. Rather than adding the un-analyzed records directly to the database, send them to a queue for processing. Then your cron job could retrieve some items from the queue for processing, and if one job takes so long to run the cron job is triggered again, the next one will run and grab the next items in the queue.
Personally, I would have the cron job retrieve a fixed number of records for processing, just to make sure you don't get the script stuck processing for a very long time in the event new records keep getting added and the processor can't keep up. Eventually it would probably finish everything but you could end up in a situation where it continues for a very long time.
You may consider creating a lock file as well that the job can look for to see if the task processor is already running. For example when the cron job starts, check for the existence of a file (e.g. processor.lock), if it exists, exit, if not, create the file, process some records, and delete the file.
Hope that helps.
Or should I retrieve a fixed amount of new records on each cronjob run and call the cronjob every certain amount of minutes?
That. And you'll have to do some trial and error metrics first to decide an optimal fixed amount.
Of course it heavily depends on what you are actually doing, how many db intensive cron jobs you are running simultaneously and what kind of setup you have. I recently spent a day looking for a Heisenbug in a very intensive script that migrated images from db to s3 (and created a few thumbs while migrating). The problem was that due to an undocumented behaviour in our ORM the connection to the database was lost at some point, as posting to s3 + thumbs generation for certain images took a little bit more than the connection time limit. It was an ugly situation, that would probably cost more than a day to identify in a recursive do it all scheme.
You'd be better off with the safe approach, even if it means a little time lost between cron executions.
Instead of using a cron job, I would use The Fat Controller to run and repeat tasks. It is basically a daemon which can run any script or application and restart it after it finishes, optionally with a delay between runs.
You can additionally specify a timeout so that long-running scripts will be stopped. This way you don't need to care about locking, long-running processes, error process and so on. It will help to keep your business logic clean.
There's more examples and use cases on the website:
http://fat-controller.sourceforge.net/
I have a Cron Job with PHP which I want to set up on my webhost, but at the moment the script takes about 20 seconds to run with only 3 users data being refreshed. If I get a 1000 users - gonna take ages. Is there an alternative to Cron Job? Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Your cron job can be as long as you want.
The main problem for you is that you must ensure the next cron job execution is not occuring while the first one is still running. You have a lot of solutions to avoid it, basically use a semaphore.
It can be a lock file, a record in database. Your cron job should check if the previous one is finished or not. A good thing is maybe sending you an email if he cannot run because of a long previous job (this way you'll have some notice alerting you that something is maybe getting wrong) By default cron jobs with bad error dstatus on exit are outputing all the standard output to the email of the account running the job, depending on how is configured the platform you could use this behavior or build an smtp connexion on the job (or store the alert in a database table).
If you want some alternatives to cron jobs you should have a look at work queues. You can mix work queues with a cron job, or use work queue in apache-php envirronment, lot of solutions, but the main idea is to make on single queue of things that should be done, and execute them one after the other (but be careful, if you handle theses tasks very slowly you'll get a big fat waiting queue).
A cron job shouldn't have any bearing on how long it's 'job' takes to complete. If you're jobs are taking 20 seconds to complete, it's PHP's fault, not cronjob.
Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Ask your webhost.
If you want to learn about optimizing php scripts, take a look at Profiling PHP Code.