I have a Cron Job with PHP which I want to set up on my webhost, but at the moment the script takes about 20 seconds to run with only 3 users data being refreshed. If I get a 1000 users - gonna take ages. Is there an alternative to Cron Job? Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Your cron job can be as long as you want.
The main problem for you is that you must ensure the next cron job execution is not occuring while the first one is still running. You have a lot of solutions to avoid it, basically use a semaphore.
It can be a lock file, a record in database. Your cron job should check if the previous one is finished or not. A good thing is maybe sending you an email if he cannot run because of a long previous job (this way you'll have some notice alerting you that something is maybe getting wrong) By default cron jobs with bad error dstatus on exit are outputing all the standard output to the email of the account running the job, depending on how is configured the platform you could use this behavior or build an smtp connexion on the job (or store the alert in a database table).
If you want some alternatives to cron jobs you should have a look at work queues. You can mix work queues with a cron job, or use work queue in apache-php envirronment, lot of solutions, but the main idea is to make on single queue of things that should be done, and execute them one after the other (but be careful, if you handle theses tasks very slowly you'll get a big fat waiting queue).
A cron job shouldn't have any bearing on how long it's 'job' takes to complete. If you're jobs are taking 20 seconds to complete, it's PHP's fault, not cronjob.
Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Ask your webhost.
If you want to learn about optimizing php scripts, take a look at Profiling PHP Code.
Related
I am working on a web application that requires php code to run at a specific date/time.
Some hypothetical examples would be sending a user an email at 09:00 on their birthday or modifying a database entry (mySQL) at a predetermined date and time.
What would be the conventional way to implement this kind of scheduling feature?
I've seen cron-jobs been used for similar requirements but would this be feasible for a large amount of scheduled tasks?
Depends on the size of your project, but usually we don't have one script do all the cron jobs, because things need to happen at different times. Some emails might send at 9pm, some database cleanup might happen at midnight, and sessions might expire every few minutes. The best thing to do is set up separate cron jobs for each thing. There are a couple exceptions for this:
Some tasks need to run very frequently, like every 30 seconds. For that we set up a cron task which checks "ps" to see whether its own process is already running, and have it just wait 30 seconds and loop.
Some tasks make more sense to run when a queue is full. For that we trigger the task when a certain user-based script executes the 100th or 200th or 300th time. For example when a user pings 100 times without doing anything else, we log them out without using a cron task, but there is a separate cron task which checks if they've been inactive for 10 minutes.
For the checking of online users who doesn't log out properly but close the browser, I want to run a cron job in every 2/3 second. (I am updating database in every 10 second when logged in)
Will it harmful for server?
Cron lowest possible frequency is 1 minute so you cannot fire anything more often with it. As for overkill - it may or may not be there, but you need to review the code and load it produces yourself.
You can't run cron jobs more frequently than once a minute so this isn't possible from a cron anyways. You'd have to have a process running in a loop with sleep to achieve this, but the idea is overkill anyways - once a minute is fine.
I'm in need of a way to create dynamic, one-off cron jobs to execute tasks at different times. Ideally, I would like to achieve this using PHP, so that when a user completes a certain action, the cron job is created and scheduled for a time that is calculated based on the time that the user completes said action. At any one time, there could be multiple cron jobs scheduled at once for different times. These cron jobs also need to be deleted upon completion.
I have tried searching around for something appropriate, however haven't encountered anything that works as I need. If anybody could point me in the right direction, that would be greatly appreciated.
Cheers
A possible solution:
Setting a Cron job that calls to CronJobManager.php every second.
Just make a regular daemon that calls for the CronJobManager.php.
Create a cronjob table in your database
The cron job table should contain these basic fields: path (to php file), run_time(datetime), last run (datetime) and type (like suicidal, if as you explain you want some cron jobs to delete themselves)
Connect CronJobManager.php with the cronjob table
Every time CronJobManager.php runs (that is, every second), loads the cron jobs. Then, comparing "now"'s time with each cron job's run_time you'll get which cron jobs to run.
For example, if cron job "foo" run_time is set to 18/04/2014 22:02:01, CronJobManager will run it when reaching that moment.
Notice that if Cron jobs executing time needs a lot of time, they'll get delayed and eventually a second or two will get lost.
Now, for every cron job that needs an execution, you would execute the related php file of that cron job, indicated in the path.
This is a general idea, and of course you would have to extend it with for example cron job states (idle, running, stop, etc).
In order to delete cron jobs you would implement this feature in the cron job object.
That is: the Cron Job class, once it has executed what it had to do, it would check its type (as defined in database). If it is 'suicidal', then it would delete the database row.
UPDATE
I updated the answer but I want to note something. If what you need is several cron jobs to run at once, in a specific second with 0 delay, then you need a cron job per task out of php that runs a specific file.
To achieve this functionality, you need to have a daemon that is running all the time that checks for these dynamic jobs and launches them. The setup is a tad complicated to put together, but if you are ready for such an endeavor, you can use the project PHP Resque Scheduler.
https://github.com/chrisboulton/php-resque-scheduler
You start up a daemon that runs all the time and then you can add jobs to a dynamic queue to be executed at any specified time in the future. I think you will find this suitable to everything you are looking to do.
I need to run some tasks continuously. These tasks consist, mainly, of retrieving specific records from the DB, analyzing and saving them. This a non-trivial analysis, which might take several seconds (more than a minute, perhaps).
I do not know how frequently will new records be saved in the DB waiting for analysis (there's another cronjob for that).
Should I retrieve records one by one calling the same analysis function again once it finishes (recursively) and try to keep the cronjob running until there are no more unanalyzed records?
Or should I retrieve a fixed amount of new records on each cronjob run and call the cronjob every certain amount of minutes?
A job queue server may work well for this scenario (See ActiveMQ or MemcacheQ for example. Rather than adding the un-analyzed records directly to the database, send them to a queue for processing. Then your cron job could retrieve some items from the queue for processing, and if one job takes so long to run the cron job is triggered again, the next one will run and grab the next items in the queue.
Personally, I would have the cron job retrieve a fixed number of records for processing, just to make sure you don't get the script stuck processing for a very long time in the event new records keep getting added and the processor can't keep up. Eventually it would probably finish everything but you could end up in a situation where it continues for a very long time.
You may consider creating a lock file as well that the job can look for to see if the task processor is already running. For example when the cron job starts, check for the existence of a file (e.g. processor.lock), if it exists, exit, if not, create the file, process some records, and delete the file.
Hope that helps.
Or should I retrieve a fixed amount of new records on each cronjob run and call the cronjob every certain amount of minutes?
That. And you'll have to do some trial and error metrics first to decide an optimal fixed amount.
Of course it heavily depends on what you are actually doing, how many db intensive cron jobs you are running simultaneously and what kind of setup you have. I recently spent a day looking for a Heisenbug in a very intensive script that migrated images from db to s3 (and created a few thumbs while migrating). The problem was that due to an undocumented behaviour in our ORM the connection to the database was lost at some point, as posting to s3 + thumbs generation for certain images took a little bit more than the connection time limit. It was an ugly situation, that would probably cost more than a day to identify in a recursive do it all scheme.
You'd be better off with the safe approach, even if it means a little time lost between cron executions.
Instead of using a cron job, I would use The Fat Controller to run and repeat tasks. It is basically a daemon which can run any script or application and restart it after it finishes, optionally with a delay between runs.
You can additionally specify a timeout so that long-running scripts will be stopped. This way you don't need to care about locking, long-running processes, error process and so on. It will help to keep your business logic clean.
There's more examples and use cases on the website:
http://fat-controller.sourceforge.net/
I am using MYSQL as my database and PHP as my programming language.I wanted to run a cron job which would run until the current system date matches the "deadline(date)" column in my database table called "PROJECT".Once the dates are same an update query has to run which would change the status(field of project table) from "open" to "close".
I am not really sure if cron jobs are the best way or I could use triggers or may be something else.Also I am using Apache as my web server and my OS is windows vista.
Also which is the best way to do it? PHP scheduler or cron jobs or any other method? can anybody enlighten me?
I think your concept needs to change.
PHP cannot schedule a job, neither can MySQL. Triggers in MySQL execute when a mysql query occurs, not at a specific time. Neither
This limitation usually isn't a problem in web development. The reason is because your PHP application should control all data going in and out. Usually, this means just the HTML that displays that data, or other formats to users, or other programs.
In your case you can think about it this way. The deadline is a set date. You can treat it as data, and save it to your database. When the deadline occurs is not important, it is that the data you have sent in your database is viewed correctly.
When a request is made to your application, check if the date of the deadline is in the past, if it is, then display that the project is closed - or update that the project is closed, just before display.
There really is no reason to update data independantly of your PHP application.
Usually, the only things you want to schedule are jobs that would affect your application in terms of load, or that need to be done only once, or where concurrency or time is an issue.
In your case none of those apply.
PS: I haven't tried PHPscheduler but I can guess it isn't a true scheduler. Cron is a deamon that sleeps until a given task is due in its queue, executes the task, then sleeps till the next one is due (at least thats what it does in the current algorithm). PHP cannot do that without the sockets and fork extensions, as special setup. So PHPscheduler is most likely just checking if a date for a task has expired, on each load of a webpage (whenever PHP executes a page). This is no different then you just checking if the date on the project has expired, without the overhead of PHPScheduler.
I would always go for a cron job for anything scheduling related.
The big bonus point is that you can echo info out as well and it get's emailed to you.
You'll find once you start using cronjobs, it's hard to stop.
cron does not exist, per se, in vista, but what does exist is the standard windows scheduling manager which you can run with a command line like "php -q -f myfile.php" which will execute the php file at the given time.
you can also use a port of the cron program, there are many out there.
if it is not critical to the second, any windows scheduling application will do, just be sure to have you PHP bin path in your PATH variable for simplicity.
For Windows CRON jobs I cannot recommend PyCron enough.
While CRON and Windows Scheduled Tasks are the tried and true ways of scheduling jobs/tasks to run on a regular basis, there are use cases where having a different scheduled task in CRON/Windows can become tedious. Namely when you want to let users schedule things to run, or for instances where you prefer simplicity/maintainability/portability/etc or all of the above.
In cases where I prefer to not use CRON/Windows for scheduled tasks, I build into the application a task scheduling system. This still requires 1 CRON job or Windows Task to be scheduled. The idea is to store Job details in the database (job name, job properties, last run time, run interval, anything else that is important for your implementation). You then schedule a "Master" job in CRON or Windows which handles running all of your other jobs for you. You'll need this master job to run at least as often as your shortest interval; if you want to be able to schedule jobs that run every minute the master job needs to run every minute.
You can then launch each scheduled job in the background from PHP with minimal effort (if you want). In memory constrained systems you can monitor memory usage or keep track of the PIDs (various methods) and limit to N jobs running at a given time.
I've had a great deal of success with this method, YMMV however based on your needs and your implementation.
how about PHPscheduler..R they not better than cronjobs? I think crons would be independent of the application hence would be difficult if one has to change the host..i am not really sure though..It would be great if anyone can comment on this!! Thanks!