I have some background tasks in my php-project. It shoud do some job if some condition satisfied. For example: if there are some orders than not yet delivered and time left to estimate delivery is less than 15 minutes, system sends notification to courier that he's probably late.
The simpliest solution - create cron task that runs php script every minute. That script will check that condition and send notification if condition is fulfilled.
Another approach is queues. I looked at gearman and rabbitmq, but as i can see they a for another usecase. They fit if you have some client that directly sends tasks. In my case i don't have any client, its just some condition in system.
And the last solution i figured is write custom php-daemon with infinite loop. In each iteration it check condition, do the job if its satisfied and sleeps for 1 minute. But there are possible problems with memory leaks, daemon restarting, etc.
So, what the best solution to this promlem in modern php?
If you do not like to use cronjob I guess your best option is to write a daemon. I like coding in Perl and if I were you I would write a daemon in Perl.
However, If I were to write the daemon in PHP, I would run another backup script in cronjob (10 min interval for example) that will check if my php daemon is running or not and restart the daemon if not running.
If cron can do the job for you, then by all means use it. Why reinvent the wheel when there is a proven tool that does the job? I wrote some
while (1) {
// run forever scripts
}
and they ran months without errors; they processed message queues which were simple mysql tables. But if you do that, you will need a cron job that checks for the while (1) process state, you will need some locking to prevent the process being started multiple times, etc. Just go with cron.
I would make a small nodejs app for this or laravel queues:
https://laravel.com/docs/5.3/queues
It's up to you.
Related
Would appreciate some help understanding typical best practices in carrying out a series of tasks using Gearman in conjunction with PHP (among other things).
Here is the basic scenario:
A user uploads a set of image files through a web-based interface. The php code responding to the POST request generates an entry in a database for each file, mostly with null entries in the columns, queues a job for each to do analysis using Gearman, generates a status page and exits.
The Gearman worker gets a job for a file and starts a relatively long-running analysis. The result of that analysis is a set of parameters that need to be inserted back into the database record for that file.
My question is, what is the generally accepted method of doing this? Should I use a callback that will ultimately kick off a different php script that is going to do the modification, or should the worker function itself do the database modification?
Everything is currently running on the same machine; I'm planning on using Gearman for background scheduling, rather than for scaling by farming out to different machines, but in any case any of the functions could connect to the database wherever it is.
Any thoughts appreciated; just looking for some insights on how this typically gets structured and what might be considered best practice.
Are you sure you want to use Gearman? I only ask because it was the defacto PHP job server about 15 years ago but hasn't been a reliable solution for quite some time. I am not sure if things have drastically improved in the last 12 months, but last time I evaluated Gearman, it wasn't production capable.
Now, on to the questions.
what is the generally accepted method of doing this? Should I use a callback that will ultimately kick off a different php script that is going to do the modification, or should the worker function itself do the database modification?
You are going to follow this general pattern with any job queue:
Collect a unit of work. In your case, it will be 1 of the images and any information about who that image belongs to, user id, etc.
Submit the work to the job queue with this information.
Job Queue's worker process picks up the work, and starts processing it. This is where I would create records in the database as you can opt to not create them on job failure.
The job queue is going to track which jobs have completed and usually the status of completion. If you are using gearman, this is the gearmand process. You also need something pickup work and process that work, I will refer to this as the job worker. The job worker is where the concurrency happens which is what i think you were referring to when you said "kick off a different php script." You can just kick off a PHP script at an interval (with supervisord or a cronjob) for a kind of poll & fork approach. It's not the most efficient approach, but it doesn't sound like it will really matter for your applications use case. You could also use pcntl_fork or pthreads in PHP to get more control over your concurrent processes and implement a worker pool pattern, but it is much more complicated than just firing off a script. If you are interested in trying to implement some concurrency in PHP, I have a proof-of-concept job worker for beanstalkd available on GitHub that implements a worker pool with both fork and pthreads. I have also include a couple of other resources on the subject of concurrency.
Job Worker (pthreads)
Job Worker (fork)
PHP Daemon Example
PHP IPC Example
I've previously used Gearman along with supervisor to manage jobs.
In this case we are using Amazon SQS which I have spent some time trying to get my head around.
I have set up a separate micro instance from our main webserver to use as an Image processing server (purely for testing at the moment, it will be upgraded and become part of a cluster before this implementation goes live)
On this micro instance I have installed PHP and ImageMagick in order to perform the image processing.
I have also written a worker script which receives the messages from Amazon SQS.
All works perfectly, however I need this script to run over and over again in order to continuously check for messages.
I don't like the thought of running a continuous loop so have started to look at other methods with little success.
So my question is what is generally considered the best practice way to do this?
I am worried about memory since PHP wasn't really designed for this, therefore it feels like running the script for a while, then stopping and restarting it might be my best bet.
I have experience using supervisor (to ensure that gearman workers kept running) and am wondering if I could simply use that to continuously execute the simple php script over and over?
My thoughts are as follows:
Set up SQS long polling so that the script checks for 20 seconds.
Use a while loop with a 20 second sleep to keep this script running for say an hour at a time
Have all this run through supervisor. When the hour is up and the loop is complete, allow the script to exit.
Supervisor should then automatically restart it
Does this sound viable? Is there a better way? What is generally considered the best practice for receiving SQS messages in PHP?
Thanks in advance
In supervisord you can set autorestart to true to have it run your command over and over again. See: http://supervisord.org/configuration.html#program-x-section-settings
Overall, using an endless while loop is perfectly fine, PHP will free your objects correctly and keep memory in check if written correctly. It can run for years without leaks (if there's a leak, you probably created it yourself, so review your code).
How do I stop a Supervisord process without killing the program it's controlling? might be of interest to you; the OP had a similar setup, with autorestart and wanted to add graceful shutdowns to it.
I have a Cron Job with PHP which I want to set up on my webhost, but at the moment the script takes about 20 seconds to run with only 3 users data being refreshed. If I get a 1000 users - gonna take ages. Is there an alternative to Cron Job? Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Your cron job can be as long as you want.
The main problem for you is that you must ensure the next cron job execution is not occuring while the first one is still running. You have a lot of solutions to avoid it, basically use a semaphore.
It can be a lock file, a record in database. Your cron job should check if the previous one is finished or not. A good thing is maybe sending you an email if he cannot run because of a long previous job (this way you'll have some notice alerting you that something is maybe getting wrong) By default cron jobs with bad error dstatus on exit are outputing all the standard output to the email of the account running the job, depending on how is configured the platform you could use this behavior or build an smtp connexion on the job (or store the alert in a database table).
If you want some alternatives to cron jobs you should have a look at work queues. You can mix work queues with a cron job, or use work queue in apache-php envirronment, lot of solutions, but the main idea is to make on single queue of things that should be done, and execute them one after the other (but be careful, if you handle theses tasks very slowly you'll get a big fat waiting queue).
A cron job shouldn't have any bearing on how long it's 'job' takes to complete. If you're jobs are taking 20 seconds to complete, it's PHP's fault, not cronjob.
Will my web host let me run a cron job which takes, for example, 10 minutes to run?
Ask your webhost.
If you want to learn about optimizing php scripts, take a look at Profiling PHP Code.
I am using MYSQL as my database and PHP as my programming language.I wanted to run a cron job which would run until the current system date matches the "deadline(date)" column in my database table called "PROJECT".Once the dates are same an update query has to run which would change the status(field of project table) from "open" to "close".
I am not really sure if cron jobs are the best way or I could use triggers or may be something else.Also I am using Apache as my web server and my OS is windows vista.
Also which is the best way to do it? PHP scheduler or cron jobs or any other method? can anybody enlighten me?
I think your concept needs to change.
PHP cannot schedule a job, neither can MySQL. Triggers in MySQL execute when a mysql query occurs, not at a specific time. Neither
This limitation usually isn't a problem in web development. The reason is because your PHP application should control all data going in and out. Usually, this means just the HTML that displays that data, or other formats to users, or other programs.
In your case you can think about it this way. The deadline is a set date. You can treat it as data, and save it to your database. When the deadline occurs is not important, it is that the data you have sent in your database is viewed correctly.
When a request is made to your application, check if the date of the deadline is in the past, if it is, then display that the project is closed - or update that the project is closed, just before display.
There really is no reason to update data independantly of your PHP application.
Usually, the only things you want to schedule are jobs that would affect your application in terms of load, or that need to be done only once, or where concurrency or time is an issue.
In your case none of those apply.
PS: I haven't tried PHPscheduler but I can guess it isn't a true scheduler. Cron is a deamon that sleeps until a given task is due in its queue, executes the task, then sleeps till the next one is due (at least thats what it does in the current algorithm). PHP cannot do that without the sockets and fork extensions, as special setup. So PHPscheduler is most likely just checking if a date for a task has expired, on each load of a webpage (whenever PHP executes a page). This is no different then you just checking if the date on the project has expired, without the overhead of PHPScheduler.
I would always go for a cron job for anything scheduling related.
The big bonus point is that you can echo info out as well and it get's emailed to you.
You'll find once you start using cronjobs, it's hard to stop.
cron does not exist, per se, in vista, but what does exist is the standard windows scheduling manager which you can run with a command line like "php -q -f myfile.php" which will execute the php file at the given time.
you can also use a port of the cron program, there are many out there.
if it is not critical to the second, any windows scheduling application will do, just be sure to have you PHP bin path in your PATH variable for simplicity.
For Windows CRON jobs I cannot recommend PyCron enough.
While CRON and Windows Scheduled Tasks are the tried and true ways of scheduling jobs/tasks to run on a regular basis, there are use cases where having a different scheduled task in CRON/Windows can become tedious. Namely when you want to let users schedule things to run, or for instances where you prefer simplicity/maintainability/portability/etc or all of the above.
In cases where I prefer to not use CRON/Windows for scheduled tasks, I build into the application a task scheduling system. This still requires 1 CRON job or Windows Task to be scheduled. The idea is to store Job details in the database (job name, job properties, last run time, run interval, anything else that is important for your implementation). You then schedule a "Master" job in CRON or Windows which handles running all of your other jobs for you. You'll need this master job to run at least as often as your shortest interval; if you want to be able to schedule jobs that run every minute the master job needs to run every minute.
You can then launch each scheduled job in the background from PHP with minimal effort (if you want). In memory constrained systems you can monitor memory usage or keep track of the PIDs (various methods) and limit to N jobs running at a given time.
I've had a great deal of success with this method, YMMV however based on your needs and your implementation.
how about PHPscheduler..R they not better than cronjobs? I think crons would be independent of the application hence would be difficult if one has to change the host..i am not really sure though..It would be great if anyone can comment on this!! Thanks!
I have a PHP script that needs to be run at certain times every weekday. Is cron or Windows task scheduler the only way to do this?
Is there a way to set this up from within another PHP script?
Depends how exact the timing needs to be. A technique I've seen used (dubbed "poor man's cron") is to have a frequently accessed script (a site's home page, for example) fire off a task on the first page load after a certain time (kept track of in the database).
If you need any sort of guaranteed accuracy, though, cron or a Windows scheduled task is really the best option. If your host doesn't support them, it's time to get a better one.
Apart from cron or Scheduled Tasks, you can have your PHP script to always run it. The process should sleep (within a loop) until the time has reached. Then, the process could execute the remaining functions/methods. Or, you can set up another script (which will act as a daemon) to check for the time and execute the other script.
Well since the web is a pull mechanism you have to have some sort of action that will trigger a PHP script to execute. cron is an option on *nix and task scheduler on windows. You could also write your own service that has a timer but only if needed, this is common on windows services for updaters, jobs etc.
One way you could do it is in the cron task just call a php script for each action needed. Or one php script that executes other tasks. The problem with web based tasks though such as PHP is timeouts. Make sure your tasks are under 60-90 seconds. If not you might look at using python , perl or ruby or even bash scripts to do the work rather than the PHP script.
cron seems like the best option for you though. You will have to call your script with wget. There are examples here: http://www.thesitewizard.com/general/set-cron-job.shtml
For instance this runs the script everyday at 11:
30 11 * * * /usr/bin/wget http://www.example.com/cron.php
Cron, of course, is by far the best way to schedule anything on *nix.
If this is in a remote server you do not have cron access to, you can setup cron/windows scheduler on your computer, to open a web browser to the page that contains the script you wish to run
You probably want to use cron (or windows scheduled tasks).
If you really wanted, you could set up another php script to run continuously with an infinite loop (with a sleep command inside the loop, say for 30 seconds or so) and then when you reach your desired day/time execute the other script via a shell command call. While possible, I can't think if a single good reason to use this method rather than cron/scheduled tasks
You can write a long running script that runs your main script in predefined times but it will be very unnecessary, error prone, and it will basically be a "cron rewrite in phph".
Using the real cron itself will be easier and a more robust solution. If you are packaging an application, you can put a file in /etc/cron.d which contains a single cron line running your application.
You'll need to use a cron job (under Linux/Unix) or a scheduled task under Windows. You could have another script running on a continuous basis which checks the time and executes a script at a specified interval, but using the OS-supplied mechanism is easier to manage and resilient to restarts, etc.
The Uniform Server project has some good suggestions on mimicking cron in environments where cron is unacceptable. Still though, if cron is at all an option, use it.