I have a process I'd like to be able to run in the background by starting up a Gearman Client any time.
I've found success by opening up two SSH connections to my server, and in one starting the worker and in the other then running the client. This produces the desired output.
The problem is that, I'd like to have a worker constantly running in the background so I can just call up a client whenever I need to have the process done. But as soon as I close the terminal which has the worker PHP file running, a call to the client does not work - the worker seems to die.
Is there a way to have the worker run constantly in the background, so calling a new client will work without having to start up a new worker?
Thanks!
If you want a program to keep running even after its parent is dead (i.e. you've closed your terminal), you must invoke it with nohup :
nohup your-command &
Quoting the relevant Wikipedia page I linked to :
nohup is a POSIX command to ignore
the HUP (hangup) signal, enabling
the command to keep running after the
user who issues the command has logged
out.The HUP (hangup) signal is
by convention the way a terminal warns
depending processes of logout.
For another (possibly more) interesting solution, see the following article : Dæmonize Your PHP.
It points to Supervisord, which makes sures a process is still running, relaunching it if necessary.
Is there a way to have the worker run constantly in the background, so calling a new client will work without having to start up a new worker?
Supervisor!
The 2009 PHP Advent Calendar has a quick article on using Supervisor (and other tricks) to create constantly-running PHP scripts without having to deal with the daemonization process in PHP itself.
Related
Heroku supports multiple types of dyno configurations and allows us to set both a web and a worker process type for an app, for example like so:
web: vendor/bin/heroku-php-apache2 web/
worker: php worker/myworker.php
The web dyno will handle the web traffic, meanwhile the worker dyno type can be used as a background job (e.g.: to process a queue.)
The docs didn't make it clear to me how these workers function, i.e.: how do they start? behave?
In particular:
Are they run just once at deployment? Or are they run repeatedly (if so when)?
Do they have a maximum execution time?
Is it okay to use an endless loop inside of them? Or should I trigger them somehow?
If I go for a simple hello world:
myworker.php
<?php
error_log( "Hello world. This is the worker talking." );
?>
Then (after deploy or restart) heroku logs --tail --ps worker shows me only this:
2017-08-31T17:46:55.353948+00:00 heroku[worker.1]: State changed from crashed to starting
2017-08-31T17:46:57.834203+00:00 heroku[worker.1]: Starting process with command `php worker/mailer.php`
2017-08-31T17:46:58.452281+00:00 heroku[worker.1]: State changed from starting to up
2017-08-31T17:46:59.782480+00:00 heroku[worker.1]: State changed from up to crashed
2017-08-31T17:46:59.773468+00:00 heroku[worker.1]: Process exited with status 0
2017-08-31T17:46:59.697976+00:00 app[worker.1]: Hello world. This is the worker talking.
Is it the expected behavior?
(I'm not very used to using PHP from the command-line, which is what workers seem to be doing and might explain some of my confusion.)
Background: I'm trying to understand how they work for my own sake, but also to help me decide whether I should use a worker or a clock for a homemade mailing/newsletter system I'm adapting to Heroku.
You probably have already find out, but here is my experience with it for anyone looking for the answer:
Are they run just once at deployment? Or are they run repeatedly (if so when)?
They will run once deployed and keep running unless an exit command is issued or if the script returns an error. On my experience, when an error occurs, it's interrupted immediately and restarted after some time (which seems random). I'm not sure if it will restart when a clean exit is executed.
Do they have a maximum execution time?
No. But keep in mind that all dynos are restarted once per day.
Is it okay to use an endless loop inside of them? Or should I trigger them somehow?
Yes. Most of my uses involves a sleep in loop followed by calling the main script function.
I need to keep a php script running and alive on my server, the script runs and checks a DB for record, processes if needed, sleeps for 3 and then loops to the top of the script in an infinite loop. The issue is launching it, if I launch it via terminal (its running on an ubuntu system) using php script.php then if the terminal session is ended the script stops running.
So how can I go about launching the script so that it will remain running in the background.
Furthermore if I set up a cron job that runs once an hour and fires off a different script that check the primary one is still running and if not restarts it, how would I get the this checker script to check that the initial script is still running (even if its in sleep).
Any assistance will be greatly appreciated
If starting the script from the web is an option, then set time limit to "unlimited" (in the script itself):
set_time_limit(0);
and set ignore_user_abort to "true":
ignore_user_abort(true);
Then you can run the script as usual from the web, and it will run forever (until the process is killed or script exits in the usual way).
Of course, you can (and MUST) protect such a starter-script by password, e.g. by using HTTP authentication via .htaccess, so that somebody cannot start many forever-running scripts, which would lay down your server.
On checking whether another process is running, see question1, question2, or try searching "[php] check if process is running" here on StackOverflow. See also http://php.net/manual/en/refs.fileprocess.process.php.
If you want to run it from the terminal and keep it running permanently, look into GNU screen. It's a virtual terminal that keeps running in the background even when you close the terminal.
$ sudo apt-get install screen
With this, you can simply do:
$ screen php myscript.php
The script will load in a screen session which you can disconnect from, and even when you close the terminal it will keep running. To check up on it, simply call:
$ screen -x
Best part is screen is free software and is in the repo of all good distros (and other *NIX's if Linux doesn't float your boat).
Cron Job would be one solution:
More details about Cron job.
Another way to do it is to use Gearman or some other taks managers like in this post
We have a web app which allows us to monitor and control our server applications. The web pages start applications by executing a shell script to start them. The problem we have run into is that if we need to restart apache, it kills any of the processes that were started by the web app.
The web pages are PHP, and are using the exec() command to call the start scripts. The start scripts start Java apps, and and run the apps with something like this:
nohup java ... &
As mentioned, PHP is running in Apache on Linux. Is there some other switch or way to start these processes which would not have them be child processes of Apache (and killed when it stops)?
CLARIFICATION
I am more familiar with Windows than with Linux. In Windows, if you want to accomplish what we are trying add the start keyword in the shell, i.e.:
start <batchfile>
When you use start, the new shell/process can be unhooked from the one that started it. Is there a Linux equivalent to the start command?
Starting long-lasting processes by PHP sounds like asking for big trouble.
You will have problems like yours, and you will have huge security implications.
Much better solution is to have your PHP pages save their intent that something needs to be run in batch mode into database table (MySQL or PostgreSQL).
Another process (probably running under more advanced credentials than apache www user) should run as daemon and constantly check database for new stuff to do and execute necessary tasks (also it could be fired by cron every few minutes).
This way, you will kill two birds with one stone.
I wrote up how to create long running processes with php on my blog however I've got to agree with mvp that this approach is far from ideal for your purposes - not just from the point of view of privilege seperation (using a setuid program or sudo solves that easily enough).
Depending opn what you're trying to achieve here, I suspect that the additional functionality in DJ Bernsteins daemontools will be a better fit.
You could use batch(1) to start your long lasting server processes.
In shell, you could do
batch << END
java yourjava.jar
END
if you have some batch shell script file, start it with
batch -f yourbatchfile
If you can improve the Java programs, you might have them call daemon(3) at their start time, perhaps using the daemon thing from Apache.
You probably want to store the daemons' process pid somewhere (e.g. in some file or database), to be able to stop them (first with kill -TERM, then with kill -QUIT, at last with kill -KILL).
Using daemon function or Java thing is probably better than using a batch
I am trying to run my php scripts in Gearman worker code but also want to monitor
besides that if they are taking more than the expected run time ,I want to kill those scripts.Each script has to run in a timely fashion(say running every 10 minutes) and the Gearman client picks ,the script which are ready to run and send s them to Gearman worker.
I tried using the following options :
1) Tried using an independent script,a normal php script which monitors the running process.
But this normal scripts will not inform Gearman that job got killed and Gearman thinks that the job that got killed is still running.
So that made me think I have to synchronize the process of monitoring and process of running php scripts in the same worker.
Also these jobs need to be restarted and the client takes care of them.
2) I am running my php scripts using the following command :
cd /home/amehrotra/include/core/background;php $workload;(this is blocking does not go to the next line until the script finishes execution).
I tried using exec , but exec does not execute the scripts
exec ("/usr/bin/php /home/amehrotra/include/core/background/$workload >/dev/null &");
3) Tried running 2 workers ,one for running php script another for monitoring but Geraman client does not connect to two workers.
Not the coolest plan, but try to use database as central place where everything is controlled.
It will take some resources and time for your workers but it is the cost to make it manageable.
Worker will need to check for commands (stop/restart) that are assigned to him via db. and he can also save some data into db so you can see what is happening.
I've created a PHP script that reads from beanstalkd and processes the jobs. No problems there.
The last thing I've got to do is just write an init script for it, so it can run as a service.
However, this has now raised another question for me. When trying to stop the service, the one obvious way of doing it would be to try and kill the process. However, if I do that, what will happen to the job, if the PHP script was halfway through processing it? So the job was reserved, but the script never succeeded or failed (to delete or bury respectively), what happens?
My guess is that the TTR will expire, and then it gets put back to the ready queue?
And bonus 2nd question, any hints on how to better manage stopping the PHP service?
When a worker process (beanstalk client) opens up a connection with beanstalkd and reserves a job, the job will be in "reserved" state until the client issues delete/release command (or) job times out.
In case, if the worker process terminates abruptly, its connection with beanstalkd will get closed and the server will immediately release all the jobs that has been reserved using that particular connection.
Ref: http://groups.google.com/group/beanstalk-talk/browse_thread/thread/232d0cac5bebe30f?hide_quotes=no#msg_efa0109e7af4672e
Any job that runs out of time, and is not buried or touched goes back into the ready queue to be reserved.
I've posted elsewhere about using Supervisord and shell scripts to run workers. It has the advantage that most of the time, you probably don't mind waiting for a little while as jobs finish cleanly. You can have supervisord kill the bash scripts that run a worker script, and when the script itself has finished, simply exits, as it can't be restarted.
Another way is to put a highest-priority (0) message into a tube that the workers listen of, that will have the workers first delete the message, and then exit. I setup the shell scripts to check for a specific return value (from exit($val);) and then they too would exit any loop in the shell scripts.
I've used these techniques for Beanstalkd and also AWS:SQS queue runners for some time, dealing with millions of jobs per day running through the system.
If you job is too valuable to lose, you can also use pcntl to wait until the job finishes and then restart/shutdown your worker. I've managed to handle all suitable pcntl signals to release the job back to tube.