I'm writing a script that builds a queue of other scripts and is supposed to manage their launch. the manager script should know which child process has finished, so it can launch other scripts waiting in the queue.
I added a "& echo $!" to get the Process Id of each child process. so I have my child processes Process Ids, and for now am using system "ps" program call to find if child processes are still running or not.
the thing is that my script currently runs only in Unix-like systems. I don't know how to fetch my children's PID in windows, and my script does not parse "tasklist" command's output in windows yet.
Is there any other way to achieve this? any in-PHP solution to find if the child process is still running? a solution to start other processes (non blocking), and check if they are still running or not.
You may find Process Control interesting for Unix environments. You may also find an example of executing programs on Windows as comment in the manual, and this points me to think of COM-objects.
What you could do is create a database or file that will hold your process ids. Every process will write his pid (process id) in the file or DB.
Use this method to acquire your php pid:
getmypid();
Your supervising process will check every now and then if the process id is still running with the following:
function is_process_running($PID) {
exec("ps $PID", $ProcessState);
return(count($ProcessState) >= 2);
}
When process is stopped you can execute the next process
and for use of windows check the comment in the manual: http://no2.php.net/manual/en/book.exec.php#87943
Have you tried proc_get_status() ? In that case you may want to spawn your child processes using proc_open(). Im not sure if this is what your looking for.
Related
I am testing my python script macro.py that runs in a loop from the terminal.
My plan is to code a Laravel application so that I can setup multiple instances of macro.py running on the server and these can be started and stopped at the any time.
Is this possible without too much difficulty?
Invoke an external script the same way it is mentioned in this question. It's very important to store the process pid in any volatile memory (database, file). The retrieved pid number can be later used to stop the process using kill -9 process_pid command.
Be careful, if your python script "breaks" in the background (between your start and stop action called from application) there is a chance that the other process will retrieve the same pid number! I recommend to store also the process startup time as well as pid number. Before killing the process do the additional check of the stored startup time and kill the process only if test passed (otherwise assume that the process stopped unexpectedly and show appropriate information in the user interface).
I want to get some data from and API and save for that user in database, this actions takes random times depending on the time and sometimes it takes even 4 hours,
I am executing the script using exec and & in the background in php,
My question is that is exec safe for long running jobs, I dont know much about fork,and linux processes etc so I dont know what happened internally on CPU cores,
Here is something I found that confused me,
http://symcbean.blogspot.com/2010/02/php-and-long-running-processes.html
Can somebody tell me if I am going in right direction with exec?
will the process be killed itself after script completion?
Thanks
Well, that article is talking about process "trees" and how a child process depends of it spawning parent.
The PHP instance starts a child process (through exec or similar). If it doesn't wait for the process output, the PHP script ends (and the response is sent to the browser, for instance), but the process will sit idling, waiting for it's child process to finish.
The problem with this is that the child process (the long running 4 hours process) is not guaranteed to finish its job, before apache decides to kill its parent process (because you have too many idle processes) and, effectively, killing its children.
The article's author then gives the suggestion of using a daemon and separate the child process from the parent process.
Edit:
Answering the question you left in the comments, here's a quick explanation of the command he uses in the article
echo /usr/bin/php -q longThing.php | at now
Starting from left to right.
echo prints to Standard Output (STDOUT) the stuff you put in front of it so...
echo /usr/bin/php -q longThing.php will print to the shell /usr/bin/php -q longThing.php
| (pipeline) feeds directly the STDOUT of a previous command to the standard input (STDIN) of the next command.
at reads commands from STDIN and executes them at a specified time. at now means the command will be executed immediately.
So basically this is the same thing as running the following sequence in the shell:
at now - Opens the at prompt
/usr/bin/php -q longThing.php - The command we want to run
^D (by pressing Control+D) - To save the job
So, regarding your questions:
Will the child process be immediately killed after the PARENT PHP script ends?
No.
Will the child process be killed at all, in some future moment?
Yes. Apache takes care of that for you.
Will the child process finish its job before being killed?
Maybe. Maybe not. Apache might kill it before its done. Odds of that happening increase with the number of idle processes and with the time the process takes to finish.
Sidenote:
I think this article does point in the right direction but I dislike the idea of spawning processes directly from PHP. In fact, PHP does not have the appropriate tools for running (long and/or intensive) bg work. With PHP alone, you have little to no control over it.
I can, however, give you the solution we found for a similar problem I faced a while ago. We created a small program that would accept and queue data processing requests (about 5 mins long) and report back when the request was finished. That way we could control how many processes could be running at the same time, memory usage, number of requests by the same user, etc...
The program was actually hosted in another LAN server, which prevented memory usage spikes slowing down the webserver.
At the front-end, the user would be informed when the request was completed through long polling,
We are running a PHP Daemon which look into a queue, receives worker jobs and spawns the worker to handle it. The workers themselves acquire a lock on a specific location before proceeding.
We spawn the Daemon as nohup background processes.
This entire architecture seems to work, except when we have to kill the processes, for whatever reason. If we kill them using -9, there is no way to trap it in the worker process and release the locks before dying.
If we use anything less than -9 (like TERM or HUP), it doesn't seem to be received by either the daemon or the worker processes.
Has anybody solved this problem in a better way?
(ps: BTW, Due to other considerations, we may not be able to change our language of implementation, so please only consider PHP based solutions)
I had related problems once too. Let me explain. I had a php 'daemon' that worked like a downloader. It accessed feeds periodically and downloads (laaaarge) content from the net. The daemon had to be stopped at a certain time, lets say 0500 in the morning to prevent it from using the whole bandwith during daytime. I decided to use a cronjob to send SIGTERM to the daemon at 0500.
In the daemon I had the following code:
pcntl_signal(SIGTERM, array($this, 'signal_handler'));
where signal_handler looked like this:
public function signal_handler($signal) {
// some cleanup code
exit(1);
}
Unfortunately this did not work :|
It took me a time to find out what's going on. The first thing I figured out was that I'll have to call the method pcntl_signal_dispatch() on init to enable signal dispatching at all. Quote from the doc (comments):
If you are running PHP as CLI and as a "daemon" (i.e. in a loop), this function must be called in each loop to check if new signals are waiting dispatching.
Ok, so far, it seemed working. But I realized quickly that under certain conditions even this will not work as expected. Sometimes the daemon could only being stopped by kill -9 - as before. :|
So what's the problem?.. Answer: My program called wget to download the files via shell_exec. The problem is, that shell_exec() blocking waits until the child process has terminated. During this blocking wait no signal processing is done, the process can only being terminated using SIGKILL - what is hard. Also a problem was that child processes had to be terminated one by one as they became zombie processes after killing the father.
My solution to this was to execute the child process using proc_open() and the use stream_select() on it's output for non blocking IO.
Now it works like a charm. :) If you need further information don't hesitate to drop a comment.
Note If you are working with PHP < 5.3 then you'll have to use `
declare(ticks=1);
instead of pcntl_signal_dispatch(). You can rfer to the the documentation of pcntl_signal() for that. But if possible you should upgrade to PHP >= 5.3
The problem was solved just by adding ticks:
// tick use required as of PHP 4.3.0
declare(ticks = 1);
Leaving this alone was causing my code not to work.
*(It's unfortunate that the documentation of pcntl_signal doesn't mention it in a lot more attention grabbing way.)*
You need to catch the signal (SIGTERM). This can be achieved via the function pcntl_signal. This will give you the option to perform any necessary functions before calling exit.
I have a PHP script that spawns between 10 and 100 PHP scripts to help share the load of some calculations.
The problem is that this parent PHP script starts in the background and runs for ever, so it has to be shut down at some point. It can be an arduous task to blindly shut down PHP processes until you hit the parent.
Is there some way I can start the parent PHP process with a unique name such as "Ping loop", or some other way of recognizing it in the top interface?
I am aware of the alternative of creating a Daemon or a shutdown script using the PID of the parent. I'm asking this question hoping to avoid having to do that.
Check what the top level PHP script is using ps axuf or a similar command.
I would store value of getmypid() into a file, when parent script starts. That's also what I've seen other programs doing, e.g., apache by default stores PID of its main process in /usr/local/apache2/logs/httpd.pid.
You can use the setproctitle() functions which comes with the proctitle pecl extension. This function sets a title for your php scripts so you can easily identify them.
You can read more from http://www.php.net/manual/en/function.setproctitle.php
From the get go, let me say that I'm trying to avoid using pcntl_fork()
For this example, lets imagine that I'm trying to fork many instances of the 'dig' command line application. In reality the same script will be used for different command line apps.
Right now I'm using php exec and appending & to the command so that bash runs it in the background.
E.g
exec("dig google.com &");
exec("dig yahoo.com &");
and so on...
This successfully creates multiple processes of dig running parallel.
The problem I'm having is that the number of processes is rising steadily until the system crashes. Essentially it's a fork bomb.
I tried to combat this by checking the number of running processes using ps ax | wc -l
and only launching more if it's below X.
E.g (running on a loop)
if 80 processes are running, i'll launch another 20.
if 70 processes are running, i'll launch another 30.
The problem is, that even with this check in place, the number of processes continues to rise until the system crashes or it hits the operating systems max user processes
Can anyone give me some hints on how I can fork effectively (mass) without raping all the system resources? I can't see why this current method isn't working tbh.
Since you have a management process, I suggest you watch over the created subprocesses. Save the PID of every subscript you start:
$running[] = exec("process www.google.com & echo $!");
Where $! will return the PID of the backgrounded process, adding it to a list in PHP. Then in your management loop, just recheck if the processes are still active:
do {
foreach ($running as $i=>$pid) {
if (!posix_getpgid($pid)) {
unset($running[$i]);
// restart
}
} }
I don't think it's very elegant or reliable. pcntl_fork is often the better approach, but you din't elaborate on your actual scripts. But maybe this works in your case.
You may also want to use uptime to check system load.