I have a php script that forks and parent calls pnctl_wait(). According to php manual pcntl_wait() should suspend execution of the current process until a child has exited. But this does not happen.
The parent process does not wait at all and executes the next line immediately.
I have tried to replicate the issue in a small sample script below
<?php
$parentpid=getmypid();
declare(ticks=1);
$apid = pcntl_fork();
if ($apid == -1) {
die('could not fork for client '.$client);
} else if ($apid) { //parent
pcntl_wait($status,WNOHANG); //Protect against Zombie children
/* Parent does not wait here */
print "PARENT $parentpid has forked $apid \n";
sleep(100);
} else { // child
$pid = getmypid();
print "CHILD $pid is sleeping\n";
sleep(40);
}
?>
You don't want the option WNOHANG here. Just use:
pcntl_wait($status);
If you pass WNOHANG pcntl_wait() will not wait for children to return. It only reports childs which have already been terminated.
The whole example should look like this:
$parentpid = getmypid();
$apid = pcntl_fork();
if ($apid == -1) {
die('could not fork for client '.$client);
} else if ($apid) {
// Parent process
print "PARENT $parentpid has forked $apid \n";
// Wait for children to return. Otherwise they
// would turn into "Zombie" processes
pcntl_wait($status);
} else {
// Child process
$pid = getmypid();
print "CHILD $pid is sleeping\n";
sleep(40);
}
Note, pcntl in php functions will not work on apache server running, So PHP don't support child processes like other languages such as fork in C++.
Related
I am running following code. It's only running parent process.
$pid = pcntl_fork();
if($pid) {
// parent process runs what is here
echo $pid;
echo "parent\n";
}
else {
// child process runs what is here
echo "child\n";
}
Output: 1553172parent
I am unable to understand, Why is not running child process.
Console script perform data import from external API. For boosting import loading performed in concurrent processes that is created by pcntl_fork command.
For communication with API cUrl is used. Communication performed by https protocol.
By some undefined reason periodically some children becomes zombie. There is no errors/warnings/notices in console and also there is not logs is written. Errors level is configured appropriately.
After investigation I suppose that there problem in curl extension since without it, with fake connection, there is no problems.
Also if run import in single process mode - there is no problems at all.
PHP: 7.2.4,
OS: Debian 9,
Curl: 7.59.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 OpenSSL/1.0.2g zlib/1.2.8 libidn/1.32 librtmp/2.3
Maybe someone encountered similar problem or know possible reasons of this strange behavior?
Pseudo code sample of child logic (main part of child showed):
while (true) {
$socket->writeRawString(Signal::MESSAGE_REQUEST_DATA);
$response = $socket->readRawString();
if (Signal::MESSAGE_TERMINATE_PROCESS === $response) {
break;
}
$response = json_decode($response, true);
if (empty($response) || empty($response['deltaId'])) {
continue;
}
$delta = $this->providerConnection->getChanges($response['deltaId']);
if(empty($delta)) {
continue;
}
$xmlReader = new \XMLReader();
$xmlReader->XML($delta);
$xmlReader->read();
$xmlReader->read();
$hasNext = true;
while ($hasNext && 'updated' !== $xmlReader->name) {
$hasNext = $xmlReader->next();
}
if ('updated' !== $xmlReader->name) {
throw new \RuntimeException('Deltas file do not contain updated date.');
}
if (strtotime($xmlReader->readString()) < $endDateTimestamp) {
$socket->writeRawString(self::SIGNAL_END_DATE_REACHED);
continue;
}
}
posix_kill(\posix_getpid(), SIGTERM);
In providerConnection->getChanges($response['deltaId']); request performed via cUrl. For work with cUrl used Php cUrl class extension
As mentioned in my comments, your problem probably is, that childprocesses that died/finished need to be collected by the parent process, or they remain as zombies.
First solution:
Install a signal handler in the parent. Something like this:
pcntl_signal(SIGCHLD, [$this, 'handleSignals']);
With a signal handler that could look like this:
/**
* #param integer $signal
*/
public function handleSignals($signal) {
switch($signal) {
case SIGCHLD:
do {
$pid = pcntl_wait($status, WNOHANG);
} while($pid > 0);
break;
default:
//Nothing to do
}
}
I normally store the pids of forked children and check them all individually with pcntl_waitpid, but this could get you going.
Second Solution:
Use a double-fork to spawn the child-processes, if the parent does not need to wait for all sub-tasks to finish. A double fork looks like this:
$pid = pcntl_fork();
if ($pid == -1) handleError();
elseif ($pid == 0) { // child
$pid = pcntl_fork();
if ($pid == -1) handleChildError();
elseif($pid == 0) { // second level child
exit(startWork()); // init will take over this process
}
// exit first level child
exit(0);
} else {
// parent, wait for first level child
pcntl_wait($pid, $status); // forked child returns almost immediatly, so blocking wait is in order
}
I give up using cUrl for my task. Today I switched to Guzzle with StreamHandler instead of cUrl and it solved all my problems.
I suppose, that due to some internal errors in cUrl, system was killing my child processes.
This is not answer to my question. It just workaround of my problem for those who may also encounter similar problem.
Topic is still open for possible suggestions/explanations.
I'm having a weird problem in PHP with symfony 1.4
I have a task that launches multiple workers, and, sometimes, I need to stop all the workers (for example, after a deployment).
I launch the task using start-stop-daemon, and I want to stop it by sending the signal SIGINT to it.
So, here is my code:
protected function execute($arguments = array(), $options = array())
{
$pid_arr = array();
$thread = $this->forkChildren($arguments, $options, $options['nb_children']);
if ($this->iAmParent())
{
declare(ticks = 1);
pcntl_signal(SIGINT, array($this, 'signalHandler'));
// Retrieve list of children PIDs
$pid_arr = $this->getChildrenPids();
// While there are still children processes
while(count($pid_arr) > 0)
{
$myId = pcntl_waitpid(-1, $status);
foreach($pid_arr as $key => $pid)
{
// If the stopped process is indeed a children of the parent process
if ($myId == $pid)
{
$this->removeChildrenPid($key);
// Recreate a child
$this->createNewChildren($arguments, $options, 1, $pid_arr);
}
}
usleep(1000000);
$pid_arr = $this->getChildrenPids();
}
}
else
$thread->run();
}
public function signalHandler($signal)
{
echo "HANDLED SIGNAL $signal\n";
foreach ($this->getChildrenPids() as $childrenPid)
{
echo "KILLING $childrenPid\n";
posix_kill($childrenPid, $signal);
}
exit();
}
What I do is quite straightforward: I fork, create N children processes, and in the parent, I add a pcntl_signal to catch the SIGINT signal. The signalHanlder function retrieves the list of children pids and sends them the same signal it just received (so SIGINT).
The problem is that the signalHandler function is never called when I send a INT signal (via kill) to the parent process. And I don't understand why!
The weird thing is that when I launch my task in cli and uses Ctrl-C, the signalHandler function is called and all the children are stopped.
So, do you understand why is this happening? Am I doing something wrong?
OK, forget about it, I found the problem just after asking the question:
I just replaced
$myId = pcntl_waitpid(-1, $status);
by
$myId = pcntl_waitpid(-1, $status, WNOHANG);
because of course, the process was hung waiting for one of the children to die.
I have a function that needs to go over around 20K rows from an array, and apply an external script to each. This is a slow process, as PHP is waiting for the script to be executed before continuing with the next row.
In order to make this process faster I was thinking on running the function in different parts, at the same time. So, for example, rows 0 to 2000 as one function, 2001 to 4000 on another one, and so on. How can I do this in a neat way? I could make different cron jobs, one for each function with different params: myFunction(0, 2000), then another cron job with myFunction(2001, 4000), etc. but that doesn't seem too clean. What's a good way of doing this?
If you'd like to execute parallel tasks in PHP, I would consider using Gearman. Another approach would be to use pcntl_fork(), but I'd prefer actual workers when it's task based.
The only waiting time you suffer is between getting the data and processing the data. Processing the data is actually completely blocking anyway (you just simply have to wait for it). You will not likely gain any benefits past increasing the number of processes to the number of cores that you have. Basically I think this means the number of processes is small so scheduling the execution of 2-8 processes doesn't sound that hideous. If you are worried about not being able to process data while retrieving data, you could in theory get your data from the database in small blocks, and then distribute the processing load between a few processes, one for each core.
I think I align more with the forking child processes approach for actually running the processing threads. There is a brilliant demonstration in the comments on the pcntl_fork doc page showing an implementation of a job daemon class
http://php.net/manual/en/function.pcntl-fork.php
<?php
declare(ticks=1);
//A very basic job daemon that you can extend to your needs.
class JobDaemon{
public $maxProcesses = 25;
protected $jobsStarted = 0;
protected $currentJobs = array();
protected $signalQueue=array();
protected $parentPID;
public function __construct(){
echo "constructed \n";
$this->parentPID = getmypid();
pcntl_signal(SIGCHLD, array($this, "childSignalHandler"));
}
/**
* Run the Daemon
*/
public function run(){
echo "Running \n";
for($i=0; $i<10000; $i++){
$jobID = rand(0,10000000000000);
while(count($this->currentJobs) >= $this->maxProcesses){
echo "Maximum children allowed, waiting...\n";
sleep(1);
}
$launched = $this->launchJob($jobID);
}
//Wait for child processes to finish before exiting here
while(count($this->currentJobs)){
echo "Waiting for current jobs to finish... \n";
sleep(1);
}
}
/**
* Launch a job from the job queue
*/
protected function launchJob($jobID){
$pid = pcntl_fork();
if($pid == -1){
//Problem launching the job
error_log('Could not launch new job, exiting');
return false;
}
else if ($pid){
// Parent process
// Sometimes you can receive a signal to the childSignalHandler function before this code executes if
// the child script executes quickly enough!
//
$this->currentJobs[$pid] = $jobID;
// In the event that a signal for this pid was caught before we get here, it will be in our signalQueue array
// So let's go ahead and process it now as if we'd just received the signal
if(isset($this->signalQueue[$pid])){
echo "found $pid in the signal queue, processing it now \n";
$this->childSignalHandler(SIGCHLD, $pid, $this->signalQueue[$pid]);
unset($this->signalQueue[$pid]);
}
}
else{
//Forked child, do your deeds....
$exitStatus = 0; //Error code if you need to or whatever
echo "Doing something fun in pid ".getmypid()."\n";
exit($exitStatus);
}
return true;
}
public function childSignalHandler($signo, $pid=null, $status=null){
//If no pid is provided, that means we're getting the signal from the system. Let's figure out
//which child process ended
if(!$pid){
$pid = pcntl_waitpid(-1, $status, WNOHANG);
}
//Make sure we get all of the exited children
while($pid > 0){
if($pid && isset($this->currentJobs[$pid])){
$exitCode = pcntl_wexitstatus($status);
if($exitCode != 0){
echo "$pid exited with status ".$exitCode."\n";
}
unset($this->currentJobs[$pid]);
}
else if($pid){
//Oh no, our job has finished before this parent process could even note that it had been launched!
//Let's make note of it and handle it when the parent process is ready for it
echo "..... Adding $pid to the signal queue ..... \n";
$this->signalQueue[$pid] = $status;
}
$pid = pcntl_waitpid(-1, $status, WNOHANG);
}
return true;
}
}
you can use "PTHREADS"
very easy to install and works great on windows
download from here -> http://windows.php.net/downloads/pecl/releases/pthreads/2.0.4/
Extract the zip file and then
move the file 'php_pthreads.dll' to php\ext\ directory.
move the file 'pthreadVC2.dll' to php\ directory.
then add this line in your 'php.ini' file:
extension=php_pthreads.dll
save the file.
you just done :-)
now lets see example of how to use it:
class ChildThread extends Thread {
public $data;
public function run() {
/* Do some expensive work */
$this->data = 'result of expensive work';
}
}
$thread = new ChildThread();
if ($thread->start()) {
/*
* Do some expensive work, while already doing other
* work in the child thread.
*/
// wait until thread is finished
$thread->join();
// we can now even access $thread->data
}
for more information about PTHREADS read php docs here:
PHP DOCS PTHREADS
if you'r using WAMP like me, then you should add 'pthreadVC2.dll' into
\wamp\bin\apache\ApacheX.X.X\bin
and also edit the 'php.ini' file (same path) and add the same line as before
extension=php_pthreads.dll
GOOD LUCK!
What you are looking for is parallel which is a succinct concurrency API for PHP 7.2+
$runtime = new \parallel\Runtime();
$future = $runtime->run(function() {
for ($i = 0; $i < 500; $i++) {
echo "*";
}
return "easy";
});
for ($i = 0; $i < 500; $i++) {
echo ".";
}
printf("\nUsing \\parallel\\Runtime is %s\n", $future->value());
Output:
.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*..*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*
Using \parallel\Runtime is easy
Have a look at pcntl_fork. This allows you to spawn child processes which can then do the separate work that you need.
Not sure if a solution for your situation but you can redirect the output of system calls to a file, thus PHP will not wait until the program is finished. Although this may result in overloading your server.
http://www.php.net/manual/en/function.exec.php - If a program is started with this function, in order for it to continue running in the background, the output of the program must be redirected to a file or another output stream. Failing to do so will cause PHP to hang until the execution of the program ends.
There's Guzzle with its concurrent requests
use GuzzleHttp\Client;
use GuzzleHttp\Promise;
$client = new Client(['base_uri' => 'http://httpbin.org/']);
$promises = [
'image' => $client->getAsync('/image'),
'png' => $client->getAsync('/image/png'),
'jpeg' => $client->getAsync('/image/jpeg'),
'webp' => $client->getAsync('/image/webp')
];
$responses = Promise\Utils::unwrap($promises);
There's the overhead of promises; but more importantly Guzzle only works with HTTP requests and it works with version 7+ and frameworks like Laravel.
I have a queue (Amazon SQS) of data that needs to be processed, and I would like to do it with multiple processes (in PHP).
I want the child workers to do something like this (pseduoish code):
while(true) {
$array = $queue->fetchNItems(10); // get 10 items
if(!count($array))
killProcess();
foreach($array as $item) {
... // process the item
$queue->remove($item);
}
sleep(2);
}
I always need 1 child process to be running, but in times of need I want to (fork?) a child process so that it can help process the queue faster.
Can someone help me with a rough PHP skeleton of what I need, or point me in the right direction?
I think I need to take a look at http://php.net/manual/en/function.pcntl-fork.php, but I'm not sure how I can use this to manage multiple processes.
When you fork a process. you make a duplicate of that process. In other words the copy (fork) contains everything the original process had (including file handles)
So how do you know if you are the parent or the forked process?
The example from the linked page shows this pretty clearly
<?php
$pid = pcntl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
// we are the parent
pcntl_wait($status); //Protect against Zombie children
} else {
// we are the child
}
?>
To extend this to what you want
<?php
$pid = pcntl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
// we are the parent
pcntl_wait($status); //Protect against Zombie children
} else {
// we are the child
while(true) {
$array = $queue->fetchNItems(10); // get 10 items
if(!count($array)) {
exit();
}
foreach($array as $item) {
... // process the item
$queue->remove($item);
}
sleep(2);
}
}
?>
This will create on forked process ( a waste in this instance ) use a loop to create multiple processes. when the child has finished exit will kill the child process. and pcntl_wait() will return allowing the parent to continue. I am not sure about php but if the parent process dies or exits it will kill the child process even if the child is not finished. hence the pcntl_wait. a more elaborate system is required if you spawn multiple children.
perhaps rather than forking you should look at the range of exec functions?
A caveat.
forking process can be wrought with problems, database handles being closed when a child exits etc. You can also kill a server with to many processes if something goes wrong. spend a lot of time playing and testing and reading.
DC
I know this is an old thread, but looked like it could use a more complete answer. This is how I generally spawn multiple processes in PHP.
A word of caution: PHP was meant to die. Meaning, the language was mean to execute for a few seconds then exit. Though, garbage cleanup in PHP has come a long way, be careful. Monitor your processes for unexpected memory consumption, or other oddities. Watch everything like a hawk for a while before you set it and forget it, and even then, still check the processes once in a while or have them automatically notify if something becomes amiss.
As I was typing this up, seemed like a good idea to slap it on github too.
When ready to run the program, I recommend, doing a tail -f on the log to see the output.
<?php
/*
* date: 27-sep-2015
* auth: robert smith
* info: run a php daemon process
* lic : MIT License (see LICENSE.txt for details)
*/
$pwd = realpath("");
$daemon = array(
"log" => $pwd."/service.log",
"errorLog" => $pwd."/service.error.log",
"pid_file" => $pwd."/",
"pid" => "",
"stdout" => NULL,
"stderr" => NULL,
"callback" => array("myProcessA", "myProcessB")
);
/*
* main (spawn new process)
*/
foreach ($daemon["callback"] as $k => &$v)
{
$pid = pcntl_fork();
if ($pid < 0)
exit("fork failed: unable to fork\n");
if ($pid == 0)
spawnChores($daemon, $v);
}
exit("fork succeeded, spawning process\n");
/*
* end main
*/
/*
* functions
*/
function spawnChores(&$daemon, &$callback)
{
// become own session
$sid = posix_setsid();
if ($sid < 0)
exit("fork failed: unable to become a session leader\n");
// set working directory as root (so files & dirs are not locked because of process)
chdir("/");
// close open parent file descriptors system STDIN, STDOUT, STDERR
fclose(STDIN);
fclose(STDOUT);
fclose(STDERR);
// setup custom file descriptors
$daemon["stdout"] = fopen($daemon["log"], "ab");
$daemon["stderr"] = fopen($daemon["errorLog"], "ab");
// publish pid
$daemon["pid"] = sprintf("%d", getmypid());
file_put_contents($daemon["pid_file"].$callback.".pid", $daemon["pid"]."\n");
// publish start message to log
fprintf($daemon["stdout"], "%s daemon %s started with pid %s\n", date("Y-M-d H:i:s"), $callback, $daemon["pid"]);
call_user_func($callback, $daemon);
// publish finish message to log
fprintf($daemon["stdout"], "%s daemon %s terminated with pid %s\n", date("Y-M-d H:i:s"), $callback, $daemon["pid"]);
exit(0);
}
function myProcessA(&$daemon)
{
$run_for_seconds = 30;
for($i=0; $i<$run_for_seconds; $i++)
{
fprintf($daemon["stdout"], "Just being a process, %s, for %d more seconds\n", __FUNCTION__, $run_for_seconds - $i);
sleep(1);
}
}
function myProcessB(&$daemon)
{
$run_for_seconds = 30;
for($i=0; $i<$run_for_seconds; $i++)
{
fprintf($daemon["stdout"], "Just being a process, %s, for %d / %d seconds\n", __FUNCTION__, $i, $run_for_seconds);
sleep(1);
}
}
?>