Related
I have a php script that forks and parent calls pnctl_wait(). According to php manual pcntl_wait() should suspend execution of the current process until a child has exited. But this does not happen.
The parent process does not wait at all and executes the next line immediately.
I have tried to replicate the issue in a small sample script below
<?php
$parentpid=getmypid();
declare(ticks=1);
$apid = pcntl_fork();
if ($apid == -1) {
die('could not fork for client '.$client);
} else if ($apid) { //parent
pcntl_wait($status,WNOHANG); //Protect against Zombie children
/* Parent does not wait here */
print "PARENT $parentpid has forked $apid \n";
sleep(100);
} else { // child
$pid = getmypid();
print "CHILD $pid is sleeping\n";
sleep(40);
}
?>
You don't want the option WNOHANG here. Just use:
pcntl_wait($status);
If you pass WNOHANG pcntl_wait() will not wait for children to return. It only reports childs which have already been terminated.
The whole example should look like this:
$parentpid = getmypid();
$apid = pcntl_fork();
if ($apid == -1) {
die('could not fork for client '.$client);
} else if ($apid) {
// Parent process
print "PARENT $parentpid has forked $apid \n";
// Wait for children to return. Otherwise they
// would turn into "Zombie" processes
pcntl_wait($status);
} else {
// Child process
$pid = getmypid();
print "CHILD $pid is sleeping\n";
sleep(40);
}
Note, pcntl in php functions will not work on apache server running, So PHP don't support child processes like other languages such as fork in C++.
I'm having a weird problem in PHP with symfony 1.4
I have a task that launches multiple workers, and, sometimes, I need to stop all the workers (for example, after a deployment).
I launch the task using start-stop-daemon, and I want to stop it by sending the signal SIGINT to it.
So, here is my code:
protected function execute($arguments = array(), $options = array())
{
$pid_arr = array();
$thread = $this->forkChildren($arguments, $options, $options['nb_children']);
if ($this->iAmParent())
{
declare(ticks = 1);
pcntl_signal(SIGINT, array($this, 'signalHandler'));
// Retrieve list of children PIDs
$pid_arr = $this->getChildrenPids();
// While there are still children processes
while(count($pid_arr) > 0)
{
$myId = pcntl_waitpid(-1, $status);
foreach($pid_arr as $key => $pid)
{
// If the stopped process is indeed a children of the parent process
if ($myId == $pid)
{
$this->removeChildrenPid($key);
// Recreate a child
$this->createNewChildren($arguments, $options, 1, $pid_arr);
}
}
usleep(1000000);
$pid_arr = $this->getChildrenPids();
}
}
else
$thread->run();
}
public function signalHandler($signal)
{
echo "HANDLED SIGNAL $signal\n";
foreach ($this->getChildrenPids() as $childrenPid)
{
echo "KILLING $childrenPid\n";
posix_kill($childrenPid, $signal);
}
exit();
}
What I do is quite straightforward: I fork, create N children processes, and in the parent, I add a pcntl_signal to catch the SIGINT signal. The signalHanlder function retrieves the list of children pids and sends them the same signal it just received (so SIGINT).
The problem is that the signalHandler function is never called when I send a INT signal (via kill) to the parent process. And I don't understand why!
The weird thing is that when I launch my task in cli and uses Ctrl-C, the signalHandler function is called and all the children are stopped.
So, do you understand why is this happening? Am I doing something wrong?
OK, forget about it, I found the problem just after asking the question:
I just replaced
$myId = pcntl_waitpid(-1, $status);
by
$myId = pcntl_waitpid(-1, $status, WNOHANG);
because of course, the process was hung waiting for one of the children to die.
I have a function that needs to go over around 20K rows from an array, and apply an external script to each. This is a slow process, as PHP is waiting for the script to be executed before continuing with the next row.
In order to make this process faster I was thinking on running the function in different parts, at the same time. So, for example, rows 0 to 2000 as one function, 2001 to 4000 on another one, and so on. How can I do this in a neat way? I could make different cron jobs, one for each function with different params: myFunction(0, 2000), then another cron job with myFunction(2001, 4000), etc. but that doesn't seem too clean. What's a good way of doing this?
If you'd like to execute parallel tasks in PHP, I would consider using Gearman. Another approach would be to use pcntl_fork(), but I'd prefer actual workers when it's task based.
The only waiting time you suffer is between getting the data and processing the data. Processing the data is actually completely blocking anyway (you just simply have to wait for it). You will not likely gain any benefits past increasing the number of processes to the number of cores that you have. Basically I think this means the number of processes is small so scheduling the execution of 2-8 processes doesn't sound that hideous. If you are worried about not being able to process data while retrieving data, you could in theory get your data from the database in small blocks, and then distribute the processing load between a few processes, one for each core.
I think I align more with the forking child processes approach for actually running the processing threads. There is a brilliant demonstration in the comments on the pcntl_fork doc page showing an implementation of a job daemon class
http://php.net/manual/en/function.pcntl-fork.php
<?php
declare(ticks=1);
//A very basic job daemon that you can extend to your needs.
class JobDaemon{
public $maxProcesses = 25;
protected $jobsStarted = 0;
protected $currentJobs = array();
protected $signalQueue=array();
protected $parentPID;
public function __construct(){
echo "constructed \n";
$this->parentPID = getmypid();
pcntl_signal(SIGCHLD, array($this, "childSignalHandler"));
}
/**
* Run the Daemon
*/
public function run(){
echo "Running \n";
for($i=0; $i<10000; $i++){
$jobID = rand(0,10000000000000);
while(count($this->currentJobs) >= $this->maxProcesses){
echo "Maximum children allowed, waiting...\n";
sleep(1);
}
$launched = $this->launchJob($jobID);
}
//Wait for child processes to finish before exiting here
while(count($this->currentJobs)){
echo "Waiting for current jobs to finish... \n";
sleep(1);
}
}
/**
* Launch a job from the job queue
*/
protected function launchJob($jobID){
$pid = pcntl_fork();
if($pid == -1){
//Problem launching the job
error_log('Could not launch new job, exiting');
return false;
}
else if ($pid){
// Parent process
// Sometimes you can receive a signal to the childSignalHandler function before this code executes if
// the child script executes quickly enough!
//
$this->currentJobs[$pid] = $jobID;
// In the event that a signal for this pid was caught before we get here, it will be in our signalQueue array
// So let's go ahead and process it now as if we'd just received the signal
if(isset($this->signalQueue[$pid])){
echo "found $pid in the signal queue, processing it now \n";
$this->childSignalHandler(SIGCHLD, $pid, $this->signalQueue[$pid]);
unset($this->signalQueue[$pid]);
}
}
else{
//Forked child, do your deeds....
$exitStatus = 0; //Error code if you need to or whatever
echo "Doing something fun in pid ".getmypid()."\n";
exit($exitStatus);
}
return true;
}
public function childSignalHandler($signo, $pid=null, $status=null){
//If no pid is provided, that means we're getting the signal from the system. Let's figure out
//which child process ended
if(!$pid){
$pid = pcntl_waitpid(-1, $status, WNOHANG);
}
//Make sure we get all of the exited children
while($pid > 0){
if($pid && isset($this->currentJobs[$pid])){
$exitCode = pcntl_wexitstatus($status);
if($exitCode != 0){
echo "$pid exited with status ".$exitCode."\n";
}
unset($this->currentJobs[$pid]);
}
else if($pid){
//Oh no, our job has finished before this parent process could even note that it had been launched!
//Let's make note of it and handle it when the parent process is ready for it
echo "..... Adding $pid to the signal queue ..... \n";
$this->signalQueue[$pid] = $status;
}
$pid = pcntl_waitpid(-1, $status, WNOHANG);
}
return true;
}
}
you can use "PTHREADS"
very easy to install and works great on windows
download from here -> http://windows.php.net/downloads/pecl/releases/pthreads/2.0.4/
Extract the zip file and then
move the file 'php_pthreads.dll' to php\ext\ directory.
move the file 'pthreadVC2.dll' to php\ directory.
then add this line in your 'php.ini' file:
extension=php_pthreads.dll
save the file.
you just done :-)
now lets see example of how to use it:
class ChildThread extends Thread {
public $data;
public function run() {
/* Do some expensive work */
$this->data = 'result of expensive work';
}
}
$thread = new ChildThread();
if ($thread->start()) {
/*
* Do some expensive work, while already doing other
* work in the child thread.
*/
// wait until thread is finished
$thread->join();
// we can now even access $thread->data
}
for more information about PTHREADS read php docs here:
PHP DOCS PTHREADS
if you'r using WAMP like me, then you should add 'pthreadVC2.dll' into
\wamp\bin\apache\ApacheX.X.X\bin
and also edit the 'php.ini' file (same path) and add the same line as before
extension=php_pthreads.dll
GOOD LUCK!
What you are looking for is parallel which is a succinct concurrency API for PHP 7.2+
$runtime = new \parallel\Runtime();
$future = $runtime->run(function() {
for ($i = 0; $i < 500; $i++) {
echo "*";
}
return "easy";
});
for ($i = 0; $i < 500; $i++) {
echo ".";
}
printf("\nUsing \\parallel\\Runtime is %s\n", $future->value());
Output:
.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*..*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*
Using \parallel\Runtime is easy
Have a look at pcntl_fork. This allows you to spawn child processes which can then do the separate work that you need.
Not sure if a solution for your situation but you can redirect the output of system calls to a file, thus PHP will not wait until the program is finished. Although this may result in overloading your server.
http://www.php.net/manual/en/function.exec.php - If a program is started with this function, in order for it to continue running in the background, the output of the program must be redirected to a file or another output stream. Failing to do so will cause PHP to hang until the execution of the program ends.
There's Guzzle with its concurrent requests
use GuzzleHttp\Client;
use GuzzleHttp\Promise;
$client = new Client(['base_uri' => 'http://httpbin.org/']);
$promises = [
'image' => $client->getAsync('/image'),
'png' => $client->getAsync('/image/png'),
'jpeg' => $client->getAsync('/image/jpeg'),
'webp' => $client->getAsync('/image/webp')
];
$responses = Promise\Utils::unwrap($promises);
There's the overhead of promises; but more importantly Guzzle only works with HTTP requests and it works with version 7+ and frameworks like Laravel.
I have code that's somewhat like this:
($i=0; $i < 100; $i++)
{
do ($foo[$i]);
}
The above is a time intensive task, and I was hoping to be able to create a function, and call it twice like below
function wrapper($start;$end)
{
($i=$start; $i < $end; $i++)
{
do ($foo[$i]);
}
}
//have both of these run in parallel
wrapper(0,50);
wrapper(51,100);
I looked at Gearman but I cannot use it as I cannot install the gearman server (since I'm on a shared server). It seems like the way to achieve this would be by forking. I tried reading up a lot about it, but documentation and support is scant. Any help / wireframe code would be appreciated.
To define my question, how could I call wrapper() passing in arguments such that it executes in a child process. Also, its important that I be able to register a callback function.
Additional Details: PHP 5.3, running on Linux server. Script is executed by cgi-fcgi.
I think this is how I am supposed to spawn a child process, but how can I use it to spawn multiple child processes? And how do I register a callback function?
$pid = pcntl_fork();
if ( $pid == -1 ) {
// Fork failed
exit(1);
} else if ( $pid ) {
// The parent process
//should I repeat this same code here to spawn another child process?
} else {
// the child process
//can I call wrapper from here and will it run in this child process?
From "Tudor Barbu's professional blog"
(http://blog.motane.lu/2009/01/02/multithreading-in-php/)
require_once( 'Thread.php' );
// test to see if threading is available
if( ! Thread::isAvailable() ) {
die( 'Threads not supported' );
}
// function to be ran on separate threads
function paralel( $_limit, $_name ) {
for ( $index = 0; $index < $_limit; $index++ ) {
echo 'Now running thread ' . $_name . PHP_EOL;
sleep( 1 );
}
}
// create 2 thread objects
$t1 = new Thread( 'paralel' );
$t2 = new Thread( 'paralel' );
// start them
$t1->start( 10, 't1' );
$t2->start( 10, 't2' );
// keep the program running until the threads finish
while( $t1->isAlive() && $t2->isAlive() ) {
}
Download Thread.php
If you are using Linux you can take advantage of the command line and do
public function newWrapperInstance($start,$end){
exec('bash -c "exec nohup setsid php-cli yourFile.php '.$start.' '.$end.' > /dev/null 2>&1 &"');
}
This will create a new instance of PHP in the background and detach itself from the exec function in the primary thread.
Warning:
The only downside is you can't control what those threads do once they are created.
I have a queue (Amazon SQS) of data that needs to be processed, and I would like to do it with multiple processes (in PHP).
I want the child workers to do something like this (pseduoish code):
while(true) {
$array = $queue->fetchNItems(10); // get 10 items
if(!count($array))
killProcess();
foreach($array as $item) {
... // process the item
$queue->remove($item);
}
sleep(2);
}
I always need 1 child process to be running, but in times of need I want to (fork?) a child process so that it can help process the queue faster.
Can someone help me with a rough PHP skeleton of what I need, or point me in the right direction?
I think I need to take a look at http://php.net/manual/en/function.pcntl-fork.php, but I'm not sure how I can use this to manage multiple processes.
When you fork a process. you make a duplicate of that process. In other words the copy (fork) contains everything the original process had (including file handles)
So how do you know if you are the parent or the forked process?
The example from the linked page shows this pretty clearly
<?php
$pid = pcntl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
// we are the parent
pcntl_wait($status); //Protect against Zombie children
} else {
// we are the child
}
?>
To extend this to what you want
<?php
$pid = pcntl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
// we are the parent
pcntl_wait($status); //Protect against Zombie children
} else {
// we are the child
while(true) {
$array = $queue->fetchNItems(10); // get 10 items
if(!count($array)) {
exit();
}
foreach($array as $item) {
... // process the item
$queue->remove($item);
}
sleep(2);
}
}
?>
This will create on forked process ( a waste in this instance ) use a loop to create multiple processes. when the child has finished exit will kill the child process. and pcntl_wait() will return allowing the parent to continue. I am not sure about php but if the parent process dies or exits it will kill the child process even if the child is not finished. hence the pcntl_wait. a more elaborate system is required if you spawn multiple children.
perhaps rather than forking you should look at the range of exec functions?
A caveat.
forking process can be wrought with problems, database handles being closed when a child exits etc. You can also kill a server with to many processes if something goes wrong. spend a lot of time playing and testing and reading.
DC
I know this is an old thread, but looked like it could use a more complete answer. This is how I generally spawn multiple processes in PHP.
A word of caution: PHP was meant to die. Meaning, the language was mean to execute for a few seconds then exit. Though, garbage cleanup in PHP has come a long way, be careful. Monitor your processes for unexpected memory consumption, or other oddities. Watch everything like a hawk for a while before you set it and forget it, and even then, still check the processes once in a while or have them automatically notify if something becomes amiss.
As I was typing this up, seemed like a good idea to slap it on github too.
When ready to run the program, I recommend, doing a tail -f on the log to see the output.
<?php
/*
* date: 27-sep-2015
* auth: robert smith
* info: run a php daemon process
* lic : MIT License (see LICENSE.txt for details)
*/
$pwd = realpath("");
$daemon = array(
"log" => $pwd."/service.log",
"errorLog" => $pwd."/service.error.log",
"pid_file" => $pwd."/",
"pid" => "",
"stdout" => NULL,
"stderr" => NULL,
"callback" => array("myProcessA", "myProcessB")
);
/*
* main (spawn new process)
*/
foreach ($daemon["callback"] as $k => &$v)
{
$pid = pcntl_fork();
if ($pid < 0)
exit("fork failed: unable to fork\n");
if ($pid == 0)
spawnChores($daemon, $v);
}
exit("fork succeeded, spawning process\n");
/*
* end main
*/
/*
* functions
*/
function spawnChores(&$daemon, &$callback)
{
// become own session
$sid = posix_setsid();
if ($sid < 0)
exit("fork failed: unable to become a session leader\n");
// set working directory as root (so files & dirs are not locked because of process)
chdir("/");
// close open parent file descriptors system STDIN, STDOUT, STDERR
fclose(STDIN);
fclose(STDOUT);
fclose(STDERR);
// setup custom file descriptors
$daemon["stdout"] = fopen($daemon["log"], "ab");
$daemon["stderr"] = fopen($daemon["errorLog"], "ab");
// publish pid
$daemon["pid"] = sprintf("%d", getmypid());
file_put_contents($daemon["pid_file"].$callback.".pid", $daemon["pid"]."\n");
// publish start message to log
fprintf($daemon["stdout"], "%s daemon %s started with pid %s\n", date("Y-M-d H:i:s"), $callback, $daemon["pid"]);
call_user_func($callback, $daemon);
// publish finish message to log
fprintf($daemon["stdout"], "%s daemon %s terminated with pid %s\n", date("Y-M-d H:i:s"), $callback, $daemon["pid"]);
exit(0);
}
function myProcessA(&$daemon)
{
$run_for_seconds = 30;
for($i=0; $i<$run_for_seconds; $i++)
{
fprintf($daemon["stdout"], "Just being a process, %s, for %d more seconds\n", __FUNCTION__, $run_for_seconds - $i);
sleep(1);
}
}
function myProcessB(&$daemon)
{
$run_for_seconds = 30;
for($i=0; $i<$run_for_seconds; $i++)
{
fprintf($daemon["stdout"], "Just being a process, %s, for %d / %d seconds\n", __FUNCTION__, $i, $run_for_seconds);
sleep(1);
}
}
?>