* SOLUTION IN THE ANSWER BELOW *
I got a problem with limiting childs in the multifork php script... Seems like last child never ends... I'm really tired and can't find the error, could You please help? It does not end most of times...
<?php
declare(ticks = 1);
$max=5;
$child=0;
function sig_handler($signo) {
global $child;
switch ($signo) {
case SIGCHLD:
$child -= 1;
echo "[-]";
}
}
pcntl_signal(SIGCHLD, "sig_handler");
$found = array(1,2,3,4,5,6,7,8,9,10,11,12);
echo "LETS GO!\n";
foreach($found as $item){
while ($child >= $max) {
sleep(1);
}
$child++;
echo "[+]";
$pid=pcntl_fork();
if($pid){
}else{ // CHILD
sleep(rand(1,5));
echo "[~]";
exit(0);
}
}
while($child != 0){
echo "($child)";
sleep(1);
}
echo "THE END.\n"
?>
Result most times is:
[+][+][+][+][+][~][-][+][~][-][+][~][-][+][~][-][+][~][-][+][~][-][+][~][~][~][-][+] (5)[-](4)(4)[~][-](3)[~][-](2)(2)[~](2)[-](1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)(1)... etc etc ...
Seems like last child does not end at all or at least it doesnt trigger sig handler...
[+] <- just before forking - count: 12
[~] <- just before child exit - count: 12
[-] <- sig handler after child exit - count: 11
Help?
PS. The weird thing is that it sometimes does end.
Ok found the solution:
function sig_handler($signo){
global $child;
switch ($signo) {
case SIGCLD:
while( ( $pid = pcntl_wait ( $signo, WNOHANG ) ) > 0 ){
$signal = pcntl_wexitstatus ($signo);
$child -= 1;
echo "[-]";
}
break;
}
}
The problem was in the way of handling signals on UNIX systems. All sigs sent in similar time are being grouped to one. The function above fixes that problem. And not only that one... when child exits immediately, it doesn't segfault ("zend_mm_heap corrupted" error in some PHP configurations/versions) anymore. And it did in previous case.
PS. I should delete this question, but I will leave the solution for everyone experiencing similar problems as multithreading in PHP scripts is very useful for some tasks.
Related
I've got the following code:
$counter = 0;
while($currentPage <= $pages) {
sleep(0.1);
flush();
$browseNodeLookup->setPage($currentPage);
try {
$xml = $apaiIO->runOperation($browseNodeLookup);
} catch(Exception $e1) {
if($counter == 20) {
break;
}
}
if($xml) {
$all_elements = iq_parse_data($xml, $itemcount, $all_elements);
$currentPage++;
} else {
if($counter == 20) {
break;
}
}
$counter++;
}
By now it looks a little bit weird already, because I just can't get that while loop to break. The problem is, my website suddenly(without any change by myself) crashed. What I did was to just remove the while loop and the website worked again. Now the problem is, sometimes the page loads, and sometimes it doesn't.
When it doesn't the script continues for so long, that with multiple clients, the server crashes and I have to manually restart apache.
Is there anything wrong with this while loop? Did I miss something?
EDIT:
I found out the mistake, it wasn't even about my while loop(allthough the fix with counter <= 20 is also true). It was about guzzle, which makes the request behind "runOperation". I had to define a timeout, because I don't know why, but it seems that something with Amazon Product API changed, so that there could be an infinite loop in guzzle.
The fix was this:
$client = new \GuzzleHttp\Client(['timeout' => 2.0]);
I should provide the whole code next time, maybe someone would have spotted this. Thank you guys.
The most likely explanation is that your code is not actually hitting your break, because the counter gets incremented every iteration, but you are only checking for exactly 20 when it's not an actual page
The following code should be changed to change this issue:
if($counter >= 20) {
break;
}
Currently your code looks a little complicated. You can simplify it in this way:
while($currentPage <= $pages) {
sleep(0.1);
flush();
$browseNodeLookup->setPage($currentPage);
try {
$xml = $apaiIO->runOperation($browseNodeLookup);
$all_elements = iq_parse_data($xml, $itemcount, $all_elements);
} finally {
$currentPage++;
}
}
From the looks of it, you're only breaking when ($counter == 20 && !$xml). You're never advancing $currentPage if that condition isn't true, so the loop will be stuck forever.
Edit for clarity:
It's likely that $counter has passed 20 by the time that $xml is falsy, which means $currentPage isn't advancing (so your while condition is still true) but because $counter is greater than 20, your break does not get hit. Change $counter == 20 to $counter >= 20.
Also if you're trying to just advance until you either hit 20 or $totalPages, whichever is lower, your code could be simplified to something like
$pages = min(20, $pages);
while ($currentPage <= $pages) {
sleep(0.1);
flush();
$browseNodeLookup->setPage($currentPage);
try {
$xml = $apaiIO->runOperation($browseNodeLookup);
$all_elements = iq_parse_data($xml, $itemcount, $all_elements);
} catch (Exception $e1) {
// Any error handling you want to do
}
$currentPage++;
}
I have php7 CLI daemon which serially parses json with filesize over 50M. I'm trying to save every 1000 entries of parsed data using a separate process with pcntl_fork() to mysql, and for ~200k rows it works fine.
Then I get pcntl_fork(): Error 35.
I assume this is happening because mysql insertion becomes slower than parsing, which causes more and more forks to be generated until CentOS 6.3 can't handle it any more.
Is there a way to catch this error to resort to single-process parsing and saving? Or is there a way to check child process count?
Here is the solution that I did based on #Sander Visser comment. Key part is checking existing processes and resorting to same process if there are too many of them
class serialJsonReader{
const MAX_CHILD_PROCESSES = 50;
private $child_processes=[]; //will store alive child PIDs
private function flushCachedDataToStore() {
//resort to single process
if (count($this->child_processes) > self::MAX_CHILD_PROCESSES) {
$this->checkChildProcesses();
$this->storeCollectedData() //main work here
}
//use as much as possible
else {
$pid = pcntl_fork();
if (!$pid) {
$this->storeCollectedData(); //main work here
exit();
}
elseif ($pid == -1) {
die('could not fork');
}
else {
$this->child_processes[] = $pid;
$this->checkChildProcesses();
}
}
}
private function checkChildProcesses() {
if (count($this->child_processes) > self::MAX_CHILD_PROCESSES) {
foreach ($this->child_processes as $key => $pid) {
$res = pcntl_waitpid($pid, $status, WNOHANG);
// If the process has already exited
if ($res == -1 || $res > 0) {
unset($this->child_processes[$key]);
}
}
}
}
}
I'm using PCNTL to multiprocess a big script in PHP on an ubuntu server.
Here is the code ( simplified and commented )
function signalHandler($signo = null) {
$pid = posix_getpid();
switch ($signo) {
case SIGTERM:
case SIGINT:
case SIGKILL:
// a process is asked to stop (from user or father)
exit(3);
break;
case SIGCHLD:
case SIGHUP:
// ignore signals
break;
case 10: // signal user 1
// a process finished its work
exit(0);
break;
case 12: // signal user 2
// a process got an error.
exit(3);
break;
default:
// nothing
}
}
public static function run($nbProcess, $nbTasks, $taskFunc, $args) {
$pid = 0;
// there will be $nbTasks tasks to do, and no more than $nbProcess children must work at the same time
$MAX_PROCESS = $nbProcess;
$pidFather = posix_getpid();
$data = array();
pcntl_signal(SIGTERM, "signalHandler");
pcntl_signal(SIGINT, "signalHandler");
// pcntl_signal(SIGKILL, "signalHandler"); // SIGKILL can't be overloaded
pcntl_signal(SIGCHLD, "signalHandler");
pcntl_signal(SIGHUP, "signalHandler");
pcntl_signal(10, "signalHandler"); // user signal 1
pcntl_signal(12, "signalHandler"); // user signal 2
for ($indexTask = 0; $indexTask < $nbTasks ; $indexTask++) {
$pid = pcntl_fork();
// Father and new child both read code from here
if ($pid == -1) {
// log error
return false;
} elseif ($pid > 0) {
// We are in father process
// storing child id in an array
$arrayPid[$pid] = $indexTask;
} else {
// We are in child, nothing to do now
}
if ($pid == 0) {
// We are in child process
$pidChild = posix_getpid();
try {
//$taskFunc is an array containing an object, and the method to call from that object
$ret = (array) call_user_func($taskFunc, $indexTask, $args);// similar to $ret = (array) $taskFunc($indexTask, $args);
$returnArray = array(
"tasknb" => $indexTask,
"time" => $timer,
"data" => $ret,
);
} catch(Exception $e) {
// some stuff to exit child
}
$pdata = array();
array_push($pdata, $returnArray);
$data_str = serialize($pdata);
$shm_id = shmop_open($pidChild, "c", 0644, strlen($data_str));
if (!$shm_id) {
// log error
} else {
if(shmop_write($shm_id, $data_str, 0) != strlen($data_str)) {
// log error
}
}
// We are in a child and job is done. Let's exit !
posix_kill($pidChild, 10); // sending user signal 1 (OK)
pcntl_signal_dispatch();
} else {
// we are in father process,
// we check number of running children
while (count($arrayPid) >= $MAX_PROCESS) {
// There are more children than allowed
// waiting for any child to send signal
$pid = pcntl_wait($status);
// A child sent a signal !
if ($pid == -1) {
// log error
}
if (pcntl_wifexited($status)) {
$statusChild = pcntl_wexitstatus($status);
} else
$statusChild = $status;
// father ($pidFather) saw a child ($pid) exiting with status $statusChild (or $status ?)
// ^^^^ ^^^^^^
// (=3) (= random number ?)
if(isset($arrayPid[$pid])) {
// father knows this child
unset($arrayPid[$pid]);
if ($statusChild == 0 || $statusChild == 10 || $statusChild == 255) {
// Child did not report any error
$shm_id = shmop_open($pid, "a", 0, 0);
if ($shm_id === false)
// log error
else {
$shm_data = unserialize(shmop_read($shm_id, 0, shmop_size($shm_id)));
shmop_delete($shm_id);
shmop_close($shm_id);
$data = array_merge($data, $shm_data);
}
// kill (again) child
posix_kill($pid, 10);
pcntl_signal_dispatch();;
}
else {
// Child reported an error
}
}
}
}
}
}
The problem I'm facing is about the value returned by wexitstatus.
To make it simple, there is a father-process, that must create 200 threads.
He makes process one at a time, and wait for a process to finish if there are more than 8 threads actually running.
I added many logs, so I see a child finished its work.
I see it calling the line posix_kill($pidChild, 10);.
I see the signal handler is called with signal user 1 (which results in an exit(0)).
I see the father awakening, but when he gets the returned code from wexitstatus, he sees a code 3, and so thinks the child got an error, whereas it has exited with code 0 !!.
The pid is the good child's pid.
Maybe I misunderstand how signals work... Any clue ?
I found the problem.
In my application, register_shutdown_function(myFrameworkShutdownFunction) was used to shut down the script "smoothly".
So an exit(0) didn't immediately stop the child process. It first went into myFrameworkShutdownFunction, and converted the return code 0 to a code 3 (because of a misconfigured variable).
Is there something in php that can halt or let the script proceed if things are ok?
In my current scripts I do like this (for this example):
$fetch = false;
if($i == 10){
echo 'I is clear'
$fetch = true;
}
if($fetch){
//Do database work here
}
echo 'This should be seen no matter what the above';
Instead of the $fetch in there, can I do something else? I don't want to stop the entire script after that like what die() or exit does.
Here's an example that should help you:
<?php
function doLoop() {
for($i=0;$i<100;$i++) {
if($i != 50) {
continue; //It's not 50, skip it
}
//Otherwise
printf("Loop: $i");
}
}
function doBreak() {
for($i=0;$i<100;$i++) {
if($i != 49) {
continue; //It's not 49 yet, continue
} //Otherwise, break
printf("Loop: $i");
break;
}
}
doLoop();
doBreak();
?>
break; can be used to end a loop when a condition is met, while continue; can also be used to skip a certain value if a condition is not met. Using die(); would stop your whole script from executing, preventing it to call anything that comes after the die(); statement because that's how the execution of the scripts pretty much go, from the top to the end.
Well the child process may exit with error but pcntl_wifexited always return true
<?php
//Sample code :
$child_pid = pcntl_fork();
if ($child_pid === 0)
{
//This is child process
$my_var = rand(1,5);
if($my_var == 2)
{
//2 is not allowed
exit(1); //exit with error
}
if($my_var == 4)
{
//4 is unknown
i_crash_now(); //crash
}
echo 'end of child' . "\n";
exit(0); //exit standard
}
else
{
sleep(1); //waiting for child with ninja code (don't do that at home, this was made by professional ninjas)
pcntl_waitpid($child_pid, $status, WNOHANG);
var_dump(pcntl_wstopsig($status)); //in case 2 > 1, in case 4 > 255, else 0
var_dump(pcntl_wifexited($status)); //always true;
}
exit(0);
I may use the signal to find errors, but I don't get what's wrong with pcntl_wifexited().
Is this related to WNOHANG option?
I imagine the pcntl_waitpid behaves similarly to the normal waitpid call.
WNOHANG forces waitpid to return 0 if no process is terminated [it's akin to a zero timeout].
Hence, exit code will appear to be normal.