PHP Pthread Multithread code slower than single thread - php

Question

I am trying to do a very simple but numerous iterations task. I choose 7 random serial numbers from an array of 324000 serial numbers and place them in another array and then search that array to see if a particular number is within it, execute another script and fwrite out how many times the looked for number is in the array.
This goes fairly fast in single thread. But when I put it in pthreads, even one single pthread running is 100x slower than single thread. The workers are not sharing any resources (i.e. the grab all info from their own folders and write info to their own folders)..fwrite bottlenecks is not the problem. The problem is with the arrays which I note below. Am I running into a cache line problem, where the arrays although they have separate variables are still sharing the same cache line? Sigh...much appreciate your help, in figuring out why the arrays are slowing it to a crawl.
<?php
class WorkerThreads extends Thread
{
private $workerId;
private $linesId;
private $linesId2;
private $c2_result;
private $traceId;
public function __construct($id,$newlines,$newlines2,$xxtrace)
{
$this->workerId = $id;
$this->linesId = (array) $newlines;
$this->linesId2 = (array) $newlines2;
$this->traceId = $xxtrace;
$this->c2_result= (array) array();
}
public function run()
{
for($h=0; $h<90; $h++) {
$fp42=fopen("/folder/".$this->workerId."/count.txt","w");
for($master=0; $master <200; $master++) {
// *******PROBLEM IS IN THE <3000 loop -very slow***********
$b=0;
for($a=0; $a<3000; $a++) {
$zex=0;
while($zex != 1) {
$this->c2_result[0]=$this->linesId[rand(0,324631)];
$this->c2_result[1]=$this->linesId[rand(0,324631)];
$this->c2_result[2]=$this->linesId[rand(0,324631)];
$this->c2_result[3]=$this->linesId[rand(0,324631)];
$this->c2_result[4]=$this->linesId[rand(0,324631)];
$this->c2_result[5]=$this->linesId[rand(0,324631)];
$this->c2_result[6]=$this->linesId[rand(0,324631)];
if(count(array_flip($this->c2_result)) != count($this->c2_result)) { //echo "duplicates\n";
$zex=0;
} else { //echo "no duplicates\n";
$zex=1;
//exit;
}
}
// *********PROBLEM here too !in_array statement, slowing down******
if(!in_array($this->linesId2[$this->traceId],$this->c2_result)) {
//fwrite($fp4,"nothere\n");
$b++;
}
}
fwrite($fp42,$b."\n");
}
fclose($fp42);
$mainfile3="/folder/".$this->workerId."/count_pthread.php";
$command="php $mainfile3 $this->workerId";
exec($command);
}
}
}
$xxTrack=0;
$lines = range(0, 324631);
for($x=0; $x<56; $x++) {
$workers = [];
// Initialize and start the threads
foreach (range(0, 8) as $i) {
$workers[$i] = new WorkerThreads($i,$lines,$lines2,$xxTrack);
$workers[$i]->start();
$xxTrack++;
}
// Let the threads come back
foreach (range(0, 8) as $i) {
$workers[$i]->join();
}
unset($workers);
}
UPDATED CODE
I was able to speed up the original code by 6x times with help from #tpunt suggestions. Most importantly what I learned is that the code is being slowed down by the calls to rand(). If I could get rid of that, then speed time would be 100x faster. array_rand,mt_rand() and shuffle() are even slower. Here is the new code:
class WorkerThreads extends Thread
{
private $workerId;
private $c2_result;
private $traceId;
private $myArray;
private $myArray2;
public function __construct($id,$xxtrace)
{
$this->workerId = $id;
$this->traceId = $xxtrace;
$c2_result=array();
}
public function run()
{
////////////////////THE WORK TO BE DONE/////////////////////////
$lines = file("/fold/considers.txt",FILE_IGNORE_NEW_LINES);
$lines2= file("/fold/considers.txt",FILE_IGNORE_NEW_LINES);
shuffle($lines2);
$fp42=fopen("/fold/".$this->workerId."/count.txt","w");
for($h=0; $h<90; $h++) {
fseek($fp42, 0);
for($master=0; $master <200; $master++) {
$b=0;
for($a=0; $a<3000; $a++) {
$zex=0;
$myArray = [];
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
while (count($myArray) !== 7) {
$myArray[rand(0,324631)] = true;
}
if (!isset($myArray[$lines2[$this->traceId]])) {
$b++;
}
}
fwrite($fp42,$b."\n");
}
$mainfile3="/newfolder/".$this->workerId."/pthread.php";
$command="php $mainfile3 $this->workerId";
exec($command);
}//END OF H LOOP
fclose($fp42);
}
}
$xxTrack=0;
$p = new Pool(5);
for($b=0; $b<56; $b++) {
$tasks[$b]= new WorkerThreads($b,$xxTrack);
$xxTrack++;
}
// Add tasks to pool queue
foreach ($tasks as $task) {
$p->submit($task);
}
// shutdown will wait for current queue to be completed
$p->shutdown();

Answer 1

Your code is just incredibly inefficient. There are also a number of problems with it - I've made a quick breakdown of some of these things below.
Firstly, you are spinning up over 500 threads (9 * 56 = 504). This is going to be very slow because threading in PHP requires a shared-nothing architecture. This means that a new instance of PHP's interpreter will need to be created for each thread you create, where all classes, interfaces, traits, functions, etc, will need to be copied over to the new interpreter instance.
Perhaps more to the point, though, is that your 3 nested for loops are performing 54 million iterations (90 * 200 * 3000). Multiply this by the 504 threads being created, and you can soon see why things are becoming sluggish. Instead, use a thread pool (see pthreads' Pool class) with a more modest amount of threads (try 8, and go from there), and cut down on the iterations being performed per thread.
Secondly, you are opening up a file 90 times per thread (so a total of 90 * 504 = 45360). You only need one file handler per thread.
Thirdly, utilising actual PHP arrays inside of Threaded objects makes them read-only. So with respect to the $this->c2_result property, the code inside of your nested while loop should not even work. Not to mention that the following check does not look for duplicates:
if(count(array_flip($this->c2_result)) != count($this->c2_result))
If you avoid casting the $this->c2_result property to an array (therefore making it a Volatile object), then the following code could instead replace your while loop:
$keys = array_rand($this->linesId, 7);
for ($i = 0; $i < 7; ++$i) {
$this->c2_result[$this->linesId[$keys[$i]]] = true;
}
By setting the values as the keys in $this->c2_result we can remove the subsequent in_array function call to search through the $this->c2_result. This is done by utilising a PHP array as a hash table, where the lookup time for a key is constant time (O(1)), rather than linear time required when searching for values (with in_array). This enables us to replace the following slow check:
if(!in_array($this->linesId2[$this->traceId],$this->c2_result))
with the following fast check:
if (!isset($this->c2_result[$this->linesId2[$this->traceId]]))
But with that said, you don't seem to be using the $this->c2_result property anywhere else. So (assuming you haven't purposefully redacted code that uses it), you could remove it altogether and simply replace the while loop at check after it with the following:
$found = false;
foreach (array_rand($this->linesId, 7) as $key) {
if ($this->linesId[$key] === $this->linesId2[$this->traceId]) {
$found = true;
break;
}
}
if (!$found) {
++$b;
}
Beyond the above, you could also look at storing the data you're collecting in-memory (as some property on the Threaded object), to prevent expensive disk writes. The results could be aggregated at the end, before shutting down the pool.
Update based up your update
You've said that the rand function is causing major slowdown. Whilst it may be part of the problem, I believe it is actually all of the code inside of your third nested for loop. The code inside there is very hot code, because it gets executed 54 million times. I suggested above that you replace the following code:
$zex=0;
while($zex != 1) {
$c2_result[0]=$lines[rand(0,324631)];
$c2_result[1]=$lines[rand(0,324631)];
$c2_result[2]=$lines[rand(0,324631)];
$c2_result[3]=$lines[rand(0,324631)];
$c2_result[4]=$lines[rand(0,324631)];
$c2_result[5]=$lines[rand(0,324631)];
$c2_result[6]=$lines[rand(0,324631)];
$myArray = (array) $c2_result;
$myArray2 = (array) $c2_result;
$myArray=array_flip($myArray);
if(count($myArray) != count($c2_result)) {//echo "duplicates\n";
$zex=0;
} else {//echo "no duplicates\n";
$zex=1;
//exit;
}
}
if(!in_array($lines2[$this->traceId],$myArray2)) {
$b++;
}
with a combination of array_rand and foreach. Upon some initial tests, it turns out that array_rand really is outstandingly slow. But my hash table solution to replace the in_array invocation still holds true. By leveraging a PHP array as a hash table (basically, store values as keys), we get a constant time lookup performance (O(1)), as opposed to a linear time lookup (O(n)).
Try replacing the above code with the following:
$myArray = [];
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
$myArray[rand(0,324631)] = true;
while (count($myArray) !== 7) {
$myArray[rand(0,324631)] = true;
}
if (!isset($myArray[$lines2[$this->traceId]])) {
$b++;
}
For me, this resulted in a 120% speedup.
As for further performance, you can (as mentioned above, again) store the results in-memory (as a simple property) and perform a write of all results at the end of the run method.
Also, the garbage collector for pthreads is not deterministic. It should therefore not be used to retrieve data. Instead, a Threaded object should be injected into the worker thread, where data to be collected should be saved to this object. Lastly, you should shutdown the pool after garbage collection (which, again, should not be used in your case).

Answer 2

despite it is unclear about your code and what $newlines and $newlines2 are, so, I am just guessing here...
something like this ?
The idea is to avoid as much as possible fopen and fwrite in your loop.
1 - open it only once in the construct.
2 - concat your chain in your loop.
3 - write it only once after the loop.
class WorkerThreads extends Thread {
private $workerId;
private $linesId;
private $linesId2;
private $c2_result;
private $traceId;
private $fp42;
private $mainfile3;
public function __construct($id, $newlines, $newlines2, $xxtrace) {
$this->workerId = $id;
$this->linesId = (array) $newlines;
$this->linesId2 = (array) $newlines2;
$this->traceId = $xxtrace;
$this->c2_result = array();
$this->fp42 = fopen("/folder/" . $id . "/count.txt", "w");
$this->mainfile3 = "/folder/" . $id . "/count_pthread.php";
}
public function run() {
for ($h = 0; $h < 90; $h++) {
$globalf42='';
for ($master = 0; $master < 200; $master++) {//<200
$b = 0;
for ($a = 0; $a < 3000; $a++) {
$zex = 0;
if ($zex != 1) {
for ($ii = 0; $ii < 6; $ii++) {
$this->c2_result[$ii] = $this->linesId[rand(0, 324631)];
}
$zex = (count(array_flip($this->c2_result)) != count($this->c2_result)) ? 0 : 1;
}
if (!in_array($this->linesId2[$this->traceId], $this->c2_result)) {
$b++;
}
}
$globalf42 .= $b . "\n";
}
fwrite($this->fp42, $globalf42);
fclose($this->fp42);
$command = "php $this->mainfile3 $this->workerId";
exec($command);
}
}
}