I am experimenting with pthreads in PHP. Referencing Joe's gist covering pools, I create a pool of worker threads, and submit instances of stackable objects. The threads and work all go as I expect, however it seems that the main thread/context does not (fully) dereference the Stackable objects, so memory usage increases indefinitely as it keeps more and more instances around.
It does seem to clear/nullify all of the properties of the stackable object, so memory usage doesn't quickly go through the roof, but it does increase forever, so this effectively can't be used for a long-running process. I hope I am just doing something wrong. Here is sample code that will demonstrate what I mean:
class W extends Worker {
public function run(){}
}
class S extends Stackable {
public $id;
public function run() {
$this->id = uniqid();
}
public function __destruct () {
// $this->id will be null here when the main thread destroys its copy of this
if ($this->id === null) {
echo "nullified stackable destroyed\n";
} else {
echo "stackable {$this->id} destroyed\n";
}
}
}
$pool = new Pool(3, W::class);
$i = 0;
while ($i++ < 10) {
$pool->submit(new S());
}
sleep(1); // to let threads finish
$pool->collect(function (S $s) {
echo "collected stackable {$s->id}\n";
return true;
});
$pool->shutdown();
echo "script exit - here come the destructions of all the accumulated stackables\n";
Many improvements have come to pthreads v3 as part of the upgrade for PHP7.
You will find the PHP7 version of this code works as expected:
<?php
class W extends Worker {
public function run(){}
}
class S extends Threaded implements Collectable {
public $id;
public function run() {
$this->id = uniqid();
$this->garbage = true;
}
public function __destruct () {
if ($this->id === null) {
echo "nullified stackable destroyed\n";
} else {
echo "stackable {$this->id} destroyed\n";
}
}
public function isGarbage() : bool { return $this->garbage; }
private $garbage = false;
}
$pool = new Pool(3, W::class);
$i = 0;
while ($i++ < 10) {
$pool->submit(new S());
}
while ($pool->collect(function (S $s) {
if ($s->isGarbage()) {
echo "collected stackable {$s->id}\n";
}
return $s->isGarbage();
})) continue;
$pool->shutdown();
echo "script exit - here come the destructions of all the accumulated stackables\n";
It is recommended that new projects use PHP7 and pthreads v3, it is vastly superior to PHP5 and pthreads v2.
Related
I'm using php-zts to perform parallel data processing, using symfony 4 and PThreads
I'm great at running multiple threads, but I'm facing a problem, I need each of the threads to be able to work with doctrine
I need to make sure that each thread is able to work with doctrine
I tried to transfer a container instance directly, but it won't work because it can't be sterilized
/console_comand.php
private function gettingStatistics(){
$pool = new \Pool(4, Autoloader::class, ["vendor/autoload.php"]);
$store = new \Threaded();
$class = new Meta();
$pool->submit(new Task($class,$store));
$pool->collect();
$pool->shutdown();
$listQuotes = array();
foreach ($store as $obj){
foreach ($obj->{'response'} as $exchange => $data){
$listQuotes[$exchange] = $data;
}
}
unset($store);
unset($interface);
return $listQuotes;
}
/Autoloader.php
<?php
namespace App\Worker;
class Autoloader extends \Worker
{
protected $loader;
public function __construct($loader)
{
$this->loader = $loader;
}
/* включить автозагрузчик для задач */
public function run()
{
require_once($this->loader);
}
/* переопределить поведение наследования по умолчанию для нового потокового контекста */
public function start(int $options = PTHREADS_INHERIT_ALL)
{
return parent::start(PTHREADS_INHERIT_NONE);
}
}
/Autoloadable.php
<?php
namespace App\Worker;
/* нормальный, автоматически загруженный класс */
class Autoloadable
{
public $response;
public function __construct($greeting)
{
$this->response = $greeting->job();
}
}
/Task.php
<?php
namespace App\Worker;
class Task extends \Threaded
{
protected $greeting;
protected $result;
public function __construct($greeting,\Threaded $store)
{
$this->greeting = $greeting;
$this->result = $store;
}
public function run()
{
$greeting = new Autoloadable($this->greeting);
$this->result[] = $greeting;
}
}
how do I pass the right doctrine to be able to work with it from the job?
there's a very similar question on github but I can't deal with it.
https://github.com/krakjoe/pthreads/issues/369
Have you tried requiring an ObjectManager instance in the __construct of Task (your last code block)?
Have a read of this article
Cannot test it atm, don't have zts setup, but I've used this to great success in other projects.
I would expect you need to do something like:
$pool = new Pool(4);
for ($i = 0; $i < 15; ++$i) {
$pool->submit(new class($objectManager) extends \Threaded
{
private $objectManager;
public function __construct(ObjectManager $objectManager)
{
$this->objectManager= $objectManager;
}
public function run()
{
// obviously replace the contents of this function
$this->objectManager->performTask;
echo 'Job\'s done.' . PHP_EOL;
}
});
}
while ($pool->collect());
$pool->shutdown();
The instantiation of the new anonymous class takes the $objectManager present in your current instance, like /console_comand.php there, and passes it to this new anonymous class to fulfill the __construct requirements.
The linked article does a better job of explaining it than I do, so please give it a read.
I am working on improving a process in php so I resorted into using multithreading using Worker to manage my threads. But the threads in the worker executes one after the other not concurrently.
This is my sample code
namespace App\Engine\Threads\ThreadClass;
class StatePaperAttendancePDFThread extends \Threaded
{
private $write_folder_name;
private $center_code;
private $paper_id;
private $title;
public function __construct($write_folder_name, $center_code, $paper_id, $title)
{
$this->write_folder_name = $write_folder_name;
$this->center_code = $center_code;
$this->paper_id = $paper_id;
$this->title = $title;
}
public function run(){
echo " Retrieving paper attendance ".$this->center_code." ".$this->paper_id." ".\Thread::getCurrentThreadId()." ".date("H:i:s:u").PHP_EOL;
$artisan = 'C:/xampp/htdocs/attendance/artisan';
$cmd = "php $artisan attendancereport:center $this->write_folder_name $this->center_code $this->paper_id $this->title";
$return = exec($cmd, $output, $return_var);
echo $return;
}
}
foreach ($centers as $i=>$center){
$center_code = $center->getCenterCode();
$thread = new StatePaperAttendancePDFThread($folder_name, $center_code, $paper_id, $title);
$worker->stack($thread);
}
$worker->start(PTHREADS_INHERIT_ALL ^ PTHREADS_INHERIT_CLASSES);
$worker->shutdown();
but when I monitor it from the CLI using the time been printed I can see that none of the threads starts together. They all starts with some seconds interval
Please what am I missing
I was able to solve my problem by creating a customized Thread Pool Executor as below. Please I am open to suggestions and Improvement
class ThreadPoolExecutor{
private $poolSize;
private $threadPool;
private $done = false;
private $workingThreads;
public function __construct($poolSize, array $threadPool)
{
$this->poolSize = $poolSize;
$this->threadPool = $threadPool;
}
public function execute()
{
$this->parametersOk();
try {
while (!empty($this->threadPool)) {
$this->extractThreads();
foreach ($this->workingThreads as $thread) {
$thread->start(PTHREADS_INHERIT_ALL ^ PTHREADS_INHERIT_CLASSES);
}
foreach ($this->workingThreads as $thread) {
$thread->join();
}
}
$this->done = true;
} catch (\Exception $ex) {
var_dump($ex->getMessage());
}
}
private function parametersOk()
{
if (!is_array($this->threadPool))
throw new \RuntimeException("threadPool expected to be an array of threads");
if (count($this->threadPool) <= 0)
throw new \RuntimeException("expected at least an element in threadPool");
foreach ($this->threadPool as $thread) {
if (!is_subclass_of($thread, \Thread::class, false)) {
throw new \RuntimeException(" an element of threadPool does not extend class \\Thread");
}
}
if ($this->poolSize > count($this->threadPool)) {
throw new \RuntimeException("The number of threads set to execute can not be greater than the threadPool");
}
return true;
}
private function extractThreads()
{
$this->workingThreads = [];
$this->workingThreads = array_slice($this->threadPool, 0, $this->poolSize);
for ($i = 0; $i < count($this->workingThreads); $i++) {
array_shift($this->threadPool);
}
}
public function isDone()
{
return $this->done;
}
}
I will appreciate any addition or correction to this.
safe storage of data. I read that for this task suits Stackable.
I inherit Stackable but data in storage is not synchronized.
AsyncOperation -- just incrementing value in storage
AsyncWatcher -- just making echo of value in storage.
Problem : data in storage is not modifying from AsyncOperation thread, storage permanently contains a -1.
I'm using pthreads.
class Storage extends Stackable {
public function __construct($data) {
$this->local = $data;
}
public function run()
{
}
public function getData() { return $this->local; }
}
class AsyncOperation extends Thread {
private $arg;
public function __construct(Storage $param){
$this->arg = $param->getData();
}
public function run(){
while (true) {
$this->arg++;
sleep(1);
}
}
}
class AsyncWatcher extends Thread {
public function __construct(Storage $param){
$this->storage = $param -> getData();
}
public function run(){
while (true) {
echo "In storage ". $this->storage ."\n";
sleep(1);
}
}
}
$storage = new Storage(-1);
$thread = new AsyncOperation($storage);
$thread->start();
$watcher = new AsyncWatcher($storage);
$watcher->start();
As you can see, Stackable class has a lot of methods, mainly used for async operations and they can help you with your issue. You should modify your async classes in this way:
class AsyncOperation extends Thread {
private $arg;
public function __construct(Storage $param){
$this->arg = $param->getData();
}
public function run(){
while (true) {
$this->arg++;
sleep(1);
}
$this->synchronized(function($thread){
$thread->notify();
}, $this);
}
}
and their usage will be like this:
$storage = new Storage();
$asyncOp = new AsyncOperation($storage);
$asyncOp->start();
$asyncOp->synchronized(function($thread){
$thread->wait();
}, $asyncOp);
var_dump($storage);
I am running 4 threads running in same time. (Threads are running work() function in same time in this case)
global $i;
$i = 1;
function work($address) {
while($i < 1000) {
$i++;
----
if($i == something) some job...
----
}
}
For some reason this don't do the job.
Threads do sometimes same circle in while, so I have later some duplicate values. (probably they have some critical section)
Any idea how to fix this ?
The counter object must be thread safe, it must also employ synchronized methods.
Follows is an example of such code:
<?php
class Counter extends Threaded {
public function __construct($value = 0) {
$this->value = $value;
}
/** protected methods are synchronized in pthreads **/
protected function increment() { return ++$this->value; }
protected function decrement() { return --$this->value; }
protected $value;
}
class My extends Thread {
/** all threads share the same counter dependency */
public function __construct(Counter $counter) {
$this->counter = $counter;
}
/** work will execute from job 1 to 1000, and no more, across all threads **/
public function run() {
while (($job = $this->counter->increment()) <= 1000) {
printf("Thread %lu doing job %d\n",
Thread::getCurrentThreadId(), $job);
}
}
protected $counter;
}
$counter = new Counter();
$threads = [];
while (($tid = count($threads)) < 4) {
$threads[$tid] = new My($counter);
$threads[$tid]->start();
}
foreach ($threads as $thread)
$thread->join();
?>
work() seems superfluous, this logic should be in the ::run function most likely.
I have run into a situation with pthreads.Consider the following script:
<?php
class W extends \Worker {
public function run()
{
echo "worker started\n";
}
}
clasds S extends \Stackable {
public function run()
{
echo "stackable executed\n";
}
}
class x{
protected $workers;
protected $i;
public function __construct()
{
$this->workers = array();
$this->i = 0;
}
public function go()
{
$thread = $this->spawn();
$j = new S;
$thread->stack($j);
//$thread->shutdown();
}
public function spawn()
{
$this->i !== 0 ? $n=$this->i++ : $n=$this->i;
$thread = new W;
$this->workers[$n] = $thread;
$thread->start();
return $thread;
}
}
$exe = new x;
$exe->go();
php crashes.2 ways to solve the issue, either:
uncommenting
$w->shutdown();
which means the workers one creates are not reusable, or commenting out
$this->c[$n] = $w;
which means not saving workers in an array.
I do not understand what is going on here.must one call shutdown on a worker?why does attempting to store
the worker in an array lead to a crash?