PHP Gearman tasks not returning jobHandle - php

We're using Gearman and when do use methods like doLowBackground or doHigh, these all return a jobHandle, but when we do tasks there is no jobHandle object. We get the GearmanTask object, instead of getting the jobHandle, we get string(0) ""
Any ideas what could cause this?
Thank you!
EDIT: Here is the code and additional info:
// $client = \GearmanClient; // servers added, all that jazz
// $workload = 'string';
$arguments = array(
'handleJob',
$workload
);
$task = call_user_func_array(array($client, $method), $arguments);
if ($task instanceof GearmanTask) {
$handles[] = $task->jobHandle();
}
$data = $client->runTasks();
The tasks run correctly but $handle is being populated with empty strings (one for each task added)
EDIT: This is the response we get:
object(GearmanTask)#294 (0) {
}
I've dumped every PECL gearman object, nothing ever displays, here's the client, populated with servers, options, etc
object(GearmanClient)#291 (0) {
}
Doesn't show anything.

A job handle is not assigned to a task until the task is received and queued by the Gearman job server.
However, you can use GearmanClient::setCreatedCallback() to get the handle once it has been queued. This must be done before both adding and running the tasks:
$client = new \GearmanClient();
$client->addServer('127.0.0.1');
$handles = array();
$client->setCreatedCallback(function (\GearmanTask $task) use (&$handles) {
$handles[] = $task->jobHandle();
});
$client->addTask('functionName', 'workload'); // ...
$client->runTasks();

Related

How do I "extract" the data from a ReactPHP\Promise\Promise?

Disclaimer: This is my first time working with ReactPHP and "php promises", so the solution might just be staring me in the face 🙄
I'm currently working on a little project where I need to create a Slack bot. I decided to use the Botman package and utilize it's Slack RTM driver. This driver utilizes ReactPHP's promises to communicate with the RTM API.
My problem:
When I make the bot reply on a command, I want to get the get retrieve the response from RTM API, so I can cache the ID of the posted message.
Problem is that, the response is being returned inside one of these ReactPHP\Promise\Promise but I simply can't figure out how to retrieve the data.
What I'm doing:
So when a command is triggered, the bot sends a reply Slack:
$response = $bot->reply($response)->then(function (Payload $item) {
return $this->reply = $item;
});
But then $response consists of an (empty?) ReactPHP\Promise\Promise:
React\Promise\Promise^ {#887
-canceller: null
-result: null
-handlers: []
-progressHandlers: & []
-requiredCancelRequests: 0
-cancelRequests: 0
}
I've also tried using done() instead of then(), which is what (as far as I can understand) the official ReactPHP docs suggest you should use to retrieve data from a promise:
$response = $bot->reply($response)->done(function (Payload $item) {
return $this->reply = $item;
});
But then $response returns as null.
The funny thing is, during debugging, I tried to do a var_dump($item) inside the then() but had forgot to remove a non-existing method on the promise. But then the var_dump actually returned the data 🤯
$response = $bot->reply($response)->then(function (Payload $item) {
var_dump($item);
return $this->reply = $item;
})->resolve();
So from what I can fathom, it's like I somehow need to "execute" the promise again, even though it has been resolved before being returned.
Inside the Bot's reply method, this is what's going on and how the ReactPHP promise is being generated:
public function apiCall($method, array $args = [], $multipart = false, $callDeferred = true)
{
// create the request url
$requestUrl = self::BASE_URL . $method;
// set the api token
$args['token'] = $this->token;
// send a post request with all arguments
$requestType = $multipart ? 'multipart' : 'form_params';
$requestData = $multipart ? $this->convertToMultipartArray($args) : $args;
$promise = $this->httpClient->postAsync($requestUrl, [
$requestType => $requestData,
]);
// Add requests to the event loop to be handled at a later date.
if ($callDeferred) {
$this->loop->futureTick(function () use ($promise) {
$promise->wait();
});
} else {
$promise->wait();
}
// When the response has arrived, parse it and resolve. Note that our
// promises aren't pretty; Guzzle promises are not compatible with React
// promises, so the only Guzzle promises ever used die in here and it is
// React from here on out.
$deferred = new Deferred();
$promise->then(function (ResponseInterface $response) use ($deferred) {
// get the response as a json object
$payload = Payload::fromJson((string) $response->getBody());
// check if there was an error
if (isset($payload['ok']) && $payload['ok'] === true) {
$deferred->resolve($payload);
} else {
// make a nice-looking error message and throw an exception
$niceMessage = ucfirst(str_replace('_', ' ', $payload['error']));
$deferred->reject(new ApiException($niceMessage));
}
});
return $deferred->promise();
}
You can see the full source of it here.
Please just point me in some kind of direction. I feel like I tried everything, but obviously I'm missing something or doing something wrong.
ReactPHP core team member here. There are a few options and things going on here.
First off then will never return the value from a promise, it will return a new promise so you can create a promise chain. As a result of that you do a new async operation in each then that takes in the result from the previous one.
Secondly done never returns result value and works pretty much like then but will throw any uncaught exceptions from the previous promise in the chain.
The thing with both then and done is that they are your resolution methods. A promise a merely represents the result of an operation that isn't done yet. It will call the callable you hand to then/done once the operation is ready and resolves the promise. So ultimately all your operations happen inside a callable one way or the other and in the broadest sense. (Which can also be a __invoke method on a class depending on how you set it up. And also why I'm so excited about short closures coming in PHP 7.4.)
You have two options here:
Run all your operations inside callable's
Use RecoilPHP
The former means a lot more mind mapping and learning how async works and how to wrap your mind around that. The latter makes it easier but requires you to run each path in a coroutine (callable with some cool magic).

Best way to offload one-shot worker threads in PHP? pthreads? fcntl?

How should I multithread some php-cli code that needs a timeout?
I'm using PHP 5.6 on Centos 6.6 from the command line.
I'm not very familiar with multithreading terminology or code. I'll simplify the code here but it is 100% representative of what I want to do.
The non-threaded code currently looks something like this:
$datasets = MyLibrary::getAllRawDataFromDBasArrays();
foreach ($datasets as $dataset) {
MyLibrary::processRawDataAndStoreResultInDB($dataset);
}
exit; // just for clarity
I need to prefetch all my datasets, and each processRawDataAndStoreResultInDB() cannot fetch it's own dataset. Sometimes processRawDataAndStoreResultInDB() takes too long to process a dataset, so I want to limit the amount of time it has to process it.
So you can see that making it multithreaded would
Speed it up by allowing multiple processRawDataAndStoreResultInDB() to execute at the same time
Use set_time_limit() to limit the amount of time each one has to process each dataset
Notice that I don't need to come back to my main program. Since this is a simplification, you can trust that I don't want to collect all the processed datasets and do a single save into the DB after they are all done.
I'd like to do something like:
class MyWorkerThread extends SomeThreadType {
public function __construct($timeout, $dataset) {
$this->timeout = $timeout;
$this->dataset = $dataset;
}
public function run() {
set_time_limit($this->timeout);
MyLibrary::processRawDataAndStoreResultInDB($this->dataset);
}
}
$numberOfThreads = 4;
$pool = somePoolClass($numberOfThreads);
$pool->start();
$datasets = MyLibrary::getAllRawDataFromDBasArrays();
$timeoutForEachThread = 5; // seconds
foreach ($datasets as $dataset) {
$thread = new MyWorkerThread($timeoutForEachThread, $dataset);
$thread->addCallbackOnTerminated(function() {
if ($this->isTimeout()) {
MyLibrary::saveBadDatasetToDb($dataset);
}
}
$pool->addToQueue($thread);
}
$pool->waitUntilAllWorkersAreFinished();
exit; // for clarity
From my research online I've found the PHP extension pthreads which I can use with my thread-safe php CLI, or I could use the PCNTL extension or a wrapper library around it (say, Arara/Process)
https://github.com/krakjoe/pthreads (and the example directory)
https://github.com/Arara/Process (pcntl wrapper)
When I look at them and their examples though (especially the pthreads pool example) I get confused quickly by the terminology and which classes I should use to achieve the kind of multithreading I'm looking for.
I even wouldn't mind creating the pool class myself, if I had a isRunning(), isTerminated(), getTerminationStatus() and execute() function on a thread class, as it would be a simple queue.
Can someone with more experience please direct me to which library, classes and functions I should be using to map to my example above? Am I taking the wrong approach completely?
Thanks in advance.
Here comes an example using worker processes. I'm using the pcntl extension.
/**
* Spawns a worker process and returns it pid or -1
* if something goes wrong.
*
* #param callback function, closure or method to call
* #return integer
*/
function worker($callback) {
$pid = pcntl_fork();
if($pid === 0) {
// Child process
exit($callback());
} else {
// Main process or an error
return $pid;
}
}
$datasets = array(
array('test', '123'),
array('foo', 'bar')
);
$maxWorkers = 1;
$numWorkers = 0;
foreach($datasets as $dataset) {
$pid = worker(function () use ($dataset) {
// Do DB stuff here
var_dump($dataset);
return 0;
});
if($pid !== -1) {
$numWorkers++;
} else {
// Handle fork errors here
echo 'Failed to spawn worker';
}
// If $maxWorkers is reached we need to wait
// for at least one child to return
if($numWorkers === $maxWorkers) {
// $status is passed by reference
$pid = pcntl_wait($status);
echo "child process $pid returned $status\n";
$numWorkers--;
}
}
// (Non blocking) wait for the remaining childs
while(true) {
// $status is passed by reference
$pid = pcntl_wait($status, WNOHANG);
if(is_null($pid) || $pid === -1) {
break;
}
if($pid === 0) {
// Be patient ...
usleep(50000);
continue;
}
echo "child process $pid returned $status\n";
}

Symfony event addListener and variable Scope/State

I'm currently using sitl/curl-easy to make a multi-curl parallel requets, but I'm having some hard time with scope / state:
// Init queue of requests
$queue = new \cURL\RequestsQueue;
... init queue options
// Set function to be executed when request will be completed
$queue->addListener('complete', function (\cURL\Event $event)
{
$response = $event->response->getContent();
// ugly :D, get the key from the object request
$key = explode("&key=", array_values($event->request->getOptions()->toArray())[0])[1];
// if response is not null, truncate it to use less space
if ($response != null){
$response = str_replace(array("\r", "\n"), "", $response);
}
>>>>>>>> DON'T WORK, $banner_holder is declared on the top of the php page
array_push($banner_holder, array("key"=> $key,"content"=>$response));
});
How to push the array of $key and $response outside the listener?
Thanks in advance.
Since you are using a closure function you will need to specify that you want to "use" the $banner_holder array like so
$queue->addListener('complete', function (\cURL\Event $event) use ($banner_holder) {

Laravel REST API and high CPU load

I am developing a simple RESTful API using Laravel 4.
I have set a Route that calls a function of my Controller that basically does this:
If information is in the database, pack it in a JSON object and return a response
Else try to download it (html/xml parsing), store it and finally pack the JSON response and send it.
I have noticed that the CPU load while doing a total of 1700 requests, only 2 at a time together, raises to 70-90%.
I am a complete php and laravel beginner and I've made the API following this tutorial, maybe I'm probably doing something wrong or it's just a proof of concept lacking of optimzations. How can I improve this code? (starting function is getGames)
Do you think the root of all problems is Laravel or I should obtain the same result even changing framework/using raw PHP?
UPDATE1 I also set a file Cache, but the CPU load is still ~50%.
UPDATE2 I set the query rate at two each 500ms and the CPU load lowered at 12%, so I guess this code is missing queue handling or something like this.
class GameController extends BaseController{
private static $platforms=array(
"Atari 2600",
"Commodore 64",
"Sega Dreamcast",
"Sega Game Gear",
"Nintendo Game Boy",
"Nintendo Game Boy Color",
"Nintendo Game Boy Advance",
"Atari Lynx",
"M.A.M.E.",
"Sega Mega Drive",
"Colecovision",
"Nintendo 64",
"Nintendo DS",
"Nintendo Entertainment System (NES)",
"Neo Geo Pocket",
"Turbografx 16",
"Sony PSP",
"Sony PlayStation",
"Sega Master System",
"Super Nintendo (SNES)",
"Nintendo Virtualboy",
"Wonderswan");
private function getDataTGDB($name,$platform){
$url = 'http://thegamesdb.net/api/GetGame.php?';
if(null==$name || null==$platform) return NULL;
$url.='name='.urlencode($name);
$xml = simplexml_load_file($url);
$data=new Data;
$data->query=$name;
$resultPlatform = (string)$xml->Game->Platform;
$data->platform=$platform;
$data->save();
foreach($xml->Game as $entry){
$games = Game::where('gameid',(string)$entry->id)->get();
if($games->count()==0){
if(strcasecmp($platform , $entry->Platform)==0 ||
(strcasecmp($platform ,"Sega Mega Drive")==0 &&
($entry->Platform=="Sega Genesis" ||
$entry->Platform=="Sega 32X" ||
$entry->Platform=="Sega CD"))){
$game = new Game;
$game->gameid = (string)$entry->id;
$game->title = (string)$entry->GameTitle;
$game->releasedate = (string)$entry->ReleaseDate;
$genres='';
if(NULL!=$entry->Genres->genre)
foreach($entry->Genres->genre as $genre){
$genres.=$genre.',';
}
$game->genres=$genres;
unset($genres);
$game->description = (string)$entry->Overview;
foreach($entry->Images->boxart as $boxart){
if($boxart["side"]=="front"){
$game->bigcoverurl = (string)$boxart;
$game->coverurl = (string) $boxart["thumb"];
} continue;
}
$game->save();
$data->games()->attach($game->id);
}
}
else foreach($games as $game){
$data->games()->attach($game->id);
}
}
unset($xml);
unset($url);
return $this->printJsonArray($data);
}
private function getArcadeHits($name){
$url = "http://www.arcadehits.net/index.php?p=roms&jeu=";
$url .=urlencode($name);
$html = file_get_html($url);
$data = new Data;
$data->query=$name;
$data->platform='M.A.M.E.';
$data->save();
$games = Game::where('title',$name)->get();
if($games->count()==0){
$game=new Game;
$game->gameid = -1;
$title = $html->find('h4',0)->plaintext;
if("Derniers jeux commentés"==$title)
{
unset($game);
return Response::json(array('status'=>'404'),200);
}
else{
$game->title=$title;
$game->description="(No description.)";
$game->releasedate=$html->find('a[href*=yearz]',0)->plaintext;
$game->genres = $html->find('a[href*=genre]',0)->plaintext;
$minithumb = $html->find('img.minithumb',0);
$game->coverurl = $minithumb->src;
$game->bigcoverurl = str_replace("/thumb/","/jpeg/",$minithumb->src);
$game->save();
$data->games()->attach($game->id);
}
}
unset($html);
unset($url);
return $this->printJsonArray($data);
}
private function printJsonArray($data){
$games = $data->games()->get();
$array_games = array();
foreach($games as $game){
$array_games[]=array(
'GameTitle'=>$game->title,
'ReleaseDate'=>$game->releasedate,
'Genres'=>$game->genres,
'Overview'=>$game->description,
'CoverURL'=>$game->coverurl,
'BigCoverURL'=>$game->bigcoverurl
);
}
$result = Response::json(array(
'status'=>'200',
'Game'=>$array_games
),200);
$key = $data->query.$data->platform;
if(!Cache::has($key))
Cache::put($key,$result,1440);
return $result;
}
private static $baseImgUrl = "";
public function getGames($apikey,$title,$platform){
$key = $title.$platform;
if(Cache::has($key)) return Cache::get($key);
if(!in_array($platform,GameController::$platforms)) return Response::json(array("status"=>"403","message"=>"non valid platform"));
$datas = Data::where('query',$title)
->where('platform',$platform)
->get();
//If this query has already been done we return data,otherwise according to $platform
//we call the proper parser.
if($datas->count()==0){
if("M.A.M.E."==$platform){
return $this->getArcadeHits($title);
}
else{
return $this->getDataTGDB($title,$platform);
}
} else{
else return $this->printJsonArray($datas->first());
}
}
}
?>
You're trying to retrieve data from others' servers. That is putting your CPU "on hold" until the data is fully retrieved. That's what is making your code be so "CPU expensive" (couldn't find other stuff that fits here =/ ), cause your script is waiting until the data is received and then release the script (CPU) work.
I strongly suggest that you make asynchronous calls. That would release your CPU to work on the code, while other part of your system is getting the information you need.
I hope that'll be some help! =D
UPDATE
To make examples, I'd have to re-factor your code (and I'm lazy as anything!). But, I can tell you for sure: If you put your request code, those who make calls to others site's XML, onto a queue you would gain a lot of free CPU time. Every request are redirected for a queue. Once they're ready, you treat them as you wish. Laravel has a beautiful way for dealing with queues.
what I would do first is to use a profiler to find out which parts would need an optimization. You can use for example this:
http://xdebug.org/docs/profiler
As well you didn't specify what kind of cpu is it, how many cores are you using? Is this a problem that your cpu is getting used that high?
you should use Laravel's Queue system along with beanstalkd for example and then monitor the queue (worker) with artisan queue:listen

Passing multiple variables to a Gearman worker function

How do you pass two variables to the same worker function? For example, say I wished to concat two strings that I pass from the client. I saw in some example code an array being used, but I can't get it to work.
<?php
$client= new GearmanClient();
$client->addServer();
$arguments = array(
"string1" => "hey",
"string2" => "there"
);
$client->addTask("string_concat", $arguments);
$client->runTasks();
?>
This tells me it's an invalid workload however (I assume cause it's an array being passed). How should I be passing them - should I create a task for each?
Then if I can't send an array, how can I use multiple variables in the worker function. I've tried like function String_Concat($job, $job2) but then I'm not sure how I'd add them to the workload()
Here is some example code if I were able to pass arrays:
<?php
$worker= new GearmanWorker();
$worker->addServer();
$worker->addFunction("string_concat", "String_Concat");
while ($worker->work());
function String_Concat($job)
{
$arguments = $job->workload();
return $arguments["string1"] . $arguments["string2"];
}
?>
What's the best way to do this? Thanks a lot!
You should serialize it.
Something like:
$data = serialize( $array );
$client->addTask("string_concat", $data);
Then, from your worker, you could do something like...
if (is_string($data) && $data = unserialize($workload)) {
} else {
// Maybe throw Exception or something?
}

Categories