I am running distributed job distribution through gearman. I am trying to convert 1000 pdf files of size 10 MB each. I am able to set it up and get the pdf files encoded.
I have the following code:
gearman_client.php (only to initialize)
<?php
class gearman_client{
static $timeout = 5000;
private static function addServerToCLient($client, $server){
$added = 0;
$client->setTimeout(self::$timeout);
$server = explode(':', $server);
$job_server = $server[0];
if(array_key_exists(1, $server))
$job_server_port = $server[1];
else
$job_server_port = 4730; //default port
try{
$added = $client->addServer($job_server, $job_server_port);
}
catch(GearmanException $e){
error_log(sprintf("GearmanException: %s .Error adding Gearman Server: %s, port: %s",$e->getMessage(), $job_server, $job_server_port), false);
}
return $added;
}
public static function initialize(){
$client = null;
$servers = explode(',', GEARMAN_JOB_SERVERS);
$added = 0;
//To accomodate Gearman bug
$flag = 0;
foreach($servers as $server){
if(!$flag){
unset($client);
$client = new GearmanClient();
}
$added = self::addServerToCLient($client, $server);
if($added)
$flag = 1;
}
if($flag)
return $client;
return 0;
}
}
Gearman client
$gmclient = gearman_client::initialize();
if(!$gmclient){
$this->error("Gearman server error","Could not create a gearman client object with a running Gearman Server");
return;
}
$pdfdata = array(
"id_host" => $this->id_host,
"id_vhost" => $this->id_vhost,
"id_contents" => $id_contents
);
$job_handle = $gmclient->doBackground("pdf_convert", json_encode($pdfdata));
if ($gmclient->returnCode() != GEARMAN_SUCCESS){
error_log("Gearman client: Bad Return code. Error while submitting background job");
}
Gearman worker
$worker = new GearmanWorker();
$worker->addServer("192.0.1.118");
$worker->addFunction("pdf_convert", function(GearmanJob $job) {
$workload = json_decode($job->workload());
$id_host = $workload->id_host;
$id_vhost = $workload->id_vhost;
$id_contents = $workload->id_contents;
contents_path($id_host, $id_vhost, $id_contents);
pdf($id_host, $id_vhost, $id_contents);
// You would then, of course, actually call this:
//mail($workload->email, $workload->subject, $workload->body);
});
while ($worker->work());
everything seems to be working well. However, I see that even though I start multiple instances of gearman worker, there is only one worker which is always busy(which I can see through the telnet interface).
If I introduce a sleep(30) in the worker code, then I start to see other workers taking up jobs.
Is this normal behavior?
Related
I use Node.js to run STOMP over WebSocket because it supports custom headers. When HTTP handshake, the server needs a cookie to check auth, not the token in the address.
As PHPer, I want to use this in PHP, but search google for a long time and found no way. can anyone help?
STOMP is not limited to Node.js, or Java, etc. There are
a STOMP PECL extension for PHP,
PHP STOMP libraries, e.g Stomp-PHP,
and general messaging libraries that support STOMP,
e.g. Enqueue's STOMP transport layer, or Laravel-Queue
Regarding your specific scenario, it's hard to come up with a straight-up answer without knowing more details but here is an example from the stomp-php library samples hat sets a custom header:
use Stomp\Client;
use Stomp\SimpleStomp;
use Stomp\Transport\Map;
// make a connection
$client = new Client('tcp://localhost:61613');
$stomp = new SimpleStomp($client);
// send a message to the queue
$body = array('city' => 'Belgrade', 'name' => 'Dejan');
$header = array();
$header['transformation'] = 'jms-map-json';
$mapMessage = new Map($body, $header);
$client->send('/queue/test', $mapMessage);
echo 'Sending array: ';
print_r($body);
$stomp->subscribe('/queue/test', 'transform-test', 'client', null, ['transformation' => 'jms-map-json']);
/** #var Map $msg */
$msg = $stomp->read();
// extract
if ($msg != null) {
echo 'Received array: ';
print_r($msg->map);
// mark the message as received in the queue
$stomp->ack($msg);
} else {
echo "Failed to receive a message\n";
}
you need a websocket stream wrapper if you want to connect using a protocol
You can use php-stomp-frame
https://github.com/jeeinn/php-stomp-frame
composer require jeeinn/php-stomp-frame
public function getStompMessage(): array
{
$stompFrame = new Frame();
$connectFrame = $stompFrame->setLogin($this->ss['login'], $this->ss['password'])
->setHeartBeat(0, 10000)->getConnect();
$subscribeFrame = $stompFrame->getSubscribe($this->ss['queue']);
$client = new Client($this->ss['host']);
$client->text($connectFrame);
$client->text($subscribeFrame);
$response = [];
$i = 0;
while ($i < 3) {
sleep(5);
try {
$message = $client->receive();
if (empty($message) || in_array($message, self::BYTE)) {
$i++;
continue;
}
$parsed = $stompFrame->parser($message);
if ($parsed['command'] == $this::STOMP_COMMAND_ERROR) {
$this->logger($this::INFO, 'STOMP return with ERROR command:', $parsed['body']);
$client->close();
break;
}
if (($parsed['command'] == $this::STOMP_COMMAND_MESSAGE) || ($parsed['body'])) {
return json_decode($parsed['body'], true);
}
} catch (Exception $e) {
$this->logger($this::ERROR, 'STOMP error', (array)$e->getMessage());
}
$i++;
}
return $response;
}
EDIT : Already fixed the error Creating default object from empty value . My question now is just suggestions how to make this process better.
I am making a tool that sends batch API request to a certain URL. I am using https://github.com/stil/curl-easy for batch API request That API will validate the phone numbers I am sending and return a result. That API can support maximum 20 request per second, supposed I have 50,000 API request, what is the best and fastest way to process things?
The current process is, I query to my database for the number of records I have which is the number of phone numbers. Then I make a loop with the number of phone numbers. Inside a single iteration, I will query 13 records from the database then pass it to the queue and process it, when the processing is finished I will query again to the database and update the fields for the the phone number based on the API response. My problem is that I think I have a logic error on my loop when I run this the processing will stop for sometime and gives me:
Creating default object from empty value
I guess that is because of my loop here's my code.
public function getParallelApi()
{
$numItems = DB::connection('mysql')->select("SELECT COUNT(ID) AS cnt FROM phones WHERE status IS NULL");
for($y = 0; $y < $numItems[0]->cnt; $y++)
{
$queue = new \cURL\RequestsQueue;
// Set default options for all requests in queue
$queue->getDefaultOptions()
->set(CURLOPT_TIMEOUT, 5)
->set(CURLOPT_RETURNTRANSFER, true)
->set(CURLOPT_SSL_VERIFYPEER, false);
// This is the function that is called for every completed request
$queue->addListener('complete', function (\cURL\Event $event) {
$response = $event->response;
$json = $response->getContent(); // Returns content of response
$feed = json_decode($json, true);
$phone_number = $feed['msisdn'];
if(isset($phone_number))
{
$status = "";$remarks = "";$network = ""; $country = "";$country_prefix = "";
if(isset($feed['status']))
{
$status = $feed['status'];
}
if(isset($feed['network']))
{
$network = $feed['network'];
}
if(isset($feed['country']))
{
$country = $feed['country'];
}
if(isset($feed['country_prefix']))
{
$country_prefix = $feed['country_prefix'];
}
if(isset($feed['remark']))
{
$remarks = $feed['remark'];
}
$update = Phone::find($phone_number);
$update->network = $network;
$update->country = $country;
$update->country_prefix = $country_prefix;
$update->status = $status;
$update->remarks = $remarks;
$update->save();
}
});
// Get 13 records
$phone = DB::connection('mysql')->select("SELECT * FROM phones WHERE status IS NULL LIMIT 13");
foreach ($phone as $key => $value)
{
$request = new \cURL\Request($this->createCurlUrl($value->msisdn));
// var_dump($this->createCurlUrl($value->msisdn)); exit;
// Add request to queue
$queue->attach($request);
}
// Process the queue
while ($queue->socketPerform()) {
$queue->socketSelect();
}
sleep(2);
$y += 13; // Move to the next 13 records
}
Session::flash('alert-info', 'Data Updated Successfully!');
return Redirect::to('upload');
}
Since the maximum is 20 request per seconds I'm just doing it 13 request per second just to be sure I won't clog their server. I am adding sleep(2); to pause a bit so that I can make sure that the queue is fully processed before moving on.
I want to check to see if Gearman daemon is running. And only then run tasks so my application does not crash.
Here's my code:
$daemonRunning = true;
while( true )
{
try
{
Yii::app()->gearman->client->ping( true );
if ( $daemonRunning === false )
{
echo "Daemon back online. Starting signature process...\n";
}
Yii::app()->gearman->client->runTasks();
}
catch( GearmanException $e )
{
echo "Daemon appears to be down. Waiting for it to come back up...\n";
$daemonRunning = false;
}
sleep(1);
}
But the issue is that ping does not throw an exception, it throws a fatal error:
PHP Error[2]: GearmanClient::ping(): flush(GEARMAN_COULD_NOT_CONNECT) 127.0.0.1:4730 -> libgearman/connection.cc:673
Though oddly, if I remove ping, and use only runTasks, then an exception is thrown.
Related:
How do I handle the error when Gearman daemon goes down while processes are running? I get the following error from PHP when I bring down the Gearman daemon:
php: libgearman/universal.cc:481: gearman_return_t connection_loop(gearman_universal_st&, const gearman_packet_st&, Check&): Assertion `&con->_packet == universal.packet_list' failed.
Aborted (core dumped)
At it's most basic, checking the status of a Gearman server can be done through command line using:
(echo status ; sleep 0.1) | nc 127.0.0.1 4730 -w 1
However as noted in this question, you can use fsocketopen to get he status of the gearman server.
// Taken from https://stackoverflow.com/questions/2752431/any-way-to-access-gearman-administration
class Waps_Gearman_Server {
/**
* #var string
*/
protected $host = "127.0.0.1";
/**
* #var int
*/
protected $port = 4730;
/**
* #param string $host
* #param int $port
*/
public function __construct($host=null,$port=null){
if( !is_null($host) ){
$this->host = $host;
}
if( !is_null($port) ){
$this->port = $port;
}
}
/**
* #return array | null
*/
public function getStatus(){
$status = null;
$handle = fsockopen($this->host,$this->port,$errorNumber,$errorString,30);
if($handle!=null){
fwrite($handle,"status\n");
while (!feof($handle)) {
$line = fgets($handle, 4096);
if( $line==".\n"){
break;
}
if( preg_match("~^(.*)[ \t](\d+)[ \t](\d+)[ \t](\d+)~",$line,$matches) ){
$function = $matches[1];
$status['operations'][$function] = array(
'function' => $function,
'total' => $matches[2],
'running' => $matches[3],
'connectedWorkers' => $matches[4],
);
}
}
fwrite($handle,"workers\n");
while (!feof($handle)) {
$line = fgets($handle, 4096);
if( $line==".\n"){
break;
}
// FD IP-ADDRESS CLIENT-ID : FUNCTION
if( preg_match("~^(\d+)[ \t](.*?)[ \t](.*?) : ?(.*)~",$line,$matches) ){
$fd = $matches[1];
$status['connections'][$fd] = array(
'fd' => $fd,
'ip' => $matches[2],
'id' => $matches[3],
'function' => $matches[4],
);
}
}
fclose($handle);
}
return $status;
}
}
In regards to your second question, I have never been able to recover a gearman worker when the connect is lost midwork. You basically have to kill the entire process running the client worker, and let supervisor take back over and restart another worker process.
Gearman client workers should be extremeley ephemeral. You should be monitoring them for memory usage regularly and auto killing them. So, if you ever hit a segfault/coredump, killing the worker is totally normal.
As mentioned, you can auto start back up your workers using Supervisor. Also check this out, it explains how to get Gearman Clients working with Supervisor
I'm trying to use PHP and SoapClient to utilize the UPS Ratings web service. I found a nice tool called WSDLInterpreter to create a starting point library for creating the service requests, but regardless what I try I keep getting the same (non-descriptive) error:
EXCEPTION=SoapFault::__set_state(array(
'message' => 'An exception has been raised as a result of client data.',
'string' => '',
'code' => 0,
Um ok, what the hell does this mean?
Unlike some of the other web service tools I have implemented, the UPS Soap wants a security block put into the header. I tried doing raw associative array data but I wasn't sure 100% if I was injecting the header part correctly.
Using the WSDLInterpreter, it pulls out a RateService class with a ProcessRate method that excepts it's own (instance) datastructure for the RateRequest and UPSSecurity portions, but all of the above generates that same error.
Here's a sample of the code that I'm using that calls classes defined by the interpreter:
require_once "Shipping/UPS/RateService.php"; // WSDLInterpreter class library
class Query
{
// constants removed for stackoverflow that define (proprietary) security items
private $_upss;
private $_shpr;
// use Interpreter code's RateRequest class to send with ProcessRate
public function soapRateRequest(RateRequest $req, UPSSecurity $upss = null)
{
$res = false;
if (!isset($upss)) {
$upss = $this->__getUPSS();
}
echo "SECURITY:\n" . var_export($upss, 1) . "\n";
$upsrs = new RateService(self::UPS_WSDL);
try {
$res = $upsrs->ProcessRate($req, $upss);
} catch (SoapFault $exception) {
echo 'EXCEPTION=' . var_export($exception, 1) . "\n";
}
return $res;
}
// create a soap data structure to send to web service from shipment
public function getRequestSoap(Shipment $shpmnt)
{
$qs = new RateRequest();
// pickup information
$qs->PickupType->Code = '01';
$qs->Shipment->Shipper = $this->__getAcctInfo();
// Ship To address
$qs->Shipment->ShipTo->Address->AddressLine = $this->__getAddressArray($shpmnt->destAddress->address1, $shpmnt->destAddress->address2);
$qs->Shipment->ShipTo->Address->City = $shpmnt->destAddress->city;
$qs->Shipment->ShipTo->Address->StateProvinceCode = $shpmnt->destAddress->state;
$qs->Shipment->ShipTo->Address->PostalCode = $shpmnt->destAddress->zip;
$qs->Shipment->ShipTo->Address->CountryCode = $shpmnt->destAddress->country;
$qs->Shipment->ShipTo->Name = $shpmnt->destAddress->name;
// Ship From address
$qs->Shipment->ShipFrom->Address->AddressLine = $this->__getAddressArray($shpmnt->origAddress->address1, $shpmnt->origAddress->address2);
$qs->Shipment->ShipFrom->Address->City = $shpmnt->origAddress->city;
$qs->Shipment->ShipFrom->Address->StateProvinceCode = $shpmnt->origAddress->state;
$qs->Shipment->ShipFrom->Address->PostalCode = $shpmnt->origAddress->zip;
$qs->Shipment->ShipFrom->Address->CountryCode = $shpmnt->origAddress->country;
$qs->Shipment->ShipFrom->Name = $shpmnt->origAddress->name;
// Service type
// TODO cycle through available services
$qs->Shipment->Service->Code = "03";
$qs->Shipment->Service->Description = "UPS Ground";
// Package information
$pkg = new PackageType();
$pkg->PackagingType->Code = "02";
$pkg->PackagingType->Description = "Package/customer supplied";
// dimensions
$pkg->Dimensions->UnitOfMeasurement->Code = $shpmnt->dimensions->dimensionsUnit;
$pkg->Dimensions->Length = $shpmnt->dimensions->length;
$pkg->Dimensions->Width = $shpmnt->dimensions->width;
$pkg->Dimensions->Height = $shpmnt->dimensions->height;
$pkg->PackageServiceOptions->DeclaredValue->CurrencyCode = "USD";
$pkg->PackageServiceOptions->DeclaredValue->MonetaryValue = $shpmnt->dimensions->value;
$pkg->PackageServiceOptions->DeclaredValue->CurrencyCode = "USD";
$pkg->PackageWeight->UnitOfMeasurement = $this->__getWeightUnit($shpmnt->dimensions->weightUnit);
$pkg->PackageWeight->Weight = $shpmnt->dimensions->weight;
$qs->Shipment->Package = $pkg;
$qs->CustomerClassification->Code = 123456;
$qs->CustomerClassification->Description = "test_rate_request";
return $qs;
}
// fill out and return a UPSSecurity data structure
private function __getUPSS()
{
if (!isset($this->_upss)) {
$unmt = new UsernameToken();
$unmt->Username = self::UPSS_USERNAME;
$unmt->Password = self::UPSS_PASSWORD;
$sat = new ServiceAccessToken();
$sat->AccessLicenseNumber = self::UPSS_ACCESS_LICENSE_NUMBER;
$upss = new UPSSecurity();
$upss->UsernameToken = $unmt;
$upss->ServiceAccessToken = $sat;
$this->_upss = $upss;
}
return $this->_upss;
}
// get our shipper/account info (some items blanked for stackoverflow)
private function __getAcctInfo()
{
if (!isset($this->_shpr)) {
$shpr = new ShipperType();
$shpr->Address->AddressLine = array(
"CONTACT",
"STREET ADDRESS"
);
$shpr->Address->City = "CITY";
$shpr->Address->StateProvinceCode = "MI";
$shpr->Address->PostalCode = "ZIPCODE";
$shpr->Address->CountryCode = "US";
$shpr = new ShipperType();
$shpr->Name = "COMPANY NAME";
$shpr->ShipperNumber = self::UPS_ACCOUNT_NUMBER;
$shpr->Address = $addr;
$this->_shpr = $shpr;
}
return $this->_shpr;
}
private function __getAddressArray($adr1, $adr2 = null)
{
if (isset($adr2) && $adr2 !== '') {
return array($adr1, $adr2);
} else {
return array($adr1);
}
}
}
It doesn't even seem to be getting to the point of sending anything over the Soap so I am assuming it is dying as a result of something not matching the WSDL info. (keep in mind, I've tried sending just a properly seeded array of key/value details to a manually created SoapClient using the same WSDL file with the same error resulting)
It would just be nice to get an error to let me know what about the 'client data' is a problem. This PHP soap implementation isn't impressing me!
I know this answer is probably way too late, but I'll provide some feedback anyway. In order to make a custom SOAP Header you'll have to override the SoapHeader class.
/*
* Auth Class to extend SOAP Header for WSSE Security
* Usage:
* $header = new upsAuthHeader($user, $password);
* $client = new SoapClient('{...}', array("trace" => 1, "exception" => 0));
* $client->__setSoapHeaders(array($header));
*/
class upsAuthHeader extends SoapHeader
{
...
function __construct($user, $password)
{
// Using SoapVar to set the attributes has proven nearly impossible; no WSDL.
// It might be accomplished with a combined SoapVar and stdClass() approach.
// Security Header - as a combined XSD_ANYXML SoapVar
// This method is much easier to define all of the custom SoapVars with attributes.
$security = '<ns2:Security xmlns:ns2="'.$this->wsse.'">'. // soapenv:mustUnderstand="1"
'<ns2:UsernameToken ns3:Id="UsernameToken-49" xmlns:ns3="'.$this->wsu.'">'.
'<ns2:Username>'.$user.'</ns2:Username>'.
'<ns2:Password Type="'.$this->password_type.'">'.htmlspecialchars($password).'</ns2:Password>'.
'</ns2:UsernameToken>'.
'</ns2:Security>';
$security_sv = new SoapVar($security, XSD_ANYXML);
parent::__construct($this->wsse, 'Security', $security_sv, false);
}
}
Then call the upsAuthHeader() before the soap call.
$client = new SoapClient($this->your_ups_wsdl,
array('trace' => true,
'exceptions' => true,
'soap_version' => SOAP_1_1
)
);
// Auth Header - Security Header
$header = new upsAuthHeader($user, $password);
// Set Header
$client->__setSoapHeaders(array($header));
I am able to make a simple php websocket server with libevent , but I am stuck when I'm trying to make it multiprocessing.
for example this is single processing
<?php
$socket = stream_socket_server ('tcp://0.0.0.0:2000', $errno, $errstr);
stream_set_blocking($socket, 0);
$base = event_base_new();
$event = event_new();
event_set($event, $socket, EV_READ | EV_PERSIST, 'ev_accept', $base);
event_base_set($event, $base);
event_add($event);
event_base_loop($base);
$GLOBALS['connections'] = array();
$GLOBALS['buffers'] = array();
function ev_accept($socket, $flag, $base) {
static $id = 0;
$connection = stream_socket_accept($socket);
stream_set_blocking($connection, 0);
$id += 1;
$buffer = event_buffer_new($connection, 'ev_read', NULL, 'ev_error', $id);
event_buffer_base_set($buffer, $base);
event_buffer_timeout_set($buffer, 30, 30);
event_buffer_watermark_set($buffer, EV_READ, 0, 0xffffff);
event_buffer_priority_set($buffer, 10);
event_buffer_enable($buffer, EV_READ | EV_PERSIST);
// we need to save both buffer and connection outside
$GLOBALS['connections'][$id] = $connection;
$GLOBALS['buffers'][$id] = $buffer;
}
function ev_error($buffer, $error, $id) {
event_buffer_disable($GLOBALS['buffers'][$id], EV_READ | EV_WRITE);
event_buffer_free($GLOBALS['buffers'][$id]);
fclose($GLOBALS['connections'][$id]);
unset($GLOBALS['buffers'][$id], $GLOBALS['connections'][$id]);
}
function ev_read($buffer, $id) {
while ($read = event_buffer_read($buffer, 256)) {
var_dump($read);
}
}
?>
But when I do this in function ev_read
function ev_read($buffer, $id) {
while ($read = event_buffer_read($buffer, 256)) {
$pid = pcntl_fork();
switch ($pid) {
case -1: // Error
die('Fork failed, your system is b0rked!');
break;
case 0: // Child
event_buffer_write($buffer,"asdawdasd");
exit(0);
break;
}
} }
it doesnt send the data...
So how can I make a multiprocessing php socket server?
While nanoserv is an excellent library, it does not use libevent. Infact the author himself has written, in his blog, that he would like to convert nanoserv to use libevent at some point of time. See his blog post here: http://blog.si.kz/index.php/2010/02/03/libevent-for-php
There is also a comment by Alix Axel on May 22 '11 at 12:19 regarding the same.
Update: A little more research led me to http://phpdaemon.net/ . It seems they are using libevent to build a whole host of network servers
Updated 30th May 2021:
Recently there has been https://www.swoole.co.uk/ & https://reactphp.org/ which provide good async libraries for sockets.
phpdaemon has a new url at https://daemon.io/
Updated 31st August 2022:
Swoole has now been forked into two separate projects: Open Swoole & Swoole