Testing to see if Gearman daemon is running - php

I want to check to see if Gearman daemon is running. And only then run tasks so my application does not crash.
Here's my code:
$daemonRunning = true;
while( true )
{
try
{
Yii::app()->gearman->client->ping( true );
if ( $daemonRunning === false )
{
echo "Daemon back online. Starting signature process...\n";
}
Yii::app()->gearman->client->runTasks();
}
catch( GearmanException $e )
{
echo "Daemon appears to be down. Waiting for it to come back up...\n";
$daemonRunning = false;
}
sleep(1);
}
But the issue is that ping does not throw an exception, it throws a fatal error:
PHP Error[2]: GearmanClient::ping(): flush(GEARMAN_COULD_NOT_CONNECT) 127.0.0.1:4730 -> libgearman/connection.cc:673
Though oddly, if I remove ping, and use only runTasks, then an exception is thrown.
Related:
How do I handle the error when Gearman daemon goes down while processes are running? I get the following error from PHP when I bring down the Gearman daemon:
php: libgearman/universal.cc:481: gearman_return_t connection_loop(gearman_universal_st&, const gearman_packet_st&, Check&): Assertion `&con->_packet == universal.packet_list' failed.
Aborted (core dumped)

At it's most basic, checking the status of a Gearman server can be done through command line using:
(echo status ; sleep 0.1) | nc 127.0.0.1 4730 -w 1
However as noted in this question, you can use fsocketopen to get he status of the gearman server.
// Taken from https://stackoverflow.com/questions/2752431/any-way-to-access-gearman-administration
class Waps_Gearman_Server {
/**
* #var string
*/
protected $host = "127.0.0.1";
/**
* #var int
*/
protected $port = 4730;
/**
* #param string $host
* #param int $port
*/
public function __construct($host=null,$port=null){
if( !is_null($host) ){
$this->host = $host;
}
if( !is_null($port) ){
$this->port = $port;
}
}
/**
* #return array | null
*/
public function getStatus(){
$status = null;
$handle = fsockopen($this->host,$this->port,$errorNumber,$errorString,30);
if($handle!=null){
fwrite($handle,"status\n");
while (!feof($handle)) {
$line = fgets($handle, 4096);
if( $line==".\n"){
break;
}
if( preg_match("~^(.*)[ \t](\d+)[ \t](\d+)[ \t](\d+)~",$line,$matches) ){
$function = $matches[1];
$status['operations'][$function] = array(
'function' => $function,
'total' => $matches[2],
'running' => $matches[3],
'connectedWorkers' => $matches[4],
);
}
}
fwrite($handle,"workers\n");
while (!feof($handle)) {
$line = fgets($handle, 4096);
if( $line==".\n"){
break;
}
// FD IP-ADDRESS CLIENT-ID : FUNCTION
if( preg_match("~^(\d+)[ \t](.*?)[ \t](.*?) : ?(.*)~",$line,$matches) ){
$fd = $matches[1];
$status['connections'][$fd] = array(
'fd' => $fd,
'ip' => $matches[2],
'id' => $matches[3],
'function' => $matches[4],
);
}
}
fclose($handle);
}
return $status;
}
}
In regards to your second question, I have never been able to recover a gearman worker when the connect is lost midwork. You basically have to kill the entire process running the client worker, and let supervisor take back over and restart another worker process.
Gearman client workers should be extremeley ephemeral. You should be monitoring them for memory usage regularly and auto killing them. So, if you ever hit a segfault/coredump, killing the worker is totally normal.
As mentioned, you can auto start back up your workers using Supervisor. Also check this out, it explains how to get Gearman Clients working with Supervisor

Related

Why timeout is not working

sorry for stupid question, but
Im trying to set the timeout for program called cirsym. Because sometimes Im facing with infinity process and I have to cancel the operation using ssh...
Would be really thankfull if someone help...
<?
/**
* Execute a command and return it's output. Either wait until the command exits or the timeout has expired.
*
* #param string $cmd Command to execute.
* #param number $timeout Timeout in seconds.
* #return string Output of the command.
* #throws \Exception
*/
function exec_timeout($cmd, $timeout) {
// File descriptors passed to the process.
$descriptors = array(
0 => array('pipe', 'r'), // stdin
1 => array('pipe', 'w'), // stdout
2 => array('pipe', 'w') // stderr
);
// Start the process.
$process = proc_open('exec ' . $cmd, $descriptors, $pipes);
if (!is_resource($process)) {
throw new \Exception('Could not execute process');
}
// Set the stdout stream to none-blocking.
stream_set_blocking($pipes[1], 0);
// Turn the timeout into microseconds.
$timeout = $timeout * 1000000;
// Output buffer.
$buffer = '';
// While we have time to wait.
while ($timeout > 0) {
$start = microtime(true);
// Wait until we have output or the timer expired.
$read = array($pipes[1]);
$other = array();
stream_select($read, $other, $other, 0, $timeout);
// Get the status of the process.
// Do this before we read from the stream,
// this way we can't lose the last bit of output if the process dies between these functions.
$status = proc_get_status($process);
// Read the contents from the buffer.
// This function will always return immediately as the stream is none-blocking.
$buffer .= stream_get_contents($pipes[1]);
if (!$status['running']) {
break;
}
// Subtract the number of microseconds that we waited.
$timeout -= (microtime(true) - $start) * 1000000;
}
// Check if there were any errors.
$errors = stream_get_contents($pipes[2]);
if (!empty($errors)) {
throw new \Exception($errors);
}
// Kill the process in case the timeout expired and it's still running.
// If the process already exited this won't do anything.
proc_terminate($process, 9);
// Close all streams.
fclose($pipes[0]);
fclose($pipes[1]);
fclose($pipes[2]);
proc_close($process);
return $buffer;
}
This is the function
private function Exec(){
if(shell_exec($this->canonical_path."cirsym_v2.1.bin") != NULL)
{
return true;
}else{
return false;
}
}

Only one worker seems to be prcoessing jobs in Gearman

I am running distributed job distribution through gearman. I am trying to convert 1000 pdf files of size 10 MB each. I am able to set it up and get the pdf files encoded.
I have the following code:
gearman_client.php (only to initialize)
<?php
class gearman_client{
static $timeout = 5000;
private static function addServerToCLient($client, $server){
$added = 0;
$client->setTimeout(self::$timeout);
$server = explode(':', $server);
$job_server = $server[0];
if(array_key_exists(1, $server))
$job_server_port = $server[1];
else
$job_server_port = 4730; //default port
try{
$added = $client->addServer($job_server, $job_server_port);
}
catch(GearmanException $e){
error_log(sprintf("GearmanException: %s .Error adding Gearman Server: %s, port: %s",$e->getMessage(), $job_server, $job_server_port), false);
}
return $added;
}
public static function initialize(){
$client = null;
$servers = explode(',', GEARMAN_JOB_SERVERS);
$added = 0;
//To accomodate Gearman bug
$flag = 0;
foreach($servers as $server){
if(!$flag){
unset($client);
$client = new GearmanClient();
}
$added = self::addServerToCLient($client, $server);
if($added)
$flag = 1;
}
if($flag)
return $client;
return 0;
}
}
Gearman client
$gmclient = gearman_client::initialize();
if(!$gmclient){
$this->error("Gearman server error","Could not create a gearman client object with a running Gearman Server");
return;
}
$pdfdata = array(
"id_host" => $this->id_host,
"id_vhost" => $this->id_vhost,
"id_contents" => $id_contents
);
$job_handle = $gmclient->doBackground("pdf_convert", json_encode($pdfdata));
if ($gmclient->returnCode() != GEARMAN_SUCCESS){
error_log("Gearman client: Bad Return code. Error while submitting background job");
}
Gearman worker
$worker = new GearmanWorker();
$worker->addServer("192.0.1.118");
$worker->addFunction("pdf_convert", function(GearmanJob $job) {
$workload = json_decode($job->workload());
$id_host = $workload->id_host;
$id_vhost = $workload->id_vhost;
$id_contents = $workload->id_contents;
contents_path($id_host, $id_vhost, $id_contents);
pdf($id_host, $id_vhost, $id_contents);
// You would then, of course, actually call this:
//mail($workload->email, $workload->subject, $workload->body);
});
while ($worker->work());
everything seems to be working well. However, I see that even though I start multiple instances of gearman worker, there is only one worker which is always busy(which I can see through the telnet interface).
If I introduce a sleep(30) in the worker code, then I start to see other workers taking up jobs.
Is this normal behavior?

Waiting for error from apns with enhanced format in php

We're using a modified version of easy-apns for sending our push messages. With the enhanced push message format the apple server responds if an error occurs, and does nothing if everything goes well.
The problem is that we have to wait for an error a certain amount of time after each message has been sent. For example if we receive no response after 1 second, we assume everything went ok.
With 20000 push messages, this takes far too long. Is there any way I can listen for errors in a faster way? For example sending to 1000 devices and then listen for errors? What happens if the connection gets closed, can I still read the error response?
Ideal would be some kind of asynchronous writing and reading, but I think that's not possible.
Here's the corresponding code:
$fp = $this->connect();
$expiry = time()+60*60;
// construct message
$msg = chr(1).pack("N",$batchid).pack("N",$expiry).pack("n",32).pack('H*',$devicetoken).pack("n",strlen($payload)).$payload;
// send message to Apple
$fwrite = fwrite($fp, $msg);
if(!$fwrite) {
// connection has been closed
$this->disconnect();
throw new Exception("Connection closed");
} else {
// read response from Apple
// Timeout. 1 million micro seconds = 1 second
$tv_sec = 1;
$tv_usec = 0;
$r = array($fp);
$we = null; // Temporaries. "Only variables can be passed as reference."
// PROBLEM: this method waits for $tv_sec seconds for a response
$numChanged = stream_select($r, $we, $we, $tv_sec, $tv_usec);
if( $numChanged === false ) {
throw new Exception("Failed selecting stream to read.");
} elseif ( $numChanged > 0 ) {
$command = ord( fread($fp, 1) );
$status = ord( fread($fp, 1) );
$identifier = implode('', unpack("N", fread($fp, 4)));
if( $status > 0 ) {
// The socket has also been closed. Cause reopening in the loop outside.
$this->disconnect();
throw new MessageException("APNS responded with status $status: {$this->statusDesc[$status]} ($devicetoken).".microtime(), $status);
} else {
// unknown response, assume ok
}
} else {
// no response, assume ok
}
}

Multi-process php with libevent

I am able to make a simple php websocket server with libevent , but I am stuck when I'm trying to make it multiprocessing.
for example this is single processing
<?php
$socket = stream_socket_server ('tcp://0.0.0.0:2000', $errno, $errstr);
stream_set_blocking($socket, 0);
$base = event_base_new();
$event = event_new();
event_set($event, $socket, EV_READ | EV_PERSIST, 'ev_accept', $base);
event_base_set($event, $base);
event_add($event);
event_base_loop($base);
$GLOBALS['connections'] = array();
$GLOBALS['buffers'] = array();
function ev_accept($socket, $flag, $base) {
static $id = 0;
$connection = stream_socket_accept($socket);
stream_set_blocking($connection, 0);
$id += 1;
$buffer = event_buffer_new($connection, 'ev_read', NULL, 'ev_error', $id);
event_buffer_base_set($buffer, $base);
event_buffer_timeout_set($buffer, 30, 30);
event_buffer_watermark_set($buffer, EV_READ, 0, 0xffffff);
event_buffer_priority_set($buffer, 10);
event_buffer_enable($buffer, EV_READ | EV_PERSIST);
// we need to save both buffer and connection outside
$GLOBALS['connections'][$id] = $connection;
$GLOBALS['buffers'][$id] = $buffer;
}
function ev_error($buffer, $error, $id) {
event_buffer_disable($GLOBALS['buffers'][$id], EV_READ | EV_WRITE);
event_buffer_free($GLOBALS['buffers'][$id]);
fclose($GLOBALS['connections'][$id]);
unset($GLOBALS['buffers'][$id], $GLOBALS['connections'][$id]);
}
function ev_read($buffer, $id) {
while ($read = event_buffer_read($buffer, 256)) {
var_dump($read);
}
}
?>
But when I do this in function ev_read
function ev_read($buffer, $id) {
while ($read = event_buffer_read($buffer, 256)) {
$pid = pcntl_fork();
switch ($pid) {
case -1: // Error
die('Fork failed, your system is b0rked!');
break;
case 0: // Child
event_buffer_write($buffer,"asdawdasd");
exit(0);
break;
}
} }
it doesnt send the data...
So how can I make a multiprocessing php socket server?
While nanoserv is an excellent library, it does not use libevent. Infact the author himself has written, in his blog, that he would like to convert nanoserv to use libevent at some point of time. See his blog post here: http://blog.si.kz/index.php/2010/02/03/libevent-for-php
There is also a comment by Alix Axel on May 22 '11 at 12:19 regarding the same.
Update: A little more research led me to http://phpdaemon.net/ . It seems they are using libevent to build a whole host of network servers
Updated 30th May 2021:
Recently there has been https://www.swoole.co.uk/ & https://reactphp.org/ which provide good async libraries for sockets.
phpdaemon has a new url at https://daemon.io/
Updated 31st August 2022:
Swoole has now been forked into two separate projects: Open Swoole & Swoole

UDP Streaming transfer from C# application to PHP webpage

I'm trying to code a C# UDP server. It receives a specific ID from the client, and return the song associated with it. The client is a PHP webpage, and stocks the bytes received into a file. Right now I'm doing some tests, trying to simply start a fake lecture of the song (just a javascript alert) when the transfer is at 2048 bytes. But I have plenty of bugs... The PHP page seems to finish the transfer into the file BEFORE having received all the data... The server continue to send bytes but the file is complete, with the good weight and all...
I know I don't have a very good english, so if you don't undersood something, just ask !
Here is the C# code:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Threading;
using System.IO;
using System.Net.Sockets;
using System.Net;
using System.Data.SQLite;
namespace cdCollector
{
public partial class Streaming : Form
{
private static List<IPAddress> clients_ = new List<IPAddress>();
public Streaming()
{
InitializeComponent();
listen();
}
public class ThreadClient
{
private static UdpClient socket_;
private static IPEndPoint ipepClient_;
private static int noChanson_;
private static SQLiteConnection connexion_;
public void setSocket(ref UdpClient socket) { socket_ = socket; }
public void setIpepClient(ref IPEndPoint ipepClient) { ipepClient_ = ipepClient; }
public void setNoChanson(int noChanson) { noChanson_ = noChanson; }
public void setConnexion(ref SQLiteConnection connexion) { connexion_ = connexion; }
public static void send()
{
try
{
while (Thread.CurrentThread.IsAlive)
{
Chanson uneChanson;
FileStream stream;
byte[] buffer = new byte[1024];
int read;
uneChanson = new Chanson(noChanson_);
uneChanson.load(ref connexion_);
stream = new FileStream("C:\\Users\\Julie\\Documents\\toune.flac", FileMode.Open, FileAccess.Read);
socket_.Send(Encoding.ASCII.GetBytes(stream.Length.ToString()), stream.Length.ToString().Length, ipepClient_);
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
socket_.Send(buffer, buffer.Length, ipepClient_);
Console.WriteLine("finished");
}
}
catch (ThreadAbortException tae)
{ }
catch (Exception)
{
Thread.CurrentThread.Abort();
}
}
}
public static void listen()
{
byte[] data = new byte[1024];
IPEndPoint ipepServer = new IPEndPoint(IPAddress.Any, 7575); // IP du serveur
IPEndPoint ipepClient = new IPEndPoint(IPAddress.Any, 0); // IP du client;
UdpClient socket = new UdpClient(ipepServer); // socket serveur
int noChanson;
SQLiteConnection connexion = new SQLiteConnection("Data Source=" + Application.StartupPath + "\\cdCollector.db");
SQLiteCommand command = new SQLiteCommand(connexion);
SQLiteDataReader dr;
Thread thread;
connexion.Open();
while (true)
{
try
{
Console.WriteLine("Waiting for a client...");
data = socket.Receive(ref ipepClient);
Console.WriteLine("Message received from {0}:", ipepClient.ToString());
Console.WriteLine(Encoding.ASCII.GetString(data, 0, data.Length));
command.CommandText = "SELECT KeyLocale FROM AssocKeys WHERE NomTable = 'Chanson' AND KeyWeb = "
+ int.Parse(Encoding.ASCII.GetString(data, 0, data.Length));
dr = command.ExecuteReader();
if (dr.HasRows)
{
dr.Read();
noChanson = dr.GetInt32(0);
dr.Close();
ThreadClient client = new ThreadClient();
client.setConnexion(ref connexion);
client.setIpepClient(ref ipepClient);
client.setNoChanson(noChanson);
client.setSocket(ref socket);
thread = new Thread(new ThreadStart(ThreadClient.send));
thread.Start();
}
else
socket.Send(Encoding.ASCII.GetBytes("Erreur: Chanson introuvable"), ("Erreur: Chanson introuvable").Length, ipepClient);
}
catch (SocketException se)
{
Console.WriteLine("Erreur Socket:" + se.Message);
}
catch (Exception ex)
{
Console.WriteLine("Erreur: " + ex.Message);
}
}
connexion.Close();
}
}
}
And the PHP code:
<?php
session_start();
$address="192.168.2.2";
$read = false;
$port = 7575;
$length = 0;
$started = false;
if (isset($port) and
($socket=socket_create(AF_INET, SOCK_DGRAM, SOL_UDP)) and
(socket_connect($socket, $address, $port)))
{
$text = "Connection successful on IP $address, port $port <br />";
$from = '';
$port = 0;
$length = 0;
socket_send( $socket, $_GET['no'], 1024, MSG_EOR );
socket_recvfrom( $socket, $buf, 1024, 12, $from, $port);
$lengthTotal = $buf;
echo "Taille prévue du fichier: " . $lengthTotal . "<br />";
if( file_exists( "temp" . $_SESSION['ID_Membre'] . ".flac" ) )
unlink("temp" . $_SESSION['ID_Membre'] . ".flac");
$file = fopen("temp" . $_SESSION['ID_Membre'] . ".flac", 'a');
$buf = null;
while( $length < $lengthTotal )
{
$length += socket_recvfrom( $socket, $buf, 1024, 12, $from, $port );
if( $length > 2048 && !$started )
{
?>
<script type="text/javascript">
<!--
alert("Lecture...");
//->
</script>
<?php
$started = true;
}
fputs($file, $buf);
flush();
}
echo "<br />" . $length . "<br />";
fclose($file);
}
else
$text="Unable to connect<pre>".socket_strerror(socket_last_error())."</pre>";
echo $text;
?>
Thanks a lot !
UDP is an inherently unreliable transport. You will need to implement acknowledgements, timeouts, retransmissions and sequence numbers on top of UDP in order to guarantee transmission of all of your data in the expected order, unless your client application can live with dropped packets. I would advise you to consider using TCP sockets instead if you need guaranteed transmission of data between server and client and don't want to have to implement all of this stuff yourself (which might need to include client-side buffering to rearrange out-of-order datagrams). If you want reliability on top of UDP, I would advise you to read a good textbook on the subject (e.g. "Unix Network Programming" by W. Richard Stevens etc.).
Pointers on TCP:
You should take a look at System.Net.Sockets.TcpClient and System.Net.Sockets.TcpListener for the C# side of things and consult the PHP documentation for info on the PHP side of things.
Using TCP sockets isn't really that much different except you'll be using send and recv (or C#/PHP equivalents) instead of send_to and recv_from. Setting up the server side of things is a little more complicated since you need to bind and listen etc. but there are plenty of resources, e.g.:
http://www.switchonthecode.com/tutorials/csharp-tutorial-simple-threaded-tcp-server
Thanks for your help. I changed what you told me, except adding 'b' to the fopen mode because my web server is on Ubuntu. I still receive plenty of errors to tell me that the client connection had to close... It seems like PHP think the download is finished and exit the loop, so it closes the connection of the socket. Also, many minutes after the page have load, the server is still sending data... I never did streaming before so I have difficulties to see where the problem is ...
Here's the new PHP code:
<?php
session_start();
$address="192.168.2.2";
$read = false;
$port = 7575;
$length = 0;
$started = false;
if (isset($port) and
($socket=socket_create(AF_INET, SOCK_DGRAM, SOL_UDP)) and
(socket_connect($socket, $address, $port)))
{
$text = "Connection successful on IP $address, port $port <br />";
$from = '';
$port = 0;
$length = 0;
socket_send( $socket, $_GET['no'], 1024, MSG_EOR );
socket_recvfrom( $socket, $buf, 1024, MSG_WAITALL, $from, $port);
$lengthTotal = $buf;
echo "Taille prévue du fichier: " . $lengthTotal . "<br />";
if( file_exists( "temp" . $_SESSION['ID_Membre'] . ".flac" ) )
unlink("temp" . $_SESSION['ID_Membre'] . ".flac");
$file = fopen("temp" . $_SESSION['ID_Membre'] . ".flac", 'a');
$buf = null;
while( $length !== FALSE && $length < $lengthTotal )
{
$length += socket_recvfrom( $socket, $buf, 1024, 12, $from, $port );
if( $length > 2048 && !$started )
{
?>
<script type="text/javascript">
<!--
alert("Lecture...");
//->
</script>
<?php
$started = true;
}
if( $length == FALSE )
echo "ERREUR";
fputs($file, $buf, $length);
flush();
}
echo "<br />" . $length . "<br />";
fclose($file);
}
else
$text="Unable to connect<pre>".socket_strerror(socket_last_error())."</pre>";
echo $text;
?>
Some points:
1.- socket_recvfrom could return FALSE if there is any error, you can check the error with false === socket_recvfrom.
2.- If you are using a windows server add b to the open mode: $file = fopen("temp" . $_SESSION['ID_Membre'] . ".flac", 'ab'); (you are writing a binary file).
3.- Use as third argument of the fputs function the value returned by the socket_recvfrom function (if this value !== FALSE).
4.- You are using the value 12 (MSG_DONTROUTE | MSG_EOR), try to use 0 or MSG_WAITALL (of course socket_recvfrom is going to wait to receive 1024 bytes).
Your reception loop must be:
$reclen = 0;
while( ($reclen !== FALSE) && ($length < $lengthTotal) )
{
$reclen = socket_recvfrom( $socket, $buf, 1024, 12, $from, $port );
if ($reclen === FALSE)
{
echo "ERREUR";
break;
}
$length += $reclen;
if( $length > 2048 && !$started )
{
?>
<script type="text/javascript">
<!--
alert("Lecture...");
//->
</script>
<?php
$started = true;
}
fputs($file, $buf, $length);
flush();
}
The problem is that you are adding the value returned by socket_recvfrom to $length if the return value is FALSE is going to add 0 to $length, that is the reason why you have to add an additional variable ($reclength).

Categories