Run function in PHP CLI script after period of inactivity - php

Question

I used Symfony2 with the RabbitMqBundle to create a worker that sends documents to ElasticSearch. Indexing documents at a one-by-one rate is much slower than using the ElasticSearch bulk API. Therefore I created a buffer that flushes the documents to ES in groups of thousand. The code looks (a bit simplified) as follows:
class SearchIndexator
{
protected $elasticaService;
protected $buffer = [];
protected $bufferSize = 0;
// The maximum number of documents to keep in the buffer.
// If the buffer reaches this amount of documents, then the buffers content
// is send to elasticsearch for indexation.
const MAX_BUFFER_SIZE = 1000;
public function __construct(ElasticaService $elasticaService)
{
$this->elasticaService = $elasticaService;
}
/**
* Destructor
*
* Flush any documents that remain in the buffer.
*/
public function __destruct()
{
$this->flush();
}
/**
* Add a document to the indexation buffer.
*/
public function onMessage(array $document)
{
// Prepare the document for indexation.
$this->doHeavyWeightStuff($document);
// Create an Elastica document
$document = new \Elastica\Document(
$document['key'],
$document
);
// Add the document to the buffer.
$this->buffer[] = $document;
// Flush the buffer when max buffersize has been reached.
if (self::MAX_BUFFER_SIZE <= ++$this->bufferSize) {
$this->flush();
}
}
/**
* Send the current buffer to ElasticSearch for indexation.
*/
public function flush()
{
// Send documents to ElasticSearch for indexation.
if (1 <= $this->bufferSize) {
$this->elasticaService->addDocuments($this->buffer);
}
// Clear buffer
$this->buffer = [];
$this->bufferSize = 0;
}
}
This all works quite nice but there is a slight problem. The queue gets filled with messages at an unpredictable rate. Sometimes 100000 in 5 minutes, sometimes not one for hours. When there are for instance 82671 documents queued, the last 671 documents don't get indexated before receiving another 329 documents which might take hours. I would like to be able to do the following:
Warning: Sci-Fi code! This obviously won't work:
class SearchIndexator
{
protected $elasticaService;
protected $buffer = [];
protected $bufferSize = 0;
protected $flushTimer;
// The maximum number of documents to keep in the buffer.
// If the buffer reaches this amount of documents, then the buffers content
// is send to elasticsearch for indexation.
const MAX_BUFFER_SIZE = 1000;
public function __construct(ElasticaService $elasticaService)
{
$this->elasticaService = $elasticaService;
// Highly Sci-fi code
$this->flushTimer = new Timer();
// Flush buffer after 5 minutes of inactivity.
$this->flushTimer->setTimeout(5 * 60);
$this->flushTimer->setCallback([$this, 'flush']);
}
/**
* Destructor
*
* Flush any documents that remain in the buffer.
*/
public function __destruct()
{
$this->flush();
}
/**
* Add a document to the indexation buffer.
*/
public function onMessage(array $document)
{
// Prepare the document for indexation.
$this->doHeavyWeightStuff($document);
// Create an Elastica document
$document = new \Elastica\Document(
$document['key'],
$document
);
// Add the document to the buffer.
$this->buffer[] = $document;
// Flush the buffer when max buffersize has been reached.
if (self::MAX_BUFFER_SIZE <= ++$this->bufferSize) {
$this->flush();
} else {
// Start a timer that will flush the buffer after a timeout.
$this->initTimer();
}
}
/**
* Send the current buffer to ElasticSearch for indexation.
*/
public function flush()
{
// Send documents to ElasticSearch for indexation.
if (1 <= $this->bufferSize) {
$this->elasticaService->addDocuments($this->buffer);
}
// Clear buffer
$this->buffer = [];
$this->bufferSize = 0;
// There are no longer messages to be send, stop the timer.
$this->flushTimer->stop();
}
protected function initTimer()
{
// Start or restart timer
$this->flushTimer->isRunning()
? $this->flushTimer->reset()
: $this->flushTimer->start();
}
}
Now, I know about the limitations of PHP not being event driven. But this is 2015 and there are solutions like ReactPHP, so this should be possible right? For ØMQ there is this function. What would be a solution that will work for RabbitMQ or independent of any message queue extension?
Solutions that I'm skeptical about:
There is crysalead/code. It simulates a timer using declare(ticks = 1);. I'm not sure wether this is a performant and solid approach. Any ideas?
I could run a cronjob that publishes a 'FLUSH' message to the same queue every 5 minutes and then explicitly flush the buffer when receiving this message but that would be cheating.

Answer 1

As I've mentioned in my comment you could use the signals. PHP allows you to register signal handlers to your scripts signals (i.e. SIGINT, SIGKILL etc.)
For your use-case you can use the SIGALRM signal. This signal will alarm your script after a certain time (that you can set) has expired. Positive side of these signals is that they are non-blocking. In other words the normal operation of your script will not be interfered with.
The adjusted solution (ticks are deprecated since PHP 5.3):
function signal_handler($signal) {
// You would flush here
print "Caught SIGALRM\n";
// Set the SIGALRM timer again or it won't trigger again
pcntl_alarm(300);
}
// register your handler with the SIGALRM signal
pcntl_signal(SIGALRM, "signal_handler", true);
// set the timeout for the SIGALRM signal to 300 seconds
pcntl_alarm(300);
// start loop and check for pending signals
while(pcntl_signal_dispatch() && your_loop_condition) {
//Execute your code here
}
Note: you can only use 1 SIGALRM signal in your script, if you set the time of your signal with pcntl_alarm the timer for your alarm will reset (without firing the signal) to its newly set value.