When long polling Azure Queue Storage with azure-sdk-for-php, if my request are more than 30 seconds apart, the library dies with this error:
PHP Notice: fwrite(): send of 277 bytes failed with errno=32 Broken pipe in ..../vendor/pear-pear.php.net/HTTP_Request2/HTTP/Request2/SocketWrapper.php on line 188
If I set the sleep function to 30 seconds, everything goes great, but I'm making LOTS of requests that I don't need.
My workers code:
use WindowsAzure\Common\ServicesBuilder;
use WindowsAzure\Common\ServiceException;
Class Worker_Task {
public $queueRestProxy;
public $servicesBuilder;
public $connectionString;
public function __construct() {
$this->connectionString = Config::get('azure.connection_string');
$this->servicesBuilder = ServicesBuilder::getInstance();
$this->queueRestProxy = $this->servicesBuilder->createQueueService($this->connectionString);
}
public function emails() {
$this->write('Processing mails...');
while(true) {
$this->queueRestProxy = $this->servicesBuilder->createQueueService($this->connectionString);
// Get message.
$listMessagesResult = $this->queueRestProxy->listMessages("emails");
$messages = $listMessagesResult->getQueueMessages();
foreach($messages as $message) {
// Process Message
$content = json_decode($message->getMessageText());
// Dispatch Email
// Delete Job
$this->deleteJob('emails', $message->getMessageId(), $message->getPopReceipt());
}
sleep(30);
}
}
}
This looks like overload or timeout problem. PHP library can't write to Azure socket. Probably you exceeding Azure load limits with infinite loop while(true) { ... }.
Don't create queueService at each loop because you closing and opening pear socket infinitely.
Try moving line:
$this->queueRestProxy = $this->servicesBuilder->createQueueService($this->connectionString);
before while(true) { :
$this->write('Processing mails...');
$this->queueRestProxy = $this->servicesBuilder->createQueueService($this->connectionString);
while(true) {
// Get message.
If that doesn't help then probably azure-sdk-for-php library incorrectly uses a socket mechanism - create new issue at https://github.com/WindowsAzure/azure-sdk-for-php OR use this approach: AzurePHP - Polling an Azure Queue
Related
I am trying to use papertrail as the logging channel in my laravel 8.X application. Whenever I use the following statement, it is throwing an exception
Log::channel('papertrail')->info('Something happened!');
socket_sendto(): unable to write to socket [1]: Operation not permitted
vendor/monolog/monolog/src/Monolog/Handler/SyslogUdp/UdpSocket.php:52
public function write($line, $header = "")
{
$this->send($this->assembleMessage($line, $header));
}
public function close(): void
{
if (is_resource($this->socket)) {
socket_close($this->socket);
$this->socket = null;
}
}
protected function send(string $chunk): void
{
if (!is_resource($this->socket)) {
throw new \RuntimeException('The UdpSocket to '.$this->ip.':'.$this->port.' has been closed and can not be written to anymore');
}
socket_sendto($this->socket, $chunk, strlen($chunk), $flags = 0, $this->ip, $this->port);
}
protected function assembleMessage(string $line, string $header): string
{
$chunkSize = static::DATAGRAM_MAX_LENGTH - strlen($header);
return $header . Utils::substr($line, 0, $chunkSize);
}
}
Please advice.
You are getting this error because the log info you are sending is too big. (It probably works most of the time, but some larger errors are too big).
Laravel is using by default UDP to send the log to Papertrail.
The Papertrail accepts logs that UDP packets of up to 1024 bytes / TCP that are up to 100,000 bytes (~100 KB) long.
https://www.papertrail.com/help/troubleshooting-remote-syslog-reachability/#message-length
A solution is to use TCP. Since Laravel 9 there is an env variable to override this:
LOG_PAPERTRAIL_HANDLER = SocketHandler::class logging.php
For older versions of Laravel, you can override the config/logging.php to use the SocketHandler::class in stead of the SyslogUdpHandler::class.
Remark: not all shared hosting providers allow to make a TLS connection.
I was trying to use Server Side Events mechanics in my project. (This is like Long Polling on steroids)
Example from "Sending events from the server" subtitle works beautifully. After few seconds, from disconnection, the apache process is killed. This method works fine.
BUT! If I try to use RabbitMQ, Apache does't get the process killed after browser disconnects from server (es.close()). And process leaves as is and gets killed only after the docker container restarts.
connection_aborted and connection_status don't work at all. connection_aborted returns only 0 and connection_status returns CONNECTION_NORMAL even after disconnect. It happens only when I use RabbitMQ. Without RMQ this functions works well.
ignore_user_abort(false) doesn't work either.
Code example:
<?php
use PhpAmqpLib\Channel\AMQPChannel;
use PhpAmqpLib\Connection\AbstractConnection;
use PhpAmqpLib\Exception\AMQPTimeoutException;
use PhpAmqpLib\Message\AMQPMessage;
class RequestsRabbit
{
protected $rabbit;
/** #var AMQPChannel */
protected $channel;
public $exchange = 'requests.events';
public function __construct(AbstractConnection $rabbit)
{
$this->rabbit = $rabbit;
}
public function getChannel()
{
if ($this->channel === null) {
$channel = $this->rabbit->channel();
$channel->exchange_declare($this->exchange, 'fanout', false, false, false);
$this->channel = $channel;
}
return $this->channel;
}
public function send($message)
{
$channel = $this->getChannel();
$message = json_encode($message);
$channel->basic_publish(new AMQPMessage($message), $this->exchange);
}
public function subscribe(callable $callable)
{
$channel = $this->getChannel();
list($queue_name) = $channel->queue_declare('', false, false, true, false);
$channel->queue_bind($queue_name, $this->exchange);
$callback = function (AMQPMessage $msg) use ($callable) {
call_user_func($callable, json_decode($msg->body));
};
$channel->basic_consume($queue_name, '', false, true, false, false, $callback);
while (count($channel->callbacks)) {
if (connection_aborted()) {
break;
}
try {
$channel->wait(null, true, 5);
} catch (AMQPTimeoutException $exception) {
}
}
$channel->close();
$this->rabbit->close();
}
}
What happens:
Browser establishes SSE connection to the server. var es = new EventSource(url);
Apache2 spawns new process to handle this request.
PHP generates a new Queue and connects to it.
Browser closes connection es.close()
Apache2 doesn't kill process and it stays as is. Queue of RabbitMQ will not be deleted. If I do some reconnections, it spawns a bunch of processes and a bunch of queues (1 reconnection = 1 process = 1 queue).
I close all tabs -- processes alive. I close browser -- the same situation.
Looks line some kind of PHP bug. Or of Apach2?
What I use:
Last Docker and docker-compose
php:7.1.12-apache or php:5.6-apache image (this happens on both versions of PHP)
Some screenshots:
Please, help me to figure out what's going on...
P.S. Sorry for my English. If you can find a mistake or typo, point to it in the comments. I'll be very grateful :)
You don't say if you're using send() or subscribe() (or both) during your server-side events. Assuming you're using subscribe() there is no bug. This loop:
while (count($channel->callbacks)) {
if (connection_aborted()) {
break;
}
try {
$channel->wait(null, true, 5);
} catch (AMQPTimeoutException $exception) {
}
}
Will run until the process is killed or the connection is remotely closed from RabbitMQ. This is normal when listening for queued messages. If you need to stop the loop at some point you can set a variable to check in the loop or throw an exception when the SSE is ended (although I find this awkward).
I used Symfony2 with the RabbitMqBundle to create a worker that sends documents to ElasticSearch. Indexing documents at a one-by-one rate is much slower than using the ElasticSearch bulk API. Therefore I created a buffer that flushes the documents to ES in groups of thousand. The code looks (a bit simplified) as follows:
class SearchIndexator
{
protected $elasticaService;
protected $buffer = [];
protected $bufferSize = 0;
// The maximum number of documents to keep in the buffer.
// If the buffer reaches this amount of documents, then the buffers content
// is send to elasticsearch for indexation.
const MAX_BUFFER_SIZE = 1000;
public function __construct(ElasticaService $elasticaService)
{
$this->elasticaService = $elasticaService;
}
/**
* Destructor
*
* Flush any documents that remain in the buffer.
*/
public function __destruct()
{
$this->flush();
}
/**
* Add a document to the indexation buffer.
*/
public function onMessage(array $document)
{
// Prepare the document for indexation.
$this->doHeavyWeightStuff($document);
// Create an Elastica document
$document = new \Elastica\Document(
$document['key'],
$document
);
// Add the document to the buffer.
$this->buffer[] = $document;
// Flush the buffer when max buffersize has been reached.
if (self::MAX_BUFFER_SIZE <= ++$this->bufferSize) {
$this->flush();
}
}
/**
* Send the current buffer to ElasticSearch for indexation.
*/
public function flush()
{
// Send documents to ElasticSearch for indexation.
if (1 <= $this->bufferSize) {
$this->elasticaService->addDocuments($this->buffer);
}
// Clear buffer
$this->buffer = [];
$this->bufferSize = 0;
}
}
This all works quite nice but there is a slight problem. The queue gets filled with messages at an unpredictable rate. Sometimes 100000 in 5 minutes, sometimes not one for hours. When there are for instance 82671 documents queued, the last 671 documents don't get indexated before receiving another 329 documents which might take hours. I would like to be able to do the following:
Warning: Sci-Fi code! This obviously won't work:
class SearchIndexator
{
protected $elasticaService;
protected $buffer = [];
protected $bufferSize = 0;
protected $flushTimer;
// The maximum number of documents to keep in the buffer.
// If the buffer reaches this amount of documents, then the buffers content
// is send to elasticsearch for indexation.
const MAX_BUFFER_SIZE = 1000;
public function __construct(ElasticaService $elasticaService)
{
$this->elasticaService = $elasticaService;
// Highly Sci-fi code
$this->flushTimer = new Timer();
// Flush buffer after 5 minutes of inactivity.
$this->flushTimer->setTimeout(5 * 60);
$this->flushTimer->setCallback([$this, 'flush']);
}
/**
* Destructor
*
* Flush any documents that remain in the buffer.
*/
public function __destruct()
{
$this->flush();
}
/**
* Add a document to the indexation buffer.
*/
public function onMessage(array $document)
{
// Prepare the document for indexation.
$this->doHeavyWeightStuff($document);
// Create an Elastica document
$document = new \Elastica\Document(
$document['key'],
$document
);
// Add the document to the buffer.
$this->buffer[] = $document;
// Flush the buffer when max buffersize has been reached.
if (self::MAX_BUFFER_SIZE <= ++$this->bufferSize) {
$this->flush();
} else {
// Start a timer that will flush the buffer after a timeout.
$this->initTimer();
}
}
/**
* Send the current buffer to ElasticSearch for indexation.
*/
public function flush()
{
// Send documents to ElasticSearch for indexation.
if (1 <= $this->bufferSize) {
$this->elasticaService->addDocuments($this->buffer);
}
// Clear buffer
$this->buffer = [];
$this->bufferSize = 0;
// There are no longer messages to be send, stop the timer.
$this->flushTimer->stop();
}
protected function initTimer()
{
// Start or restart timer
$this->flushTimer->isRunning()
? $this->flushTimer->reset()
: $this->flushTimer->start();
}
}
Now, I know about the limitations of PHP not being event driven. But this is 2015 and there are solutions like ReactPHP, so this should be possible right? For ØMQ there is this function. What would be a solution that will work for RabbitMQ or independent of any message queue extension?
Solutions that I'm skeptical about:
There is crysalead/code. It simulates a timer using declare(ticks = 1);. I'm not sure wether this is a performant and solid approach. Any ideas?
I could run a cronjob that publishes a 'FLUSH' message to the same queue every 5 minutes and then explicitly flush the buffer when receiving this message but that would be cheating.
As I've mentioned in my comment you could use the signals. PHP allows you to register signal handlers to your scripts signals (i.e. SIGINT, SIGKILL etc.)
For your use-case you can use the SIGALRM signal. This signal will alarm your script after a certain time (that you can set) has expired. Positive side of these signals is that they are non-blocking. In other words the normal operation of your script will not be interfered with.
The adjusted solution (ticks are deprecated since PHP 5.3):
function signal_handler($signal) {
// You would flush here
print "Caught SIGALRM\n";
// Set the SIGALRM timer again or it won't trigger again
pcntl_alarm(300);
}
// register your handler with the SIGALRM signal
pcntl_signal(SIGALRM, "signal_handler", true);
// set the timeout for the SIGALRM signal to 300 seconds
pcntl_alarm(300);
// start loop and check for pending signals
while(pcntl_signal_dispatch() && your_loop_condition) {
//Execute your code here
}
Note: you can only use 1 SIGALRM signal in your script, if you set the time of your signal with pcntl_alarm the timer for your alarm will reset (without firing the signal) to its newly set value.
I have a function that needs to go over around 20K rows from an array, and apply an external script to each. This is a slow process, as PHP is waiting for the script to be executed before continuing with the next row.
In order to make this process faster I was thinking on running the function in different parts, at the same time. So, for example, rows 0 to 2000 as one function, 2001 to 4000 on another one, and so on. How can I do this in a neat way? I could make different cron jobs, one for each function with different params: myFunction(0, 2000), then another cron job with myFunction(2001, 4000), etc. but that doesn't seem too clean. What's a good way of doing this?
If you'd like to execute parallel tasks in PHP, I would consider using Gearman. Another approach would be to use pcntl_fork(), but I'd prefer actual workers when it's task based.
The only waiting time you suffer is between getting the data and processing the data. Processing the data is actually completely blocking anyway (you just simply have to wait for it). You will not likely gain any benefits past increasing the number of processes to the number of cores that you have. Basically I think this means the number of processes is small so scheduling the execution of 2-8 processes doesn't sound that hideous. If you are worried about not being able to process data while retrieving data, you could in theory get your data from the database in small blocks, and then distribute the processing load between a few processes, one for each core.
I think I align more with the forking child processes approach for actually running the processing threads. There is a brilliant demonstration in the comments on the pcntl_fork doc page showing an implementation of a job daemon class
http://php.net/manual/en/function.pcntl-fork.php
<?php
declare(ticks=1);
//A very basic job daemon that you can extend to your needs.
class JobDaemon{
public $maxProcesses = 25;
protected $jobsStarted = 0;
protected $currentJobs = array();
protected $signalQueue=array();
protected $parentPID;
public function __construct(){
echo "constructed \n";
$this->parentPID = getmypid();
pcntl_signal(SIGCHLD, array($this, "childSignalHandler"));
}
/**
* Run the Daemon
*/
public function run(){
echo "Running \n";
for($i=0; $i<10000; $i++){
$jobID = rand(0,10000000000000);
while(count($this->currentJobs) >= $this->maxProcesses){
echo "Maximum children allowed, waiting...\n";
sleep(1);
}
$launched = $this->launchJob($jobID);
}
//Wait for child processes to finish before exiting here
while(count($this->currentJobs)){
echo "Waiting for current jobs to finish... \n";
sleep(1);
}
}
/**
* Launch a job from the job queue
*/
protected function launchJob($jobID){
$pid = pcntl_fork();
if($pid == -1){
//Problem launching the job
error_log('Could not launch new job, exiting');
return false;
}
else if ($pid){
// Parent process
// Sometimes you can receive a signal to the childSignalHandler function before this code executes if
// the child script executes quickly enough!
//
$this->currentJobs[$pid] = $jobID;
// In the event that a signal for this pid was caught before we get here, it will be in our signalQueue array
// So let's go ahead and process it now as if we'd just received the signal
if(isset($this->signalQueue[$pid])){
echo "found $pid in the signal queue, processing it now \n";
$this->childSignalHandler(SIGCHLD, $pid, $this->signalQueue[$pid]);
unset($this->signalQueue[$pid]);
}
}
else{
//Forked child, do your deeds....
$exitStatus = 0; //Error code if you need to or whatever
echo "Doing something fun in pid ".getmypid()."\n";
exit($exitStatus);
}
return true;
}
public function childSignalHandler($signo, $pid=null, $status=null){
//If no pid is provided, that means we're getting the signal from the system. Let's figure out
//which child process ended
if(!$pid){
$pid = pcntl_waitpid(-1, $status, WNOHANG);
}
//Make sure we get all of the exited children
while($pid > 0){
if($pid && isset($this->currentJobs[$pid])){
$exitCode = pcntl_wexitstatus($status);
if($exitCode != 0){
echo "$pid exited with status ".$exitCode."\n";
}
unset($this->currentJobs[$pid]);
}
else if($pid){
//Oh no, our job has finished before this parent process could even note that it had been launched!
//Let's make note of it and handle it when the parent process is ready for it
echo "..... Adding $pid to the signal queue ..... \n";
$this->signalQueue[$pid] = $status;
}
$pid = pcntl_waitpid(-1, $status, WNOHANG);
}
return true;
}
}
you can use "PTHREADS"
very easy to install and works great on windows
download from here -> http://windows.php.net/downloads/pecl/releases/pthreads/2.0.4/
Extract the zip file and then
move the file 'php_pthreads.dll' to php\ext\ directory.
move the file 'pthreadVC2.dll' to php\ directory.
then add this line in your 'php.ini' file:
extension=php_pthreads.dll
save the file.
you just done :-)
now lets see example of how to use it:
class ChildThread extends Thread {
public $data;
public function run() {
/* Do some expensive work */
$this->data = 'result of expensive work';
}
}
$thread = new ChildThread();
if ($thread->start()) {
/*
* Do some expensive work, while already doing other
* work in the child thread.
*/
// wait until thread is finished
$thread->join();
// we can now even access $thread->data
}
for more information about PTHREADS read php docs here:
PHP DOCS PTHREADS
if you'r using WAMP like me, then you should add 'pthreadVC2.dll' into
\wamp\bin\apache\ApacheX.X.X\bin
and also edit the 'php.ini' file (same path) and add the same line as before
extension=php_pthreads.dll
GOOD LUCK!
What you are looking for is parallel which is a succinct concurrency API for PHP 7.2+
$runtime = new \parallel\Runtime();
$future = $runtime->run(function() {
for ($i = 0; $i < 500; $i++) {
echo "*";
}
return "easy";
});
for ($i = 0; $i < 500; $i++) {
echo ".";
}
printf("\nUsing \\parallel\\Runtime is %s\n", $future->value());
Output:
.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*..*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*.*
Using \parallel\Runtime is easy
Have a look at pcntl_fork. This allows you to spawn child processes which can then do the separate work that you need.
Not sure if a solution for your situation but you can redirect the output of system calls to a file, thus PHP will not wait until the program is finished. Although this may result in overloading your server.
http://www.php.net/manual/en/function.exec.php - If a program is started with this function, in order for it to continue running in the background, the output of the program must be redirected to a file or another output stream. Failing to do so will cause PHP to hang until the execution of the program ends.
There's Guzzle with its concurrent requests
use GuzzleHttp\Client;
use GuzzleHttp\Promise;
$client = new Client(['base_uri' => 'http://httpbin.org/']);
$promises = [
'image' => $client->getAsync('/image'),
'png' => $client->getAsync('/image/png'),
'jpeg' => $client->getAsync('/image/jpeg'),
'webp' => $client->getAsync('/image/webp')
];
$responses = Promise\Utils::unwrap($promises);
There's the overhead of promises; but more importantly Guzzle only works with HTTP requests and it works with version 7+ and frameworks like Laravel.
I'm trying to create a PHP file, which wouldn't run if it's already running. Here's the code I'm using:
<?php
class Test {
private $tmpfile;
public function action_run() {
$this->die_if_running();
$this->run();
}
private function die_if_running() {
$this->tmpfile = #fopen('.refresher2.pid', "w");
$locked = #flock($this->tmpfile, LOCK_EX|LOCK_NB);
if (! $locked) {
#fclose($this->tmpfile);
die("Running 2");
}
}
private function run() {
echo "NOT RUNNNING";
sleep(100);
}
}
$test = new Test();
$test->action_run();
The problem is, when I run this from console, it works great. But when I try to run it from browser, many instances can run simultaneously. This is on Windows 7, XAMPP, PHP 5.3.2. I guess OS is thinking that it's the same process and thus the functionality falls. Is there a cross-platform way to create a PHP script of this type?
Not really anything to promising. You can't use flock for that like this.
You could use system() to start another (php) process that does the locking for you. But drawbacks:
You need to do interprocess communication. Think about a way how to tell the other program when to release the lock etc. You can use stdin for messenging und use 3 constants or something. In this case it's still rather simple
It's bad for performance because you keep creating processes which is expensive.
Another way would be to start another program that runs all the time. You connect to it using some means of IPC (probably just use a tcp channel because it's cross-platform) and allow this program to manage file acces. That program could be a php script in an endless loop as well, but it will probably be simpler to code this in Java or another language that has multithreading support.
Another way would be to leverage existing ressources. Create a dummy database table for locks, create an entry for the file and then do table-row-locking.
Another way would be not to use files, but a database.
I had a similar problem a while ago.
I needed to have a counter where the number returned was unique.
I used a lock-file and only if this instance was able to create the lock-file was it allowed to read the file with the current number.
Instead of counting up perhaps you can allow the script to run.
The trick is to let try a few times (like 5) with a small wait/sleep in between.
function GetNextNumber()
{
$lockFile = "lockFile.txt";
$lfh = #fopen($lockFile, "x");
if (!$lfh)
{
$lockOkay = false;
$count = 0;
$countMax = 5;
// Try ones every second in 5 seconds
while (!$lockOkay & $count < $countMax)
{
$lfh = #fopen($lockFile, "x");
if ($lfh)
{
$lockOkay = true;
}
else
{
$count++;
sleep(1);
}
}
}
if ($lfh)
{
$fh = fopen($myFile, 'r+') or die("Too many users. ");
flock($fh, LOCK_EX);
$O_nextNumber = fread($fh, 15);
$O_nextNumber = $O_nextNumber + 1;
rewind($fh);
fwrite($fh, $O_knr);
flock($fh, LOCK_UN);
fclose($fh);
unlink($lockFile); // Sletter lockfilen
}
return $O_nextNumber;
}