I need to notify the client(s) whether some of the clients made changes on the database. I consider it to be a single-directional connection (it's only the server who sends events), so I do not need WebSockets.
The logic is as follows:
Client fetches the common API (let's say /api.php?action=add&payload=...), and with this fetch the change on DB is being made.
After mysqli_result returned true, the server-sent event should be triggered to notify other clients (or their service-workers) about DB changed.
The caveats are:
My SSE logic is placed on a separate file (/sse.php) to not make a mess in API logic and to listen to separate endpoint clientside. So I need to trigger SSE from api.php and push the result from there to all clients.
In sse.php I defined a function for pushing which takes the message from api.php as a param. Whatever I've tried, this function was never called.
I strongly want to avoid making daemons and any kind of infinite loops. But without them, the connection for the clientside listener (eventSource) closes and reopens every 3 seconds.
The current state of this all:
api.php
<?php
header("Access-Control-Allow-Origin: *");
include 'connection.php'; // mysqli credentials and connection object
include_once 'functions.php'; // logics
$action = isset($_GET['a']) ? $_GET['a'] : 'fail';
switch ($action) {
case "add":
$result = api__handle_add(); // adds data to DB and returns stringified success/failure result
require "./sse.php"; // file with SSE logic with ev__ namespace
ev__send_data($result); // this defined function should send message to clients
break;
// ...
case "testquery":
require "./sse.php";
ev__send_data("Test event!!! Ololo");
break;
case "fail": default:
header('HTTP/1.1 403 Forbidden', true, 403);
}
sse.php
<?php
header("Access-Control-Allow-Origin: *");
header("Cache-Control: no-cache");
header("Content-Type: text/event-stream");
header("Connection: keep-alive");
$_fired = false; // flag checking for is anything happening right now
while (1) { // the infinite loop I strongly want to get rid of
if (!$_fired) {
usleep(5000); // nothing to send if nothing is fired, but I should keep the connection alive somehow
} else {
// here I should call 'ev__send_data($message_from_api_php)'
// thus I require this file to `api.php` and try to call 'ev__send_data' from there.
}
ob_flush();
flush();
}
function ev__send_data(string $msg) {
global $_fired;
$_fired = true;
echo "data: $msg". PHP_EOL;
echo PHP_EOL;
ob_flush();
flush();
usleep(1000);
$_fired = false;
}
The clientside implementation is typical, with onopen, onerror and onmessage handlers of eventSource. I think I don't need to take it here as it has no differences from thousands of examples I've seen here on SO and at other sources.
To conclude: I want to trigger SSE's from outside SSE-file and pass pieces of data from outside, and I want to get rid of while (1) {} and emit events only when something really happens. I've come across thousands of tutorials and SO topics, and found nothing but typical "tutorial cases" about how to send server time to the client every n seconds and stuff. And I think I don't need WebSockets (considering that my local dev environment is Win10 + XAMPP, my prod environment will be something Linux-like and I don't think I can implement my own WebSocket without third-party dependencies which I don't want to install).
Do I have any chance?
EDIT
I found a little different approach. Now my server event generation depends on boolean flags that were settled in MySQL database in separate flags table. When I need an event to be emitted, API changes definite flag (for the very this example, when one client had submitted some changes in DB, API also makes the query to flags table and sets flag DB_CHANGED to 1). And sse.php makes query to flags table each iteration of the infinite loop (every 5 seconds) and emits events depending on the flags' state, and after emitting, sets the corresponding flag to 0 via one more query. This kinda works.
Example of current sse.php:
<?php
header("Cache-Control: no-cache");
header("Content-Type: text/event-stream");
require_once "connection.php"; // mysqli connection object
set_time_limit(600);
stream();
function stream() {
global $connection;
while (1) { // oh that infinite loop
$flags = []; // there will be fetched flags from DB
$flags_sqli = $connection->query("SELECT * FROM `flags`");
while ($flag = $flags_sqli->fetch_assoc()) {
$flags[$flag['key_']] = ev__h__to_bool_or_string($flag['value_']);
}
if ($flags['ON_TEST']) {
ev__send_data("Test event!!! Ololo");
ev__update_flag("ON_TEST", "0");
} elseif ($flags['ON_DB_ADD']) {
ev__handle_add();
} else {
echo PHP_EOL; // keep connection alive - push single "end of line"
}
ob_flush();
flush();
sleep(5); // interval per iteration - 5 secs
}
}
function ev__send_data(string $msg) : void {
echo "data: ".date("d-m-Y")." ".date("H:i:s").PHP_EOL;
echo "data: $msg".PHP_EOL;
echo PHP_EOL;
}
function ev__handle_add() : void {
global $connection;
$table = $connection->query('SELECT `value_` FROM `flags` WHERE `key_` = "LAST_ADDED_TABLE"')->fetch_row()[0];
$query = "SELECT * FROM `banners_".$table."` WHERE `id` = (SELECT MAX(`id`) FROM `banners_".$table."`)";
$row = $connection->query($query)->fetch_assoc();
echo "event: db_add_row".PHP_EOL;
echo 'data: { "manager": '.$row['manager_id'].', "banner_name": "'.$row['name'].'", "time": "'.date("d-m-Y").' '.date("H:i:s").'" }'.PHP_EOL;
echo PHP_EOL;
ev__update_flag("ON_DB_ADD", "0");
}
function ev__update_flag(string $key, string $value) {
global $connection;
$query = "UPDATE `flags` SET `value_` = '$value' WHERE `flags`.`key_` = '$key';";
$connection->query($query);
}
function ev__h__to_bool_or_string($value) {
return $value === "0" || $value === "1"
? boolval(intval($value))
: $value;
}
Therefore, I didn't get rid of the infinite loop. Considering that there will be a very few concurrent connections (max 3-5 sessions per one time) (when in prod), this very case should not make performance troubles. But what if it would be some kind of hard-loaded service? As an answer, I want to see optimization tips for the case of Server-Sent Events.
How should I multithread some php-cli code that needs a timeout?
I'm using PHP 5.6 on Centos 6.6 from the command line.
I'm not very familiar with multithreading terminology or code. I'll simplify the code here but it is 100% representative of what I want to do.
The non-threaded code currently looks something like this:
$datasets = MyLibrary::getAllRawDataFromDBasArrays();
foreach ($datasets as $dataset) {
MyLibrary::processRawDataAndStoreResultInDB($dataset);
}
exit; // just for clarity
I need to prefetch all my datasets, and each processRawDataAndStoreResultInDB() cannot fetch it's own dataset. Sometimes processRawDataAndStoreResultInDB() takes too long to process a dataset, so I want to limit the amount of time it has to process it.
So you can see that making it multithreaded would
Speed it up by allowing multiple processRawDataAndStoreResultInDB() to execute at the same time
Use set_time_limit() to limit the amount of time each one has to process each dataset
Notice that I don't need to come back to my main program. Since this is a simplification, you can trust that I don't want to collect all the processed datasets and do a single save into the DB after they are all done.
I'd like to do something like:
class MyWorkerThread extends SomeThreadType {
public function __construct($timeout, $dataset) {
$this->timeout = $timeout;
$this->dataset = $dataset;
}
public function run() {
set_time_limit($this->timeout);
MyLibrary::processRawDataAndStoreResultInDB($this->dataset);
}
}
$numberOfThreads = 4;
$pool = somePoolClass($numberOfThreads);
$pool->start();
$datasets = MyLibrary::getAllRawDataFromDBasArrays();
$timeoutForEachThread = 5; // seconds
foreach ($datasets as $dataset) {
$thread = new MyWorkerThread($timeoutForEachThread, $dataset);
$thread->addCallbackOnTerminated(function() {
if ($this->isTimeout()) {
MyLibrary::saveBadDatasetToDb($dataset);
}
}
$pool->addToQueue($thread);
}
$pool->waitUntilAllWorkersAreFinished();
exit; // for clarity
From my research online I've found the PHP extension pthreads which I can use with my thread-safe php CLI, or I could use the PCNTL extension or a wrapper library around it (say, Arara/Process)
https://github.com/krakjoe/pthreads (and the example directory)
https://github.com/Arara/Process (pcntl wrapper)
When I look at them and their examples though (especially the pthreads pool example) I get confused quickly by the terminology and which classes I should use to achieve the kind of multithreading I'm looking for.
I even wouldn't mind creating the pool class myself, if I had a isRunning(), isTerminated(), getTerminationStatus() and execute() function on a thread class, as it would be a simple queue.
Can someone with more experience please direct me to which library, classes and functions I should be using to map to my example above? Am I taking the wrong approach completely?
Thanks in advance.
Here comes an example using worker processes. I'm using the pcntl extension.
/**
* Spawns a worker process and returns it pid or -1
* if something goes wrong.
*
* #param callback function, closure or method to call
* #return integer
*/
function worker($callback) {
$pid = pcntl_fork();
if($pid === 0) {
// Child process
exit($callback());
} else {
// Main process or an error
return $pid;
}
}
$datasets = array(
array('test', '123'),
array('foo', 'bar')
);
$maxWorkers = 1;
$numWorkers = 0;
foreach($datasets as $dataset) {
$pid = worker(function () use ($dataset) {
// Do DB stuff here
var_dump($dataset);
return 0;
});
if($pid !== -1) {
$numWorkers++;
} else {
// Handle fork errors here
echo 'Failed to spawn worker';
}
// If $maxWorkers is reached we need to wait
// for at least one child to return
if($numWorkers === $maxWorkers) {
// $status is passed by reference
$pid = pcntl_wait($status);
echo "child process $pid returned $status\n";
$numWorkers--;
}
}
// (Non blocking) wait for the remaining childs
while(true) {
// $status is passed by reference
$pid = pcntl_wait($status, WNOHANG);
if(is_null($pid) || $pid === -1) {
break;
}
if($pid === 0) {
// Be patient ...
usleep(50000);
continue;
}
echo "child process $pid returned $status\n";
}
I have a simple AJAX call that retrieves text from a file, pushes it into a table, and displays it. The call works without issue when testing on a Mac running Apache 2.2.26/PHP 5.3 and on an Ubuntu box running Apache 2.2.1.6/PHP 5.3. It does not work on RedHat running Apache 2.2.4/PHP 5.1. Naturally, the RedHat box is the only place where I need it to be working.
The call returns 200 OK but no content. Even if nothing is found in the file (or it's inaccessible), the table header is echoed so if permissions were a problem I would still expect to see something. But to be sure, I verified the file is readable by all users.
Code has been redacted and simplified.
My ajax function:
function ajax(page,targetElement,ajaxFunction,getValues)
{
xmlhttp=new XMLHttpRequest();
xmlhttp.onreadystatechange=function()
{
if (xmlhttp.readyState===4 && xmlhttp.status===200)
{
document.getElementById(targetElement).innerHTML=xmlhttp.responseText;
}
};
xmlhttp.open('GET','/appdir/dir/filedir/'+page+'_funcs.php?function='+ajaxFunction+'&'+getValues+'&'+new Date().getTime(),false);
xmlhttp.setRequestHeader('cache-control','no-cache');
xmlhttp.send();
}
I call it like this:
ajax('pagename','destelement','load_info');
And return the results:
// Custom file handler
function warn_error($errno, $errstr) {
// Common function for warning-prone functions
throw new Exception($errstr, $errno);
}
function get_file_contents() {
// File operation failure would return a warning
// So handle specially to suppress the default message
set_error_handler('warn_error');
try
{
$fh = fopen(dirname(dirname(__FILE__))."/datafile.txt","r");
}
catch (Exception $e)
{
// Craft a nice-looking error message and get out of here
$info = "<tr><td class=\"center\" colspan=\"9\"><b>Fatal Error: </b>Could not load customer data.</td></tr>";
restore_error_handler();
return $info;
}
restore_error_handler();
// Got the file so get and return its contents
while (!feof($fh))
{
$line = fgets($fh);
// Be sure to avoid empty lines in our array
if (!empty($line))
{
$info[] = explode(",",$line);
}
}
fclose($fh);
return $info;
}
function load_info() {
// Start the table
$content .= "<table>
<th>Head1</th>
<th>Head2</th>
<th>Head3</th>
<th>Head4</th>";
// Get the data
// Returns all contents in an array if successful,
// Returns an error string if it fails
$info = get_file_contents();
if (!is_array($info))
{
// String was returned because of an error
echo $content.$info;
exit();
}
// Got valid data array, so loop through it to build the table
foreach ($info as $detail)
{
list($field1,$field2,$field3,$field4) = $detail;
$content .= "<tr>
<td>$field1</td>
<td>$field2</td>
<td>$field3</td>
<td>$field4</td>
</tr>";
}
$content .= "</table>";
echo $content;
}
Where it works, the response header indicates the connection as keep-alive; where it fails, the connection is closed. I don't know if that matters.
I've looked all over SO and the net for some clues but "no content" issues invariably point to same-origin policy problems. In my case, all content is on the same server.
I'm at a loss as to what to do/where to look next.
file_get_contents() expects a parameter. It does not know what you want, so it returned false. Also, you used get_file_contents() which is the wrong order.
This turned out to be a PHP version issue. In the load_info function I was using filter_input(INPUT_GET,"value"), but that was not available in PHP 5.1. I pulled that from my initial code post because I didn't think it was part of the problem. Lesson learned.
I am trying to build an configuration parser for my application I installed APC today, but everytime I try to put an serialized object in the store, it does not get in there and does not. (I am checking with apc.php for my version[3.1.8-dev] on PHP 5.3.16 [My Dev Environment], so I am sure that the data is not in the cache). this is how I pass the data to the cacher:
// The data before the caching
array (
'key' => md5($this->filename),
'value' => serialize($this->cfg)
);
// The caching interface
function($argc){
$key = $argc['key'];
Cache\APC::getInstance()->set($key,$argc['value']);
}
// The caching method described above
public function set($key, $val) {
if (apc_exists($key)) {
apc_delete ($key);
return apc_store($key, $val);
}
else
return false;
}
// the constructor of the configuration class.
// It 1st looks for the configuration in
// the cache if it is not present performs the reading from the file.
public function __construct($filename = '/application/config/application.ini',
$type = self::CONFIG_INI)
{
if (defined('SYSTEM_CACHE') && SYSTEM_CACHE === 'APC'){
$key = md5($filename);
$cfg = APC::getInstance()->get($key);
if (!empty($cfg)) {
print "From Cache";
$this->cfg = unserialize($cfg);
return;
} else {
print "From File";
}
}
}
I did a few tests and there is not a problem with the MD5() key (which I thought while writing this question) nor with APC itself. I am really stuck on this one, nothing odd in the logs, so if anyone can give me at least some directions will be very appreciated.
Thanks in advance!
The problem is was in my code:\
public function set($key, $val) {
/*
*
* If the key exists in the cache delete it and store it again,
* but how could it exist when the else clause is like that...
*/
if (apc_exists($key)) {
apc_delete ($key);
return apc_store($key, $val);
}
// This is very wrong in the current case
// cuz the function will never store and the if will
// never match..
else
return false;
}
NOTE:
Always think and keep your eyes open, if you still can't find anything get off the PC and give yourself a rest. Get back after 10-15 minutes and pown the code. It helps! :D
I'm trying to update a variable in APC, and will be many processes trying to do that.
APC doesn't provide locking functionality, so I'm considering using other mechanisms... what I've found so far is mysql's GET_LOCK(), and php's flock(). Anything else worth considering?
Update: I've found sem_acquire, but it seems to be a blocking lock.
/*
CLASS ExclusiveLock
Description
==================================================================
This is a pseudo implementation of mutex since php does not have
any thread synchronization objects
This class uses flock() as a base to provide locking functionality.
Lock will be released in following cases
1 - user calls unlock
2 - when this lock object gets deleted
3 - when request or script ends
==================================================================
Usage:
//get the lock
$lock = new ExclusiveLock( "mylock" );
//lock
if( $lock->lock( ) == FALSE )
error("Locking failed");
//--
//Do your work here
//--
//unlock
$lock->unlock();
===================================================================
*/
class ExclusiveLock
{
protected $key = null; //user given value
protected $file = null; //resource to lock
protected $own = FALSE; //have we locked resource
function __construct( $key )
{
$this->key = $key;
//create a new resource or get exisitng with same key
$this->file = fopen("$key.lockfile", 'w+');
}
function __destruct()
{
if( $this->own == TRUE )
$this->unlock( );
}
function lock( )
{
if( !flock($this->file, LOCK_EX | LOCK_NB))
{ //failed
$key = $this->key;
error_log("ExclusiveLock::acquire_lock FAILED to acquire lock [$key]");
return FALSE;
}
ftruncate($this->file, 0); // truncate file
//write something to just help debugging
fwrite( $this->file, "Locked\n");
fflush( $this->file );
$this->own = TRUE;
return TRUE; // success
}
function unlock( )
{
$key = $this->key;
if( $this->own == TRUE )
{
if( !flock($this->file, LOCK_UN) )
{ //failed
error_log("ExclusiveLock::lock FAILED to release lock [$key]");
return FALSE;
}
ftruncate($this->file, 0); // truncate file
//write something to just help debugging
fwrite( $this->file, "Unlocked\n");
fflush( $this->file );
$this->own = FALSE;
}
else
{
error_log("ExclusiveLock::unlock called on [$key] but its not acquired by caller");
}
return TRUE; // success
}
};
You can use the apc_add function to achieve this without resorting to file systems or mysql. apc_add only succeeds when the variable is not already stored; thus, providing a mechanism of locking. TTL can be used to ensure that falied lockholders won't keep on holding the lock forever.
The reason apc_add is the correct solution is because it avoids the race condition that would otherwise exist between checking the lock and setting it to 'locked by you'. Since apc_add only sets the value if it's not already set ( "adds" it to the cache ), it ensures that the lock can't be aquired by two calls at once, regardless of their proximity in time. No solution that doesn't check and set the lock at the same time will inherently suffer from this race condition; one atomic operation is required to successfully lock without race condition.
Since APC locks will only exist in the context of that php execution, it's probably not the best solution for general locking, as it doesn't support locks between hosts. Memcache also provides an atomic add function and thus can also be used with this technique - which is one method of locking between hosts. Redis also supports atomic 'SETNX' functions and TTL, and is a very common method of locking and synchronization between hosts. Howerver, the OP requests a solution for APC in particular.
If the point of the lock is to prevent multiple processes from trying to populate an empty cache key, why wouldn't you want to have a blocking lock?
$value = apc_fetch($KEY);
if ($value === FALSE) {
shm_acquire($SEMAPHORE);
$recheck_value = apc_fetch($KEY);
if ($recheck_value !== FALSE) {
$new_value = expensive_operation();
apc_store($KEY, $new_value);
$value = $new_value;
} else {
$value = $recheck_value;
}
shm_release($SEMAPHORE);
}
If the cache is good, you just roll with it. If there's nothing in the cache, you get a lock. Once you have the lock, you'll need to double-check the cache to make sure that, while you were waiting to get the lock, the cache wasn't repopulated. If the cache was repopulated, use that value & release the lock, otherwise, you do the computation, populate the cache & then release your lock.
Actually, check to see if this will work better then Peter's suggestion.
http://us2.php.net/flock
use an exclusive lock and if your comfortable with it, put everything else that attempted to lock the file in a 2-3 second sleep. If done right your site will experience a hang regarding the locked resource but not a horde of scripts fighting to cache the samething.
If you don't mind basing your lock on the filesystem, then you could use fopen() with mode 'x'. Here is an example:
$f = fopen("lockFile.txt", 'x');
if($f) {
$me = getmypid();
$now = date('Y-m-d H:i:s');
fwrite($f, "Locked by $me at $now\n");
fclose($f);
doStuffInLock();
unlink("lockFile.txt"); // unlock
}
else {
echo "File is locked: " . file_get_contents("lockFile.txt");
exit;
}
See www.php.net/fopen
I realize this is a year old, but I just stumbled upon the question while doing some research myself on locking in PHP.
It occurs to me that a solution might be possible using APC itself. Call me crazy, but this might be a workable approach:
function acquire_lock($key, $expire=60) {
if (is_locked($key)) {
return null;
}
return apc_store($key, true, $expire);
}
function release_lock($key) {
if (!is_locked($key)) {
return null;
}
return apc_delete($key);
}
function is_locked($key) {
return apc_fetch($key);
}
// example use
if (acquire_lock("foo")) {
do_something_that_requires_a_lock();
release_lock("foo");
}
In practice I might throw another function in there to generate a key to use here, just to prevent collision with an existing APC key, e.g.:
function key_for_lock($str) {
return md5($str."locked");
}
The $expire parameter is a nice feature of APC to use, since it prevents your lock from being held forever if your script dies or something like that.
Hopefully this answer is helpful for anyone else who stumbles here a year later.
EAccelerator has methods for it; eaccelerator_lock and eaccelerator_unlock.
Can't say if this is the best way to handle the job, but at least it is convenient.
function WhileLocked($pathname, callable $function, $proj = ' ')
{
// create a semaphore for a given pathname and optional project id
$semaphore = sem_get(ftok($pathname, $proj)); // see ftok for details
sem_acquire($semaphore);
try {
// capture result
$result = call_user_func($function);
} catch (Exception $e) {
// release lock and pass on all errors
sem_release($semaphore);
throw $e;
}
// also release lock if all is good
sem_release($semaphore);
return $result;
}
Usage is as simple as this.
$result = WhileLocked(__FILE__, function () use ($that) {
$this->doSomethingNonsimultaneously($that->getFoo());
});
Third optional argument can come handy if you use this function more than once per file.
Last but not least it isn't hard to modify this function (while keeping its signature) to use any other kind of locking mechanism at a later date, e.g. if you happen to find yourself working with multiple servers.
APC is now considered unmaintained and dead. It's successor APCu offers locking via apcu_entry. But be aware, that it also prohibits the concurrent execution of any other APCu functions. Depending on your use case, this might be OK for you.
From the manual:
Note: When control enters apcu_entry() the lock for the cache is acquired exclusively, it is released when control leaves apcu_entry(): In effect, this turns the body of generator into a critical section, disallowing two processes from executing the same code paths concurrently. In addition, it prohibits the concurrent execution of any other APCu functions, since they will acquire the same lock.
APCu has apcu_entry since 5.1.0, can implement a lock mechanism with it now:
/** get a lock, will wait until the lock is available,
* make sure handle deadlock yourself :p
*
* useage : $lock = lock('THE_LOCK_KEY', uniqid(), 50);
*
* #param $lock_key : the lock you want to get it
* #param $lock_value : the unique value to specify lock owner
* #param $retry_millis : wait befor retry
* #return ['lock_key'=>$lock_key, 'lock_value'=>$lock_value]
*/
function lock($lock_key, $lock_value, $retry_millis) {
$got_lock = false;
while (!$got_lock) {
$fetched_lock_value = apcu_entry($lock_key, function ($key) use ($lock_value) {
return $lock_value;
}, 100);
$got_lock = ($fetched_lock_value == $lock_value);
if (!$got_lock) usleep($retry_millis*1000);
}
return ['lock_key'=>$lock_key, 'lock_value'=>$lock_value];
}
/** release a lock
*
* usage : unlock($lock);
*
* #param $lock : return value of function lock
*/
function unlock($lock) {
apcu_delete($lock['lock_key']);
}
What I've found, actually, is that I don't need any locking at all... given what I'm trying to create is a map of all the class => path associations for autoload, it doesn't matter if one process overwrites what the other one has found (it's highly unlikely, if coded properly), because the data will get there eventually anyway. So, the solution turned out to be "no locks".