I have to subscribe to all the channels of my Redis db and simultaneously read data from another hash in the same db node. The following is the code I have written for this using phpredis:
$notif = new Worker;
try {
// connect redis
$notif->connectRedisServer();
// start the worker processing
$notif->startWorkerProcess();
}
catch(Exception $e) {
exit('Exception: '.$e->getMessage());
}
class Worker {
private $redis;
public $recentHash = 'tracker.data.recent';
public $notifHash = 'tracker.settings.data';
public $serverConnectionDetails = { //Redis server connection details
};
private $redis;
// redis database used for getting recent record of every imei
const RECENTDATA_DB = 1;
public function connectRedisServer() {
if(!empty($this->redis))
return $this->redis; // already connected
$this->redis = new Redis();
if(!$this->redis->pconnect(
$this->serverConnectionDetails['redis']['host'],
$this->serverConnectionDetails['redis']['port']
))
return 'Redis connection failed.';
if(isset($this->serverConnectionDetails['redis']['password'])
&&
!$this->redis->auth($this->serverConnectionDetails['redis']['password'])
)
return 'Redis authentication failed.';
return $this->redis;
}
public function startWorkerProcess() {
$this->redis->select(self::RECENTDATA_DB);
$this->redis->pSubscribe(array('*'), array($this, 'processRedisData'));
}
public function processRedisData($redis, $pattern, $chan, $msg) {
$message = rtrim((string) $msg,",");
$tData = json_decode($message, true);
$tId = (int) $tData['tID'];
echo "Recent Published Data:";
var_dump($tData);
$data = $this->getNotifSettings($tId);
if(!empty($data)) {
echo "Redis Settings: ";
var_dump($trackerSettings);
}
}
public function getNotifSettings($tId) {
$data = $this->redis->hGet($this->notifHash, $tId); //This command doesn't return any data even if it exists for the $tId
if($data === false)
return null;
$data = json_decode($data, true);
return $data; // Always comes up as null
}
}
The problem here is that once I get subscribed to all the channels on db1 in Redis. I don't get any result if I try to run HGET, even though the data for the given key exists in the db. I have put additional comments in the code above to explain where the problem is. Check getNotifSettings() function.
Any help will be appreciated.
Related
The objective is to continually collect data of the current temperature. But a separate process should analyse the output of that data because I have to tweak the algorithm a lot but I want to avoid downtime so stopping the process is a no-go.
The problem is when I separate these processes, that process 2 would either continually have to make calls to the database or read from a local file to do something with the output generated by 1 process but I want to act upon it immediately and that is expensive in terms of resources.
Would it be possible to reload the class into memory somehow when the file changes by for example writing a function that keeps calculating the MD5 of the file, and if it changes than reload the class somehow? So this separate class should act as a plugin. Is there any way to make that work?
Here is a possible solution. Use Beanstalk (https://github.com/kr/beanstalkd).
PHP class to talk to Beanstalk (https://github.com/pda/pheanstalk)
Run beanstalk.
Create a process that goes into an infinite loop that reads from a Beanstalk queue. (Beanstalk queues are called "Tubes"). PHP processes are not meant to be run for a very long time. The main reason for this is memory. The easiest way to do handle this is to restart the process every once in a while or if memory gets to a certain threshold.
NOTE: What I do is to have the process exit after some fixed time or if it uses a certain amount of memory. Then, I use Supervisor to restart it.
You can put data into Beanstalk as JSON and decode it on the receiving end. The sending and receiving processes need to agree on that format. You could store your work payload in a database and just send the primary key in the queue.
Here is some code you can use:
class BeanstalkClient extends AbstractBaseQueue{
public $queue;
public $host;
public $port;
public $timeout;
function __construct($timeout=null) {
$this->loadClasses();
$this->host = '127.0.0.1';
$this->port = BEANSTALK_PORT;
$this->timeout = 30;
$this->connect();
}
public function connect(){
$this->queue = new \Pheanstalk\Pheanstalk($this->host, $this->port);
}
public function publish($tube, $data, $delay){
$payload = $this->encodeData($data);
$this->queue->useTube($tube)->put($payload,
\Pheanstalk\PheanstalkInterface::DEFAULT_PRIORITY, $delay);
}
public function waitForMessages($tube, $callback=null){
if ( $this->timeout ) {
return $this->queue->watchOnly($tube)->reserve($this->timeout);
}
return $this->queue->watchOnly($tube)->reserve();
}
public function delete($message){
$this->queue->delete($message);
}
public function encodeData($data){
$payload = json_encode($data);
return $payload;
}
public function decodeData($encodedData) {
return json_decode($encodedData, true);
}
public function getData($message){
if ( is_string($message) ) {
throw new Exception('message is a string');
}
return json_decode($message->getData(), true);
}
}
abstract class BaseQueueProcess {
protected $channelName = ''; // child class should set this
// The queue object
public $queue = null;
public $processId = null; // this is the system process id
public $name = null;
public $status = null;
public function initialize() {
$this->processId = getmypid();
$this->name = get_called_class();
$this->endTime = time() + (2 * 60 * 60); // restart every hour
// seconds to timeout when waiting for a message
// if the process isn't doing anything, timeout so they have a chance to do housekeeping.
$queueTimeout = 900;
if ( empty($this->queue) ) {
$this->queue = new BeanstalkClient($queueTimeout);
}
}
public function receiveMessage($queueMessage) {
$taskData = $this->queue->getData($queueMessage);
// debuglog(' Task Data = ' . print_r($taskData, true));
if ( $this->validateTaskData($taskData) ) {
// process the message
$good = $this->didReceiveMessage($taskData);
if ( $good !== false ) {
// debuglog("Completing task {$this->taskId}");
$this->completeTask($queueMessage);
}
else {
$this->failTask($queueMessage);
}
}
else {
// Handle bad message
$this->queue->delete($queueMessage);
}
}
public function run() {
$this->processName = $this->channelName;
// debuglog('Start ' . $this->processName);
// debuglog(print_r($this->params, true));
while(1) {
$queueMessage = $this->queue->waitForMessages($this->channelName);
if ( ! empty($queueMessage) ) {
$this->receiveMessage($queueMessage);
}
else {
// empty message
// a timeout
// // debuglog("empty message " . get_called_class());
}
$memory = memory_get_usage();
if( $memory > 20000000 ) {
// debuglog('Exit '.get_called_class().' due to memory. Memory:'. ($memory/1024/1024).' MB');
// Supervisor will restart process.
exit;
}
elseif ( time() > $this->endTime ) {
// debuglog('Exit '.get_called_class().' due to time.');
// Supervisor will restart process.
exit;
}
usleep(10);
}
}
public function completeTask($queueMessage) {
//
$this->queue->delete($queueMessage);
}
public function failTask($queueMessage) {
//
$this->queue->delete($queueMessage);
}
}
class MyProcess extends BaseQueueProcess {
public function initialize() {
$this->channelName = 'Temperature';
parent::initialize();
}
public function didReceiveMessage($taskData) {
// debuglog(print_r($taskData, true));
// process data here
// return false if something went wrong
return true;
}
}
//Sender
class WorkSender {
const TubeName = 'Temperature';
const TubeDelay = 0; // Set delay to 0, i.e. don't use a delay.
function send($data) {
$c = BeanstalkClient();
$c->publish(self::TubeName, $data, self::TubeDelay);
}
}
When i run gearman upto three workers on single server is working fine but when i start the 4th one php code is working fine but can't detect the new worker and also not clearing the job queue.
protected function createWorker()
{
$this->worker = new \GearmanWorker();
$config = $this->app->config->job_remote;
$this->worker->addServer($config['host'], $config['port']);
return $this->worker;
}
public function listen($eventType, $callback)
{
if (!($this->worker instanceof \GearmanWorker)){
$this->worker = $this->createWorker();
}
$this->worker->addFunction($eventType, $callback);
return $this->worker;
}
public function doWork($worker)
{
if (!($worker instanceof \GearmanWorker)){
$worker = $this->createWorker();
}
$this->worker = $worker;
while (1) {
$this->worker->work();
$this->app->log->debug($this->worker->returnCode());
if ($this->worker->returnCode() != \GEARMAN_SUCCESS) {
break;
}
}
}
First I am calling 'listen' method and then 'doWork' method
Client Side Code:
protected function createClient()
{
$this->client = new \GearmanClient();
$config = $this->app->config->job_remote;
$this->client->addServer($config['host'], $config['port']);
return $this->client;
}
public function addTask($eventType, array $params)
{
if (!($this->client instanceof \GearmanClient)){
$this->client = $this->createClient();
}
// add single task in queue
$this->client->addTaskBackground($eventType, serialize($params));
// Run task
$this->client->runTasks();
}
Using GearmanManager You can run upto any no. of workers untill it slows down your computer and uses 100% cpu and 100% memory.
Gearman.ini
[GearmanManager]
worker_dir=./workers
count=50
dedicated_count=1
max_worker_lifetime=3600
auto_update=1
log_file=./logs/WebAnalyzerWorker.log
max_runs_per_worker=3
timeout=600
Run Gearman worker
./vendor/brianlmoon/gearmanmanager/pecl-manager.php -c ./gearman.ini -vvvvv
Worker CLass
<?php
include dirname(__DIR__)."/vendor/autoload.php";
use Services\Log;
use Services\ServiceInitialization;
use Lib\Technology\FindWhoisRecords;
use Illuminate\Database\Capsule\Manager as Capsule;
class DomainDetailFetchJob{
public function beforeRun()
{
ServiceInitialization::loadConfig();
ServiceInitialization::loadDatabaseConfiguration();
}
public function run($job, &$log)
{
$log[] = "Starting whois job";
$collection = "whois";
$this->beforeRun();
ServiceInitialization::$config->all();
$workload = $job->workload();
//workload
$whois = new FindWhoisRecords($workload);
if($whois->whois_result["err_code"]==0) {
$log[] = "Whois Info fetch successful";
//success save the details
$whois->whois_result["result"]["workload"] = $workload;
Capsule::table($collection)->where("workload", $workload)->update($whois->whois_result["result"], ['upsert' => true]);
}
else {
$log[] = "Whois Info fetch failed";
$logger = new Log();
$logger->write($whois->whois_result["err_msg"], "error", "Whois Record Job");
unset($logger);
}
}
}
Gearman Client
$client = new GearmanClient();
$client->addServer();
$client->doBackground("DomainDetailFetchJob", $url);
I'm creating my own custom SessionHandler to store my session information in a postgresql 9.3 database and I'm having a problem where the session data passed to the write() method isn't being written to the database, but the session name is??
Things that I know for a fact
My custom class is handling the sessions when session_start() is called - as tested with echoing output from the various methods and no session files are being created in /tmp
The $session_data arg in write() contains the proper serialized string as shown by echoing the contents in the write() method.
$session_name is being written to the database just fine and so is a BLANK serialized string a:0:{}.
Things I'm confused about:
Echoing the contents of $_SESSION['test_var1'] shows the correct value stored, even if read() is empty or returning no value??
If the session name is saved in the DB just fine, why isn't the session data?
Server Configuration
OS: Gentoo
Database: Postgresql 9.3
Web Server: NGINX 1.7.6
PHP 5.5.18 connected to NGINX via FPM
PHP ini session settings
session.save_handler = user
session.use_strict_mode = 1
session.use_cookies = 1
session.cookie_secure = 1
session.use_only_cookies = 1
session.name = _g
session.auto_start = 0
session.serialize_handler = php_serialize
class SessionManagement implements SessionHandlerInterface {
private $_id = '';
private $_link = null;
public function __construct() {
session_set_save_handler(
[$this, 'open'],
[$this, 'close'],
[$this, 'read'],
[$this, 'write'],
[$this, 'destroy'],
[$this, 'gc']
);
}
public function open($save_path, $session_id) {
echo 'open called<br/>';
$this->_id = $session_id;
$this->_link = new PDO('pgsql:host=' . $_SERVER['DB_HOST'] . ';dbname=' . $_SERVER['DB_DB'],
$_SERVER['DB_USER'],
$_SERVER['DB_PASS']);
}
public function close() {
echo 'close called<br/>';
}
public function destroy($session_id) {
echo 'destroying '.$session_id, '<br/>';
}
public function gc($maxlifetime) {
echo 'GC called<br/>';
}
public function read($session_name) {
$name = $this->_id.'_'.$session_name;
$sql = 'SELECT session_data FROM sessions WHERE session_name = :name';
if ($rel = $this->_link->prepare($sql)) {
if ($rel->execute([':name' => $name])) {
return $rel->fetch(PDO::FETCH_ASSOC)['session_data'];
} else {
return false;
}
} else {
return false;
}
return '';
}
public function write($session_name, $session_data) {
echo 'Session data: '.$session_data.'<br/>';
$name = $this->_id . '_' . $session_name;
$data = $session_data;
$sql = "SELECT 1 FROM sessions WHERE session_name = :name";
if ($rel = $this->_link->prepare($sql)) {
if ($rel->execute([':name' => $name])) {
if ($rel->rowCount()) {
echo 'Updating...<br/>';
$sql = 'UPDATE sessions SET session_data = :data WHERE session_name = :name';
if ($rel = $this->_link->prepare($sql)) {
if ($rel->execute([':name' => $name, ':data' => $data])) {
echo 'Update success...<br/>';
} else {
echo 'Update failed...<br/>';
var_dump($rel->errorInfo());
}
}
} else {
echo 'Inserting...<br/>';
$sql = 'INSERT INTO sessions (session_name, session_data) ';
$sql .= 'VALUES(:name, :data)';
if ($rel = $this->_link->prepare($sql)) {
if ($rel->execute([':name' => $name, ':data' => $data])) {
echo 'Insert success...<br/>';
} else {
echo 'Insert failed...<br/>';
var_dump($rel->errorInfo());
}
}
}
}
}
}
}
Test code:
new SessionManagement();
session_start();
$_SESSION['test_var1'] = 'some test data';
session_write_close(); // Making sure writing is finished
echo $_SESSION['test_var1'];
Output via test page
open called
Session data: a:1:{s:9:"test_var1";s:14:"some test data";}
Inserting...
Insert success...
close called
some test data
Relevant database fields
session_name: _g_h8m64bsb7a72dpj56vgojn6f4k3ncdf97leihcqfupg2qtvpbo20
session_data: a:0:{}
I'm not sure if this is a database issue or a PHP issue. I've been messing with this for a few days now and decided it was time to ask the community. Hopefully someone has some insight as to what the problem is.
I think you must initialize PDO object outside of the Open function handler and the class itself
try to access to your PDO Object with a Global value or through a static variable.
This is my implementation with MYSQL for my project :
class Core_DB_SessionHandler implements SessionHandlerInterface
{
protected $options = array(); // Options de la session
protected static $db = NULL; // Acceder a la BDD
public function __construct($options, $pdo) {
$this->options = $options;
self::$db = $pdo;
}
public function open($savePath, $sessionName) {
$now = time();
$req = self::$db->prepare("DELETE FROM tbl_sessions WHERE expire < '{1}' ");
$req->execute(array($now));
return TRUE;
}
public function close() {
$this->gc(ini_get('session.gc_maxlifetime'));
}
public function read($id) {
$now = time();
$stmt = self::$db->query("SELECT data FROM tbl_sessions WHERE sid = '$id AND expire < '$now'");
$result = $stmt->fetchColumn();
return $result;
}
public function write($id, $data) {
if (array_key_exists('TIMEOUT', $_SESSION)) {
$newExp = $_SESSION['TIMEOUT'];
}
else {
$newExp = time() + $this->options['time_limiter'];
}
try {
$req = self::$db->prepare('INSERT INTO tbl_sessions (sid, data, expire) VALUES (:sid, :data, :expire)
ON DUPLICATE KEY UPDATE data = :data, expire = :expire');
$vals = array('sid' => $id, 'data' => $data, 'expire' => $newExp);
$req->execute($vals);
return TRUE;
}
catch (PDOException $e) {
throw new Core_Exception(sprintf('PDOException was thrown when trying to write the session data: %s', $e->getMessage()), 0, $e);
}
}
public function destroy($id) {
$stmt = self::$db->prepare("DELETE FROM tbl_sessions WHERE sid = '{1}'");
$stmt->execute(array($id));
//return ($stmt->rowCount() === 1) ? true : false;
return TRUE;
}
public function gc($maxlifetime) {
$now = time();
$req = self::$db->prepare("DELETE FROM tbl_sessions WHERE expire < '{1}' ");
$req->execute(array($now));
return TRUE;
}
}
and i initialize handler like this :
$handler = new Core_DB_SessionHandler($MyOptions, $MyPDO);
if (PHP5) {
if (!session_set_save_handler($handler, TRUE)) {
throw new Core_Exception('Erreur lors de l\'init des sessions !');
}
}
nb : In your Table structure don't use autoincrement for ID
Well, I've solved my problem, but it's hard to say if this fix was actually the problem in the first place.
In the read method, I changed the follow:
return $rel->fetch(PDO::FETCH_ASSOC)['session_data'];
to
$data $rel->fetch(PDO::FETCH_ASSOC)['session_data'];
return $data;
After this the session was writing $session_data to the database without any problem. That's all well and dandy, but it doesn't explain why it didn't work in the first place. I mainly say this because upon switching the statement back everything continued to work. As in, I can't reproduce the issue now. So it's hard for me to even say that this was the problem in the first place.
Hopefully this helps someone. I've been unable to find more information about it, but it something does show up I'll be sure to add it here.
I'm using phpredis in my program, store something in the redis server, get them when the same request comes(in the same day), but I always get empty result. Can anyone give me some enlightenment? Here is the code of Cache class I'm using:
<?php
class Cache
{
public static function getInstance()
{
static $instance = null;
null == $instance && $instance = new self();
return $instance;
}
protected function __construct()
{
}
protected function getR()
{
static $r = NULL;
if (NULL == $r) {
$r = new Redis();
try {
$r->pconnect(HOST, PORT, 5);
} catch(Exception $ex) {
//log
try {
$api->connect(HOST, PORT, 5);
} catch (Exception $ex) {
//log
}
}
}
return $r;
}
public function getValue($key)
{
$result = array();
$r = $this->getR();
if(!empty($r)) {
try{
$result = $r->hKeys($key);
$r->setTimeout($keys, 86400);
} catch (Exception $ex){
//log
}
}
return $result; // return true
}
public function setValue($key, $value)
{
$result = false;
$r = $this->getR();
if(!empty($r)) {
try{
$result = $r->hMset($key, $value);
} catch (Exception $ex){
//log
}
}
}
}
?>
EDIT:
I checked the key-values with redis-cli, found something wired: the key-value data was stored in db 5 while I thought it should be in DB 0 by default without select statement, but the program retrieved db 0, of course nothing returned. Now I'm wondering why the data went to DB 5 given that I've not selected DB.
Finally, I've figured out what happend here. Before I stored my key–value pair, there was some code which also had communicated with Redis server and it had explicitly selected the DB 5, and the default DB of my redis connection was affected by last context, so my data was stored in DB 5. By coincidence, when I wanted to retrive my data, the last redis connection used DB 0, of course I got nothing.
The script works fine and is setting the data, but the website code is unable to use it and is instead setting its own memcached values. My website code is written in codeIgniter framework. I don't know why this is happening.
My script code :-
function getFromMemcached($string) {
$memcached_library = new Memcached();
$memcached_library->addServer('localhost', 11211);
$result = $memcached_library->get(md5($string));
return $result;
}
function setInMemcached($string,$result,$TTL = 1800) {
$memcached_library = new Memcached();
$memcached_library->addServer('localhost', 11211);
$memcached_library->set(md5($string),$result, $TTL);
}
/*---------- Function stores complete product page as one function call cache -----------------*/
function getCachedCompleteProduct($productId,$brand)
{
$result = array();
$result = getFromMemcached($productId." product page");
if(true==empty($result))
{
//------- REST CODE storing data in $result------
setInMemcached($productId." product page",$result,1800);
}
return $result;
}
Website Code :-
private function getFromMemcached($string) {
$result = $this->memcached_library->get(md5($string));
return $result;
}
private function setInMemcached($string,$result,$TTL = 1800) {
$this->memcached_library->add(md5($string),$result, $TTL);
}
/*---------- Function stores complete product page as one function call cache -----------------*/
public function getCachedCompleteProduct($productId,$brand)
{
$result = array();
$result = $this->getFromMemcached($productId." product page");
if(true==empty($result))
{
// ----------- Rest Code storing data in $result
$this->setInMemcached($productId." product page",$result,1800);
}
return $result;
}
This is saving data in memcached. I checked by printing inside the if condition and checking the final result
Based on the CodeIgniter docs, you can make use of:
class YourController extends CI_Controller() {
function __construct() {
$this->load->driver('cache');
}
private function getFromMemcached($key) {
$result = $this->cache->memcached->get(md5($key));
return $result;
}
private function setInMemcached($key, $value, $TTL = 1800) {
$this->cache->memcached->save(md5($key), $value, $TTL);
}
public function getCachedCompleteProduct($productId,$brand) {
$result = array();
$result = $this->getFromMemcached($productId." product page");
if( empty($result) ) {
// ----------- Rest Code storing data in $result
$this->setInMemcached($productId." product page",$result,1800);
}
return $result;
}
}
Personally try to avoid 3rd party libraries if it already exists in the core framework. And I have tested this, it's working superbly, so that should fix this for you :)
Just remember to follow the instructions at http://ellislab.com/codeigniter/user-guide/libraries/caching.html#memcached to set the config as needed for the memcache server