I have a daemon script running in PHP. This script continuously hits an endpoint and gets the response from a resource and checks for a response.
The response is typically
{"a": 1, "b": 2, "c": 3, "status" : "true"}
most of the time.
Occasionally, it could turn out that the response might change to
{"a": 1, "b": 2, "c": 3, "status" : "false"}
I have logic to read values of a, b and c and perform operations based on them.
My question is, is there a way to somehow cache this response instead of hitting the end point over and over at the daemon level?
Or is there a way to cache the json response from the server level?
Use each element of your object as an array key:
$responses[$a][$b][$c]['status'] = true;
The cache would be a simple:
if (isset($responses[$a][$b][$c]['status'])) {
... used cached value
} else {
... compute new value and store in cache
}
Though you might want some extra logic in there to expire older entries, other you'll end up sucking up memory until things die.
If you want to cache the response from your daemon here are some options
1 - If on PHP < 5.5 , you can use apc's user cache
Here is how save some random piece of text (does not matter if its JSON or an object , you will be able to retrieve it fine)
apc_add('cache_key',$save_this_var, TTL);
OR
apc_store('cache_key',$save_this_var, TTL);
apc_add will not overwrite if you save something else with the exact same key , whereas apc_store will overwrite
TTL is the time (in seconds) you want to cache the data for
Now you can retrieve the cache like so
$my_cache = apc_fetch('cache_key');
$my_cache has the variable you saved earlier
2 - If you are using PHP 5.5 , then make use apc's user cache by adding the apcu module ,it works exactly like apc cache I explained above ,just a name change (apcu does not cache opcode whereas apc in PHP < 5.4 does)
3 - If you are in a cloud or multiserver environment , you can use memcached as well , but if on single machine , i would go with apc or apcu
Related
How to convert redis-cli script load $LUA_SCRIPT into Predis methods?
Follow is the lua script:
local lock_key = 'icicle-generator-lock'
local sequence_key = 'icicle-generator-sequence'
local logical_shard_id_key = 'icicle-generator-logical-shard-id'
local max_sequence = tonumber(KEYS[1])
local min_logical_shard_id = tonumber(KEYS[2])
local max_logical_shard_id = tonumber(KEYS[3])
local num_ids = tonumber(KEYS[4])
if redis.call('EXISTS', lock_key) == 1 then
redis.log(redis.LOG_INFO, 'Icicle: Cannot generate ID, waiting for lock to expire.')
return redis.error_reply('Icicle: Cannot generate ID, waiting for lock to expire.')
end
--[[
Increment by a set number, this can
--]]
local end_sequence = redis.call('INCRBY', sequence_key, num_ids)
local start_sequence = end_sequence - num_ids + 1
local logical_shard_id = tonumber(redis.call('GET', logical_shard_id_key)) or -1
if end_sequence >= max_sequence then
--[[
As the sequence is about to roll around, we can't generate another ID until we're sure we're not in the same
millisecond since we last rolled. This is because we may have already generated an ID with the same time and
sequence, and we cannot allow even the smallest possibility of duplicates. It's also because if we roll the sequence
around, we will start generating IDs with smaller values than the ones previously in this millisecond - that would
break our k-ordering guarantees!
The only way we can handle this is to block for a millisecond, as we can't store the time due the purity constraints
of Redis Lua scripts.
In addition to a neat side-effect of handling leap seconds (where milliseconds will last a little bit longer to bring
time back to where it should be) because Redis uses system time internally to expire keys, this prevents any duplicate
IDs from being generated if the rate of generation is greater than the maximum sequence per millisecond.
Note that it only blocks even it rolled around *not* in the same millisecond; this is because unless we do this, the
IDs won't remain ordered.
--]]
redis.log(redis.LOG_INFO, 'Icicle: Rolling sequence back to the start, locking for 1ms.')
redis.call('SET', sequence_key, '-1')
redis.call('PSETEX', lock_key, 1, 'lock')
end_sequence = max_sequence
end
--[[
The TIME command MUST be called after anything that mutates state, or the Redis server will error the script out.
This is to ensure the script is "pure" in the sense that randomness or time based input will not change the
outcome of the writes.
See the "Scripts as pure functions" section at http://redis.io/commands/eval for more information.
--]]
local time = redis.call('TIME')
return {
start_sequence,
end_sequence, -- Doesn't need conversion, the result of INCR or the variable set is always a number.
logical_shard_id,
tonumber(time[1]),
tonumber(time[2])
}
redis-cli script load $LUA_SCRIPT
I tried
$predis->eval(file_get_contents($luaPath), 0);
or
class ListPushRandomValue extends \Predis\Command\ScriptCommand {
public function getKeysCount() {
return 0;
}
public function getScript() {
$luaPath = Aa::$vendorRoot .'/icicle/id-generation.lua';
return file_get_contents($luaPath);
}
}
$predis->getProfile()->defineCommand('t', '\Controller\ListPushRandomValue');
$response = $predis->t();
But both of above showed error below.
ERR Error running script (call to f_5849d008682280eed4ec67b97ba50ae546fc5e8d): #user_script:19: user_script:19: attempt to perform arithmetic on local 'end_sequence' (a table value)
First, let me say that I am not an expert on LUA but I have just dealt with Redis and LUA script to implement simple locking and noticed a few errors in the question.
There is a conversion process between Redis and LUA that should be reviewed: Conversion. I think this will help with the error given.
On the this call:
$predis->eval(file_get_contents($luaPath), 0);
You pass the contents of the script and access keys but don't pass in any keys to evaluate. This call is more correct:
$predis->eval(file_get_contents($luaPath), 4, oneKey, twoKey, threeKey, fourKey);
This might actually be the reason for the above error. Hope this helps someone in the future.
I'm adding a key using Memcached like so:
$valueToStore = time(); // some number
$success = $memcached->add( 'test_key', $valueToStore, 20 ); // cache for 20 seconds
But it's always succeeding when I call it in a different session, even before 20 seconds have passed. According to the docs at http://php.net/manual/en/memcached.add.php, it should be returning FALSE until the key expires (because the key already exists).
I'm running on a single development server with plenty of free cache space. Any idea what might be happening?
php -v returns: PHP 5.5.9-1ubuntu4.3
memcached version 2.1.0
libmemcached version 1.0.8.
You need to be distinct if you are using the Memcache class or the Memcached class. Your cache design is a bit strange. You should be checking the cache to first see if the item is there. If the item is not then store it. Also Memcache has some strange behavior on using the boolen type as the third argument. You should MEMCACHE_COMPRESSED. I think you are using Memcache.
To illustrate how to fix your problem:
$in_cache = $memcached->get('test_key');
if($in_cache)
return $in_cache;
else
$valueToStore = time();
$memcached->add('test_key', $valueToStore, MEMCACHE_COMPRESS, 20);
I have an classical apache server delivering php files, and a nodeJS server (with socket.io, but whithout express/connect) used for real-time event management on that PHP website. I sometimes need to authenticate the clients connecting to the nodeJS server, but this authentication is lost when the user reloads the page, because it also reloads the socket.io client (I store the socket ID on the server, which gets lost at each refresh)
The question is: Is there a way to keep the connection alive in socket.io, or a way to link the apache PHP sessions and the nodeJS server? Or maybe a way to keep this authentication using cookies (knowing I must store sensitive data like user passwords and keys)?
You can use memcached as your session storage handler in PHP. Memcached is a simple key value store that can be accessed via TCP; there is a memcached module available for Node.js.
PHP stores the session in memcached by using the session id as the key. The session data (value) stored in memcached is a serialized PHP object, with a slight twist. You can read more about this unusual serialization at the SO question "Parse PHP Session in Javascript". Luckily though, there is already an NPM module out there: php-unserialize.
Now for the How-To.
Assumptions
memcached is accessible at 127.0.0.1:11211
php.ini (or php.d/memcache.ini) is configured with: session.save_handler='memcached' and session.save_path='tcp://127.0.0.1:11211'
you have installed the required NPM modules (2): npm install memcached php-unserialize
you're ok with CLI
Prepare
First, just to get some test data to work with, save the following php script (s.php):
<?php
session_start();
$_SESSION['some'] = 'thing';
echo session_id()."\n";
print_r($_SESSION);
Execute it with php s.php, and it should put stuff in stdout:
74ibpvem1no6ssros60om3mlo5
Array
(
[some] => thing
)
Ok, now we know the session id (74ibpvem1no6ssros60om3mlo5), and have confirmed that the session data was set. To confirm it is in memcached, you can run memcached-tool 127.0.0.1:11211 dump which provides a dump of known key:value pairs, for example I have two in my test bed:
Dumping memcache contents
Number of buckets: 1
Number of items : 3
Dumping bucket 2 - 3 total items
add 74ibpvem1no6ssros60om3mlo5 0 1403169638 17
some|s:5:"thing";
add 01kims55ut0ukcko87ufh9dpv5 0 1403168854 17
some|s:5:"thing";
So far we have 1) created a session id in php, 2) stored session data from php in memcached, and 3) confirmed the data exists via CLI.
Retrieval with Node.js
This part is actually really easy. Most of the heavy-lifting has already been done by the NPM modules. I cooked up a little Node.js script that runs via CLI, but you get the picture:
var Memcached = require('memcached');
var PHPUnserialize = require('php-unserialize');
var mem = new Memcached('127.0.0.1:11211'); // connect to local memcached
var key = process.argv[2]; // get from CLI arg
console.log('fetching data with key:',key);
mem.get(key,function(err,data) { // fetch by key
if ( err ) return console.error(err); // if there was an error
if ( data === false ) return console.error('could not retrieve data'); // data is boolean false when the key does not exist
console.log('raw data:',data); // show raw data
var o = PHPUnserialize.unserializeSession(data); // decode session data
console.log('parsed obj:',o); // show unserialized object
});
Assuming the above is saved as m.js, it can be run with node m.js 74ibpvem1no6ssros60om3mlo5 which will output something like:
fetching data with key: 74ibpvem1no6ssros60om3mlo5
raw data: some|s:5:"thing";
parsed obj: { some: 'thing' }
Warnings/Gotchas
One of my PHP applications stores some binary data in the session values (i.e. encrypted), but the keys and the normal session object remain intact (as in the example above). In this case, memcached-tool <host:port> dump printed a malformed serialized session string to stdout; I thought this might be isolated to stdout, but I was wrong. When using PHPUnserialize.unserializeSession, it also had trouble parsing the data (delimited by |). I tried a few other session deserialization methods out on the net, but did not have any success. I would assume memcached is maintaining the correct data internally since it works with the native PHP session save handler, so, at the time of this writing, I'm not quite sure if it is the deserialization methods or if the memcached NPM module simply isn't retrieving/interpreting the data correctly. When sticking with non-binary data like ascii or utf-8, it should work as intended.
I think this link will be of some help to you
https://simplapi.wordpress.com/2012/04/11/php-nodejs-session-share-memcache/
Though the thread is old I would like to recommend what I used for my project.
Instead of memcached you can also use Redis for session handling.
I have used the phpredis as php redis client.Instead of storing session to files you can save in Redis. Most of the heavy lifting will be done by apache. For every request apache will append the session values to the cookies.And it reads the session values from every request and validates it.
Setting required to save the php session to redis is also very simple.
session.save_handler = redis
session.save_path = "tcp://host1:6379?weight=1, tcp://host2:6379?weight=2&timeout=2.5, tcp://host3:6379?weight=2"
That's it.This will make php save the sessions to redis instead of the file. This will also move the session that are stored in files to redis.
If your project stores session in database - some do - then you can consider using database as a transfer medium.
If analysis in your particular case shows promise, then node-mysql (or similar) can be used - see this: link
The answer from zamnuts helped me make since of doing authentication and was the approach I was already taking. Thanks for that.
The reason I am posting is the for me some reason when using :
var PHPUnserialize = require('php-unserialize');
Kept giving me error
SyntaxError: String length mismatch
I am not sure why? I wrote a function that does the job for me and wanted to share in case it may help someone else.
function Unserialize(data){
var result = {};
if(data !== undefined){
var preg = data.replace(/(^|s:[0-9]+:)|(^|i:)|(^|b:)|(")|(;$)/g,'').split(';');
var a = [];
preg.forEach(function(value){
a.push(value.split('|'));
});
var b = [];
a.forEach(function(value){
if(Array.isArray(value)){
Array.prototype.push.apply(b, value);
}else{
b.push(value);
}
});
var arr_A = [];
var arr_B = [];
b.forEach(function(value, k){
if(k % 2 == 0){
arr_A.push(value);
}else{
arr_B.push(value);
}
});
if (arr_A == null) return {};
for (var i = 0, l = arr_A.length; i < l; i++) {
if (arr_B) {
result[arr_A[i]] = arr_B[i];
} else {
result[arr_A[i][0]] = arr_A[i][1];
}
}
}
return result;
}
I just call it like this:
var PHPUnserialize = Unserialize;
memcached.get(key, function(err, data){
var memData = PHPUnserialize(data);
console.log(memData.is_logged_in);
});
You should be able to modify the regex to suit your needs fairly easily.
I'm moving some part of my site from relational database to Redis and need to insert milions of keys in possibly short time.
In my case, data must be first fetched from MySQL, prepared by PHP and then added to corresponding sorted sets (time as a score + ID as a value). Currently I'm taking adventage of phpredis multi method with Redis::PIPELINE parameter. Despite noticeable speed improvements it turned out to block reads and slow down loading times while doing import.
So here comes the question - is using pipeline in phpredis an equivalent to the mass insertion described in http://redis.io/topics/mass-insert?
Here's an example:
phpredis way:
<?php
// All necessary requires etc.
$client = Redis::getClient();
$client->multi(Redis::PIPELINE); // OR $client->pipeline();
$client->zAdd('key', 1, 2);
...
$client->zAdd('key', 1000, 2000);
$client->exec();
vs protocol from redis.io:
cat data.txt | redis-cli --pipe
I'm one of the contributors to phpredis, so I can answer your question. The short answer is that it is not the same but I'll provide a bit more detail.
What happens when you put phpredis into Redis::PIPELINE mode is that instead of sending the command when it is called, it puts it into a list of "to be sent" commands. Then, once you call exec(), one big command buffer is created with all of the commands and sent to Redis.
After the commands are all sent, phpredis reads each reply and packages the results as per each commands specification (e.g. HMGET calls come back as associative arrays, etc).
The performance on pipelining in phpredis is actually quite good, and should suffice for almost every use case. That being said, you are still processing every command through PHP, which means you will pay the function call overhead by calling the phpredis extension itself for every command. In addition, phpredis will spend time processing and formatting each reply.
If your use case requires importing MASSIVE amounts of data into Redis, especially if you don't need to process each reply (but instead just want to know that all commands were processed), then the mass-import method is the way to go.
I've actually created a project to do this here:
https://github.com/michael-grunder/redismi
The idea behind this extension is that you call it with your commands and then save the buffer to disk, which will be in the raw Redis protocol and compatible with cat buffer.txt | redis-cli --pipe style insertion.
One thing to note is that at present you can't simply replace any given phpredis call with a call to the RedisMI object, as commands are processed as variable argument calls (like hiredis), which work for most, but not all phpredis commands.
Here is a simple example of how you might use it:
<?php
$obj_mi = new RedisMI();
// Some context we can pass around in RedisMI for whatever we want
$obj_context = new StdClass();
$obj_context->session_id = "some-session-id";
// Attach this context to the RedisMI object
$obj_mi->SetInfo($obj_context);
// Set a callback when a buffer is saved
$obj_mi->SaveCallback(
function($obj_mi, $str_filename, $i_cmd_count) {
// Output our context info we attached
$obj_context = $obj_mi->GetInfo();
echo "session id: " . $obj_context->session_id . "\n";
// Output the filename and how many commands were sent
echo "buffer file: " . $str_filename . "\n";
echo "commands : " . $i_cmd_count . "\n";
}
);
// A thousand SADD commands, adding three members each time
for($i=0;$i<1000;$i++) {
$obj_mi->sadd('some-set', "$i-one", "$i-two", "$i-three");
}
// A thousand ZADD commands
for($i=0;$i<1000;$i++) {
$obj_mi->zadd('some-zset', $i, "member-$i");
}
// Save the buffer
$obj_mi->SaveBuffer('test.buf');
?>
Then you can do something like this:
➜ tredismi php mi.php
session id: some-session-id
buffer file: test.buf
commands : 2000
➜ tredismi cat test.buf|redis-cli --pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 2000
Cheers!
Let's say I cache data in a PHP file in PHP array like this:
/cache.php
<?php return (object) array(
'key' => 'value',
);
And I include the cache file like this:
<?php
$cache = include 'cache.php';
Now, the question is will the cache file be automatically cached by APC in the memory? I mean as a typical opcode cache, as all .php files.
If I store the data differently for example in JSON format (cache.json), the data will not be automatically cached by APC?
Would apc_store be faster/preferable?
Don't mix APC's caching abilities with its ability to optimize intermediate code and cache compiled code. APC provides 2 different things:
It gives a handy method of caching data structures (objects,
arrays etc), so that you can store/get them with apc_store and
apc_fetch
It keeps a compiled version of your scripts so that the
next time they run, they run faster
Let's see an example for (1): Suppose you have a data structure which takes 1 second to calculate:
function calculate_array() {
sleep(1);
return array('foo' => 'bar');
}
$data = calculate_array();
You can store its output so that you don't have to call the slow calculate_array() again:
function calculate_array() {
sleep(1);
return array('foo' => 'bar');
}
if (!apc_exists('key1')) {
$data = calculate_array();
apc_store('key1', $data);
} else {
$data = apc_fetch('key1');
}
which will be considerably faster, much less than the original 1 second.
Now, for (2) above: having APC will not make your program run faster than 1 second, which is the time that calculate_array() needs. However, if your file additionally needed (say) 100 milliseconds to initialize and execute, simply having enabled APC will make it need (approx) 20 millisecond. So you have an 80% increase in initialization/preparation time. This can make quite a difference in production systems, so simply installing APC can have a noticeable positive impact on your script's performance, even if you never explicitly call any of its functions
If you are just storing static data (as in your example), it would be preferable to use apc_store.
The reasoning behind this is not so much whether the opcode cache is faster or slower, but the fact you are using include to fetch static data into scope.
Even with an opcode cache, the file will still be checked for consistency on each execution. PHP will not have to parse the contents, but it will have to check whether the file exists, and that it hasn't changed since the opcode cache was created. Filesystem checks are resource expensive, even if it is only to stat a file.
Therefore, of the two approaches I would use apc_store to remove the filesystem checks completely.
Unlike the other answer I would use the array-file-solution (the first one)
<?php return (object) array(
'key' => 'value',
);
The reason is, that with both solutions you are on the right side, but when you let the caching up to APC itself you don't have to juggle around with the apc_*()-functions. You simply include and use it. When you set
apc.stat = 0
you avoid the stat-calls on every include too. This is useful for production, but remember to clear the system-cache on every deployment.
http://php.net/apc.configuration.php#ini.apc.stat
Oh, not to forget: With the file-approach it works even without APC. Useful for the development setup, where you usually shouldn't use any caching.