I'm adding a key using Memcached like so:
$valueToStore = time(); // some number
$success = $memcached->add( 'test_key', $valueToStore, 20 ); // cache for 20 seconds
But it's always succeeding when I call it in a different session, even before 20 seconds have passed. According to the docs at http://php.net/manual/en/memcached.add.php, it should be returning FALSE until the key expires (because the key already exists).
I'm running on a single development server with plenty of free cache space. Any idea what might be happening?
php -v returns: PHP 5.5.9-1ubuntu4.3
memcached version 2.1.0
libmemcached version 1.0.8.
You need to be distinct if you are using the Memcache class or the Memcached class. Your cache design is a bit strange. You should be checking the cache to first see if the item is there. If the item is not then store it. Also Memcache has some strange behavior on using the boolen type as the third argument. You should MEMCACHE_COMPRESSED. I think you are using Memcache.
To illustrate how to fix your problem:
$in_cache = $memcached->get('test_key');
if($in_cache)
return $in_cache;
else
$valueToStore = time();
$memcached->add('test_key', $valueToStore, MEMCACHE_COMPRESS, 20);
Related
Story
I have 3 memcached servers running where I shutdown the one or the other to investigate how PHP-memcached behaves upon a server not beeing reachable.
I have defined 4 servers in PHP, 1 to simulate a server that is mostly offline (spare server). When I shutdown 1 server (=> 2 are still online), the third ->get() gives me a result.
When I shutdown one more server (=> 1 is still online), it won't find objects pushed to that last server.
Sample output
First run, 3 of 4 servers up:
Entity not found in cache on 1st try: NOT FOUND
Entity not found in cache on 2nd try: NOT FOUND
Entity not found in cache on 3rd try: NOT FOUND
Entity not found in cache on 4th try: NOT FOUND
Second run, 3 of 4 servers up:
Entity found in Cache: SUCCESS
Third run, 2 of 4 servers up:
Entity not found in cache on 1st try: CONNECTION FAILURE
Entity not found in cache on 2nd try: SERVER IS MARKED DEAD
Entity not found in cache on 3rd try: NOT FOUND
Entity not found in cache on 4th try: NOT FOUND
Fourth run, 1 of 4 servers up:
Entity not found in cache on 1st try: CONNECTION FAILURE
Entity not found in cache on 2nd try: SERVER IS MARKED DEAD
Entity not found in cache on 3rd try: CONNECTION FAILURE
Entity not found in cache on 4th try: SERVER IS MARKED DEAD
Although there is one server left online and I do push my object to memcached everytime it does not find any in cache, it is not able to find the key anymore.
I think it should also work with only a single server left.
Can you explain this behaviour to me?
It looks like it is not possible to implement something that is safe even when I shutdown 19 of 20 servers.
Sidequestion: libketama is not really maintained anymore, is it still good to use it? The logic behind the lib was rather good and is also used in the varnish caching server.
Appendix
My Script:
<?php
require_once 'CachableEntity.php';
require_once 'TestEntity.php';
echo PHP_EOL;
$cache = new Memcached();
$cache->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$cache->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
$cache->setOption(Memcached::OPT_SERVER_FAILURE_LIMIT, 1);
$cache->setOption(Memcached::OPT_REMOVE_FAILED_SERVERS, true);
$cache->setOption(Memcached::OPT_AUTO_EJECT_HOSTS, true);
$cache->setOption(Memcached::OPT_TCP_NODELAY, true);
//$cache->setOption(Memcached::OPT_RETRY_TIMEOUT, 10);
$cache->addServers([
['localhost', '11212'],
['localhost', '11213'],
['localhost', '11214'],
['localhost', '11215'], // always offline
]);
$entityId = '/test/test/article_123456789.test';
$entity = new TestEntity($entityId);
$found = false;
$cacheKey = $entity->getCacheKey();
$cacheResult = $cache->get($cacheKey);
if (empty($cacheResult)) {
echo 'Entity not found in cache on 1st try: ' . $cache->getResultMessage(), PHP_EOL;
$cacheResult = $cache->get($cacheKey);
if (empty($cacheResult)) {
echo 'Entity not found in cache on 2nd try: ' . $cache->getResultMessage(), PHP_EOL;
$cacheResult = $cache->get($cacheKey);
if (empty($cacheResult)) {
echo 'Entity not found in cache on 3rd try: ' . $cache->getResultMessage(), PHP_EOL;
$cacheResult = $cache->get($cacheKey);
if (empty($cacheResult)) {
echo 'Entity not found in cache on 4th try: ' . $cache->getResultMessage(), PHP_EOL;
$entity
->setTitle('TEST')
->setText('Hellow w0rld. Lorem Orem Rem Em M IpsuM')
->setUrl('http://www.google.com/content-123456789.html');
$cache->set($cacheKey, $entity->serialize(), 120);
}
}
else { $found = true; }
}
else { $found = true; }
}
else { $found = true; }
if ($found === true) {
echo 'Entity found in Cache: ' . $cache->getResultMessage(), PHP_EOL;
$entity->unserialize($cacheResult);
echo 'Title: ' . $entity->getTitle(), PHP_EOL;
}
echo PHP_EOL;
The behaviour you are experiencing is consistent. When a server is not available it's first marked with a failure and then marked as dead.
The problem is that apparently it would only be coherent if you had set Memcached::OPT_SERVER_FAILURE_LIMIT value to 2 while you have set this to 1. This would have explained why you have two error lines per unreachable server (CONNECTION FAILURE, SERVER IS MARKED AS DEAD)
This seems to be related with timeout. Adding a usleep() after a failure with a matching OPT_RETRY_TIMEOUT value will enable the server to be dropped from the list (see the following bug comment)
The value does not replicate to the next server because only keys are distributed.
Note that OPT_LIBKETAMA_COMPATIBLE does not use libketama, but only reproduces the same algorithm, which means that it does not matter if libketama is no longer active while this is the recommended configuration in PHP documentation:
It is highly recommended to enable this option if you want to use consistent hashing, and it may be enabled by default in future releases.
EDIT:
In my understanding of your post, the message "Entity found in Cache: SUCCESS" only appears on the second run (1 server offline) because there's no change from the previous command and the server hosting this key is still available (so memcached consider from the key that the value is stored on either the 1st, 2nd or 3rd server). Let's call those servers John, George, Ringo and Paul.
In the third run, at start, memcached deduces from the key which one of the four servers owns the value (e.g. John). It asks John twice before giving up because it's now off. Its algorithm then only considers 3 servers (not knowing that Paul is already dead) and deduces that George should contain the value.
George answers twice that it does not contain the value and then store it.
But on the fourth run, John, George and Paul are off. Memcached tries John twice, and then tries George twice. It then stores in Ringo.
The problem here is that the unavailable servers are not memorized between different runs, and that within the same run you have to ask twice a server before it's removed.
Redundancy
Since Memcached 3.0.0, there is a redundancy configuration.
It can be made in the extension config file.
/etc/php/7.0/mods-available/memcached.ini (may be different among operating systems)
memcache.redundancy=2
with ini_set('memcache.redundancy', 2)
This parameter is not really documented, you may replace "2" by the number of servers, this will add slight overhead with extra writes.
Loosing 19/20 servers
With redundancy, you may loose some servers and keep a "read success".
Notes:
loosing 95% of a server pool will cause a stress on the remaining ones.
Cache servers are made for performances, "a large number of servers can slow down a client from a few angles"
libketama
Github repository has not received any commits since 2014. Libketama is looking for a new maintainer https://github.com/RJ/ketama
How to convert redis-cli script load $LUA_SCRIPT into Predis methods?
Follow is the lua script:
local lock_key = 'icicle-generator-lock'
local sequence_key = 'icicle-generator-sequence'
local logical_shard_id_key = 'icicle-generator-logical-shard-id'
local max_sequence = tonumber(KEYS[1])
local min_logical_shard_id = tonumber(KEYS[2])
local max_logical_shard_id = tonumber(KEYS[3])
local num_ids = tonumber(KEYS[4])
if redis.call('EXISTS', lock_key) == 1 then
redis.log(redis.LOG_INFO, 'Icicle: Cannot generate ID, waiting for lock to expire.')
return redis.error_reply('Icicle: Cannot generate ID, waiting for lock to expire.')
end
--[[
Increment by a set number, this can
--]]
local end_sequence = redis.call('INCRBY', sequence_key, num_ids)
local start_sequence = end_sequence - num_ids + 1
local logical_shard_id = tonumber(redis.call('GET', logical_shard_id_key)) or -1
if end_sequence >= max_sequence then
--[[
As the sequence is about to roll around, we can't generate another ID until we're sure we're not in the same
millisecond since we last rolled. This is because we may have already generated an ID with the same time and
sequence, and we cannot allow even the smallest possibility of duplicates. It's also because if we roll the sequence
around, we will start generating IDs with smaller values than the ones previously in this millisecond - that would
break our k-ordering guarantees!
The only way we can handle this is to block for a millisecond, as we can't store the time due the purity constraints
of Redis Lua scripts.
In addition to a neat side-effect of handling leap seconds (where milliseconds will last a little bit longer to bring
time back to where it should be) because Redis uses system time internally to expire keys, this prevents any duplicate
IDs from being generated if the rate of generation is greater than the maximum sequence per millisecond.
Note that it only blocks even it rolled around *not* in the same millisecond; this is because unless we do this, the
IDs won't remain ordered.
--]]
redis.log(redis.LOG_INFO, 'Icicle: Rolling sequence back to the start, locking for 1ms.')
redis.call('SET', sequence_key, '-1')
redis.call('PSETEX', lock_key, 1, 'lock')
end_sequence = max_sequence
end
--[[
The TIME command MUST be called after anything that mutates state, or the Redis server will error the script out.
This is to ensure the script is "pure" in the sense that randomness or time based input will not change the
outcome of the writes.
See the "Scripts as pure functions" section at http://redis.io/commands/eval for more information.
--]]
local time = redis.call('TIME')
return {
start_sequence,
end_sequence, -- Doesn't need conversion, the result of INCR or the variable set is always a number.
logical_shard_id,
tonumber(time[1]),
tonumber(time[2])
}
redis-cli script load $LUA_SCRIPT
I tried
$predis->eval(file_get_contents($luaPath), 0);
or
class ListPushRandomValue extends \Predis\Command\ScriptCommand {
public function getKeysCount() {
return 0;
}
public function getScript() {
$luaPath = Aa::$vendorRoot .'/icicle/id-generation.lua';
return file_get_contents($luaPath);
}
}
$predis->getProfile()->defineCommand('t', '\Controller\ListPushRandomValue');
$response = $predis->t();
But both of above showed error below.
ERR Error running script (call to f_5849d008682280eed4ec67b97ba50ae546fc5e8d): #user_script:19: user_script:19: attempt to perform arithmetic on local 'end_sequence' (a table value)
First, let me say that I am not an expert on LUA but I have just dealt with Redis and LUA script to implement simple locking and noticed a few errors in the question.
There is a conversion process between Redis and LUA that should be reviewed: Conversion. I think this will help with the error given.
On the this call:
$predis->eval(file_get_contents($luaPath), 0);
You pass the contents of the script and access keys but don't pass in any keys to evaluate. This call is more correct:
$predis->eval(file_get_contents($luaPath), 4, oneKey, twoKey, threeKey, fourKey);
This might actually be the reason for the above error. Hope this helps someone in the future.
I have a daemon script running in PHP. This script continuously hits an endpoint and gets the response from a resource and checks for a response.
The response is typically
{"a": 1, "b": 2, "c": 3, "status" : "true"}
most of the time.
Occasionally, it could turn out that the response might change to
{"a": 1, "b": 2, "c": 3, "status" : "false"}
I have logic to read values of a, b and c and perform operations based on them.
My question is, is there a way to somehow cache this response instead of hitting the end point over and over at the daemon level?
Or is there a way to cache the json response from the server level?
Use each element of your object as an array key:
$responses[$a][$b][$c]['status'] = true;
The cache would be a simple:
if (isset($responses[$a][$b][$c]['status'])) {
... used cached value
} else {
... compute new value and store in cache
}
Though you might want some extra logic in there to expire older entries, other you'll end up sucking up memory until things die.
If you want to cache the response from your daemon here are some options
1 - If on PHP < 5.5 , you can use apc's user cache
Here is how save some random piece of text (does not matter if its JSON or an object , you will be able to retrieve it fine)
apc_add('cache_key',$save_this_var, TTL);
OR
apc_store('cache_key',$save_this_var, TTL);
apc_add will not overwrite if you save something else with the exact same key , whereas apc_store will overwrite
TTL is the time (in seconds) you want to cache the data for
Now you can retrieve the cache like so
$my_cache = apc_fetch('cache_key');
$my_cache has the variable you saved earlier
2 - If you are using PHP 5.5 , then make use apc's user cache by adding the apcu module ,it works exactly like apc cache I explained above ,just a name change (apcu does not cache opcode whereas apc in PHP < 5.4 does)
3 - If you are in a cloud or multiserver environment , you can use memcached as well , but if on single machine , i would go with apc or apcu
I use "memcached" to store php sessions.
It is important, that request must be synchronously (to avoid duplicate transactions or operations), but while using "memcached" session, "session locking" not works.
is some method to lock "memcached" session until one request will executed?
There's nothing built in no, but you can write stuff yourself to make your code atomic.
$key = 'lockable_key_name';
$lockkey = $key.'##LOCK';
if($memcached->add($lockkey, '', 60)) {
$storedvalue = $memcached->get($key);
// do something with $storedvalue
$memcached->set($key, $newvalue);
// release
$memcached->delete($lockkey);
}
In your code you could check for the lock by doing:
if(!$memcached->get($lockkey)) {
// then do something
}
If the get method returns false then there's no lock, or the operation has hung and passed the 60 second timeout specified in the add call above.
Since you were asking for credible/official sources:
The memcached extension supports session locking since version 3.0.4, according to the changelog document on the PECL extension page: http://pecl.php.net/package-info.php?package=memcache&version=3.0.4
If you happen to run an earlier version (it means that your version of the memcached extension is more than 4 years old), you are out of luck and should upgrade.
Maybe try something like $(field_name)_is_locked = true when you start then when you are done $(field_name)_is_locked = false and pass the variable to the server when you update it.
On my website there is a php function func1(), which gets some info from other resources. It is very costly to run this function.
I want that when Visitor1 comes to my website then this func1() is executed and the value is stored in $variable1=func1(); in a text file (or something, but not a database).
Then a time interval of 5 min starts and when during this interval Visitor2 visits my website then he gets the value from the text file without calling the function func1().
When Visitor3 comes in 20 min, the function should be used again and store the new value for 5 minutes.
How to make it? A small working example would be nice.
Store it in a file, and check the file's timestamp with filemtime(). If it's too old, refresh it.
$maxage = 1200; // 20 minutes...
// If the file already exists and is older than the max age
// or doesn't exist yet...
if (!file_exists("file.txt") || (file_exists("file.txt") && filemtime("file.txt") < (time() - $maxage))) {
// Write a new value with file_put_contents()
$value = func1();
file_put_contents("file.txt", $value);
}
else {
// Otherwise read the value from the file...
$value = file_get_contents("file.txt");
}
Note: There are dedicated caching systems out there already, but if you only have this one value to worry about, this is a simple caching method.
What you are trying to accomplish is called caching. Some of the other answers you see here describe caching at it's simplest: to a file. There are many other options for caching depending on the size of the data, needs of the application, etc.
Here are some caching storage options:
File
Database/SQLite (yes, you can cache to a database)
MemCached
APC
XCache
There are also many things you can cache. Here are a few:
Plain Text/HTML
Serialized data such as PHP objects
Function Call output
Complete Pages
For a simple, yet very configurable way to cache, you can use the Zend_Cache component from the Zend Framework. This can be used on it's own without using the whole framework as described in this tutorial.
I saw somebody say use Sessions. This is not what you want as sessions are only available to the current user.
Here is an example using Zend_Cache:
include ‘library/Zend/Cache.php’;
// Unique cache tag
$cache_tag = "myFunction_Output";
// Lifetime set to 300 seconds = 5 minutes
$frontendOptions = array(
‘lifetime’ => 300,
‘automatic_serialization’ => true
);
$backendOptions = array(
‘cache_dir’ => ‘tmp/’
);
// Create cache object
$cache = Zend_Cache::factory(‘Core’, ‘File’, $frontendOptions, $backendOptions);
// Try to get data from cache
if(!($data = $cache->load($cache_tag)))
{
// Not found in cache, call function and save it
$data = myExpensiveFunction();
$cache->save($data, $cache_tag);
}
else
{
// Found data in cache, check it out
var_dump($data);
}
In a text file. Oldest way of saving stuff (almost). Or do a cronjob to run the script with the function each 5 minutes independently on the visits.
Use caching, such as APC!
If the resource is really big, this may not be the best option and a file may then indeed be better.
Look at:
apc_store
apc_fetch
Good luck!