Using the newer PHP pecl/memcached extension. Calls to Memcached::setOption() like;
$m = new Memcached();
$m->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
are costing between 150 to 500ms - just in making the call to setOption() and as we're not using persistent connections but rather doing this on every request, it hurts.
Delving deeper, setting Memcached::OPT_DISTRIBUTION to Memcached::DISTRIBUTION_CONSISTENT ends up calling update_continuum() in libmemcached which appears to be fairly intensive, although we're only passing a list of 15 memcached servers in, so somewhat surprising to see it take between 150 to 500ms to rebuild the continuum data structure.
Could it be setting this option is only suitable for persistent connections, where it's called only once while making the initial connection? Or is this a bug libmemcached?
Using the newer pecl/memcached extension 1.0.1 with libmemcached 0.38
Thanks.
libmemcached 0.38 is fairly old at this point. So is pecl/memcached 1.0.1. Could you try pecl/memcached 2.0.0b1 release from github?
Same issue with pecl/memcached 2.2.0 (latest from PECL as of writing). The only solution for me was to switch from DISTRIBUTION_CONSISTENT to:
$memcached->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$memcached->setOption(Memcached::OPT_REMOVE_FAILED_SERVERS, true);
It seems to work fine:
var_dump($memcached->getServerByKey($key)['host']);
// string '192.168.56.1' (length=12) <== dead host
$memcached->set($key, $result, 3600);
var_dump($memcached->getServerByKey($key)['host']);
// string '127.0.0.1' (length=9) <== working host
var_dump($memcached->getLastErrorMessage());
// string 'SUCCESS' (length=7)
Related
I have a script in python receiving continuously data from sensors.
I need to publish the last data at request on a webpage using php.
Apache, php and python are all on the same linux install (actually on an raspberry).
I'm not interested in storing previous data and I'm a bit concerned about data corruption on writing on SD. I would prefer to reduce complexity and increase speed refresh (avoid sql).
Could a text file written in ramfs / tmpfs solve the problem? or there is a method to share memory like memcache to share also global variables?
Any practical example or howto will be really well-accepted.
I think you may try System V shared memory.
As example:
In Python side:
python -m pip install sysv_ipc
then somewhere in the python script:
import sysv_ipc
...
KEY=20171220
sysv_memory=sysv_ipc.SharedMemory(KEY, sysv_ipc.IPC_CREAT, 0666, 256)
...
sysv_memory.write('1234'+'\0')
Then in the PHP side:
$SHARED_MEMORY_KEY = 20171220;
...
$shmId = shmop_open($SHARED_MEMORY_KEY, 'a', 0666, 256);
...
$test_string = shmop_read($shmId, 0, 0);
I can get the $test_string as '1234' successfully in PHP.
Here is a solution using memcached that works on Raspbian 10 (buster), PHP 7.3.19, and Python 3.7.3:
1. Install memcached
apt-get install memcached php-memcached
pip install pymemcache
These commands install memchached server and client modules for PHP and Python.
2. PHP code
$m = new Memcached();
// connect
$m->addServer('127.0.0.1', 11211);
// get a value
$value = $m->get('key');
// set a value
$m->set('key', 'value');
// clean up
$m->quit();
3. Python code
from pymemcache.client import base
# connect
m = base.Client(('127.0.0.1', 11211))
# get a value
value = m.get('key')
# set a value
m.set('key', 'value')
# clean up
m.close()
Note:
I used default memcached settings here. If you need to change them edit sudo nano /etc/memcached.conf and restart the daemon: sudo /etc/init.d/memcached restart.
You can use any interoperable format like json or msgpack.
Whenever you generate data in python, add it to a caching layer like memcache/redis using json format ( and preferably a gzip compressed version), then your PHP script can unserialize the json data and display it in the app.
Clearly memcache as a means to share data different application will work.
You will not have any corrupted data for sure as all the memcache operation are atomic. memcache atomic discussion could be useful.
On memcached's FAQ:
Is memcached atomic? Aside from any bugs you may come across, yes all commands are internally atomic. Issuing multiple sets at the same time has no ill effect, aside from the last one in being the one that sticks.
Note: Running memcache service might consume considerable amount of memory.
Hope it helps!
Simple setup:
1 node running twemproxy (vcache:22122)
2 nodes running memcached (vcache-1, vcache-2) both listening on 11211
I have the following twemproxy config:
default:
auto_eject_hosts: true
distribution: ketama
hash: fnv1a_64
listen: 0.0.0.0:22122
server_failure_limit: 1
server_retry_timeout: 600000 # 600sec, 10m
timeout: 100
servers:
- vcache-1:11211:1
- vcache-2:11211:1
The twemproxy node can resolve all hostnames. As part of testing I took down vcache-2. In theory for every attempt to interface with vcache:22122, twemproxy will contact a server from the pool to facilitate the attempt. However, if one of the cache nodes is down, then twemproxy is supposed to "auto eject" it from the pool, so subsequent requests will not fail.
It is up to the app layer to determine if a failed interface attempt with vcache:22122 was due to infrastructure issue, and if so, try again. However I am finding that on the retry, the same failed server is being used, so instead of subsequent attempts being passed to a known good cache node (in this case vcache-1) they are still being passed to the ejected cache node (vcache-2).
Here's the php code snippet which attempts the retry:
....
// $this is a Memcached object with vcache:22122 in the server list
$retryCount = 0;
do {
$status = $this->set($key, $value, $expiry);
if (Memcached::RES_SUCCESS === $this->getResultCode()) {
return true;
}
} while (++$retryCount < 3);
return false;
-- Update --
Link to Issue opened on Github for more info: Issue #427
I can't see anything wrong with your configuration. As you know the important settings are in place:
default:
auto_eject_hosts: true
server_failure_limit: 1
The documentation suggests connection timeouts might be an issue.
Relying only on client-side timeouts has the adverse effect of the original request having timedout on the client to proxy connection, but still pending and outstanding on the proxy to server connection. This further gets exacerbated when client retries the original request.
Is your PHP script closing the connection and retrying before twemproxy failed its first attempt and removed the server from the pool? Perhaps adding a timeout value in the twemproxy lower than the connection timeout used in PHP solves the issue.
From your discussion on Github though it sounds like support for healthcheck, and perhaps auto ejection, aren't stable in twemproxy. If you're building against old packages you might be better to find a package which has been stable for some time. Is mcrouter (with interesting article) suitable?
For this feature to work please merge with this repo/branch
https://github.com/charsyam/twemproxy/tree/feature/heartbeat
to have this specific commit
https://github.com/charsyam/twemproxy/commit/4d49d2ecd9e1d60f18e665570e4ad1a2ba9b65b1
here is the PR
https://github.com/twitter/twemproxy/pull/428
after that recompile it
Connecting to Cassandra with PHP is really a pain. The documentations at Apache and DataStax are extremely poorly written - for Windows users.
I have Thrift installed (I believe!) via Chocolatey. But I am still not able to compile php code for Cassandra using thrift.
If you look at this link ,
now we can compile php code for Cassandra using thrift I used command: d:\cassandra\trift\thrift.exe --gen php
d:\cassandra\interface\cassandra.thrift
So what is cassandra.thrift and where does it come from?? WHAT should I put inside it??
If I follow the instruction exactly, I get this error,
Could not open input file: d:\cassandra\interface\cassandra.thrift
So what is going on?
How do I make this work?
I have tried to install DataStax PHP Driver for Apache Cassandra and that documentation even worst.
Why PHP modules do not come with Cassandra like it does for MongoDB? Most of the independent drivers I found are outdated, not supported anymore or abandoned.
EDIT:
From the README,
Install the PHP extension
Installing with pecl
The PHP driver is not published to the official PECL repository yes.
You can still install it using pecl by specifying the provided
package.xml file path as the argument to pecl install command.
Install the 2.0 version of the C/C++ driver
not published to the official PECL repository yes - is it yes or yet?
Obtaining Build Dependencies
CMake
Git
ActiveState Perl
Python v2.7.x
I have downloaded and installed. Then, what? In Building the Driver,
A batch script has been created to detect installed versions of Visual...
What? Where does A batch script suddenly come from??
Then,
First you will need to open a “Command Prompt” (or Windows SDK Command
Prompt) to execute the batch script.
Usage: VC_BUILD.BAT [OPTION...]
--DEBUG Enable debug build
--RELEASE Enable release build (default)
--DISABLE-CLEAN Disable clean build
....
What are these bunch of '--' for?
To build 32-bit shared library:
VC_BUILD.BAT --X86 To build 64-bit shared library:
VC_BUILD.BAT --X64
Where does .BAT come from? What should I put inside it? Where should I run it from??
After all, what are those Build Dependencies for? How do I use them??
Just hope that someone can write a proper guide then the guide above - it is frightening! (if you compare the guides in MongoDB, it is far better and professional)
EDIT 2:
First error when I run the .bat from my desktop,
I have git installed already but I still have this error,
After fixing git issue above, I have a new one - it just frozen there, nothing happens,
The IDL file cassandra.thrift is usually part of the cassandra package, but you can find it following the link above. The link points to trunk, you may want another version.
After you downloaded the right version of that file or better found it in your downloaded Cassandra package in the interface folder, generate the code as outlined in the documentation you have. The rest should be easy.
Why PHP modules do not come with Cassandra like it does for MongoDB? Most of the independent drivers I found are outdated, not supported anymore or abandoned.
I'm not really sure about that, but my guess is that the fact CQL is promoted heavily for a while now instead of using the raw Thrift API - the latter is a complex task, while CQL is more easy to use - is one of the key factors why it is this way. It more or less eliminates the need for another wrapper.
PS: Just to be sure:
thrift.exe --gen php d:\cassandra\interface\cassandra.thrift
could not open input file cassandra.thrift
Of course you point to the right drives, folders and files, do you?
Forget about Thrift and the 'beta', I found a better solution. It very straight forward and extremely easy!
Example codes,
require_once 'lib/Cassandra/Cassandra.php';
$cassandra = new Cassandra();
$s_server_host = '127.0.0.1'; // Localhost
$i_server_port = 9042;
$s_server_username = 'admin'; // We don't use username
$s_server_password = 'password'; // We don't use password
$s_server_keyspace = 'demo'; // We don't have created it yet
$cassandra->connect($s_server_host, $s_server_username, $s_server_password, $s_server_keyspace, $i_server_port);
// Tests if the connection was successful:
if ($cassandra) {
// Select:
// Queries a table.
$cql = "SELECT * FROM users;";
// Launch the query.
$results = $cassandra->query($cql);
// Update:
// Prepares a statement.
$stmt = $cassandra->prepare('UPDATE users SET first_name = ?, last_name = ? where id = ?');
// Executes a prepared statement.
$values = array('first_name' => 'Fred', 'last_name' => 'Smith', 'id' => '1');
$result = $cassandra->execute($stmt, $values);
// Insert:
// Prepares a statement.
$stmt = $cassandra->prepare('INSERT INTO users (id, first_name, last_name)
VALUES (:id, :first_name, :last_name)');
// Executes a prepared statement.
$values = array('first_name' => 'John', 'last_name' => 'Robinson', 'id' => '4');
$result = $cassandra->execute($stmt, $values);
// Delete:
// Prepares a statement.
$stmt = $cassandra->prepare('DELETE FROM users WHERE id = :id');
// Executes a prepared statement.
$values = array('id' => '4');
$result = $cassandra->execute($stmt, $values);
// Closes the connection.
$cassandra->close();
}
Update:
The datastax PHP driver is now GA and binaries are available for download (no need to build it yourself):
https://github.com/datastax/php-driver
Regarding the DataStax PHP driver, the instructions are being improved per your feedback as I type.
Because this Driver is in Beta, we do not yet have pre compiled binaries that you can simply download. They will be available once the driver is GA. For now you will have to build them yourself.
The process for building the binaries is very straight forward. 1) Install the dependencies 2) run vc_build.bat.
You can find the vc_build.bat here (just right click save as from your browser):
https://raw.githubusercontent.com/datastax/php-driver/master/ext/vc_build.bat
I have a simple example where I set a value for 5 seconds. The problem is that after 5 seconds; I still get back a value when I expected 'false'.
$memcache = new Memcache;
$memcache->connect('localhost', 11211) or die ("Could not connect");
$memcache->set('foo', 'bar', 0, 5); // 5 seconds expiry
var_dump($memcache->get('foo')); // bar
sleep(10);
var_dump($memcache->get('foo')); // still shows bar
Here is the memcache server version
Server's version: 1.4.13
Late to the game, but in your code it looks like you are passing "0" for the expiration (not "5"), which translates to "never expire" Specifically:
$memcache->set('foo', 'bar', 0, 5); // 5 seconds expiry
Should be:
$memcache->set('foo', 'bar', 5); // 5 seconds expiry
Unless I'm misunderstanding the PHP documentation located here, which shows that the set command takes three parameters:
public bool Memcached::set ( string $key , mixed $value [, int $expiration ] )
Edit: Whoops, I see that you're using the Memcache extension and not Memcached, which does have four parmaters. Perhaps try using the MEMCACHE_COMPRESSED constant instead of 0 to see if it works:
$memcache->set('foo', 'bar', MEMCACHE_COMPRESSED, 5); // 5 seconds expiry
There was a bug in the memcached server including version 1.4.13 through at least 1.4.14 that, once the memcached server had been running for some time, it would sometimes get into a mode where it would fail to properly expire values.
Restarting the service fixed it for me, and I'm hopeful that the newer versions fix it more permanently.
As the code looks fine - the next chain down the line is to look at either your version of the memcache PHP extension not working, or memcached server itself.
This get a but tricky. Easiest is to rule out memcached server first. (There's a php interface you can install - but that won't help you work outwhich bit.) So...
In terminal (or command window in Windows) type
telnet localhost 11211
(Note - telnet client is not installed in Windows by default - go to "control panel", "turn windows features on or off" and add from there.)
This gives you access to memcached.
Then type
stats items
which lists the memory items in memcached.
Hopefully you've only got one slab, so note its number and type
stats cachedump [Number] 0
And this will list what's recorded in cache.
If this still shows "bar" (in encoded format) then it's memcached server that's not working - upgrade for a newer version.
If this doesn't show "bar" (or, preferably, the item just doesn't exist - you get ERROR instead) then it'll be the memcache extension to PHP that's not working. Again, check your version of that.
When done, type
quit
Alternative is to check out "memcached" (php extension) and rewrite your PHP code with those classes. It's newer. If that still fails, it's definately memcached server; if that works that it was the php memcache extension.
How can I use memcache in Joomla? I'm a newbie in this field so please be descriptive with your answer.
You will need to install memcached on your server and will probably need root access to do so. You can get memcached from http://www.danga.com/memcached/. It requires libevent, which can be downloaded here: http://www.monkey.org/~provos/libevent/
Finally, you'll need to get the PHP PECL extension for memcache. To install this, you need to go to the server where PHP is installed and run this command:
pecl install memcache
Again, you will most likely need root access to your server to do this. After you have libevent, memcached, and the PECL extensions installed, go to the Global Configuration in Joomla and choose Memory Cache as the Cache Handler under Cache Settings. After you save the Global Configuration, open it again and more inputs should appear underneath the Cache Handler input. Set Memory Chache Server to localhost and the port to 11211. This should match the parameters you use to run memcached from the command line.
EDIT: You can also use XCache not only to store data in a way similar to Memcache, but it will also cache the opcode generated by PHP. This way, instead of reading the PHP code from disk and parsing it each time, it will hold the code in memory for the next request.
Be sure to select XCache as the Cache Handler in Global Configuration. Read this for information on installing XCache: http://xcache.lighttpd.net/wiki/InstallFromSource
In order to make Joomla to use memcache for session caching you need to manually edit the configuration.php and change this line:
public $session_handler = 'database';
to this one:
public $session_handler = 'memcache';
And this is what is missing everywhere, you need to add a new option memcache_settings:
public $memcache_settings = 'a:3:{s:10:"persistent";s:1:"0";s:11:"compression";s:1:"0";s:7:"servers";a:1:{i:0;a:2:{s:4:"host";s:9:"127.0.0.1";s:4:"port";s:5:"11211";}}}';
This is a serialized multy-dimentianal array. I use this code to generate the above string:
$a = array(
"persistent" => "0",
"compression" => "0",
"servers" => array(
"0" => array(
"host" => "127.0.0.1", "port" => "11211")
)
);
echo(serialize($a));
If you don't add the memcache_settings option your sessions will never work with memcache.
These settings in configuration.php work for memcached in Joomla 3.3+
public $cache_handler = 'memcached';
public $memcached_server_host = '127.0.0.1';
public $memcached_server_port = '11211';
public $memcached_persist = '1';
public $memcached_compress = '1';
public $cachetime = '15';
public $session_handler = 'memcached';
public $session_memcached_server_host = '127.0.0.1';
public $session_memcached_server_port = '11211';
To install memcached on a Debian system:
apt-get install memcached php5-memcached
(you can also use memcache in place of each occurence of memcached in the above public variables & install the older php5-memcache extension)
Upgrading to php5.5 will give you a builtin Zend Opcode Cache - this could be used with APCu (APCu is the Alternative PHP Cache with the Opcode Cache removed) - to cache userland locally if you do not need a distributed memory cache (memcached)
For a single VPS APCu has a lower memory footprint & is a more suitable cache (especially with php5.5) & can be set in configuration.php with:
public $cache_handler = 'apc';
public $session_handler = 'apc';
This how to might also offer some help for Joomla 2.5 as it points to the Joomla Admin Screen to use memcache http://www.siteground.com/tutorials/supercacher/joomla_memcached.htm
It's better to not use APC for backend cache. This can cause a lot of fragmentation. I have only seen it degrade performance of Joomla.