I just designed an admin console for a social networking website. My boss now wants me to cache the results of several MySQL queries that build these results (for 24 hours). The site uses Memcached (with Wordpress W3 total cache) and XCache. I wanted to know what's the best way to do this.
Here is an example of one such query and how I am getting the results (basically I am returning aggregate stats on users, which means my results are fairly simple, eg:
//users who registered in last 365 days
$users_reg_365 = "select ID
from wp_users
where user_registered > NOW() - interval 365 day";
then use the wpdb query class to get the results:
$users_reg_365 = $wpdb->get_results($users_reg_365);
then display the result in the dashboard:
<li><?php echo "Total users who registered within last 365 days: <span class='sqlresult'>" . sizeof($users_reg_365) . "</span>"; ?></li>
My understanding of Memcached/XCache is that it basically stores strings, so would it make sense to just cache sizeof($users_reg_365)?
The last wrinkle is that our Wordpress site uses W3 total cache, which leverages Memcached, and the boss asked me not to use Memcached but XCache instead, but I find the docs a bit confusing. What's the best way to solve this problem? Can SQL itself be told to 'remember' certain queries like this, or is memory caching the way to go?
Thanks!
You can find more about the differences of both here:
Difference between Memcache, APC, XCache and other alternatives I've not heard of
An example how you could
<?php
$m = new Memcached();
$m->addServer('localhost', 11211);
// cache 24hrs
$cache_expire = 86400;
// users is your key
$users_reg_365 = $m->get('users_reg_365');
if (empty($users_reg_365)) {
$users_reg_365 = "select ID from wp_users where user_registered > NOW() - interval 365 day";
$m->set('users_reg_365', $cache_expire);
}
If you need to exactly refresh the cache at middle night change the value of $cache_expire.
You can refer to the full reference of memcached at
http://www.php.net/manual/en/memcached.get.php
Hm. I'm not sure how your caching is configured inside of WordPress. If you have WordPress's object cache set up to use Memcache(d)/XCache for persistent caching, you could do something like this:
$key = 'custom-key-to-look-up-later';
$data = sizeof( $users_reg_365 );
$group = 'group-id';
$expire = 60 * 60 * 24 // time in seconds before expiring the cache.
wp_cache_set( $key, $data, $group, $expire );
Later you can look up that value like this:
$data = wp_cache_get( $key, $group );
if( ! is_wp_error( $data ) )
// the data is good. do something with it.
You can find the docs on these functions here.
Before you begin, though, make sure WordPress is set up to work with Memcache(d) or XCache. :)
IF you want to store the results of any array just serialize/jsonencode it and store as that..
Related
I've been searching for a suitable PHP caching method for MSSQL results.
Most of the examples I can find suggest storing the results in an array, which would then get included to page. This seems great unless a request for the content was made at the same time as it being updated/rebuilt.
I was hoping to find something similar to ASP's application level variables, but far as I'm aware, PHP doesn't offer this functionality?
The problem I'm facing is I need to perform 6 queries on page to populate dropdown boxes. This happens on the vast majority of pages. It's also not an option to combine the queries. The cached data will also need to be rebuilt sporadically, when the system changes. This could be once a day, once a week or a month. Any advice will be greatly received, thanks!
You can use Redis server and phpredis PHP extension to cache results fetched from database:
$redis = new Redis();
$redis->connect('/tmp/redis.sock');
$sql = "SELECT something FROM sometable WHERE condition";
$sql_hash = md5($sql);
$redis_key = "dbcache:${sql_hash}";
$ttl = 3600; // values expire in 1 hour
if ($result = $redis->get($redis_key)) {
$result = json_decode($result, true);
} else {
$result = Db::fetchArray($sql);
$redis->setex($redis_key, $ttl, json_encode($result));
}
(Error checks skipped for clarity)
I'm writing a chat program for a site that does live broadcasting, and like you can guess with any non application driven chat it relies on a looping AJAX call to get new information (messages) in my case once every 2 seconds. My JSON that is being created via PHP and populated by SQL is of some concern to me, while it shows no noticeable impact on my server at present I cannot predict what adding several hundred users to the mix may do.
<?PHP
require_once("../../../../wp-load.php");
global $wpdb;
$table_name = $wpdb->prefix . "chat_posts";
$posts = $wpdb->get_results("SELECT * FROM ". $table_name ." WHERE ID > ". $_GET['last'] . " ORDER BY ID");
echo json_encode($posts);
?>
There obviously isn't much wiggle room as far as optimizing the code itself, but I am a little worried about how well the Wordpress SQL engine is written and if it will bog my SQL down once it gets to the point where it is receiving 200 requests every 2 seconds. Would caching the json encoded results of the DB query to a file then age checking it against new calls to the PHP script and either re-constructing the file with a new query or passing the files contents based on its last modification date be a better way to handle this? At that point I am putting a bigger load on my file-system but reducing my SQL load to one query every 2 seconds regardless of number of users.
Or am I already on the right path with just querying the server on every call?
So this is what I came up with, I went the DB only route for a few tests and while response was snappy, it didn't scale well and connections quickly got eaten up. So I decided to write a quick little bit of caching logic. So far it has worked wonderfully and seems to allow me to scale my chat as big as I want.
$cacheFile = 'cache/chat_'.$_GET['last'].'.json';
if (file_exists($cacheFile) && filemtime($cacheFile) + QUERY_REFRESH_RATE > time())
{
readfile($cacheFile);
} else {
require_once("../../../../wp-load.php");
$timestampMin = gmdate("Y-m-d H:i:s", (time() - 7200));
$sql= "/*qc=on*/" . "SELECT * FROM ". DB_TABLE ."chat_posts WHERE ID > ". $_GET['last'] . " AND timestamp > '".$timestampMin."' ORDER BY ID;";
$posts = $wpdb->get_results($sql);
$json = json_encode($posts);
echo $json;
file_put_contents($cacheFile,$json);
}
Its also great in that it allows me to run my formatting functions against messages such as parsing URL's into actual links and such with much less overhead.
I am wondering how I would use Cache (MemCache) to display how many registered and active users I have on my website. I currently use:
return mysql_result(mysql_query("SELECT COUNT(`user_id`) FROM `users` WHERE `active` = 1"), 0);
And this isn't good to use because on each page load, it's going to query the database. Which isn't good. So how would I go upon making a "memcache" or something of that sort, where it would update that number to all users, after say 10 minutes or so?
I googled how but I couldn't really get a good how to on it.
Thanks!
I'll give you the guidelines rather than a full documentation/tutorial.
Before trying to install Memcached, you should use MySQLi or PDO instead of mysql_* functions. Checkout the warning message in this documentation.
You should install Memcached and its PHP extension.
Once it is done, make sure it is enabled by calling the function phpinfo(). If Memcached isn't there the installation failed.
Here is an explicit code snippet
$mc = new Memcached();
$mc->addServer('localhost', 11211); // add a new server to the pool
// you will want to vary this key according to the query
// if two different queries use the same key your functions will return unexpected data
$cacheKey = 'a-custom-key-that-defines-the-data-within-it';
// how long will it take for the cache to expire in seconds : an hour
$cacheLifetime = 3600;
$data = $mc->get($cacheKey); // we get the data from cache
// check if Memcached was able to retrieve the data
if ($mc->getResultCode() === Memcached::RES_FAILURE) {
// if for some reasons the data is not there we get it and reinject into memcached
$data = mysql_result(mysql_query("SELECT COUNT(`user_id`) FROM `users` WHERE `active` = 1"), 0);
$mc->set($cacheKey, $data, $cacheLifetime); // save the data in the defined $cacheKey
}
return $data;
i want make an unique counter which tracks following, every referrer and ip from where the user comes. Memcached should store that data and than it should be entered into the database after 5 mins or so. And also it should check for duplicate entries that not twice the ip is written to the DB
I have an idea to do it without memcache, but than every sec something would be written into the database which would make the site slow.
I can cache basic SQL with memcache but i have no clue how make that above, just learned coding. So im a totally noob :)
Thank you for your help.
As I understand it - you are using a RDBMS to store the information for long term. I can suggest 2 things:
1. Store information in key/value store such as Memcached and write it to RDBMS periodically through a background process (CRON job).
With this solution you would do something like this in your application code:
$m = new Memcached();
$m->addServer('localhost', 11211);
$m->setOption(Memcached::OPT_COMPRESSION, false);
$m->append('some_unique_id', '|' . time() . ':' . base64_encode($_SERVER['HTTP_REFERER']));
And then you would have a CRON job doing something like this:
$m = new Memcached();
$m->addServer('localhost', 11211);
foreach($list_of_unique_ids as $uid)
{
$raw_data = $m->get($uid);
$m->delete($uid);
$uid_safe = mysql_real_escape_string($uid);
$items = explode('|', $raw_data);
$sql = array();
foreach($items as $rec)
{
if (!empty($rec))
{
list($timestamp, $ref_base64) = explode(':', $rec, 2);
$ref = base64_decode($ref_base64);
$ref_safe = mysql_real_escape_string($ref);
$sql[] = "('" . $uid_safe . "', " . (int) $timestamp . ", '" . $ref_safe . "')";
}
}
mysql_query("INSERT INTO some_table (user_id, ts, ref) VALUES " . implode(',', $sql));
}
2. Use a more advanced key/value store such as Redis or MongoDB as primary means of long-term storage.
Just pick a database engine with huge write throughput and call it a day.
> i want make an unique counter which
> tracks following, every referrer and
> ip from where the user comes.
> Memcached should store that data and
> than it should be entered into the
> database after 5 mins or so. And also
> it should check for duplicate entries
> that not twice the ip is written to
> the DB
I would advise you to use redis to implement this because redis has all the commands needed to do this efficiently and has persistent snapshots. To count you simply use the incrby/incr => decr/decbry commands.
If you installed memcached on your box, then installing redis is going to be a sinch. Just make will be enough. A popular client to connect to redis from PHP is predis.
If you can not install software you also have the option to use the free plan(5 MB memory, 1 Database, but no backups) from http://redistogo.com. Than you need to do the backups to MySQL manually because snapshots are disabled with free plan(probably better of getting a dedicated box or buying mini plan for $5/month).
I have a Facebook application with 250K DAU. I use Memcached to cache the files and MySql selects because without that my server is going crazy.
I have set up the code like this:
$memcache = new Memcache;
$memcache->connect('127.0.0.1', 11211) or die ("Could not connect");
// FOR FILES
$doc = $memcache->get('mem_index_html');
if ($doc == null)
{
$doc = new Document( HTML.'index.html' );
$memcache->set('mem_index_html', $doc, 0, 86400);
}
// FOR SQL
$sql=sprintf('count(qtn_id) FROM tb_questions where qtn_lang="EN"));
$total_pages = $memcache->get($sql);
if ($total_pages == null)
{
$query_id = DB::query($sql);
list( $total_pages ) = DB::nextRow( $query_id );
DB::free( $query_id );
$memcache->set($sql, $total_pages, 0, 86400);
}
The problem is that after some time – usually 24 hours, errors are starting to show. No log is created but I receive a timeouts if I try to reload the page. I assume that there is a lots of traffic in that moments, the Memcached is not working correct and everything is slowing down.
My temporary solution is to Flush the memcache - $memcache->flush(), but that is not good solution.
Am I doing something wrong? Is there something to optimize or I don’t know. I found searching thru web this http://code.google.com/p/memcached/wiki/TutorialCachingStory about making more that one Memecached servers. Is that the best solution?
Thanks for your answers.
Darko
The fourth parameter in the $memcache->set command represents the expiration timeout. Once this is exceeded, the object stored will no longer be available. Since you set it to 24 hours they would not be available after that period unless you updated them.
If the expiry parameter is set to 0, the item will never be expire, but it can still be deleted if memcached is filled up and need to free some space.
If the value is less than 30 days (30 * 24 * 3600 = 2592000), the value provided is treated as a offset from the current time.
If the value is larger than that it will be interpreted as absolute time.