I need to know if I can improve the way I cache my api calls from the inside of my CodeIgniter app.The way I do it right now is like this, in a hmvc pattern:
Controller HOME == calls to => module app/application/modules/apis/controllers/c_$api == loads library => app/application/libraries/$api ==> Library returns response to module's controller_X, the controller invokes the view with the data it has
//Note: My app does not use twitter api, but others
Inside the apis module is where the all the apc caching is happening, like so:
// Load up drivers
$this->load->library('driver');
$this->load->driver('cache', array('adapter' => 'apc'));
// Get Tweets from Cache
$tweets = $this->cache->get('my_tweets');
if ( ! $tweets)
{
// No tweets in the cache, so get new ones.
$url = 'http://api.twitter.com/1/statuses/user_timeline.json?screen_name=gaker&count=5';
$tweets = json_decode(file_get_contents($url));
$this->cache->save('my_tweets',$tweets, 300);
}
return $tweets;
as explained in this article: http://www.gregaker.net/2011/feb/12/codeigniter-reactors-caching-drivers/
So I was wondering:
Having 3 scenarios: home, query, result; in each module apis's controller, do you think it would be a good idea to implement cache for each controller with all the scenarios? example:
//for each api1, api2 ... apiX, apply this:
//home
$this->cache->save('api_home',$api_home, 300);
//query
$this->cache->save("api_$query", $api_{$query}, 300); // I don't know for sure if $api_{$query} works or not, so don't hang me because I haven't tried it.
//result
$this->cache->save("api_$queryId", $api_{$queryId}, 300);
Even though I cached the api call, do you think I should cache the result in the controller that is calling the api module controller, with the same 3 scenarios (home, query and result)? Like so:
//modules/{home,fetch,article}/controllers/{home,fetch,article}.php
//home
$homeData['latest'][$api] = modules::run("apis/c_$api/data", array('action'=>'topRated'));
$this->cache->save('home_data', $home_data, 300);
//query
$searchResults[$api] = modules::run("apis/c_$api/data", $parameters);
$this->cache->save("search_results_$query", $search_results_{$query}, 300);
//article page
$result = modules::run("apis/c_$api/data", $parameters);
$this->cache->save("$api_article_$id", ${$api}_article_{$id}, 300);
So, what do you think? Is it a good practice the mentioned above, or just an awful stupid one?
//Note, the suggested caching ideas were not tested... so, I don't know if ${$api}_article_{$id} will work or not (even though I suppose it will)
IMHO It is a good idea to cache api results if you don't need real time data. If you don't care that you won't see new data for an hour, then by all means cache it for an hour. So for your first question, you just need to ask yourself: "How fresh does the content need to be for my application?" and implement caching accordingly.
For the second question: I don't see a lot of value in caching content if it's only been manipulated in simple ways. At that point you're using up space in your cache and not getting a lot of value. But if there are database, or other api calls being made using that data, then yes they should be cached using a technique similar to the above.
If you're that worried about processor load (the only reason to cache content after manipulation) you're best bet is to look at something like Varnish or CloudFront.
Related
I am working on a TYPO3 project where I have to dynamically disable caching based on a condition. It is a very specific usecase, that will not happen a lot.
I planned to use a USER_INT function, where I would perform the check and disable the cache if necessary. The USER_INT function works flawlessly, it is being called on every page load.
The thing is, I can not disable the cache, or at least I do not know how.
The code, I have right now:
page = PAGE
page {
typeNum = 0
adminPanelStyles = 0
11 = USER_INT
11.userFunc = [COMPANY_NAMESPACE]\PageHandler->checkCache
And in the function I perform the check:
public function checkCache($content,$conf){
global $TSFE;
$id = $TSFE->id;
if($this->checkIfDisableCache($id)){
//$TSFE->set_no_cache(); // <---- first I tried this one
$TSFE->no_cache=true; // <-----after a while I got despoerate and tried to disable it directly
}
}
I also tried to play with the config, it did not work.
The funny thing is, if I set it directly in typoscript:
config.no_cache = 1
it works, but since the check is rather complex, I want to use PHP to determine, if the cache should be disabled.
I know I am doing something wrong, I just don't know what. Any help would be appretiated :)
I don't think either of the previous answers really explain the situation. You have sort of a catch-22 here, in that your USER_INT is executed after the page cache entry has been generated. The way it works internally is everything that can be cached gets rendered first, and every USER_INT then outputs a marker in the HTML source which gets replaced afterwards. This way the cache can contain the version with markers and those can be rendered without having to render the whole page.
So what you need to do in this case if you want the page cache to be disabled only in some conditions, is to use a custom TypoScript condition that is capable of setting config.no_cache = 1 only under special circumstances. That way you prevent generating a cache entry if the condition is met, but preserve full caching and cached output for every other request.
https://docs.typo3.org/typo3cms/TyposcriptSyntaxReference/TypoScriptParserApi/CustomConditions/Index.html
Note that it is still recommended that you instead create the parts of your page that must not be cached, as USER_INT objects. Having a use case where you in some cases need to disable the entire page cache indicates a possible misunderstanding of how the caching framework and/or USER_INT works. Hopefully the above explains those parts a bit.
if you look at the pibase (AbstractPlugin) code you will see that probably setting $conf['useCacheHash']and $conf['no_cache'] should be done.
https://api.typo3.org/typo3cms/current/html/_abstract_plugin_8php_source.html#l00190
If you create this object as USER_INT, it will be rendered non-cached, outside the main page-rendering.
https://docs.typo3.org/typo3cms/TyposcriptReference/ContentObjects/UserAndUserInt/Index.html
Our site uses the Vimeo PHP library (https://github.com/vimeo/vimeo.php).
Currently I'm calling the library within snippets, e.g.:
require_once("____/autoload.php");
$vimeo = new \Vimeo\Vimeo(____AuthKeys, etc.___);
...
$videos = $vimeo->request('/me/albums/____)['body']['data']
...
But this means way more calls to the API than necessary ... right?
Vimeo recommends caching the response, but I'm not sure how to do that in modx.
I'm guessing the first 3 lines only need to be run once, then cached ... until we make changes to our Vimeo account (add videos, albums, etc.)
What's the best way to accomplish this?
The only part that changes from snippet to snippet is the $vimeo->request... portion ... is there a way to only have that at the start of our snippets?
You can use getCache to cache the complete output for a longer period of time, but if you want to cache data inside your snippet, you can use the modCacheManager for that.
For example, that might look like this:
require_once("____/autoload.php");
$vimeo = new \Vimeo\Vimeo(____AuthKeys, etc.___);
...
$cacheManager = $modx->getCacheManager();
$videos = $cacheManager->get('vimeo_videos');
if (empty($videos)) {
$videos = $vimeo->request('/me/albums/____')['body']['data']
$cacheManager->set('vimeo_videos', $videos, 3600);
}
// Process $videos further
That will cache the data for one hour (note the 3600 in the set call).
Im looking for an elegant way to hand over data/params when using $f3->reroute();
I have multiple routes configured in a routes.ini:
GET #sso: /sso/first [sync] = Controller\Ccp\Sso->first, 0
GET #map: /map [sync] = Controller\MapController->second, 3600
Now I reroute(); to #map route, from first();
class Sso {
public function first($f3){
$msg = 'My message!';
if( !empty($msg) ){
$f3->reroute('#map');
}
}
}
Is there any "elegant" way to pass data (e.g. $msg) right into $MapController->second(); ?
I donĀ“t want to use $SESSION or the global $f->set('msg', $msg); for this.
This isn't an issue specific to fat-free-framework, but web in general. When you reroute, you tell the browser to redirect the user's browser page using a 303 header redirect code.
Take a minute to read the doc regarding re-routing: http://fatfreeframework.com/routing-engine#rerouting
There seems to be some contradicting information in your question, which leads me to question the purpose of what you are trying to achieve.
If you are rerouting, you can either use the session, cookies, or use part of the url to pass messages or references to a message.
If you do not need to redirect, but just want to call the function without changing the passed parameters, you could abstract the content of the function and call that function from both routes. You could also use the $f3 globals, which are a great way of passing data between functions in cases where you don't want to pass the data using the function call. is there a reason why you don't want to to use this? The data is global for the single session, so there is no security concern, and the data gets wiped at the end of the request, so there is very little extra footprint or effect on the server.
If you're alright with not using #map_name in re-routes you can do something like this:
$f3->reroute('path/?foo=bar');
Not the prettiest I'll admit. I wish $f3->reroute('#path_name?foo=bar') would work.
I have code similar to what's below (it's example code). Using Guzzle, I make multiple calls to the same site to see if a document exists. If the document exists, I save it. As I make each call, the memory usage goes up. Eventually, if the number of requests is high enough, I run out of memory. I used memory_get_peak_usage to track where the memory use was happening, and it's the Guzzle client.
The code works as expected, but I cannot find a way to tell the Guzzle Client to "reset and dump all previous requests". I'm pretty sure it's caching the results in memory, but as I've written them out to a file, I know I won't be needing said results. How do I dump them?
FWIW, my current solution is making a new client duplicating the parameters of the original one, and unsetting it periodically. It works, but it's ugly.
$client = new \Guzzle\Http\Client('some_url');
for ($i=0; $i<10000; $i++)
{
try {
$pdf = $client->get( $pdf_name )->send();
$this->filesystem->put(
$pdf_name,
$pdf->getBody( true )
);
} catch ( ClientErrorResponseException $ex ) {
}
}
Based on a cursorary glance at the source code for the bundle the Guzzle client is making use of Doctrine's filesystem cache. References:
Bundle/Resources/config/cache.xml
Bundle/DependencyInjection/MisdGuzzleExtension.php
Bundle/DependencyInjection/Configuration.php
The Bundle documentation also provides information on Caching. So, in theory to remove/disable the cache, all you have to do is remove the reference to <argument type="service" id="misd_guzzle.cache.filesystem"/> from the the addSubscriber section of your MyBundle/Resources/config/services.xml
I find the SqlCacheDependency very useful when writing C# ASP.NET applications, and would love to use something similar in my PHP applications. Can anyone suggest something?
SqlCacheDependency caches the page page output forever, until the specified table(s) are modified in the database.
Here's the basic jist of what happens in ASP.NET:
SqlCacheDependency SqlDep = null;
// Check the Cache for the SqlSource key.
// If it isn't there, create it with a dependency
// on a SQL Server table using the SqlCacheDependency class.
if (Cache["MyCache"] == null) {
SqlDep = new SqlCacheDependency("DatabaseName", "TableName");
// Perform action to be cached ...
var result = DatabaseIntensiveFunction();
Cache.Insert("MyCache", result, SqlDep);
}
else {
// The data was retrieved from the Cache.
}
// Output page from cache.
Output(Cache["MyCache"]);
So does anyone know of any MySql table dependency technique? - much cleaner than time-based caching.
Why not use something like Memcache or APC for this?
Edit: I also just found the MySQLnd Query Cache plugin for 5.3.3.