I have a controller that is being called lots of times (thousands per minute), and I need to log every call without losing the response speed.
I have a piece of code as follows:
$redis = Redis::connection();
$redis->pipeline(function($pipe) use ($type, $redis)
{
// usual
$pipe->incr($type);
// check unique list
$len = $pipe->lLen($type.'_unique_list');
$list = $pipe->lRange($type.'_unique_list', 0, $len);
if(!in_array($this->uid, $list)) {
$pipe->rPush($type . '_unique_list', $this->uid);
$pipe->incr($type . '_unique');
}
});
In the other place I get data from Redis and display them.
The problem is, that while I use $pipe->lLen and $pipe->lRange the numbers won't change (the interesting moment is that neither $type nor $type . '_unique' change).
I've tried replacing $len with PHP_INT_MAX, but the problem remains the same. I've also tried adding $pipe->exec(); in the end, but it didn't help as well.
If I replace $pipe->lRange with $redis->lRange, everything starts working but awfully slowly, because each redis call waits for response.
How could I solve this situation?
UPD: I found out that $list taken with $pipe returns Redis object, not array. So the question is, how could I check if the key exists in Redis list without retrieving the list itself.
i suggest use a hash instead a list if need check the key existence.
lists are for add many items and sort by input order, not by key
Hashes are by key, and has the method hexist
Related
I'm using Redis for PHP.
I need to check if the key exists in redis list, and if it does not, add it. For now my code looks as follows:
$redis = Redis::connection();
$redis->pipeline(function($pipe) use ($type, $redis)
{
$list = $pipe->lRange($type.'_unique_list', 0, -1);
if(!in_array($this->uid, $list)) {
$pipe->rPush($type . '_unique_list', $this->uid);
}
});
The problem is that $list taken from $pipe returns Redis object, not array, and in_array does not work. But if I use $redis->lRange, the script becomes too slow.
So my question is, if there is any possibility to check if the key exists in Redis list without retrieving the list into the script? Some special Redis command which I can't find in docs? Or maybe I could replace in_array with something else in this particular situation?
Wrongish answer: you can call LINDEX instead of doing the search in the client.
Righter answer: scanning a linked list is always an expensive operation (O(N)) whether it is server- or client-side. Consider using a different data structure, e.g. a Set, for that purpose if your N is large.
I have a query in a CakePHP 3.0 table with a formatResults() method applied to it.
In order to carry out this calculation, data is required from a table in a different database (Currencies).
I am currently using this code to gather the data and pass it to the function:
$currencies = TableRegistry::get('Currencies');
$currencyValues = $currencies
->findByCurrno($options['currency'])
->cache('currency'.$options['currency']);
$currencyValues = $currencyValues->first()->toArray()
$query->formatResults(function ($results) use ($currencyValues) {
return $results->map(function($row) use ($currencyValues) {
$row['converted_earnings'] = $row['earned'] / $currencyValues['cur'.$row['currency']];
$row['converted_advances'] = $row['advances'] / $currencyValues['cur'.$row['currency']];
return $row;
});
});
The problem is, this code seems to take a very long time to execute, even though it is only iterating through a few hundred rows of data.
Further investigation revealed that if I do not collect the data from the 'currencies' table and instead declare $currencyValues as an array with fixed numbers the code takes a full second less to execute.
Through commenting out parts of the code, I have isolated this to be the cause of the problem:
$currencyValues = $currencies
->findByCurrno($options['currency'])
->cache('currency'.$options['currency']);
If I remove this part of the code then everything runs quickly, and as soon as I add it in (even if I do not use the data it returns and use the static array) the page takes far longer to load. It should be noted that this problem occurs whether or not I use the ->cache() method, and that the query itself reports that it takes 0-1ms in the sql dump.
My question is - why is this slowing down my execution so much and how can I stop it? It is pretty much a requirement that I use the data from this table for the operation, so I am looking for a way to speed it up.
Good morning.
I'm currently trying to build a very basic caching system for one of my scripts. The cache is JSON data and contains only 1 key and it's value, but many individual fields, something like this;
{"Item1":"Item1 Description"}
{"Item2":"Item2 Description"}
{"Item3":"Item3 Description"}
What I'm intending to do is;
First check if a cache file is available
Then check if an item exists in the cache
Then add the new item along with it's description if it's not already in the cache...
...or return the item description if it's not there.
All data being stored is strings. The cache file doesn't store any other type of data.
I've put together a basic function but I'm having trouble getting it functioning;
function ItemIsInCache($CacheFile, $ItemId) {
if(file_exists($CacheFile)) {
$json = json_decode(file_get_contents($CacheFile, true));
if(in_array($ItemId, $json)) { // <<
$itemname = array_search($ItemId, $json);
return itemname;
} else {
$item[$itemId] = GrabItemName($ItemId);
$itemname = array_search($ItemId, $json); // <<
return $itemname;
}
} else {
$item[$ItemId] = GrabItemName($ItemId);
$ejson = json_encode($item);
file_put_contents($CacheFile, $ejson);
return $item[$ItemId];
}
}
Notes
GrabItemName is a different function that returns the description data based on the $ItemId.
The warnings I'm getting are Wrong datatype for second argument in both array_search() and in_array(), on lines 4 and lines 9 respectively (those are the line numbers in the above code - due to the nature of my script these numbers are later on) -- for simplicity, I've marked the problem lines with // <<.
The function is running in a loop which I've no problems with. The problems lie within this function.
What currently happens
Right now, if the cache doesn't exist, it creates it and adds the first item from the loop to the cache file in it's respective JSON format (that fires since the cache file doesn't exist, so after the final else statement).
However, items from the loop after that don't get added, presumably because the file exists and there's something wrong with the code.
The last part of the function works exactly as I want it to but the first part does not.
Expected behaviour with fixed code
Check cache > Return description if item exists ELSE add new item to cache.
The items and their associated descriptions will NOT change, but I'm pulling them from a rate limited API, and I need to ensure I cache whatever I can for everyones benefit.
So, any ideas what I'm doing wrong with the function? I'm sure it's something incredibly simple that I'm overlooking.
Your file is not JSON for an erray. The correct JSON for an array is
[
{"Item1":"Item1 Description"},
{"Item2":"Item2 Description"},
{"Item3":"Item3 Description"}
]
You're missing the brackets around the array, so you just get a single object.
When creating the initial file, you need to do:
$ejson = json_encode(array($item));
so that it's initialized as an array of one item, not just an item.
I have this while loop, that basically loops through a lot of records in a database, and inserts the data in another:
$q = $con1->query($users1) or die(print_r($con2->errorInfo(),1));
while($row = $q->fetch(PDO::FETCH_ASSOC)){
$q = $con2->prepare($users2);
$q->execute(array($row['id'], $row['username'])) or die(print_r($con2-errorInfo(),1));
}
(The script has been shortened for easy reading - the correct one has a much longer array)
I would like to do this more graphical, and show a progress bar on how far it has went, instead of just seeing a page loading for a few minutes (there are ~20.000 rows in this one - I have tables with much more data)
I get that you could get the total number from the old database, and I could also easily put the current number into a variable like this:
$q = $con1->query($users1) or die(print_r($con2->errorInfo(),1));
$i = 0;
while($row = $q->fetch(PDO::FETCH_ASSOC)){
$q = $con2->prepare($users2);
$q->execute(array($row['id'], $row['username'])) or die(print_r($con2-errorInfo(),1));
$i++;
}
But now I need to actually fetch $i and display it - or something like it.
How is this "easily" done?
The code for the progress bar can either be in the same document as the while loop, or in another if easier.
You can do a "master" file that does an ajax to this first file to run a single query. You could get all the entry id's in this master file, and then pass it as a parameter to the second file that does a single query. Store these ids in a javascript array.
Create a function that does this, and when the first ajax is done, move to the second element of the id array, and do another ajax with a second parameter. That's how magento imports are done by the way :)
If you need further explanations, let me know, I tried my best to explain, but may have not been perfectly clear.
// you generate this javascript array using php.
// let's say you have all the ids that have to be processed in $Ids php array.
Ids = [<?php echo implode(',', $Ids); ?>];
function doAjax(i) {
$.ajax({ // using jquery for simplicity
'url': "ajax.php?id=" + Ids[i],
}).done(function(){
if ( i >= 0 ) {
// at the point you know you're at ((Ids.length-i)/(Ids.length) * 100) percent of the script
// so you can do something like this:
// $('.progressbar').css('width', ((Ids.length-i)/(Ids.length) * 100) + '%');
doAjax(i-1);
}
});
}
doAjax(Ids.length); // starting from the last entry
So, just to explain what this does. It starts by declaring a global javascript array that has all the ids that will need to be changed.
Then I declare a recursive ajax function, this way we can make sure that only one ajax runs at any single time (so the server doesn't blow up), and we can have a fairly accurate progress. This ajax function does the following:
Sends a request to ajax.php?id=xxx - where xxx is one of the ids in the javascript array.
In the file, we get the id ($_GET['id']), you take it from the old database, and insert it in the new one. This is only for one entry.
when the ajax is done, it goes to the done() function. Since we start the doAjax() function with the last element, we do the next iteration doAjax(i-1). Since we're going backwards in the array, we check if the key is positive. If it's not, the script will stop.
That's about it.
You can't. The php is first interpreted by the server and then send to the user as HTML-Code.
The only possibility would be creating a html-page and call the php-script with AJAX.
I'm using Wordpress and developed some site-specific plugins for it, additionally my theme is customized to fit the requirements of the plugins in the backend.
The last days I fiddled with transients in Wordpress. In some tutorials they're saying "If your're using custom queries and their results are cachable: Use a transient". Sounds good but I'm wondering when to use transients to get a real advantage.
I mean, even when using transients there have to be at least two queries in the background, haven't it? The first one for checking the validity, second one for the transient itself.
So is it really useful to use a transient i.e. for a custom WP_Query?
Thanks a lot for your help and thoughts.
Seems fairly straightforward. It's a literal class helper that allows you to store objects in a 'memcache' type fashion. You first set the transient
function do_something_here($callback_param = 'value'){
$key = 'do_something_' . $callback_param;//set the name of our transient equal to the value of the callback param being passed in the function.
$my_query = get_transient($myKey); //if we've stored this request before, then use it.
if($my_query !=== false){
//we found a previous existing version of this query. let's use it.
return $my_query;
}else{
//it doesn't exist, we need to build the transient.
//do our database querying here, global $wpdb; etc
//We are going to pretend our returned variable is 'george'
$value = george;
$length = 60*60*24; //how long do we want the transient to exist? 1 day here.
set_transient($key, $value, $length);
return $value;
}
}
Now that we have created our trigger and bound it to the name of '$key', we can access it anytime by using the exact value that key implies (which we declared earlier).
echo 'I wanted to do something, so : ' . do_something('value') . ' is what i did! ';
By utilizing this format you can hold queries in a 'cache' like world and use them to generate your responses. This is similar in a way to using 'trigger' events in MySql. Infact, this is a PORTION of a TECHNIQUE commonly referred to as long polling.