I have a query in a CakePHP 3.0 table with a formatResults() method applied to it.
In order to carry out this calculation, data is required from a table in a different database (Currencies).
I am currently using this code to gather the data and pass it to the function:
$currencies = TableRegistry::get('Currencies');
$currencyValues = $currencies
->findByCurrno($options['currency'])
->cache('currency'.$options['currency']);
$currencyValues = $currencyValues->first()->toArray()
$query->formatResults(function ($results) use ($currencyValues) {
return $results->map(function($row) use ($currencyValues) {
$row['converted_earnings'] = $row['earned'] / $currencyValues['cur'.$row['currency']];
$row['converted_advances'] = $row['advances'] / $currencyValues['cur'.$row['currency']];
return $row;
});
});
The problem is, this code seems to take a very long time to execute, even though it is only iterating through a few hundred rows of data.
Further investigation revealed that if I do not collect the data from the 'currencies' table and instead declare $currencyValues as an array with fixed numbers the code takes a full second less to execute.
Through commenting out parts of the code, I have isolated this to be the cause of the problem:
$currencyValues = $currencies
->findByCurrno($options['currency'])
->cache('currency'.$options['currency']);
If I remove this part of the code then everything runs quickly, and as soon as I add it in (even if I do not use the data it returns and use the static array) the page takes far longer to load. It should be noted that this problem occurs whether or not I use the ->cache() method, and that the query itself reports that it takes 0-1ms in the sql dump.
My question is - why is this slowing down my execution so much and how can I stop it? It is pretty much a requirement that I use the data from this table for the operation, so I am looking for a way to speed it up.
Related
I've been spent hours trying to figure out how I'm supposed to get around this error in my scenario. I'm trying to run a few queries in sequence.
I have read: codeigniter : Commands out of sync; you can't run this command now, however I am not able to update /system/database/drivers/mysqli/mysqli_result.php
I've tried:
$this->db->reset_query();
$this->db->close();
$this->db->initialize();
$this->db->reconnect();
mysqli_next_result( $this->db->conn_id );
$query->free_result();
But they either give me the same error, or different errors which I will detail in the comments of my code.
The way my code is organized- I have a make_query method that takes a bunch of search options and figures out which tables to join and fields to search based on those. Sometimes, I just want to count results, sometimes I want all of the resulting data, sometimes I just want distinct values for certain fields or to group by certain fields. So I call make_query with my options, and then decide what to select afterwards. I have also been saving my query text to report, so that users can see what query is being run.
One interesting thing I have noted is that when I have no options set (ie there are no WHERE clauses in my SQL), I do not get this error and the multiple queries are able to run with no problem!
Here is my code:
//Get rows from tblPlots based on criteria set in options array
public function get_plot_data($options=array()){
$this->db->save_queries = TRUE;
log_message('debug','before make query 1 get plot data');
$this->make_query($options,'plot');
log_message('debug','after make query 1 get plot data');
$this->db->distinct();
//Now, set select to return only requested fields
if (!empty($options['fields']) and $options['fields']!="all" ){
//Because other tables could be joined, add table name to each select
array_walk($options['fields'], function(&$value, $key) { $value = 'tblPlots.'.$value;} );
$this->db->select($options['fields']);
}
if (!empty($options['limit']) and $options['limit']>0){
$this->db->limit($options['limit'], $options['offset']);
}
//Get the resulting data
$result=$this->db->get('tblProgram')->result();
$query_text = $this->db->last_query(); //tried removing this but didn't help
log_message('debug','query text '.$query_text);
$this->db->save_queries = FALSE;
//get the number of rows
//$this->db->reset_query();
//$this->db->close();
//$this->db->initialize();
//$this->db->reconnect();
//mysqli_next_result( $this->db->conn_id );
//$this->db->free_result(); //Call to undefined method CI_DB_mysqli_driver::free_result()
//$result->free_result(); //Call to a member function free_result() on array
log_message('debug','before make query 2');
$this->make_query($options,"plot");
$this->db->select('pkProgramID');
log_message('debug','before count results');
//this is where my code errors out trying to do the next step:
$count=$this->db->count_all_results('tblProgram');
}
I'm not including the code for make_query because it is long and calls other functions, but it runs successfully the first time and outputs the SQL query I would expect. If it would be helpful, I can include that code as well.
I'm not sure how to correctly call free_result() given that the CI documentation only gives an example using query('SQL QUERY') and I am building the query using Active Record, so I'm not sure how to use free result in my scenario? Maybe that's the issue?
$query2 = $this->db->query('SELECT name FROM some_table');
$query2->free_result();
Thank you for any help!
I ended up posting on a CI forum that suggested https://www.youtube.com/watch?v=yPiBhg6r5B0 which worked! Also I ended up having to join my tables differently to avoid timeouts (I think I was having trouble because I was using AJAX to call many queries asynchronously and one of them was timing out). Hope this helps someone!
I have a controller that is being called lots of times (thousands per minute), and I need to log every call without losing the response speed.
I have a piece of code as follows:
$redis = Redis::connection();
$redis->pipeline(function($pipe) use ($type, $redis)
{
// usual
$pipe->incr($type);
// check unique list
$len = $pipe->lLen($type.'_unique_list');
$list = $pipe->lRange($type.'_unique_list', 0, $len);
if(!in_array($this->uid, $list)) {
$pipe->rPush($type . '_unique_list', $this->uid);
$pipe->incr($type . '_unique');
}
});
In the other place I get data from Redis and display them.
The problem is, that while I use $pipe->lLen and $pipe->lRange the numbers won't change (the interesting moment is that neither $type nor $type . '_unique' change).
I've tried replacing $len with PHP_INT_MAX, but the problem remains the same. I've also tried adding $pipe->exec(); in the end, but it didn't help as well.
If I replace $pipe->lRange with $redis->lRange, everything starts working but awfully slowly, because each redis call waits for response.
How could I solve this situation?
UPD: I found out that $list taken with $pipe returns Redis object, not array. So the question is, how could I check if the key exists in Redis list without retrieving the list itself.
i suggest use a hash instead a list if need check the key existence.
lists are for add many items and sort by input order, not by key
Hashes are by key, and has the method hexist
I've got SugarCrm plugin which is exporting data to external service. I'm using logic hooks for updated/deleted/new Contacts, but I've got problem with synchronizing already existing data. I have to extract all the data from the SugarCRM and there are two SugarBean methods I've tried to use: get_full_list() and get_list(). First one gives me the full Contact list, but I need to send it in batches 1000 Contacts in one Json max, the second method returns only first page of the Contacts (depends on config settings 10 - 1000max entries).
I'm using this method ATM:
// prepare contacts data from SugarBean
$bean = BeanFactory::getBean($module);
$contactResults = $bean->get_full_list();
Then foreach on $contactResults and save the data I want to the required format and send it as a Json via postrequest. I've tried to find the solution to split it into batches, but Im stuck :( Neither get_full_list or get_list seems to work for me.
Any suggestions? Maybe someone solved this issue already?
Thanks in advance!
It sounds to me like your problem is creating batches? If not please be more specific about what isn't working.
For splitting an array into batches, you may want to have a look at https://php.net/manual/en/function.array-chunk.php
Also get_list supports retrieving later pages. It is defined like this: function get_list($order_by = "", $where = "", $row_offset = 0, $limit=-1, $max=-1, $show_deleted = 0, $singleSelect=false, $select_fields = array()).
That means for the second page you could specify $row_offset = 1000, for the third page make it 2000, etc. So basically run a loop that calls get_list with $limit = 1000 and increases an initial $row_offset of 0 by 1000 after each iteration, until less than 1000 records or null is returned by the function.
Here are some general hints if you run into problems with processing those beans:
If the problem you're having is incomplete data, try loading each bean manually by using its ID. Some Sugar functions don't load all (special) fields by default.
If things seem to just fail for no reason, make sure to check your PHP log for errors. Maybe loading as many beans at once could possibly cause problems with your PHP's max_execution_time or memory_limit.
I'm having a big head ache with Laravel's chunk and each functions for breaking up result sets.
I have a table, that has a column processed with a value of 0. If I run the following code, it goes through all 13002 records.
Record::where(['processed' => 0])->each(function ($record) {
Listing::saveRecord($record);
}, 500);
This code will run through all 13002 records. However, if I add in some code to mark a record as processed, things go horribly pear shaped.
Record::where(['processed' => 0])->each(function ($record) {
$listing_id = Listing::saveRecord($record);
$record->listing_id = $listing_id;
$record->processed = 1;
$record->save();
}, 500);
When this code runs, only 6002 records are processed.
From my understand of things, that on each iteration of of the chunk (each runs through chunk), that it's executing a new statement.
I've come from using Yii2 and I'm mostly happy with the move, except for this hiccup, which has me pulling my hair out. Yii2 has similar functions (each and batch), but they seem to use result sets and pointers, so even if you update the table while you're processing your results, it doesn't effect your result set.
Is there actually a better way to do this in Laravel?
Try this
Records::where('processed',0)->chunk(100, function($records){
foreach($records as $record)
{
// do your stuff...
}
});
https://laravel.com/docs/5.3/queries#chunking-results
Sorry about indentation, on my phone and that doesnt work apperently..
I have this while loop, that basically loops through a lot of records in a database, and inserts the data in another:
$q = $con1->query($users1) or die(print_r($con2->errorInfo(),1));
while($row = $q->fetch(PDO::FETCH_ASSOC)){
$q = $con2->prepare($users2);
$q->execute(array($row['id'], $row['username'])) or die(print_r($con2-errorInfo(),1));
}
(The script has been shortened for easy reading - the correct one has a much longer array)
I would like to do this more graphical, and show a progress bar on how far it has went, instead of just seeing a page loading for a few minutes (there are ~20.000 rows in this one - I have tables with much more data)
I get that you could get the total number from the old database, and I could also easily put the current number into a variable like this:
$q = $con1->query($users1) or die(print_r($con2->errorInfo(),1));
$i = 0;
while($row = $q->fetch(PDO::FETCH_ASSOC)){
$q = $con2->prepare($users2);
$q->execute(array($row['id'], $row['username'])) or die(print_r($con2-errorInfo(),1));
$i++;
}
But now I need to actually fetch $i and display it - or something like it.
How is this "easily" done?
The code for the progress bar can either be in the same document as the while loop, or in another if easier.
You can do a "master" file that does an ajax to this first file to run a single query. You could get all the entry id's in this master file, and then pass it as a parameter to the second file that does a single query. Store these ids in a javascript array.
Create a function that does this, and when the first ajax is done, move to the second element of the id array, and do another ajax with a second parameter. That's how magento imports are done by the way :)
If you need further explanations, let me know, I tried my best to explain, but may have not been perfectly clear.
// you generate this javascript array using php.
// let's say you have all the ids that have to be processed in $Ids php array.
Ids = [<?php echo implode(',', $Ids); ?>];
function doAjax(i) {
$.ajax({ // using jquery for simplicity
'url': "ajax.php?id=" + Ids[i],
}).done(function(){
if ( i >= 0 ) {
// at the point you know you're at ((Ids.length-i)/(Ids.length) * 100) percent of the script
// so you can do something like this:
// $('.progressbar').css('width', ((Ids.length-i)/(Ids.length) * 100) + '%');
doAjax(i-1);
}
});
}
doAjax(Ids.length); // starting from the last entry
So, just to explain what this does. It starts by declaring a global javascript array that has all the ids that will need to be changed.
Then I declare a recursive ajax function, this way we can make sure that only one ajax runs at any single time (so the server doesn't blow up), and we can have a fairly accurate progress. This ajax function does the following:
Sends a request to ajax.php?id=xxx - where xxx is one of the ids in the javascript array.
In the file, we get the id ($_GET['id']), you take it from the old database, and insert it in the new one. This is only for one entry.
when the ajax is done, it goes to the done() function. Since we start the doAjax() function with the last element, we do the next iteration doAjax(i-1). Since we're going backwards in the array, we check if the key is positive. If it's not, the script will stop.
That's about it.
You can't. The php is first interpreted by the server and then send to the user as HTML-Code.
The only possibility would be creating a html-page and call the php-script with AJAX.