I'm having a big head ache with Laravel's chunk and each functions for breaking up result sets.
I have a table, that has a column processed with a value of 0. If I run the following code, it goes through all 13002 records.
Record::where(['processed' => 0])->each(function ($record) {
Listing::saveRecord($record);
}, 500);
This code will run through all 13002 records. However, if I add in some code to mark a record as processed, things go horribly pear shaped.
Record::where(['processed' => 0])->each(function ($record) {
$listing_id = Listing::saveRecord($record);
$record->listing_id = $listing_id;
$record->processed = 1;
$record->save();
}, 500);
When this code runs, only 6002 records are processed.
From my understand of things, that on each iteration of of the chunk (each runs through chunk), that it's executing a new statement.
I've come from using Yii2 and I'm mostly happy with the move, except for this hiccup, which has me pulling my hair out. Yii2 has similar functions (each and batch), but they seem to use result sets and pointers, so even if you update the table while you're processing your results, it doesn't effect your result set.
Is there actually a better way to do this in Laravel?
Try this
Records::where('processed',0)->chunk(100, function($records){
foreach($records as $record)
{
// do your stuff...
}
});
https://laravel.com/docs/5.3/queries#chunking-results
Sorry about indentation, on my phone and that doesnt work apperently..
Related
I've been spent hours trying to figure out how I'm supposed to get around this error in my scenario. I'm trying to run a few queries in sequence.
I have read: codeigniter : Commands out of sync; you can't run this command now, however I am not able to update /system/database/drivers/mysqli/mysqli_result.php
I've tried:
$this->db->reset_query();
$this->db->close();
$this->db->initialize();
$this->db->reconnect();
mysqli_next_result( $this->db->conn_id );
$query->free_result();
But they either give me the same error, or different errors which I will detail in the comments of my code.
The way my code is organized- I have a make_query method that takes a bunch of search options and figures out which tables to join and fields to search based on those. Sometimes, I just want to count results, sometimes I want all of the resulting data, sometimes I just want distinct values for certain fields or to group by certain fields. So I call make_query with my options, and then decide what to select afterwards. I have also been saving my query text to report, so that users can see what query is being run.
One interesting thing I have noted is that when I have no options set (ie there are no WHERE clauses in my SQL), I do not get this error and the multiple queries are able to run with no problem!
Here is my code:
//Get rows from tblPlots based on criteria set in options array
public function get_plot_data($options=array()){
$this->db->save_queries = TRUE;
log_message('debug','before make query 1 get plot data');
$this->make_query($options,'plot');
log_message('debug','after make query 1 get plot data');
$this->db->distinct();
//Now, set select to return only requested fields
if (!empty($options['fields']) and $options['fields']!="all" ){
//Because other tables could be joined, add table name to each select
array_walk($options['fields'], function(&$value, $key) { $value = 'tblPlots.'.$value;} );
$this->db->select($options['fields']);
}
if (!empty($options['limit']) and $options['limit']>0){
$this->db->limit($options['limit'], $options['offset']);
}
//Get the resulting data
$result=$this->db->get('tblProgram')->result();
$query_text = $this->db->last_query(); //tried removing this but didn't help
log_message('debug','query text '.$query_text);
$this->db->save_queries = FALSE;
//get the number of rows
//$this->db->reset_query();
//$this->db->close();
//$this->db->initialize();
//$this->db->reconnect();
//mysqli_next_result( $this->db->conn_id );
//$this->db->free_result(); //Call to undefined method CI_DB_mysqli_driver::free_result()
//$result->free_result(); //Call to a member function free_result() on array
log_message('debug','before make query 2');
$this->make_query($options,"plot");
$this->db->select('pkProgramID');
log_message('debug','before count results');
//this is where my code errors out trying to do the next step:
$count=$this->db->count_all_results('tblProgram');
}
I'm not including the code for make_query because it is long and calls other functions, but it runs successfully the first time and outputs the SQL query I would expect. If it would be helpful, I can include that code as well.
I'm not sure how to correctly call free_result() given that the CI documentation only gives an example using query('SQL QUERY') and I am building the query using Active Record, so I'm not sure how to use free result in my scenario? Maybe that's the issue?
$query2 = $this->db->query('SELECT name FROM some_table');
$query2->free_result();
Thank you for any help!
I ended up posting on a CI forum that suggested https://www.youtube.com/watch?v=yPiBhg6r5B0 which worked! Also I ended up having to join my tables differently to avoid timeouts (I think I was having trouble because I was using AJAX to call many queries asynchronously and one of them was timing out). Hope this helps someone!
Summary
This is a script (CakePHP 2.10.18 - LAMP dedicated server with PHP 5.3) that loads information from 2 MySQL tables, and then does some process of the data to output it to excel.
Table 1 has users, and Table 2 has info about those users (one record per user). The script has the goal of grabbing the record of a user from Table 1, grabbing its related info from Table 2, and put it in an excel row (using PHPExcel_IOFactory library for this).
The information extracted of those tables is of around 8000 records from each, the tables themselves have 100K and 300K total records respectively. All the fields in those tables are ints and small varchars with the exception of one field in the second table (datos_progreso seen in the code below), which is a text field and contains serialized data, but nothing big.
The issue is that if I run the script for the full 16000 records I get an Internal Server Error (without really any explanation in the logs), if I run the script for 1000 records it all works fine, so this seems to point out it's a resources issue.
I've tried (among other things that I will explain at the end) increasing the memory_limit from 128M to 8GB (yes you read that right), max_execution_time from 90 to 300 seconds, and max_input_vars from 1000 to 10000, and that isn't solving the problem.
My thoughts are that the amount of data isn't that huge to cause the resources to run out, but I've tried optimizing the script in several ways and can't get it to work. The only time I get it to work is by running it on a small portion of the records like I mention above.
I would like to know if there's something script-wise or php-configuration-wise I can do to fix this. I can't change the database tables with the information by the way.
Code
This is just the relevant bits of code that I think matter, the script is longer:
$this->Usuario->bindModel(
array('hasMany' => array(
'UsuarioProgreso' => array('className' => 'UsuarioProgreso', 'foreignKey' => 'id_usuario', 'conditions' => array('UsuarioProgreso.id_campania' => $id_campania)))
));
$usuarios = $this->Usuario->find('all', array(
'conditions'=>array('Usuario.id_campania'=>$id_campania, 'Usuario.fecha_registro >'=>'2020-05-28'),
'fields'=>array('Usuario.id_usuario', 'Usuario.login', 'Usuario.nombre', 'Usuario.apellido', 'Usuario.provincia', 'Usuario.telefono', 'Usuario.codigo_promocion'),
'order'=>array('Usuario.login ASC')
));
$usuario = null;
$progreso_usuario = null;
$datos_progreso = null;
$i = 2;
foreach ($usuarios as $usuario) {
if (isset($usuario['UsuarioProgreso']['datos_progreso'])) {
$datos_progreso = unserialize($progreso['UsuarioProgreso']['datos_progreso']);
$unit = 1;
$column = 'G';
while ($unit <= 60) {
if (isset($datos_progreso[$unit]['punt']))
$puntuacion = $datos_progreso[$unit]['punt'];
else
$puntuacion = ' ';
$objSheet->getCell($column.$i)->setValue($puntuacion);
$column++;
$unit++;
}
$nivel = 1;
$unidad_nivel = array(1 => 64, 2 => 68, 3 => 72, 4 => 76, 5 => 80, 6 => 84);
while ($nivel <= 6) {
$unidad = $unidad_nivel[$nivel];
if (isset($datos_progreso[$unidad]['punt']))
$puntuacion = $datos_progreso[$unidad]['punt'];
else
$puntuacion = ' ';
$objSheet->getCell($column.$i)->setValue($puntuacion);
$column++;
$nivel++;
}
}
//Free the variables
$usuario = null;
$progreso_usuario = null;
$datos_progreso = null;
$i++;
}
What I have tried
I have tried not using bindModel, and instead just load the information of both tables separately. So loading all the info of users first, looping through it, and on each loop grab the info for that specific user from Table 2.
I have tried also something similar to the above, but instead of loading all the info at once for the users from Table 1, just load first all their IDs, and then loop through those IDs to grab the info from Table 1 and Table 2. I figured this way I would use less memory.
I have also tried not using CakePHP's find(), and instead use fetchAll() with "manual" queries, since after some research it seemed like it would be more efficient memory-wise (didn't seem to make a difference)
If there's any other info I can provide that can help understand better what's going on please let me know :)
EDIT:
Following the suggestions in the comments I've implemented this in a shell script and it works fine (takes a while but it completes without issue).
With that said, I would still like to make this work from a web interface. In order to figure out what's going on, and since the error_logs aren't really showing anything relevant, I've decided to do some performance testing myself.
After that testing, these are my findings:
It's not a memory issue since the script is using at most around 300 MB and I've given it a memory_limit of 8GB
The memory usage is very similar whether it's via web call or shell script
It's not a timeout issue since I've given the script 20 minutes limit and it crashes way before that
What other setting could be limiting this/running out that doesn't fail when it's a shell script?
The way I solved this was using a shell script by following the advice from the comments. I've understood that my originally intended approach was not the correct one, and while I have not been able to figure out what exactly was causing the error, it's clear that using a web script was the root of the problem.
I have a MySQL stored procedure that updates data across a set of tables (basically for one record in the principal table and related records in a set of child tables). It's called via AJAX through a PHP function. (That is, the AJAX call is to a PHP page, which ultimately calls this SP.) It was working fine, but now I'm trying to make it do one more thing and running into the "Commands out of sync; you can't run this command now" error.
The change is to store one more item in the principal table, but to do so may require adding an item to a child table (called ActionTopic). The page lets the user either choose from a dropdown or type in a new value. I've added two parameters to the SP: one is the PK chosen in the dropdown, the other is the new value typed in. In the SP, I've added the code below. It checks whether there was a new value typed in. If so, it calls another SP that checks whether the value typed in is already in the table and, if not, adds it. (I've tried with the code to check and add the record inline rather than in a separate SP and I have the same problem.)
if cNewTopic <> '' then
-- First, make sure the new topic isn't already there
call aoctest.AddActionTopic(cNewTopic);
-- SELECT #iTopicID := iID FROM ActionTopic WHERE UPPER(Description) = UPPER(cNewTopic);
SET #iTopicID = LAST_INSERT_ID();
else
SET #iTopicID = Topic;
end if;
The page works if the user makes a choice from the dropdown. The problem only occurs when the user types in a new value. Even when I get the error, everything else works as expected. The new value is added to the child table, and the parent table and other children are updated as expected.
Interestingly, if I call the SP in MySQL Workbench with the same parameters (after ensuring that the new value isn't in the new table), it runs without error. The only odd thing I've noticed is that I get two rows in the Output section of MySQL Workbench rather than one. Both show the call to the SP. The first shows "1 row(s) returned" and a period of time, while the second shows "0 row(s) returned" and "-/0.000 sec". A call to the SP in MySQL Workbench where the new value is already in the table also shows two rows in the Output section, but the second one shows "1 row(s) returned".
Not sure whether any of the other code is needed here. If you think I need to show more, please ask.
UPDATE: Based on the comment from Pete Dishman, I took a harder look at where the error is occurring. It's not the original SP call giving an error. It's the next call to MySQL, which is still inside the Ajax call.
The code processing the result already had this code:
//not sure why next line should be needed.
mysqli_next_result($conn);
I tried both simply doubling the call to mysqli_next_result (that is, two in a row) and putting it into a loop along the lines Pete suggested. With two calls, I still get the same error. With a loop, I wait 30 seconds and then get error 500: Internal server error.
UPDATE 2: I tried with a loop for mysqli_more_results() (similar to the one in Pete Dishman's reply) and echoing a counter inside the loop. The code brought my internet connection to a crawl and I finally had to break out of it, but there were dozens of iterations of the loop. Just tried the following:
mysqli_free_result($result);
$result = mysqli_store_result($conn);
mysqli_free_result($result);
if (mysqli_more_results($conn)) {
$result = mysqli_store_result($conn);
mysqli_free_result($result);
}
$allresult = getsubmissions($conn);
Found a noticeable delay before it failed.
Even if you can't tell me what's wrong, I'd appreciate ideas for how to debug this.
This may be because the stored procedure is returning multiple result sets (the two rows in workbench).
When querying from php you need to retrieve both result sets before you can send another query, otherwise you get the "commands out of sync" error.
The documentation seems to imply this is only the case when using mysqli_multi_query but we have it in our code when using mysqli_real_query.
Our query code is along the lines of:
mysqli_real_query($conn, $sql);
$resultSet = mysqli_store_result($conn);
while (!is_null($row = mysqli_fetch_array($resultSet, MYSQLI_BOTH)))
{
$results[] = $row;
}
mysqli_free_result($resultSet);
// Check for any more results
while (mysqli_more_results($conn))
{
if (mysqli_next_result($conn))
{
$result = mysqli_store_result($conn);
if ($result !== FALSE)
{
mysqli_free_result($result);
}
}
}
return $results;
The code would be different obviously if you're using PDO, but the same principle may apply (See http://php.net/manual/en/pdostatement.nextrowset.php)
I've solved my own problem by reworking the code that processes the result as follows:
if (mysqli_more_results($conn)) {
mysqli_free_result($result);
mysqli_next_result($conn);
$result = mysqli_store_result($conn);
mysqli_free_result($result);
if (mysqli_more_results($conn)) {
mysqli_next_result($conn);
$result = mysqli_store_result($conn);
if (!is_null($result) and gettype($result)!== 'boolean') {
mysqli_free_result($result);
}
}
}
$allresult = getsubmissions($conn);
I have a query in a CakePHP 3.0 table with a formatResults() method applied to it.
In order to carry out this calculation, data is required from a table in a different database (Currencies).
I am currently using this code to gather the data and pass it to the function:
$currencies = TableRegistry::get('Currencies');
$currencyValues = $currencies
->findByCurrno($options['currency'])
->cache('currency'.$options['currency']);
$currencyValues = $currencyValues->first()->toArray()
$query->formatResults(function ($results) use ($currencyValues) {
return $results->map(function($row) use ($currencyValues) {
$row['converted_earnings'] = $row['earned'] / $currencyValues['cur'.$row['currency']];
$row['converted_advances'] = $row['advances'] / $currencyValues['cur'.$row['currency']];
return $row;
});
});
The problem is, this code seems to take a very long time to execute, even though it is only iterating through a few hundred rows of data.
Further investigation revealed that if I do not collect the data from the 'currencies' table and instead declare $currencyValues as an array with fixed numbers the code takes a full second less to execute.
Through commenting out parts of the code, I have isolated this to be the cause of the problem:
$currencyValues = $currencies
->findByCurrno($options['currency'])
->cache('currency'.$options['currency']);
If I remove this part of the code then everything runs quickly, and as soon as I add it in (even if I do not use the data it returns and use the static array) the page takes far longer to load. It should be noted that this problem occurs whether or not I use the ->cache() method, and that the query itself reports that it takes 0-1ms in the sql dump.
My question is - why is this slowing down my execution so much and how can I stop it? It is pretty much a requirement that I use the data from this table for the operation, so I am looking for a way to speed it up.
Good morning.
I'm currently trying to build a very basic caching system for one of my scripts. The cache is JSON data and contains only 1 key and it's value, but many individual fields, something like this;
{"Item1":"Item1 Description"}
{"Item2":"Item2 Description"}
{"Item3":"Item3 Description"}
What I'm intending to do is;
First check if a cache file is available
Then check if an item exists in the cache
Then add the new item along with it's description if it's not already in the cache...
...or return the item description if it's not there.
All data being stored is strings. The cache file doesn't store any other type of data.
I've put together a basic function but I'm having trouble getting it functioning;
function ItemIsInCache($CacheFile, $ItemId) {
if(file_exists($CacheFile)) {
$json = json_decode(file_get_contents($CacheFile, true));
if(in_array($ItemId, $json)) { // <<
$itemname = array_search($ItemId, $json);
return itemname;
} else {
$item[$itemId] = GrabItemName($ItemId);
$itemname = array_search($ItemId, $json); // <<
return $itemname;
}
} else {
$item[$ItemId] = GrabItemName($ItemId);
$ejson = json_encode($item);
file_put_contents($CacheFile, $ejson);
return $item[$ItemId];
}
}
Notes
GrabItemName is a different function that returns the description data based on the $ItemId.
The warnings I'm getting are Wrong datatype for second argument in both array_search() and in_array(), on lines 4 and lines 9 respectively (those are the line numbers in the above code - due to the nature of my script these numbers are later on) -- for simplicity, I've marked the problem lines with // <<.
The function is running in a loop which I've no problems with. The problems lie within this function.
What currently happens
Right now, if the cache doesn't exist, it creates it and adds the first item from the loop to the cache file in it's respective JSON format (that fires since the cache file doesn't exist, so after the final else statement).
However, items from the loop after that don't get added, presumably because the file exists and there's something wrong with the code.
The last part of the function works exactly as I want it to but the first part does not.
Expected behaviour with fixed code
Check cache > Return description if item exists ELSE add new item to cache.
The items and their associated descriptions will NOT change, but I'm pulling them from a rate limited API, and I need to ensure I cache whatever I can for everyones benefit.
So, any ideas what I'm doing wrong with the function? I'm sure it's something incredibly simple that I'm overlooking.
Your file is not JSON for an erray. The correct JSON for an array is
[
{"Item1":"Item1 Description"},
{"Item2":"Item2 Description"},
{"Item3":"Item3 Description"}
]
You're missing the brackets around the array, so you just get a single object.
When creating the initial file, you need to do:
$ejson = json_encode(array($item));
so that it's initialized as an array of one item, not just an item.