I use CodeIgniter, and when an insert_batch does not fully work (number of items inserted different from the number of items given), I have to do the inserts again, using insert ignore to maximize the number that goes through the process without having errors for existing ones.
When I use this method, the kind of data I'm inserting does not need strict compliance between the number of items given, and the number put in the database. Maximize is the way.
What would be the correct way of a) using insert_batch as much as possible b) when it fails, using a workaround, while minimizing the number of unnecessary requests?
Thanks
The Correct way of inserting data using insert_batch is :
CI_Controller :
public function add_monthly_record()
{
$date = $this->input->post('date');
$due_date = $this->input->post('due_date');
$billing_date = $this->input->post('billing_date');
$total_area = $this->input->post('total_area');
$comp_id = $this->input->post('comp_id');
$unit_id = $this->input->post('unit_id');
$percent = $this->input->post('percent');
$unit_consumed = $this->input->post('unit_consumed');
$per_unit = $this->input->post('per_unit');
$actual_amount = $this->input->post('actual_amount');
$subsidies_from_itb = $this->input->post('subsidies_from_itb');
$subsidies = $this->input->post('subsidies');
$data = array();
foreach ($unit_id as $id => $name) {
$data[] = array(
'date' => $date,
'comp_id' => $comp_id,
'due_date' => $due_date,
'billing_date' => $billing_date,
'total_area' => $total_area,
'unit_id' => $unit_id[$id],
'percent' =>$percent[$id],
'unit_consumed' => $unit_consumed[$id],
'per_unit' => $per_unit[$id],
'actual_amount' => $actual_amount[$id],
'subsidies_from_itb' => $subsidies_from_itb[$id],
'subsidies' => $subsidies[$id],
);
};
$result = $this->Companies_records->add_monthly_record($data);
//return from model
$total_affected_rows = $result[1];
$first_insert_id = $result[0];
//using last id
if ($total_affected_rows) {
$count = $total_affected_rows - 1;
for ($x = 0; $x <= $count; $x++) {
$id = $first_insert_id + $x;
$invoice = 'EBR' . date('m') . '/' . date('y') . '/' . str_pad($id, 6, '0', STR_PAD_LEFT);
$field = array(
'invoice_no' => $invoice,
);
$this->Companies_records->add_monthly_record_update($field,$id);
}
}
echo json_encode($result);
}
CI_Model :
public function add_monthly_record($data)
{
$this->db->insert_batch('monthly_record', $data);
$first_insert_id = $this->db->insert_id();
$total_affected_rows = $this->db->affected_rows();
return [$first_insert_id, $total_affected_rows];
}
AS #q81 mentioned, you would split the batches (as you see fit or depending on system resources) like this:
$insert_batch = array();
$maximum_items = 100;
$i = 1;
while ($condition == true) {
// code to add data into $insert_batch
// ...
// insert the batch every n items
if ($i == $maximum_items) {
$this->db->insert_batch('table', $insert_batch); // insert the batch
$insert_batch = array(); // empty batch array
$i = 0;
}
$i++;
}
// the last $insert_batch
if ($insert_batch) {
$this->db->insert_batch('table', $insert_batch);
}
Edit:
while insert batch already splits the batches, the reason why you have "number of items inserted different from the number of items given" might be because the allowed memory size is reached. this happened to me too many times.
Related
I am inserting data to the database using insert_batch() function.
I want to split the process.
I mean if I want to create 10,000 serial numbers. but 1,000 rows at a time, it should run the create process 10 times in a loop.
How can I do that?
$serial_numbers = $this->serial_numbers_model->generate_serial_numbers($product_id, $partner_id, $serial_number_quantity, $serial_number_start);
$issued_date = date("Y-m-d H:i:s");
$inserted_rows = 0;
foreach ($serial_numbers as $sn) {
$check_number = $this->serial_numbers_model->generate_check_number();
$first_serial_number = reset($serial_numbers);
$last_serial_number = end($serial_numbers);
$inserted_rows++;
$insert_data[] = array(
'partner_id' => $partner_id,
'product_id' => $product_id,
'check_number' => $check_number,
'serial_number' => $sn,
'issued_time' => $issued_date,
'serial_number_status' => CREATE_STATUS
);
}
$this->serial_numbers_model->insert_batch($insert_data);
}
Probably your serial_numbers_model->insert_batch() is just a wrapper around Codeigniter's native insert_batch()? The code below uses the native one for clarity, replace it with yours as required.
// Track how many in your batch, and prepare empty batch array
$count = 0;
$insert_data = [];
foreach ($serial_numbers as $sn) {
// ... your code, prepare your data, etc ...
$count++;
$insert_data[] = array(
// ... your data ...
);
// Do you have a batch of 1000 ready?
if ($count === 1000) {
// Yes - insert it
$this->db->insert_batch('table', $insert_data);
// $this->serial_numbers_model->insert_batch($insert_data);
// Reset the count, and empty the batch, ready to start again
$count = 0;
$insert_data = [];
}
}
// Watch out! If there were 1001 serial numbers, the first 1000 were handled,
// but the last one hasn't been inserted!
if (sizeof($insert_data)) {
$this->db->insert_batch('table', $insert_data);
}
I am just learning PHP and using the Codeigniter framework.
I'm using this code in my controller when inserting new data and it's working.
//INSERT MULTI ROWS TABEL harga_inventori
$kode_inventori = $_POST['kode_inventori'];
$result = array();
foreach($_POST['kode_kategori_pelanggan'] AS $key => $val){
$result[] = array(
"kode_kategori_pelanggan" => $_POST['kode_kategori_pelanggan'][$key],
"kode_inventori"=>$kode_inventori,
"harga" => $_POST['harga_jual'][$key],
"diskon" => $_POST['diskon'][$key]
);
}
$res = $this->db->insert_batch('harga_inventori', $result);
redirect("inventori");
But when I'm using the same pattern for the updating function, it's not working at all.
for($i = 0; $i < count($_POST['harga_jual'][$i]); $i++) {
if($_POST['kode_kategori_pelanggan'][$i] != '') {
$res[] = array(
'harga' => $_POST['harga_jual'][$i],
'diskon' => $_POST['diskon'][$i],
);
$this->db->where('kode_inventori',$_POST['kode_inventori']);
$this->db->where('kode_kategori_pelanggan',$_POST['kode_kategori_pelanggan'][$i]);
$this->db->update('harga_inventori',$res);
}
}
redirect("inventori");
I'm trying the update_batch() but I got many errors, so I'm using for loop and updating a single row at a time.
What could be the problem here?
Its only a typo. You should pass the array $res not $data, and change $res[] to $res since it is not needed. You should also check with isset() to prevent errors of undefined:
for($i = 0; $i < count($_POST['harga_jual'][$i]); $i++) {
if(isset($_POST['kode_kategori_pelanggan'][$i])) {
$res = array(
'harga' => $_POST['harga_jual'][$i],
'diskon' => $_POST['diskon'][$i],
);
$this->db->where('kode_inventori',$_POST['kode_inventori']);
$this->db->where('kode_kategori_pelanggan',$_POST['kode_kategori_pelanggan'][$i]);
$this->db->update('harga_inventori',$res);
}
}
redirect("inventori");
It would help to see some of your data to know what you are trying to do. It seems a little strange that you initiate the counter $i at the same time using it as an array key in the for loop like this: for($i = 0; $i < count($_POST['harga_jual'][$i]); $i++)
It would help to know how the data in your $_POST looks like. You could try to remove the [$i] in your for loop. I would also check each POST variable it they are set before using them, something like:
for($i = 0; $i < count($_POST['harga_jual']); $i++) {
if(isset($_POST['kode_kategori_pelanggan'][$i])) {
// CHECK IF POST VARIABLES IS SET AND IF NOT SET A DEFAULT VALUE (IN THIS EXAMPLE AN EMPTY STRING):
$harga_jual = isset($_POST['harga_jual'][$i]) ? $_POST['harga_jual'][$i] : '';
$diskon = isset($_POST['diskon'][$i]) ? $_POST['diskon'][$i] : '';
$kode_inventori = isset($_POST['kode_inventori']) ? $_POST['kode_inventori'] : '';
$kode_kategori_pelanggan = $_POST['kode_kategori_pelanggan'][$i]; // ALREADY CHECKED ISSET ABOVE...
$data = array(
'harga' => $harga_jual,
'diskon' => $diskon,
);
$this->db->where('kode_inventori',$kode_inventori);
$this->db->where('kode_kategori_pelanggan', $kode_kategori_pelanggan);
$this->db->update('harga_inventori', $data);
}
}
redirect("inventori");
I have MySQL with Float Values which are pressure parametters. I need to calculate average. But the code I make returns me average "empty" in my json response.
Here is the json response I get:
{"result":{"pressure":{"min":[{"Id":"2","presion":"0","Insertado":"2016-08-16 16:20:08"},{"Id":"5","presion":"0","Insertado":"2016-08-16 18:09:04"}],"max":[{"Id":"3","presion":"55","Insertado":"2016-08-16 16:22:14"}],"avg":[]},"last_entry":{"Id":"8","presion":"50","Insertado":"2016-08-16 18:28:45"}}}
As it shows. avg give empty!.
This is code in PHP
$press_values = get_press_values($json_object->result);
// get pressure result set with respected values
$press_result = get_press_result_set_from_values($json_object->result,$press_values);
// get latest entry
$latest_entry = get_latest_date_entry($json_object->result);
// Wrap results in an array
$output_result = array(
'pressure' => $press_result,
'last_entry' => $latest_entry
);
}
Then.
function get_press_values($result){
$min = -1;
$max = -1;
$avg = -1;
// get all pressure values
$pressures = array_map(function($result_item) {
return intval($result_item->presion);
}, $result);
if($pressures){
$min = min($pressures);
$max = max($pressures);
$avg = intval(calculate_average($pressures));
}
return array(
'min' => $min,
'max' => $max,
'avg' => $avg
);
}
Then:
function get_press_result_set_from_values($array,$value){
$min_objs = array();
$max_objs = array();
$avg_objs = array();
foreach ($array as $item) {
if($item->presion == $value['min']){
$min_objs[] = $item;
}
if($item->presion == $value['max']){
$max_objs[] = $item;
}
if($item->presion == $value['avg']){
$avg_objs[] = $item;
}
}
return array(
'min' => $min_objs,
'max' => $max_objs,
'avg' => $avg_objs,
);
}
then:
function calculate_average($arr) {
$total = 0;
$count = count($arr); //total numbers in array
foreach ($arr as $value) {
$total = $total + $value; // total value of array numbers
}
$average = ($total/$count); // get average value
return $average;
}
This part in your code:
if($item->presion == $value['avg']){
$avg_objs[] = $item;
}
will only add an item if the presion perfectly matches the average, which can be very unlikely, depending on your data of course.
update: For example you have the values 50, 51, 54, 55. The average is 53. Your code will not find any $item->presion which is equal to 53, because the average doesn't need an item that has that value (in contrast to min and max).
You should check if you actually mean the median (which has to be determined in a different way ...).
update2: to get the average into your result, you have to change
$avg_objs = array(); to $avg_objs = $value['avg'];. and remove the part from your code that I posted above.
Also, if you want your average to not be an integer, you should change the line
$average = ($total/$count);
to
$average = (float)$total / $count;
and remove the intval from:
$avg = intval(calculate_average($pressures));
I have this xml file named flight-itinerary.xml. A scaled-down version is shown below.
<itin line="1" dep="LOS" arr="ABV">
<flt>
<fltav>
<cb>1</cb>
<id>C</id>
<av>10</av>
<cur>NGN</cur>
<CurInf>2,0.01,0.01</CurInf>
<pri>15000.00</pri>
<tax>30800.00</tax>
<fav>1</fav>
<miles></miles>
<fid>11</fid>
<finf>0,0,1</finf>
<cb>2</cb>
<id>J</id>
<av>10</av>
<cur>NGN</cur>
<CurInf>2,0.01,0.01</CurInf>
<pri>13000.00</pri>
<tax>26110.00</tax>
<fav>1</fav>
<miles></miles>
<fid>12</fid>
<finf>0,0,0</finf>
</fltav>
</flt>
</itin>
The complete file contains 8 itinerary <itin> elements. The <fltav> element of each of the <itin> elements contains 11 of the <cb>1</cb> to <finf>0,0,1</finf> groups.
And below is the code I am using to process the file:
<?php
function processFlightsData()
{
$data = array();
$dom= new DOMDocument();
$dom->load('flight-itinerary.xml');
$classbands = $dom->getElementsByTagName('classbands')->item(0);
$bands = $classbands->getElementsByTagName('band');
$itineraries = $dom->getElementsByTagName('itin');
$counter = 0;
foreach($itineraries AS $itinerary)
{
$flt = $itinerary->getElementsByTagName('flt')->item(0);
$dep = $flt->getElementsByTagName('dep')->item(0)->nodeValue;
$arr = $flt->getElementsByTagName('arr')->item(0)->nodeValue;
$time_data = $flt->getElementsByTagName('time')->item(0);
$departure_day = $time_data->getElementsByTagName('ddaylcl')->item(0)->nodeValue;
$departure_time = $time_data->getElementsByTagName('dtimlcl')->item(0)->nodeValue;
$departure_date = $departure_day. ' '. $departure_time;
$arrival_day = $time_data->getElementsByTagName('adaylcl')->item(0)->nodeValue;
$arrival_time = $time_data->getElementsByTagName('atimlcl')->item(0)->nodeValue;
$arrival_date = $arrival_day. ' '. $arrival_time;
$flight_duration = $time_data->getElementsByTagName('duration')->item(0)->nodeValue;
$flt_det = $flt->getElementsByTagName('fltdet')->item(0);
$airline_id = $flt_det->getElementsByTagName('airid')->item(0)->nodeValue;
$flt_no = $flt_det->getElementsByTagName('fltno')->item(0)->nodeValue;
$flight_number = $airline_id. $flt_no;
$airline_type = $flt_det->getElementsByTagName('eqp')->item(0)->nodeValue;
$stops = $flt_det->getElementsByTagName('stp')->item(0)->nodeValue;
$av_data = $flt->getElementsByTagName('fltav')->item(0);
$cbs = iterator_to_array($av_data->getElementsByTagName('cb')); //11 entries
$ids = iterator_to_array($av_data->getElementsByTagName('id')); //ditto
$seats = iterator_to_array($av_data->getElementsByTagName('av')); //ditto
$curr = iterator_to_array($av_data->getElementsByTagName('cur')); //ditto
$price = iterator_to_array($av_data->getElementsByTagName('pri')); //ditto
$tax = iterator_to_array($av_data->getElementsByTagName('tax')); //ditto
$miles = iterator_to_array($av_data->getElementsByTagName('miles')); //ditto
$fid = iterator_to_array($av_data->getElementsByTagName('fid')); //ditto
$inner_counter = 0;
for($i = 0; $i < count($ids); $i++)
{
$data[$counter][$inner_counter] = array
(
'flight_number' => $flight_number,
'flight_duration' => $flight_duration,
'departure_date' => $departure_date,
'departure_time' => substr($departure_time, 0, 5),
'arrival_date' => $arrival_date,
'arrival_time' => substr($arrival_time, 0, 5),
'departure_airport_code' => $dep,
'departure_airport_location_name' => get_airport_data($dep, $data_key='location'),
'arrival_airport_code' => $arr,
'arrival_airport_location_name' => get_airport_data($arr, $data_key='location'),
'stops' => $stops,
'cabin_class' => $ids[$i]->nodeValue,
'ticket_class' => $ids[$i]->nodeValue,
'ticket_class_nicename' => formate_ticket_class_name($ids[$i]->nodeValue),
'available_seats' => $seats[$i]->nodeValue,
'currency' => $curr[$i]->nodeValue,
'price' => $price[$i]->nodeValue,
'tax' => $tax[$i]->nodeValue,
'miles' => $miles[$i]->nodeValue,
);
++$inner_counter;
}
return $data;
}
?>
Now, the outer loop iterates 8 times for each <itin> element, and during each iteration of the outer loop, the inner loop iterates 11 times, resulting in a total of 88 iterations per pass and causing serious performance issues. What I am looking for is a faster method of processing the file. Any helps will be greatly appreciated.
I don't think the loop is the bottle-neck. You should check your operations that are called within the loop, get_airport_data and formate_ticket_class_name.
Trying your code (without the auxiliary operations) on a number of itin elements takes less than a second, check this fiddle: http://phpfiddle.org/main/code/7fpi-b3ka (Note that the XML might not be similar to yours, I've guessed a lot of elements that were missing).
If there are operations that are called which increases the processing time substantially, try to call the operation with bulk data or cache the responses.
I have the following code,
This is where i process the textfield input type wherein i type in the ingredient of an recipe.
The proper format of how one should input the ingredient is like this.
5-ml milk, 4-ml water, 2-pcs carrot
There are always delimiter in between since i have different columns for quantity, unit of measurement and the name.
I have no problem with the separation. I have another table. Which is named ingredients.
This is where i populate different ingredients like milk and put in what food group they belong and their nutritional info(what it gives off, like calcium).
Anyhow, I get a database error, Somehow, when i try to insert after all that processing, both the type and nutrition are null. I've thought about it, maybe my query was wrong? but it didnt give off any warning or notice. Though it only gave me notice/warning that both the values are undefined when i tried to declare them at array $comp which are the values to be inserted.
I get this error,
Error Number: 1048
Column 'nutrition' cannot be null
INSERT INTO component (componentname, quantity, unit,
nutrition, type) VALUES ('milk', '5', 'ml', NULL, NULL)
Filename: C:\www\KG\system\database\DB_driver.php
Line Number: 330
The code:
function insertrecipe()
{
$ingredients = $this->input->post('ingredients');
$recipename = $this->input->post('recipename');
$recipe = array(
'recipename' => $recipename,
'description' => $this->input->post('description'),
'instruction' => $this->input->post('instructions'),
'serving' => $this->input->post('serving'),
);
$this->db->insert('recipe', $recipe);
$this->ingredient($ingredients,$recipename);
}
function ingredient($ingredients,$recipename)
{
$ids = array();
$first_slice = explode(',',$ingredients);
$secondslice = $this->second_slice($first_slice);
foreach($secondslice as $qty => $iname)
{
$arr = explode('-',$qty);
$third_slice[$arr[1]] = $arr[0];
$sql = $this->db
->select('nutrition,group')
->where('ingname', $iname)
->from('ingredients')
->get()
->result_array();
$result = $this->db->query($sql);
foreach($result->result_array() as $row)
{
$ingredient_type = $this->get_ingredient_type($row['group']);
}
foreach($third_slice as $measure => $num)
{
$comp = array(
'componentname' => $iname,
'quantity' => $num,
'unit' => $measure,
'nutrition' => $nutri,
'type' => $ingredient_type
);
if($insert2 = $this->db->insert('component',$comp))
{
$latest_id = $this->db->insert_id();
$ids[] = $latest_id;
}
}
}
}
function second_slice($data)
{
foreach($data as $key => $value)
{
$ingredient = explode(' ', trim($value));
$second_slice[$ingredient[0]] = $ingredient[1];
}
return $second_slice;
}
function get_ingredient_type($data)
{
//foreach($data->result_array() as $row)
//{
if($data['group'] == "vegetable")
{
$type = 1;
}
else if($data['group'] == "fruit")
{
$type = 2;
}
else if($data['group'] == "dairy")
{
$type = 3;
}
else if($data['group'] == "seafood")
{
$type = 4;
}
else
{
$type = 5;
}
//}
return $type;
}
The database table of component has the following columns.
componentid componentname quantity unit nutrition type
nutrition is varchar unit is int. I guess they're not null or cols that dont accept null values.
I've separated different foreach loops into functions. I originally thought the error was because i had 3-5 for each loop within that function alone. so i decided to separate them into functions.