Dears,
i have an excel file with 5K rows and i'm importing it to my table in the DB successfully.
But the error, when the system finish all the rows, it keeps looping and the page doesn't stop running and not redirecting to my view.
My controller:
if($request->hasFile('import_file')){
$path = $request->file('import_file')->getRealPath();
$data = \Excel::load($path)->get();
foreach ($data as $key => $row) {
$res = policies::where('phone', '=', $row['phone'])
->where('draft_no', '=', $row['draftno'])
->where('due_date', '=', $duedate)
->select('id')->get()->toArray();
if(empty($res)) {
$polic = new policies();
$polic->cust_id = $row['custno'];
$polic->policy = '';
$polic->bord_date = $borddate;
$polic->client_id = $row['clientid'];
$polic->client_no = $row['clientno'];
$polic->client_name = $row['clientname'];
$polic->draft_no = $row['draftno'];
if ($row['status'] == '') {
$polic->status = '';
} else {
$polic->status = $row['status'];
}
$polic->due_date = $duedate;
if ($row['curno'] == 'USD') {
$polic->currency = 1;
} else {
$polic->currency = 0;
}
$polic->amount = $row['amnt'];
$polic->zone = $row['zone'];
$polic->broker_id = $row['brokercode'];
$polic->broker_name = $row['brokername'];
$polic->remarks = $row['remarks'];
$polic->phone = $row['phone'];
$polic->insured_name = $row['insname'];
// $polic->cust_id = $row['valuedate'];
$polic->address = ''; //address
if (trim($row['status']) == 'P') {
$polic->paid_at = date('Y-m-d');
}
$polic->new = 1; //address
$polic->save();
}
else {
//am updating the imported date in the DB
}
what is very strange that in my localhost is working fine, but in digitaloceans cloud, keep looping without redirecting.
Thanks for your help.
I can be because you have 5000 rows to insert and 5000 insert operation consumes lots of memory. What you can try is batch insert operation.
In your policies.php make all fields fillable
protected $fillable=['cust_id ','policy','bord_date','client_id','client_no','client_name ','draft_no','bord_date','status','due_date','currency','amount','zone','broker_id','broker_name','remarks','phone','insured_name','address','paid_at','new'];
And on your excel file import use exists rather than getting collections.
if($request->hasFile('import_file')){
$path = $request->file('import_file')->getRealPath();
$data = \Excel::load($path)->get();
$data=[];
$i=0;
foreach ($data as $key => $row) {
$res = policies::where('phone', '=', $row['phone'])
->where('draft_no', '=', $row['draftno'])
->where('due_date', '=', $duedate)
->exists();
if(!$res) {
$i++;
$data[$i]['cust_id ']=$row['custno'];
$data['policy'] = '';
$data['bord_date'] = $borddate;
$data[$i]['client_id'] = $row['clientid'];
$data[$i]['client_no'] = $row['clientno'];
$data[$i]['client_name'] = $row['clientname'];
$data[$i]['draft_no'] = $row['draftno'];
if ($row['status'] == '') {
$data[$i]['status'] = '';
} else {
$data[$i]['status'] = $row['status'];
}
$data[$i]['due_date'] = $duedate;
if ($row['curno'] == 'USD') {
$data[$i]['currency'] = 1;
} else {
$data[$i]['currency'] = 0;
}
$data[$i]['amount'] = $row['amnt'];
$data[$i]['zone'] = $row['zone'];
$data[$i]['broker_id'] = $row['brokercode'];
$data[$i]['broker_name'] = $row['brokername'];
$data[$i]['remarks'] = $row['remarks'];
$data[$i]['phone'] = $row['phone'];
$data[$i]['insured_name'] = $row['insname'];
// $data[$i]['cust_id'] = $row['valuedate'];
$data[$i]['address'] = ''; //address
if (trim($row['status']) == 'P') {
$data[$i]['paid_at'] = date('Y-m-d');
}
$data[$i]['new'] = 1; //address
}
else {
//am updating the imported date in the DB
}
}
policies::insert($data);
Related
I am unable to enter multiple data, it enter only single data. I have tried using for loop and then entering data, using 3 user and 2 task, there is an error previously offset.
public function add($postData)
{
// dd($postData);
$c = count($postData['user_name']);
$t = count($postData['task_name']);
for ($i = 0; $i < $c; $i++) {
$user_name = $postData['user_name'][$i];
$user_email = $postData['user_email'][$i];
$data['insert']['user_name'] = $user_name;
$data['insert']['user_email'] = $user_email;
}
for ($j = 0; $j < $t; $j++) {
$task_name = $postData['task_name'][$j];
$data['insert']['task_name'] = $task_name;
}
$data['insert']['name'] = $postData['name'];
$data['insert']['description'] = $postData['description'];
$data['insert']['customer_name'] = $postData['customer_name'];
$data['insert']['billing_method'] = $postData['billing_method'];
$data['insert']['dt_created'] = DT;
$data['table'] = PROJECT;
$result = $this->insertRecord($data);
if ($result == true) {
$response['status'] = 'success';
$response['message'] = 'Project created';
} else {
$response['status'] = 'danger';
$response['message'] = DEFAULT_MESSAGE;
}
return $response;
}
As Per lack of question details attached, I am supposing that you want to insert multiple task entry with project name, description etc.
Here is updated code:
<?php
// dd($postData);
$username = $postData['username'];
$user_email = $postData['user_email'];
$task_name = $postData['task_name'];
foreach ($username as $key => $value) {
$data['insert']['name'] = $postData['name'];
$data['insert']['description'] = $postData['description'];
$data['insert']['customer_name'] = $postData['customer_name'];
$data['insert']['billing_method'] = $postData['billing_method'];
$data['insert']['username'] = $value;
$data['insert']['user_email'] = $user_email[$key];
$data['insert']['task_name'] = $task_name[$key];
$data['insert']['dt_created'] = DT;
$data['table'] = PROJECT;
$result = $this->insertRecord($data);
}
In this code section, the query gets all the teachers and appends their assigned classes within the object
The weird thing is it works perfectly on local but not online, it is almost it doesn't wait for the while loop
<?php
$SID = mysqli_real_escape_string($con, htmlspecialchars($_POST["SID"], ENT_QUOTES));
$allTeachersArr = [];
$test = [];
$q = "SELECT onderwysers.*,klasse.Klas,klasse.Datum,klasse.Vak FROM `onderwysers` LEFT JOIN klasse ON onderwysers.ID = klasse.Teacher WHERE `SID` = '$SID'";
// echo $q;
$result = $con->query($q);
if ($result->num_rows > 0) {
while ($row = $result->fetch_assoc()) {
$subjectItem = new stdClass();
$filtered_array = [];
$isIn = false;
$var1 = $row['ID'];
$var2 = $row['Teacher'];
if (count($allTeachersArr) > 0)
foreach ($allTeachersArr as $item) {
if ($item->ID == $var1 && $item->Teacher == $var2) {
// do something
$filtered_array = $item;
$isIn = true;
}
}
if ($isIn == true) {
$notAdd = false;
// echo "updates";
$subjectItem->Klas = $row["Klas"];
$subjectItem->Datum = $row["Datum"];
$subjectItem->Vak = $row["Vak"];
foreach ($filtered_array->displaySubjects as $item) {
if ($item->Klas == $row["Klas"] && $item->Vak == $row["Vak"] && $item->Datum == $row["Datum"]) {
// do something
$filtered_array = $item;
$notAdd = true;
}
}
if ($notAdd == false) {
$temp = $filtered_array->displaySubjects;
array_push($temp, $subjectItem);
$filtered_array->displaySubjects = $temp;
}
} else {
//add new entry
$subjectItem->ID = $row["ID"];
$subjectItem->Title = $row["Title"];
$subjectItem->Name = $row["Name"];
$subjectItem->Surname = $row["Surname"];
$subjectItem->Afkorting = $row["Afkorting"];
$subjectItem->IdentityNumber = $row["IdentityNumber"];
$subjectItem->Geslag = $row["Geslag"];
$subjectItem->ContactDetails = $row["ContactDetails"];
$subjectItem->Email = $row["Email"];
$subjectItem->RegKlas = $row["RegKlas"];
$subjectItem->Status = $row["Status"];
$subjectItem->UID = $row["UID"];
$subjectItem->displaySubjects = [];
array_push($allTeachersArr, $subjectItem);
}
}
}
echo json_encode($allTeachersArr);
$con->close();
THE EXPECTED RETURNED DATA(This is local, online returns nothing)
I have to get data from fingerprint machine. Everything in my code run and works, but if I show all employee data, it loads very slow.
I have create datatables by codeigniter and ajax call to view. My code like below
// Datatables Variables
$draw = intval($this->input->get("draw"));
$start = intval($this->input->get("start"));
$length = intval($this->input->get("length"));
$year = $this->input->post("year");
$month = $this->input->post("month");
$dept = $this->input->post("dept");
$where = $year.'-'.$month.'-';
$shifts = $this->m_general->get_data('tb_shift')->result();
$rows = $this->m_employee->attendance($where)->result_array();
if($dept=='5'){
$pin_emp = $this->m_general->get_data('tb_employee')->result();
}else{
$pin_emp = $this->m_general->check_data('tb_employee',array('emp_dept'=>$dept))->result();
}
$loc_machine= $this->m_general->get_data('tb_pkm')->result();
$roaster = $this->m_employee->list_schedule_emp($where)->result_array();
$holiday = $this->m_general->get_data('tb_holiday')->result();
$total_days = tglakhir($year,$month);
$dates = array();
$people = array();
$data = array();
foreach ($rows as $row) {
if (empty($row['date']) || empty($row['pin'])) {
continue; // no date or sn, nothing to print for this row...
}
// in array assures unique values
if (!in_array($row['date'], $dates)) {
$dates[] = $row['date'];
}
// this may seem convoluted, take your time to understand
$people[$row['date']][$row['pin']] = $row;
$array_emp[$row['date']][$row['pin']][] = $row;
}
foreach ($roaster as $rs) {
if (empty($rs['date']) || empty($rs['id_nip'])) {
continue; // no date or sn, nothing to print for this row...
}
// this may seem convoluted, take your time to understand
$get_rs[$rs['date']][$rs['id_nip']] = $rs;
$dt_rs[$rs['date']][$rs['id_nip']][] = $rs;
}
foreach($pin_emp AS $q){
$array_pin[]=$q->emp_pin;
$details[$q->emp_pin] = $q;
}
foreach($holiday AS $hday){
$get_hday[$hday->date_holiday] = $hday;
}
foreach($loc_machine AS $list_machine){
$lm[$list_machine->id_machine] = $list_machine;
}
for($m=1; $m <= $total_days; $m++){
if($m<10){
$m = '0'.$m;
}else{
$m = $m;
}
$date[]=$m;
}
foreach($shifts AS $shift_e){
$array_shift[$shift_e->id_shift] = $shift_e;
$ar_shift[]=$shift_e->id_shift;
}
foreach ($array_pin AS $id){
$name = $details[$id]->emp_name;
$shift = $details[$id]->emp_shift;
$id_machine = $details[$id]->emp_reg_sn;
if(isset($lm[$id_machine])){
$loc_machine= $lm[$id_machine]->pkm_name;
}
$date_range = array();
foreach($date AS $date_col){
$day = date('D', strtotime($year.'-'.$month.'-'.$date_col));
$today = date('Y-m-d');
$full_day = $year.'-'.$month.'-'.$date_col;
$s_in = '';
$s_out = '';
$time_in = '';
$time_out = '';
$t_in = '';
$t_out = '';
$finger_tap = '';
$diff_in = '';
$diff_out = '';
$diff = null;
$diff2 = null;
$late = '00:00:00';
$r = $date_col;
$title = '';
$date_holiday= '';
if (ISSET($get_hday[$full_day])==TRUE) {
$date_holiday = $get_hday[$full_day]->date_holiday;
$title = $get_hday[$full_day]->note_holiday;
}
foreach ($dates as $key) {
$shift_detail= $array_shift[$shift]->code_shift;
if(isset($get_rs[$full_day][$id]['id_shift_emp']) && $get_rs[$full_day][$id]['id_shift_emp']!=''){
$shift = $get_rs[$full_day][$id]['id_shift_emp'];
$shift_detail = $array_shift[$shift]->code_shift;
}
if (isset($people[$key][$id]['date']) && $people[$key][$id]['date'] == $key) {
foreach ($array_emp[$key][$id] AS $dtl){
if(isset($get_rs[$key][$id]['id_shift_emp']) && $get_rs[$key][$id]['id_shift_emp']!=''){
$shift = $get_rs[$key][$id]['id_shift_emp'];
}
$shift = $dtl['emp_shift'];
$date_time = $dtl['date_time'];
if(($shift==5)OR($shift==7)OR($shift==9)OR($shift==11)OR($shift==12)OR($shift==14)){
$r = $date_col+1;
}
if(substr($key,8,2)==$date_col){
$status_code= $dtl['status_code'];
$status_note= $dtl['att_status'];
$first_in = strtotime($year.'-'.$month.'-'.$date_col.' '.$array_shift[$shift]->first_check);
$last_out = strtotime($year.'-'.$month.'-'.$r.' '.$array_shift[$shift]->last_check);
$shift_in = $array_shift[$shift]->in_shift;
$shift_out = $array_shift[$shift]->out_shift;
$s_in = strtotime($year.'-'.$month.'-'.$day.' '.$shift_in);
$s_out = strtotime($year.'-'.$month.'-'.$day.' '.$shift_out);
$finger_tap = strtotime($date_time);
$shift_detail= $array_shift[$shift]->code_shift;
foreach($ar_shift AS $id_shift){
if ($shift==$id_shift) {
$currdiff = abs($finger_tap - $first_in);
if (is_null($diff) || $currdiff < $diff) {
$diff = $currdiff;
$time_in = date('H:i:s',strtotime($date_time));
$t_in = strtotime($date_time);
if(($t_in-$s_in)>0){
$diff_in = gmdate('H:i:s',$t_in-$s_in);
}else{
$diff_in = $late;
}
}
$currdiff2 = abs($finger_tap - $last_out);
if (is_null($diff2) || $currdiff2 < $diff2) {
$diff2 = $currdiff2;
$time_out = date('H:i:s',strtotime($date_time));
$t_out = strtotime($date_time);
if(($t_out-$s_out)>0){
$diff_out = $late;
}else{
$diff_out= gmdate('H:i:s',$s_out-$t_out);
}
}
}
}
}
if(($shift==5)OR($shift==7)OR($shift==9)OR($shift==11)OR($shift==12)OR($shift==14)){
if ($day==$r) {
$shift_out = $array_shift[$shift]->out_shift;
$s_out = strtotime($year.'-'.$month.'-'.$r.' '.$shift_out);
$finger_tap = strtotime($date_time);
foreach($ar_shift AS $id_shift){
if ($shift==$id_shift) {
$currdiff2 = abs($finger_tap - $last_out);
if (is_null($diff2) || $currdiff2 < $diff2) {
$diff2 = $currdiff2;
$time_out = date('H:i:s',strtotime($date_time));
$t_out = strtotime($date_time);
if(($t_out-$s_out)>0){
$diff_out = $late;
}else{
$diff_out= gmdate('H:i:s',$s_out-$t_out);
}
}
}
}
}
}
$status = '<i class="material-icons col-green">check_box_outline_blank</i>'.$shift_detail.'';
if($year.'-'.$month.'-'.$date_col < $today){
$status = '<i class="material-icons col-red">fiber_manual_record</i>'.$shift_detail.'';
}
if(($day == 'Sun') OR ($day == 'Sat')){
$status='<a href="javascript:change_status_att(\''.$id.'\',\''.$name.'\',\''.$full_day.'\')" class="btn btn-md bg-red waves-effect" data-toggle="tooltip" data-placement="top" title="Libur">off</button>';
}
if($date_holiday == $full_day){
$status = '<i class="material-icons col-red">today</i>OFF';
}
if($time_in!=''){
if($time_in==$time_out){
$status = '<i class="material-icons col-red">brightness_6</i>'.$shift_detail.'';
}
if ($status_code!='') {
$status = '<i class="material-icons col-orange">sim_card_alert</i>'.$shift_detail.'';
}
}
if($time_in!=$time_out){
$status = '<i class="material-icons col-green">check_box</i>'.$shift_detail.'';
}
if($time_in=='00:00:00'){
$status = '<i class="material-icons col-orange">sim_card_alert</i>'.$shift_detail.'';
}
}
}
}
$date_range[] = $status;
}
$first = array(
''.$id.'',
$name,
$loc_machine
);
$data[] = array_merge($first,$date_range);
}
$output = array(
"draw" => $draw,
"recordsTotal" => '',
"recordsFiltered" =>'' ,
"data" => $data
);
$this->output->set_content_type('application/json')->set_output(json_encode($output));
}
This code runs and works, but need about 10 second for load.
Can somebody help Me what I have to do to be fast this load?
UPDATE
after I analysis my code, the slow load data because datatables say "data was truncated". But when I load on small data(filtered), It was fine.
problem still same when I get all data from database.
To make it faster, you first need to know what is slow. To do that, you need to use debugging tools, like this one and timestamps.
Then, once you get logs with the duration of your processes, you can analyze it to know which part of your code is slow.
Then, try to optimize it. And if you can’t, edit your question to focus on the slow part or post a new one.
We are using the finicity API to get statements and transaction. We have followed below steps, but we are getting "aggregationStatusCode:185" while fetching the customer accounts [ https://api.finicity.com/aggregation/v1/customers/['customer_id']/accounts ].
To get the custommer account I am using https://api.finicity.com/aggregation/v1/customers/['customer_id']/accounts.
Then we are fetching statements for account using "https://api.finicity.com/aggregation/v1/customers/['customer_id']/accounts/['account_id']/statement?index=1; this loops for 6 times. We need 6 months statement for each account.
Then, we are fetching transaction for account using "https://api.finicity.com/aggregation/v1/customers/['customer_id']/accounts/['account_id']/transactions?fromDate=".$from."&toDate=".strtotime(date('Y-m-d'))."&limit=".$transactionLimit."&includePending=true";
We are facing the below two issues.
1. Getting timeout error, when selecting more than 2 accounts.
2. Getting "aggregationStatusCode:185" for some live accounts.
Below are the code for reference.
function get_statements(){
$FinicityAppToken=$_POST['FinicityAppToken'];
$customerId=$_POST['customerId'];
$leadId=$_POST['leadId'];
$finicityConnectUrl=$_POST['finicityConnectUrl'];
$NoOFStatements = 6;
$transactionLimit = 100;
$transactionsMonth = 1;
$url="https://api.finicity.com/aggregation/v1/customers/".$customerId."/accounts";
$response=callAPI('POST', $url,false,$FinicityAppToken);
$response = json_decode($response);
$allstatements=array();
if($response){
if(isset($response->accounts)){
$accounts = array();
$array = array();
foreach($response->accounts as $account){
if ($account->aggregationStatusCode == 0 && $account->aggregationSuccessDate != '' && $account->aggregationAttemptDate != '') {
for($i=1;$i<=$NoOFStatements;$i++) {
$url="https://api.finicity.com/aggregation/v1/customers/".$customerId."/accounts/".$account->id."/statement?index=".$i;
$stateresponse=callAPI('GET', $url,false,$FinicityAppToken);
$temp = array();
$temp['accountId'] = $account->id;
$temp['statement'] = base64_encode($stateresponse);
$statements[]=$temp;
}
$act = array();
$month = date('m') - $transactionsMonth;
$from = strtotime(date('Y').'-'.$month.'-1');
$url="https://api.finicity.com/aggregation/v1/customers/".$customerId."/accounts/".$account->id."/transactions?fromDate=".$from."&toDate=".strtotime(date('Y-m-d'))."&limit=".$transactionLimit."&includePending=true";
$data['url'] = $url;
$transactions = json_decode(callAPI('GET', $url,false,$FinicityAppToken));
$act['account']['accountNumber'] = $account->number;
unset($account->number);
$act['account']['currencySymbol'] = $account->currency;
unset($account->currency);
if(empty((array) $account->detail)) {
$account->detail = (object) transactionsDetailsEmpty();
}
$act['account'] = json_decode(json_encode($account), true);
if(isset($transactions->transactions)) {
$act['transactions'] = $transactions->transactions;
} else {
$act['transactions'] = array();
}
// echo "step";
$accounts[] = $act;
} else {
$data['success'] = 1;
$data['response']=$response ;
$data['finicityConnectUrl'] = $finicityConnectUrl;
$data['leadId'] = $leadId;
$data['FinicityAppToken'] = $FinicityAppToken;
$data['customerId'] = $customerId;
echo json_encode($data);
die();
}
$j++;
}
$array['accounts'] = $accounts;
$array['leadId'] = $leadId;
$data['accounts'] = $array;
$url="https://api.finicity.com/aggregation/v1/customers/".$customerId;
$response=callAPI('DELETE', $url,false,$FinicityAppToken);
}
}
}
Is it possible to insert in to two tables at once? I need to insert some data in to a table (students) and then based on the primary key insert in to another (enrollments). Is this possible?
The enrollments table however pulls other primary key from another table called schools (id).
public function add()
{
if(isset($_POST['submit'])) {
$this->student_validation();
if($this->form_validation->run() === TRUE) {
$data = $this->_get_posted_student_data();
$insert_id = $this->student->insert('students', $data);
if($insert_id) {
$this->__insert_enrollment($insert_id);
success($this->lang->line('insert_success'));
redirect('web/index/' . $data['front_school_id']);
} else {
error($this->lang->line('insert_failed'));
redirect('web/admission');
}
} else {
$this->data['post'] = $_POST;
}
}
$this->data['schools'] = $this->schools;
$this->data['add'] = TRUE;
$this->layout->title($this->lang->line('add') . ' ' . $this->lang->line('student') . ' | ' . SMS);
$this->layout->view('web/index', $this->data);
}
private function __insert_enrollment($insert_id)
{
$data = array();
$school = $this->student->get_school_by_id($this->input->post('front_school_id'));
$data['student_id'] = $insert_id;
$data['school_id'] = $this->input->post('front_school_id');
//$data['class_id'] = $this->input->post('class_id');
//$data['section_id'] = $this->input->post('section_id');
$data['academic_year_id'] = $school->academic_year_id;
//$data['roll_no'] = $this->input->post('roll_no');
$data['created_at'] = date('Y-m-d H:i:s');
//$data['created_by'] = logged_in_user_id();
$data['status'] = 1;
$this->db->insert('enrollments', $data);
}
private function _get_posted_student_data()
{
$items = array();
$school_id = $this->session->userdata('front_school_id');
$items[] = 'first_name';
$items[] = 'middle_name';
$items[] = 'last_name';
$items[] = 'gender';
$items[] = 'religion';
$items[] = 'dobpin';
$items[] = 'address';
$items[] = 'immunization_update';
$items[] = 'nationality';
$items[] = 'valid_permit';
$items[] = 'relation_father';
$items[] = 'father_name';
$items[] = 'father_phone';
$items[] = 'father_email';
$items[] = 'father_address';
$items[] = 'relation_mother';
$items[] = 'mother_name';
$items[] = 'mother_phone';
$items[] = 'mother_email';
$items[] = 'mother_address';
$items[] = 'other_sibling';
$items[] = 'name_sibling';
$items[] = 'class_sibling';
$items[] = 'name';
$items[] = 'relation';
$items[] = 'child';
$items[] = 'application_name';
$data = elements($items, $_POST);
$data['dob'] = date('Y-m-d', strtotime($this->input->post('dob')));
$data['admission_date'] = date('Y-m-d', strtotime($this->input->post('admission_date')));
$data['age'] = floor((time() - strtotime($data['dob'])) / 31556926);
$data['modified_at'] = date('Y-m-d H:i:s');
//$data['modified_by'] = logged_in_user_id();
$data['created_at'] = date('Y-m-d H:i:s');
//$data['created_by'] = logged_in_user_id();
$data['status'] = 1;
// create user
//$data['user_id'] = $this->student->create_user();
if($_FILES['birth_certificate']['name']) {
$data['birth_certificate'] = $this->_upload_birth_certificate();
}
if($_FILES['bill']['name']) {
$data['bill'] = $this->_upload_bill();
}
if($_FILES['immunization']['name']) {
$data['immunization'] = $this->_upload_immunization();
}
if($_FILES['photo']['name']) {
$data['photo'] = $this->_upload_photo();
}
return $data;
}
You need $this->db->insert_id() to get last inserted id which will get from this.
$this->db->insert('students', $data);
$insert_id = $this->db->insert_id();
if(!empty($insert_id)) {
$this->__insert_enrollment($insert_id);
success($this->lang->line('insert_success'));
redirect('web/index/' . $data['front_school_id']);
} else {
error($this->lang->line('insert_failed'));
redirect('web/admission');
}