Slow upload with .csv files and PHP - php

I've created a function to upload csv files using PHP and insert its contents into a Mongo database.
The upload is taking to much time to be completed and I don't have idea how to build a better code to do this.
Bellow my function in PHP:
public function importEmail($file,$list_name,$header){
$data = $this->ReadCSV($file, $has_header);
global $Email;
$falhas = 0;
array_shift($data);
if($data === false) {
return false;
} else {
foreach($data as $key => $value) {
$Email->email = $value[$header["email"]];
$Email->first_name = $value[$header["fName"]];
$Email->last_name = $value[$header["lName"]];
$Email->gender = strtoupper($value[$header["gender"]]);
$Email->status = $value[$header["status"]] !="" ? $value[$header["status"]] : "1";
$Email->list_name = $list_name;
$Email->opt_out = $v = (strtoupper($value[$header["opt"]]) != "" ? strtoupper($value[$header["opt"]]) : "N");
if(!$Email->update(trim($value[$header["email"]]))) $falhas++;
}
}
return $falhas === 0 ? true : false;
}
EDITED
This is the update() method which is responsible to insert records into MongoDB
public function update($id) {
global $mongo, $gClient;
$update = array(
'$set' => array(
'userData.name' => $this->first_name,
'userData.lastName' => $this->last_name,
'userData.gender' => $this->gender,
'userData.optOut' => $this->opt_out == "Y" || $this->opt_out == "N" ? $this->opt_out : "Y",
'userData.lastUpdate' => (int)(microtime(true) * 1000),
'userData.status' => $this->status == 0 || $this->status == 1 ? +$this->status : 0
),
'$setOnInsert' => array(
'userData.registerDate' => (int)(microtime(true) * 1000)
)
);
if(is_string($this->list_name) && strlen($this->list_name) > 0)
$update['$push'] = array(
'userData.meta' => array(
'type' => 'listname',
'name' => $this->list_name
)
);
$result = $mongo->{$gClient->slug}->usersHistory->update(array(
'userEmail' => $id
), $update, array(
'upsert' => true
));
if(!$this->email) $this->email = $id;
$mongo->{$gClient->slug}->usersHistory->update(array(
'userEmail' => $id
), array(
'$set' => array(
'userEmail' => $this->email
)
));
return $result['n'] > 0;
}
Those .csv files are always bigger than 50k (lines).
How can I build something faster than this method?

Related

Parsing returns an empty value

I make a parser of items from DotA 2 user inventory in the Steam service. Every time I try to parse user data, I get an empty value:
{"success":true,"items":[]}, but there are items in my Steam inventory.
My function to parse items:
public function loadMyInventory() {
if(Auth::guest()) return ['success' => false];
$prices = json_decode(Storage::get('prices.txt'), true);
$response = json_decode(file_get_contents('https://steamcommunity.com/inventory/'.$this->user->steamid64.'/570/2?l=russian&count=5000'), true);
if(time() < (Session::get('InvUPD') + 5)) {
return [
'success' => false,
'msg' => 'Error, repeat in '.(Session::get('InvUPD') - time() + 5).' сек.',
'status' => 'error'
];
}
//return $response;
$inventory = [];
foreach($response['assets'] as $item) {
$find = 0;
foreach($response['descriptions'] as $descriptions) {
if($find == 0) {
if(($descriptions['classid'] == $item['classid']) && ($descriptions['instanceid'] == $item['instanceid'])) {
$find++;
# If we find the price of an item, then move on.
if(isset($prices[$descriptions['market_hash_name']])) {
# Search data
$price = $prices[$descriptions['market_hash_name']]*$this->config->curs;
$class = false;
$text = false;
if($price <= $this->config->min_dep_sum) {
$price = 0;
$text = 'Cheap';
$class = 'minPrice';
}
if(($descriptions['tradable'] == 0) || ($descriptions['marketable'] == 0)) {
$price = 0;
$class = 'minPrice';
$text = 'Not tradable';
}
# Adding to Array
$inventory[] = [
'name' => $descriptions['market_name'],
'price' => floor($price),
'color' => $this->getRarity($descriptions['tags']),
'tradable' => $descriptions['tradable'],
'class' => $class,
'text' => $text,
'classid' => $item['classid'],
'assetid' => $item['assetid'],
'instanceid' => $item['instanceid']
];
}
}
}
}
}
Session::put('InvUPD', (time() + 5));
return [
'success' => true,
'items' => $inventory
];
}
But should return approximately the following value:
{"success":true,"items":[{"classid":"2274725521","instanceid":"57949762","assetid":"18235196074","market_hash_name":"Full-Bore Bonanza","price":26}]}
Where my mistake?
First of all, you are iterating on descriptions for every assets, which is assets*descriptions iteration, it's quite a lot, but you can optimize this.
let's loop once for descriptions and assign classid and instanceid as object key.
$assets = $response["assets"];
$descriptions = $response["descriptions"];
$newDescriptions=[];
foreach($descriptions as $d){
$newDescriptions[$d["classid"]][$d["instanceid"]] = $d;
}
this will give as the ability to not loop over description each time, we can access the description of certain asset directly $newDescriptions[$classid][$instanceid]]
foreach($assets as $a){
if(isset($newDescriptions[$a["classid"]]) && isset($newDescriptions[$a["classid"]][$a["instanceid"]])){
$assetDescription = $newDescriptions[$a["classid"]][$a["instanceid"]];
$inventory = [];
if(isset($prices[$assetDescription["market_hash_name"]])){
$price = $prices[$assetDescription['market_hash_name']]["price"]*$this->config->curs;
$class = false;
$text = false;
if($price <= $this->config->min_dep_sum) {
$price = 0;
$text = 'Cheap';
$class = 'minPrice';
}
if(($assetDescription['tradable'] == 0) || ($assetDescription['marketable'] == 0)) {
$price = 0;
$class = 'minPrice';
$text = 'Not tradable';
}
$inventory["priceFound"][] = [
'name' => $assetDescription['market_name'],
'price' => floor($price),
'color' => $this->getRarity($assetDescription['tags']),
'tradable' => $assetDescription['tradable'],
'class' => $class,
'text' => $text,
'classid' => $a['classid'],
'assetid' => $a['assetid'],
'instanceid' => $a['instanceid']
];
}else{
$inventory["priceNotFound"][] = $assetDescription["market_hash_name"];
}
}
}
About your mistake:
are you Sure your "prices.txt" contains market_hash_name?
I don't see any other issue yet, operationg on the data you have provided in comment, I got print of variable $assetDescription. Please doublecheck variable $prices.

how to count matched id from excel and database

i have an excel file and i want to upload it and save data to database.
And data will be updated based on IC Number (Identity card).
If in excel file there are 4 IC are matched with IC from the database, it will count and there will be a notification if 4 data match and update to database.
The question is how to count matched ic from excel and database?
it always show zero (0)
if($request->hasFile('file')){
$extension = File::extension($request->file->getClientOriginalName());
if ($extension == "xlsx" || $extension == "xls" || $extension == "csv") {
$path = $request->file->getRealPath();
$data = Excel::load($path, function($reader) {
})->get();
if(!empty($data) && $data->count()){
foreach ($data as $value) {
$insert['data'] = [
'no' => $value->no,
'name' => $value->name,
'ic_no' => $value->ic_no,
'disburse_date' => $value->disburse_date,
'product_type_description' => $value->product_type_description,
'amount_release' => $value->amount_release,
'amount' => $value->amount,
'date_disburse' => $value->date_disburse,
'net_disbursement' => $value->net_disbursement,
];
if(!empty($insert))
{
$user = Auth::user()->id;
$request = new DisburseFromMbsb;
//$request->cus_id = $data->id_cus;
$request->ic_no = $value['ic_no'];
$request->name = $value['name'];
$request->disburse_date = $value['disburse_date'];
$request->product_type_description = $value['product_type_description'];
$request->amount_release = $value['amount_release'];
$request->amount = $value['amount'];
$request->date_disburse = $value['date_disburse'];
$request->net_disbursement = $value['net_disbursement'];
$request->user_id = $user;
$request->save();*/
Loandisburse::where('ic', $value['ic_no'])->where('stage','W11')->where('status_upload',0)->update(array(
'status_upload'=>'1',
'amount_release' => $value['amount_release'],
'amount' => $value['amount'],
'date_disburse' => $value['date_disburse'],
'net_disbursement' => $value['net_disbursement'],
'disburse_date' => $value['disburse_date'],
'product_type_description' => $value['product_type_description']
));
$number = Loandisburse::where('ic', $value['ic_no'])->where('stage','W11')->where('status_upload',1)->count(); // this is my code to count
}
}
return redirect('/upload/amount')->with(['update' => 'Data saved successfully '.$number.' rows' ]);
}
this is my code to count
$number = Loandisburse::where('ic', $value['ic_no'])->where('stage','W11')->where('status_upload',1)->count();
Assuming all Code you wrote is correct (I have no idea how the Excel Package works ),
there are a few lines that need altering and you are good to go.
First you have to declare the variable number outside the scope of forEach
After that, instead of
$number = Loandisburse.......
You have to use
$number += Loandisburse.........
In order not to overwrite your previous values.
That means you have to try the following code:
if($request->hasFile('file')){
$extension = File::extension($request->file->getClientOriginalName());
$number = 0;
if ($extension == "xlsx" || $extension == "xls" || $extension == "csv") {
$path = $request->file->getRealPath();
$data = Excel::load($path, function($reader) {
})->get();
if(!empty($data) && $data->count()){
foreach ($data as $value) {
$insert['data'] = [
'no' => $value->no,
'name' => $value->name,
'ic_no' => $value->ic_no,
'disburse_date' => $value->disburse_date,
'product_type_description' => $value->product_type_description,
'amount_release' => $value->amount_release,
'amount' => $value->amount,
'date_disburse' => $value->date_disburse,
'net_disbursement' => $value->net_disbursement,
];
if(!empty($insert)) {
$user = Auth::user()->id;
$request = new DisburseFromMbsb;
$request->ic_no = $value['ic_no'];
$request->name = $value['name'];
$request->disburse_date = $value['disburse_date'];
$request->product_type_description = $value['product_type_description'];
$request->amount_release = $value['amount_release'];
$request->amount = $value['amount'];
$request->date_disburse = $value['date_disburse'];
$request->net_disbursement = $value['net_disbursement'];
$request->user_id = $user;
$request->save();
Loandisburse::where('ic', $value['ic_no'])
->where('stage','W11')
->where('status_upload',0)->update(array(
'status_upload'=>'1',
'amount_release' => $value['amount_release'],
'amount' => $value['amount'],
'date_disburse' => $value['date_disburse'],
'net_disbursement' => $value['net_disbursement'],
'disburse_date' => $value['disburse_date'],
'product_type_description' => $value['product_type_description']
));
$number += Loandisburse::where('ic', $value['ic_no'])
->where('stage','W11')
->where('status_upload',1)
->count(); // this is my code to count
}//Closing If Not Empty
}//Closing For Each
}//Closing If
}//Closing If
return redirect('/upload/amount')->with(['update' => 'Data saved successfully '.$number.' rows' ]);
}//Closing If

getting multiple customer details In codeigniter

i have written this code to receive data from the Android device. it was inserted just one customer data I need to receive multiple customer details if app goes offline. but it was inserting one data into DB in offline mode also.how can i change this for multiple customer data insertions.
function index_post($customerID = false) {
if ($customerID) {
//update the record
$updateData = array();
$allowedparams = array('streetid' => 'streetid', 'name' => 'name', 'mobile' => 'mobile', 'adhaar' => 'adhaar', 'profession' => 'profession', 'address' => 'address', 'pincode' => 'pincode', 'nearby' => 'nearby', 'paddress' => 'paddress', 'isOwned' => 'isOwned');
foreach ($allowedparams as $k => $v) {
if (!$this->IsNullOrEmptyString($this->post($k, true))) {
$updateData[$v] = $this->post($k, true);
}
}
if ($this->model_customer->update($customerID, $updateData)) {
$data = array('status' => true, 'messsage' => 'cusotmer updated succesfully');
$http_code = REST_Controller::HTTP_OK;
} else {
$data = array('status' => false, 'messsage' => 'cusotmer failed to update.');
$http_code = REST_Controller::HTTP_INTERNAL_SERVER_ERROR;
}
} else {
//insert the record
$allowedparams = array('streetid' => 'streetid', 'name' => 'name', 'mobile' => 'mobile', 'adhaar' => 'adhaar', 'profession' => 'profession', 'address' => 'address', 'pincode' => 'pincode', 'cycle' => 'cycle', 'nearby' => 'nearby', 'paddress' => 'paddress', 'isOwned' => 'isOwned');
$requiredParam = array('streetid', 'name', 'mobile', 'cycle');
$insertdata = array();
foreach ($allowedparams as $k => $v) {
if (in_array($k, $requiredParam)) {
//check if its not null
if ($this->post($k) == null || trim($this->post($k)) == '') {
$data = array('status' => false, 'message' => $k . ' parameter missing or empty');
$http_code = REST_Controller::HTTP_BAD_REQUEST;
break;
}
}
$insertData[$v] = $this->post($k, true);
}
if ($customerID = $this->model_customer->create($insertData)) {
$data['customerID'] = $this->_frameCustomer2($this->model_customer->get($customerID)); //you need to put
$http_code = REST_Controller::HTTP_OK;
} else {
$data = array('status' => false, 'message' => 'unable to create customer');
$http_code = REST_Controller::HTTP_INTERNAL_SERVER_ERROR;
}
}
$this->response($data, $http_code);
}
private function _frameCustomer2($c) { //get value from index_get
$data = array();
$data['id'] = $c->id;
$data['name'] = $c->name;
$data['street'] = array('id' => $c->streetid);
$data['creationDate'] = $c->creationdate;
$data['mobile'] = $c->mobile;
$data['adhaar'] = $c->adhaar;
$data['profession'] = $c->profession;
$data['isOwned'] = ($c->isOwned == 1) ? true : false;
$data['address'] = $c->address;
$data['pincode'] = $c->pincode;
$data['status'] = $c->status;
$data['cycle'] = $c->cycle;
$data['balance'] = $c->balance;
$data['creditAvailable'] = $c->creditbalance;
$data['nearby'] = $c->nearby;
$data['accountNumber'] = $c->accountnumber;
$data['permanentAddress'] = $c->paddress;
$data['lastVisit'] = $this->model_customer->lastVisit($c->id);
return $data;
}
and my part of model function is
function create($insertdata = array()) { //new customer insert
if ($this->db->insert('customer', $insertdata)) {
return $this->db->insert_id();
} else {
return false;
}
}
function update($customerID = 0, $updateData = array()) {
$this->db->where('id', $customerID);
if ($this->db->update('customer', $updateData) && $this->db->affected_rows() == 1) {
return true;
} else {
return false;
}
Instead of customer Id, you can ask the mobile developers to send data in the form of array. In both online and offline. In case of online there will be just one element in the request array.
function index_post() {
$request_data = $this->request->body;
foreach($request_data as $key => $value)
{
//Check if customer id is present
if(Customer present)
{
Update the record
} else {
Insert the record
}
}
}

zend multicheckboxes form controller

Merged with multicheckboxes zend form multi populate.
i did do the form with multicheckboxs and it works fine when insert or update but the my problem is how to populate all the multi boxes that is checked form the database this is the code but it doesn't show just one checkbox is checked
$id = (int) $this->_request->getParam('id');
//The incoming request
$request = $this->getRequest();
//initialize form
$form = new Admin_Form_DownFooterTab();
//instance of db
$db = Zend_Db_Table::getDefaultAdapter();
if ($this->getRequest()->isPost()) {
if ($form->isValid($request->getPost())) {
if (isset($id) && $id != "") {
$gettag = $form->getValue('tag_id');
$gettags = count($gettag);
try {
//shift the orders to
$select = $db->select()->from(array('t' => 'tab'), array('t.id',
't.title',
't.tab_position',
't.order',
't.is_active',
't.is_deleted'))->where('id = ?', $id);
$currentTab = $db->fetchRow($select);
$var3 = array('tab.order' => new Zend_Db_Expr('tab.order - 1'));
$var4 = array('tab.order >= ' . $currentTab['order'], 'is_active=1', 'is_deleted=0', 'tab_position=4');
$db->update('tab', $var3, $var4);
$var = array('tab.order' => new Zend_Db_Expr('tab.order + 1'));
$var2 = array('tab.order >= ' . $form->getValue('order'), 'is_active=1', 'is_deleted=0', 'tab_position=4');
$db->update('tab', $var, $var2);
$db->delete('tab_tag', array('tab_id = ?' => $id));
foreach ($gettag as $value) {
$db->insert('tab_tag',
array(
'tag_id' => $value,
'tab_id' => $id
));
}
$db->update('tab', array(
'title' => $form->getValue('title'),
'body' => $form->getValue('body'),
'is_active' => $form->getValue('is_active'),
'banner_link' => $form->getValue('banner_link'),
'tab_path' => $form->getValue('tab_path'),
'link_open' => $form->getValue('link_open'),
'tab_position' => $form->getValue('tab_position'),
'tab_parent' => $form->getValue('tab_parent')
),
array('id =?' => $id));
$this->flash('Tab Updated', 'admin/tab');
} catch (Exception $e) {
$this->flash($e->getMessage(), 'admin/tab');
}
} else {
try {
$formValues = $form->getValues();
$formValues['created_by'] = 1;
$formValues['created_date'] = date('Y/m/d H:i:s');
$formValues['is_deleted'] = 0;
$var = array('tab.order' => new Zend_Db_Expr('tab.order + 1'));
$var2 = array('tab.order >= ' . $form->getValue('order'), 'is_active=1', 'is_deleted=0', 'tab_position=4');
$db->update('tab', $var, $var2);
foreach ($gettag as $value) {
$db->insert('tab_tag',
array(
'tag_id' => $value,
'tab_id' => $id
));
}
$db->insert('tab', array(
'title' => $form->getValue('title'),
'body' => $form->getValue('body'),
'is_active' => $form->getValue('is_active'),
'banner_link' => $form->getValue('banner_link'),
'tab_path' => $form->getValue('tab_path'),
'link_open' => $form->getValue('link_open'),
'tab_position' => $form->getValue('tab_position'),
'tab_parent' => $form->getValue('tab_parent')
));
$this->flash('Tab inserted', 'admin/tab');
} catch (Exception $e) {
$this->flash($e->getMessage(), 'admin/tab');
}
}
}
}
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
if (isset($id) && $id != "") {
$values = $db->fetchRow("SELECT * FROM tab WHERE id = ?", $id);
$form->populate($values);
}
$this->view->form = $form;
}
$ddlCat_parent->setMultiOptions($cats);
$this->view->form = $form;
}

PHP Variables for if statements

Ok, I am wanting to do something like this:
$which = !empty($param_id) ? "['groups'][$param_id]" : "['groups']";
And than I'd like it to be able to do something like so...
$all_groups . $which = array(
-1 => array(
'id' => '-1',
'name' => $txt['parent_guests_only'],
'checked' => in_array('-1', $checked) || in_array('-3', $checked),
'is_post_group' => false,
)
And I need it to build an array like so, if !empty($param_id)
$all_groups['groups'][$param_id] = array(the array info);
But if $param_id is empty it should do the this instead:
$all_groups['groups'] = array(the array info);
I don't think I can concatenate it or can I?
Can someone please help me here? This is happening many many times throughout a function, so I don't want to use if... else... statements every single time. Would be too many, thinking of a 1 fast approach for all of them.
Thanks :)
EDIT, here is the function in question:
function ListGroups($checked = array(), $unallowed = array(), $order = array(), $param_id = 0)
{
global $context, $smcFunc, $txt;
// We'll need this for loading up the names of each group.
if (!loadLanguage('ManageBoards'))
loadLanguage('ManageBoards');
if (empty($checked))
return array();
$all_groups['groups'][$param_id] = array();
if (!in_array('-1', $unallowed))
// Guests
$all_groups['groups'][$param_id] = array(
-1 => array(
'id' => '-1',
'name' => $txt['parent_guests_only'],
'checked' => in_array('-1', $checked) || in_array('-3', $checked),
'is_post_group' => false,
)
);
if (!in_array('0', $unallowed))
{
// Regular Members
if (!empty($all_groups['groups']))
$all_groups['groups'][$param_id] += array(
0 => array(
'id' => '0',
'name' => $txt['parent_members_only'],
'checked' => in_array('0', $checked) || in_array('-3', $checked),
'is_post_group' => false,
)
);
else
$all_groups['groups'][$param_id] = array(
0 => array(
'id' => '0',
'name' => $txt['parent_members_only'],
'checked' => in_array('0', $checked) || in_array('-3', $checked),
'is_post_group' => false,
)
);
}
// Load membergroups.
$request = $smcFunc['db_query']('', '
SELECT group_name, id_group, min_posts
FROM {db_prefix}membergroups
WHERE id_group > {int:is_zero}',
array(
'is_zero' => 0,
)
);
while ($row = $smcFunc['db_fetch_assoc']($request))
{
if (!in_array($row['id_group'], $unallowed))
{
$all_groups['groups'][(int) $param_id][(int) $row['id_group']] = array(
'id' => $row['id_group'],
'name' => trim($row['group_name']),
'checked' => in_array($row['id_group'], $checked) || in_array('-3', $checked),
'is_post_group' => $row['min_posts'] != -1,
);
}
}
$smcFunc['db_free_result']($request);
// Let's sort these arrays accordingly!
if (!empty($order))
{
$all_groups['groups'][$param_id] = sortGroups($all_groups['groups'][$param_id], $order);
$context['group_order' . $param_id] = implode(', ', $order);
}
else
{
$context['group_order' . $param_id] = '';
sort($all_groups['groups'][$param_id]);
$x = 0;
foreach ($all_groups['groups'][$param_id] as $key => $value)
{
$x++;
$context['group_order' . $param_id] .= $x < count($all_groups['groups'][$param_id]) ? $value['id'] . ', ' : $value['id'];
}
}
return $all_groups['groups'][$param_id];
}
I need to do a check for !empty($param_id), if so, it needs to build the $all_groups['groups'] array without the $param_id.
So will need to add in a check for if (!empty($params_id)) build the array like so: $all_groups['groups'][$params_id] else build it like this instead: $all_groups['groups']. I don't want a bunch of if... else... statements in here, just a 1 or 5 liner would be GREAT!
Thanks Guys :)
Don't overcomplicate it. :)
$array = array(
/* contents */
);
if (!empty($param_id)) {
$all_groups['groups'][$param_id] = $array;
} else {
$all_groups['groups'] = $array;
}
I don't know what $all_groups['groups'] looks like before this; if it was empty, I'd shorten this to:
$all_groups['groups'] = array(
/* contents */
);
if (!empty($param_id)) {
$all_groups['groups'] = array($param_id => $all_groups['groups']);
}
if (!empty($param_id)) {
$which = &$all_groups['groups'][$param_id]
} else {
$which = &$all_groups['groups'];
}
$which = array(
-1 => array(
'id' => '-1',
'name' => $txt['parent_guests_only'],
'checked' => in_array('-1', $checked) || in_array('-3', $checked),
'is_post_group' => false,
);
unset($which); // unset $which, if you want to use this variable
// in this scope once again
Generally speaking, references are the solution. See #zerkms' answer.
However, if at all possible, I would try to redesign your data structures such that you don't have to resort to this type of conditional behavior. For example, using default as the default missing key:
$which = !empty($param_id) ? $param_id : 'default';
$all_groups['groups'][$which] = array( ... );
I don't know if it's possible in your case, but this could be easier to manage.
Other potential solutions:
$tmp = array( ... );
if ($cond)
$foo = $tmp;
else
$bar = $tmp;
or:
function make_array( ... )
{
return array( ... );
}
if ($cond)
$foo = make_array( ... );
else
$bar = make_array( ... );

Categories