How to optimize Postgres for bulk insert data? - php

I have a table that must be optimized for bulk insert data over 100 00 rows for execution time.
Table columns
I try to insert data using PHP, where each row is element of array:
$dataset = [["columnindex" => 1, "rowindex" => 2, "type" => "num", "value" => 400], ...]
Problem is when I try to insert array with 100 rows the Postgres does not work, also PDO does not return any errors.
I use insert from Laravel:
SessionPrepared::insert($dataset);
If to slice array it is adeed to db:
$dataset = array_slice($dataset, 0, 10);

Related

Efficient solution to generating an array in PHP, which extracts unique data from one array, based on data from another

Writing in PHP, I have 2 arrays, each created from SQL queries.
The first query runs through a table that has multiple pieces of data that correspond to various quiz attempts. The table has a column for the user's Email, the activity ID (which represents a quiz attempt) and another 2 columns for data relating to the attempt (for example 'percentage achieved' or 'quiz ID'):
UserEmail ActID ActKey ActMeta
joB#gm.com 2354 Percentage 98
joB#gm.com 2354 Quiz ID 4
boM#hm.com 4567 Percentage 65
boM#hm.com 4567 Quiz ID 7
Once queried, this first array ($student_quiz_list) stores the selected data in the form of
[[UserEmail, ActID, ActKey, ActMeta], [UserEmail, ActID, ActKey, ActMeta], [UserEmail, ActID, ActKey, ActMeta]...]
where each pair of sub-arrays corresponds to a single quiz attempt.
The second table that is queried has two columns that relate to the quizzes themselves. The first column is the Quiz ID and the second is the Quiz name.
Quiz ID Quiz Name
4 Hardware
7 Logic
Once queried, this second array ($quiz_list) stores the selected data in the form of
[[ID, Name], [ID, Name]...]
What I need to do is create a 3rd array (from the 2 above) which holds the user's email and percentage score
[email, percentage], [email, percentage]...]
but with each sub-array corresponding to a unique actID (so basically the user's percentage in each quiz they attempted without duplicates) and (this is the challenging bit) only for quizzes with certain ID values, in this case, let's say quiz ID 4.
In PHP, what would be the most efficient solution to this? I continually create arrays with duplicates and cannot find a neat solution which provides the outcome desired.
Any help would be greatly received.
Try this code as the example and let me know.
$student_quiz_list=array(
array(
'UserEmail'=>'joB#gm.com','ActID'=>'2354','ActKey'=>'Percentage','ActMeta'=>'90',
),
array(
'UserEmail'=>'joB#gm.com','ActID'=>'2354','ActKey'=>'QuizID','ActMeta'=>'4',
),
array(
'UserEmail'=>'boM#hm.com','ActID'=>'4567','ActKey'=>'Percentage','ActMeta'=>'98',
),
array(
'UserEmail'=>'boM#hm.com','ActID'=>'4567','ActKey'=>'QuizID','ActMeta'=>'7',
),
);
$final_array=array();
foreach( $student_quiz_list as $row){
if($row['ActKey']=='Percentage'){
$final_array[]=array('UserEmail'=>$row['UserEmail'],
'ActMeta'=>$row['ActMeta']
) ;
}
}
echo"<pre>"; print_r($final_array); echo"</pre>";
As commenter #Nico Haase suggested, you can do most of the logic in SQL. You didn't respond to my comment, so I suppose a user can have multiple attempts per quiz ID:
SELECT
UserEmail,
ActMeta
FROM
your_table # replace with your table name
WHERE
ActKey = 'Percentage'
AND ActID IN (
# subselection with table alias
SELECT
t2.ActID
FROM
your_table t2 # replace with your table name
WHERE
t2.ActKey = 'Quiz ID'
AND t2.ActMeta = 2 # insert your desired quiz ID here
AND t2.ActID = ActID
)
(Query tested with MySQL/MariaDB)
For the case that you cannot change the SQL part, here is how you can process your data in PHP. But consider that a large dataset could exceed your server capabilities, so I would definitely recommend the solution above:
// Your sample data
$raw = [
['UserEmail' => 'joB#gm.com', 'ActID' => 2354, 'ActKey' => 'Percentage' , 'ActMeta' => 98],
['UserEmail' => 'joB#gm.com', 'ActID' => 2354, 'ActKey' => 'Quiz ID', 'ActMeta' => 4],
['UserEmail' => 'joB#gm.com', 'ActID' => 4567, 'ActKey' => 'Percentage' , 'ActMeta' => 65],
['UserEmail' => 'joB#gm.com', 'ActID' => 4567, 'ActKey' => 'Quiz ID', 'ActMeta' => 7],
];
// Extract the corresponding ActIDs for a QuizID
$quiz_id = 4;
$act_ids = array_column(
array_filter(
$raw,
function($item) use ($quiz_id) {
return $item['ActMeta'] == $quiz_id;
}
),
'ActID'
);
// Get the entries with ActKey 'Percentage' and an ActID present in the previously extracted set
$percentage_entries = array_filter(
$raw,
function($item) use ($act_ids) {
return $item['ActKey'] === 'Percentage' && in_array($item['ActID'], $act_ids);
}
);
// Map over the previous set to get the array into the final form
$final = array_map(
function($item) {
return [$item['UserEmail'], $item['ActMeta']];
},
$percentage_entries
);

How to Update Rows (with different data) in one DB Query Laravel

I'm making a gallery with sortable photos with Laravel and jQuery UI Sortable.
My function in the controller gets a nice array:
$items = [0 => 22, 1 => 25, 2 => 45];
But there will be approx 150 - 200 photos in one gallery. Is there any chance to set one DB Query instead 150 - 200? Because my controller makes this at the moment...
<?php
foreach($photos['item'] as $position => $id){
Photo::where('id', $id)->update(['position' => $position]);
}
But it creates approx 150 - 200 DB queries, which is awful.
Edit #1
Basically I need something like this (two corresponding arrays with ids and positions):
$ids = [22, 24, 25, 34];
$positions = [0, 1, 2, 3];
Photos::where('id', $ids)->update(['position'] => $positions);
But I can't find anything about this approach.
Take a look here: Eloquent model mass update.
Basically, you are looking for a mass or bulk update.

How to increment and update column in one eloquent query

Is it possible to update a timestamp (besides updated_at) and increment a column in one query? I obviously can
->increment('count')
and separately
->update(['last_count_increased_at' => Carbon::now()])
but is there an easy way to do both together.
Product::where('product_id', $product->id)
->update(['count'=> $count + 1, 'last_count_increased_at' => Carbon::now()];
Without having to query and get the count first?
You can specify additional columns to update during the increment or decrement operation:
Product::where('id',$id)
->increment('count', 1, ['increased_at' => Carbon::now()]);
It is more eloquent solution.
You can use the DB::raw method:
Product::where('product_id', $product->id)
->update([
'count'=> DB::raw('count+1'),
'last_count_increased_at' => Carbon::now()
]);
With Laravel 8 you can now achieve this in a single query to create or update on duplicate key.
$values = [
'name' => 'Something',
'count' => 1,
];
$uniqueBy = ['name'];
$update = ['count' => DB::raw('count+1')];
Model::upsert($values, $uniqueBy, $update);
If the model exists count will be incremented, if it is inserted count will equal 1. This is done on the DB level, so only one query involved.
Read more about upserts: https://laravel.com/docs/8.x/eloquent#upserts

project the sum of values in a mongo subdocument

I have a Mongo Collection that I'm trying to aggregate in which I need to be able to filter the results based on a sum of values from a subdocument. Each of my entries has a subdocument that looks like this
{
"_id": <MongoId>,
'clientID': 'some ID',
<Other fields I can filter on normally>
"bidCompData": [
{
"lineItemID": "210217",
"qtyBid": 3,
"priceBid": 10.25,
"qtyComp": 0
"description": "Lawn Mowed"
"invoiceID": 23
},
{
<More similar entries>
}
]
}
What I'm trying to do is filter on the sum of qtyBid in a given record. For example, my user could specify that they only want records that have a total qtyBid across all of the bidCompData that's greater than 5. My research shows that I can't use $sum outside of the $group stage in the pipeline but I need to be able to sum just the qtyBid values for each individual record. Presently my pipeline looks like this.
array(
array('$project' => $basicProjection), //fields to project calculated earlier using the input parameters.
array('$match' => $query),
array('$group' => array(
'_id' =>
array('clientID' => '$clientID'),
'count' => array('$sum' => 1)
)
)
I tried having another group and an unwind before the group I presently have in my pipeline so that I could get the sum there but it doesn't let me keep my fields besides the id and the sum field. Is there a way to do this without using $where? My database is large and I can't afford the speed hit from the JS execution.

Update Duplicate Data when insert data in laravel

Hi I've search and try to solve the issue but failed to grab solution sorry for taking your time.
I am inserting a set of data in laravel with the query builder to insert multiple data at a time.
DB::table('table')->insert(
array(
array(
'col1' => 'data1',
'col2' => 'data1'
),
array(
'col1' => 'data2',
'col2' => 'data2'
),
)
);
Is there any way to check if exist the col1,col2 value in table if exist then update otherwise insert. I wanted to get return true or false from the query result if all data successfully update or inserted. I wanted to solve it with laravel query builder.
Thanks
As far as I know, the Laravel query builder does not support this.
You could do a raw MySQL query:
INSERT INTO table (a,b,c) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE c=c+1;
https://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
But that is very messy and MySQL specific.
Instead i would suggest you use the Eloquent ORM:
// Retrieve the flight by the attributes, or create it if it doesn't exist...
$flight = App\Flight::firstOrCreate(['name' => 'Flight 10']);
http://laravel.com/docs/5.1/eloquent#inserting-and-updating-models

Categories