Predicting future IDs used before saving to the DB - php

I am saving a complex dataset in Laravel 4.2 and I am looking for ways to improve this.
$bits has several $bobs. A single $bob can be one of several different classes. I am trying to duplicate a singular $bit and all its associated $bobs and save all of this to the DB with as few calls as possible.
$newBit = $this->replicate();
$newBit->save();
$bobsPivotData = [];
foreach ($this->bobs as $index => $bob) {
$newBob = $bob->replicate();
$newBobs[] = $newBob->toArray();
$bobsPivotData[] = [
'bit_id' => $newBit->id,
'bob_type' => get_class($newBob),
'bob_id' => $newBob->id,
'order' => $index
];
}
// I now want to save all the $bobs kept in $newBobs[]
DB::table('bobs')->insert($newBobs);
// Saving all the pivot data in one go
DB::table('bobs_pivot')->insert($bobsPivotData);
My problem is here, that I cant access $newBob->id before I have inserted the $newBob after the loop.
I am looking for how best to reduce saves to the DB. My best guess is that if I can predict the ids that are going to be used, I can do all of this in one loop. Is there a way I can predict these ids?
Or is there a better approach?

You could insert the bobs first and then use the generated ids to insert the bits. This isn't a great solution in a multi-user environment as there could be new bobs inserted in the interim which could mess things up, but it could suffice for your application.
$newBit = $this->replicate();
$newBit->save();
$bobsPivotData = [];
foreach ($this->bobs as $bob) {
$newBob = $bob->replicate();
$newBobs[] = $newBob->toArray();
}
$insertId = DB::table('bobs')->insertGetId($newBobs);
$insertedBobs = DB::table('bobs')->where('id', '>=', $insertId);
foreach($insertedBobs as $index => $newBob){
$bobsPivotData[] = [
'bit_id' => $newBit->id,
'bob_type' => get_class($newBob),
'bob_id' => $newBob->id,
'order' => $index
];
}
// Saving all the pivot data in one go
DB::table('bobs_pivot')->insert($bobsPivotData);
I have not tested this, so some pseudo-code to be expected.

Related

How to optimize large data inserts using laravel eloquent

Good day, basically I want to insert some related data all at once using eloquent. My current code is:
$allStudies = Study::chunk(50, function ($studies) use ($request, $questionData, $answerData) {
foreach ($studies as $study) {
$evaluationInsert = Evaluation::create([
'study_id' => $study->id,
'questionnaire' => $request->questionnaire,
'description' => $request->description
]);
$evaluationQuestions = $evaluationInsert
->question()
->createMany($questionData);
foreach ($evaluationQuestions as $question) {
$question->answer()->createMany($answerData);
}
}
});
The result of $allStudies is a collection of Study model that currently have around 150-ish data. $questionData is just a static array of arrays that consist of 38 elements and $answerData is an array of arrays that have 4 elements which consist the answer options of each questions. However, the code does work but it takes too long time to execute because of big loops and increasing the timeout in php seems not an ideal way to solve this. What is the elegant way to solve this kind of case?

Laravel Bulk Update for multiple record ids

I want to mass update my records in Laravel but the records not getting updated. I have a different record for each Id. Below is what I am trying.
$ids = [5,6,8,9],
$updated_array = [
['name' => 'tarun'],
['name' => 'Akash'],
['name' => 'Soniya'],
['name' => 'Shalu'],
];
Model::whereIn('id', $ids)->update($updated_array);
Mass updates are used when you're trying to update multiple rows to the same values. You cannot mass update with different values.
Therefore, this would work, but would update all matching records to name of 'tarun':
Model::whereIn('id', $ids)->update(['name' => 'tarun']);
For your example, you could do:
foreach($ids as $key => $id) {
Model::where('id', $id)->update($updated_array[$key]);
}
But as far as I know, there's no way to do this without running 4 queries in Laravel and writing a raw SQL statement to accomplish this would even be messy.
You can use Laravel Upsert for mass update.
For example :
User::query()->upsert([
['id' => 1, 'email' => 'dadan#example.com'],
['id' => 2, 'email' => 'satria#example.com'],
], 'email');
This feature available in Laravel 8 or newer
Some good solutions for this issue are on the following post: https://github.com/laravel/ideas/issues/575
1) Can create custom function to do this, which uses raw SQL, as per barryvdh comment in the post.
public static function updateValues(array $values)
{
$table = MyModel::getModel()->getTable();
$cases = [];
$ids = [];
$params = [];
foreach ($values as $id => $value) {
$id = (int) $id;
$cases[] = "WHEN {$id} then ?";
$params[] = $value;
$ids[] = $id;
}
$ids = implode(',', $ids);
$cases = implode(' ', $cases);
$params[] = Carbon::now();
return \DB::update("UPDATE `{$table}` SET `value` = CASE `id` {$cases} END, `updated_at` = ? WHERE `id` in ({$ids})", $params);
}
This apparently increase performance by 13x
2) As per another comment, another idea is to do it on a per record basis, but do the whole thing as a single transaction, which makes it much faster.
DB::beginTransaction();
// your loop and updates;
if( !$user )
{
rollbackTransaction();
} else {
// Else commit the queries
commitTransaction();
}
3) There is a laravel library also that appears to try and solve this issue. https://github.com/mavinoo/laravelBatch
Note: I have not tried or tested any of the above solutions.
I think the only way to do this without n queries would be to
(optional) backup table
grab the records and create an array with all the updated data
bulk delete the records
bulk insert from array
That's 3 queries.
Iterating through n records is not practical for my application either; I was hoping there was an alternative but it looks like I'm going to have to implement this.
We can Insert and update batch (bulk) in laravel using this Package
Laravel Batch
Please try the below code:
$data_to_be_updated = [ ['id'=>5,'name' => 'tarun'], ['id'=>6, 'name' => 'Akash'],
['id'=>8, 'name' => 'Soniya'], ['id'=>9,'name' => 'Shalu']
];
foreach ($data_to_be_updated as $key => $value) {
$data = Model::where('id',$value['id'])->first();
if ($data) {
$data->name = $value['name'];
$data->save();
}
}
you can do it like this above mentioned:-
foreach($ids as $key => $id) {
Model::where('id', $id)->update($updated_array[$key]);
}

How do I optimise laravel seeds beyond mass insertion for faster seeding/

So I am developing a laravel application and I am trying to get my seeds optimised so that they run faster.
http://bensmith.io/speeding-up-laravel-seeders
This guide helped a ton. According to this, I should minimise the number of SQL queries by doing mass insertions and it cut down the time to 10% of the original seeding time which is awesome.
So now I am doing something like:
$comments = [];
for ($i = 0; $i < 50; $i++) {
$bar->advance();
$comments[] = factory(Comment::class)->make([
'created_at' => Carbon\Carbon::now(),
'updated_at' => Carbon\Carbon::now(),
'comment_type_id' => $comment_types->shuffle()->first()->id,
'user_id' => $users->shuffle()->first()->id,
'commentable_id' => $documents->shuffle()->first()->id,
])->toArray();
}
Comment::insert($comments);
This works like a charm. It gets the queries down to a single one.
But then I have other seeders where I to work with dumps and they are more complex:
$dump = file_get_contents(database_path('seeds/dumps/serverdump.txt'));
DB::table('server')->truncate();
DB::statement($dump);
$taxonomies = DB::table('server')->get();
foreach($taxonomies as $taxonomy){
$bar->advance();
$group = PatentClassGroup::create(['name' => $taxonomy->name]);
preg_match_all('/([a-zA-Z0-9]+)/', $taxonomy->classes, $classes);
foreach(array_pop($classes) as $key => $class){
$type = strlen($class) == 4 ? 'GROUP' : 'MAIN';
$inserted_taxonomies[] = PatentClassTaxonomy::where('name', $class)->get()->count()
? PatentClassTaxonomy::where('name', $class)->get()->first()
: PatentClassTaxonomy::create(['name' => $class, 'type' => $type]);
}
foreach($inserted_taxonomies as $inserted_taxonomy){
try{
$group->taxonomies()->attach($inserted_taxonomy->id);
}catch(\Exception $e){
//
}
}
}
So here when I am attaching taxonomies to groups, I use the native eloquent code so taking the record and mass inserting is difficult.
Yes, I can fiddle around and figure out a way to mass insert that too but my problem is that I have to write and optimise all seeds and every part of those seeds to mass insert.
Is there a way, where I can listen to DB queries laravel is trying to execute while seeding. I know I can do something like this:
DB::listen(function($query) {
//
});
But it would still be executed right. What I would like to do is somehow catch the query in a variable, add it to a stack and then execute the whole stack when the seed is coming to an end. Or in between too since I might need some ids for some seeds. What is a good workaround this? And how to really optimise the seeds in laravel with a smart solution?

Check performance of variable initializing

I have loop like following and its run for more than 6000 records,
foreach ($csv as $value) {
$research = ResearchData::create(array('company_id' => Input::get('company'), 'date' => Input::get('date')));
}
in here i used 2 values company_id and date.
i want to know what is the most good way to use this from follow codes
................1....................
$company_id=Input::get('company_id');
$date=Input::get('date');
foreach($csv as value){
$research=ResearchData::create(array('company_id'=>$company_id,'date'=>$date));
}
................2...................
foreach ($csv as $value) {
$research = ResearchData::create(array('company_id' => Input::get('company'), 'date' => Input::get('date')));
}
From a performance point of view, number 1 will be faster, but only because Input::get will take a tiny little bit longer as it does some checks, an array concatenation and eventually grabs something from an array. This will take a completely negligible amount of time, but option 1 does this once whereas option 2 will do this for every iteration of the loop
From any other point of view (code clarity, documentation etc) it's completely opinion based.
You can do a bulk insert. I didn't do a performance check, but I expect a better performance. Check below:
$company_id=Input::get('company_id');
$date=Input::get('date');
$data = array_fill(0, count($csv) - 1, ['company_id' => $company_id, 'date' => $date]); // skip the large foreach
ResearchData::insert(array_values($data)); // skip the numeric keys
Documentation:
http://php.net/array_filter
http://laravel.com/docs/4.2/queries#inserts

How to import Excelsheets in Symfony doctrine entity

I want to import an ExcelSheet to my DB using Symfony/Doctrine (imported ddeboer data-import bundle)
What is best practice to import the data and first check if the data is already imported?
I was thinking of two possibilities:
1)
$numarray = $repo->findAllAccounts();
$import = true;
foreach ($reader as $readerobjectkey => $readervalue) {
foreach ($numarray as $numkey){
if (($numkey->getNum() == $readervalue['number'])){
$import = false;
}
}
if($import){
$doctrineWriter ->disableTruncate()
->prepare()
->writeItem(
array(
'num' => $readervalue['number'],
'name' => $readervalue['name'],
'company' => $companyid
)
)
->finish();
2)
foreach ($reader as $row =>$value ) {
// check if already imported
$check = $this->checkIfExists($repo,'num', $value['number']);
if ($check){
echo $value['number']." Exists <br>";
}else{echo $value['number']." new Imported <br>";
$doctrineWriter ->disableTruncate()
->prepare()
->writeItem(
array(
'num' => $value['number'],
'name' => $value['name'],
'company' => $companyid
)
)
->finish();
public function checkIfExists($repo, $field, $value){
$check = $repo->findOneBy(array($field => $value));
return $check;
Problem is with big exceldatasheets (3000 rows +) with both solutions i get a timeout....
Error: Maximum execution time of 30 seconds exceeded
in general: for performance issues: is it prefered to generate 1000 queries to check if value exists (findOneBy) or to use two foreach loops to compare values?
Any help would be awesome!
Thx in advance...
You can try to check the filemtime of the file : http://php.net/manual/en/function.filemtime.php
I'm not sure if it would work properly but it worth a shot to try it and see if the modified date works as expected.
Otherwise you should think of another way that checking the data like this, it would take lot of resources to do so. maybe adding some metadata to the excel file :
http://docs.typo3.org/typo3cms/extensions/phpexcel_library/1.7.4/manual.html#_Toc237519906
Any other way than looping or querying database for large data is better.

Categories