CakePHP 3 fast insertion/updating of records - php

Question

I am trying to insert/update +/- 10k rows with a foreach loop. The complete loop duration is about 3-5minutes. Are there any tips on my code to do the insertion of update faster? The $rows are retrieved from a xls file converted to domdocument.
foreach($rows as $key => $row)
{
if($key < 1){continue;}
$cells = $row -> getElementsByTagName('td');
foreach ($cells as $cell) {
$project_id = $cells[0]->nodeValue;
$title = $cells[1]->nodeValue;
$status = $cells[2]->nodeValue;
$projectmanager = $cells[3]->nodeValue;
$engineer = $cells[4]->nodeValue;
$coordinator = $cells[5]->nodeValue;
$contractor_a = $cells[6]->nodeValue;
$contractor_b = $cells[7]->nodeValue;
$gsu = $cells[9]->nodeValue;
$geu = $cells[10]->nodeValue;
$query = $this->Projects->find('all')->select(['project_id'])->where(['project_id' => $project_id]);
if ($query->isEmpty()) {
$project = $this->Projects->newEntity();
$project->title = $title;
$project->project_id = $project_id;
$project->status = $status;
$project->projectmanager = $projectmanager;
$project->engineer = $engineer;
$project->coordinator = $coordinator;
$project->contractor_a = $contractor_b;
$project->contractor_b = $contractor_a;
$project->gsu = date("Y-m-d H:i:s");
$project->geu = date("Y-m-d H:i:s");
$project->gsm = date("Y-m-d H:i:s");
$project->gem = date("Y-m-d H:i:s");
if ($this->Projects->save($project)) {
//$this->Flash->success(__('The project has been saved.'));
continue;
}else{
debug($project->errors());
}
}else{
continue;
$query->title = $title;
$query->status = $status;
$query->projectmanager = $projectmanager;
$query->engineer = $engineer;
$query->coordinator = $coordinator;
$query->contractor_a = $contractor_b;
$query->contractor_b = $contractor_a;
$query->gsu = $gsu;
$query->geu = $geu;
if ($this->Projects->save($query)) {
//$this->Flash->success(__('The project has been saved.'));
continue;
}
}
}
//$this->Flash->error(__('The project could not be saved. Please, try again.'));
}

Answer 1

For faster bulk inserts don't use entities but rather generate insert queries directly.
https://book.cakephp.org/3.0/en/orm/query-builder.html#inserting-data

Answer 2

Ello, my vriend.
The TableClass->save() method is useful when saving one single record, in your case, you should use TableClass->saveMany() instead.
For this to happen, you need to treat your entities as arrays inside your foreach.
After the foreach, you will use another method from the tableclass (newEntities) to convert the array into entities before finally save them.
Basic example:
//Lets supose our array after our foreach become something like this:
$all_records =
[
//Each item will be an array, not entities yet
[
'name' => 'I.N.R.I.',
'year' => '1987',
'label' => 'Cogumelo',
'country' => 'Brazil',
'band' => 'Sarcófago',
'line_up' => '[{"name":"Wagner Antichrist","role":"Vomits, Insults"},{"name":"Gerald Incubus","role":"Damned Bass"},{"name":"Z\u00e9der Butcher","role":"Rotten Guitars"},{"name":"D.D. Crazy","role":"Drums Trasher"}]'
],
//Another record coming in..
[
'name' => 'Eternal Devastation',
'year' => '1986',
'label' => 'Steamhammer',
'country' => 'Germany',
'band' => 'Destruction',
'line_up' => '[{"name":"Marcel Schmier","role":"Vocals, Bass"},{"name":"Mike Sifringer","role":"Guitars"},{"name":"Tommy Sandmann","role":"Drums"}]'
]
];
//Time to get the tableclass...
$albums = TableRegistry::get('Albums');
//Time to transform our array into Album Entities
$entities = $albums->newEntities($all_records);
//Now, we have transformed our array into entities on $entities, this is the variable we need to save
if(!$albums->saveMany($entities))
{
echo "FellsBadMan";
}
else
{
echo "FellsGoodMan";
}
You can read more about here